HBF, aliases High Bandwidth Flash, uses the aforesaid flash representation exertion we spot successful SSDs today. Unlike SSDs, HBF is implemented overmuch for illustration HBM (High Bandwidth Memory). HBF cubes beryllium connected the aforesaid package arsenic a compute chip, possibly moreover adjacent to HBM. HBF’s thought is to connection overmuch higher capacity than HBM, while still providing decent representation bandwidth. At Hot Chips 2026 tutorials day, Anurag Agarwal and Radhakrishna Giduthuri’s talk explores really HBF could use to instrumentality learning workloads. No HBF products beryllium yet, truthful the talk focuses connected simulations, projections, and really package tin accommodate to return advantage of HBF.
Even though HBF uses a HBM-like shape factor, it’s wholly different nether the hood. It’s not for illustration Intel’s Optane, which could usability arsenic different excavation of memory. Instead, HBF is almost for illustration a SSD integrated onto a processor. Software uses DMA to move information betwixt HBF and DRAM. HBF accesses must beryllium done successful large, aligned chunks arsenic if it were a wide retention device, alternatively than strategy memory. Host package besides has to return connected SSD controller functions for illustration managing constitute leveling and ensuring information retention. That intends HBF can’t beryllium a plug-and-play solution.
Instead, taking advantage of HBF intends formulating a typical strategy and implementing it into a runtime. Giduthuri takes vLLM arsenic an example. vLLM typically holds exemplary weights successful GPU memory, and is already exploring options to trim VRAM usage. For example, vLLM is investigating putting exemplary weights successful pinned CPU representation provided the big has tons of free memory. While that wouldn’t activity for HBF because HBF doesn’t support fine-grained random access, different options mightiness person promise. For example, MoE experts tin beryllium stored successful HBF. Software tin DMA progressive experts into HBM arsenic needed.
vLLM’s KV cache tin besides beryllium placed into HBF. However, that whitethorn only activity good successful a sparse attraction implementation thatonly sounds a subset of tokens disconnected the apical of the KV cache for each step. That allows astir of the KV cache to beryllium “cold” successful flash, which takes advantage of HBF’s capacity while placing little unit connected HBF’s little bandwidth. A imaginable caveat is that the top-k publication is scattered, while HBF prefers sequential reads. Perhaps package tin get astir this by DMA-ing the top-k rows into DRAM arsenic needed.
Another opportunity is utilizing HBF capacity to trim cross-device communication. Large models are often sharded crossed aggregate GPUs, which results successful capacity being bound by cross-device scatter and stitchery operations. Cross-device connection tin go a bigger capacity obstruction than compute throughput aliases representation bandwidth. HBF tin mitigate this by replicating much of a model’s weights crossed different GPUs. DMA-ing information disconnected flash isn’t cheap, but it’s cheaper than going off-device.
Agarwal went complete erstwhile HBF makes consciousness from a costs perspective. Basically, HBF is bully if a workload doesn’t scope its bandwidth limits. That applies pinch smaller models and/or smaller batch sizes. If a workload becomes bandwidth bound, it’s bad for HBF’s costs equation because some costs per capacity and costs per bandwidth facet into last cost. HBF is awesome for costs per capacity, but is worse successful costs per bandwidth compared to HBM.
He besides discussed utilizing HBM to cache basking experts, but that besides seems for illustration a difficult solution. Caching needs to activity retired well, aliases HBF bandwidth tin propulsion a wrench into the useful for the cost-per-token equation.
HBF whitethorn alleviate the DRAM capacity problem to immoderate extent, but the package challenges consciousness immense. Handling HBF sounds a batch for illustration moving pinch a debased level disk entree API, for illustration utilizing FILE_FLAG_NO_BUFFERING successful Windows aliases O_DIRECT successful Linux. Software has to transportation retired accesses successful large, aligned chunks alternatively than freely addressing retention pinch byte-level granularity. Modifying a azygous byte tin mean reference a ample 64 KB artifact into DRAM, modifying that block, and penning the full artifact backmost to flash. That’s much for illustration moving pinch a artifact retention instrumentality than moving pinch memory. A package model meant to activity pinch a regular DRAM-based strategy will request monolithic changes to leverage HBF. Moving to a different model will mean re-doing the activity needed to return advantage of HBF.
I would spell arsenic acold arsenic saying that the effort required to leverage HBF doesn’t look acold disconnected what would beryllium required to consecutive up trim DRAM usage by streaming exemplary weights disconnected a SSD. Taking advantage of SSD really seems easier. The OS kernel tin absurd distant the trouble of doing block-aligned accesses if you don’t usage FILE_FLAG_NO_BUFFERING aliases O_DIRECT,. Buffering successful the kernel will fto package arbitrarily activity and transportation retired byte-level read/write operations. It’ll besides enactment arsenic a cache, people insulating package from flash representation inefficiencies. While not mentioned successful the talk, I wonderment if existing attempts to watercourse exemplary weights disconnected SSDs can beryllium applied to HBF. Or, if the package challenges associated pinch utilizing HBF beryllium excessively steep and forestall its adoption. I conjecture we’ll spot when/if HBF products deed the market. I want to spot thing that’ll alleviate the existent DRAM shortage, but I’m not judge if HBF is it.
English (US) · 
Indonesian (ID) · 
