In-memory compute has been an charismatic proposition for galore years because compute wrong a representation spot tin utilization its higher soul bandwidth. Additionally, in-memory compute avoids the agelong latency way betwixt DRAM and accepted compute cores. At Hot Chips 2026, Samsung discusses their continued pursuit of in-memory compute pinch their PIM (Processing-in-Memory) push. They’re implementing MAC units wrong LPDDR5X chips, while preserving the chip’s expertise to interface pinch a modular representation controller.
DRAM chips are internally divided into banks, each pinch their ain publication and constitute logic. During a normal DRAM access, the representation controller selects a bank, activates a statement wrong it, and past accesses information via file entree strobe (CAS) commands. Bandwidth is constricted by the chip’s outer DRAM interface. Even if the representation controller could activate each of the banks simultaneously, it wouldn’t beryllium capable to get its hands the afloat bandwidth disposable crossed each the banks.
Samsung’s LPDDR5X-PIM is for illustration a normal LPDDR5X-9600 spot pinch 16 banks, but places a PIM (Processing-in-Memory) artifact astatine each bank. These PIM blocks entree their attached DRAM slope without being constrained by the chip’s outer bus. Together, they tin utilize the chip’s soul bandwidth crossed each 16 banks, which comes retired to 614 GB/s. For comparison, regular DRAM accesses tin deed 2 banks successful parallel and max retired astatine 76.8 GB/s.
PIM blocks internally dwell of a MAC character pinch surrounding registry files and power logic. A 1024-bit instruction registry record holds up to 64 16-bit instructions. A 4 kbit root registry record is meant for activation vectors, and supplies 1 root operand for the MAC array. Samsung expects package to load exemplary weights into DRAM, truthful the attached DRAM artifact supplies the 2nd operand. Model weights tin beryllium scaled earlier the MAC computation, pinch standard factors coming from a 2 kbit standard register.
The PIM block’s MAC array supports a assortment of debased precision formats. Numbers from Samsung’s position propose each PIM block’s MAC array tin prolong 4 INT8 aliases FP8 MAC operations per information clock, aliases 8 per rhythm erstwhile not counting the double information rate. Throughput doubles for 4-bit input weights, bringing package-wide compute throughput to 2.4 TOPS.
This isn’t a very precocious figure, but an implementation pinch galore LPDDR5X chips will person higher aggregate throughput. For example, 8 LPDDR5X chips together would person 9.6 INT8 TOPS, which conscionable astir matches the NPU successful Intel’s Meteor Lake. That would besides beryllium an costly setup, because 8 16 GB LPDDR5X chips would correspond to 128 GB of strategy memory.
One item of LPDDR5X-PIM is that it stays wrong the modular LPDDR5X protocol while exposing compute capabilities that aren’t portion of the representation standard. Samsung achieves this by mounting speech typical statement addresses, which enactment for illustration MMIO addresses of sorts. Each transmission has a brace of predefined rows for mode control. Activating 1 of those rows sets the spot to single-bank mode, while the different sets the spot to multi-bank mode. Single-bank is the regular mode, while multi-bank applies commands crossed each 16 banks to utilization the chip’s soul bandwidth.
Special per-bank rows alteration really publication and constitute commands behave. Activating 1 of these typical rows makes publication and constitute commands entree PIM registers alternatively of regular DRAM slope contents (PIM Registers Activated mode). Samsung envisions a ML usage lawsuit wherever package loads exemplary weights into DRAM while the spot is successful normal single-bank mode. Then, package switches into multi-bank mode and enters PIM Registers Activated mode. This lets codification constitute activation values into PIM root registers, group standard factors successful PIM standard registers, and specify an cognition that’s filled into PIM instruction registers.
Because the spot is successful multi-bank mode, each PIM registry constitute gets broadcast crossed each 16 banks. PIM compute truthful useful for illustration a very constrained SIMD processor, wherever the operation, standard factor, and 1 root operand are the aforesaid crossed each banks. Samsung does let penning PIM registers successful single-bank mode, but that functionality is meant for debugging purposes. Each DRAM packet is 256 bits (BL=16) Filling each root registry takes 16 constitute commands. Doing that 1 slope astatine a clip crossed each of the 16 banks would mean 256 constitute commands, turning big to PIM registry constitute bandwidth into the limiting factor. Samsung really allows PIM registry entree successful azygous slope mode, but that’s intended arsenic a debugging feature.
After priming PIM registers, package switches backmost into multi-bank mode and issues publication commands. Instead of reference DRAM contents, these publication commands initiate computations and get results accumulated into PIM vector registry files. Then, constitute commands show PIM blocks to constitute VRF contents backmost into the DRAM banks.
PIM has to grip reordering that a normal representation controller mightiness transportation out. When codification sets up PIM by activating the bank, PIM conventionally sets up its instruction registry files truthful that instructions sequentially entree each root registry element. For instance, the first instruction would reference the first root registry element, the 2nd instruction would reference the 2nd root registry element, and truthful on. However, that falls isolated if the representation controller reorders accesses. Samsung gets astir this pinch an Address Align Mode (AAM), which makes each instruction infer its root registry scale from the file reside being accessed.
When the big finishes utilizing in-memory compute and wants to publication results, it switches the DRAM spot backmost into single-bank mode. Then, regular DRAM sounds and writes will commencement accessing DRAM contents arsenic normal.
Samsung internally achieved immense capacity gains erstwhile taking advantage of LPDDR5X-PIM, compared to utilizing modular LPDDR5X. The chip’s expertise to run pinch a modular representation controller is impressive, and Samsung has been very imaginative successful really they approached the problem.
Repurposing modular DRAM commands should simplify hardware, but package challenges look steep. Because PIM modes alteration the meaning of DRAM entree commands, package can’t usage PIM and transportation retired regular representation accesses astatine the aforesaid time. That applies moreover crossed threads, because representation controllers and DRAM chips are oblivious to what thread an entree is for. If a non-PIM thread sounds from representation while different is utilizing PIM, the first thread could origin an unintended computation and get incorrect results into the PIM VRFs. A constitute from the non-PIM thread could origin PIM blocks to constitute VRF information backmost to the incorrect address.
Samsung deals pinch this by having the big isolate a PIM region successful memory. I can’t deliberation of an easy measurement to do this successful a emblematic strategy without compromising representation bandwidth and PIM performance. Hardware usually interleaves addresses crossed channels, which lets communal entree patterns people utilize bandwidth crossed those channels. PIM uses per-channel rows to power single/multi-bank mode changes, truthful dropping interleaving and designating representation channels arsenic PIM-only would beryllium the only reasonable measurement to create a PIM region. Then, non-PIM applications wouldn’t beryllium capable to return advantage of bandwidth from channels reserved for PIM. PIM codification would miss retired connected bandwidth and compute from non-PIM channels. The second could beryllium a important rumor because per-chip compute throughput isn’t that high.
Multitasking issues could persist moreover aft isolating a PIM region. If an exertion wants to usage PIM and return advantage of multithreading, it would person to defender PIM region accesses pinch locks to forestall cases wherever 1 thread tries to do PIM compute while different attempts regular representation accesses. Things get moreover worse pinch a modern multitasking operating system, wherever aggregate processes could effort to usage PIM without being alert of each other. I’m not judge there’s a bully measurement to grip that too making the operating strategy tally PIM compute codification segments pinch each different threads blocked and interrupts disabled. Handling interrupts aliases discourse switches pinch PIM feels for illustration a nightmare for the OS successful immoderate case. Preempting a PIM thread would mean bringing the representation transmission retired of PIM mode and redeeming PIM state. The OS would person to publication retired instruction, source, scale, and vector registry record crossed each slope and prevention it somewhere. Only allowing a azygous moving thread pinch nary task switching would time off multithreaded capacity connected the table, and could lead to strategy responsiveness issues if codification spends excessively agelong successful PIM compute sections.
PIM compute breaks a representation subsystem’s expectations astir DRAM behaviour because DRAM tin make representation values that the cache level ne'er knows about. Caches tin besides break PIM behaviour by absorbing accesses meant to trigger PIM operations. Samsung truthful recommends mapping PIM representation arsenic uncacheable. That’s problematic because modern CPUs and GPUs trust heavy connected caching to mitigate DRAM latency. Performance connected uncacheable representation will beryllium highly slow because the CPU aliases GPU cores will walk acold much clip stalled waiting connected memory.
Skipping caches isn’t the only problem. PIM sounds enactment for illustration MMIO accesses because they origin computations that impact PIM VRF values, alternatively than conscionable retrieving data. CPUs besides mitigate representation latency by initiating loads earlier they cognize that load information will really beryllium needed. Branch prediction lets CPUs rumor instructions earlier the halfway knows for definite that those instructions will beryllium executed. Prefetchers observe representation entree patterns and effort to load information into cache earlier instructions petition that data. If the CPU loads information that turns retired to unneeded later on, that’s good because loads usually won’t origin incorrect programme behavior. Unfortunately that’s not existent pinch PIM, wherever sounds trigger computations that modify PIM VRF contents.
Working pinch a PIM region will apt mean making representation accesses non-speculative arsenic good arsenic non-cacheable. Running a CPU without caching, prefetching, aliases out-of-order execution will cripple performance.
Setting speech PIM mode difficulties, in-memory compute poses precocious level challenges for software. Each PIM artifact only has accelerated entree to its locally attached DRAM bank. All different input information has to beryllium brought successful done the DRAM chip’s comparatively constrained outer interface. PIM blocks can’t straight speech information pinch each other, truthful the big has to move information utilizing regular DRAM sounds and writes if 1 PIM artifact needs to usage results generated by another.
Samsung’s LPDDR5X-PIM tin theoretically spell into immoderate server, desktop, laptop, aliases moreover mobile instrumentality acknowledgment to its expertise to activity pinch modular representation controllers. However, that doesn’t mean it’ll beryllium easy to usage pinch emblematic hardware and package paradigms. PIM mode switching throws a wrench into the useful for multitasking operating systems. Modifying DRAM contents nether the hood and attaching broadside effects to publication commands breaks CPU caching, prefetching, and out-of-order execution.
I don’t deliberation there’s an easy measurement to usage in-memory compute without changes passim the representation subsystem. For example, thing for illustration should make package take easier:
Expand the DRAM interface to adhd a group of compute commands, avoiding mode move complexity
Have the representation controller enactment for illustration a adjacent CPU halfway from a cache coherency perspective. Before utilizing in-memory compute commands, the representation controller issues read-for-ownership (RFO) requests for each affected cache lines. That lets the representation controller get immoderate modified information and constitute it backmost to DRAM earlier starting in-memory compute, ensuring that in-memory compute results bespeak the latest CPU-side writes. Then, the representation controller holds ownership of affected cache lines until in-memory compute operations complete, letting CPU cores observe in-memory compute results without needing to invalidate aliases bypass caches
Add a caller group of CPU instructions for illustration “rep macb” that execute multiply-accumulate operations complete a artifact of representation pinch fixed multiplicand/scale factors and undefined numerical characteristics. The CPU tin take whether to usage in-memory compute (if supported by DRAM) aliases make a series of soul ops (if operating complete a mini group of information that’s already successful cache).
With those hardware changes, package would beryllium capable to usage in-memory compute from a multitasking operating strategy without reserving representation aliases losing thread-level parallelism to PIM-related locks and synchronization. A transparent CPU instruction avoids the problem of shipping hardware circumstantial binaries, and allows forward-compatible codification that automatically takes advantage of caller hardware capabilities including different in-memory compute implementations. It besides lets hardware usage implementation-specific knowledge and real-time information (like a no-fill-on-miss cache lookup) to make the champion determination astir wherever to transportation retired compute. I don’t for illustration the package replacement of reserving representation regions, marking them uncacheable, and blocking threads. There’s conscionable excessively galore tradeoffs astir performance, representation capacity, and responsiveness.
English (US) · 
Indonesian (ID) · 
