High-Performance Array-Backed LRU Hash Table

Concurrent LRU Hash Table optimized for:

• multi-core scalability
• predictable tail latency
• NUMA architectures
• zero runtime allocations

1. The Problem: The Standard Library Bottleneck
2. The Solution: Core Architecture & Algorithms
3. Benchmarks & Scaling Performance
4. When This Table May Not Be the Best Fit
5. Quick Start API Overview
6. Project Structure
7. Building trial code
8. Conclusion & Future Hardware Extrapolation
9. License & Contributing

A high-performance concurrent LRU hash array designed for demanding systems programming workloads specified arsenic caching layers, web infrastructure, and kernel components. By leveraging shard-based parallelism and cache-friendly representation layouts, the implementation delivers precocious throughput successful environments wherever modular room containers degrade nether contention.

• Zero Runtime Allocations: Pre-allocated level arrays destruct heap fragmentation and OS-level fastener stalls.
• Custom TTAS Spinlocks: Replaces std::shared_mutex to destruct OS discourse switches, achieving 14x+ throughput and sub-microsecond tail latencies. (Note: User-mode uses a civilization TTAS Spinlock for earthy speed, while Kernel-mode relies connected EX_PUSH_LOCK).
• Sharded Architecture: Eliminates world fastener convoys, scaling linearly pinch beingness CPU halfway counts.
• NUMA-Aware Memory: Distributes shard allocations crossed beingness CPU sockets to maximize representation controller bandwidth.
• Lock-Free Destruction: Payloads are explicitly destroyed extracurricular the synchronization boundary, ensuring level tail latencies.
• Lazy LRU Promotion: A tunable "Safe Zone" bypasses exclusive fastener upgrades connected basking reads, yielding an ~20% throughput boost.
• Custom Allocators (User-Mode): Supports template-injected allocators for domain-specific representation management.
• Dual Environment Ready: Full cross-platform user-mode support alongside a dedicated Windows 10+ Kernel implementation (IRQL < DISPATCH_LEVEL).

The implementation prioritizes mechanical sympathy, cache locality, fastener scalability, and predictable representation behavior, making it suitable for demanding environments specified as:

• High-Frequency Trading (HFT) infrastructure
• Storage subsystem caches
• Real-time web routing
• Kernel / driver components
• High-throughput web servers

The implementation provides O(1) average-time operations for insertion, lookup, and removal while maintaining a strict aliases probabilistic Least Recently Used (LRU) eviction policy.

Typical concurrent LRU implementations (e.g., combining std::unordered_map + std::list protected by a world std::shared_mutex) suffer from terrible architectural flaws on modern high-core-count CPUs:

1. Global Lock Contention: A azygous fastener creates a catastrophic "lock convoy," wherever adding threads really decreases full throughput.
2. Pointer Chasing: Node traversal crossed the heap destroys L1/L2 cache locality.
3. Allocator Overhead: Every insertion/eviction triggers heap allocation/deallocation (new/delete), resulting successful representation fragmentation and OS-level fastener stalls.
4. False Sharing: Unaligned representation structures origin adjacent CPU cores to invalidate each other's L1 cache lines, silently destroying performance.

This task solves the modular room bottlenecks done a operation of sharding, flat-array representation management, and lock-free demolition techniques.

This sketch illustrates the architecture of a LRU hash array that eliminates world fastener contention by partitioning information into independent, cache-aligned shards. Each shard operates autonomously pinch its ain exclusive TTAS spinlock, metadata counters, and a contiguous "Mega-Block" of representation containing the bucket and node arrays. Within these arrays, some the hash collision chains and the doubly-linked LRU queues are constructed utilizing 32-bit array indices alternatively than modular 64-bit pointers, which halves the structural memory overhead and improves L1/L2 cache locality during hot-path operations.

At a precocious level, the array is walled into independent shards, each managing its ain hash array and LRU chain:

To maximize L1/L2 cache deed rates, the soul node building explicitly separates information based connected entree wave during traversal:

The Hot Path (First Cache Line): Variables captious for navigating collision chains and verifying matches (Hash, HashNext, LruPrev, and the Key) are tightly packed into the first hardware cache statement (64 bytes aliases 128 bytes depending connected architecture). This ensures that the CPU tin scan heavy hash buckets successful a azygous representation fetch without triggering costly main-memory stalls.

The Cold Path: Variables required only aft a successful cardinal lucifer aliases during an eviction (Value*, LastPromoted, LruNext) are pushed disconnected to secondary cache lines. This guarantees that the representation controller ne'er wastes bandwidth fetching payload pointers aliases property metrics for nodes that are simply being passed complete during a lookup scan.

The array is divided into independent, isolated shards. Each shard contains its ain hash buckets, LRU list, spinlock, and capacity limits.

• Shard count is dynamically scaled based connected processor topology (shards ≈ CPU cores × 32).
• Threads are routed utilizing a MurmurHash3-style avalanche mixer (MixHash) to unit entropy into the little bits. This ensures azygous shard distribution nether emblematic hash quality regardless of the value of the user-provided hash function: shard = MixHash(hash) & (ShardCount - 1).
• This ensures azygous workload distribution and avoids world fastener contention by design.

Instead of allocating nodes individually connected the heap, each nodes and buckets are pre-allocated successful contiguous level arrays (Mega-Blocks).

• Zero Runtime Allocations: Once initialized, the array ne'er calls caller aliases delete.
• Relative 32-bit Indices: Linked lists (LRU chains and Hash collisions) are implemented utilizing 32-bit array indices alternatively of 64-bit pointers. This cuts the structural representation overhead successful half and dramatically increases the number of nodes that fresh wrong the CPU's L1/L2 cache.
• NUMA Awareness: The user-mode array utilizes VirtualAllocExNuma (Windows) aliases libnuma (Linux) to administer shard allocations evenly crossed beingness CPU sockets, maximizing representation controller bandwidth.

Memory layout is strictly controlled to respect hardware-specific cache alignment, i.e., 64 bytes connected x86_64 and ARM64, and 128 bytes connected Apple silicon.

False Sharing Prevention:

Shards are explicitly padded to hardware-specific cache statement boundaries (64-byte aliases 128-byte). A thread locking Shard A will ne'er invalidate the cache statement for a thread accessing Shard B.

Hot/Cold Path Struct Packing:

Variables required for hash traversal packed into the first information of the first cache line. The CPU fetches these together successful a azygous read, guaranteeing a cache deed during deep collision concatenation probing.

• Out-of-Lock Destruction: Deadlocks and latency spikes are avoided by guaranteeing that personification codification ne'er executes wrong the synchronization boundary. Evicted nodes are detached, the fastener is dropped, and only past is the payload destructed/released.

• Lazy LRU Promotion (Generation Counter): Traditional LRUs beforehand items to the MRU caput connected each read, requiring an exclusive write-lock. This implementation uses a probabilistic Generation counter. If a publication hits a "hot" item, the promotion is skipped, allowing the thread to complete the publication instantly.

• Optional Proactive Trimming: The array is afloat autonomous; erstwhile a shard reaches capacity, Add() automatically performs inline LRU eviction to make room. Therefore, a dedicated background trimming thread is not required for continuous operation. However, to guarantee ultra-flat P99/P99.9 tail latencies connected your foreground basking path, you tin optionally invoke Trim() during comparatively idle cycles aliases from a inheritance worker. Proactively trimming progressive items down to a little watermark (e.g., 85%) ensures foreground insertions consistently hit warm, pre-allocated free nodes alternatively than paying the structural execution costs of inline eviction.

The Array-backed array replaces std::shared_mutex pinch a civilization Spinlock designed specifically for microscopic captious sections. It implements the TTAS shape to strictly forestall MESI protocol autobus floods ("Cache Line Bouncing") connected multi-socket / multi-core systems.

Supported Spin Policies: To accommodate different execution environments, the fastener behaviour is injected astatine compile-time:

• AdaptiveSpinPolicy (Default): Maximizes throughput by spinning concisely successful user-space, falling backmost to a forced OS deschedule to forestall deadlocks during terrible contention.

• ExponentialBackoffPolicy (Opt-in): Implements a dynamic, self-tuning backoff strategy for high-contention environments. Instead of polling the fastener astatine a changeless rate, waiting threads double their hardware region batches (1, 2, 4... up to MAX_BACKOFF_PAUSES) aft each grounded attempt.

To debar synchronization overhead connected mini information sets, the implementation automatically scales down progressive shards for smaller capacities, enforcing a minimum of 64 items per shard.

This ensures that mini tables do not suffer unnecessary fastener aliases memory fragmentation costs while still preserving the aforesaid API and behavior.

To beryllium the architecture, the Custom Array Table was benchmarked against the modular implementation (std::unordered_map + std::list) crossed 3 chopped hardware topologies:

Intel Core i7-1165G7 (4 Cores / 8 Threads, Low Power) Intel Core i7-8086K (6 Cores / 12 Threads, High Clock) Intel Core i7-12700H (14 Cores / 20 Threads, Big.LITTLE)

Workload: 1,000,000 capacity, Mix of Add/Lookup/Remove/Trim, 0% Safe Zone. Multi-Threaded Scaling (The "Lock Convoy" Collapse)

The Array-Backed array scales positively pinch beingness hardware, whereas the modular room implementation exhibits antagonistic scaling nether dense mixed contention.

At utmost percentiles, the Array-Backed array maintains low-microsecond latency, bypassing the terrible latency spikes characteristic of modular OS-mediated locks.

At the 99.9th percentile, we measurement the worst-case algorithmic contention (e.g., heavy hash collisions aliases fastener upgrades). The civilization array maintains low-microsecond latency.

At the 99.99th percentile (P99.99), we observe the existent costs of OS-mediated locking.

(Note: The jump to 20.6µs connected the i7-12700H astatine P99.99 is the hardware signature of the OS Thread Director migrating a thread betwixt a P-Core and an E-Core, forcing an L1/L2 cache flush).

Under dense mixed workloads (simultaneous reads, writes, and evictions), the architectural differences create a compounding capacity gap. As halfway counts increase, the modular array loses throughput owed to contention, while the sharded Array-Backed array accelerates

Strict LRU caches suffer nether dense publication contention because each publication requires an exclusive fastener upgrade to update the LRU head.

By utilizing a Generation counter, the cache probabilistically "ages" items. If an point is accessed but hasn't aged past the threshold, the promotion is skipped, allowing the thread to complete the publication instantly. By utilizing a microscopic exclusive fastener without the overhead of reader-to-writer upgrade hazards, it achieves higher throughput than traditional Reader-Writer fastener implementations.

Workload: 95% Read / 5% Write connected the 6-Core i7-8086K.

On Linux guests successful virtualized environments (VMware, AWS, Azure), spinning vCPUs tin trigger VM-Exit storms if the lock-holder is preempted by the hypervisor (LHP). Standard room locks (std::shared_mutex) suffer severely from this owed to dense OS discourse switching.

To combat this, the array relies connected its civilization spinlocks. Both disposable policies efficaciously mitigate LHP compared to the modular library, but their capacity characteristics vary significantly depending connected the impermanent Linux distribution and its underlying CPU scheduler:

Benchmark: Linux Guests connected VMware (Windows 10 Host, 4 vCPUs)

Testing crossed different distributions reveals that the optimal fastener strategy is highly limited connected the impermanent OS:

• Ubuntu: The ExponentialBackoffPolicy offers somewhat amended P99.9 tail latency stability, trading highest scaling throughput for stricter latency bounds.
• Fedora: The AdaptiveSpinPolicy outperforms the yielding attack crossed the board, delivering amended P99.9 stableness and P99.99 tail latency.

Recommendation for Cloud Deployments: Since hypervisor configurations and Linux CPU schedulers (CFS aliases EEVDF) respond otherwise to userspace spinning versus difficult yielding, do not blindly default to the yielding argumentation connected Linux. It is highly recommended to floor plan some policies connected your circumstantial target OS and hypervisor operation to find which yields the champion tail latency for your workload.

How to Toggle Policies (Linux Only): Windows handles LHP natively, truthful this mitigation is strictly for Linux deployments. By default, the build uses AdaptiveSpinPolicy. You tin opt into the yielding lock utilizing conditional compilation during the build step:

All benchmarks were produced connected Windows 10/11 utilizing the test_um suite included successful this repository, compiled pinch Microsoft Visual Studio 2026 utilizing AdaptiveSpinPolicy SpinLock hold policy.

While this architecture excels nether dense concurrent workloads, it is not a metallic bullet. In immoderate scenarios, a modular room creation (such arsenic std::unordered_map + std::list) may beryllium the much due choice:

• Extremely Small Tables (< ~100 items): While the array internally reduces shard counts erstwhile capacity is beneath 1024 entries, the baseline overhead of avalanche hashing, atomic reference counting, and shard routing tin predominate connected microscopic datasets. In these cases, a elemental STL-based LRU protected by a std::mutex is often faster.
• Strictly Single-Threaded Workloads: This array is explicitly designed to lick multi-threaded locking bottlenecks. In purely single-threaded environments, a modular STL-based LRU may outperform it. The modular containers are highly optimized for uncontended execution, whereas the Array-Backed array still incurs the fixed overhead of atomic operations, representation barriers, and reader-writer fastener acquisitions.
• Highly Memory-Constrained Environments: To execute zero runtime allocations and forestall OS fastener stalls, this array pre-allocates level "Mega-Blocks" for its full maximum capacity upfront. If your situation cannot spend to pre-allocate the maximum imaginable representation footprint, you must usage a accepted node-based instrumentality that allocates representation connected demand.

Values must inherit aliases instrumentality an intrusive reference counting interface (AddRef() and Release()).

The repository is organized into chopped layers to abstracted the halfway hash array logic from the environment-specific wrappers and trial suites:

• km/: Contains the Kernel-Mode LRU Hash array implementation.

  • LRUHashTable.h: The superior header for usage successful Windows Driver environments.

• um/: Contains the User-Mode LRU Hash array implementation.

  • lru_hash_table.h: The cross-platform header for Linux, macOS, and Windows applications.
  • lru_string_key.h: High-performance string-based cardinal implementation.

• test_common/: Shared trial logic utilized by some kernel and user-mode capacity tests.

  • std_lru_hash_table.h: A wrapper for modular room comparisons.
  • test_lru_hash_common.h: Shared capacity tests and validation logic.

• test_km/: Test codification for verifying kernel-mode logic wrong a user-mode capacity trial harness.

  • TestLRUHashKm.cpp: The driver-logic trial harness, strictly requiring MSVC 2022/2026 connected Windows.

• test_um/: Cross-platform user-mode capacity trial suite.

  • test_lru_hash.cpp: The superior benchmark and validation instrumentality utilized connected Linux, macOS, and Windows.

• test_drv/: Actual Windows Kernel driver capacity trial codification for deployment connected target systems.

  • TestLruDrv.cpp: Windows Kernel driver capacity trial code

• sample_um/ : User-mode sample code.

  • sample_lru_hash.cpp: User-mode C++ sample showing really to usage businesslike string-based cardinal pinch LRU Hash table.

Requires a C++20 compliant compiler (GCC aliases Clang).

Linux Dependencies: The Linux user-mode implementation utilizes libnuma to hindrance shard allocations to beingness CPU sockets, mimicking the representation controller routing of the Windows kernel implementation. You must instal the NUMA improvement headers earlier building:

Using the Build Script (Recommended)

./build.sh [--clean | -c] [--hybrid] [--type Release | Debug] [--compiler g++ | clang++]

Using CMake The included CMakeLists.txt automatically detects your platform, handles libnuma linking connected Linux, and configures optimized build flags.

Manual Compilation If you for illustration building without CMake, guarantee you see the -pthread and -lnuma (Linux only) flags for due linking.

Optimizing for Linux Guest VMs: Cross-platform thread scaling wrong virtual machines often hits bottlenecks owed to impermanent OS scheduler behaviour and modular room allocator contention. This implementation addresses these bottlenecks and improves multi-threaded scaling done 2 circumstantial mitigations:

Hard vCPU Yielding: Standard sched_yield() (via std::this_thread::yield()) is often treated arsenic a no-op by the Linux CFS scheduler erstwhile nary different threads are waiting connected that specific vCPU. This causes "busy-spins" that trigger VM-Exit storms. The civilization ExponentialBackoffPolicy mitigates this by forcing a difficult discourse move via a 500-nanosecond nanosleep once the exponential backoff period is met.

Allocator Contention (jemalloc): The default glibc malloc heavy throttles concurrent allocations. To forestall allocator bottlenecks connected Linux, it is highly recommended to link against jemalloc. This replaces the contention-heavy strategy heap pinch a sharded, lock-free allocation strategy that complements the table's soul sharding.

Recommended Step: Running the Benchmark connected Linux To guarantee the trial suite tin accurately benchmark tail latencies, use NUMA node affinity, and negociate thread priorities without requiring afloat guidelines privileges, it is highly recommended to assistance the test_lru_hash executable the CAP_SYS_NICE capacity earlier execution.

Native Visual Studio Solution (.slnx/.sln) and Project (.vcxproj) files are included successful the repository.

User-Mode Build utilizing Visual Studio 2022 aliases later.

• Note: If building pinch Visual Studio 2022, you must manually alteration the Platform Toolset to v143 successful the task properties.
• Requires C++20.
• NUMA support is handled natively via VirtualAllocExNuma.

Kernel-Mode: Build utilizing Visual Studio 2022 and the Windows Driver Kit (WDK 11).

• Requires C++17.
• The implementation utilizes EX_PUSH_LOCK and performs NUMA-aligned allocations via ExAllocatePool3.
• IRQL Restriction: Since the synchronization bound uses push locks (which run astatine <= APC_LEVEL), the existent kernel implementation can only beryllium utilized astatine IRQL < DISPATCH_LEVEL (i.e., PASSIVE_LEVEL aliases APC_LEVEL). It is not safe for usage wrong DPC routines aliases hardware interrupt handlers.

Per Amdahl's Law, modular global-lock LRU implementations are heavy constricted by their sequential synchronization overhead, inevitably starring to fastener convoys nether dense contention. By utilizing a sharded architecture to isolate synchronization, Array-Backed array minimizes that sequential fraction, allowing throughput to standard positively pinch parallel load.

Based connected this mechanical sympathy, the capacity advantage of this architecture will go moreover much pronounced connected modern hardware topologies:

• Massive L3 Caches (e.g., AMD 3D V-Cache / Server CPUs): Utilizing 32-bit array indices halves the structural representation footprint. This allows importantly larger moving sets to reside within L3 SRAM, deferring main-memory latency penalties.

• High Parallelism & NUMA (Threadripper / EPYC / Xeon): Avalanche hashing and NUMA-aware beingness representation allocation evenly disperse workloads crossed beingness sockets. This sustains near-linear scaling good past the period wherever modular implementations degrade.

• Symmetric Core Scaling & Cache Coherency (Ryzen / Threadripper): At utmost halfway counts, the "Lazy Promotion" optimization skips exclusive fastener upgrades connected basking reads. This keeps targeted cache lines successful the Shared (S) authorities wrong the MESI protocol, allowing aggregate cores crossed different chiplets to cache the aforesaid nodes locally without triggering cross-die invalidation traffic.

This task is licensed nether the Apache License, Version 2.0.

You whitethorn not usage this record isolated from successful compliance pinch the License. You whitethorn get a transcript of the License at: http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable rule aliases agreed to successful writing, package distributed nether the License is distributed connected an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either definitive aliases implied. See the LICENSE record for the circumstantial connection governing permissions and limitations.