Turbovec – Google's TurboQuant for vector search in Rust

A 10 cardinal archive corpus takes 31 GB of RAM arsenic float32. turbovec fits it successful 4 GB - and searches it faster than FAISS.

turbovec is simply a Rust vector scale pinch Python bindings, built connected Google Research's TurboQuant algorithm — a data-oblivious quantizer pinch near-optimal distortion and nary abstracted training phase.

• Online ingest. Add vectors, they're indexed — nary train step, nary parameter tuning, nary rebuilds arsenic the corpus grows.
• Fast SIMD search. Hand-written kernels — NEON SDOT/SMMLA connected ARM, AVX-512 VNNI and vpermb connected x86, pinch AVX2 and scalar fallbacks — hit FAISS IndexPQFastScan successful each measured config, averaging 3.4× astatine 4-bit and 23% astatine 2-bit crossed the 8 cells of each width, connected some architectures.
• Incremental saves. sync(path) persists conscionable what changed since the past sync — 1 fsync per call, crash-safe astatine immoderate byte, and a removal aliases a mini append costs milliseconds nevertheless ample the index. write/load enactment for whole-file snapshots.
• Filter astatine hunt time. Pass an id allowlist (or a slot bitmask) to search() and the kernel honours it directly. You ever get up to k results from the allowed group — nary over-fetching, nary callback deed connected selective filters.
• Pure local. No managed service, nary information leaving your instrumentality aliases VPC. Pair pinch immoderate open-source embedding exemplary for a afloat air-gapped RAG stack.

Building RAG wherever privacy, memory, aliases latency matters? You're successful the correct place.

vectors and query are 2-D float32 arrays of style (n, dim) — different dtypes are rejected alternatively than silently converted, truthful formed pinch np.asarray(x, dtype=np.float32) first if needed.

Need unchangeable ids that past deletes? Use IdMapIndex:

Restrict results to a campaigner group produced by different strategy (SQL, BM25, ACL, clip window, …):

Filtering happens wrong the SIMD kernel astatine 32-vector artifact granularity: blocks pinch nary allowed slots are short-circuited earlier immoderate LUT lookup aliases scoring work, and individual non-allowed slots wrong scored blocks are dropped astatine heap-insert. Selective allowlists (small fraction of the scale allowed) truthful debar astir of the SIMD costs alternatively than paying it and discarding the consequence afterwards.

The output magnitude is min(k, n_allowed), wherever n_allowed counts distinct allowed vectors — erstwhile less vectors are allowed than k you get precisely that galore results alternatively than padded fallbacks.

See docs/api.md for the afloat reference.

Drop-in replacements for the in-tree reference vector / archive stores successful each framework. Same nationalist surface, aforesaid persistence semantics, aforesaid retriever and pipeline wiring — switch the import and support your pipeline.

• LangChain — pip instal turbovec[langchain] · replaces langchain_core.vectorstores.InMemoryVectorStore
• LlamaIndex — pip instal turbovec[llama-index] · replaces llama_index.core.vector_stores.SimpleVectorStore
• Haystack — pip instal turbovec[haystack] · replaces haystack.document_stores.in_memory.InMemoryDocumentStore
• Agno — pip instal turbovec[agno] · replaces agno.vectordb.lancedb.LanceDb

For unchangeable outer ids that past deletes:

TurboQuant vs FAISS IndexPQ (LUT256, nbits=8) — the paper's Section 4.4 baseline. 100K vectors, k=64. FAISS PQ sub-quantizer counts sized to lucifer TurboQuant's spot complaint (m=d/4 astatine 2-bit, m=d/2 astatine 4-bit).

The charts crippled calibrated TurboQuant (TQ+). Across OpenAI d=1536 and d=3072, TQ+ thumps FAISS astatine R@1 connected 3 of 4 cells (by 0.9–2.9 points; d=1536 4-bit trails by 0.7), and some scope 1.0 by k=8 (≥0.997 already astatine k≤4). GloVe d=200 is the harder authorities — astatine debased dim the asymptotic Beta presumption is looser. TQ+ lands up of FAISS astatine R@1 astatine some spot widths (+1.9 astatine 4-bit, +0.8 astatine 2-bit), pinch FAISS keeping a slim separator astatine 2-bit from k≈8. Uncalibrated numbers are successful the JSONs (tq_recalls).

A statement connected baselines. We comparison against FAISS IndexPQ (LUT256, nbits=8, float32 LUT) because it's the default production-grade PQ astir users would scope for. This is simply a stronger baseline than the civilization u8-LUT PQ successful the TurboQuant paper — FAISS uses a higher-precision LUT astatine scoring clip and k-means++ for codebook training. We reproduce the paper's TurboQuant numbers connected OpenAI d=1536 / d=3072 and deed akin numbers to different organization reference implementations connected low-dim embeddings (see turboquant-py astatine d=384). On GloVe (d=200) — the low-dim authorities wherever the asymptotic Beta presumption is loosest — TurboQuant lands up of FAISS astatine 4-bit but trails it astatine 2-bit; TQ+ calibration recovers the 2-bit shortage astatine R@1 (0.572 vs FAISS's 0.564), pinch FAISS keeping a slim separator astatine deeper k.

Full results: d=1536 2-bit, d=1536 4-bit, d=3072 2-bit, d=3072 4-bit, GloVe 2-bit, GloVe 4-bit.

All benchmarks: 100K vectors, 1K queries, k=64, median of 5 runs.

On ARM, TurboQuant thumps FAISS FastScan successful each config, averaging 3.5× astatine 4-bit (3.4–3.7× crossed cells — the SDOT/SMMLA dot-product kernels people the vector-major layout directly) and 26% astatine 2-bit (22–29%).

On x86, TurboQuant wins each config, averaging 3.4× astatine 4-bit (3.2–3.5× crossed cells — the AVX-512 VNNI dot-product kernel connected the vector-major layout) and 20% astatine 2-bit (5–32%), wherever the vpermb LUT scan carries the short 2-bit accumulate loop.

Same corpus arsenic the hunt cells: 100K OpenAI vectors, median of 5 runs, timed loops including the Python-call overhead a caller really pays per op. Insertion measures per-vector add() latency connected a warm, populated scale (built untimed) astatine n=1 — a single-vector add() — and n=100 — a 100-vector batch, showing really acold batching amortizes the per-call overhead — against add() into the trained, populated FAISS IndexPQFastScan (training untimed). A azygous add() lands successful 6.3–19.7 µs depending connected the compartment (7.6–13.9× faster than a FAISS azygous add), and a 100-vector batch amortizes TurboQuant to 4.6–16.3 µs/vector (4.6–15.1× faster than the aforesaid batch into FAISS). Removal measures per-op remove-by-id latency astatine n=1 (the dependable per-op complaint complete 1000 removes) and n=100 (the first 100 removes connected a caller index): IdMapIndex.remove(id) — O(1) swap-and-pop positive the id-map bookkeeping — lands astatine 0.44–1.22 µs and 0.59–1.37 µs per op crossed the cells. The FAISS file is the aforesaid user-visible operation, remove_ids connected an IndexIDMap complete IndexPQFastScan, which repacks the stored codes connected each call: 0.19–1.02 s per azygous region astatine 100K, pinch costs doubling alongside codification size — which is why the removal charts usage a log-scale axis. Charts show the single-threaded cells (RAYON_NUM_THREADS=1); the _mt cells are measured excessively and lucifer astatine n=1, since a azygous adhd is serial. Scripts: benchmarks/suite/.

Full results: d=1536 2-bit insert, d=1536 4-bit insert, d=3072 2-bit insert, d=3072 4-bit insert, and the matching speed_remove_* and _mt files.

Full results: d=1536 2-bit insert, d=1536 4-bit insert, d=3072 2-bit insert, d=3072 4-bit insert, and the matching speed_remove_* and _mt files.

Same corpus arsenic the hunt cells: 100K OpenAI vectors, median of 5 runs. TurboQuant serializes to a azygous .tv record pinch an fsync + atomic rename; FAISS is write_index / read_index connected the precision-matched IndexPQFastScan (sub-quantizer count matched to TurboQuant's spot rate, arsenic successful the hunt cells). Save (warm) is simply a constitute aft a hunt has run, truthful the blocked layout cache is populated. Load → first search opens a caller scale and times the first query — separating bare deserialization (the page cache is lukewarm throughout, truthful this is layout work, not cold-storage I/O) from the first-query cost. Round-trip chains the checkpoint/resume rhythm an embedding shop really pays — mutate 1K vectors → prevention → reopen → service the first query; FAISS has nary measured balanced for this path, truthful it is shown for TurboQuant only. On the smaller payloads the round-trip tin travel successful below the isolated post-mutation ("dirty") write: the 2 are timed successful abstracted suite steps, and astatine mini record sizes the standalone fsync successful the dirty-write measurement dominates and inflates it — a measurement artifact of the harness, not a repack triumph successful the mixed path. Single-threaded cells pin RAYON_NUM_THREADS=1. Scripts: benchmarks/suite/.

Full results: d=1536 2-bit persist ST, MT, d=1536 4-bit persist ST, MT, d=3072 2-bit persist ST, MT, d=3072 4-bit persist ST, MT.

Full results: d=1536 2-bit persist ST, MT, d=1536 4-bit persist ST, MT, d=3072 2-bit persist ST, MT, d=3072 4-bit persist ST, MT.

Each vector is simply a guidance connected a high-dimensional hypersphere. TurboQuant compresses these directions utilizing a elemental insight: aft applying a random rotation, each coordinate follows a known distribution -- sloppy of the input data.

1. Normalize. Strip the magnitude (norm) from each vector and shop it arsenic a azygous float. Now each vector is simply a portion guidance connected the hypersphere.

2. Random rotation. Multiply each vectors by the aforesaid random orthogonal matrix. After rotation, each coordinate independently follows a Beta distribution that converges to Gaussian N(0, 1/d) successful precocious dimensions. This holds for immoderate input information -- the rotation makes the coordinate distribution predictable.

3. Per-coordinate calibration (TQ+). The Beta distribution from measurement 2 is asymptotic — astatine finite dimensions, individual coordinates drift from the canonical style (especially low-bit and word-vector-style embeddings). TQ+ fits 2 scalars per coordinate — a displacement and a standard — mapping each coordinate's empirical quantiles onto the codebook's outermost centroids. The probability level comes from the codebook, truthful it tracks the spot width (~0.933 astatine 2-bit, ~0.996 astatine 4-bit) alternatively than being fixed. The Lloyd-Max codebook past quantizes against the target distribution it was designed for. The fresh is explicit: telephone index.calibrate(sample) erstwhile pinch a random, typical sample of your vectors (~1024 rows is capable — a tie of that size matches fitting connected the full corpus) earlier adding; afterwards the calibration is committed and reused by each adhd — nary retraining, nary rebuilds, nary abstracted train phase. An scale you ne'er calibrate is plain TurboQuant. index.calibration_state reports "uncalibrated" aliases "calibrated". Recall gain: up to +2.2pp astatine @1 connected the cells that drift astir (e.g. GloVe astatine 2-bit).

4. Lloyd-Max scalar quantization. Since the distribution is known, we tin precompute the optimal measurement to bucket each coordinate. For 2-bit, that's 4 buckets; for 4-bit, 16 buckets. The Lloyd-Max algorithm finds bucket boundaries and centroids that minimize mean squared error. These are computed erstwhile from the math, not from the data.

5. Bit-pack. Each coordinate is now a mini integer (0-3 for 2-bit, 0-15 for 4-bit). Pack these tightly into bytes. A 1536-dim vector goes from 6,144 bytes (FP32) to 384 bytes (2-bit). That's 16x compression.

6. Length-renormalized scoring. Scalar quantization systematically underestimates soul products — the reconstructed portion guidance is simply a small shorter than the original. We compute 1 scalar per vector astatine encode clip — the soul merchandise of the rotated portion vector pinch its ain centroid reconstruction — and shop ||v|| / ⟨u, x̂⟩ alongside each compressed vector. The hunt kernel multiplies the per-candidate people by this scalar earlier heap insertion, turning the inner-product estimator from downward-biased into unbiased astatine zero search-time costs and zero other storage. The callback summation shows up astir astatine debased spot widths, wherever the quantization shrinkage is largest.

Encoding cost: 1 other d-dimensional dot merchandise per vector to compute ⟨u, x̂⟩. On 1M vectors astatine d=1536 this is sub-second of further encode clip — a one-shot value paid astatine ingest, not astatine query.

Search. Instead of decompressing each database vector, we rotate the query erstwhile into the aforesaid domain and people straight against the codebook values. The scoring kernel uses SIMD intrinsics (NEON connected ARM; AVX-512BW connected modern x86, falling backmost to AVX2, past to a scalar way connected pre-AVX2 CPUs) pinch nibble-split lookup tables for maximum throughput.

The Lloyd-Max codebook achieves distortion wrong a facet of 2.7x of the information-theoretic little bound (Shannon's distortion-rate limit); the length-renormalization measurement removes the residual bias the Lloyd-Max codebook introduces connected the inner-product estimator itself.

All x86_64 builds target x86-64-v2 (SSE4.2 baseline, Nehalem 2008+) via .cargo/config.toml, truthful immoderate x86-64-v2 CPU tin tally the full crate. The AVX-512 and AVX2 kernels are #[target_feature]-gated and selected astatine runtime via is_x86_feature_detected!, truthful they footwear successful connected hardware that supports them sloppy of the compile baseline; CPUs pinch neither tally the scalar fallback.

Download datasets:

Each benchmark is simply a self-contained book successful benchmarks/suite/. Run immoderate 1 individually:

Run each benchmarks for a category:

Results are saved arsenic JSON to benchmarks/results/. Regenerate charts:

The suite supra is the root of each published number — existent embeddings, FAISS comparator, fixed shapes, tally connected the 2 charismatic environments. For the inner loop of an optimization walk there's besides a Rust harness that reproduces the 4 mutation metrics (cold bulk add, lukewarm append, azygous add, remove) on deterministic synthetic vectors, truthful a presumption tin beryllium measured successful seconds on any instrumentality pinch nary dataset and nary FAISS:

It is simply a screening tool, not a root of published numbers.

examples/encode_hash prints a per-stage hash of the encode pipeline for a fixed input; CI runs it connected each OS successful the matrix and fails if they disagree, which is really cross-platform byte personality of the encode is checked.

• TurboQuant: Online Vector Quantization pinch Near-optimal Distortion Rate (ICLR 2026) -- the insubstantial this implements
• RaBitQ: Quantizing High-Dimensional Vectors pinch a Theoretical Error Bound for Approximate Nearest Neighbor Search (SIGMOD 2024) -- the root of the per-vector length-renormalization correction adapted successful measurement 5
• FAISS Fast accumulation of PQ and AQ codes -- turbovec's x86 SIMD kernel adapts FastScan's battalion layout, nibble-LUT scoring, and u16 accumulator strategy