TL;DR: We fused SuperKMeans and e-means, resulting in 3× faster clustering of high-dimensional vectors without norm trade-offs. Available now at https://github.com/cwida/SuperKMeans.
Recently, there has been a lot of buzz about clustering of vector embeddings: EVōC, SuperKMeans, QGKmeans, and fair final week, e-means, o-means, and s-means. Among these, e-means is a big deal: it promises the identical norm as Lloyd’s alongside lone 2 epochs. Why is this a big agreement for vector search?
Well, clustering is a key construction obstacle for large-scale vector indexes (e.g., IVF, ScANN), which is exactly the bottleneck that users have to delay on before issuing similarity hunt queries.
A twelvemonth ago, we released SuperKMeans. SuperKMeans accelerates Lloyd’s k-means by 2—10× by avoiding scanning vector dimensions during clustering, without trade-offs in clustering quality. SuperKMeans has been quickly adopted by industry (ParadeDB, Zilliz, FAISS), showing how timely it is to enhance clustering achievement for high-dimensional vectors. However, e-means is additionally breathtaking since it tackles attempt by operating small epochs. Traditionally, clustering for vector embeddings is standardized to 10 epochs. e-means promises the identical (or better) clustering norm with fair 2 epochs. I would propose checking out the fantastic website made by e-means’ authors: https://e-means.github.io/
Interestingly, in their paper, the authors propose that SuperKMeans’ idea of pruning dimensions is orthogonal to e-means. We agree: there is nothing concerning these two ideas that opposes the other. Naturally, this put us to work!
The Setup
We have extended our clustering archive to execute e-means and SuperEMeans (open-sourced at https://github.com/cwida/SuperKMeans) and have measured their benefits.
The datasets: We volition be using two datasets: i) 5M 1536-dimensional OpenAI embeddings from the VectorDBBench benchmark: https://huggingface.co/datasets/makneeee/openai_large_5m. (27 GB). ii) The archetypal 10M MS MARCO passages embedded alongside the Cohere Embed V3 example (1024-dimensional vectors): https://huggingface.co/datasets/Cohere/msmarco-v2.1-embed-english-v3 (40 GB).
The index: We volition build an IVF indicator alongside a point-per-cluster proportion of 1:256, resulting in about 20,000 centroids for OpenAI and 40,000 centroids for the Cohere dataset.
The setup: For this experiment, I volition use a sole Zen 5 device alongside 16 cores and 128 GB of RAM (r8a.4xlarge | $1.278/hr). However, our SuperKMeans archive plant on any CPU (e.g., Intel, Graviton, Zen).
The Results
The following storyline compares the end-to-end clustering runtime of Lloyd’s (10 iterations), SuperKMeans (10 iterations), e-means (2 epochs), and SuperEMeans (2 epochs). SuperEMeans substantially accelerates e-means, up to 1.8× faster.

But… is it a high-quality index?
Let’s difference the centroids obtained by all method in conditions of how fine they execute on vector hunt tasks. For this analysis, we matter 1000 queries to our IVF index. Then, we measure the proportion of top-100 retrieved results that are in the true top-100 results during scanning 1% of the accessible clusters (recall@100). We have additionally compared the balance of clusters by counting the figure of extend calculations performed whenever scanning those clusters. The results for OpenAI are summarized in the array below:
| Algorithm | Runtime (s) | Avg. Recall@100 at 1% of probed centroids | Avg. Vectors scanned at 1% of probed centroids |
|---|---|---|---|
| Lloyd’s k-means (10 iters) | 985 | 0.962 | 52 053 |
| SuperKMeans (10 iters) | 232 | 0.962 | 51 993 |
| e-means (2 epochs) | 203 | 0.967 | 58 184 |
| SuperEMeans (2 epochs) | 128 | 0.967 | 58 198 |
| e-means (2 epochs, author’s implementation) | 333 | 0.967 | 58 111 |
Recall@100 is slightly higher in e-means. However, we end up scanning about 10% additional vectors during searches. To equivalent the figure of vectors scanned, we have to investigation slightly small clusters in e-means, about 0.90% of the total clusters. This results in recall@100 = 0.964 during scanning 51,962 vectors, resulting in an ultimately improved trade-off than traditional k-means.
The narrative is the identical for Cohere:
| Algorithm | Runtime (s) | Avg. Recall@100 at 1% of probed centroids | Avg. Vectors scanned at 1% of probed centroids |
|---|---|---|---|
| Lloyds k-means (10 iters) | 2504.4 | 0.919 | 112 714 |
| SuperKMeans (10 iters) | 623.3 | 0.919 | 112 547 |
| e-means (2 epochs) | 525.8 | 0.923 | 135 380 |
| SuperEMeans (2 epochs) | 295.3 | 0.923 | 135 463 |
| e-means (2 epochs, author’s implementation) | 927.1 | 0.925 | 136 185 |
Again, we have to investigation slightly small clusters in e-means, about 0.90% of the total clusters. This results in recall@100 = 0.918 during scanning 121,511 vectors. Here, the gap is not favorable for e-means, which scans 8% additional vectors for the identical recall.
Why does e-means scan additional vectors?
In e-means, clusters are additional bloated than in Lloyd’s k-means in the regions of the vector area queried most often. Yet e-means reaches a lesser WCSS (MSE), since that goal doesn’t reward stable clusters, during vector hunt pays for all item it scans. Note that the disbursal is latency at a fixed figure of probed clusters, not retrieval norm (since matching the figure of vectors scanned yields approximately the identical recall).
As for the algorithmic logic why, the imbalance comes from e-means’ mini-batch updates. A collection that gains fair a few points in the first batches stays alongside fair a few points for the remainder of the clustering.
To counteract this, e-means introduces a revival stage that, all five batches, re-seeds clusters whose use in a lot cut below 0.1× the average use throughout clusters. If this happens, that collection is eliminated and a ample collection is divided into two. We established this threshold too conservative, leaving many small clusters stuck alongside few points throughout batches. Raising the revival threshold from 0.1 to 0.3 brings collection balance and scan disbursal nearer to Lloyd’s flat in the Cohere dataset.
| Algorithm | Avg. Recall@100 at 1% of probed centroids | Avg. Vectors scanned at 1% of probed centroids |
|---|---|---|
| k-means (10 iters) | 0.919 | 112 714 |
| e-means (2 epochs, revival t=0.1) | 0.923 | 135 380 (1.20× higher) |
| e-means (2 epochs, revival t=0.3) | 0.918 | 119 548 (1.06× higher) |
More Epochs
At 5 epochs (the author’s default in their implementation), the speedup from SuperEMeans is remarkable: up to 3× faster!
| Cohere | Runtime (s) | Avg. Recall@100 at 1% of probed centroids | Avg. Vectors scanned at 1% of probed centroids |
|---|---|---|---|
| e-means (5 epochs) | 1283 | 0.927 | 136 212 |
| SuperEMeans (5 epochs) | 436 (3× faster) | 0.927 | 136 191 |
| OpenAI | Runtime (s) | Avg. Recall@100 at 1% of probed centroids | Avg. Vectors scanned at 1% of probed centroids |
|---|---|---|---|
| e-means (5 epochs) | 513 | 0.968 | 58 078 |
| SuperEMeans (5 epochs) | 162 (3.2× faster) | 0.968 | 58 086 |
e-means is an engaging advancement in clustering algorithms. I would affection to see additional advancements that pay notice to collection balance. This is a key aspect for vector hunt achievement that doesn’t necessarily align alongside the WCSS / MSE goal of k-means.
Anyways, I am enthusiastic to see anywhere we go from here! As the authors additionally citation in their paper, approximate project can accelerate large-scale clustering (+1B points), anywhere equal 1 era can disbursal hours. In our library, we execute hierarchical k-means, which has additionally seen achievement on vector-embedding datasets (implemented in NVIDIA’s cuVS library). However, it didn’t activity fairly fine alongside e-means. I doubtful this power additionally be because of how cluster’s revival works.
Another choice is to collection quantized vectors. For example, quantize 32-bit floats to 8-bit integers, or 1-bit RaBitQ codes before clustering. We have explored this in another investigation paper alongside notable results.
SuperKMeans and e-means detailed specialized reports are accessible on arXiv.
Reproducibility
The origin code to reproduce the results of this blog article are accessible at: https://github.com/lkuffo/reproducibility-superemeans