Nowadays, astir group look to over-engineer their RAG stack. They jump consecutive to embeddings, vector databases, and reranking pipelines. Meanwhile, their users conscionable want to find the doc that says “How to reset my password.”
In engineering, there’s ever the correct instrumentality for the correct problem. In AI Retrieval Systems it’s not different.
Before we dive into recipes, let’s found erstwhile you should usage each approach. The cardinal factors are:
1. Data Freshness Requirements - Real-time updates (news, societal media) favour approaches pinch easy re-indexing. Daily aliases play updates activity good pinch hybrid approaches. A unchangeable corpus (monthly aliases quarterly updates) makes pre-embedding sensible.
2. Corpus Characteristics - High churn (more than 10% changes daily) intends you should debar afloat pre-embedding. Stable documents activity good pinch pre-embedding. Long-tail distribution (90% ne'er accessed) intends on-the-fly wins.
3. Query Patterns - Keyword-heavy queries should commencement pinch full-text search. Semantic aliases conversational queries use from embeddings. Mixed patterns request hybrid approaches.
4. Scale & Performance - Less than 1000 queries per time intends elemental approaches are sufficient. 1K to 10K queries per time requires selective optimization. More than 10K queries per time justifies afloat optimization.
5. Team Capabilities - No ML expertise intends enactment pinch full-text positive query rewriting. Some ML acquisition makes hybrid hunt manageable. Having an ML squad disposable makes precocious approaches viable.
Now, let’s look astatine the look book. Start astatine the top. Move down only erstwhile you person information proving you request to.
Good aged BM25. Elasticsearch. Postgres full-text search. The worldly that existed earlier “embedding” became a verb.
You’re conscionable starting out. Your users constitute keyword-style queries (”pandas merge dataframe”). Exact matches matter (”invoice #12345”). You want zero ML complexity. Your corpus has proprietary terminology (more connected this later).
Zero API costs. Fast (under 10ms). Easy to debug (you tin spot precisely why a archive matched). Surprisingly effective (handles galore usage cases). No chunking strategy needed – useful pinch afloat documents. No information complexity – easy to trial and validate. No exemplary deprecation consequence (BM25 doesn’t change).
Misses synonyms (”car” vs “automobile”). Fails connected semantic queries (”How do I...?”). Can’t understand intent beyond keywords.
In my experience, this handles a important information of usage cases. Don’t skip this step. You mightiness beryllium amazed really acold you tin get.
When you jump consecutive to embeddings, you instantly look questions like: What chunk size? (512 tokens? 1024?) What overlap? (50 tokens? 100?) Semantic chunking aliases fixed-size? How do I measure if my chunking is good?
With full-text search, you skip each of this. Your documents are your documents. Search conscionable works.
Use an LLM to toggle shape messy personification queries into cleanable keyword searches.
Most “semantic search” problems are really query formulation problems.
Users inquire questions conversationally. Vocabulary mismatch (users opportunity “fix bugs”, docs opportunity “debugging”). You person soul motto (your model called “Atlas”). You want elasticity to iterate quickly connected query strategies.
~$0.001 per query (using GPT-4o-mini for query rewriting)
An LLM tin region stopwords (”how do I” becomes nothing). It tin adhd synonyms (”car” becomes “car automobile vehicle”). It tin construe domain position (”speed up code” becomes “optimize performance”). It tin decompose analyzable queries (”read CSV and plot” becomes [”read CSV”, “plot data”]). It tin study from your glossary (via strategy prompt).
With embeddings, if results aren’t good, you request to set chunking strategy, re-embed full corpus, tally regression tests connected your eval set, and dream it improved.
With query rewriting, if results aren’t good, you set the strategy prompt. That’s it. Test immediately.
Even better, you tin create a loop:
The supplier tin iterate, learn, and accommodate – each without re-embedding anything.
Say your institution has a Python model called “Atlas.” If you usage general-purpose embeddings:
General embedding exemplary (trained connected internet): “Atlas” = [vectors pointing toward: Greek mythology, maps, geography] Your existent Atlas docs = [vectors astir information processing] Similarity score: 0.15 (terrible!)The exemplary has nary thought your “Atlas” exists. It falls backmost to what it learned successful training. But pinch query rewriting:
For proprietary terms, nonstop keyword matching thumps semantic understanding.
Use BM25 to get candidates (top 50-100), past rerank pinch embeddings (top 10).
BM25 is accelerated and awesome astatine keyword matching. Embeddings are bully astatine semantic understanding. Together, they screen each other’s weaknesses.
Users inquire semantic questions (”find alternatives to X”). BM25 positive query rewriting unsocial isn’t cutting it (you person information proving this). You tin tolerate 100-500ms latency. Your corpus is comparatively unchangeable (not changing each minute).
Let’s do the mathematics pinch existent pricing (OpenAI text-embedding-3-small astatine $0.02 per 1M tokens):
Embedding 50 docs per query (avg 500 tokens each) intends 50 docs × 500 tokens = 25,000 tokens
Cost: 25,000 × $0.00002 = ~$0.0005 per query. At 1,000 queries per time × 30 days = ~$15 per month.
Actually beautiful reasonable. But there’s a catch: latency.
Embedding 50 documents on-the-fly adds 200-500ms per query. For user-facing search, that’s noticeable. This is wherever the existent trade-off lives – not cost, but speed.
When you present embeddings, you request to determine really to chunk your documents (fixed-size? semantic? by section?). You request to find what chunk size and overlap to use. You request to grip chunks that span important context.
This adds complexity that axenic full-text hunt avoids.
If your information changes frequently, why salary to re-embed everything?
High archive churn (more than 10% of docs updated daily). Real-time content (news, societal media, unrecorded updates). You’re experimenting pinch embedding models (no re-indexing needed). Data freshness is critical (documents must beryllium up-to-date). Small K for reranking (20-50 docs).
On-the-fly / online (1000 queries/day, 50 docs/query): - Embedding cost: ~$15/month (ongoing) - Storage: $0 (just shop text) - Latency: 200-500ms per query - Freshness: Perfect (always current) - Model switching: Easy (just alteration the API call)Here’s thing group don’t talk astir enough: embedding models get deprecated.
OpenAI deprecated text-embedding-ada-002 successful favour of text-embedding-3. If you pre-embedded 10 cardinal documents pinch the aged model, you now request to re-embed each 10 cardinal documents pinch the caller model, update your vector database, tally regression tests connected your information set, validate that value didn’t degrade, grip the cutover period, and woody pinch immoderate API changes.
You virtually conscionable alteration 1 statement of code. Done.
Latency. You’re embedding documents connected each query. This is only viable if you’re okay pinch 200-500ms latency, K is mini (reranking 20-50 docs, not 500), and your usage lawsuit favors freshness complete speed.
Pre-embed often accessed documents (”hot tier”), embed rarely-accessed documents on-the-fly (”cold tier”).
Access patterns travel Pareto distribution. 20% of docs get 80% of traffic.
Clear entree patterns (some docs are accessed measurement much than others). Medium-to-large corpus (more than 100K documents). Mix of unchangeable and changing content. Need bully latency for communal queries. Want to minimize re-embedding connected exemplary updates.
Fast for 80% of queries (hit pre-embedded cache). Fresh for rarely-accessed docs. Only re-embed basking tier erstwhile switching models (20% of corpus). Adapts to changing entree patterns. Best latency/cost/flexibility trade-off.
When your embedding exemplary gets deprecated:
Full pre-embedding: Re-embed 1M docs × $0.01 = $10,000 + downtime Hot/cold tiers: Re-embed 200K docs × $0.01 = $2,000 + minimal downtime On-the-fly: Change 1 statement of codification = $0 + zero downtimeEmbed everything upfront. Store successful vector database. Search pinch ANN (approximate nearest neighbors).
Very precocious query measurement (more than 10K queries per day). Need nether 50ms latency. Very unchangeable corpus (under 5% churn per month). Access shape is wide (no agelong tail). You person ML squad to negociate infrastructure.
Pre-embedding (1M docs): - One-time embedding: 1M docs × 500 tokens × $0.00002 = $10 - Storage: 1M × 1536 dims × 4 bytes = 6GB (~$10-30/month) - Search latency: nether 50ms (blazing fast!) - Freshness: Only arsenic caller arsenic past re-indexDocuments alteration often (more than 10% per week). You’re experimenting pinch embedding models. Low query measurement (under 1K queries per day). You haven’t tried simpler approaches first.
This is wherever afloat pre-embedding hurts the most. When you request to move models, you look downtime (your hunt is degraded while re-embedding), compute cost (re-embedding millions of documents), testing burden (full regression trial suite connected caller embeddings), chunking reevaluation (maybe caller exemplary useful amended pinch different chunk sizes?), and risk (what if the caller exemplary is worse for your domain?).
This is overkill for astir systems. I’ve seen teams walk months optimizing their vector database setup erstwhile query rewriting would person solved 90% of their problems.
But if you’re Pinterest, Shopify, aliases handling monolithic standard pinch a unchangeable corpus, this is wherever you extremity up.
Here’s wherever things get spicy. We’ve been discussing single-intent queries: “How do I merge dataframes?”
But existent users inquire worldly like: “How do I publication a CSV file, cleanable missing data, and crippled the results?”
That’s 3 abstracted intents. Searching for this arsenic 1 query is for illustration trying to find a edifice that serves pizza, sushi, and tacos. Good luck.
Modern agentic RAG systems (Perplexity, ChatGPT search) grip this elegantly:
Break down the query.
Route each sub-query optimally
Combine results into coherent answer
Each sub-query is focused and precise, starring to amended retrieval. Parallel execution intends little latency (max, not sum). Adaptive routing results successful little costs (only analyzable queries salary for LLM). Structured output provides amended UX.
Without decomposition
LLM rewriting full analyzable query: $0.005
Embedding 50 docs: $0.025
Total: $0.03
With decomposition
Decompose: $0.001
Sub-query 1 (simple): $0
Sub-query 2 (simple): $0
Sub-query 3 (complex): $0.001
Total: $0.002
15x cheaper, amended quality.
This is wherever agentic retrieval really shines. The supplier tin intelligently determine which sub-queries request costly processing (embeddings) and which tin beryllium handled pinch inexpensive methods (simple preprocessing + BM25).
Okay, you’ve publication this far. You conscionable want to know: “What should I build?”
Start here: Do you person hunt astatine all? If not, build BM25 first. Seriously. Stop reference and build it. If you do person search, continue.
Measure your baseline. Run your existent hunt for 2-4 weeks and cod personification feedback. Are users happy pinch the results? If yes, stop. You’re done. Go vessel features. If no, continue.
What’s the main complaint?
If users opportunity “Can’t find docs that intelligibly exist,” effort query rewriting first. At $0.001 per query pinch zero re-indexing, it’s worthy testing. Run an A/B trial for 2 weeks. If you spot bully improvement, support it and you’re done. If it’s not enough, continue.
If users opportunity “Results are okay but not great,” A/B trial hybrid hunt (sparse positive embedding rerank). Is the added latency worthy it? If yes, determine connected implementation. If your information changes frequently, usage on-the-fly embedding. If you person clear basking docs, usage hot/cold tiers. If you person a unchangeable corpus and precocious scale, usage afloat pre-embedding. If the latency isn’t worthy it, optimize query rewriting further instead.
If users opportunity “Need amended semantic understanding,” usage hybrid hunt and take your attack based connected your situation. High churn (more than 10% per day) intends on-the-fly. Medium standard pinch clear patterns intends hot/cold tiers. Massive standard pinch unchangeable information intends afloat pre-embedding.
Key determination factors:
Full-text pinch query rewriting offers cleanable information freshness pinch debased setup complexity and query latency nether 50ms. Model switching is trivial, nary chunking is needed, and it useful for astir usage cases.
On-the-fly embedding provides cleanable information freshness pinch debased setup complexity but higher query latency of 200-500ms. Model switching is trivial, chunking is needed, and it’s champion for precocious churn scenarios.
Hot/cold tiers supply mixed information freshness pinch mean setup complexity and query latency of 50-100ms. Model switching is easy, chunking is needed, and it offers balanced capacity for varied needs.
Full pre-embedding has old information until reindex pinch precocious setup complexity but query latency nether 50ms. Model switching is painful, chunking is needed, and it’s designed for monolithic standard operations.
The 80/20 rule: 60% of systems should extremity astatine full-text positive query rewriting. 25% request hybrid pinch on-the-fly aliases hot/cold. 10% request afloat pre-embedding. 5% request civilization solutions.
Bottomline: Don’t beryllium the personification who builds the 5% solution for a 60% problem.
English (US) · 
Indonesian (ID) · 
