What RAG Actually Costs to Run in Production: A Full Cost Breakdown

The costs model

The assumptions, stated successful afloat truthful the exemplary is reproducible. Corpus: 100,000 documents averaging 1,500 tokens each (150M tokens of text), chunked astatine 512 tokens pinch 64-token overlap into astir 335,000 chunks (171M embedded tokens; overlap causes a 1.14x description ). Per query: a 32-token mobility embedding, reranking of 20 retrieved candidates (about 10,400 input tokens, 40 output), and reply procreation complete the apical 5 chunks (about 2,900 input tokens, 350 output). Models: Qwen3 Embedding 0.6B ($0.04 per 1M tokens), DeepSeek V4 Flash for reranking ($0.068 successful / $0.168 retired per 1M), gpt-oss-120b for answers ($0.10 / $0.70) pinch Llama 3.3 70B ($0.65 / $0.65) arsenic the alternative. Months are 30 days.

One-time ingestion costs

Ingesting 100,000 documents costs nether $10 without enrichment and nether $20 pinch it. If the corpus doubles, these numbers double; they ne'er go the problem.

Monthly costs astatine 3 postulation levels

Cost per query, all-in pinch gpt-oss-120b, is each full divided by that tier’s monthly query volume: astir $0.0033 astatine debased postulation ($98 ÷ 30,000 queries, wherever the fixed database costs dominates), $0.0017 astatine mid postulation ($496 ÷ 300,000), and $0.0013 astatine precocious postulation ($4,000 ÷ 3,000,000). Database scheme prices are typical existent tiers; corroborate against the managed database plans for your region and engine.

Why the measure looks for illustration this

Three things autumn retired of the table, and each 1 contradicts a communal assumption.

First, embeddings and retention are cheap. Query embeddings ne'er scope moreover 1% of the bill, and the vectors adhd astir a dollar of retention connected apical of the database scheme (the database statement successful the array is the full managed cluster, which astir applications tally anyway). Trying to prevention money present is optimizing the incorrect statement item. It besides intends re-embedding the full corpus aft a chunking alteration costs astir $7. You tin spend to experiment.

Second, exemplary prime for procreation is simply a 4x swing. gpt-oss-120b and Llama 3.3 70B tally identical prompts successful this pipeline, and the quality astatine 100K queries per time is $1,605 versus $6,338 per month. That makes procreation exemplary action the highest-leverage costs determination successful the full system. Evaluate whether the cheaper exemplary answers your questions good capable earlier defaulting to the bigger name; successful RAG, wherever the model’s occupation is reference provided discourse alternatively than recalling facts, smaller models adjacent overmuch of the value gap.

Third, astatine precocious postulation the reranker softly becomes the largest statement item. This surprises group because reranking feels for illustration a insignificant post-processing step, but it sounds each 20 campaigner chunks per query, astir 4 times much tokens than the reply exemplary sees. Token volume, not exemplary price, is what makes it expensive.

How to little the bill

The reranking measure has straightforward controls. Rerank 10 candidates alternatively of 20 and the costs halves, usually pinch small value nonaccomplishment if retrieval is decent. Rerank conditionally, only erstwhile the apical vector scores are adjacent together and the ordering is genuinely ambiguous, and you skip the measurement wholly for the mostly of easy queries. Or usage the managed knowledge guidelines reranker (BGE Reranker v2 m3 astatine $0.01 per cardinal tokens, astir a seventh of the LLM reranker’s effective rate) if you adopt the managed stack.

Generation has levers too. Prompt caching, supported connected serverless inference, discounts the repeated strategy punctual connected models that support it, though retrieved chunks disagree per query and will not deed the cache. Capping max_tokens and instructing the exemplary to reply concisely straight cuts the astir costly tokens you buy.

There is besides a crossover constituent worthy knowing. Serverless pricing scales linearly pinch postulation forever, but dedicated inference does not: a azygous NVIDIA H100 runs $4.41 per hour, astir $3,220 per month, flat, sloppy of query volume. When your monthly serverless procreation and reranking walk approaches that number (in this model, location adjacent the 100K-queries-per-day tier), dedicated capacity for an open-weight exemplary starts winning connected value and gives you accordant latency astatine the aforesaid time. Below that, serverless wins because you salary thing for idle.

How the economics comparison to moving this elsewhere

The token prices successful this exemplary are not unusually low. For gpt-oss-120b, Fireworks AI, Groq, and Together AI each published $0.15 per 1M input and $0.60 per 1M output arsenic of July 2026, against DigitalOcean’s $0.10 and $0.70 (verified pricing and compatibility differences are successful our comparison of OpenAI-compatible conclusion APIs). Which is cheaper depends connected your postulation shape, and RAG has a unique one: this pipeline sends astir 8 input tokens for each output token, because the exemplary sounds 5 chunks to constitute 1 answer. Input-heavy postulation favors the supplier pinch the cheaper input rate. On this model’s token mix, procreation costs $0.000535 per query connected DigitalOcean versus $0.000645 astatine the $0.15/$0.60 providers: $1,605 versus $1,935 per period astatine 100K queries per day. Routing done an aggregator for illustration OpenRouter adds its 5.5% in installments interest connected apical of whichever supplier serves the request.

The larger quality is structural alternatively than per-token. A RAG pipeline is not conscionable inference: it needs a vector database, and astir conclusion providers do not tally one. Building this stack connected an inference-only supplier intends a 2nd vendor for vector retention (a dedicated vector database, aliases self-hosted pgvector connected rented compute), a 2nd bill, abstracted entree control, and a web hop betwixt retrieval and procreation that sits connected your latency way for each azygous query. Running Managed PostgreSQL and the conclusion endpoint successful 1 relationship removes the cross-vendor hop, keeps the full pipeline’s walk connected 1 invoice, and intends the database backing your vectors has the aforesaid backups, failover, and monitoring arsenic the remainder of your infrastructure. The per-token differences supra are existent but small; the operational consolidation is usually the stronger argument.

The pipeline down the numbers

Everything supra is grounded successful a actual pipeline, and the remainder of this article builds it. The ingestion half runs erstwhile (and again erstwhile documents change): publication documents, divided them into chunks, make an embedding per chunk, and constitute chunk matter positive vector into PostgreSQL. The serving half runs connected each question: embed the question, propulsion the nearest chunks, rerank them, and walk the survivors to a exemplary that writes the answer.

Documents ──> Chunking ──> Embeddings API ──> Managed PostgreSQL (pgvector) │ User mobility ──> Embeddings API ──> vector hunt ─────┘ │ top 20 chunks │ LLM reranker │ top 5 chunks │ Serverless Inference (answer)

Everything model-related goes done 1 endpoint, https://inference.do-ai.run/v1, which is OpenAI-compatible: the charismatic OpenAI SDK useful aft changing the guidelines URL and API key, and you switch models by changing a string. The pipeline uses open-weight models throughout, which is what keeps the costs exemplary supra wherever it is.

To travel on you request a DigitalOcean relationship pinch a model entree key and a prepaid serverless conclusion balance, a Managed PostgreSQL cluster, and Python 3.10+ pinch openai, psycopg2-binary, and tiktoken.

Chunking

Language models and embedding models some activity amended connected focused passages than connected full documents, and vector hunt retrieves astatine the granularity you embed. Chunking is wherever you group that granularity, and it straight drives the 171M-token embedding fig successful the costs model.

We usage fixed-size chunks of 512 tokens pinch a 64-token overlap. Fixed-size chunking is not the cleverest strategy, but it is predictable, cheap, and difficult to get wrong, which is what you want for the first accumulation version. The overlap exists truthful a condemnation falling connected a chunk bound appears intact successful astatine slightest 1 chunk; it is besides why 150M tokens of matter becomes 171M tokens of embedding input. Overlap is not free, and this is wherever you salary for it.

import tiktoken enc = tiktoken.get_encoding("cl100k_base") CHUNK_TOKENS = 512 OVERLAP_TOKENS = 64 def chunk_text(text: str, doc_id: str) -> list[dict]: tokens = enc.encode(text) stride = CHUNK_TOKENS - OVERLAP_TOKENS chunks = [] for i, commencement in enumerate(range(0, len(tokens), stride)): model = tokens[start : commencement + CHUNK_TOKENS] if len(window) < 32: # skip trailing fragments break chunks.append({ "doc_id": doc_id, "chunk_index": i, "text": enc.decode(window), }) return chunks

Two applicable notes. Keep doc_id and chunk_index from the start; you request them for citations, re-indexing, and deletions. And defy the impulse to chunk small: chunks of 100 to 200 tokens retrieve precisely but often deficiency capable discourse to reply from, which pushes you into fetching much chunks and paying much per query.

Embeddings

An embedding turns a chunk of matter into a vector specified that semantically akin texts extremity up adjacent each other. This is what lets “how do I reset my password” find a chunk titled “credential betterment procedure” moreover though they stock nary keywords.

The serverless Embeddings API offers respective unfastened models. We usage Qwen3 Embedding 0.6B astatine $0.04 per cardinal input tokens: it handles agelong inputs comfortably, performs good connected multilingual and retrieval benchmarks, and produces 1024-dimensional vectors. If your corpus is English-only and costs matters much than quality, BGE-M3 and E5 Large V2 are disposable astatine $0.02 per cardinal tokens, which would trim the already-small $6.86 successful half.

import os from openai import OpenAI client = OpenAI( base_url="https://inference.do-ai.run/v1", api_key=os.environ["DIGITALOCEAN_INFERENCE_KEY"], ) def embed_batch(texts: list[str]) -> list[list[float]]: resp = client.embeddings.create( model="qwen3-embedding-0.6b", input=texts, ) return [item.embedding for point in resp.data]

Send chunks successful batches (32 to 64 per petition balances throughput against petition size) and adhd retry logic pinch exponential backoff; immoderate ingestion occupation spanning hundreds of thousands of requests will deed a transient nonaccomplishment eventually.

Optional: enrich chunks pinch batch conclusion astatine half price

Raw chunks suffer document-level context. A chunk that says “the limit was raised to 50” retrieves poorly because thing successful it says what limit, aliases where. A communal hole is to prepend a short LLM-generated discourse statement to each chunk earlier embedding it: which archive it came from and what it is about.

This is offline, latency-insensitive work, which makes it a fresh for batch inference; really the discount useful and really to submit, monitor, and download a occupation are covered successful that portion and successful the batch conclusion guide, truthful present we instrumentality to the RAG-specific question: which parts of this pipeline beryllium successful a batch job.

The norm is simple: batch thing that runs earlier users show up. Chunk enrichment qualifies precisely because nary 1 is waiting connected it; whether the occupation finishes successful 1 hr aliases twenty-four changes thing astir the pipeline. Query-time activity (retrieval, reranking, reply generation) ne'er qualifies, because a personification is connected the different end. Two constraints style the fresh here: batch presently supports only OpenAI and Anthropic matter models connected the chat endpoints, truthful this 1 measurement uses a commercialized exemplary moreover though the remainder of the pipeline is open-weight, and embedding requests themselves cannot spell done batch (at $0.04 per cardinal tokens, they do not request to; the LLM enrichment step, not embedding, is wherever ingestion money really goes).

Each petition successful the batch record asks a mini exemplary to situate 1 chunk:

def enrichment_request(chunk: dict, doc_title: str) -> dict: return { "custom_id": f'{chunk["doc_id"]}-{chunk["chunk_index"]}', "method": "POST", "url": "/v1/chat/completions", "body": { "model": "gpt-5-nano", "messages": [{ "role": "user", "content": ( f"Document title: {doc_title}\n\n" f"Chunk:\n{chunk['text']}\n\n" "Write 1 condemnation situating this chunk wrong the " "document, for usage arsenic a hunt discourse prefix. " "Output only the sentence." ), }], "max_tokens": 80, }, }

For 335,000 chunks (roughly 650 input and 60 output tokens per petition pinch GPT-5 nano), the occupation costs astir $18.92 astatine serverless rates and $9.46 done batch: the 50% discount doing its occupation connected the 1 ingestion measurement that involves an LLM. Prepend the returned condemnation to each chunk earlier embedding, and some the embedding and the reranker spot the added context.

This measurement is optional. Ship without it first, look astatine your retrieval failures, and adhd it if chunks retrieve poorly for deficiency of context. Re-embedding afterward costs different $7, inexpensive capable to dainty arsenic an research alternatively than a commitment.

Vector retention successful Managed PostgreSQL

You request location to support 335,000 vectors and their text, and to find the nearest vectors to a query quickly. A dedicated vector database is 1 option, but if you are already moving PostgreSQL, the pgvector extension turns the database you person into the vector shop you need, pinch backups, failover, and entree power already handled by the managed platform. DigitalOcean Managed PostgreSQL supports some vector (pgvector) and vectorscale arsenic standard extensions.

CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE chunks ( id BIGSERIAL PRIMARY KEY, doc_id TEXT NOT NULL, chunk_index INT NOT NULL, text TEXT NOT NULL, embedding vector(1024) NOT NULL, UNIQUE (doc_id, chunk_index) ); CREATE INDEX chunks_embedding_idx ON chunks USING hnsw (embedding vector_cosine_ops);

The HNSW scale is what makes hunt fast. Without it, each query scans each 335,000 vectors; pinch it, queries traverse a chart and return successful single-digit milliseconds astatine this scale. Build the scale aft bulk-loading the data, not before: inserting into an existing HNSW scale is overmuch slower than building it erstwhile complete a afloat table.

The ingestion loop ties the first 3 stages together:

import psycopg2 from psycopg2.extras import execute_values conn = psycopg2.connect(os.environ["DATABASE_URL"]) def store_chunks(chunks: list[dict], embeddings: list[list[float]]): rows = [ (c["doc_id"], c["chunk_index"], c["text"], emb) for c, emb in zip(chunks, embeddings) ] with conn.cursor() as cur: execute_values( cur, """INSERT INTO chunks (doc_id, chunk_index, text, embedding) VALUES %s ON CONFLICT (doc_id, chunk_index) DO UPDATE SET matter = EXCLUDED.text, embedding = EXCLUDED.embedding""", rows, template="(%s, %s, %s, %s::vector)", ) conn.commit()

The upsert connected (doc_id, chunk_index) makes re-ingestion idempotent: rerun a archive and its chunks are replaced, not duplicated. When a archive is deleted from the source, delete its rows by doc_id.

On sizing, which is wherever the database statement successful the costs array comes from: 335,000 vectors astatine 1,024 dimensions and 4 bytes per magnitude is astir 1.4 GB, positive the HNSW chart and chunk text, landing the array astir 4 GB connected disk. Single-node Managed PostgreSQL starts astatine $15 per month, good for development. For production, HNSW hunt capacity depends connected the scale staying successful memory, truthful prime a scheme pinch RAM comfortably supra your scale size and adhd a standby node for precocious readiness ($30 per period and up per node). Additional retention is $0.21 per GiB per month, which for this corpus rounds to astir a dollar.

Retrieval

Serving a mobility starts by embedding it pinch the aforesaid exemplary utilized astatine ingestion (not optional; vectors from different models unrecorded successful different spaces and comparing them is meaningless), past asking PostgreSQL for the nearest chunks:

def retrieve(question: str, k: int = 20) -> list[dict]: q_emb = embed_batch([question])[0] with conn.cursor() as cur: cur.execute( """SELECT doc_id, chunk_index, text, 1 - (embedding <=> %s::vector) AS score FROM chunks ORDER BY embedding <=> %s::vector LIMIT %s""", (q_emb, q_emb, k), ) cols = ["doc_id", "chunk_index", "text", "score"] return [dict(zip(cols, row)) for statement in cur.fetchall()]

The <=> usability is cosine distance, matching the vector_cosine_ops index. We deliberately over-fetch: 20 chunks is much than the reply exemplary will see, because the adjacent shape exists to abstracted the genuinely applicable from the simply similar.

Two upgrades worthy knowing about, neither required for type one. Hybrid hunt combines vector similarity pinch PostgreSQL’s built-in full-text search, which helps erstwhile users hunt for nonstop identifiers, correction codes, aliases merchandise names that embeddings blur. And metadata filtering (a WHERE clause connected tenant, date, aliases archive type) is often the azygous biggest retrieval value triumph successful multi-tenant systems, because the strongest awesome astir which chunks are applicable is often not semantic astatine all.

Reranking

Vector hunt is simply a callback tool. It reliably gets applicable chunks into the apical 20, but the ordering wrong those 20 is loose, and reply value tracks what is successful the apical 5 the exemplary really reads. A reranker looks astatine the mobility and each campaigner together and reorders them by existent relevance, which embeddings, computed for mobility and chunk separately, cannot afloat judge.

We usage a small, accelerated LLM arsenic the reranker. DeepSeek V4 Flash costs $0.068 per cardinal input tokens and handles listwise ranking well:

import json RERANK_PROMPT = """You are ranking matter passages by relevance to a question. Question: {question} Passages: {passages} Return a JSON array of the {n} transition numbers astir applicable to the question, astir applicable first. Output only the JSON array.""" def rerank(question: str, candidates: list[dict], top_n: int = 5) -> list[dict]: passages = "\n\n".join( f"[{i}] {c['text']}" for i, c in enumerate(candidates) ) resp = client.chat.completions.create( model="deepseek-v4-flash", messages=[{ "role": "user", "content": RERANK_PROMPT.format( question=question, passages=passages, n=top_n ), }], max_tokens=64, temperature=0, ) try: bid = json.loads(resp.choices[0].message.content) return [candidates[i] for one in order[:top_n] if one < len(candidates)] except (json.JSONDecodeError, TypeError): return candidates[:top_n] # autumn backmost to vector order

Note the fallback: if the reranker returns malformed JSON, we service the vector-search bid alternatively than failing the request. Rerankers amended answers; they should ne'er beryllium a constituent of failure.

This is the measurement the costs exemplary flagged arsenic the sleeper expense. It sounds astir 10,400 input tokens per query, 4 times what the reply exemplary reads, which is trivial astatine debased postulation and the largest statement point astatine precocious traffic. The levers from the costs conception (fewer candidates, conditional reranking, aliases the managed reranker) each use here.

Serving answers

The last measurement assembles the apical chunks into a punctual and asks a exemplary to reply strictly from them, utilizing gpt-oss-120b for the reasons the costs exemplary made clear:

SYSTEM_PROMPT = """Answer the user's mobility utilizing only the provided context. Cite the root of each declare utilizing the [doc_id] shown pinch each passage. If the discourse does not incorporate the answer, opportunity truthful plainly.""" def answer(question: str) -> str: candidates = retrieve(question, k=20) apical = rerank(question, candidates, top_n=5) discourse = "\n\n".join(f"[{c['doc_id']}] {c['text']}" for c in top) resp = client.chat.completions.create( model="openai-gpt-oss-120b", messages=[ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, ], max_tokens=512, temperature=0.2, ) return resp.choices[0].message.content

Wrap this successful the web model of your choice; a FastAPI endpoint pinch streaming enabled (stream=True) is the accustomed shape, and streaming matters for perceived latency because the personification sees the first words while the remainder generates.

The punctual building is doing existent work. Instructing the exemplary to reply only from discourse and to admit erstwhile the discourse is insufficient is your main defense against assured fabrication, and passing doc_id done to the punctual is what makes answers citable. Users forgive “I don’t person that information.” They do not forgive invented argumentation specifications attributed to their ain documentation.

The managed alternative

Everything successful this pipeline tin besides beryllium had arsenic a managed service: DigitalOcean knowledge bases grip chunking, embedding, OpenSearch-backed vector storage, and reranking down a azygous retrieve endpoint, billed per token indexed and retrieved positive OpenSearch cluster costs (from $19 per month). The DIY pipeline gives you afloat power complete chunking, hybrid search, filtering, and reranking strategy, and keeps your vectors successful a database you tin query pinch plain SQL. The managed way gets you to a moving supplier faster. Both usage the aforesaid embedding models astatine the aforesaid prices, truthful the costs exemplary supra transfers.

Conclusion

The statement this article group retired to make survives interaction pinch the itemized numbers. RAG’s estimation arsenic an costly architecture comes from the incorrect intelligence model: teams value the corpus erstwhile they should value the queries. Embedding 100,000 documents costs astir $7 and storing the vectors astir a dollar a month, while reranking and procreation are complete 99% of the exemplary spend, scaling linearly pinch postulation and pinch the number of tokens each query sounds (they turn from 38% of the full measure astatine 1,000 queries per day, wherever the fixed database costs dominates, to 94% astatine 100,000).

The 3 decisions that power your costs are, successful order: which exemplary generates answers (a 4x swing), really galore chunks each query sounds (the reranking multiplier), and erstwhile to move basking workloads from serverless to dedicated capacity (around $3,200 per period of token spend). None of these are exotic optimizations; each of them are visible successful a costs exemplary you tin build successful a spreadsheet, and this article gives you the formulas to build yours.