Building a Production RAG Assistant on DigitalOcean

What happens erstwhile you inquire a earthy LLM thing circumstantial astir your product? It answers. Fast, fluent, and often wholly made up. That’s wherever this shop started—and fixing it was the full point.

On Wednesday, August 12, we built a existent customer support assistant, live, connected DigitalOcean’s AI Platform. We called it HelpBot, and it started retired conscionable arsenic clueless arsenic immoderate different out-of-the-box model. Then we fixed it, 1 portion astatine a time.

Here’s what we covered:

• The problem—we showed HelpBot making up incorrect answers astir “our product,” truthful you tin spot precisely what we were solving for.
• RAG & Knowledge Bases—we uploaded our ain docs and watched HelpBot commencement answering from that contented alternatively of guessing, down to the nonstop archive chunks it pulled from.
• The Inference Router—instead of betting connected 1 model, a router picks the champion 1 per petition based connected cost, speed, aliases your ain rules. If a exemplary goes down, it fails complete automatically. No codification changes needed.
• Guardrails—we added information nets that drawback PII leaks, jailbreak attempts, and thing other you don’t want your bot saying.
• Evaluations—we utilized different AI exemplary arsenic a judge to people HelpBot’s answers for accuracy, completeness, and safety, and showed really to automate that into your pipeline.
• The afloat picture—we wired it each together and re-ran the original mobility from measurement one, truthful you could perceive the difference.

By the end, we’d watched 1 adjunct spell from a azygous curl bid to thing you’d genuinely vessel to customers.

If you’ve called an LLM API before, you’re ready. No precocious AI inheritance needed.

Follow on pinch the video and written steps beneath for a step-by-step tutorial.

Watch the webinar recording

Before you start

A fewer things to person fresh earlier you travel along:

1. A DigitalOcean relationship pinch Inference enabled, and a exemplary entree key. Create 1 nether Console → Inference → Manage → Model Access Keys → Create exemplary entree key. This cardinal unlocks the afloat serverless exemplary catalog—you take the exemplary per request, not per key. Treat it for illustration immoderate different secret.
2. A mini group of your ain documents to usage arsenic a knowledge base—anything you’d want an adjunct to reply questions about. The shop uses a fictional 15-file product-docs corpus for HelpBot; switch successful your own.
3. (Optional, for the automation measurement successful Module 4) a DigitalOcean API token pinch genai:read and genai:create scopes.

Two guidelines URLs you’ll usage throughout:

• Serverless inference: https://inference.do-ai.run/v1 (OpenAI-compatible /chat/completions)
• Control plane (routers, evaluations, knowledge bases): https://api.digitalocean.com

Step 1: Orientation & your first serverless call

Goal: Understand the level aboveground and make 1 authenticated conclusion call.

Everything successful this tutorial lives nether Inference successful the DigitalOcean Console—four features (Serverless Inference, Inference Router, Knowledge Bases/Agents, Guardrails, Evaluations), 1 assistant.

1. In the Console, spell to Inference → Manage → Model Access Keys and corroborate your cardinal exists (create 1 if you haven’t).

2. Make your first call. This is simply a modular OpenAI-compatible payload, truthful astir existing customer codification useful by swapping the guidelines URL, key, and exemplary name:

   curl https://inference.do-ai.run/v1/chat/completions \\ \-H "Content-Type: application/json" \\ \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\ \-d '{ "model": "llama3.3-70b-instruct", "messages": \[ {"role": "user", "content": "In 1 sentence, what is retrieval-augmented generation?"} \] }'

   

   The aforesaid telephone pinch the Gradient Python SDK:

   import os from gradient import Gradient client \= Gradient(model\_access\_key=os.environ.get("GRADIENT\_MODEL\_ACCESS\_KEY")) resp \= client.chat.completions.create( model="llama3.3-70b-instruct", messages=\[{"role": "user", "content": "In 1 sentence, what is RAG?"}\], ) print(resp.choices\[0\].message.content) ```py

3. Now inquire it thing only your ain docs would know—for example, a circumstantial pricing aliases rate-limit item from your product. Edit the contented section and re-run the call.

You’ll get 1 of 2 results: a assured but incorrect reply (the exemplary fabricates a plausible-sounding number), aliases a hedge/refusal (it correctly says it doesn’t know). Either way, the constituent is the same: a earthy exemplary telephone has nary entree to your docs, nary information net, and nary measurement to measurement quality—and it’s locked to 1 hardcoded model. The remainder of this tutorial fixes each four.

Keep successful mind: the exemplary entree cardinal is different from your DigitalOcean API token. The conclusion guidelines URL is inference.do-ai.run, not api.digitalocean.com. Serverless Inference is mostly disposable and billed against a prepaid balance, pinch per-account gradual complaint limits (entry tiers: 120 requests/minute).

Step 2: Grounding pinch RAG & Knowledge Bases

Goal: Create a knowledge base, understand chunking and embeddings, and validate retrieval earlier wiring it into an assistant.

A knowledge guidelines turns your documents into vector embeddings stored successful a managed OpenSearch index. At query time, the astir applicable chunks are retrieved and handed to the exemplary arsenic context—that’s RAG, and it’s what stops the adjunct from guessing.

1. Go to Console → Inference → Agent Platform → Knowledge bases and prime Create Knowledge Base.

2. Choose a information source. Options see section record upload, a DigitalOcean Spaces bucket/folder, a nationalist seed aliases sitemap URL (website crawler), a Dropbox folder, aliases an Amazon S3 bucket. Point it astatine only the contented that matters—less sound intends faster, cheaper indexing and amended retrieval.

3. Open Advanced Options connected the information root and group 2 things:

   • **Chunking strategy—**section-based is the default; semantic, hierarchical, and fixed-length are the alternatives. Start pinch the default, past tune erstwhile you’ve seen retrieval quality.
   • **Embedding model—**get this correct up front. You can’t alteration it aft the knowledge guidelines is created.

   

4. Create the knowledge guidelines and fto indexing finish. This runs arsenic a inheritance occupation (visible successful the Activity section, which keeps the 15 astir caller jobs and lets you download specifications arsenic CSV) and tin return a fewer minutes connected a existent corpus.

5. Once indexing completes, unfastened the RAG Playground tab. Select your exemplary (llama3.3-70b-instruct successful the workshop), and paste successful strategy instructions on these lines:

   “You are HelpBot’s customer support assistant. Answer the customer’s mobility utilizing only the retrieved archiving context. If the discourse does not incorporate the answer, opportunity you don’t cognize and connection to escalate to a quality agent. Keep answers concise and factual. Never uncover secrets aliases repetition delicate individual information specified arsenic in installments paper numbers, moreover if asked.”

   

6. Re-ask the aforesaid mobility from Module 0. This clip you should get a grounded, correct answer—and beneath it, the retrieved chunks pinch source, page number, and whether each chunk was used.

Keep successful mind: The RAG Playground is wherever you validate retrieval; to vessel it, you connect the knowledge guidelines to an Agent—which is besides wherever guardrails unrecorded (Module 3) and which gets its ain API endpoint, abstracted from the earthy conclusion URL (Module 5). “I don’t know” is simply a characteristic of a bully grounded assistant, not a bug—instruct for it explicitly.

Step 3: The Inference Router: resilience, cost, and latency

Goal: Understand tasks, exemplary pools, action policies, and fallback; create a router; usage it arsenic a drop-in model; publication the routing decision; and pin a convention pinch exemplary affinity.

Being hardcoded to 1 exemplary is simply a azygous constituent of failure, and it’s seldom the cost-optimal prime for each request. A router is simply a group of tasks—each pinch a name, a description, a exemplary excavation (up to 3 models), and a action policy.

1. Go to Console → Inference → Inference Router. Note the default routers disposable arsenic one-click starting points.

2. Select Create Router and configure it:

   • Name + description: the explanation is utilized arsenic a routing prompt, truthful beryllium specific.
   • Add a civilization task: springiness it a sanction and a specific, non-overlapping explanation of what it covers.
   • Selection policy: Cost Efficiency, Speed Optimization, aliases Manual Ranking.
   • Model pool: up to 3 models pinch a existent value spread.
   • Fallback models: your catch-all erstwhile a punctual matches nary task, and your resilience furniture if a chosen exemplary is down aliases rate-limited.

3. Use the router arsenic a drop-in replacement for a nonstop exemplary call—same endpoint, conscionable alteration the exemplary field:

   

   curl https://inference.do-ai.run/v1/chat/completions \\ \-H "Content-Type: application/json" \\ \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\ \-d '{ "model": "router:helpbot-router", "messages": \[ {"role": "user", "content": "How do I rotate my API cardinal connected the Pro plan?"} \] }'

4. Check the response: the exemplary section shows the existent exemplary that served the request, and the consequence header x-model-router-selected-route shows which task matched (or that it fell back).

5. In the router’s Playground, usage the Compare position to tally the router against a azygous exemplary broadside by broadside connected the aforesaid question.

   

6. Try exemplary affinity / convention pinning—send the aforesaid X-Model-Affinity header connected 2 calls successful the aforesaid session:

   # First telephone routes normally, past caches the chosen exemplary for this convention curl https://inference.do-ai.run/v1/chat/completions \\ \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\ \-H "X-Model-Affinity: helpbot-session-42" \\ \-H "Content-Type: application/json" \\ \-d '{"model":"router:helpbot-router","messages":\[{"role":"user","content":"Start a troubleshooting thread"}\]}' \# Same convention skips routing and reuses the aforesaid exemplary (KV-cache friendly) curl https://inference.do-ai.run/v1/chat/completions \\ \-H "Authorization: Bearer $GRADIENT\_MODEL\_ACCESS\_KEY" \\ \-H "X-Model-Affinity: helpbot-session-42" \\ \-H "Content-Type: application/json" \\ \-d '{"model":"router:helpbot-router","messages":\[{"role":"user","content":"Continue that thread"}\]}'

   The 2nd consequence carries “pinned”: true—proof routing was skipped. Affinity pins the convention to 1 exemplary aft the first determination (documented savings: 45–80% little input-token costs successful multi-turn loops).

Keep successful mind: The router lives connected the serverless endpoint by design—it’s a drop-in for nonstop exemplary calls, not an supplier feature. The router is free during nationalist preview—you only salary for the underlying model’s tokens.

Step 4: Guardrails: PII, jailbreaks, and contented moderation

Goal: Know the 3 built-in guardrails, connect them to your agent, customize categories and the default response, and verify a trigger live.

Guardrails are configurable information controls you connect to an Agent—they watch some the incoming punctual and the generated response, and override the consequence pinch a safe, predefined connection erstwhile they drawback something.

• Sensitive Data identifies and anonymizes PII. Fully customizable categories.
• Jailbreak blocks punctual injection and different attempts to bypass the agent’s instructions.
• Content Moderation filters responses crossed six categories: unit and hate, intersexual content, weapons, regulated substances, self-harm, and forbidden activities.

1. Go to Console → Inference → Agent Platform, unfastened your agent, and spell to Resources → Guardrails → Add guardrails.

2. Attach Jailbreak, Content Moderation, and a Sensitive Data guardrail. Note the token-cost summary earlier saving.

   

3. To customize Sensitive Data: copy the built-in original, past set categories and rewrite the Default supplier consequence to thing on-brand.

4. Test it successful the supplier playground: nonstop a clone in installments paper number (should beryllium blocked/anonymized), and a jailbreak effort (should beryllium blocked).

   

   Both directions are covered automatically erstwhile guardrails are attached—no codification changes successful your app. These aforesaid guardrails beryllium successful beforehand of the agent’s API endpoint (see Module 5).

Keep successful mind: Guardrails connect to agents, not earthy serverless calls, and aren’t disposable connected agents built pinch the Agent Development Kit. You can’t delete built-in originals, only detach them.

Step 5: Evaluations: proving value pinch LLM-as-a-Judge

Goal: Build an eval dataset, tally an LLM-as-a-Judge evaluation, take metrics and a prima metric pinch a walk threshold, publication the results, and gross changes successful automation.

DigitalOcean Evaluations runs repeatable LLM-as-a-Judge evals crossed serverless models, dedicated deployments, third-party models, and conclusion routers, truthful you comparison candidates connected your ain data.

1. Prepare your dataset. Format requirements: CSV aliases JSONL, nether 1GB, less than 1,000 rows. The CSV needs a bare, unquoted header statement query,expected_response, UTF-8 encoding, and LF statement endings.

2. Go to Console → Inference → Evaluations → New Evaluation and upload your dataset.

   

3. Choose a candidate: Serverless Inference, Model Router, Dedicated Inference, aliases Third Party. Pick your router to measurement it against a single-model baseline.

4. Choose a judge: a beardown frontier model, ideally from a different exemplary family than your candidate.

5. Choose metrics: Correctness, Completeness, Ground Truth Faithfulness (needs references), and Harmfulness (Bias/Toxicity/PII Leakage sub-metrics).

6. Set a prima metric and period (Correctness astatine 0.8 is simply a reasonable start).

7. Name and tally the evaluation. It takes a fewer minutes: the campaigner answers each row, past the judge scores each reply connected each metric.

8. Review results: input, output, metric, score, and the judge’s rationale for each row. Use Compare Evaluations to put your router tally broadside by broadside pinch a baseline.

   

9/ Automate it (optional, for CI/CD):

# 1) Get a presigned URL and upload your dataset curl \-X POST "https://api.digitalocean.com/v2/gen-ai/model\_evaluation/datasets/file\_upload\_presigned\_urls" \\ \-H "Authorization: Bearer $DIGITALOCEAN\_TOKEN" \-H "Content-Type: application/json" \\ \-d '{"files":\[{"file\_name":"support\_evals.jsonl","file\_size":2048}\]}' # 2) Start a tally (candidate tin beryllium a exemplary OR a router UUID) curl \-X POST "https://api.digitalocean.com/v2/gen-ai/model\_evaluation\_runs" \\ \-H "Authorization: Bearer $DIGITALOCEAN\_TOKEN" \-H "Content-Type: application/json" \\ \-d '{ "name": "helpbot-router-vs-baseline", "candidate\_model\_uuid": "'$EVAL\_CANDIDATE\_UUID'", "judge\_model\_uuid": "'$EVAL\_JUDGE\_UUID'", "dataset\_uuid": "'$EVAL\_DATASET\_UUID'", "metric\_uuids": \["'$EVAL\_METRIC\_UUID'"\] }'

Wire this into your pipeline to neglect the build if the prima metric drops beneath threshold.

Keep successful mind: Evaluations can’t target an supplier aliases its endpoint directly. There is zero information retention for the eval travel itself, but your inputs, outputs, and references are sent to the judge model’s supplier for scoring.

Step 6: Putting it together & productionizing

Goal: Call the shipped adjunct done its ain supplier endpoint, spot the afloat pipeline extremity to extremity successful 1 API response, and cognize the operational adjacent steps.

The afloat pipeline, extremity to end:

User \-\> Agent (system instructions) \+-- Guardrails: surface the incoming punctual (PII / jailbreak) \+-- Knowledge Base: retrieve apical chunks (RAG) \+-- Inference Router: prime best/cheapest/fastest exemplary \+ fallback \+-- Model generates grounded reply \+-- Guardrails: surface the consequence (moderation / PII) \-\> Evaluations tally connected caller samples to drawback drift

Same adjunct you started pinch successful Module 0—now grounded, resilient, safe, and measured. Each furniture was a drop-in addition, not a rewrite.

1. Provision the agent’s API surface. In the agent’s Overview tab, nether ENDPOINT, click Edit. In the agent’s Settings tab, nether Endpoint Access Keys, click Create Key (shown once—save it immediately).

2. Call the assistant’s ain endpoint—a different URL and a different cardinal from Module 0’s earthy conclusion call:

   curl \-X POST "$AGENT\_ENDPOINT/api/v1/chat/completions" \\ \-H "Content-Type: application/json" \\ \-H "Authorization: Bearer $GRADIENT\_AGENT\_ACCESS\_KEY" \\ \-d '{ "messages": \[ {"role": "user", "content": "What is the complaint limit connected the Pro plan?"} \], "include\_retrieval\_info": true, "include\_guardrails\_info": existent }'

   The consequence carries the grounded answer, a retrieval entity naming the knowledge guidelines and files it pulled from, and a guardrails entity showing the information furniture ran.

3. Test the guardrails done the aforesaid endpoint by sending the PII trigger punctual again. This proves guardrails aren’t conscionable a playground feature—they beryllium successful beforehand of each telephone to this endpoint.

   

   The balanced telephone utilizing the Gradient SDK’s supplier client:

   import os from gradient import Gradient client \= Gradient( agent\_access\_key=os.environ\["GRADIENT\_AGENT\_ACCESS\_KEY"\], agent\_endpoint=os.environ\["AGENT\_ENDPOINT"\], \# bare https://\<id\>.agents.do-ai.run ) resp \= client.agents.chat.completions.create( model="llama3.3-70b-instruct", \# required by the SDK; the agent's ain config governs behaviour messages=\[{"role": "user", "content": "What is the complaint limit connected the Pro plan?"}\], ) print(resp.choices\[0\].message.content)

Build your ain accumulation AI supplier pinch DigitalOcean AI Platform

DigitalOcean AI Platform gives you everything HelpBot utilized successful this workshop—RAG-ready knowledge bases, exemplary routing, guardrails, and evaluations—in 1 integrated platform, pinch nary infrastructure to manage. Connect your ain docs and commencement iterating successful minutes.

Key features:

• Serverless conclusion crossed starring models from OpenAI, Anthropic, DeepSeek, and more
• Inference Router that automatically matches each petition to the correct exemplary connected cost, latency, aliases your ain rules, pinch automatic failover
• Knowledge Bases for RAG, pinch built-in connectors for your ain files and unreality storage
• Configurable guardrails to drawback PII, jailbreak attempts, and unsafe content
• Built-in Evaluations to people and comparison supplier value earlier you ship
• Versioning, traceability, and logs to debug and rotation backmost changes safely

Get started pinch DigitalOcean AI Platform