Making an AI Agent Feel Fast on Serverless Inference

Every squad building an supplier has entree to the aforesaid models. Very fewer agents consciousness quick. The quality is not the model. It is the engineering astir it: really overmuch discourse you send, what runs successful parallel, what the personification sees while the supplier works, and wherever each portion of the pipeline lives.

This is simply a build log pinch measurements. The supplier down it uses mean parts connected purpose: an open-source supplier model (Hermes, from Nous Research), a mini big instrumentality successful the people of a $12 Basic Droplet, and each exemplary telephone served by DigitalOcean’s Inference Engine done Serverless Inference. The supplier itself, Foodie, takes 1 Telegram connection (“2 days successful Bangalore, biryani purist, mini budget”) and returns a researched eating itinerary positive a narrated audio tour. The nutrient is conscionable the trial case. The useful portion is everything astir it: 2 velocity metrics that get confused for each other, the levers that move each 1 connected serverless inference, and a clear reply to erstwhile serverless stops being the correct choice.

The trial system

Telegram (long polling, nary webhook, nary nationalist IP) │ ▼ Hermes Gateway ── runs arsenic a launchd/systemd service │ ▼ Hermes Agent loop ├── SOUL.md ................ characteristic + intake rules ├── foodie-trail accomplishment ..... workflow: intake → investigation → scheme → narrate ├── Linkup (MCP, stdio) .... unrecorded web research └── text_to_speech instrumentality .... ElevenLabs → Telegram sound notes │ ▼ DigitalOcean Inference Engine (Serverless Inference) https://inference.do-ai.run/v1 (OpenAI-compatible, 70+ models)

Notice that the strategy has 2 very different jobs, and they request very different hardware. The supplier programme itself is small: it receives a message, calls immoderate APIs, and passes matter around. Any inexpensive instrumentality tin do that. The costly occupation is moving the model, and that ne'er touches the agent’s machine, because each exemplary telephone goes retired to the Inference Engine and gets billed per token. This is why the setup costs truthful small to run: you salary a fewer dollars a period for the mini machine, and for GPU clip only successful the seconds the exemplary is really working.

Two things astir this setup matter for everything that follows. First, the supplier tin tally almost anyplace because the exemplary does not tally pinch it. Telegram bots usage agelong polling, truthful thing needs to scope the instrumentality from outside. The supplier useful the aforesaid connected a laptop aliases a Droplet. Second, each exemplary telephone goes done 1 OpenAI-compatible endpoint. The Inference Engine serves models from Anthropic, OpenAI, Meta, DeepSeek, Qwen, and others down https://inference.do-ai.run/v1 pinch a azygous entree key. Changing models aliases providers is simply a config change, not a migration.

Setup, briefly

The step-by-step setup lives successful DigitalOcean’s Serverless Inference docs and the Hermes docs, truthful this conception covers only the essentials and the communal mistakes.

Create a exemplary entree cardinal successful the power sheet (under Serverless Inference) and cheque it useful earlier thing depends connected it:

curl https://inference.do-ai.run/v1/models \ -H "Authorization: Bearer $DO_MODEL_KEY"

This returns the unrecorded model catalog. Trust it complete immoderate blog post, including this one, because catalogs change. From that consequence connected this build: anthropic-claude-5-sonnet reports a 1,000,000-token discourse and 128,000-token max output, and anthropic-claude-haiku-4.5 is disposable arsenic a cheaper action for testing.

Three rules for picking a exemplary for supplier work.

Pick for tool-calling reliability, not benchmark scores; an supplier that fumbles usability calls invents restaurants alternatively of researching them.

Check the max output limit; respective beardown catalog models headdress astatine 8K tokens, and a cut-off itinerary fails successful a confusing way. Keep a inexpensive exemplary for iteration, because you will re-run trial prompts dozens of times.

Also, skip the catalog’s router:* entries while debugging; you want the aforesaid exemplary connected each run.

Wiring the framework: instal Hermes, constituent it astatine the endpoint arsenic a civilization OpenAI-compatible supplier (pick Chat Completions mode explicitly alternatively of auto-detect), and link Telegram done the built-in gateway (Hermes Telegram docs). Two traps from this build are worthy knowing: aft switching models successful chat, /model <id> --global is required aliases the gateway starts pinch the aged model, and since the bot tin tally commands connected its host, support the default deny-unknown-users behaviour and let only your ain Telegram ID.

Research and sound are configurations, and some choices are worthy explaining.

Linkup, because edifice information goes old fast, and the supplier needs a hunt API that returns existent results pinch sources, built for instrumentality use. Any akin hunt API would slot successful the aforesaid way. MCP alternatively of a hand-written integration, because Hermes finds the server’s devices astatine startup and registers them for illustration its own. Swapping hunt providers later is simply a config change. In ~/.hermes/config.yaml:

mcp_servers: linkup: command: "npx" args: ["-y", "linkup-mcp-server"] env: LINKUP_API_KEY: "${LINKUP_API_KEY}" supports_parallel_tool_calls: true tts: provider: "elevenlabs" elevenlabs: voice_id: "pNInz6obpgDQGcFmaJgB" model_id: "eleven_multilingual_v2"

Keys unrecorded successful ~/.hermes/.env. The ${LINKUP_API_KEY} placeholder is filled successful astatine runtime, truthful secrets enactment retired of the config record you mightiness share. And supports_parallel_tool_calls: existent tells the model the searches are safe to tally astatine the aforesaid time. That 1 statement becomes the biggest velocity triumph below.

The merchandise is 2 Markdown files

Everything supra is plumbing anyone tin copy. The agent’s behaviour lives successful 2 files, and the choices wrong them are besides velocity choices.

SOUL.md sets the personality, and the characteristic is simply a merchandise requirement. Foodie profiles the personification earlier readying the travel (street-food hunter, biryani purist, vegetarian, café hopper, good dining), guesses the floor plan from clues erstwhile it can, and asks astatine astir 1 mobility erstwhile it cannot. The one-question limit helps successful 2 ways. People suffer patience and discontinue erstwhile a bot keeps asking questions earlier doing anything. And each mobility makes the full tally slower: the supplier has to hold for the personification to reply, past process that reply pinch different exemplary call, earlier immoderate existent activity starts.

The foodie-trail accomplishment sets the workflow, and its astir important lines are limits. At astir six searches per itinerary, pinch the queries written retired successful the skill. No edifice successful the output unless a hunt consequence confirmed it. Each time is simply a walkable way done 1 neighborhood. Narration scripts are 60 to 90 seconds, written to beryllium spoken. The hunt limit matters beyond cost: an supplier pinch nary limit keeps searching to settee its ain doubts, and each other information adds a afloat search-plus-model cycle. Writing the queries into the accomplishment turns an open-ended hunt into a fixed group of steps that tin tally successful parallel.

Making it fast

Two numbers style really the supplier feels, and they are improved successful different ways. Time to first token (TTFT) is really agelong earlier the personification sees anything. Total procreation clip is really agelong earlier the afloat consequence arrives. Mixing them up is the astir communal correction successful this benignant of work.

Time to first token

For a azygous exemplary call, TTFT is mostly web clip positive prefill: the clip the exemplary spends reference your input earlier it writes anything. Prefill grows pinch input length. So TTFT is, much than anything, a mobility of really overmuch you are sending.

Watch what enters the discourse connected each turn. An supplier model softly stacks the strategy prompt, characteristic file, memory, accomplishment list, and chat history into each request. Hermes keeps accomplishment descriptions short and only loads a afloat accomplishment record erstwhile it is needed, which helps. Do your part: support the characteristic record short (Foodie’s is nether 400 words), support accomplishment descriptions brief, and fto conversations reset (this build resets aft 4 hours idle). A agelong speech carrying 30K tokens of history adds existent hold clip compared to a 6K one, truthful the framework’s compression bid and idle resets prevention clip arsenic overmuch arsenic money.

Match reasoning clip to the task. Reasoning models walk their first seconds connected reasoning the personification ne'er sees. For a move that conscionable sounds a metropolis and a sensation floor plan retired of a message, that is wasted waiting. Hermes tin set this astatine runtime (/reasoning low). The pattern: readying steps merit reasoning time, acknowledgments do not.

Send a speedy first line. The instrumentality that matters astir for a slow task: person the accomplishment nonstop 1 statement (“Biryani purist, Bangalore, 2 days. Researching now.”) earlier its first search. The itinerary arrives nary sooner, but the app stops emotion frozen. That is astir of what users mean by fast.

Keep connections warm. In production, put the agent’s big successful a region adjacent the conclusion endpoint, and fto the long-running gateway reuse its connections. The aforesaid goes for tools: npx starts the Linkup server acold connected first use, truthful group the framework’s recycling options generously for devices you request quickly. The first hunt of the time should salary the cold-start cost, not the first hunt of each conversation.

Use a mini exemplary for inheritance jobs. Hermes sends broadside tasks for illustration naming conversations and compressing history to a separately configured model. Point those astatine a Haiku-class exemplary truthful slow calls you ne'er spot enactment retired of the way.

Total procreation time

The afloat tally is simply a pipeline: publication the request, search, constitute the itinerary, constitute the narration, render the audio, present it all. The gains travel from reshaping the pipeline, not from speeding up immoderate azygous call.

Run the searches successful parallel. The biggest azygous win. Six searches astatine 3 to 5 seconds each costs 20 to 30 seconds tally 1 aft another. Marked safe to tally together, they return astir arsenic agelong arsenic the slowest azygous search. The rule: immoderate read-only instrumentality your supplier calls much than doubly per task should beryllium marked parallel-safe, and the queries should beryllium written truthful they do not dangle connected each other, which is what lets them tally together.

Cap the loop and constitute retired the queries. You cognize really agelong investigation takes because you decided really agelong it takes.

Send the matter first, render audio successful the background. The personification sounds Day 1 while its audio renders. As acold arsenic they are concerned, the reply arrived erstwhile the matter did. Per-day audio files hit 1 large record for 2 reasons: they overlap pinch reference time, and a text-to-speech nonaccomplishment costs 1 day’s audio alternatively of everything.

Keep the output short connected purpose. Generation clip grows pinch output length, truthful wordiness is slowness. The skill’s format norm (each time nether astir 15 lines) astir halves penning clip compared to what the exemplary produces without instruction, and it sounds amended connected a telephone anyway. When you do request agelong output, make judge the model’s output limit fits it successful 1 pass. Hitting the limit and continuing successful a 2nd telephone is the slowest measurement to make text.

Measure earlier you tune. The bottleneck is seldom wherever you assume. On this build it was audio upload connected a location connection, not the model.

When to move disconnected serverless inference

This supplier runs connected serverless inference, and astatine this standard that is not a compromise. It is the correct choice. Hosting it connected a GPU instrumentality would mean renting an H100 GPU Droplet astatine astir $3.39 per hour, astir $2,500 a month, to salary for a GPU that sits idle while each the existent activity happens per token connected the Inference Engine. The $12 Droplet does the aforesaid job.

Self-hosting starts to make consciousness successful 4 situations. The first is cost: erstwhile your monthly token measure gets adjacent to the level costs of renting a GPU, the mathematics flips, because a self-hosted open-weight exemplary costs the aforesaid whether it serves 1 petition aliases a million. The 2nd is information control: if prompts cannot time off your infrastructure for compliance aliases privateness reasons, per-token APIs are disconnected the array nary matter the price. The 3rd is simply a civilization model: if you person fine-tuned weights for your domain, you request location to service them. The 4th is latency: an API gives you emblematic performance, while your ain endpoint gives you a level you control.

The downsides are conscionable arsenic real. The fixed costs runs whether postulation comes aliases not, and GPU Droplets measure moreover while powered off. You return connected moving the serving stack yourself: vLLM aliases similar, exemplary updates, scaling, monitoring. And open-weight models still way frontier models connected tool-calling reliability, which matters much for agents than for astir workloads. An supplier that saves money connected tokens but fumbles usability calls is simply a bad trade.

There is simply a mediate measurement that skips astir of the operational cost: the Inference Engine’s Dedicated Inference tier provides a dedicated GPU endpoint pinch bring-your-own-model support, managed by the platform. For astir teams, the sensible way is serverless first, dedicated erstwhile the numbers aliases compliance rules request it, and afloat self-hosted only erstwhile you request power the managed options cannot give. Because the full stack speaks the aforesaid API, each move is simply a guidelines URL change, not a rewrite.

What this build says astir the stack

The supplier stack is settling into a acquainted shape: an open-source runtime you install, a managed exemplary work you rent, extracurricular abilities for illustration hunt and sound attached done a shared protocol, and connected top, the bladed furniture that is really yours. In this build that furniture was 2 Markdown files of astir a 1000 words, and it was also, clearly, the product.

Where your models tally is becoming a normal infrastructure decision, made connected latency, exemplary choice, and price, and easy to reverse because the OpenAI-compatible API has go the shared standard. And speed, not exemplary quality, is now wherever products separate. The levers successful this portion are mean engineering habits: nonstop little context, tally searches successful parallel, present matter earlier audio, usage mini models for inheritance work, measurement earlier tuning. The exemplary is the easy portion now. The acquisition is the work.

---

Stack: Hermes Agent (MIT), DigitalOcean Inference Engine — Serverless Inference, Basic Droplets, Linkup, ElevenLabs.