Process a Million Documents Overnight: Batch Inference End-to-End

Say you person 1 cardinal support tickets sitting successful entity storage, and by tomorrow greeting you request each 1 classified by rumor type and summarized successful a paragraph. Sending them done a real-time chat completions API 1 astatine a clip is the evident first idea, and it is the incorrect one. At modular complaint limits the requests unsocial return much than a day, you salary afloat value for each token, and 1 web nonaccomplishment astatine 3 AM tin extremity your book halfway through.

Here is the statement this article makes: astir LLM workloads are not conversations. Sorting documents into categories, summarizing them, adding tags aliases fields to aged records, moving evaluations, moving done a backlog: these are throughput problems, and the interactive API everyone reaches for first is simply a latency-optimized instrumentality being applied to a throughput job. Teams that dainty the real-time endpoint arsenic the only endpoint salary astir double for the privilege of an reply cipher is waiting for. Batch inference, an execution exemplary wherever you package each cardinal requests into files, manus them to the platform, and cod results erstwhile the occupation finishes, is the instrumentality really built for this style of work, and it should beryllium the default for immoderate occupation wherever the deadline is simply a clip of time alternatively than a number of seconds.

To beryllium clear astir what this portion is up front: a cost-modeling methodology and a moving pipeline. The batch costs exemplary is built from documented limits and database prices. The nonaccomplishment modes and model-selection numbers are measured: we ran this article’s punctual building against existent documents (news articles arsenic stand-ins, pinch a matching class list) complete the serverless real-time API, twice, and study what we found. The million-ticket occupation is simply a worked illustration chosen to make each number concrete, and each calculation is shown truthful you tin rerun the exemplary pinch your ain archive counts, token sizes, and exemplary choices. Model prices change; cheque DigitalOcean’s Inference Pricing page earlier budgeting a existent job.

The occupation we are running

To support the numbers concrete, present is the project:

• 1,000,000 plain-text documents, averaging astir 1,200 tokens each (roughly 900 words)
• For each document: delegate 1 of 8 categories and constitute a 3-4 condemnation summary
• Deadline: results fresh the adjacent morning
• Model: GPT-5 mini done DigitalOcean’s serverless inference platform, which offers it astatine $0.25 per cardinal input tokens and $2.00 per cardinal output tokens astatine real-time rates. Batch requests are billed astatine up to half those rates, for reasons the billing conception covers.

One creation determination up front: classification and summarization hap successful a azygous petition per document, not two. Two abstracted calls would double your petition count and re-send each archive twice. Instead, the punctual asks the exemplary to return 1 JSON entity containing some the class and the summary. That halves the measure and simplifies the pipeline, astatine the costs of a somewhat longer prompt.

With punctual instructions added, each petition carries astir 1,500 input tokens and produces astir 200 output tokens. Across a cardinal documents, that is 1.5 cardinal input tokens and 200 cardinal output tokens.

Before penning immoderate code, it is worthy checking whether the real-time API could moreover decorativeness successful time.

DigitalOcean’s serverless conclusion complaint limits astatine Tier 3 and Tier 4 are 600 requests per infinitesimal and astir 800K to 2M tokens per infinitesimal (see the relationship tier array connected the Inference Limits page). One cardinal requests astatine 600 per infinitesimal takes astir 27.8 hours, and that assumes zero retries and a book that ne'er falls behind. The token limit is conscionable arsenic restrictive: 1.7 cardinal full tokens astatine 2M tokens per infinitesimal is astir 14 hours of steady, perfectly timed requests.

You could technologist astir this: petition a tier increase, tune a complaint limiter, checkpoint progress, shard the work. Teams do, and it is usually wasted effort, because the complaint limits are not an obstacle here. They are a awesome that a latency-optimized API is the incorrect execution exemplary for a throughput job.

Batch conclusion avoids each of this. Batch jobs usage a abstracted quota (by default, you tin taxable up to 10 cardinal tokens per exemplary per account) and tally connected isolated capacity astatine little scheduling priority, truthful a moving batch occupation does not devour your real-time quota aliases degrade latency for your accumulation applications. You besides do not constitute a complaint limiter, a retry loop pinch backoff, aliases a checkpoint file. The level retries transient errors (429, 408, 5xx) up to 2 times pinch exponential backoff connected its own, and grounded requests onshore successful an correction record alternatively of crashing thing (see the batch conclusion conception of the Inference Features page).

The trade-off is speed. A batch occupation has a 24-hour completion window, and you person nary power complete erstwhile wrong that model your requests run. If immoderate portion of your workload needs an reply successful seconds, that portion stays connected the real-time API. Everything other is simply a campaigner for batch.

What you request earlier starting

Three prerequisites, each one-time setup:

• A DigitalOcean relationship astatine Tier 3 aliases higher. Tier 1 and Tier 2 accounts do not person entree to Anthropic models aliases OpenAI models (except the open-weight gpt-oss models), and batch conclusion only supports OpenAI and Anthropic commercialized models pinch matter prompts (see the Inference Limits page for tier exemplary entree and the batch conclusion limits for supported models).
• A affirmative prepaid balance. Serverless inference, including batch, is prepaid: usage is deducted from your balance, and if it hits $0 entree is suspended. For this job, load capable to screen the estimate successful the billing conception positive margin.
• A model entree key, created from the DigitalOcean power panel. Every API telephone beneath authenticates pinch it against the serverless conclusion guidelines URL, https://inference.do-ai.run.

One constraint to scheme around: each batch occupation uses a azygous model, and multi-model batch jobs are not supported (see batch conclusion limits). This is not the aforesaid arsenic DigitalOcean’s Inference Router, which picks a best-fit exemplary per request; that characteristic applies to real-time inference, not batch jobs. So if you want to nonstop easy documents to a cheaper exemplary and difficult ones to a stronger model, that is 2 abstracted batch jobs, not one.

Planning astir the limits

Batch conclusion has 3 limits that style really you divided a cardinal documents:

• Maximum 50,000 requests per input file
• Maximum 200 MB per input file
• Maximum 10 cardinal tokens submitted per exemplary per relationship (the default; you tin petition an increase)

At first glimpse the divided looks easy: 1,000,000 ÷ 50,000 = 20 files. But each record must besides enactment nether 200 MB, truthful activity done the numbers successful 3 steps.

Step 1: How large is 1 line? Each statement holds 1 document, the punctual instructions, and a mini JSON wrapper. One token is astir 4 characters.

Step 2: Which limit fills up first?

The size headdress fills up first: only astir 31,500 documents fresh successful a file, good beneath the 50,000-request cap. In this project, record size decides the split.

Step 3: Choose the split, pinch room to spare.

Forty files intends forty batch jobs, which sounds for illustration a batch but changes almost thing successful the code, because the submission and polling logic is simply a loop either way.

Last, cheque the token quota. The occupation needs 1.5 cardinal input tokens positive astir 200 cardinal for output, 1.7 cardinal total. That is good nether the 10 cardinal default, truthful each 40 jobs tin beryllium submitted astatine once.

Building the input files

Each statement of a batch input record is 1 self-contained request. For OpenAI-provider jobs connected DigitalOcean, the statement follows the OpenAI Batch API shape: a custom_id, a method, a URL, and the petition assemblage you would person sent to the real-time endpoint (see the input record format successful DigitalOcean’s batch conclusion guide).

Two specifications matter much than they look:

First, custom_id is the only cardinal that links a consequence backmost to its root document. Results do not travel backmost successful input order. Use your existent archive ID, ne'er an array scale that intends thing aft the database is re-sorted. Duplicate custom_id values wrong a record neglect validation, truthful a unchangeable unsocial ID solves some problems astatine once.

Second, headdress the output length. The summary should beryllium 3-4 sentences, truthful 500 tokens is plenty. Note the parameter names: GPT-5 models usage max_completion_tokens connected the chat completions endpoint, not the older max_tokens, and they do not judge a civilization temperature. The headdress besides covers the model’s soul reasoning tokens, truthful group reasoning_effort to minimal for a elemental task for illustration this; it keeps reasoning tokens adjacent zero and the output measure predictable. Without a cap, 1 overly agelong completion wastes output tokens, multiplied by nevertheless galore documents trigger it.

The pursuing book sounds archive records (ID positive text), builds the petition lines, and rolls complete to a caller record each 25,000 requests:

import json SYSTEM_PROMPT = ( "You categorize and summarize documents. Respond pinch a azygous JSON object: " '{"category": "<one of: billing, bug_report, feature_request, account, ' 'security, performance, documentation, other>", ' '"summary": "<3-4 condemnation summary>"} ' "The class worth must beryllium precisely 1 of the 8 listed strings, " "lowercase. Never invent different category; if unsure, usage \"other\"." ) CHUNK_SIZE = 25_000 def write_batch_files(documents, prefix="batch_input"): """documents yields (doc_id, text) tuples. Returns database of record paths.""" paths, out, count, portion = [], None, 0, 0 for doc_id, matter in documents: if count % CHUNK_SIZE == 0: if out: out.close() portion += 1 way = f"{prefix}_{part:03d}.jsonl" retired = open(path, "w", encoding="utf-8") paths.append(path) statement = { "custom_id": doc_id, # your existent archive ID "method": "POST", "url": "/v1/chat/completions", "body": { "model": "gpt-5-mini", "messages": [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": text}, ], "max_completion_tokens": 500, "reasoning_effort": "minimal", }, } out.write(json.dumps(line, ensure_ascii=False) + "\n") count += 1 if out: out.close() return paths

Before uploading anything, validate locally. A surgery statement aliases a copy custom_id fails the full record astatine the validation stage, and uncovering that retired aft upload wastes a cycle. A ten-line cheque that parses each line, verifies the required keys, and confirms custom_id characteristic saves you that wasted cycle.

Uploading files and creating jobs

Submission is simply a three-step process per file, and the bid matters. The afloat flow, pinch charismatic examples successful Python, JavaScript, and cURL, is documented successful DigitalOcean’s batch conclusion guide.

Step one: petition a record intent. POST /v1/batches/files pinch a record sanction ending successful .jsonl returns a file_id and a presigned upload URL. The file_id is valid for up to 30 days and reusable crossed jobs; the upload URL expires successful astir 15 minutes, truthful upload promptly. If you miss the window, petition a caller intent.

Step two: PUT the earthy JSONL bytes to the presigned URL. Use Content-Type: application/octet-stream aliases omit the header entirely. Presigned URLs are signature-sensitive, and a nonstandard contented type for illustration application/jsonl tin break the signature match.

Step three: create the batch occupation pinch the file_id. This measurement performs a cheque against entity retention and fails if the upload has not finished, truthful ne'er reorder steps 2 and three. The create telephone takes the supplier (openai aliases anthropic), the completion model (only 24h is presently accepted), an endpoint that must lucifer the url connected each statement for OpenAI jobs (omit it for Anthropic jobs), and a request_id you generate.

That request_id is simply a information cardinal that prevents copy jobs, and for a 40-job overnight tally it matters. If your submission book hits a web correction and retries, the aforesaid request_id returns the existing occupation alternatively of creating a copy that would double-bill 25,000 documents. Build it from the record sanction alternatively than generating a random UUID each time, truthful rerunning the full book is besides safe:

import hashlib import os import uuid import requests from pydo import Client client = Client(token=os.environ["DIGITALOCEAN_TOKEN"]) def submit_file(path): # 1. Reserve a file_id and presigned upload URL. intent = client.batches.files.create(file_name=os.path.basename(path)) file_id, upload_url = intent["file_id"], intent["upload_url"] # 2. PUT the earthy JSONL bytes. octet-stream keeps the signature valid. with open(path, "rb") as fh: put = requests.put( upload_url, data=fh, headers={"Content-Type": "application/octet-stream"}, timeout=300, ) put.raise_for_status() # 3. Create the job. request_id derived from the record sanction makes # the full book safe to rerun without duplicating jobs. request_id = str(uuid.UUID( hashlib.md5(f"doc-pipeline-2026-08:{path}".encode()).hexdigest() )) batch = client.batches.create( file_id=file_id, provider="openai", endpoint="/v1/chat/completions", completion_window="24h", request_id=request_id, ) return batch["batch_id"] batch_ids = {} for way in sorted(paths): # paths from write_batch_files() batch_ids[path] = submit_file(path) print(f"submitted {path} -> {batch_ids[path]}")

Persist the batch_ids mapping to disk (a JSON record is fine). It is the only authorities your pipeline needs to past a restart.

Monitoring 40 jobs

Each occupation moves done a fixed group of states: validating (file structure, unsocial IDs, token counts), queued (waiting for capacity), in_progress, and past 1 of 4 last states. completed intends each petition was processed, moreover if immoderate individual requests failed; those failures are successful the correction file, not reflected successful the occupation status. grounded intends a systemic problem, usually full validation failure. expired intends the 24-hour model ran out. cancelled is self-explanatory. Importantly, expired and cancelled are not full losses: everything that completed earlier the occupation ended is preserved, downloadable, and billed. Only unprocessed requests are dropped, and you are not charged for them.

Polling intends checking the job’s position connected a repeating schedule: your book asks the API for the existent status, waits, and asks again until the occupation is done. The batch API does not nonstop you a notification erstwhile a occupation finishes, truthful this loop is really you find out. There is nary logic to cheque often; erstwhile a infinitesimal crossed each jobs is plentifulness for an overnight run:

import time def wait_for_jobs(batch_ids, poll_seconds=60): pending = set(batch_ids.values()) terminal = {"completed", "failed", "expired", "cancelled"} states = {} while pending: for bid in list(pending): b = client.batches.retrieve(bid) position = b["status"] counts = b.get("request_counts", {}) print(f"{bid} {status:12} " f"{counts.get('completed', 0)}/{counts.get('total', 0)}") if position in terminal: states[bid] = status pending.discard(bid) if pending: time.sleep(poll_seconds) return states

Run it, spell to sleep. The request_counts section gives you per-job advancement if you want a dashboard, but for astir teams, reference the log successful the greeting is enough.

Handling failures

Failures hap astatine 3 levels, and each has a different fix.

Request-level failures are individual lines that could not beryllium processed: a archive that exceeds the model’s discourse window, a contented argumentation rejection, a severely formatted punctual that slipped done validation. These do not neglect the job. Each 1 is written to an correction record pinch its custom_id and an correction code:

{"custom_id": "doc-88213", "error": {"code": "context_length_exceeded", "message": "Request exceeded maximum discourse length."}} {"custom_id": "doc-90142", "error": {"code": "content_policy_violation", "message": "Request was blocked by contented moderation."}}

The correction codes show you what to do. context_length_exceeded documents request truncation aliases splitting earlier resubmission. Content argumentation rejections request a quality look. Anything impermanent was already retried doubly by the level earlier landing here, truthful simply resubmitting those IDs arsenic 1 last mini batch occupation is reasonable. If your input files are clean, the correction record should beryllium small; successful our 200-completion information complete the real-time API, zero requests grounded astatine the API level. The entries that do look are mostly oversized documents you should person caught successful validation.

Job-level failures are rarer. If a occupation expires pinch activity remaining, you tin create a continuation occupation that processes only the requests that grounded aliases were ne'er reached, without re-running (or re-paying for) completed work. The aforesaid copy protection from request_id covers the submission side: a create telephone retried aft a web correction returns the original batch job.

The billing norm is simple: you are charged only for completed requests. Expired, cancelled, aliases guardrail-blocked requests that ne'er produced output costs nothing.

Retrieving and joining results

When a occupation reaches a terminal state, GET /v1/batches/{batch_id}/results returns presigned URLs for the output record and, if 1 exists, the correction file. Two operational specifications here: the presigned URLs are short-lived, truthful download instantly aft fetching them alternatively than storing the URLs for later, and output files are retained for up to 30 days aft completion, aft which they are permanently deleted. Downloading and archiving results to your ain retention should beryllium portion of the pipeline, not an afterthought.

Each output statement carries the custom_id, the afloat API consequence (including per-request token usage), and an correction section that is null connected success. The subordinate backmost to your root documents is simply a dictionary lookup, and summing the usage fields arsenic you spell gives you an nonstop token count to cheque against your bill:

import json import requests as http CATEGORIES = {"billing", "bug_report", "feature_request", "account", "security", "performance", "documentation", "other"} def collect_results(batch_ids, out_path="results.jsonl"): total_in = total_out = failures = 0 with open(out_path, "w", encoding="utf-8") as out: for path, bid in batch_ids.items(): links = client.batches.results.retrieve(bid) if not links.get("result_available"): print(f"{bid}: results not ready, canvass again later") continue resp = http.get(links["output_file_url"], timeout=300) resp.raise_for_status() for statement in resp.text.splitlines(): rec = json.loads(line) if rec.get("error"): failures += 1 continue usage = rec["response"]["usage"] total_in += usage["prompt_tokens"] total_out += usage["completion_tokens"] contented = rec["response"]["choices"][0]["message"]["content"] try: parsed = json.loads(content) except json.JSONDecodeError: failures += 1 # exemplary returned non-JSON; queue for retry continue if parsed.get("category") not in CATEGORIES: failures += 1 # valid JSON, invalid class value continue out.write(json.dumps({ "doc_id": rec["custom_id"], "category": parsed["category"], "summary": parsed["summary"], }, ensure_ascii=False) + "\n") print(f"input tokens: {total_in:,} output tokens: {total_out:,} " f"failures: {failures:,}")

Note the 2nd nonaccomplishment mode handled here: the petition succeeded but the exemplary returned thing that is not valid JSON. With a strict strategy punctual this is uncommon (we measured zero successful 200 information completions), but crossed a cardinal completions “rare” still happens, and the cheque costs nothing. Queue those custom_ids pinch the error-file IDs for the cleanup batch.

The 3rd check, class not successful CATEGORIES, exists because we ran this punctual building against existent documents (50 news articles, pinch an eight-value news-category database successful spot of the summons categories) and recovered a nonaccomplishment the modeled pipeline missed: the exemplary returns perfectly valid JSON pinch a class that is not connected the list. In a 50-document information complete the real-time API, GPT-5 nano invented an out-of-list class 4 times successful 50 and GPT-5 mini twice, producing labels for illustration “science” and “humanitarian” that nary downstream subordinate would recognize. Valid JSON is not valid data; validate the values, not conscionable the structure.

We past tightened the punctual (the “never invent different category” condemnation above) and re-ran the aforesaid 50 documents. The consequence is worthy knowing earlier you spot punctual fixes astatine scale: mini’s violations dropped from 2 to 0, while nano’s stayed astatine 4 retired of 50. Prompt-level fixes are model-dependent; the code-level cheque is not. If you take the cheaper model, scheme for its measured usurpation complaint (8% successful our test) pinch a retry aliases normalization pass. Retrying 8% of requests adds astir 8% to nano’s bill, which still leaves it astir 5x cheaper than mini.

The itemized bill

Batch tokens connected DigitalOcean are billed astatine up to half the serverless (real-time) rates for OpenAI and Anthropic models (see the batch conclusion conception of the Inference Pricing page). The little complaint is not a promotion; it is scheduling economics. Batch jobs tally astatine little privilege and stock off-peak GPU capacity (per the batch conclusion limits), truthful activity that tin hold 24 hours fills hardware that would different beryllium idle betwixt real-time peaks. A petition that must reply successful 2 seconds is much costly to service than 1 that tin tally astatine 4 AM, and the pricing reflects that.

Token usage is the only batch complaint the pricing page lists; location are nary abstracted fees for record upload, storage, occupation creation, aliases polling. At the afloat batch rate, GPT-5 mini costs $0.125 per cardinal input tokens and $1.00 per cardinal output tokens.

Here is the complete measure for the occupation arsenic specified. The guidelines rates are GPT-5 mini’s serverless prices ($0.25 input, $2.00 output per cardinal tokens) from the Inference Pricing page, halved for batch; each costs is past tokens multiplied by rate:

That is astir $0.0004 per document. The identical workload astatine real-time rates costs $375.00 for input positive $400.00 for output, aliases $775.00 total. The $387.50 quality is the value of urgency: what this occupation would salary for answers successful seconds erstwhile the existent deadline is tomorrow morning. Most pipelines ne'er inquire that question, which is why astir pipelines overpay. The useful framing is not “can we spend real-time” but “what is the deadline, really.”

Model prime moves this number much than thing other successful the pipeline. The aforesaid occupation connected different batch-eligible models, astatine afloat batch rates:

For straightforward classification, GPT-5 nano astatine $77.50 for the full cardinal is worthy evaluating first. The correct process is to tally typical documents done each campaigner astatine real-time rates, people the outputs broadside by side, and only past perpetrate the million. We did precisely that for this article: 50 documents (news articles arsenic stand-ins) done some GPT-5 nano and GPT-5 mini pinch this punctual structure, doubly (once earlier and erstwhile aft the punctual tightening described earlier), for nether a dollar successful total. What we measured:

• JSON validity was perfect: 200 completions crossed the 2 models and 2 runs, zero parse failures. The nonaccomplishment mode astatine this tier is not surgery JSON.
• Category subject was not. Nano returned an out-of-list class connected 4 of 50 documents successful some runs; mini did doubly successful the first tally and zero times aft the punctual fix. Nano’s complaint did not amended pinch the stricter prompt.
• On summary quality, judged archive by document, mini was intelligibly aliases somewhat amended connected 14 to 23 of 50 (depending connected really strictly you score), the remainder were ties, and nano was ne'er amended connected a azygous document. The shape was consistent: mini retains specifics that nano drops, specified arsenic names, dollar figures, and lawsuit details.

The verdict falls retired of the numbers. For classification-dominant activity astatine scale, nano positive a validation-and-retry walk is the logical prime astatine a 5th the price. When the summaries provender thing a quality will read, salary for mini. Either way, an information that costs little than a dollar settled a four-figure determination pinch measurements alternatively of instinct; skipping it is really teams extremity up paying Claude Haiku prices for GPT-5 nano work, aliases shipping a cardinal bladed summaries astatine immoderate price.

Two billing caveats. Serverless conclusion is prepaid, truthful the equilibrium must beryllium loaded earlier the occupation runs, and batch pricing is stated arsenic “up to” half the real-time rate, truthful corroborate the effective complaint for your exemplary connected the pricing page earlier you perpetrate the workload.

When batch thumps real-time, and erstwhile to self-host

The determination comes down to a fewer questions you tin reply earlier penning immoderate code.

Batch is the correct instrumentality erstwhile each of these are true:

• Nobody is waiting connected an individual response. Results needed wrong 24 hours, but not wrong seconds.
• The workload is ample capable that complaint limits aliases costs matter. Below a fewer 1000 requests, the real-time API is simpler and the savings are excessively mini to matter.
• Requests are independent. Each statement is self-contained; batch has nary system for 1 petition to dangle connected another’s output. Multi-step chains request orchestration extracurricular the batch API, typically 1 batch occupation per step.
• Text in, matter out. Multimodal requests and image procreation are not supported for batch connected DigitalOcean.
• One exemplary per occupation fits your routing. Mixed-model pipelines mean aggregate jobs.

Stay connected real-time erstwhile latency matters astatine all, erstwhile petition measurement is small, aliases erstwhile you request features batch does not support (streaming, aliases provider-specific features for illustration extended thinking).

Self-hosting connected dedicated GPUs is the 3rd option, and the mathematics is different alternatively than automatically better. DigitalOcean’s dedicated inference runs an H100 astatine $4.41 per hr and an 8x H100 node astatine $30.32 per hour. Self-hosting starts to triumph successful 3 situations. First, open-source models: batch conclusion only supports OpenAI and Anthropic commercialized models, truthful a nightly million-document occupation connected Llama 3.3 aliases Qwen belongs connected dedicated GPUs aliases serverless per-token open-source pricing instead. Second, steady, dense use: if the overnight occupation runs each nighttime and mostly fills the hardware, a dedicated endpoint astatine a fewer 100 dollars per nighttime of GPU clip tin costs little than per-token pricing for ample token volumes, and scale-to-zero intends you extremity paying erstwhile the queue is empty. Third, information control: erstwhile documents cannot time off infrastructure you control, per-token commercialized APIs are disconnected the array sloppy of price. The costs self-hosting adds backmost are engineering clip (serving stack, batching logic, monitoring, retries: everything the batch API conscionable did for you) and the consequence of idle GPUs. If your measurement is uneven aliases the occupation is occasional, per-token batch pricing wins connected full costs moreover erstwhile the earthy GPU mathematics looks close.

Closing

The pipeline supra is astir 150 lines of Python, and astir of it is regular information handling alternatively than instrumentality learning: divided the input to respect record limits, usage existent archive IDs arsenic custom_ids, build petition IDs from record names truthful reruns are safe, download results earlier the URLs and the 30-day retention tally out, and cheque token usage against the bill. The level handles the parts that are really difficult astatine this scale: retries, capacity scheduling, and isolation from your accumulation traffic.

The wont worthy building is to benignant each LLM workload by 1 question: is anyone waiting for this answer? When a personification is waiting, salary real-time rates for real-time behavior. When cipher is, and for backlogs, evaluations, tagging jobs, and reports cipher is, batch should beryllium the default and real-time the objection you justify. Most teams person it backwards: they dainty the interactive API arsenic the only API and latency tolerance arsenic thing to disregard alternatively than a creation input. On this job, that framing is worthy $387.50 retired of $775.00, and it required nary cleverness, nary infrastructure, and nary exemplary changes. The activity was going to tally overnight anyway; it should beryllium priced that way.

References

DigitalOcean documentation

• Inference Pricing: each exemplary rates utilized successful this article (GPT-5 mini $0.25/$2.00, GPT-5 nano $0.05/$0.40, Claude Haiku 4.5 $1.00/$5.00 per 1M tokens), the “up to 50%” batch discount, dedicated GPU pricing (H100 $4.41/hour, 8x H100 $30.32/hour), prepaid billing, and the norm that only completed requests are charged.
• Inference Limits: relationship tier complaint limits (600 requests and 800K-2M tokens per infinitesimal astatine Tiers 3-4), batch limits (50,000 requests and 200 MB per file, 10 cardinal tokens per exemplary per account, 24-hour window), tier exemplary access, and batch postulation isolation.
• Inference Features: automatic retries of transient errors, correction files, continuation jobs, and duplicate-safe occupation creation.
• How to Use Batch Inference: the input record format, three-step upload flow, presigned URL life (~15 minutes), record ID validity (up to 30 days), occupation states, results retrieval, output retention (up to 30 days), and cancellation billing.
• Use Serverless Inference, Manage Model Access Keys, and Manage Serverless Inference Prepayment: setup prerequisites.

Provider and competitor documentation

• OpenAI Batch API guide: 50% discount, 50,000 requests and 200 MB per batch, 24-hour window.
• Anthropic Message Batches: 50% discount, 100,000 requests aliases 256 MB per batch, 29-day consequence retention, emblematic completion nether 1 hour.
• OpenRouter Batch API quickstart: beta status, typically 50% off, inline petition format, 24-hour window, 30-day consequence retention.
• Together AI batch inference: up to 50% disconnected pinch the afloat discount constricted to selected models, open-weight catalog.
• Fireworks AI Batch API: level 50% disconnected serverless rates, open-weight and civilization models.

Measured data: the nonaccomplishment rates, category-violation counts, token counts, and model-quality comparison travel from the authors’ ain information runs (two runs of 50 documents each done GPT-5 nano and GPT-5 mini complete DigitalOcean’s serverless real-time API, August 2026), not from immoderate documentation.