Show HN: Spinf-serve – inference server for bulk scoring and decisions

Bulk classification and decision conclusion on one GPU.

spinf-serve says a part of satisfied formerly and answers many questions concerning it in the identical pass. Each answer is a probability complete the options you give, not generated text. It runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API, the identical API as the hosted spinf service. Apache 2.0.

We built this motor for our own work: a regular market indicator peruse from millions of news articles, all part asked dozens of narrow, calibrated questions. A serving motor built for conversation re-reads the satisfied for all inquiry and generates one token at a time, which made that job dilatory and expensive. spinf-serve lays all inquiry out as a branch of one peruse of the satisfied and packs many contents into one onward pass. Bulk classification is anywhere it started and anywhere it is strongest.

The identical scheme is what is now called a decision model: a example that returns a probability complete a fixed set of answers (yes or no, a label, a level) alternatively of a paragraph. Any open education example plant this way in spinf-serve, and dedicated decision models run as they are. The hosted spinf API grew out of this engine; we are publishing it since group asked to run it on their own hardware.

It is not a conversation server. Nothing is generated unless you ask for it, and it is built for throughput fairly than for the latency of a sole request.

Linux, an NVIDIA GPU alongside 24 GB and fp8 tensor cores (Ada, Hopper or Blackwell: RTX 4090 / 5090, L4, L40S, RTX 6000 Ada, H100), a controller supporting CUDA 13, Python 3.12. Ampere cards run alongside int8 weights instead (configuration).

pip instal "spinf-serve[server]" spinf-serve assist spinf/gemma-4-12b-fp8 # downloads the checkpoint on archetypal start; serves on harbor 8000

Or the image, alongside the NVIDIA Container Toolkit installed (the quantity keeps the example and the compile caches; arguments following the depiction name are those of spinf-serve serve):

docker run --gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve

Then ask. Two classifications and one extraction from a sole peruse of the content:

curl http://localhost:8000/v1/score \ -H "Content-Type: application/json" \ -d '{  "model": "spinf/gemma-4-12b-fp8",  "messages": [  { "role": "user",  "content": "Customer assistance message:\nI was charged twice for September, delight refund among the charges." }  ],  "scoring": {  "queries": [  { "id": "topic",  "template": "\n\nTopic (billing, technical, account, shipping, or cancellation):{?}",  "options": [" billing", " technical", " account", " shipping", " cancellation"] },  { "id": "refund",  "template": "\n\nIs the client asking for a refund?\nAnswer:{?}",  "options": [" yes", " no"] },  { "id": "month",  "template": "\n\nQ: Which duration was charged twice?\nA: \"{*}\"" }  ]  }  }'

For all scored inquiry the reply gives all option's probability, and the identical probability on bare content (real numbers from the 12B model):

{ "id": "refund", "combinations": [ { "values": {}, "options": [ { "text": " yes", "logprob": -0.90, "p": 0.72, "floor_p": 0.60 }, { "text": " no", "logprob": -1.86, "p": 0.28, "floor_p": 0.40 } ], "residual": 0.44 } ] }

p is the probability among your options. floor_p is the identical inquiry alongside no content, so p - floor_p is what the satisfied added (here topic moves " billing" from 0.47 to 0.91). residual is the probability that went to words outside your options. For duration the reply lists the most apt answers alongside their probabilities ("September" first). The identical body, alongside an API key, runs against api.spinf.com; the OpenAI SDKs activity through /v1/chat/completions with the scoring obstacle in extra_body (quickstart).

• One read, many questions. The satisfied is peruse once; all inquiry is a branch of the identical continue and expenses its own tokens only. Placeholders rotate one template into a grid of questions.
• Scores you can threshold. A probability per choice alongside its flat and log-probability; a decision is a threshold you choose on your own data (calibration).
• Extraction from the identical read: {*} text, {n} numbers and {$} amounts decoded alongside their probabilities, or liberated content generated into a slot.
• The identical mark any alternatively is in the batch. With the fp8 checkpoints, a content's scores are bit-identical alone or packed alongside thousands of others, throughout calls, days and servers. The system_fingerprint says whenever two responses are not comparable.
• Images, audio and video frames in the content, for the models that peruse them.
• Prompt packs. Ready-made, calibrated inquiry sets for a job (content moderation, triage, merchandise listings, reviews, brand, entertainment), preloaded and named in a petition (below).
• Hosted or self-hosted. A client switches between this server and api.spinf.com by changing the basis URL and the key. spinf-client talks to both; the agent skills instruct a coding delegate the API, this server and the group format.

Measured on one RTX 4090 alongside Gemma 4 12B in fp8: 24 Wikinews articles of concerning 470 tokens, 56 two-option questions each, 1,344 questions in all (the community job set in fixtures/), sent complete HTTP to spinf-serve and to vLLM 0.26 with prefix caching. The manuscript and the complete results are in benchmarks/.

vLLM already reuses all part throughout its 56 questions through the prefix cache. What it cannot do is answer them in one onward pass: it runs 1,344 single-token requests, during spinf-serve lays the 56 questions out as branches of one read and packs the 24 articles into two 16k-token passes. The gap grows alongside the figure of questions per content.

In practice, per 4090 alongside the 12B model: the spinf/moderation group (106 questions per message, 22 categories) scores concerning 20,000 messages per hour; a news part asked 74 questions is peruse at nearly 700 questions per second. Extraction slots add decoding activity (a searched slot expenses concerning as much as 200 to 600 tokens of scoring, a generated one up to concerning 2,000) but portion the read.

A job is rarely one question: satisfied moderation is 106 of them, all calibrated against normal traffic and combined into category scores and a decision. A prompt pack is one document holding the questions, how the satisfied is presented to the model, how the answers rotate into scores and how the scores rotate into a decision. The server preloads packs and a petition names one ("packs": [{"id": "spinf/moderation"}]), getting the pack's categories and decision rear next to its own questions, from the identical read.

The authoritative packs are in spinf-prompts, loaded by default. The server loads packs from any GitHub repository or local directory stated alongside --packs; PACKS.md lists every pack we cognize of, ours and others', and an issue adds yours. Format, petition sectors and sources: docs/prompt-packs.md.

Each is a checkpoint we quantized and calibrated, published on the Hugging Face Hub with its formula (quantizations/); the motor pins the revision it was parity-tested on. Any another Hub checkpoint of a supported architecture family runs by its path, quantized at burden time.

docs/: quickstart, examples, scoring API, configuration, writing questions, calibration, prompt packs, FAQ; the guides (decision models, probabilities you can act on, few-shot classification, sentiment, study coding, moderation, intent routing); and the engineering documents on how the motor plant (architecture, batch invariance, precision, the Qwen stack, extraction).

Version 0.1. The motor (scoring, extraction, generation, media, lot invariance, the two example stacks) is complete and parity-tested against recorded references; the server speaks the community API of the hosted service; the four checkpoints are public. What is motionless missing is listed in docs/README.md.

Questions, ideas and bugs go to the issues of this repository. There is no forum or chat, and nothing is too small.

spinf_serve/ holds the motor (protocol, compiler, packer, scorer, extract, service), the served models' manifests, the packs component (pure Python, additionally used by the client and the hosted API), one example stack per architecture family in models/, the quantization schemes in quant/, the HTTP server in server/ and the spinf-serve command. quantizations/ has one directory per published checkpoint (recipe, calibration output, example card), fixtures/ the public parity fixture and references, benchmarks/ the throughput comparison. The map of the code is docs/ARCHITECTURE.md.

uv sync --extra server uv run pytest # CPU tests (the authoritative packs' aureate fixtures need ../spinf-prompts or SPINF_PROMPTS_DIR) uv run ruff inspect . && uv run ruff format --check . CUDA_VISIBLE_DEVICES=0 uv run spinf-serve parity --fixture fixtures/wikinews_20261024.json.gz \ --reference fixtures/wikinews_20261024.spinf-gemma-4-12b-fp8.json.gz --passes 2

Every alter that touches a kernel, a compile border or a quantization scheme ships alongside the GPU parity run: bit-identical log-probabilities on the recorded citation of all published checkpoint (fixtures/). Dependencies are pinned exactly; they alter lone together alongside a parity run. See CONTRIBUTING.md.

Apache 2.0 (LICENSE, NOTICE). The models we run and re-quantize, the projects we build on, the techniques we implement and the data we checked on are credited in ACKNOWLEDGEMENTS.md.