How We Made a Text-to-Speech Model Respond in Sub-50 ms

TL;DR

Our Qwen3-TTS 1.7B CustomVoice implementation achieves 10 requests per 2nd (RPS) and sub-50 sclerosis p95 time-to-first-audio (TTFA) while maintaining real-time playback connected a azygous NVIDIA H100 SXM.

We comparison 5 implementations: ours, vLLM-Omni, SGLang-Omni, VoxServe, and M*, nether Poisson open-loop traffic. After tuning each implementation for low-latency streaming, ours is the only 1 to execute sub-50 ms p95 TTFA. We support sub-50 sclerosis p95 TTFA done 10 RPS and support it below 100 sclerosis moreover astatine 20 RPS.

Our strategy produces astir 630 characters per second astatine 10 RPS. At $4.29 per hr for a 1× H100 SXM instance, this translates to ~$2 per 1M characters astatine afloat utilization. For comparison, ElevenLabs V3 is $100 / 1M and Cartesia Sonic 3.5 is $49 / 1M astatine a higher TTFA.

We unfastened root the implementation and benchmark. Our methodology is explained below.

Defining “Real-time” TTS

Let’s commencement by discussing what a real-time TTS server needs to achieve. We deliberation it’s a four-part problem:

1. Low Audible TTFA: Time from petition dispatch to the first audible sample must beryllium low.
2. Zero underruns: Once playback starts, the customer must not tally retired of buffered audio.
3. Capacity: 1 and 2 must clasp arsenic RPS increases.
4. Non-malformed output: Speech must beryllium intelligible.

We take Qwen3-TTS CustomVoice 1.7B because it is 1 of the astir celebrated TTS models pinch a permissive license.

Based connected the supra definition, we target low p95 audible TTFA pinch zero underruns while maintaining precocious RPS connected a azygous NVIDIA H100 SXM.

All benchmarks tally for 5 minutes nether Poisson open-loop postulation to approximate existent workloads, pursuing Fireworks AI’s LLM benchmark. Each motor receives the complete matter successful a azygous HTTP request, while audio output remains streamed. We observe audible TTFA, reconstruct playback from received PCM, and measure the completed audio utilizing Deepgram STT.

How Do Other Engines Perform?

The array beneath shows the upstream/default consequence astatine 1 RPS for each engine. We only use changes for compatibility successful this run.

These defaults person important room for improvement. We tune each serving motor for its ain latency, continuity, quality, and capacity requirements.

1. Remove starring silence

The first PCM returned by a exemplary tin incorporate tens of milliseconds of soundlessness earlier the first sustained sound. This spread pushes audible TTFA backmost for illustration so:

We adhd a move trim. It detects sustained reside from short RMS windows, removes samples earlier onset, and streams the remaining audio normally. This alteration improves TTFA by ~80ms but does not make exemplary conclusion itself faster.

2. Tune framework accumulation

We besides tune really galore codec frames are collected earlier decoding and releasing an audio chunk.

Smaller first chunks trim TTFA, but supply little playback headroom and create much predominant decoder work. Larger chunks are easier to batch and make continuous playback safer, but hold the first audible output. A useful configuration truthful starts pinch a mini chunk and increases the chunk size for later output.

The nonstop knobs disagree by engine: vLLM-Omni exposes settings specified arsenic codec_chunk_frames and codec_chunk_ramp; the different engines supply balanced chunk aliases stride controls. We iterate complete these values to find the config that champion matches: debased p95 TTFA, zero underruns and unchangeable behaviour arsenic load increases.

Performance aft tuning existing serving engines

The pursuing array shows the selected no-underrun floor plan for each motor aft leading-silence and frame-accumulation tuning.

VoxServe reaches sub-50 sclerosis p95 TTFA astatine 1 RPS, while the different 3 engines do not. By astir 6 RPS, each motor is astatine astir 100 sclerosis p95 TTFA aliases higher.

How We Optimized Qwen3-TTS

We first request to understand Qwen3-TTS architecture. It is simply a 3-part exemplary performing hierarchical multi-codebook generation. The Talker predicts the first codebook token for each audio frame, the Code Predictor generates the remaining 15 codebook tokens, and the causal Codec converts codebook tokens into waveform samples.

Each module has its ain compute profile, batching behavior, and latency requirements. Rather than optimizing each module successful isolation, we attraction connected a broader question: really should a serving strategy coordinate these heterogeneous tasks?

1. Bringing 3 modules nether 1 scheduler

Most Qwen3-TTS serving implementations are divided into 2 stages: the Talker and Code Predictor tally together, while the Codec runs separately. This separation enables token procreation and waveform decoding to overlap crossed requests.

We return this a measurement further. We expose the Talker, Code Predictor, and Codec arsenic 3 independently schedulable tasks. The cardinal is not simply splitting them into parts, but bringing each 3 onto a shared scheduling aboveground managed by 1 scheduler. This creation draws inspiration from M* (arXiv).

With this setup, the scheduler tin determine whether to tally the Talker, beforehand the Code Predictor, aliases prioritize a Codec occupation that is approaching its playback deadline. It tin besides batch requests waiting for the aforesaid module. Instead of pursuing a fixed execution order, we tin rearrange activity according to urgency.

Combining the Talker and Code Predictor whitethorn look much businesslike because it removes an intermediate boundary. However, the mixed cognition tin go a non-preemptible portion of activity that blocks much urgent Code Predictor aliases Codec jobs. Keeping the modules abstracted creates shorter units of activity and gives the scheduler much opportunities to interleave requests.

2. Scheduling astir the needs of reside streaming

Speech streaming has 2 chopped notions of urgency.

Before the first chunk of audio arrives, each millisecond increases TTFA, truthful we request to prioritize this path. But erstwhile playback begins, the extremity changes: the adjacent chunk only needs to get earlier the existent audio finishes playing. Producing it earlier provides nary user-visible benefit.

Thus, we springiness precocious privilege to requests that person not produced their first audio, while established streams go urgent only arsenic they attack a playback deadline.

Running each urgent petition unsocial would destruct batching efficiency. Instead, our scheduler selects an urgent petition arsenic an anchor and fills the remainder of the batch pinch compatible work. This helps the captious petition meet its deadline while making effective usage of the GPU.

This argumentation useful particularly good because each 3 modules stock a scheduling surface, allowing the scheduler to take some the petition and the pipeline shape to advance.

3. Exploiting the regular building of the Code Predictor

The Code Predictor is an autoregressive transformer, but its execution is unusually regular. It ever performs a fixed number of steps (15) per framework to capable the remaining audio codebooks.

We utilization its fixed building to preallocate its KV cache and seizure the full frame-generation loop arsenic a azygous CUDA graph. We besides usage a Triton attraction kernel specialized for its short, bounded context.

By replacing a host-driven series pinch a fixed GPU program, we little latency and simplify the execution system.

4. Rebuilding the Codec astir cached state

The Qwen3-TTS Codec is made up of Transformers and CNNs. Generating the adjacent audio chunk depends connected some the Transformer discourse and convolutional authorities from erstwhile chunks.

A naive implementation reprocesses the afloat framework history connected each update, many times decoding aged audio arsenic the utterance grows.

To debar this, we usage a state-cache-based Codec. Each petition retains the Transformer discourse and convolutional authorities needed by the adjacent chunk. Incremental decoding past reuses this cached authorities and processes only recently arrived frames alternatively of replaying the afloat history.

Initializing the authorities cache from the first framework adds overhead and hurts TTFA. We truthful usage afloat decoding for the first audio, past move to state-cached incremental decoding for businesslike sustained playback.

We likewise alteration chunk sizes complete the people of a request. Smaller chunks fto playback statesman quickly, while larger chunks amended batching and GPU ratio during sustained playback.

5. Additional serving optimizations

We seizure CUDA graphs for a predefined group of batch sizes. If a fresh cohort exceeds the largest captured batch size, we divided it crossed scheduling turns alternatively than falling backmost to eager mode.

We besides debar unnecessary CPU–GPU synchronization. For example, while EOS is suppressed, procreation cannot terminate, truthful we defer the termination cheque until EOS is enabled. This lets the CPU hole and taxable consequent activity without waiting for the GPU.

Finally, we support input streaming for modular speech-to-speech systems. As an upstream LLM generates tokens, the TTS exemplary tin statesman synthesizing reside earlier receiving the complete response, reducing end-to-end latency.

What’s Next?

Qwen3-TTS is conscionable the opening of our activity connected multimodal inference. We scheme to widen our scope to image, video, and world models, arsenic good arsenic fine-tuning. Our eventual imagination is to simulate the world 1:1 done realtime multimodal inference.

We are a squad of experts successful multimodal AI investigation and infrastructure. Our unfastened TTS model, Dia, has been downloaded complete 2 cardinal times and has classed #1 connected Hugging Face. Our squad of ex-YC, ex-KRAFTON, and ex-NAVER engineers has published investigation astatine NeurIPS and ICLR and earned 3 IOI and ICPC World Finals golden medals. Nari Labs is backed by Y Combinator.

If you want to activity pinch america connected thing multimodal, let’s chat.