Experience and lessons learned from serving multi-stage qwen3-omni in vLLM-Omni

Qwen3-Omni combines multimodal understanding with speech generation. This post explains how vLLM-Omni serves it as a staged pipeline and optimizes each stage for online workloads.

TL;DR

vLLM-Omni’s Qwen3-Omni serving stack includes:

• A three-stage pipeline: Thinker for multimodal reasoning, Talker for speech codec generation, and Code2Wav for waveform reconstruction.
• OpenAI-compatible serving: /v1/chat/completions is the primary endpoint for Qwen3-Omni text and audio generation.
• Batching, CUDA Graphs, async chunk, and replicas: stage-level batching and per-stage graph capture on Thinker, Talker, and Code2Wav improve high-concurrency throughput; async-chunk Thinker-to-Talker and Talker-to-Code2Wav handoffs emit partial payloads instead of waiting for full-stage completion; and Talker/Code2Wav replicas scale the speech-generation stages.
• Performance validation: controlled benchmark sweeps and DFX perf runs show lower audio TTFP (time to first audio packet), lower audio RTF (real-time factor), and higher throughput as each optimization layer is enabled.

Quickstart

The default Qwen3-Omni deploy profile is resolved automatically when serving the model with --omni:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --port 8091

For explicit configuration, pass the staged deploy profile:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --port 8091 \
  --deploy-config vllm_omni/deploy/qwen3_omni_moe.yaml

The bundled profile includes a platforms: section. vLLM-Omni detects the runtime backend (CUDA, NPU, ROCm or XPU) and merges the matching deltas automatically — no extra CLI flag is required. The same launch command works across hardware.

Requests should use /v1/chat/completions. Set modalities in the request body to declare output types — e.g. ["text"] for text only, or ["text", "audio"] for text plus speech.

For deployment options, async-chunk settings, and multi-replica layouts, see the Qwen3-Omni online serving guide.

Qwen3-Omni Serving Model

Text-only LLM serving is one loop: prefill, decode, detokenize. Qwen3-Omni adds two speech stages after multimodal reasoning, each with a different compute profile:

Thinker   -> multimodal understanding + text generation
Talker    -> hidden states and embeddings to RVQ codec codes
Code2Wav  -> codec codes to waveform audio

Figure 1: Qwen3-Omni serving in vLLM-Omni is a staged dataflow: Thinker produces text and hidden states, Talker produces codec codes, and Code2Wav reconstructs audio.

Optimization Overview

Different parts of the Qwen3-Omni pipeline hit different bottlenecks. vLLM-Omni does not apply one fixed recipe; each optimization targets a specific stage or handoff. The walkthrough that follows takes each one in turn — in the order we validated it — and answers three questions in order: Why the problem exists, Why it works (how the mechanism removes that problem), and What you gain (the measured payoff).

Technique	Target stage / path	Problem it addresses	Primary benefit
Stage decomposition	Thinker → Talker → Code2Wav	Three stages with very different compute profiles share one loop, so a single batching/graph/device policy lets the slowest sub-path gate the rest — capping throughput and blocking per-stage tuning and scaling	Independent runtime policy per stage
AR + Code2Wav batching	Talker MTP path, Code2Wav async chunks	Single-request micro-work leaves SMs idle between launches at high concurrency, capping GPU occupancy and req/s	Higher occupancy and req/s
CUDA Graph	Thinker / Talker / Code2Wav decode paths	Repeated CPU-side kernel dispatch on every decode step inflates TPOT and keeps audio RTF above real-time	Lower TPOT and audio RTF; ~4× throughput jump in sweep
Async chunk	Thinker→Talker, Talker→Code2Wav	Full-payload stage barriers force Code2Wav to wait for a full Talker payload, delaying first audio and inflating audio TTFP	Pipelined handoffs; largest audio TTFP reduction
Async omni output	Client-facing audio emission	Synchronous client I/O makes decode workers block on emission between chunks, wasting GPU time and lowering throughput	Throughput recovery without audio TTFP regression
Stage replicas	Talker, Code2Wav	Talker and Code2Wav saturate and queue while Thinker still has headroom, becoming the tail bottleneck under load	Horizontal scale on bottleneck stages only
Hot-path cleanup	Talker code predictor, connector payloads	Per-step Python, allocation, and sync overhead compounds over long utterances, inflating E2EL and audio TTFP	Lower per-step latency; stacks with all layers above

We validated each layer with a controlled benchmark sweep on Seed-TTS en (Qwen3-Omni-30B-A3B-Instruct, 4/32/64/128 prompts at concurrency 1/8/16/32, 5 warmups, three visible GPUs mapped as 0/1/2). Each configuration restarted the server with an isolated deploy profile and added one optimization on top of the previous row. Batch through Async output pin one stage per GPU (Thinker / Talker / Code2Wav on GPUs 0 / 1 / 2, single replica each); the Stage replicas row keeps Thinker on GPU 0 and runs 2× Talker + 2× Code2Wav on GPUs 1 and 2. The table below summarizes concurrency 32; Validation Results charts all four concurrency levels.

Step	Config added	Talker / Code2Wav replicas	Req/s	Mean audio TTFP	Mean audio RTF
Baseline	Batch	1 / 1	2.0	4975 ms	0.98
+ CUDA Graph	Graph capture on Thinker, Talker, Code2Wav	1 / 1	7.9 (+295%)	1798 ms (−64%)	0.38 (−61%)
+ Async chunk	Async-chunk stage handoffs	1 / 1	9.0 (+14%)	497 ms (−72%)	0.39
+ Async output	Async omni output path	1 / 1	7.8 (−13%)	464 ms (−7%)	0.31 (−21%)
+ Stage replicas	2× Talker + 2× Code2Wav	2 / 2	11.1 (+42%)	482 ms	0.31

Figure 2: Qwen3-Omni performance comes from optimizing the staged dataflow, stage runtime, and decode hot path together.

Optimization Stack, Stage by Stage

Each optimization builds on the previous one, so the numbers reported in each step assume every layer above it is already enabled.

1. Stage Decomposition and Batching: The Baseline

Why. Qwen3-Omni is not one homogeneous decode loop: Thinker does multimodal AR text generation, Talker runs a codec-predictor AR path, and Code2Wav runs parallel vocoder decode. Folding these three very different workloads into a single serving path forces the same batching policy, graph policy, and device layout on all of them — and lets the slowest sub-path gate the rest. Separating the stages removes that coupling, but it exposes a second problem: the speech path still spends most of its GPU time on single-request micro-work. Each Talker decode step is a short code-predictor forward and each Code2Wav chunk is a small vocoder forward, so at concurrency 32, running them one request at a time leaves SMs idle between launches and never amortizes the fixed per-step cost.

Why it works. Stage decomposition breaks the coupling: stage boundaries become first-class serving objects, connectors define what crosses each one (hidden states, embeddings, codec codes, chunk metadata), and the scheduler can schedule, batch, and graph each stage on its own critical path — so no single policy is forced on all three and the slowest sub-path no longer gates the rest.Batching can reduce queuing and closes the idle-SM gap.

What you gain. Explicit stages let vLLM-Omni treat each component as an independent runtime — separate max_num_seqs, sampling params, connectors, graph/eager policy, and optional replicas — which is the prerequisite for every optimization below. On top of that, per-stage batching raises GPU occupancy on the speech-generation side: more requests share the same Talker MTP invocation and the same Code2Wav forward.

2. CUDA Graph: Per-Stage Decode Capture

Why. Batching raised occupancy, but each decode step still paid repeated CPU-side kernel dispatch. Qwen3-Omni runs three decode-heavy stages; Talker alone may execute hundreds of short steps per utterance, and each step previously re-launched the same stable operator sequence from Python. At concurrency 32, that launch tax dominated TPOT and kept audio RTF above real-time even after batching.

Why it works. CUDA Graph removes the per-step kernel dispatch that dominated TPOT: it captures a fixed operator sequence once and replays it with minimal CPU work. Each stage has a different capture point, but the principle is the same: decode shapes bucket into stable (batch, seq, frames) profiles, so the runtime records the graph at warmup and reuses it on the hot path.

Stage 0 — Thinker: vLLM outer decode graph

The Thinker is an autoregressive multimodal stage (LLM_AR). When enforce_eager is false, it uses vLLM’s standard CUDA Graph capture on the decode path — the same mechanism as text-only LLM serving. This removes repeated CPU-side kernel dispatch during long Thinker generations.

The default deploy profile leaves stage 0 on graph-friendly settings on CUDA. Platform overrides can disable outer capture where the backend is unsafe: for example, XPU sets enforce_eager: true and max_cudagraph_capture_size: 0 on stage 0.

Stage 1 — Talker: outer decode graph + compiled code predictor

The Talker stage also runs through vLLM’s outer CUDA Graph path when enforce_eager: false. Each Talker decode step additionally invokes the code predictor — a short re-prefill transformer that emits RVQ codec codes. That inner path is optimized separately:

• torch.compile fuses the 5-layer predictor forward (dynamic=False, epilogue_fusion=False) so RMSNorm/RoPE stay numerically aligned with the reference path while still reducing kernel count per step.
• On CUDA, the code predictor does not enable a second manual CUDA Graph layer by default (use_cuda_graphs=False), because that would conflict with vLLM’s Talker CUDAGraphWrapper. The outer Talker graph and compiled inner forward are complementary: one captures the AR stage loop, the other fuses the codec-prediction micro-forward.

Optional prefix-graph buckets (code_predictor_prefix_graphs in connector config) can capture additional stable predictor shapes when explicitly enabled.

Stage 2 — Code2Wav: inner vocoder graph

Code2Wav is a generation stage (LLM_GENERATION), not an AR loop. Its graph path is an inner CUDAGraphDecoderWrapper rather than vLLM’s outer wrapper:

# Enabled during weight load when stage enforce_eager is false
self.code2wav.enable_cudagraph(
    codec_chunk_frames=chunk_frames,
    codec_left_context_frames=left_frames,
)

Shape bucketing from connector config. Before warmup, the wrapper reads codec_chunk_frames and codec_left_context_frames from the stage connector config. Capture enumerates the (batch, num_quantizers, frames) buckets that async-chunk and full-payload decode will hit at runtime — including the smaller first chunk from initial_codec_chunk_frames.

Vocoder warmup. precompute_snake_caches() runs before graph capture so SnakeBeta activations do not pay repeated setup inside the captured decode loop.

Chunk dispatch. In async-chunk mode, chunked_decode_streaming delegates stable chunks to _cudagraph_wrapper.chunked_decode_with_cudagraph; full-payload paths use the wrapper’s batched decode entry points when shapes match captured buckets.

What you gain. Turning on CUDA Graph for all three stages in the benchmark sweep raises req/s from 2.0 to 7.9 (+295%), cuts mean audio TTFP from 4975 ms to 1798 ms, and drops mean audio RTF from 0.98 to 0.38. Most of the win comes from removing launch overhead across Thinker text generation, Talker codec decode, and Code2Wav vocoder forwards together — not from Code2Wav alone.

3. Async Chunk: Pipelined Inter-Stage Handoffs

Why. CUDA Graph made each stage faster, but the pipeline was still barrier-synchronized: Talker could not start until Thinker finished, and Code2Wav could not emit audio until Talker accumulated a full payload. First-audio latency therefore tracked full Thinker generation plus full Talker prefill — even when only a few codec frames were needed to produce the first audible chunk.

Why it works. Async chunk replaces the full-payload barrier with pipelined partial handoffs. Thinker emits embedding rows incrementally; Talker accumulates codec frames and slices them on initial_codec_chunk_frames / codec_chunk_frames boundaries; Code2Wav decodes each slice with codec_left_context_frames for continuity. The async scheduler overlaps chunk transfer with stage compute, so each stage starts work while the previous one is still decoding — first audio is ready after a few codec frames instead of a full Thinker generation plus Talker prefill.

Connector chunk policy. The deploy config controls output chunk cadence independently from Code2Wav decode context:

initial_codec_chunk_frames: 4
codec_chunk_frames: 25
codec_left_context_frames: 25

This lets the connector emit a small first codec chunk for lower audio TTFP while Code2Wav keeps enough left context for audio continuity across chunk boundaries.

Incremental handoffs. On the worker connector path, Thinker→Talker tracks put_req_chunk plus pending chunk-0 prefill state, and Talker→Code2Wav accumulates codec rows in code_prompt_token_ids before slicing — so each boundary ships only new frames instead of replaying the full payload.

What you gain. Async chunk is the largest audio TTFP win in the sweep: mean audio TTFP drops from 1798 ms (CUDA Graph) to 497 ms, and from 4975 ms (Batch alone) overall. Req/s rises to 9.0 because pipelined handoffs keep Talker and Code2Wav busy while Thinker is still decoding.

4. Async Output: Non-Blocking Client Emission

Why. Async chunk speeds inter-stage handoffs, but the client-facing emission path can still stall stage workers. If Code2Wav must synchronously serialize, buffer, and push each audio chunk through the HTTP serving loop before starting the next decode step, GPU time is lost to I/O and Python scheduling even though the stage-to-stage pipeline is already incremental.

Why it works. use_async_omni_output decouples chunk production from chunk delivery. Code2Wav can finish a vocoder forward and hand audio to a non-blocking output path while Talker and the connector keep advancing. The serving loop surfaces packets to /v1/chat/completions asynchronously instead of making decode workers wait on client I/O.

What you gain. On top of async chunk in the benchmark sweep, async output keeps mean audio TTFP near 464 ms while lowering mean audio RTF from 0.39 to 0.31 at concurrency 32. Throughput is not the primary metric here — req/s moves from 9.0 to 7.8 — because decoupling client emission trades a small amount of batching efficiency for smoother stage scheduling and a lower real-time factor.

5. Stage Replicas: Scaling Talker and Code2Wav

Why. Under concurrent speech generation, Thinker, Talker, and Code2Wav do not saturate equally. Thinker handles one multimodal prompt per request; Talker and Code2Wav execute many short decode steps and vocoder forwards per utterance. At concurrency 32, a single Talker/Code2Wav instance becomes the tail bottleneck even when Thinker still has headroom — replicating the entire pipeline would waste memory duplicating the multimodal stage.

Why it works. Stage replication is horizontal scaling applied only where queue depth builds up. One Thinker on GPU 0 fans out to two Talker replicas on GPUs 1 and 2; two Code2Wav replicas share the same pair of GPUs. Load spreads across the speech-generation side without copying the full three-stage pipeline per replica. The benchmark deploy config keeps one Thinker and runs two replicas each for Talker and Code2Wav:

{
  "stage_overrides": {
    "1": {"num_replicas": 2, "devices": "1,2"},
    "2": {"num_replicas": 2, "devices": "1,2"}
  },
  "extra_cli_args": ["--async-chunk"]
}

Figure 3: Async chunk and stage replicas target the speech-generation side of the pipeline, where Talker and Code2Wav can become the bottleneck under concurrent load.

What you gain. Adding replicas on top of async output reaches 11.1 req/s (+42%) at concurrency 32 — the highest throughput in the sweep — while keeping mean audio TTFP near 482 ms and mean audio RTF near 0.31. The tracked multi-replica perf suite separately targets 2100 ms mean audio TTFP and 0.39 mean audio RTF for long text-plus-audio workloads at concurrency 32.

6. Hot-Path Cleanup: Talker Decode and Connector Payloads

Why. Batching, graphs, async chunk, and replicas address structural bottlenecks, but the Talker decode loop still executed hundreds of small per-step costs: Python generate() dispatch, repeated torch.cat while building connector payloads, and device-to-host reads of decode state that the next step immediately needed back on GPU.

Why it works. Each fix targets a repeated micro-cost on the critical path:

Connector payload construction. Large async chunks previously paid for repeated torch.cat while accumulating Thinker and Talker payloads. Passing decode embeddings per token and trimming redundant payload assembly removes allocation and copy work on every chunk boundary.

Talker code predictor rewrite. The older path used Hugging Face generate() for very short codec-predictor sequences. That added Python dispatch, dynamic allocation, and KV-cache overhead on every Talker decode step. The optimized path uses re-prefill with SDPA, native GQA, inline top-k sampling, cached module references, and torch.compile on the inner transformer (epilogue_fusion=False for fp32-safe RMSNorm/RoPE). On CUDA this compile path sits below vLLM’s Talker CUDA Graph rather than adding a conflicting second graph layer; see §2 above.

GPU-resident decode state. Intermediate keys such as hidden_states.last, hidden_states.trailing_text, embed.tts_pad_projected, and codes.audio stay in model_intermediate_buffer so the next decode step reuses GPU-resident tensors instead of forcing device-to-host synchronization — eliminating round trips that would otherwise serialize every Talker step.

Numerical precision on the code predictor. Where audio quality is sensitive, RMSNorm variance and RoPE stay in fp32, and per-call embedding buffers avoid cross-request aliasing. Qwen3-Omni configures the wrapper with parallel embedding, stored sampling, and returned projection buffers so Talker can feed the next decode step without rebuilding state on CPU.

What you gain. Hot-path cleanup removes overhead that scales with utterance length. In a long-context single-request test, E2EL dropped from 21.28 s to 7.37 s, audio TTFP from 3197 ms to 1796 ms, and audio RTF from 0.71 to 0.28. These changes stack with the layers above and show up in the DFX perf suite baselines rather than as a separate sweep row.

Validation Results

The charts below plot the same benchmark sweep summarized in Optimization Overview — Batch, CUDA Graph, Async chunk, Async output, and Stage replicas — across all four concurrency levels (1/8/16/32), each starting from the Batch baseline.

Figure 4: Request throughput (req/s) at c=1 (orange), c=8 (blue), c=16 (purple), and c=32 (green). Stage replicas reach 11.1 req/s at c=32 and 7.8 req/s at c=16, up from 2.0 req/s (Batch at c=32).

Figure 5: Mean audio RTF at c=1, 8, 16, and 32. Batch stays near real-time at high concurrency (RTF 0.87–0.98); async chunk and replicas keep c=16/c=32 RTF at or below ~0.40.

Figure 6: Mean audio TTFP in milliseconds (log scale) at c=1, 8, 16, and 32. Async chunk drops c=32 TTFP from ~4975 ms (Batch) to ~497 ms.

Cross-Workload DFX Perf Results

The local sweep above isolates each layer on one workload. To confirm the gains generalize, the following results come from the Qwen3-Omni DFX perf artifacts for Qwen/Qwen3-Omni-30B-A3B-Instruct using the openai-chat-omni backend. They reflect the accumulated optimization stack above. The percentages compare the measured run against the tracked perf-suite baseline for the same workload. Lower is better for TTFT, E2EL, audio TTFP, and audio RTF; higher is better for throughput.

Workload	Key result	Change vs. baseline
Long text + audio output, 128 prompts, concurrency 32, 2500 input / 900 output tokens	80.43 audio-s/s, 0.399 mean audio RTF, 1065 ms mean audio TTFP	TTFT -20.3%, E2EL -26.3%, audio RTF -29.1%
Audio-input multimodal, 10 prompts at 0.1 req/s	3.15 audio-s/s, 0.117 mean audio RTF, 396 ms mean audio TTFP	TTFT -12.7%, E2EL -11.6%, audio RTF -10.3%
Image + video multimodal, 40 prompts at 0.5 req/s	13.74 audio-s/s, 0.146 mean audio RTF, 332 ms mean audio TTFP	TTFT -9.2%, E2EL -24.0%, audio RTF -19.8%
Image + video + audio multimodal, 100 prompts at 1.0 req/s	29.07 audio-s/s, 0.186 mean audio RTF, 553 ms mean audio TTFP	TTFT -14.3%, E2EL -34.5%, audio RTF -31.4%

The same infrastructure also benefits text-only routing through Qwen3-Omni. In the text-only long-output workload with 128 prompts at concurrency 32, mean E2EL dropped from the tracked 18.15 s baseline to 14.15 s (-22.1%).

The multi-replica perf suite separately validates the deployment shape where stages 1 and 2 each run two replicas. That suite tracks long text-plus-audio serving at concurrency 8, 16, 24, and 32, with baseline targets for TTFT, TPOT, audio TTFP, and audio RTF. The Stage replicas row in the benchmark sweep above shows the same scaling idea in practice: replicating Talker and Code2Wav improves throughput and latency without duplicating the Thinker stage.

The important result is that the gains show up across different request shapes: long text-plus-audio generation, audio input, image/video input, and mixed image/video/audio input. That is what we want from a pipeline optimization. It improves the common dataflow instead of only helping one narrow prompt pattern.

Acknowledgements

We thank the Qwen3-Omni contributors in vLLM-Omni, including Haiyan Wu, Taichang Zhou, Canlin Guo, Ruirui Yang, Ziming Huang, Wengang Zheng, Lianhao Xu, Han Gao, Junhong Liu, Samit Huang, Hao Chen, Alex Brooks, Chenguang Zheng, Peiqi Yin, Wenjing Chen, Nick Cao, Shunyang Li, Yong Yang, Divyansh Singhvi, Yueqian Lin, Dayu Qiu, Roger Wang and Hongsheng Liu, for their contributions and feedback.

---

References

• Qwen3-Omni pipeline topology in vLLM-Omni: pipeline.py
• Qwen3-Omni model wrapper in vLLM-Omni: qwen3_omni.py
• Qwen3-Omni stage input processors: stage_input_processors/qwen3_omni.py
• Qwen3-Omni deploy profile: qwen3_omni_moe.yaml
• Qwen3-Omni async-chunk perf config: test_qwen3_omni_async_chunk.json
• Qwen3-Omni multi-replica perf config: test_qwen3_omni_multi_replicas.json
• Qwen3-Omni model repository: Qwen/Qwen3-Omni-30B-A3B-Instruct

If you are interested in Qwen3-Omni serving or omni-modality inference, join the #vllm-omni channel in vLLM Slack, or open an issue in vLLM-Omni GitHub.