Streaming conversational modelling

Rates, frames and clocks

A speech system runs several clocks at once, and "the frame rate" can mean any of them. Keep the clocks apart and every number in this chapter becomes easy to check. The anchor to remember is 12.5 Hz, which is one frame every 80 ms.

The ladder from waveform to LLM

The whole chain for one Whisper-style input path:

Other encoders reach 12.5 Hz by other routes. Step-Audio 2 runs a frozen audio encoder at 25 Hz and a 2x adaptor, giving 12.5 Hz (arXiv 2507.16632).

The reason to stop at 12.5 Hz is sequence length. Thirty minutes is 1,800 s, so 50 Hz gives $1800 \times 50 = 90{,}000$ positions (derived). At 12.5 Hz the same audio is $1800 \times 12.5 = 22{,}500$ positions (derived).

Codec frame rates, tokens per second, bitrate

A codec frame is a vector in time. A token is one integer id. A residual quantiser (RVQ) turns each frame into one id per codebook, so token rate and frame rate differ.

Mimi has $f = 12.5$ Hz, $N = 8$ and $V = 2048 = 2^{11}$, so $12.5 \times 8 = 100$ tokens/s and $100 \times 11 = 1100$ bps. That matches the 1.1 kbps in the Mimi paper (arXiv 2410.00037). Confirmed

Text rate, audio rate and duplex clocks

Text is slow and audio is fast, so a model that emits both must choose how to align them. This section shows the two alignments and the clock a duplex model adds on top.

Text tokens against audio frames

Text is slow. Moshi puts text on the same 12.5 Hz grid as audio and fills the empty frames with a PAD token. In English conversational speech padding is about 65% of text-stream tokens (arXiv 2410.00037), so about 35% of frames carry a real token, roughly $0.35 \times 12.5 \approx 4.4$ text tokens/s (derived).

Interleaving models drop the shared grid and fix an order instead. GLM-4-Voice (arXiv 2412.02612) alternates 13 text tokens with 26 speech tokens (1:2). At 12.5 Hz, 26 speech tokens last $26 / 12.5 = 2.08$ s, so the text runs at $13 / 2.08 = 6.25$ tokens/s (derived).

Duplex clocks

A duplex model needs one clock that both directions share. In Moshi, every 80 ms step reads the user's 8 audio codes and emits the agent's text token and 8 audio codes. There is no turn state; silence is just tokens. One temporal-transformer pass plus up to 17 depth sub-steps must finish before the next frame is due. Inference If the temporal pass were free, the budget is $80 / 17 \approx 4.7$ ms per sub-step (derived).

Other systems decide less often than they run:

Latency in frames, video alignment and vocabulary

Latency is easiest to reason about in frames. The same clock arithmetic also aligns audio with video, and fixes the vocabulary for the interview.

Seconds to frames, and Moshi's latency

Divide any delay by 0.08 s to count frames.

Moshi's 160 ms is one 80 ms frame plus one 80 ms acoustic-delay step: $2 \times 80 = 160$ ms (derived decomposition). The paper reports about 200 ms in practice. With acoustic delay 2 the paper states a theoretical latency of 8 timesteps, 640 ms (arXiv 2410.00037). Confirmed

Video alignment

Video frames run at 25 or 30 fps, against 12.5 Hz for audio tokens. At 25 fps there are exactly 2 video frames per audio frame. At 30 fps there are $30 / 12.5 = 2.4$, which is not an integer, so the two streams cannot share a frame grid (derived).

Video latents are coarser than pixel frames. Wan 2.1's causal VAE has a latent length of about $1 + T/4$ for $T$ frames (arXiv 2503.20314), so 25 fps becomes 6.25 latent frames per second and 30 fps becomes 7.5 (derived).

Ovi aligns the two token rates without resampling. It runs RoPE on both streams and scales the audio RoPE frequencies by $31/157 \approx 0.197$, so 31 video latent frames line up with 157 audio tokens (arXiv 2510.01284). That is about $157 / 31 \approx 5.06$ audio tokens per video latent frame (derived).

Say it right in the interview

Timing logic: choosing input and output rates

Two rates matter when you design a speech model: how fast audio enters the LLM, and how fast the LLM must emit audio frames. They are set by different forces, and the output rate is the one that fixes your latency floor. The clocks themselves are defined in Rates, frames and clocks; this section covers how to choose them. Numbers are cited to the paper; arithmetic is marked derived, reasoning is marked Inference.

The input path, step by step

1. 16 kHz waveform, cut into 25 ms windows with a 10 ms hop: 100 Hz mel frames (arXiv 2212.04356).
2. A Whisper-style encoder has a stride-2 convolution: 50 Hz, 20 ms per frame (arXiv 2212.04356; 50 Hz is derived).
3. A pooling or adapter stage cuts the rate again, to 10 to 25 Hz before the LLM.

Inference Input can be fast and continuous because the audio already exists. The encoder reads it in parallel with no sampling loop, so a high rate only costs encoder compute. Output must be generated one step at a time, each step conditioned on the last, so every extra frame per second is an extra serial LLM step. The pooling stage exists to move that cost out of the LLM, where each position is expensive.

The output path: what fixes the rate

The output rate is the frame rate of the audio codec (or tokenizer) the LLM predicts. Each frame becomes one id per codebook, so $\text{tokens/s} = f \times Q$, and the bitrate adds the bits per id:

Mimi: $12.5 \times 8 \times 11 = 1100$ bps (derived), matching the 1.1 kbps in the paper (arXiv 2410.00037). The paper states frame and stride correspond to 80 ms, and that 8 codebooks at 12.5 Hz give "a sequence length of 100 steps per second", against around 3 to 4 text tokens per second for English.

The deeper table of codec bitrates is in Codec frame rates, tokens per second, bitrate.

How to choose the output rate

• Sequence length and compute. Mimi's 100 steps per second is the figure the paper weighs against 3 to 4 text tokens per second (arXiv 2410.00037). VibeVoice's 7.5 Hz is chosen to cut decoding steps (arXiv 2508.19205).
• Reconstruction quality. WavTokenizer scores PESQ 1.70 and UTMOS 3.60 at 40 tokens/s against 2.37 and 4.05 at 75 (arXiv 2408.16532, Table 1). Pooling X-Codec 2 to a 25 Hz variant raises UTMOSv2 on seven languages, for example Dutch 2.168 to 2.457 (arXiv 2601.20185).
• Intelligibility and LM difficulty. SiTok WER is 23.0 at 6.25 Hz, 4.06 at 12.5 Hz and 3.05 at 25 Hz (arXiv 2602.06602). TaDiCodec reaches 6.25 Hz with WER 3.02, but removing the text input at decode causes "significant performance degradation under extremely low token rate" (arXiv 2508.16790). A probing study measures WavTokenizer WER at 11.37% (75 tokens/s) against 24.36% (40 tokens/s) (arXiv 2606.16969, Table 1); it says models below 50 Hz without architectural accommodations degrade, and it attributes its own DAC cliff to a training misconfiguration, so it is not evidence of an intrinsic floor.
• Text-rate matching. GLM-4-Voice alternates 13 text tokens and 26 speech tokens, a 1:2 ratio chosen so text generation is "consistently faster than speech" (arXiv 2412.02612). Details: text and audio clocks.
• Reaction granularity. Inference One frame is the smallest unit the model can emit or react to: 80 ms at 12.5 Hz, 40 ms at 25 Hz.

Why 12.5 Hz

Three pressures meet at 12.5 Hz, and a fourth sets the cost.

• Sequence length and compute. Voxtral cuts its 50 Hz encoder output with a 4x adapter to 12.5 Hz (arXiv 2507.13264). Thirty minutes (1,800 s) at 50 Hz is $1800 \times 50 = 90{,}000$ positions; at 12.5 Hz it is 22,500 (derived).
• Text-rate matching. Moshi puts English at around 3 to 4 text tokens per second against 100 audio steps per second at 8 codebooks (arXiv 2410.00037). At 12.5 Hz that is about $12.5 / 4 \approx 3$ frames per text token (derived).
• The quality knee. SiTok WER is 23.0, 4.06 and 3.05 at 6.25, 12.5 and 25 Hz (arXiv 2602.06602). Going from 12.5 to 25 Hz buys little; going to 6.25 Hz costs a lot.
• Reaction quantum. An 80 ms frame is shorter than the median 151 ms by which human listeners start speaking before the other turn ends in smooth floor transfers (arXiv 2608.25218). Inference So the frame is not the binding constraint on human-like timing.
• The cost. Pooling discards fine acoustic detail. Kimi-Audio keeps continuous 50 Hz Whisper features alongside its 12.5 Hz discrete semantic tokens (arXiv 2504.18425).

Who emits what, per frame

The LLM does not predict one text token per input speech frame. How text lines up with frames depends on the design:

So text and speech are 1:1 per frame only in time-aligned designs, and that is why those designs pad the text stream.

Decoupling the backbone rate from the output rate

If every codebook needed its own backbone step, Q codebooks at 12.5 Hz would force a $12.5 \times Q$ Hz backbone. Designs avoid this by letting one backbone step cover many tokens:

• Moshi: a depth transformer fills K = 2Q + 1 = 17 sub-tokens within each 80 ms frame (arXiv 2410.00037).
• Qwen3-Omni Talker: a linear head predicts codebook 0 per frame, and an MTP module generates the residual codebooks (arXiv 2509.17765).
• VITA-Audio: ten audio tokens in a single forward pass, 3 to 5x speed-up (arXiv 2505.03739).
• MusicGen: a delay pattern shifts codebook k by k-1 steps, so one pass predicts all codebooks (arXiv 2306.05284).
• Patches: VibeVoice and DiTAR predict a patch per step, see continuous autoregression.
• DiffuPlex predicts multiple future user and assistant frames in one backbone wake: 1.46x and 1.59x wall-clock speedups (arXiv 2610.12214).

Minimums and rules of thumb

1. Pool the encoder to 12.5 to 25 Hz before the LLM (arXiv 2507.13264, 2509.17765, 2412.02612, 2504.18425).
2. Default output rate 12.5 Hz (arXiv 2410.00037, 2412.02612, 2509.08753, 2602.06602); SiTok shows a steep WER penalty below it.
3. Go to 25 Hz when reconstruction is the bottleneck (arXiv 2412.10117, 2601.20185; SiTok has its best WER at 25 Hz, arXiv 2602.06602).
4. A lower frame rate is cheaper for the LM (arXiv 2306.06546, 2410.00037).
5. When interleaving, let text lead speech (arXiv 2412.02612 at 1:2; Moshi's 3 to 4 text tokens/s, arXiv 2410.00037).
6. With many codebooks, emit several tokens per backbone step (arXiv 2410.00037, 2509.17765, 2505.03739, 2610.12214).
7. Inference The minimum reaction time is one frame. Practical chunks are larger: Moshi's latency is 160 ms in theory and 200 ms in practice (arXiv 2410.00037); SyncLLM uses chunks of 160 to 240 ms (arXiv 2409.15594).
8. Inference from the source 6.25 Hz is viable only when the transcript is available at decode time (arXiv 2508.16790, ablation).

Worked example

Derived from the rates above, not measured.

The 25 Hz design reacts twice as finely but runs the backbone twice as often (derived). The 12.5 Hz design runs the backbone half as often but adds a depth pass inside each 80 ms deadline. See the per-frame deadline.

Producing text and speech together: Moshi

Voice-to-voice models that "reason and talk" must emit text and audio at the same time. The sources show three coupling patterns. This section covers Moshi, the delayed parallel stream; the next covers delay as a knob and fixed interleaving. The control head that decides when to speak is in the full-duplex sections.

Moshi: the codec and the clock

Moshi (arXiv 2410.00037) models speech with Mimi tokens at 12.5 Hz. Mimi has 8 RVQ levels of 2048 codes. The first level is a semantic code distilled from WavLM; the other seven are acoustic. The codec is causal, with an 80 ms frame. Confirmed

Each 80 ms frame holds 17 streams: 1 text token, 8 Moshi audio tokens and 8 user audio tokens ($K = 2Q + 1 = 17$ with $Q = 8$). The user stream is just another input: the model hears the user continuously while it speaks.

Two transformers per frame

• Temporal transformer: 4096-dim, 32 layers, run once per frame over time. It is initialised from the Helium text LM (about 2.1T training tokens) and outputs a context vector.
• Depth transformer: 1024-dim, 6 layers, random initialisation. It turns the context vector into up to 17 sub-tokens inside the frame.
• The split keeps the big model at 12.5 steps per second. Only the small model runs 17 times per step. Inference

Inner Monologue: text, PAD and delays

Moshi predicts a text token for the agent's own speech on the same 12.5 Hz grid. Frames with no new word carry PAD. The paper also defines an EPAD token, inserted before the next word to mark the end of the padding. Padding is about 65% of text-stream tokens in English conversational speech.

Text is the first sub-step of each frame, so that frame's audio tokens are conditioned on it (derived from the sub-step order: text, Moshi audio, user audio). Inference

Latency follows from the delay: 160 ms theoretical with acoustic delay 1, about 200 ms in practice. The paper also gives 640 ms for acoustic delay 2. The recipe: Helium text LM, then 7M h of audio transcribed with Whisper large-v3 (single stream), then multi-stream post-training on diarised audio, a Fisher fine-tune (2,000 h of phone conversation), and instruction tuning on more than 20k h of synthetic speech (arXiv 2410.00037). Reported

Ways to model streaming ASR and TTS in speech LLMs

Streaming ASR and streaming TTS each split into a handful of families, and each family answers two questions. For ASR: what sets the latency floor, and who decides when to emit a word? For TTS: does the text arrive complete or incrementally, how far does the decoder look ahead, and how many tokens come out per step? Numbers are the authors' own, on different hardware and test sets, so compare shapes, not digits. Background is in Streaming approaches for ASR, Streaming and low latency, Moshi and Delay and interleaving.

Streaming ASR families

Chunked (SpeechLLM-XL). A decoder-only model reads audio in fixed chunks and emits the chunk's text followed by an end-of-chunk token. Training uses CTC forced alignment and an auxiliary CTC loss weighted 0.5. Chunk length is the latency dial: test clean / other WER is 2.5 / 6.5 at 2.56 s, 2.7 / 6.7 at 1.28 s and 3.1 / 7.8 at 0.32 s (arXiv 2410.03752).

Wait token (Speech ReaLLM). An RNN-T-style blank or wait token is predicted after every input embedding, so the LLM decides per frame whether to emit. Latency is the 240 ms input frame plus 960 ms of encoder right context. The 80M model reaches WER 3.0 / 7.4 on LibriSpeech, with RTF 0.79 under greedy decoding (arXiv 2406.09569).

Transducer with an LLM predictor (Transducer-Llama). The transducer keeps its streaming alignment and uses a Llama as the prediction network. With 160 ms segments and a Llama3-8b predictor, WER is 2.47 / 6.53 against 3.65 / 8.96 for a standard RNN-T. The paper uses a weak-to-strong LM swap and MWER fine-tuning (arXiv 2412.16464).

Delayed stream (Kyutai DSM). Text and audio share a clock and the text is simply delayed, 2.5 s by default for ASR. DSM-ASR scores 1.7 / 4.3 on LibriSpeech clean / other, against 3.3 / 4.7 for streaming Parakeet-TDT-v2, with delay precision of about 300 ms (arXiv 2509.08753). This is the family from Delay and interleaving.

Read/write policies. These wrap an offline model. LocalAgreement takes the longest common prefix of n = 2 successive updates over a buffer of about 30 s trimmed at sentence boundaries, at 3.3 s latency on long-form audio in the abstract (Table 3: 3.62 s at a 1.0 s minimum chunk; arXiv 2307.14743). Simul-Whisper stops decoding at the end of the received audio using cross-attention, with truncation detection (arXiv 2406.10052). Its abstract reports 1.46% average degradation at a 1 s chunk; Table 1 gives 2.34 for Large-v2, so I treat the abstract figure with care.

Unified streaming and offline (Qwen3-ASR). The AuT encoder runs at 12.5 Hz with a flash-attention window of 1 to 8 s, so one model serves both modes. The 1.7B model averages 2.69 offline and 3.33 streaming (2 s chunks in the evaluation); the 0.6B model reaches a time to first token as low as 92 ms with vLLM (arXiv 2601.21337).

Inference Two families are not covered because I found no primary source: CTC partials with LLM correction, and learned read/write agents.

Streaming TTS families

1. Full text, causal decoder (Qwen3-TTS). The tokenizer runs at 12.5 Hz and is causal with no look-ahead, so a packet of 4 tokens covers 320 ms. For the 0.6B model, time to first packet is 93 ms for the LM plus 4 ms of decode, 97 ms in total; the 1.7B model takes 101 ms (Table 2; arXiv 2601.15621). The Sesame CSM README documents a single generate call that returns a complete clip and no streaming mode (github.com/SesameAILabs/csm).

2. Incremental text, interleaved. CosyVoice 2 streaming mixes text and speech tokens at N = 5 to M = 15 (arXiv 2412.10117). SyncSpeech starts speaking after 2 text tokens, where CosyVoice 2 needs 5; its first packet takes 0.06 s against 0.22 s for CosyVoice 2 on LibriSpeech test-clean (arXiv 2502.11094). SpeakStream emits each word's speech after the word, using forced-aligned boundaries, at about 30 ms TTS latency (arXiv 2505.19206). This family fits a TTS that consumes another LLM's streamed output.

3. AR LM plus chunk-wise causal flow. CosyVoice 2 trains one flow-matching model under four masks: full-causal, chunk-M, chunk-2M and non-causal. Each mask sets how much future context a chunk sees (arXiv 2412.10117). The paper gives latency formulas, not a measured first packet. Kimi-Audio cuts audio into chunks of about 1 s and appends the next n semantic tokens (for example 4) from the following chunk before decoding. This is training-free and delays the first chunk by n tokens (arXiv 2504.18425, section 2.4).

4. Multi-token prediction (VITA-Audio). The MCTP module predicts about 10 audio tokens per forward pass. The audio token delay is 0, against 1 for Moshi and 13 for GLM-4-Voice (Table 1). The streaming first chunk drops from 236 ms to 53 ms (Table 9; arXiv 2505.03739).

5. Fixed ratio in dialogue models. GLM-4-Voice interleaves 13 text : 26 speech tokens in 0.8 s blocks (arXiv 2412.02612). MinMo uses 5 semantic : 15 speech tokens, with about 600 ms end to end (arXiv 2501.06282). OmniFlatten uses 2 : 10 and reports 193 ms against Moshi at 553 ms (arXiv 2410.17799). LLaMA-Omni 2 reads R = 3 and writes W = 10 tokens; its first speech chunk takes about 583 ms on 7B, the sum of LLM, TTS, flow and vocoder time (arXiv 2505.02625).

Inference A sixth family, non-autoregressive or flow models run chunk by chunk, is not covered because I found no primary streaming paper for F5 or ZipVoice.

How to choose

Inference This maps onto the unified speech LLM design. The listen head is a delayed-stream or chunked ASR. The speak head is incremental-text interleaved TTS, consuming the tokens that the downstream reasoning LLM streams out.

Delay as a knob and interleaved decoders

Moshi's text-audio offset is a design choice, and other systems make the same choice differently. This section covers delay as a knob (DSM, Hibiki) and the fixed-ratio interleaving families.

DSM: delay as a knob

Delayed Streams Modeling (arXiv 2509.08753) pre-aligns streams and applies a fixed delay per stream. In the paper's framing, ASR corresponds to the text stream being delayed relative to audio, and TTS to the opposite.

Inference Delay becomes a per-stream setting, not an architecture. ASR needs audio lookahead before it commits a word. TTS needs text lookahead before it commits prosody. For "text and voice at once", the same decoder with a chosen offset is the lever.

Hibiki: per-word delay

Hibiki (arXiv 2502.03382) is a multistream decoder-only model for speech translation. It sets a delay per word, using the perplexity of an off-the-shelf text translation model (MADLAD-3B) on source prefixes. The target is required to lag the source by at least 2 s. The authors report LAAL of 3.4 s on short-form and 5.0 s on long-form audio, against Seamless at 2.8 s and 4.2 s. On the same tables Hibiki scores ASR-BLEU 35.5 against Seamless at 33.8 (short-form) and 26.6 against 23.9 (long-form).

Interleaving and streaming decoders

Reported All latencies are the authors' own figures under different hardware and definitions.

Inference Moshi aligns text to the clock (padding fills the gaps). Interleaving aligns text to the order (no clock, so no padding, but no free time axis). Thinker-Talker separates the two jobs: the Thinker can finish a thought before the Talker voices it. I found no source here that shows sustained reasoning and live speech together in one duplex model.

Full-duplex models compared

A full-duplex model listens while it speaks. Backchannels, overlaps and barge-in then come from the model's own decisions, not from a VAD timer. The sources group the designs by where the "speak or listen" decision lives.

Survey levels L0 to L3

A 2026 survey (arXiv 2606.19453) defines four levels. L0 is an external scheduler around a half-duplex model. L1 is a control head reading the LLM's hidden states. L2 encodes the duties in the token stream itself. L3 is representation-level and not realised yet. Its line to remember: "architecture sets capacity, training realizes it".

Inference My mapping: control-head models are L1; dual-stream, flattened, chunked and action-channel models are L2.

Families and numbers

Reported Most figures are authors' own. On DuplexSLA-Bench (self-reported, arXiv 2605.20755), backchannel accuracy is 0.33% for gpt-realtime-1.5 (semantic-vad-high), 40.00% for gemini-3.1-flash-live and 98.33% for DuplexSLA. Instruct-FD (arXiv 2607.20460) finds the best of six full-duplex systems follows turn-taking instructions 64.4% of the time.

Wiring the decision and the turn-taking feel

The same families can be read as three wirings of one decision. The wiring explains most of what makes a duplex model feel like, or unlike, a person talking.

Three ways to wire the decision

• Dual-stream: no separate decision; speaking and listening are both tokens on every frame.
• Flattened: one sequence, so who speaks is decided by which role's tokens come next.
• Control head: a small classifier reads hidden states and gates a mostly unchanged LLM. This is why Freeze-Omni can keep its LLM frozen.

The 2026 survey (arXiv 2606.19453) calls SyncLLM an L2-shaped but "apparent (not substantive)" full-duplex system.

What causes the turn-taking feel

Inference Two causes follow from the sources, both about the decision and not about speech quality:

1. Decisions quantised at chunk boundaries. A decision waits for the next boundary, $c/2$ on average for chunk length $c$ (derived). That is 40 ms for an 80 ms frame, 120 ms for a 240 ms chunk and 500 ms for a 1.0 s chunk.
2. Scarce interaction data. The 2025 survey names synchronous data scarcity (arXiv 2509.14515). The 2026 survey finds that observed behaviour is constrained by the interaction patterns in training and evaluation (arXiv 2606.19453).

A related finding: audio foundation models miss when to speak, interrupt too aggressively and rarely backchannel (Talking Turns, arXiv 2503.01174). Inference My reading: the speech path is not the bottleneck. The bottleneck is the duplex decision, plus ingesting user audio while the model is speaking its own.

Turn-taking prediction and the human baseline

Turn-taking is a prediction problem: when does the other speaker stop, and should I start? The field moved from stand-alone predictors to end-of-turn detectors in cascades, and now to duplex models that fold the decision into generation.

Predictors: TurnGPT and VAP

• TurnGPT (Ekstedt and Skantze, Findings of EMNLP 2020, arXiv 2010.10874): a transformer language model that predicts turn shifts in spoken dialogue from text.
• Voice Activity Projection, VAP (Ekstedt and Skantze, Interspeech 2022, arXiv 2205.09812): a self-supervised predictive objective on the voice activity of both speakers, so turn-taking models train without labels.
• Multilingual VAP (LREC-COLING 2024, arXiv 2403.06487): English, Mandarin and Japanese. A model trained on one language predicts poorly on the others; one multilingual model performs on par with the monolingual ones and learns to identify the input language.

End-of-turn models in production

Reported These are vendor numbers, measured on different benchmarks and conditions, so do not rank the vendors from this table. The eot-bench framing is worth borrowing: false-cutoff rate at a fixed latency budget (300 or 600 ms), and mean latency at a fixed false-cutoff target (5% or 10%). It compares systems at an operating point, not at a single threshold.

The human baseline

On smooth transfers, humans start speaking a median of 151 ms before the other turn ends (TurnBench, arXiv 2608.25218). Define the floor-transfer offset as

A cascade that waits for silence can only produce $\Delta > 0$. A negative offset needs prediction, which is what VAP and duplex models do. Inference

TurnBench (arXiv 2608.25218, Aug 2026) is an annotated benchmark of end-of-turn (EOT) and interruption detection. The authors report about 30 h of dual-channel dyadic speech in 154 dialogues, each triple-annotated. A segment is gold when at least 2 of 3 annotators give the same label with endpoints within $\pm 200$ ms. Results are recall, false-positive rate (FPR) and median latency, across 14 turn-taking systems.

The TurnBench authors state the open problem: no current system matches human timing without excessive false positives.

Backchannels, barge-in and self-voice

A good duplex agent tells a backchannel from a barge-in, and does not mistake its own voice for the user's. Both are decision problems with an echo problem underneath.

Backchannel against barge-in

A backchannel ("mm-hm") signals attention and leaves the floor with the speaker. A barge-in takes the floor. Inference A characteristic failure follows: the system treats a backchannel as a barge-in and stops talking.

• Reported DuplexSLA-Bench (arXiv 2605.20755, Table 6, context-prefill setting, authors' numbers): backchannel accuracy 0.33% for gpt-realtime-1.5 with semantic VAD, 40.00% for gemini-3.1-flash-live, 98.33% for DuplexSLA.
• Talking Turns (ICLR 2025, arXiv 2503.01174): in a user study, audio foundation models sometimes miss when to speak, interrupt too aggressively and rarely backchannel. It uses a supervised turn-event model as judge, with test benchmarks curated from Switchboard.
• Full-Duplex-Bench v1.5 (arXiv 2507.23159) scores user interruption, user backchannel, talking to others and background speech. It measures categorical behaviour, stop and response latency, and prosodic adaptation.

Inference Two numbers define good barge-in handling: stop latency, the time from user onset to the last audible agent sample, and the false-yield rate, how often a backchannel, laugh or background voice stops the agent. Optimising only one of them breaks the other.

The full metric proposals, with operating points, are in conversational evaluation; training for these behaviours is in online RL and rewards.

Self-voice and echo

A duplex model hears the speaker output it just produced, unless something removes it. MinMo's duplex module still requires separate acoustic echo cancellation (AEC) and VAD modules (arXiv 2501.06282). SALMONN-omni (arXiv 2411.18138) is instead described as listening to its own generated speech and background sounds while speaking, and SALM-Duplex (arXiv 2505.15670) targets barge-in and turn-taking.

Inference A model that hears its own output needs an explicit reference-signal path: the playback signal goes in as an extra input, so the model or the AEC can subtract what it said from what the microphone heard. Measure stop latency on echo-cancelled output, or a barge-in test passes for the wrong reason.

Streaming inference: deadlines and first packets

A duplex model is a real-time loop, not a request-response call. It must produce a frame on a fixed clock for as long as the session lasts, even when nobody is talking. This section turns the earlier rates into engineering budgets.

The per-frame deadline

Inference Moshi (arXiv 2410.00037) runs at 12.5 Hz, so it has a hard deadline of 80 ms per frame: one temporal pass plus up to 17 depth sub-steps (the paper's K = 17 streams). A missed deadline is an audible gap, because the audio must keep flowing.

The standard check is the real-time factor (RTF), compute time divided by audio time:

RTF below 1 is necessary and not sufficient: the worst frame must still fit in 80 ms. MiniCPM-o 4.5 reports RTF 0.21 (INT4) to 0.27 (FP16) with llama.cpp-omni on an RTX 4090 (arXiv 2604.27393). Reported That is 210 to 270 ms of compute per second of audio (derived).

LLaMA-Omni 2 (arXiv 2505.02625) shows the chunk version of the same arithmetic. Speech tokens run at 25 Hz, so each is 40 ms and a 10-token chunk is 400 ms. The authors report a 542.7 to 663.3 ms first chunk, so roughly 140 to 260 ms of overhead sits on top of the chunk (derived). Inference

The deadline can be relaxed by waking the backbone less often than once per frame. DiffuPlex (arXiv 2610.12214) predicts several future user and assistant frames in one backbone wake, using rolling masked diffusion over an 80 ms interaction interval. The authors report 1.46× (LISTEN) and 1.59× (SPEAK) deployment wall-clock speedups, and 1.61× and 1.80× for the core LM alone.

Chunk size trade-offs

MiniCPM-o's ablation (arXiv 2604.27393) finds 1.0 s gives the best balance. The authors report that shorter chunks leave less modelling budget for control and generation, and that too-short chunks lack enough information per window, causing substantial degradation. Inference Shorter chunks cut the average wait for a decision ($c/2$, derived) but pay the per-step cost more often. Smaller is not automatically better.

First packet against chunk

The first packet is the time until the first playable audio. The chunk is the granularity after that. They are different numbers.

Streaming inference: memory and many sessions

Memory, not compute, often sets how many duplex sessions one GPU can hold. This section covers the cache, long sessions and batching.

KV-cache growth

Inference This arithmetic is derived under stated assumptions: full multi-head attention, bf16 storage, and only the temporal transformer counted (32 layers, model dimension 4096, from Moshi's Table 1, arXiv 2410.00037). The depth transformer's cache can be discarded each frame.

The leading 2 is keys plus values; the trailing 2 is bytes per bf16 value. At 12.5 frames per second that is 512 KiB $\times 12.5 \approx 6.3$ MiB per second, or about 23 GB for one hour (derived). Grouped-query attention would shrink it.

The cache grows linearly with session length and with the number of sessions. Inference For many concurrent sessions, cache memory, not weights, sets capacity.

Long sessions

• The Moshi paper states the model follows multi-turn conversations across 5 minutes (arXiv 2410.00037). Ask what happens beyond that before claiming more.
• One windowing pattern for audio is block-local attention in Qwen2.5-Omni's streaming codec DiT: 2 blocks of lookback and 1 of lookahead (arXiv 2503.20215).
• For video, Self Forcing (arXiv 2506.08009) uses a rolling KV cache for autoregressive video extrapolation at inference. It is a related pattern for unbounded streams, but it is a video model.

Batching many duplex sessions

Inference A turn-based server pays nothing for an idle chat. A duplex session is always active: it needs one frame every 80 ms, including silence. Batching sessions into one GPU step helps throughput, but the whole batched step still has to finish inside 80 ms. The sustainable batch size is the largest $B$ with $t_{\text{step}}(B) < 80$ ms.

DSM-TTS reports 150 ms latency at batch size 1 and about 380 ms at batch size 32 (arXiv 2509.08753, Table 6), so latency figures depend on load.

Streaming decoders and the latency budget

The decoder streams too, and its lookahead is latency. This section ends with the end-to-end budget and a checklist.

Streaming codec decoding

The decoder must also stream, and any lookahead it needs is added latency.

Inference One block of lookahead costs one block of delay. A fully causal decoder such as Mimi's pays none, which is part of why 160 ms is achievable.

Where latency hides

Checklist

• Report latency with its definition: theory, first packet, first chunk, turn-taking delay or total.
• State batch size, GPU and whether the run was cold start.
• Measure the worst frame, not just RTF; a missed 80 ms deadline is audible.
• Compute the KV-cache cost per session before promising concurrency.
• Test barge-in on echo-cancelled output and report stop latency.
• Add the decision wait $c/2$ when comparing chunked and per-frame systems.
• Run a session past the reported context (Moshi: 5 minutes) to see what happens at the limit.

Pre-training and post-training speech-to-speech models

Nearly every open speech-to-speech model follows one recipe. Start from a strong text LLM, add speech tokens, pretrain on ASR, TTS and interleaved speech-text, run spoken-dialogue SFT, then apply DPO or RL on the text side and sometimes the speech side. The real choice is whether the text backbone stays frozen or trains on a mixed corpus.

Every recipe below is the authors' own account in the cited paper. For the TTS-only post-training view (WER and SIM rewards, hacking), see Post-training and evaluation for speech generation. The reward details for online RL are in the next section.

Stage-by-stage recipes

Reported All rows are the authors' own accounts.

Frozen backbone or mixed text: fighting forgetting

Confirmed Adding speech capability can degrade knowledge and reasoning even for text inputs, and more for spoken queries (arXiv 2509.14930). SALAD (arXiv 2510.13632) reports that speech-adapted LLMs underperform their text counterparts and attributes the gap to forgetting and cross-modal misalignment.

Interleaved synthetic pretraining and spoken instruction data

Interleaved pretraining (arXiv 2411.17607) synthesises interleaved speech tokens from text spans with a text-to-token model and a 12.5 Hz supervised tokenizer. It used 1T tokens including 600B synthetic interleaved. Spoken QA rose from 13% (Moshi) to 31%.

Spoken instruction data is synthetic almost everywhere: an LLM writes the dialogue and TTS renders it. Moshi uses over 20k h of synthetic speech (arXiv 2410.00037), and Step-Audio 2 adds synthesised speech-to-speech data in its cooldown (arXiv 2507.16632). Inference The risk is TTS voice and prosody leakage into the model's own speaking style; mix in real recorded turns and many TTS voices.

Preference, RL and distillation

• DPO on text tokens. Moshi-Aligned (arXiv 2506.21463) applies length-normalised DPO to text-token probabilities only, because audio-token probabilities are unstable. It uses 93,490 pairs (27% timing, 73% content), judged by Mistral Large 2. Averaged QA moved 36.1 to 39.2 and safety 70.9 to 77.8 (Moshi-Instruct to Moshi-Aligned).
• Talker DPO. Qwen2.5-Omni ranks preference triplets by WER and punctuation-pause error. Qwen3-Omni runs DPO on multilingual pairs in a four-stage Talker.
• RL. Step-Audio 2 runs PPO with a binary conciseness reward on the thinking trace (60 iterations), PPO with a learned reward model (120), then GRPO for audio perception (400). Qwen3-Omni uses GSPO with rule-based and model-based rewards.
• Distillation. Qwen3-Omni distils strong to weak: off-policy, then on-policy KL against Qwen3-32B and 235B teachers.

Duplex post-training and evaluation gates

DuplexPO (arXiv 2607.07148) runs GRPO-style RL on selected windows from long human conversations. Its four-part reward (FCDR) covers turn initiation, backchanneling, yielding and regularised participation, with a Qwen2.5-7B-Instruct backbone. Reported Intelligence gains are small (Llama Questions 75.3 versus 72.0 for the SFT baseline, Table 4). The point is that timing is optimised without touching content.

Synthesis

Proposed recipe for a full-duplex text+voice model

Online RL and rewards for speech generation

The common pattern: the policy is a codec or discrete-token LM (sometimes flow matching), sampled in groups. Rewards come from the decoded audio: ASR error for intelligibility, speaker-verification (SV) cosine for similarity, a quality or preference predictor for naturalness. The main failure is reward hacking on any single optimised score. Duplex timing is the newest axis, with per-axis rewards derived from VAD events.

The TTS-side summary (DiffRO, F5R-TTS, hacking signatures, listening gates) is in Post-training and evaluation for speech generation. This section adds reward implementations, the duplex rewards, and infrastructure.

Reward taxonomy and concrete implementations

Duplex timing rewards

• DuplexPO (arXiv 2607.07148). The Factorized Conversational Dynamics Reward covers turn initiation, backchanneling, yielding and regularised participation. It optimises only windows around annotated agent events, GRPO-style.
• Kyutai multi-faceted alignment (arXiv 2606.11167). Reported RL post-training of Moshi and PersonaPlex with one reward per axis (pause handling, turn-taking, backchanneling, user interruption) plus an LLM-based response-quality reward against semantic degradation. Per the official model card, GRPO uses rule-based rewards, segments come from Meta's Seamless Interaction corpus via VAD, and evaluation uses Full-Duplex-Bench v1 and v2. Checkpoints: kyutai/moshika-rl-seamless and kyutai/personaplex-rl-seamless.
• ASPIRin (arXiv 2604.10065). GRPO with a binary speaking-or-silence mapping of text tokens. The authors report over 50% fewer duplicate n-grams than standard GRPO.
• Multi-reward DPO for blockwise duplex. Semantic, audio-quality and emotion-consistency rewards aggregated in one DPO objective, studied on blockwise duplex models (arXiv 2601.19063).

Algorithms

GRPO dominates for codec LMs: sample a group of $G$ rollouts per prompt, normalise rewards within the group, and keep a KL penalty to the SFT reference. Standard notation for the advantage of rollout $i$:

$$\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}$$

With several rewards, a common choice is to normalise each reward within the group before summing. Inference GROW (arXiv 2608.03215) standardises its rewards within the group; I have not checked other papers.

Reward hacking: cases and guards

Inference The guards in the right column are my pairing of cases with mitigations, except where a source is named.

Infrastructure and cost

Every rollout must be decoded to a waveform, then passed through an ASR and an SV model. Inference A SpeechJudge-style reward model dominates the cost. Cheaper options are token-level rewards, cached reference embeddings and a small ASR for the reward.

• FunAudioLLM/CosyVoice: GRPO training support and an RL model, per the README.
• zai-org/GLM-TTS: multi-reward GRPO TTS README.
• AmphionTeam/SpeechJudge: the generative reward model.
• Kyutai RL checkpoints: kyutai/moshika-rl-seamless, kyutai/personaplex-rl-seamless.

Bridge from GRPO on streaming ASR

Practical recipes

Monitor: per-reward mean and variance, fraction of zero-advantage groups, held-out metrics, F0 spread, silence ratio, speaking rate, duration distribution, KL, clip fraction and end-of-sequence behaviour. Stop when held-out metrics degrade even if the train reward rises, and spot-check audio.

Evaluating conversational systems

Evaluation moved from demos to automated overlap benchmarks (Full-Duplex-Bench) and annotated dual-channel benchmarks (TurnBench, arXiv 2608.25218, Aug 2026). Latency alone is a weak number: it only means something at a stated false-interruption rate. The open problem, per the TurnBench authors, is that no current system matches human timing without excessive false positives.

Latency definitions and operating points

LiveKit's eot-bench (LiveKit blog) reports two views of the same detector: the false-cutoff rate at a fixed latency budget (300 or 600 ms), and the mean latency at a fixed false-cutoff target (5% or 10%). Reported For LiveKit's own v1, the blog gives 9.9% false cutoffs at 300 ms and 4.5% at 600 ms, and 543 ms mean latency at 5% and 295 ms at 10%. It is a vendor benchmark, but the operating-point framing is reusable for any turn-taking system.

Vendor latency claims are not comparable. Deepgram's Flux docs state "~260ms end-of-turn detection" inside the STT model, without a stated benchmark. That cannot be compared with the eot-bench operating points above.

Benchmarks

Reported numbers

• TurnBench. Confirmed The VAP baseline scores end-of-turn recall 0.845, false-positive rate (FPR) 0.055, median latency 368 ms; interruption recall 0.945, FPR 0.107, median 994 ms. Humans on smooth transfers start a median 151 ms before the current turn ends.
• DuplexSLA-Bench (arXiv 2605.20755). Reported Authors' backchannel accuracy, context-prefill setting: gpt-realtime-1.5 0.33% (semantic VAD), gemini-3.1-flash-live 40.00%, DuplexSLA 98.33%.
• Instruct-FD (arXiv 2607.20460). Reported The authors report that the best of six full-duplex systems achieves only 64.4% adherence to turn-taking instructions.
• PersonaPlex (arXiv 2602.06053). Reported The authors report 0.170 s smooth turn-taking and 0.240 s user-interruption latency for the released 7B checkpoint, against 0.265 s and 0.257 s for Moshi.

Ten proposed latency-aware and interaction-based metrics

Inference This list is my proposal, built on the benchmarks above. Every timing metric uses gold event labels and echo-cancelled agent output, never the system's own VAD.

The set separates timing from content, in line with DuplexPO, and ties latency to an operating point.