Quick read

Foundations: the S2S design space

• Every S2S system answers five questions: input rep, output rep, text/speech coupling, duplex scheme, acoustic renderer.
• Core tension: speech encoders emit 50 Hz frames (Whisper in Voxtral; arXiv 2507.13264) while text is far sparser.
• Each choice tries to give the LLM a text-like-cost sequence without losing prosody, speaker identity or timing.
• If the backbone sees embeddings and emits tokens for a separate renderer, the LLM can be swapped, frozen or lightly tuned (Freeze-Omni freezes it; arXiv 2411.00774).
• Open problem: keeping text-level intelligence once the model must speak.
• URO-Bench reports that open-source spoken dialogue models lag their backbone LLMs on instruction following and suffer catastrophic forgetting (arXiv 2502.17810).

Full chapter

Speech understanding: input and ASR

• Most models feed the LLM continuous embeddings from an ASR-pretrained encoder plus a small projector: Voxtral (arXiv 2507.13264), Qwen3-Omni (arXiv 2509.17765), Granite-speech (arXiv 2505.08699), Step-Audio 2 (arXiv 2507.16632), Canary-Qwen.
• Discrete tokens: GLM-4-Voice uses a single-codebook VQ at 12.5 Hz, 175 bps (arXiv 2412.02612).
• Codec tokens: Moshi's Mimi runs at 12.5 Hz with 8 codebooks (arXiv 2410.00037); Inference this suits duplex.
• 12.5 Hz converges: Whisper emits 50 Hz frames, and 30 minutes at 50 Hz is a 90k-token sequence, so Voxtral adds a 4x downsampling adapter (arXiv 2507.13264).
• Inference Downsampling to 10-12.5 Hz is the main sequence-length lever and costs fine acoustic detail, which motivates hybrids like Kimi-Audio (arXiv 2504.18425).
• Canary-Qwen freezes the LLM and trains encoder, projection and LoRA; the card reports 5.63 mean WER and RTFx 418 on the Open ASR Leaderboard (NVIDIA model card).

Inference Chunk size and lookahead trade latency against WER.

Full chapter

How audio and ASR text meet in the LLM

• Prefix: projected audio frames, then a task prompt, then text; text is predicted given the audio (Qwen2-Audio, arXiv 2407.10759; Voxtral, arXiv 2507.13264).
• Chunk-interleaved: audio chunk, its text, then a $ end-of-chunk token; 1.28 s chunks, CTC forced alignment for targets (SpeechLLM-XL, arXiv 2410.03752).
• Per-frame wait: after each 240 ms embedding the model emits zero or more words, then BLANK (Speech ReaLLM, arXiv 2406.09569).
• Transducer: the LLM is the non-blank predictor; encoder frames enter through the joint network (Transducer-Llama, arXiv 2412.16464).
• Time-aligned streams: per-stream embeddings summed into one vector per 80 ms step; ASR text is delayed, 2.5 s by default in DSM (Moshi, arXiv 2410.00037; DSM, arXiv 2509.08753).
• Hybrid input: Whisper features (50 to 12.5 Hz) added to semantic-token embeddings; the Introduction says concatenated, the method sections say added (Kimi-Audio, arXiv 2504.18425).
• Audio positions: Qwen3-Omni TM-RoPE gives audio temporal IDs of 80 ms each (arXiv 2509.17765).

Full section

Whisper in one page-half

• Encoder-decoder Transformer, 680,000 h weakly supervised; one model does ASR, X-to-English translation and language ID via task tokens (arXiv 2212.04356).
• Hz ladder: 16 kHz audio, 100 Hz log-mel (80 bins; 128 in large-v3), 50 Hz encoder frames (1,500 per 30 s window), 12.5 Hz after a 4x adapter.
• Training: web transcripts filtered for machine-generated text, audio language ID checked against the transcript; large-v2 trained 2.5x more epochs with SpecAugment.
• large-v3 used 1M h weakly labelled plus 4M h pseudo-labelled with large-v2 (OpenAI model card).
• Long-form: sliding 30 s windows, beam 5, temperature fallback when log-probability is below -1 or compression ratio above 2.4, no-speech threshold 0.6.
• Limits (model card): hallucination, repetitive text, not real-time out of the box; the fixed 30 s window and non-causal encoder make streaming hard.
• large-v3-turbo cuts the decoder from 32 to 4 layers (model card).
• Speech LLMs reuse the encoder (Qwen2-Audio, arXiv 2407.10759; Voxtral, arXiv 2507.13264; GLM-4-Voice makes it causal, arXiv 2412.02612).
• Qwen3-Omni replaced it with AuT, trained from scratch on 20M h of supervised audio (arXiv 2509.17765).
• Streaming the encoder: causal fine-tune (CarelessWhisper, arXiv 2508.12301; GLM-4-Voice, arXiv 2412.02612), a windowed encoder with a 1 to 8 s window (AuT, arXiv 2601.21337), or a wrapper that re-decodes a buffer (whisper_streaming, arXiv 2307.14743).
• Look-ahead (right context) trades latency for accuracy: NVIDIA's cache-aware FastConformer reports RNNT WER 9.5% at 0 ms against 6.3% at 1,360 ms, and chunk-aware look-ahead beats per-layer look-ahead at the same latency (arXiv 2312.17279). Look-ahead section

Full section

Speech generation: codecs, AR, flow, streaming

• The tokenizer fixes the LM's sequence length and the quality ceiling; the trade is rate against fidelity.
• Mimi is causal, 12.5 Hz, 1.1 kbps, first codebook distilled from WavLM (arXiv 2410.00037).
• WavTokenizer runs at 40 or 75 tokens/s (arXiv 2408.16532); VibeVoice uses continuous 7.5 Hz latents (arXiv 2508.19205).
• A controlled frame-rate ablation traces the reported quality cliff at 6.25 Hz to the training configuration, not a fundamental barrier (arXiv 2606.16969).
• The dominant high-quality design is hybrid: an AR LM for content and prosody, flow matching for acoustics (CosyVoice 2, arXiv 2412.10117; Qwen3-TTS, arXiv 2601.15621; FlashTTS, arXiv 2606.09141).
• Inference Flow matching won because it has a plain regression target, straighter paths, fewer steps and natural infilling.
• Continuous AR (MELLE, arXiv 2407.08551; DiTAR, arXiv 2502.03930; VibeVoice) removes quantization loss but adds per-frame diffusion or flow cost, and RL is harder without discrete tokens (arXiv 2608.03215).
• Inference Latency levers: for flow decoders, step distillation (FlashTTS 325 ms at 2 NFE; its table lists 843 ms for CosyVoice 2 at 10 NFE); for AR, lower token rate plus a causal codec.

Full chapter

Speech-to-speech systems: architectures, duplex, multilingual

• Six coupling families, ordered by how tightly speech is fused into the LLM: cascaded, half-cascade, Thinker-Talker, interleaved, parallel streams, codec-free.
• Speaking costs reasoning quality; causes reported are catastrophic forgetting and modality inequivalence (arXiv 2509.14930).
• Mitigations: a frozen LLM (arXiv 2411.00774), cross-modal distillation from a text teacher (arXiv 2509.14930), layer-wise hidden-state alignment with RL (arXiv 2601.05543).
• Inference Half-cascade or Thinker-Talker with a frozen or lightly tuned LLM is the pragmatic route; native interleaved is the frontier.
• Duplex schemes: dual-stream (Moshi, arXiv 2410.00037; dGSLM, arXiv 2203.16502), time-multiplexed (OmniFlatten, arXiv 2410.17799), state head (Freeze-Omni; MinMo, arXiv 2501.06282), shared timeline (MiniCPM-o 4.5, arXiv 2604.27393), cascade-duplex (DuplexCascade, arXiv 2603.09180).
• Data is the bottleneck: Fisher is about 2,000 h of two-channel telephone speech (arXiv 2410.00037); unlabelled audio scales (Kimi-Audio more than 13M h, arXiv 2504.18425).
• Benchmarks: Full-Duplex-Bench (arXiv 2503.04721) and v1.5 (arXiv 2507.23159), VoiceBench (arXiv 2410.17196), URO-Bench (arXiv 2502.17810); evaluation is maturing but not comparable across papers.
• Multilingual is lopsided: Qwen3-Omni has text in 119 languages, speech understanding in 19 and speech generation in 10 (arXiv 2509.17765).
• Text streams serve as supervision, scaffold, alignment and capability transfer.

Reported The DuplexSLA authors report 98.33% backchannel accuracy on their own benchmark vs 40% or less for the closed-source baselines they tested (arXiv 2605.20755). NVIDIA's PersonaPlex model card lists a smooth turn-taking rate of 0.908 and latencies of 0.170 and 0.240 on Full-Duplex-Bench, without units (model card).

Full chapter

Streaming conversation: rates, Moshi and full-duplex models

• Moshi runs text, its own audio and user audio on one 12.5 Hz clock: 17 streams per 80 ms frame (arXiv 2410.00037).
• A 4096-dim, 32-layer temporal transformer runs once per frame; a 1024-dim, 6-layer depth transformer fills the sub-tokens.
• Moshi's text stream is about 65% PAD, so it fits the audio clock; GLM-4-Voice instead interleaves 13 text : 26 speech tokens (arXiv 2412.02612).
• Moshi's latency is 160 ms in theory and about 200 ms in practice; DSM applies a fixed delay per stream, 2.5 s default for ASR text (arXiv 2509.08753).
• Duplex families: dual-stream (Moshi), chunked (SyncLLM, arXiv 2409.15594), flattened (OmniFlatten, arXiv 2410.17799), control head (MinMo, arXiv 2501.06282; Freeze-Omni, arXiv 2411.00774), action channel (DuplexSLA, arXiv 2605.20755).
• Latency figures use different definitions and are not comparable across systems.
• Tokens per second is frame rate times codebooks, and bitrate adds the bits per token: $12.5 \times 8 \times 11 = 1100$ bps for Mimi (derived).
• Lower output rates shorten sequences but cost intelligibility: SiTok WER is 23.0, 4.06 and 3.05 at 6.25, 12.5 and 25 Hz (arXiv 2602.06602). Timing logic

Full chapter

Streaming ASR and TTS: the modelling families

ASR families:

• Chunked with an end-of-chunk token: WER 2.7 / 6.7 at 1.28 s chunks, 3.1 / 7.8 at 0.32 s (SpeechLLM-XL, arXiv 2410.03752).
• Per-frame wait token: 240 ms frame plus 960 ms right context, WER 3.0 / 7.4 (Speech ReaLLM, arXiv 2406.09569).
• Transducer with LLM predictor: 160 ms segments, WER 2.47 / 6.53 (Transducer-Llama, arXiv 2412.16464).
• Delayed text stream: 2.5 s default delay, WER 1.7 / 4.3 (Kyutai DSM, arXiv 2509.08753).
• Read/write policy on Whisper: LocalAgreement, 3.3 s long-form latency (arXiv 2307.14743).
• One encoder, dynamic 1 to 8 s window: 2.69 offline, 3.33 streaming (Qwen3-ASR, arXiv 2601.21337).

TTS families:

• Full text, causal decoder: 4-token packet = 320 ms, 97 ms first packet (Qwen3-TTS, arXiv 2601.15621).
• Incremental text, interleaved: starts after 2 text tokens, 0.06 s against 0.22 s (SyncSpeech, arXiv 2502.11094).
• AR plus chunk-wise causal flow: four masks and N = 5 : M = 15 in CosyVoice 2 (arXiv 2412.10117).
• Multi-token prediction: audio delay 0 against 1 (Moshi) and 13 (GLM-4-Voice); first chunk 236 to 53 ms (VITA-Audio, arXiv 2505.03739).
• Fixed dialogue ratio: LLaMA-Omni 2 first speech chunk about 583 ms (arXiv 2505.02625).

Full section

Streaming conversation: training, RL rewards and evaluation

• Recipe: strong text LLM, speech tokens, ASR/TTS and interleaved pretraining, spoken SFT, then DPO or RL.
• The real choice is frozen LLM (Freeze-Omni, MinMo) or a mixed text corpus (Moshi, Kimi-Audio, Step-Audio 2).
• DPO goes on text-token probabilities because audio-token probabilities are unstable (Moshi-Aligned, arXiv 2506.21463).
• RL: Step-Audio 2 uses PPO then GRPO (arXiv 2507.16632); Qwen3-Omni uses GSPO (arXiv 2509.17765).
• Duplex rewards are timing events on audio: DuplexPO (arXiv 2607.07148) and Kyutai's alignment of Moshi and PersonaPlex (arXiv 2606.11167), the latter adding an LLM response-quality reward against semantic degradation.
• Kyutai publishes its RL-tuned checkpoints (model card).
• Evaluate on Full-Duplex-Bench v1 (arXiv 2503.04721), v1.5 (arXiv 2507.23159), v2 (arXiv 2510.07838) and TurnBench (arXiv 2608.25218).
• Humans on smooth transfers start speaking a median 151 ms before the other turn ends (TurnBench).
• Inference Report latency at a fixed false-yield rate, measured from gold end-of-turn.

Full chapter

Control and quality: steering, post-training, evaluation

• Five control channels, usually combined: reference clip, natural-language instruction, inline tags, duration targets, style guidance at sampling time.
• IndexTTS2 takes timbre from one prompt and emotional tone from a style prompt, so they can come from different speakers, and offers explicit-token-count duration control (arXiv 2506.21619).
• The IndexTTS2 authors motivate exact durations by video dubbing; Inference this is the hook for avatar sync.
• Inline tags: ElevenLabs bracket tags such as [whispers], less reliable for deliveries the voice was not trained on (ElevenLabs docs).
• Post-training uses WER and speaker-similarity rewards: DiffRO scores codec tokens with no waveform in the loop (arXiv 2505.17589), F5R-TTS applies GRPO to flow outputs (arXiv 2504.02407), GROW does on-policy RL on the flow-matching objective (arXiv 2608.03215).
• The recurring failure is reward hacking: GRPO on CER and NLL rewards collapsed prosody into monotone speech; iterative DPO with a few hundred human preference pairs per round fixed it (arXiv 2509.18531).
• RRPO reports that DiffRO-style reward models can be exploited with acoustic artifacts (arXiv 2512.04552).
• Inference Always pair automatic rewards with a naturalness term and listening tests.
• Synthesia Express-2 (4 Sep 2025) pairs Express-Voice with Express-Video; the blind-test result is a company claim with no scores (Synthesia post) Reported.

Full chapter

Multimodal generation: audio-visual diffusion and AR models

• Inference Video backbones share one recipe: 3D causal VAE, DiT, flow matching. Examples: Wan 2.1 (arXiv 2503.20314), HunyuanVideo 1.5 (arXiv 2511.18870), Seedance 1.5 pro (arXiv 2512.13507).
• Audio-driven avatars add audio conditioning to video backbones: OmniHuman-1.5 (arXiv 2508.19209), HunyuanVideo-Avatar (arXiv 2505.20156), MultiTalk (arXiv 2505.22647), InfiniteTalk (arXiv 2508.14033).
• Ovi (arXiv 2510.01284) generates audio and video jointly: twin DiT towers, bidirectional cross-attention in every block, audio RoPE scaled to match 31 video latent frames against 157 audio tokens.
• UniVerse-1 takes the other route: pretrained Wan2.1 and ACE-Step experts joined by lightweight cross-modal MLP connectors (arXiv 2509.06155).
• Streaming distils a bidirectional teacher into a causal few-step student with a KV cache (CausVid, Self Forcing, LiveTalk); error accumulation over long rollouts is the failure mode.
• AR image models split into discrete (VAR, arXiv 2404.02905; Emu3, arXiv 2409.18869), continuous with a diffusion head (MAR, arXiv 2406.11838; Fluid, arXiv 2410.13863), and hybrids (Transfusion, arXiv 2408.11039; Show-o2, arXiv 2506.15564; BAGEL, arXiv 2505.14683).
• Inference MELLE, DiTAR and VibeVoice are the audio analogue of MAR and Fluid.

Full chapter

Multimodal generation: tokenizers and multimodal LLMs

• Inference The tokenizer matters more than the generator: the consensus is a causal, heavily compressed latent regularised toward semantic features.
• Inference Discrete codecs stay the default for audio LMs; continuous VAE latents stay the default for diffusion and flow.
• Discrete examples: MAGVIT-v2 LFQ (arXiv 2310.05737), FSQ (arXiv 2309.15505), DAC (arXiv 2306.06546); continuous examples: Wan VAE (arXiv 2503.20314), Stable Audio Open AE (arXiv 2407.14358).
• Best reconstruction is not best downstream: DASB reports discrete audio representations are less robust than continuous ones (arXiv 2406.14294).
• AVTok is a 1D joint audio-video tokenizer (arXiv 2606.30811, Reported); VideoPoet concatenated MAGVIT-v2 and SoundStream vocabularies instead (arXiv 2312.14125).
• Open multimodal LLMs share one template: vision encoder, token-reducing connector, decoder LLM (Qwen3-VL, arXiv 2511.21631; InternVL3.5, arXiv 2508.18265).
• Qwen3-Omni's TM-RoPE gives audio one temporal ID per 80 ms and adjusts video frame IDs to real timestamps; Qwen2.5-Omni interleaved audio and video in fixed 2 s chunks (arXiv 2509.17765, arXiv 2503.20215).
• StreamingVLM keeps a fixed-size KV cache and runs at up to 8 FPS on one H100 (arXiv 2510.09608).

Full chapter

Common problems: speech and audio-visual

Ranking, fixes and metrics are my judgement Inference.

• LLM-based TTS can repeat or drop words; certain cross-attention heads learn the alignment and nothing forces it to stay monotonic, and CTC loss plus attention priors encourage monotonic attention (arXiv 2406.17957) Reported.
• Sync-C and Sync-D are based on SyncNet (arXiv 2403.06421); THEval's authors report limited correlation with human evaluation and unreasonable values (arXiv 2511.04520) Reported.
• Synthesia's documented name fix is a per-word pronunciation entry, respelling or IPA (script docs) Confirmed.
• A 2026 survey calls a backchannel misread as a barge-in the most-cited duplex failure in user feedback (arXiv 2606.19453) Reported.
• The Express-2 post gives no quantitative results, failure cases or latency figures (post) Confirmed.

Full section

System design: the unified speech LLM

• Inference One speech model owns everything audio: streaming ASR with emotion tags, turn-taking, backchannels, barge-in and expressive TTS on one 12.5 Hz clock.
• Inference Reasoning stays in a swappable text LLM behind a tool call: transcript, tags and turn state go in, streamed text comes back and is spoken.
• Inference Fillers and backchannels come from the speech LLM, so the extra LLM hop is covered; the hop and two models to serve are the honest costs.
• Inference The speech LLM, not the text LLM, yields on a real barge-in, ignores a user "mm-hm", and sends the played and heard text back.
• Precedents Confirmed: Kyutai DSM (ASR and TTS as delays of one architecture; arXiv 2509.08753), Step-Audio 2 tool calls (arXiv 2507.16632), Qwen3-Omni Thinker-Talker (arXiv 2509.17765), Freeze-Omni (arXiv 2411.00774) and MinMo (arXiv 2501.06282) with small decision modules beside a text LLM.
• DuplexSLA is the precedent for an action channel: 160 ms chunk clock, 98.33% backchannel accuracy, self-reported (arXiv 2605.20755) Reported.
• Training data: unlabelled audio (Moshi 7M h, Kimi-Audio over 13M h), interleaved synthetic speech-text (spoken QA 13% to 31%; arXiv 2411.17607), and dual-channel conversation (Fisher, about 2,000 h; arXiv 2410.00037).
• Then DPO on text tokens (arXiv 2506.21463) and timing RL (DuplexPO, arXiv 2607.07148; Kyutai, arXiv 2606.11167).
• Inference Native full-duplex is the research track; the independent ASR, LLM and TTS cascade is the baseline and fallback.

Full section

System design: voice agent and talking avatar

• Inference Target: a unified speech LLM with a downstream text LLM (previous page). The cascade below is the baseline, launch fallback and comparison point.
• A cascaded voice agent is three models plus turn logic, run as overlapping streams; latency comes from overlap, not any single model.
• Inference Use WebRTC to the client (LiveKit and Pipecat package this) with a control data channel; persistent bidirectional streams between servers, with data and control planes separate.
• Replace a bare VAD timeout with a semantic end-of-turn model, such as Pipecat's Smart Turn (docs) Reported. Inference It shifts the dead-air versus cut-off curve but does not remove it.
• Inference Never feed raw partials to the LLM; start it at the end-of-turn candidate and cancel on change.
• Inference Chunk LLM text for TTS with a small first chunk at a clause boundary, then full sentences.
• Barge-in: cancel LLM and TTS, flush buffers, truncate context to what was actually played, return to listening. OpenAI's Realtime API exposes truncation as conversation.item.truncate (guide) Reported.
• Echo cancellation is the precondition; the W3C echoCancellation property removes speaker sound from the microphone (W3C spec) Confirmed.
• Inference Avatar: the speech LLM drives a causal few-step renderer, audio is the clock, the cascade is the fallback and native duplex is the research track.
• Streaming renderers: CausVid 9.4 FPS at 1.3 s initial latency (arXiv 2412.07772); Self Forcing 17 FPS on one H100 (arXiv 2506.08009); author-reported, not comparable.

Cascade fallback budget (my illustrative targets, no source):

Full chapter

System design: audio-visual avatar platform

• Two products, two SLOs: batch video (render takes 3-10 minutes, API docs; 1080p, 30 fps, Express-2 post) and interactive agents ("typically under one second", Synthesia) Confirmed.
• Custom avatars sit behind a live consent video that must match the photo (personal avatars docs) Confirmed.
• Inference Shared consent-gated identity and voice stores; consent is a hard pipeline stage.
• Batch: Express-Voice, Express-Animate, Express-Eval and Express-Render, distilled to two diffusion steps (Express-2 post) Confirmed. Dubbing chain (ASR, MT, duration-controlled TTS) Inference.
• Inference Audio-then-motion cascade in production; joint audio-video (Ovi, arXiv 2510.01284; Seedance 1.5 pro, arXiv 2512.13507) as a research track; causal few-step student for interactive.
• Inference Interactive voice: a unified speech LLM with a text LLM behind a tool call; the ASR, LLM, TTS cascade is the fallback and baseline.
• One clock Inference: Ovi scales audio RoPE by 31/157 to align 31 video latent frames with 157 audio tokens (arXiv 2510.01284); Qwen3-Omni assigns one temporal ID per 80 ms of audio (arXiv 2509.17765).
• Perceiving the user: StreamingVLM keeps a fixed-size KV cache, up to 8 FPS on one H100 (arXiv 2510.09608).
• Cross-paper Sync-C is not comparable, so build an internal benchmark first Inference.
• GRPO on CER and NLL rewards collapsed prosody into monotone speech, so guard rewards against hacking (arXiv 2509.18531).

Full section

Model cheat sheet

Every cell comes from the cited paper; years are arXiv first-submission years, latency figures are the authors' own, and "n/s" means no sourced value.

Five sentences to remember

Inference My synthesis of the table, not a sourced claim.

1. Several systems here run audio at about 12.5 Hz (Moshi, GLM-4-Voice, Kimi-Audio, Step-Audio 2, Qwen3-Omni).
2. Output is often an LM for content plus a flow or ConvNet renderer.
3. Text is the scaffold and the supervision.
4. Duplex looks limited by data and evaluation more than by architecture.
5. Latency comes from token rate, NFE and chunking.

Full chapter