Speech-to-speech modelling
Speech-to-speech systems
S2S architectures with the LLM as a downstream component
Six families, ordered by how tightly speech is fused into the LLM. The first three are the most conservative designs; the last three fuse speech into the LLM most tightly.
The six families
| Family | Models | How text and speech are produced together |
|---|---|---|
| Cascaded | ASR, LLM, TTS | Sequential; latencies add (Moshi, arXiv 2410.00037, cites a latency of several seconds) |
| Half-cascade | LLaMA-Omni 2 (arXiv 2505.02625), Freeze-Omni (arXiv 2411.00774) | Speech encoder into text LLM; a streaming speech decoder consumes LLM text or hidden states |
| Thinker-Talker | Qwen3-Omni (arXiv 2509.17765), Qwen3.5-Omni (arXiv 2604.15804) | Thinker produces text; Talker conditions on it and emits speech codes |
| Interleaved | GLM-4-Voice (arXiv 2412.02612), Step-Audio 2 (arXiv 2507.16632), VITA-Audio (arXiv 2505.03739) | One sequence alternates text and audio tokens |
| Parallel streams | Moshi (arXiv 2410.00037), Kimi-Audio (arXiv 2504.18425) | Moshi: text tokens run time-aligned with audio tokens (Inner Monologue). Kimi-Audio: shared transformer layers, then parallel text and audio heads |
| Codec-free | SALMONN-omni (arXiv 2411.18138) | Embeddings instead of codec tokens; a "thinking" mechanism facilitates asynchronous text and speech generation |
Key facts per model
- LLaMA-Omni 2: 0.5B to 14B on Qwen2.5, Whisper-large-v3 encoder, a streaming speech decoder inspired by CosyVoice 2, trained on 200K multi-turn dialogues; the authors report beating GLM-4-Voice (arXiv 2505.02625).
- Freeze-Omni: frozen LLM, so the authors report speech-mode intelligence at the same level as the text mode of the backbone, plus chunk-wise state prediction for duplex dialogue (arXiv 2411.00774).
- Qwen3-Omni: text in 119 languages, speech understanding in 19, speech generation in 10 (arXiv 2509.17765).
- Qwen3.5-Omni (Apr 2026): hybrid-attention MoE Thinker and Talker, 256k context, and ARIA, which dynamically aligns text and speech units to stabilise streaming speech and prosody (arXiv 2604.15804). The authors claim SOTA on 215 audio and audio-visual subtasks.
- Kimi-Audio: continuous audio features in, discrete tokens out, more than 13M h of audio pretraining (arXiv 2504.18425).
- MiniCPM-o 2.6: 8B, built from SigLip-400M, Whisper-medium-300M, ChatTTS-200M and Qwen2.5-7B, with a time-division multiplexing mechanism for streaming (OpenBMB model card).
- Commercial: OpenAI's Realtime API model "works directly with audio" (OpenAI docs). Google's Live API offers native audio output models (Gemini API docs).
The intelligence gap
Speaking costs reasoning quality. Causes reported: catastrophic forgetting and modality inequivalence (arXiv 2509.14930), and a bottleneck in condensing redundant speech into stable late-layer decisions (arXiv 2603.01502). Mitigations: a frozen LLM (arXiv 2411.00774), cross-modal distillation from a text teacher (arXiv 2509.14930), layer-wise hidden-state alignment with RL (arXiv 2601.05543), EchoX (arXiv 2509.09174).
Inference Half-cascade or Thinker-Talker with a frozen or lightly tuned LLM looks like the pragmatic route to keep text-level quality. Native interleaved is the frontier.
"I think of the LLM as swappable. The real design choice is how text and speech are coupled in time: sequentially, by a Thinker-Talker handoff, interleaved, or as parallel streams."
Full-duplex and backchannelling
Full duplex means the model listens and speaks at once, so it can overlap, interrupt and backchannel without explicit turn logic. This is the target capability for the Synthesia voice role.
Schemes
| Scheme | Models | Mechanism |
|---|---|---|
| Dual-stream | Moshi (arXiv 2410.00037), dGSLM (arXiv 2203.16502) | Own and user audio as parallel streams, no explicit speaker turns. dGSLM is textless, trained on 2000 h of two-channel Fisher audio, and generates speech, laughter and other paralinguistic signals in both channels |
| Time-multiplexed / flattened | OmniFlatten (arXiv 2410.17799) | One flattened sequence; three stages: modality alignment, half-duplex, full-duplex; backbone architecture unchanged |
| State head | Freeze-Omni (arXiv 2411.00774), MinMo (arXiv 2501.06282) | Freeze-Omni: chunk-wise state prediction on the LLM's hidden state. MinMo: a duplex predictor plus a duplex alignment stage, about 8B parameters, 1.4M h of speech |
| Shared timeline | MiniCPM-o 4.5 (arXiv 2604.27393) | 9B parameters; "Omni-Flow" aligns omni-modal inputs and outputs on a shared temporal axis; runs on edge devices with under 12 GB RAM (authors) |
| Cascade-duplex | DuplexCascade (arXiv 2603.09180) | VAD-free cascade that aims for duplex behaviour while keeping cascade strengths |
Reported Self-reported results. The DuplexSLA authors report 98.33% backchannel accuracy on their own benchmark, against 40% or less for the closed-source baselines they tested (arXiv 2605.20755). NVIDIA's PersonaPlex model card lists a smooth turn-taking rate of 0.908 and latencies of 0.170 and 0.240 on Full-Duplex-Bench, without units on the card (model card; paper arXiv 2602.06053).
Benchmarks
- Full-Duplex-Bench v1 (arXiv 2503.04721): pause handling, backchanneling, turn-taking, interruption management; automatic metrics.
- v1.5 (arXiv 2507.23159): overlap scenarios (user interruption, user backchannel, talking to others, background speech), stop and response latency. It finds two strategies: responsive and floor-holding.
- ECHO (arXiv 2609.17360): paired Chinese yield-or-keep turn-taking benchmark; the authors find three of four systems over-yield on backchannels.
- VoiceBench (arXiv 2410.17196): multi-faceted evaluation of LLM-based voice assistants.
- URO-Bench (arXiv 2502.17810): open-source spoken dialogue models lag behind their backbone LLMs on instruction following.
Evaluation is maturing but not comparable across papers.
Data is the bottleneck
| Source | Detail |
|---|---|
| Fisher | About 2,000 h of two-channel telephone conversation, 8 kHz; Moshi upsamples it to 24 kHz (arXiv 2410.00037) |
| Synthetic dialogue via TTS | LLaMA-Omni 2 (200K dialogues, arXiv 2505.02625), Hibiki (aligned synthetic data, arXiv 2502.03382), Freeze-Omni (60,000 text Q&A, arXiv 2411.00774), SyncLLM (212k h synthetic plus 2k h real, arXiv 2409.15594) |
| Unlabelled audio | Kimi-Audio more than 13M h (arXiv 2504.18425); MinMo 1.4M h (arXiv 2501.06282); Moshi 7M h (arXiv 2410.00037) |
A 2026 survey (arXiv 2606.19453) proposes an L0-L3 hierarchy and an IDLE/LISTEN/SPEAK/WAIT/DUAL state machine. It finds that observed behaviour is constrained by the interaction patterns in training and evaluation, and that public training-data coverage is limited. A 2025 survey (arXiv 2509.14515) names synchronous data scarcity, architectural divergence and evaluation gaps as the main challenges.
"Architecture is mostly solved: dual-stream or state-head. The bottleneck is dual-channel conversational data with real overlap and backchannels, and a benchmark everyone shares."
Multimodal, multilingual S2S
Multilingual S2S is lopsided: models understand far more languages than they speak. Text is the bridge that makes the rest possible.
Coverage asymmetry
| Model | Text | Speech understanding | Speech generation |
|---|---|---|---|
| Qwen3-Omni (arXiv 2509.17765) | 119 languages | 19 | 10 |
| Omnilingual ASR (arXiv 2511.09690) | n/a | 1,600+ languages (ASR only) | n/a |
| SeamlessStreaming (arXiv 2312.05187) | 96 target languages in text output | 101 source languages | 36 target languages |
Qwen3-Omni (arXiv 2509.17765) and Qwen3.5-Omni (arXiv 2604.15804) each report speech generation in 10 languages. Inference Generation is likely the constraint, because it needs clean TTS-grade data per language.
Translation and code-switching
- Hibiki (arXiv 2502.03382): simultaneous speech-to-speech translation with multistream modelling. It uses per-word delays from text machine translation to build aligned synthetic data. The authors' model has 2.7B parameters and translates French to English (the pair they evaluate).
- SeamlessStreaming (arXiv 2312.05187): direct streaming translation from 101 source languages into speech in 36 target languages.
No dedicated study was found in the sources used here. Treat code-switching as an open question to test, not a solved capability.
How text streams serve S2S
| Role | Mechanism |
|---|---|
| Supervision | ASR transcripts and TTS text give cheap aligned targets for speech understanding and generation, so unpaired audio can still be used (Kimi-Audio, over 13M h, arXiv 2504.18425) |
| Scaffold | Text is generated first or alongside speech: Moshi's time-aligned Inner Monologue (arXiv 2410.00037), Thinker text before Talker (arXiv 2509.17765), Step-Audio 2 fixed-ratio interleaving (arXiv 2507.16632), ARIA alignment (arXiv 2604.15804) |
| Alignment | Per-word delays (Hibiki, arXiv 2502.03382) and delayed streams (DSM, arXiv 2509.08753) turn transcripts into time-aligned training data |
| Capability transfer | A frozen text LLM (Freeze-Omni, arXiv 2411.00774) or text-teacher distillation (arXiv 2509.14930) carries text-level multilingual knowledge into speech |
Inference For a multilingual product, understand broadly with a multilingual encoder, generate in a short list of languages you can evaluate with native raters, and use the text stream to hold language identity.
"Understanding scales through ASR-style data; generation does not. The text stream is both the supervision signal and the scaffold that lets generation borrow the LLM's multilingual ability."