Speech-to-speech modelling
Reference
Cheat sheet: key models
Every cell comes from the cited paper (arXiv). Years are the arXiv first-submission year; latency figures are the authors' own reports. A dash means I have no sourced value for that cell.
| Model | Year | Input rep | Output rep | Renderer | Streaming | Why it matters |
|---|---|---|---|---|---|---|
| Moshi (arXiv 2410.00037) | 2024 | Mimi codec tokens | Mimi codec tokens plus text | Mimi decoder | Yes; 160 ms theoretical, 200 ms in practice | First real-time full-duplex spoken LLM (authors' claim); Inner Monologue |
| dGSLM (arXiv 2203.16502) | 2022 | Textless, two-channel audio | Textless speech | — | — | First textless dialogue model (authors' claim); trained on 2000 h of Fisher |
| Voicebox (arXiv 2306.15687) | 2023 | Text plus audio context | Speech (infilled) | Flow matching, non-autoregressive | — | Flow-matching infilling; 50K hours of speech |
| F5-TTS (arXiv 2410.06885) | 2024 | Text padded with filler tokens | Speech | DiT flow matching | — | Fully non-autoregressive flow TTS; RTF 0.15 |
| CosyVoice 2 (arXiv 2412.10117) | 2024 | Text | FSQ speech tokens | Chunk-aware causal flow matching | Yes | LM plus flow; streaming and non-streaming in one model |
| CosyVoice 3 (arXiv 2505.17589) | 2025 | Text | Speech tokens | — | — | Differentiable reward model for post-training; 1M hours of data |
| GLM-4-Voice (arXiv 2412.02612) | 2024 | Discrete 12.5 Hz tokens | Discrete tokens | Flow matching plus HiFi-GAN | Yes | Interleaved text and speech; 175 bps tokenizer |
| Freeze-Omni (arXiv 2411.00774) | 2024 | Speech encoder | Text, then speech | Speech decoder | Yes | Frozen LLM keeps text intelligence |
| OmniFlatten (arXiv 2410.17799) | 2024 | Flattened tokens | Flattened tokens | — | Yes | Three-stage post-training on a flattened stream |
| Kimi-Audio (arXiv 2504.18425) | 2025 | Continuous features, 12.5 Hz tokens | Discrete tokens | Flow-matching detokenizer plus BigVGAN | Yes, chunk-wise | More than 13M hours of pretraining audio |
| Step-Audio 2 (arXiv 2507.16632) | 2025 | Audio encoder, 12.5 Hz output | Text and audio tokens | Flow matching plus HiFi-GAN | — | Calls tools: web search, audio search |
| LLaMA-Omni 2 (arXiv 2505.02625) | 2025 | Speech encoder | Text, then speech | Streaming speech decoder | Yes | Trained on only 200K dialogues |
| Qwen3-Omni (arXiv 2509.17765) | 2025 | AuT encoder | Multi-codebook codec frames | Causal ConvNet | Yes; 234 ms theoretical first packet | Thinker-Talker MoE; ConvNet replaces block-wise diffusion |
| Qwen3-TTS (arXiv 2601.15621) | 2026 | Text | 25 Hz or 12.5 Hz tokens | Block-wise DiT (25 Hz) or causal ConvNet (12.5 Hz) | Yes; 97 ms first packet (12.5 Hz) | Two tokenizers; Apache 2.0 |
| Llasa (arXiv 2502.04128) | 2025 | Text | Single-codebook VQ codec tokens | — | — | Single Transformer; inference-time search with verifiers |
| MELLE (arXiv 2407.08551) | 2024 | Text | Continuous mel frames | — | — | Autoregressive without vector quantization |
| VibeVoice (arXiv 2508.19205) | 2025 | Text | Continuous latents, 7.5 Hz tokenizer | Diffusion head | — | Up to 90 minutes and 4 speakers |
| IndexTTS2 (arXiv 2506.21619) | 2025 | Text plus timbre and style prompts | — | — | — | Duration control; emotion and timbre disentangled |
| Hibiki (arXiv 2502.03382) | 2025 | Source speech, multistream | Text and audio tokens | — | Yes, simultaneous | Simultaneous speech translation, French to English |
| MiniCPM-o 4.5 (arXiv 2604.27393) | 2026 | Omni-modal streams | Speech | — | Yes, full duplex | 9B parameters; edge devices with under 12 GB RAM |
| FlashTTS (arXiv 2606.09141) | 2026 | Text | Tokens, then mel | Mean-flow decoder, 2 NFE | Yes; 325 ms first packet | Lagged multi-track; no sentence-level buffering |
Five sentences to remember
Inference My synthesis of the table, not a sourced claim.
- Several systems here run audio at about 12.5 Hz (Moshi, GLM-4-Voice, Kimi-Audio, Step-Audio 2, Qwen3-Omni).
- Output is often an LM for content plus a flow or ConvNet renderer.
- Text is the scaffold and the supervision.
- Duplex looks limited by data and evaluation more than by architecture.
- Latency comes from token rate, NFE and chunking.