Materials Speech-to-speech: literature review

Speech-to-speech: literature review

Control and quality

Steering voice, tone and emotion

Five control channels, usually combined: a reference clip, a natural-language instruction, inline tags, duration targets, and style guidance at sampling time.

Text content Reference audio speaker, timbre NL instruction, tags style, emotion Duration target exact timing Token LM conditioned on all four Decoder tokens audio
Figure: each control channel enters the token language model as a separate condition; the decoder turns the resulting tokens into audio.

Control channels

Channel Examples Notes
In-context cloning Synthesia Express-Voice Synthesia describes a two-stage transformer with autoregressive and non-autoregressive parts (about 800M parameters each), operating on graphemes with Descript's RVQ tokenizer. It is conditioned on reference audio, with no fixed speaker embedding (Synthesia Express-2 post)
NL instructions CosyVoice 3, Parler-TTS, IndexTTS2, Hume Octave, gpt-4o-mini-tts, Qwen-Audio-3.0-TTS CosyVoice 3 prepends a natural-language description and an <endofprompt> token to the text; instruction-following data grew from 1,500 to 5,000 hours (arXiv 2505.17589). Parler-TTS reads gender, speaking rate, pitch and reverberation from the text description, with no reference audio; Parler-TTS Large is 2.3B parameters (Parler-TTS README). Hume Octave 1 takes a natural-language description and a separate speed multiplier from 0.5 to 2.0 (Hume docs). OpenAI's TTS guide shows an instructions field on gpt-4o-mini-tts. Qwen-Audio-3.0-TTS offers free-style instructions and inline tags (arXiv 2607.23938)
Inline tags ElevenLabs audio tags, CosyVoice 3, Qwen-Audio-3.0-TTS ElevenLabs lists bracket tags such as [whispers]; its docs say a tag for a delivery the voice was not trained on may be less reliable (ElevenLabs docs). CosyVoice 3 supports vocal bursts between text tokens and feature tags such as [laughter] and [breath] (arXiv 2505.17589)
Duration control IndexTTS2 (arXiv 2506.21619) Explicit token count for exact timing, or free mode that keeps prompt prosody
Style guidance CFG on the style or text condition Inference Reuses the CFG term from the flow-matching material in chapter 03 (Speech generation), with a style-only unconditional branch

Disentanglement

IndexTTS2 separates emotion from timbre: timbre comes from one prompt and emotional tone from a style prompt, so the two can come from different speakers (arXiv 2506.21619). Its text-based emotion control comes from fine-tuning Qwen3. The authors report better word error rate, speaker similarity and emotional fidelity than other zero-shot TTS models. Step-Audio-EditX takes a different route: large-margin synthetic data instead of representation-level disentanglement (arXiv 2511.03601).

In CosyVoice's own tests, the instruct model raised the share of generated speech that an emotion2vec classifier labelled sad, angry, surprised and disgusted, compared with the base model (arXiv 2407.05407, Table 10).

Duration control and lip-sync

The IndexTTS2 authors motivate exact durations by video dubbing, where speech must match a video block length (arXiv 2506.21619). Inference This is the hook for avatar sync.

Evaluation

emotion2vec (arXiv 2312.15185) is a self-supervised speech-emotion representation, usable as the basis for emotion similarity. Synthesia reports measuring Express-Voice with WavLM for speaker similarity and emotion2vec for emotional similarity.

Synthesia context

Synthesia's Express-2 (4 Sep 2025) pairs Express-Voice with Express-Video. The post reports a blind test with 100 native English-speaking evaluators across 17 accents, in which Express-Voice rated highest for matching the original speaker's identity, rhythm and accent (Reported company claim). The post gives no scores.

Gap

In-context cloning internals, speaker embeddings and timbre/prosody/content disentanglement were not researched beyond IndexTTS2 and Express-Voice.

In the interview, say:

"I would combine a reference clip for identity, an NL or tag channel for emotion, and a duration mode for sync, with emotion kept disentangled from timbre so style transfers across speakers."

Prompting speech LLMs to shape the output speech

A speech LLM takes several prompts at once, and each one controls a different property of the output. Persona text sets what the agent is, a reference clip sets who it sounds like, an instruction sets how it delivers a line, and the user's own audio nudges its prosody. Steering voice, tone and emotion covers the TTS-side channels; this section covers the speech-to-speech level.

System / persona text Voice reference audio Style instruction Inline tags in text Numeric parameters Conversation audio Speech LLM one context Speech out role, behaviour timbre, identity tone, pace, accent local delivery rate, pauses user's prosody (streamed) tokens
Figure: six prompt channels enter one speech LLM; the conversation audio is streamed and the rest are set by the application.

Prompt channels

Channel Example prompt Controls Consumed as Example system
System / persona text "You are Qwen-Omni, a smart voice assistant created by Alibaba Qwen." Role, background, scenario, behaviour Text tokens; PersonaPlex forces them on the agent text channel while the agent audio channel stays silent (arXiv 2602.06053) PersonaPlex, Qwen3-Omni (user_system_prompt, README), OpenAI Realtime instructions (docs)
Voice reference audio A short speech sample Timbre, speaker identity, speaking style PersonaPlex: audio on the agent audio channel, before the text prompt (arXiv 2602.06053). The model card calls it "a sequence of audio tokens" that sets vocal characteristics and speaking style (model card) PersonaPlex, Synthesia Express-Voice (conditioned on reference audio, no fixed speaker embedding; post), Qwen3-Omni (three preset voices via speaker, no clip)
Style instruction "Speak in a cheerful and positive tone." Tone, intonation, accent, whispering, speed of speech Natural-language text next to the content. CosyVoice 3 prepends it to the input text, followed by <||endofprompt||> (arXiv 2505.17589). Parler-TTS feeds the description as separate input_ids from the spoken text, which is prompt_input_ids (README) gpt-4o-mini-tts (docs), OpenAI Realtime (prompting guide), Gemini TTS style value (docs), CosyVoice 3, Parler-TTS
Inline tags / markup [laughter], <short pause>, <break time="250ms"/> Event or delivery at one point in the text Part of the spoken text. CosyVoice 3 supports vocal bursts between text tokens and tags such as [laughter], [breath] (arXiv 2505.17589) ElevenLabs [whispers] (docs), Gemini TTS angle-bracket tags, Google Cloud TTS SSML (prosody, break, emphasis)
Numeric parameters speed, 0.5 to 2.0 Rate and pauses, outside the model's wording API fields. OpenAI says its Realtime speed "changes playback rate, not how the model composes speech", so a faster-sounding delivery needs a prompt instruction (guide) Hume Octave: use speed instead of describing pace (docs), Google SSML rate
Conversation audio The user sounds upset or hurried Response style, re-entry prosody after an interruption Streamed input audio in the same context. Gemini Live's affective dialog lets the model "adapt its response style to the input expression and tone" (Live API docs) Gemini Live (enable_affective_dialog), duplex models

The Hume speed range of 0.5 to 2.0 is from the Octave docs cited in the steering section. Gemini TTS takes a free-text style value (examples: "warm and enthusiastic", "whispered urgently") in a speech_metadata annotation, and treats the text field as a verbatim transcript, so stage directions go in the style field (Gemini speech generation docs).

How the model consumes a prompt

  • Everything is context. PersonaPlex concatenates a voice segment and a text segment as a "hybrid system prompt" ahead of the dialogue, with delimiters marking where the system prompt ends. Loss is masked on the system prompt during training. The voice prompt comes first so it can be prefilled when no cloning is needed (arXiv 2602.06053).
  • Instruction in front, content behind. CosyVoice 3 prepends a description and an end token to the text to synthesise (arXiv 2505.17589). Its fine-tuning instructions name speaker and style (for example "You are Speaker A. Please talk to me happily."), and these fields are randomly masked.
  • Edit with an instruction. Step-Audio-EditX takes the original audio plus a descriptive command, and can apply several editing rounds to one clip (arXiv 2511.03601).
  • Inference Classifier-free guidance on the style or instruction condition would act as a strength knob: higher guidance exaggerates the style, lower guidance follows the voice's default. See the CFG material in the speech generation chapter; no source here measures it for prompts.

Where prompting sits in the unified design

In the unified speech LLM design, session setup and per-turn hints are separate concerns.

Inference (design, not sourced)
  • Fix persona text and voice at session start; OpenAI Realtime documents that the voice cannot change once the model has emitted audio.
  • Let the reasoning LLM return a short style field with each reply, such as style: "warm, slightly slower", and pass it to the speech LLM as a style instruction.
  • Keep the style field to a few words; Hume advises instructions of no more than 100 characters.
  • Let the speech LLM adapt prosody to the user's audio on its own; do not ask the reasoning LLM to guess it from a transcript.
  • Gate voice prompts on consent. OpenAI builds custom voices from a consent recording; Synthesia has the voice speaker read a randomly generated passcode (help article).

Pitfalls

Pitfall Evidence
Tags the voice was not trained on ElevenLabs: "Tags land more readily when the delivery is already in the voice's training data" (docs)
Stage directions read aloud ElevenLabs notes the model may speak delivery guides in the text. Qwen3-Omni's README asks for output of spoken content only, with no descriptions of actions or emotions
Prompt cannot replace the voice OpenAI: prompting "can steer accent, pacing, and delivery, but it cannot fully replace voice design"
Emotion carried by the reference clip Inference A clip carries timbre and delivery together. IndexTTS2 adds a gradient reversal layer and a speaker classifier to disentangle them (arXiv 2506.21619)
Instruction gaps Instruct-FD: the best of six full-duplex systems reaches only 64.4% adherence on turn-taking instructions, scored by an LLM-based judge (arXiv 2607.20460)
Brittle adaptation Full-Duplex-Bench v1.5 compares speech before and after an interruption. GPT-4o speeds up by 19.04 words per minute; Moshi by 59.55 with falling pitch, which the authors call hurried and brittle (arXiv 2507.23159)
Persona drift PersonaPlex trails Gemini on its GPT-4o-judged role-adherence benchmark; author-reported (arXiv 2602.06053)

To measure adherence, use emotion2vec for emotion similarity (arXiv 2312.15185) and audio-LLM judges for style. AudioJudge reports significant verbosity and positional biases that need mitigation (arXiv 2507.12705), so keep human listening in the loop.

In the interview, say:

"I treat prompting as separate channels: persona text and a voice clip fixed per session, a short style instruction per turn, inline tags for local effects, and the user's audio for adaptation. A reasoning LLM can emit a style hint with each reply, and the speech LLM applies it. I would measure adherence with emotion similarity and calibrated judges, because instruction following is still the weak spot."

Post-training and evaluation for speech generation

Post-training uses ASR and speaker-verification rewards. The recurring failure is reward hacking.

TTS policy Speech samples ASR: WER Speaker SIM Emotion, naturalness Policy update updated policy generates the next rollouts Human listening audit for reward hacking rollouts rewards
Figure: the RL loop for TTS. Automatic rewards score each rollout; human listening tests audit the rewards and gate release.

Methods

Method Idea
Seed-TTS RL (arXiv 2406.02430) RL for robustness, speaker similarity, controllability
DiffRO (CosyVoice 3; arXiv 2505.17589, 2507.05911) Computes rewards on neural codec tokens rather than synthesized audio; Gumbel-Softmax makes the reward differentiable; a multi-task reward model adds emotion and quality control
F5R-TTS (arXiv 2504.02407) Converts flow outputs to Gaussians so GRPO applies; WER and SIM rewards; 29.5% relative WER reduction, 4.6% relative SIM gain (authors)
Flow-GRPO (arXiv 2505.05470) ODE-to-SDE conversion that keeps the model's marginals, enabling the sampling that online RL needs for flow-matching models (text-to-image)
GROW (arXiv 2608.03215) Group-relative advantage-weighted on-policy RL that acts on the standard flow-matching objective, avoiding the ODE-to-SDE conversion and per-step likelihood ratios; on DiTAR the authors report WER 2.016 to 1.558 and SIM 0.676 to 0.715
SwanVoice (arXiv 2605.30993) DiffusionNFT post-training with phone-level and speaker-similarity rewards
FlowTTS-GRPO (arXiv 2606.23190) Online RL for flow-matching TTS: ODE-to-SDE conversion, weighted multi-objective reward; tested on CosyVoice 3.0 and F5-TTS
Step-Audio-EditX (arXiv 2511.03601) Large-margin synthetic data for iterative emotion and style editing, with no embedding priors or auxiliary modules
Preference optimisation EASPO: stepwise preference optimisation for diffusion TTS, scoring noisy intermediate states (arXiv 2509.25416)

Reward hacking

Trap: automatic rewards get gamed
  • RRPO reports that DiffRO-style reward models can be exploited with acoustic artifacts that earn spurious reward but degrade perceptual quality (arXiv 2512.04552).
  • A prosody study found GRPO on CER and NLL rewards lowered error rates but collapsed prosody into monotone speech; adding speaker similarity destabilised training. Its fix was iterative DPO with a few hundred human preference pairs per round (arXiv 2509.18531).
  • FlowSE-GRPO (speech enhancement): a single reward gives fast metric gains but degrades audio fidelity; multi-metric rewards mitigate this (arXiv 2601.16483).
Inference

Inference Always pair automatic rewards with a naturalness term and listening tests.

Metrics

Metric Notes
WER Seed-TTS-eval uses Whisper-large-v3 (English) and Paraformer-zh (Mandarin)
SIM Seed-TTS-eval uses WavLM-large fine-tuned for speaker verification. Inference Other benchmarks may use other verification models, so quote the variant
UTMOS Predicted MOS used as a naturalness check alongside WER and SIM (for example in GROW, arXiv 2608.03215)
emotion2vec Emotion similarity (arXiv 2312.15185)
Full-Duplex-Bench (arXiv 2503.04721) Pause handling, backchanneling, turn-taking and interruption management for spoken dialogue models
Human listening Needed because the automatic metrics are optimisable

The GRPO experience from online RL at Amazon transfers directly. The new part is making rollouts differentiable or likelihood-tractable for flow models.

In the interview, say:

"I would use WER and SIM rewards plus an emotion or naturalness term, watch for monotone prosody as a hacking signature, and gate releases on human listening tests."