Common problems

Speech generation problems

Thirteen failure modes recur in TTS and voice cloning. Most come from three places: weak text-to-audio alignment, conditioning on a flawed reference, and stochastic sampling. The table is the quick map; the subsections below cover the ones worth explaining aloud.

Grounding labels follow the rest of the page. Synthesia facts cite its Express-2 post and its docs; research claims cite arXiv. Mechanisms marked Inference are my reading, not a source. The Metric and Mitigation columns of the table are suggestions unless a subsection cites them.

A1 Hallucination and alignment

LLM-based TTS models can repeat words, drop words or produce misaligned speech Reported. The authors of arXiv 2406.17957 find that certain cross-attention heads implicitly learn the text-to-speech alignment, and that nothing in the loss forces that alignment to stay monotonic. Regotron adds a monotonic alignment loss to Tacotron 2 for the same class of problem (arXiv 2204.13437).

GOAT reports a strong positive correlation between hallucination and model uncertainty, and over 50% lower character error rate on challenging test cases (arXiv 2508.15442) Reported.

Flow-matching TTS is non-autoregressive, so it has no free-running attention to stall (see diffusion and flow). The cost is that the total duration must be chosen before generation, and that choice can be wrong Inference.

A3 Mispronunciation

Names, acronyms, numbers and heteronyms fail when grapheme-to-phoneme rules or text normalisation guess wrong Inference. Express-Voice operates on graphemes directly, so any fix has to live in the model or in the script Confirmed (Express-2 post).

Synthesia's documented workaround is a per-word pronunciation entry Confirmed. The script docs give a respelling example, "sin-THEE-zhuh" for Synthesia, and offer two previews: an IPA version and a respelling that replaces the text before it reaches the voice model. Constraints from the docs and the help article:

• A saved pronunciation does not automatically apply to instances of the word added to the script later.
• "Apply to all" covers every instance in the current video.
• "Add to glossary" applies across the workspace and is available on Enterprise plans only.
• Pronunciation reliability varies by voice and language; regenerating the scene can help.

The pattern is a manual patch around a grapheme model. A production fix would add a per-customer dictionary applied before synthesis and a fixed name list for regression tests Inference.

A2 Long-form drift

Over minutes, autoregressive generation can lose speaker identity or flatten prosody because errors compound and nothing re-anchors the voice Inference. Measure speaker similarity in sliding windows against the first window, and plot it against time. Mitigate by synthesising in chunks and re-conditioning each chunk on the original reference.

A6 and A8 Reference leakage

Express-Voice clones in context: the reference audio conditions the model with no fine-tuning Confirmed (Express-2 post). The same mechanism that copies timbre can copy emotion, background noise and microphone channel Inference.

• Detect leakage with emotion2vec similarity between prompt and output, and a noise-floor check.
• Gate prompts at upload: reject clipped, noisy or reverberant audio, or denoise first.
• Prefer neutral, clean references when the target style is set by the script.

A7 Metric pitfalls and A11 codecs

Synthesia names WavLM for speaker similarity and emotion2vec for emotional similarity in its voice evaluation, and publishes no scores Confirmed (Express-2 post). A similarity score depends on which speaker encoder computes it, so scores from different encoders do not compare Inference.

Express-Voice uses Descript's residual vector-quantised tokenizer Confirmed. A metallic or buzzy timbre that survives every sampling setting points at the tokenizer, not the LM Inference. See codecs and tokenizers for the rate trade-offs.

A9 Sampling instability and reranking

Higher temperature plausibly raises the hallucination rate of A1, so a few samples fail even when the average looks good Inference. The cheap fix is best-of-N: sample several candidates, score each, return the best. Score with ASR error plus a speaker-similarity term.

Synthesia applies the same idea to video: Express-Eval scores audio-motion alignment and picks the best motion candidate Confirmed (see audio-visual problems). Pushing the reranking signal into training is online RL, which invites reward hacking. One report finds that GRPO on transcription signals (CER and NLL) lowers error rates but collapses prosody into monotone speech (arXiv 2509.18531) Reported. See online RL rewards for composite rewards and KL control.

A4, A5, A10, A12, A13 in brief

• Accents. Express-Voice training data was curated for accent coverage, including YODAS and LibriLight. Synthesia reports a blind test with 100 native English-speaking evaluators across 17 accents and top ratings, but names no comparison systems and gives no scores Reported (Express-2 post). Report per accent and per language pair Inference.
• Prosody. Express-2 offers voice styles for the same voice (for example Excited and Neutral) Confirmed. The script docs add a pause tool with durations from 0.1 s to 99 s (default 1 s) (script docs). That is coarse control, and a flat read needs a script edit Inference.
• Chunk seams. Streaming trades first-packet time against boundary quality Inference. See streaming and low latency.
• Languages. Synthesia's own pages disagree on language counts. The Synthesia 3.0 post says dubbing covers 30+ languages, while the title of its AI dubbing page says 130+. Quote the number only with its page.
• Misuse. Synthesia states that it strictly prohibits cloning voices without explicit consent Confirmed (voice cloning page).

Speech-to-speech and duplex problems

End-to-end and full-duplex models add failures that cascades do not have. The model must listen while it speaks, decide when to talk, and keep its text intelligence while operating on audio tokens. Synthesia's Video Agents "talk, listen, and act in real time" inside a video Confirmed (Synthesia 3.0 post), so these failures reach its interactive product, not only research systems.

Intelligence drop and forgetting

Adding speech capability can degrade an LLM's knowledge and reasoning even when inputs stay textual, and performance drops further on spoken queries (arXiv 2509.14930) Reported. The SALAD authors name the resulting shortfall the "text-speech understanding gap" (arXiv 2510.13632). Two recipes exist: freeze the text LLM (Freeze-Omni, arXiv 2411.00774), or keep a high share of text in the training mix (Moshi trains half of the time on text-only batches, arXiv 2410.00037; Kimi-Audio pretrains on text-only, audio-only and interleaved data, arXiv 2504.18425).

Interleaved synthetic speech-text data lifted spoken question answering from 13% (Moshi) to 31% in one report (arXiv 2411.17607) Reported. Gate every stage on a strict text QA score so forgetting shows up early Inference. See streaming conversation.

Multi-turn degradation

Full-Duplex-Bench v2 tests daily, correction, entity-tracking and safety task families. Its authors report that the tested systems often get confused when people talk at the same time, struggle with corrections and sometimes lose track of who or what is being discussed (arXiv 2510.07838) Reported. FLEXI reports significant gaps between open-source and commercial models in emergency awareness, turn terminating and interaction latency (arXiv 2509.22243) Reported.

Backchannel misread as barge-in

A 2026 survey calls this the single most-cited failure mode in user feedback: the system mistakes the user's acknowledgement ("uh-huh") for a barge-in and stops (arXiv 2606.19453) Reported. Talking Turns finds that audio foundation models can interrupt too aggressively and rarely backchannel themselves (arXiv 2503.01174). The open problem is the trade-off between responding early and yielding falsely Inference.

DuplexSLA reports backchannel accuracy of 98.33%, against 40.00% for gemini-3.1-flash-live and 0.33% for gpt-realtime-1.5 with semantic VAD (arXiv 2605.20755, Table 6) Reported. These numbers are self-reported by the DuplexSLA authors on their own scenarios. Quote them as a claim, not a ranking.

Self-echo, latency spikes and quantisation

• Self-echo. A model that ingests user audio while speaking can hear its own voice. Mitigate with acoustic echo cancellation and by conditioning on the model's own output stream Inference.
• Chunk quantisation. SyncLLM alternates 160 ms chunks of user and assistant tokens (arXiv 2409.15594), and the survey calls it an example of "apparent (not substantive)" full duplex, since transitions occur only at chunk boundaries (arXiv 2606.19453) Reported. Turn-taking feel plausibly suffers when decisions are quantised at 80 to 240 ms, trained on heuristic labels, or starved of backchannels in the data Inference.
• Latency spikes. Reported latencies are measured under different conditions and do not compare Inference. Track p95 and p99, because a conversation is judged by its worst pauses. See streaming and low latency.

Interruption recovery and truncation

On barge-in the agent must stop, and its context must record only the words actually spoken. Otherwise it believes it said a sentence the user never heard. Measure stop latency from the gold onset to the last audible agent sample on echo-cancelled output, plus the fraction of agent speech after onset Inference.

Long-session context and synthetic data

Token-level duplex models grow the KV cache continuously. A back-of-envelope for Moshi's temporal transformer uses its published shape of 32 layers and 4096 dimensions (arXiv 2410.00037) and assumes full multi-head attention in bf16: $2 \times 32 \times 4096 \times 2$ bytes is about 512 KiB per frame, or roughly 23 GB per hour at 12.5 frames per second Inference. Grouped-query attention shrinks it.

Duplex training data is often synthetic: an LLM writes dialogue and TTS renders both sides. SyncLLM trains on 212k hours of synthetic dialogue and 2k hours of real speech (arXiv 2409.15594), and PersonaPlex on synthetic conversations from open LLMs and TTS (arXiv 2602.06053) Reported. Risks are TTS voice and prosody leakage, over-regular inserted backchannels, and circularity, since inserting backchannels needs dialogue that already contains them Inference. DuplexPO applies RL over selected windows of human conversations to learn when to speak separately from what to say (arXiv 2607.07148) Reported. See online RL rewards for per-axis timing rewards and the hacking risks.

Audio-visual problems, grounded in Synthesia

Synthesia's Express-2 stack (September 2025) has four parts: Express-Voice for speech, Express-Animate for motion, Express-Eval for motion scoring, and Express-Render for frames Confirmed (Express-2 post, 4 September 2025). Each stage can introduce its own failures, and later stages cannot always repair them. The figure shows where each problem (B1 to B11 and A-codes from the previous section) enters; the placement is my mapping Inference.

B1 Lip-sync, and why Sync-C misleads

Hard cases plausibly include bilabial closures (p, b, m), fast speech and non-English audio Inference. The standard metrics are Sync-C and Sync-D, also written LSE-C and LSE-D, which are based on SyncNet (arXiv 2403.06421). Evidence that they mislead:

• THEval's authors say existing metrics show limited correlation with human evaluation and unreasonable values, such as ground truth being widely outperformed (arXiv 2511.04520).
• In a psychophysical comparison of metrics on talking-head videos, only LSE-O showed moderate effectiveness (arXiv 2403.06421).

Both are Reported results from papers. One proposed replacement uses an audio-visual speech representation expert (AV-HuBERT) for a lip-sync loss and three new evaluation metrics (arXiv 2405.04327). Sync-C values from different papers use different protocols, so I treat cross-paper tables as unreliable Inference. See metrics and data.

B2, B3 Identity drift, flicker and colour drift

Express-Render is described as keeping identity consistent over long sequences, with no metric attached Confirmed (Express-2 post). A related evaluation framework studies lip leakage, where the identity reference image influences the lips instead of the driving audio (arXiv 2511.08613) Reported. Open avatar work offers fixes for long sequences: sparse-frame conditioning to preserve identity (InfiniteTalk, arXiv 2508.14033) and a multi-step retrospective self-correction mechanism against accumulated error (SoulX-FlashTalk, arXiv 2512.23379) Reported. Flicker around teeth and mouth interior is a plausible weak spot Inference. The mechanisms are explained in streaming and real-time.

B4, B5, B6 Gesture and expressiveness

Express-2 avatars combine facial expressions and lip sync with natural hand and body gestures, and the post publishes no failure rates Confirmed. Express-Animate generates gestures from audio alone, which decouples motion from appearance Confirmed. Audio-only conditioning plausibly yields rhythm without intent Inference. OmniHuman-1.5 makes the same argument: earlier avatar models synchronise with low-level cues such as audio rhythm and lack semantic understanding of emotion and intent, so it adds MLLM-written semantic guidance (arXiv 2508.19209) Reported.

Express-2 also exposes performance intensity and temporal diversity controls Confirmed. To measure voice-face mismatch, compare emotion2vec on the audio with a facial-expression classifier on the video Inference.

B7 Dubbing

Synthesia's dubbing translates uploaded video into 30+ languages with frame-accurate lip-sync Confirmed (Synthesia 3.0 post). The post gives no lip-sync or voice-retention metrics.

The hard constraint is duration: translated speech is often longer than the source line, so the system must shorten the translation, speed the speech, or retime the video Inference. Measure the fraction of segments outside a timing budget, then lip-sync quality after retiming.

B8, B9 Streaming, resolution and compression

The Express-2 post gives no latency numbers. It says generation times should stay similar after the two-step distillation, and describes Express-Render-Turbo as coming later Confirmed. The quality versus latency trade-off is unpublished Inference.

Causal streaming students show their own failures. LiveTalk reports that Self Forcing, the leading on-policy distillation approach, shows flickering, black frames and quality degradation under multimodal conditioning, and proposes an improved distillation recipe (arXiv 2512.23576) Reported. Error accumulation plausibly grows with rollout length, and audio and video clocks can drift apart Inference. Output is 1080p at 30 fps with no stated duration limit Confirmed. Because encoder settings can change metric values, pin the encode settings before scoring Inference.

B11 Safety

Synthesia requires a live consent video that must show the same person as the photo, and does not accept uploaded consent videos Confirmed (personal avatars docs). It states that it strictly prohibits cloning voices without explicit consent (voice cloning page) and moderates content at the point of creation Confirmed. Its AI governance page lists a Content Integrity Policy, EU AI Act Article 50 labelling and Content Authenticity Initiative engagement. The page does not mention C2PA or watermarking.

Express-2 component to problem map

Priorities, metrics and questions

Not every failure deserves equal effort. This ranking weighs how visible the failure is to a customer, how often it occurs in a voice and avatar product, and how cheaply it can be measured. The ranking, mitigations and metrics are my own judgement Inference, built on the sourced problems in the previous sections.

Why this order

• Items 1 and 2 are audible on first listen and cheap to test automatically, so they belong in every regression run.
• Items 3 to 7 are the avatar-specific problems; they need human rating because the automatic metrics are weak.
• Items 8 to 10 depend on product shape (dubbing, streaming, Video Agents) and rank higher if that product is the focus.
• Any metric that is optimised in training needs a held-out judge, or reward hacking will hide regressions (see online RL rewards).

Questions to ask Synthesia about failure modes

1. Which failure modes do you track in production, and which of them have an automatic metric?
2. How do you evaluate lip-sync, given the known weaknesses of Sync-C, and does it differ for non-English dubbing?
3. What does Express-Eval catch in practice: gesture plausibility, lip-sync, or both, and how often does it change the pick?
4. For Express-Voice, how do you protect against emotion or noise leakage from customer reference audio?
5. For Video Agents, what is the p95 first-frame latency, and how do you handle barge-in and backchannels?