Speech generation

Codecs and speech tokenizers

The tokenizer fixes the sequence length the LM sees and the ceiling on audio quality. Four families matter.

Qwen3-TTS ships two tokenizers: 25 Hz single-codebook and 12 Hz multi-codebook (arXiv 2601.15621).

What "first codebook distilled from WavLM" means

Mimi trains its first quantizer to carry phonetic content, not just sound (arXiv 2410.00037). The teacher is WavLM, a self-supervised speech model producing 1024-dim embeddings at 50 Hz. During training, the first quantizer's output passes through a linear projection to 1024 dims, and a cosine-distance loss pulls it toward WavLM's embedding for the same moment. WavLM's features are average-pooled (stride 4, kernel 8) from 50 Hz to Mimi's 12.5 Hz.

Mimi is fully causal, while WavLM sees the whole utterance. The teacher is used only in training, so its look-ahead never reaches inference. Distilling into level 1 of a normal RVQ made semantic and acoustic goals conflict, because later levels encode level 1's residual. Mimi therefore runs one plain VQ for semantics in parallel with a 7-level RVQ for acoustics and sums them; the authors report distillation improves phonetic discriminability (ABX) and the split gives a better semantic-acoustic trade-off.

Trade-offs

Evidence on low frame rates

A controlled frame-rate ablation reproduces a quality cliff at 6.25 Hz. Its authors trace the cliff to the training configuration (fixed clip duration gives too few tokens at low rates), not to a fundamental barrier. After the fix, WER degrades smoothly with phonemic load down to 3.1 Hz and 1.6 Hz (arXiv 2606.16969). Another paper pools X-Codec-2.0 from 50 Hz to a 25 Hz latent rate and reports a 0.29 UTMOSv2-based MOS gain (arXiv 2601.20185).

How codecs are trained: VQ-VAE, RVQ and continuous latents

An RVQ codec is a VQ-VAE-family autoencoder: encoder, discrete bottleneck, decoder, trained end to end. It is not a VAE. The VQ-VAE paper does view its model as a VAE with a deterministic posterior and a uniform prior, which makes the KL term a constant ($\log K$) that is dropped from training (arXiv 1711.00937). So no KL loss and no sampling appear in an RVQ codec. Continuous-latent tokenizers are the VAE counterpart: they keep a Gaussian bottleneck and a KL term instead of a codebook. What these tokenizers produce is described in Codecs and speech tokenizers.

VQ-VAE: the base recipe

The encoder output $z_e(x)$ is replaced by its nearest codebook vector $e_k$, with $k = \arg\min_j \lVert z_e(x) - e_j \rVert_2$, and the decoder reads $z_q(x) = e_k$ (arXiv 1711.00937). The argmin has no gradient, so the decoder's gradient at $z_q$ is copied unchanged to $z_e$ (a straight-through estimator). Reconstruction therefore never updates the codebook, so two extra terms are added. The codebook loss pulls code vectors toward encoder outputs, and the commitment loss keeps encoder outputs from drifting away from their codes.

Source: VQ-VAE, Eq. 3 (arXiv 1711.00937). Here $\operatorname{sg}$ is the stop-gradient operator, identity in the forward pass and zero partial derivatives in the backward pass. The decoder trains on the first term only, the encoder on the first and third, and the codebook on the second. The paper uses $\beta = 0.25$ and reports little change for $\beta$ from 0.1 to 2.0. Its appendix describes an EMA update for the codebook instead of the second term (decay $\gamma = 0.99$), but states the main experiments did not use it (arXiv 1711.00937). Inference The first term is a log-likelihood to maximise; in a codec it becomes a reconstruction error to minimise.

RVQ: SoundStream, EnCodec, DAC

Residual vector quantization chains $N_q$ quantizers, each coding what the previous ones missed (SoundStream, Algorithm 1; arXiv 2107.03312).

SoundStream (arXiv 2107.03312) trains this end to end with the following recipe.

• Quantizer dropout: for each example, sample $n_q$ uniformly in $[1, N_q]$ and use only the first $n_q$ quantizers. One model then serves many bitrates.
• Codebooks use EMA updates (following VQ-VAE-2) with k-means initialisation on the first batch. A code with no assigned frames for several batches is replaced by a random frame from the batch. The EMA decay is 0.99, and codes whose EMA falls below 2 are replaced.
• Generator loss: adversarial, feature matching and multi-scale spectral reconstruction, with $\lambda_{adv} = 1$, $\lambda_{feat} = 100$, $\lambda_{rec} = 1$. Its generator loss (Eq. 6) lists no commitment term.
• Streamable by design: all convolutions are causal, and the encoder strides (2, 4, 5, 8) give one embedding per 320 samples. With $N_q = 8$, each codebook has 1024 entries.

EnCodec keeps the recipe and changes the training signal (arXiv 2210.13438).

• It adds a commitment loss $\sum_c \lVert z_c - q_c(z_c) \rVert_2^2$ over the residual steps, with gradient only into the quantizer input.
• It uses a multi-scale STFT discriminator with five window lengths (2048 to 128).
• A loss balancer rescales each loss's gradient with respect to the output $\hat x$. With weights $\lambda_i$, reference norm $R$ and the EMA $\langle \lVert g_i \rVert_2 \rangle_\beta$ of each gradient norm, it backpropagates the following. The commitment loss is excluded.

Source: EnCodec, Eq. 5 (arXiv 2210.13438), with $R = 1$ and $\beta = 0.999$. At 24 kHz EnCodec produces 75 latent steps per second. It uses up to 32 codebooks of 1024 entries, and trains for 1.5, 3, 6, 12 and 24 kbps by picking a multiple of 4 codebooks per batch (arXiv 2210.13438).

DAC (arXiv 2306.06546) targets codebook collapse, where many codes go unused. It does code lookup in a low-dimensional space (8 or 32 dimensions) while the code embedding stays 1024-dimensional. It L2-normalises encoder outputs and codes, so distance becomes cosine similarity. With this it trains on plain VQ-VAE codebook and commitment losses (weights 1.0 and 0.25), without k-means initialisation or random restarts. The authors found that applying quantizer dropout to every example hurts full-bandwidth quality, and apply it with probability $p = 0.5$ instead. Other weights are 15.0 for the multi-scale mel loss, 2.0 for feature matching and 1.0 for adversarial, with no loss balancer. Reported setup: 44.1 kHz audio at 8 kbps, 86 Hz frame rate, 9 codebooks (arXiv 2306.06546).

FSQ: a codebook-free alternative

Finite scalar quantization bounds each of $d$ channels and rounds it to one of $L$ integer levels (arXiv 2309.15505).

Source: FSQ, Sec. 3 (arXiv 2309.15505). The implicit codebook is the product of the per-channel level sets. Gradients pass through the rounding with a straight-through estimator, implemented as $x + \operatorname{sg}(\operatorname{round}(x) - x)$. The reconstruction loss alone pushes information across bins, so there is no codebook loss, no commitment loss and no EMA. The authors report codebook usage near 100% for most models, without auxiliary losses (arXiv 2309.15505). CosyVoice 2 swaps VQ for FSQ in its speech tokenizer, which works at 25 Hz (arXiv 2412.10117). That tokenizer is an ASR encoder with FSQ inserted (SenseVoice-Large), trained to predict text, so it is supervised rather than a reconstruction autoencoder. The authors report that the FSQ tokenizer fully uses its codebook (arXiv 2412.10117).

Mimi: adversarial-only training

Mimi uses 8 quantizers with 2048 entries at 12.5 Hz, 1.1 kbps, with quantizer dropout for bitrate scalability (arXiv 2410.00037). It applies quantization to only 50% of training sequences and passes unquantized embeddings to the decoder otherwise. It also drops the reconstruction loss and keeps only the feature-matching and discriminator losses. The authors report that this lowers objective scores (VisQOL 1.84) but sounds better: MUSHRA 81.0 against 58.8 for the EnCodec-style mix of reconstruction and adversarial losses (arXiv 2410.00037). The semantic distillation loss and the split RVQ are covered in What "first codebook distilled from WavLM" means.

Continuous latents: the VAE side

A VAE learns an encoder $q_\phi(z \mid x)$ and a decoder $p_\theta(x \mid z)$ by maximising a variational lower bound (arXiv 1312.6114).

Source: Kingma and Welling, Eq. 3 (arXiv 1312.6114). The reparameterisation trick draws $z = \mu + \sigma \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$, which makes the sample differentiable. The KL term regularises the posterior toward the prior. Speech and audio tokenizers keep this shape and add the codec's adversarial and spectral losses.

Prior, posterior and the reparameterisation trick

• Prior: in the paper's Gaussian example, the prior over latents is a centred isotropic Gaussian, $p(z) = \mathcal{N}(0, I)$ (arXiv 1312.6114).
• Posterior: the encoder outputs a mean and standard deviation per latent dimension, so $q_\phi(z \mid x) = \mathcal{N}(\mu, \sigma^2 I)$ (arXiv 1312.6114).
• With both Gaussian, the KL term has a closed form and needs no sampling (Eq. 10, arXiv 1312.6114):

Sampling is made differentiable by moving the randomness into a parameter-free noise variable (Section 2.4, arXiv 1312.6114):

Now $z$ is a deterministic function of $\mu$ and $\sigma$, with $\partial z / \partial \mu = 1$ and $\partial z / \partial \sigma = \epsilon$ (derived), so reconstruction gradients flow back into the encoder. The KL pulls $\mu$ toward 0 and $\sigma$ toward 1, keeping the latent space smooth and bounded.

Inference At generation time nothing samples from $\mathcal{N}(0, I)$: the LM's diffusion or flow head generates the next latent from context, and the decoder turns it into audio. The learned generator replaces the simple prior; the prior only shapes the latent space during tokenizer training.

Inference The small KL weight in Stable Audio Open suggests the KL acts as a light regulariser, not a tight bottleneck. The paper does not say this.

Why go continuous: MELLE argues that vector quantization, designed for compression, sacrifices fidelity compared with continuous representations (arXiv 2407.08551). MELLE itself regresses mel frames with a variational latent sampling module, and leaves VAE latents for future work (arXiv 2407.08551). The cost is a generation head that handles continuous targets, as described in Continuous autoregression, and harder RL, covered in Online RL rewards.

Comparison

How an LM then models the codebooks is in Training an LM on RVQ tokens.

Training an LM on RVQ tokens: flattening, parallel, delay, depth

Once the codec exists, the LM must model $K$ codebooks per frame. MusicGen defines a codebook pattern as a partition of all (time, codebook) pairs into steps, and the model predicts every position in a step in parallel (arXiv 2306.05284). Its two reference patterns, with frame $t$ and codebook $k$, are below.

Source: MusicGen, Eqs. 5 and 6 (arXiv 2306.05284). In the delay pattern, codebook $k$ is shifted by $k - 1$ steps. Step $s$ therefore predicts codebook 1 of frame $s$, codebook 2 of frame $s - 1$, and so on. Each token sees the lower codebooks of its own frame as context, and the sequence stays near $T$ steps long. Inference From Eq. 6, the delay pattern needs $T + K - 1$ steps for $T$ frames. MusicGen's 30 s ablation reports 1500 steps for delay and parallel. Positions absent from a step get a special token, and the first input is the sum of special tokens (arXiv 2306.05284).

Inference The latency column is my reading of the step counts, except the NAR and SoundStorm figures, which the papers state. FAD values are MusicGen's 30 s ablation (lower is better).

• MusicGen reports that flattening improves generation but costs far more, and that delay reaches similar performance at a fraction of the cost. Its ablation table lists 1500 steps for delay, 6000 for flattening, and FAD of 0.96 and 0.86 respectively (arXiv 2306.05284).
• VALL-E uses an autoregressive model for the first of 8 quantizers and a non-autoregressive model for the other seven, called once per quantizer. The authors note that early quantizers recover acoustic properties such as speaker identity and later ones learn fine acoustic detail (arXiv 2301.02111).
• Moshi's RQ-Transformer runs a Temporal Transformer over $S$ steps and a smaller Depth Transformer over at most $K$ sub-tokens per step, instead of one model over $K S$ steps. With 8 codebooks at 12.5 Hz, a flat sequence would be 100 steps per second of audio (arXiv 2410.00037).
• Moshi also delays the acoustic tokens against the semantic token by $\tau$ steps: $V_{s,1} = A_{s,1}$ and $V_{s,q} = A_{s-\tau,q}$ for $q > 1$. The authors found a delay of 1 or 2 steps greatly improves quality, since it lets the larger Temporal Transformer model the semantic-acoustic dependence (arXiv 2410.00037). For how Moshi's streams are timed, see Timing logic.
• Qwen3-Omni's Talker predicts one codec frame per step: a linear head for the zeroth codebook, then an MTP module for all residual codebooks (arXiv 2509.17765).
• SoundStorm sums the embeddings of all tokens in a frame, so the sequence length equals the number of frames, whatever the number of quantizers. It uses a bidirectional Conformer with one head per RVQ level, and fills in levels coarse to fine (arXiv 2305.09636).

Generation paradigms

There are four ways to generate speech from text or from an LM state. The dominant high-quality design is the hybrid.

AR codec LMs

• Llasa: a single Llama-style Transformer over X-Codec2 tokens, 1B/3B/8B, with inference-time search using speech-understanding verifiers (arXiv 2502.04128).
• Spark-TTS: Qwen2.5 LLM plus BiCodec (arXiv 2503.01710).
• CosyVoice 2: an LLM predicts speech tokens, then a chunk-aware causal flow-matching model decodes them (arXiv 2412.10117).
• Newer entries: IndexTTS2 (arXiv 2506.21619), Step-Audio-EditX (LLM-based audio editing with reinforcement learning; arXiv 2511.03601), Higgs Audio v2 (pretrained on over 10 million hours of audio, per the Boson AI repo).

Multi-codebook handling

An RVQ frame has several codes, so the model needs a trick.

• Sesame CSM: a Llama backbone models the zeroth Mimi codebook from interleaved text and audio, and a smaller audio decoder with a linear head per codebook models the rest; Mimi runs at 12.5 Hz (Sesame blog).
• Moshi: $K = 2Q + 1 = 17$ streams per step with $Q = 8$ codebooks (8 user audio, 8 own audio, 1 text). Its RQ-Transformer splits temporal and per-codebook (depth) modelling (arXiv 2410.00037).
• Qwen3-Omni's Talker uses an MTP module to emit codec frames (arXiv 2509.17765).

NAR notes

SoundStorm uses bidirectional attention and confidence-based parallel decoding to generate codec tokens, and the authors report 30 s of audio in 0.5 s on a TPU-v4 (arXiv 2305.09636). MaskGCT is two-stage (text to semantic tokens, then semantic to acoustic tokens), mask-and-predict in both, with no explicit alignment or phone-level duration model, trained on 100K h (arXiv 2409.00750).

The "LM for content, flow for acoustics" split

• Inference Why the LM: text streaming, interleaving with text, reuse of LLM serving stacks, in-context learning.
• Inference Why flow: it can restore acoustic detail lost to quantization, and CosyVoice 2 shows it can decode chunk by chunk (arXiv 2412.10117).

From tokens to waveform: decoder paths

Mel is one route to audio, not the only one. Four routes are in use, and they differ in what the LM predicts and what turns it into samples. Codecs and flow decoders are covered in the neighbouring sections; this one compares the endings.

Details on the LM side are in Codecs and speech tokenizers, Continuous autoregression and Diffusion and flow matching.

Codec decoder or vocoder

A codec decoder is trained together with its encoder and quantizer to rebuild the waveform from the codes. DAC is an example: a neural codec that turns audio into discrete tokens, with about 90x compression of 44.1 kHz audio at 8 kbps (arXiv 2306.06546). A vocoder is a separate model that maps an acoustic representation, usually a mel spectrogram, to a waveform. HiFi-GAN is a GAN vocoder that generates raw waveforms (arXiv 2010.05646). BigVGAN is a GAN vocoder built to generalise to unseen speakers and recording conditions without fine-tuning (arXiv 2206.04658).

Is the STFT reversible?

Inference Mel features suit models but drop phase and fine frequency detail, so classical inversion is slow and artefact-prone; learned vocoders, or codec decoders that skip spectrograms entirely, replaced it.

What phase adds to the signal

Magnitude says how much energy sits at each frequency in a frame. Phase says where each component is in its cycle, so it fixes how the components line up in time.

• Transients and waveform shape: a plosive or onset is sharp because many components peak at the same instant; keeping magnitude and scrambling phase smears that energy across the frame. Inference
• Timing inside a frame: the group delay, $\tau(\omega) = -\,d\phi(\omega)/d\omega$, locates energy in time more finely than the frame length (standard definition).
• Fine pitch: the instantaneous frequency, the rate of phase change over time, places a harmonic more precisely than the FFT bin spacing (standard definition).
• Frame consistency: overlapping STFT frames must agree for overlap-add to rebuild the signal; inconsistent phase is what phase retrieval has to repair (see the table above). Inference

The research record on how much phase matters:

• Oppenheim and Lim argue that magnitude and phase play different roles, that many important features of a signal survive when only the phase is kept, and that a finite-length signal can be reconstructed from phase alone to within a scale factor; their examples include images and speech (Oppenheim and Lim, "The Importance of Phase in Signals", Proc. IEEE 69(5), 1981).
• Paliwal and Alsteris challenge the traditional view that magnitude dominates for short (speech-length) windows: in their listening tests the short-time phase spectrum can contribute to intelligibility comparably to the magnitude spectrum when the window function and analysis-modification-synthesis parameters are chosen well (Paliwal and Alsteris, Eurospeech 2003; Speech Communication 2005).
• The same line of work reports a mismatch for machines: with an MFCC front-end, recognition of phase-only stimuli at short windows was much worse than of magnitude-only stimuli (Alsteris and Paliwal, Interspeech 2004 and 2005).

Inference Models avoid predicting phase directly because it wraps every $2\pi$ and looks noisy frame to frame, so most systems model magnitude or mel and let a vocoder, Griffin–Lim or a codec decoder supply consistent phase; Vocos is the exception that generates the spectral coefficients itself (arXiv 2306.00814).

Why route 2 exists, and why route 1 streams

Inference Semantic tokens are cheap for an LM to predict but drop voice detail. A flow model restores it, conditioned on a speaker prompt (CosyVoice 2 uses reference speech and a speaker embedding, arXiv 2412.10117). The cost is extra stages and look-ahead, as in Kimi-Audio, where the look-ahead delays only the first chunk (arXiv 2504.18425). Chunk-wise streaming of this kind is covered in Streaming and low latency.

Route 1 has one stage after the LM, and that stage can be causal. Qwen3-Omni can output the waveform immediately after the Talker generates each token (arXiv 2509.17765). Inference The price is on the LM side: it must predict every acoustic codebook of each frame. Moshi uses a depth transformer for that and Qwen3-Omni an MTP module, as described in Generation paradigms.

Why not predict samples directly

Inference A waveform at 24 kHz has 24,000 samples per second. Mimi emits 12.5 frames per second (arXiv 2410.00037), so one frame stands for 1,920 samples. Predicting samples one by one would make the sequence 1,920 times longer than a frame-rate LM sees. That is why current systems predict tokens or latents and leave sample-level detail to a decoder.

Diffusion and flow matching for speech

Flow matching is the objective behind many recent TTS systems. It plays two roles: fully NAR TTS, and the acoustic decoder after an AR LM.

Diffusion objective (DDPM)

The forward process mixes clean data $x_0$ with Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$. The network $\epsilon_\theta$ learns to predict that noise from the noisy input and the timestep $t$.

Here $\bar\alpha_t$ is the cumulative noise schedule, so it fixes how much signal survives at step $t$. Sampling runs the learned denoiser backwards from pure noise.

This is the noise-prediction objective of the DDPM paper (arXiv 2006.11239).

Conditional flow matching objective

Flow matching regresses a network onto a velocity field. With a linear (optimal-transport) path, $x_0$ is noise and $x_1$ is data.

The target velocity $x_1 - x_0$ is constant along the path, so training is plain regression. There is no noise schedule and no SNR weighting.

Sampling integrates the learned ODE from noise to data:

The linear path and velocity regression follow Flow Matching with optimal-transport paths (arXiv 2210.02747).

Classifier-free guidance (CFG)

CFG pushes the output toward the condition $c$ by extrapolating away from an unconditional prediction $\varnothing$. The guidance scale $w$ sets the strength.

The cost is two forward passes per sampling step, so CFG doubles inference compute. The same form applies to a diffusion network by swapping $v_\theta$ for $\epsilon_\theta$.

Classifier-free guidance was introduced for diffusion models (arXiv 2207.12598); the form above applies it to a velocity network.

In speech, ZipVoice targets the CFG cost (arXiv 2506.13053). Lipman et al. show OT paths train and sample faster than diffusion paths (arXiv 2210.02747).

Where it sits

Counterexamples and traps

ASR has little flow work. Drax is the clearest, using discrete flow matching for NAR ASR (arXiv 2510.04162).

How flow-matching TTS works, and getting high-fidelity audio out

A flow-matching TTS model learns to fill a masked span of a mel spectrogram, given text and the audio around the gap. At inference the gap is the whole utterance after a reference clip, and an ODE solver fills it from noise. The objective and CFG are derived in Diffusion and flow matching; this section walks the pipeline step by step, then lists the levers that decide how good the final audio sounds.

Training: text-guided infilling

Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text (arXiv 2306.15687). Its loss covers only the masked frames; the unmasked frames are fed in and the masked ones are set to zero. Voicebox expands phones to frame level using forced-alignment durations. E2 TTS drops that: it pads the character sequence with a filler token up to the mel length, so no alignment or grapheme-to-phoneme step is needed (arXiv 2406.18009). F5-TTS keeps the filler padding but refines the text with four ConvNeXt V2 blocks before concatenating it with the speech input, and uses a DiT backbone with adaLN-zero (arXiv 2410.06885).

Duration: fix the length first

A non-autoregressive model must know the output length before it starts. Voicebox predicts phone durations with a separate regression model (arXiv 2306.15687). In E2 TTS the total length is the prompt length plus the generated length, and that length can be chosen freely. Its main experiments still used a separately trained regression duration model, for a fair comparison with Voicebox (arXiv 2406.18009). F5-TTS skips the model and estimates duration from the ratio of the prompt transcript length to the text length (arXiv 2410.06885).

Inference The F5 paper does not spell out which quantity the ratio scales. The natural reading is the prompt's audio duration. Either way, the estimate also acts as a speech-rate control: a shorter total gives faster speech. E2 TTS tested rates from 0.7 to 1.3 (arXiv 2406.18009).

Zero-shot voice cloning is infilling

At inference the reference clip becomes the unmasked context at the start. E2 TTS builds the input from the prompt's mel features, a zero-filled block for the part to generate, and the character sequence (prompt transcript, then target text, then fillers) (arXiv 2406.18009). The model continues the clip in the same voice, so no separate speaker encoder is part of that description. The same infilling objective gives Voicebox its other tasks, such as noise removal and content editing (arXiv 2306.15687).

Sampling: ODE steps, CFG and Sway Sampling

Sampling integrates $\mathrm{d}x/\mathrm{d}t = v_\theta(x,t)$ from noise with a fixed-step solver. Voicebox reports high quality in under 10 NFEs, and its default midpoint step of 0.0625 gives NFE 32 (arXiv 2306.15687). CFG on the conditions (see the shared definition above) roughly doubles the cost, since the model runs once with and once without them. F5-TTS drops the masked speech with probability 0.3 and then speech and text together with probability 0.2 during training, and reports RTF 0.15 at 16 NFE and 0.31 at 32 NFE for 10 s of speech on an RTX 3090 (arXiv 2410.06885).

F5-TTS adds Sway Sampling, an inference-only warp of the time grid that needs no retraining. It draws $u \sim U[0,1]$ and maps it to

which places more solver steps at small $t$. The authors argue that the early steps sketch the contour of the speech. In their ablation (Table 5, 32 NFE, LibriSpeech-PC) WER falls from 2.84 to 2.41 and speaker similarity rises from 0.62 to 0.66. They report that it also helps E2 TTS (arXiv 2410.06885).

The AR plus flow hybrid: CosyVoice 2

Here an AR language model writes speech tokens and the flow model only renders acoustics (arXiv 2412.10117). The flow model is conditioned on semantic tokens, a speaker embedding from an external 3D-Speaker model, and a masked speech feature, which at inference is the reference mel. Tokens run at 25 Hz and are up-sampled by two to match the 50 Hz mel at 24 kHz. A look-ahead layer (a right-padded 1-D convolution) precedes the up-sampling. Training masks 70 to 100% of the final mel frames, as in the NAR models.

The network is a causal convolutional Transformer UNet. For streaming, it samples one of four attention masks per training example, uniformly: non-causal, full-causal, chunk-M (past plus M future frames) and chunk-2M. Inference uses a cosine time schedule $t := 1 - \cos(\tfrac{1}{2}t\pi)$, which gives more steps to the initial stage, CFG strength 0.7 and 10 NFE. A pre-trained vocoder, not named in the paper, then makes the waveform. Inference Because the tokens fix the content and its length, this variant needs no duration estimate. Chunked use of this decoder is covered in Streaming and low latency.

Output: mel plus vocoder, or latent plus decoder

The flow model's output is a mel (or a latent) that still has to become samples. The routes and their trade-offs are compared in From tokens to waveform. Everything below concerns the last two boxes of the inference row.

Getting high fidelity out

Sample rate and bandwidth. The systems above use 24 kHz mels (E2, F5, CosyVoice 2) and Mimi uses 24 kHz audio (arXiv 2410.00037). DAC targets 44.1 kHz and trains on speech, music and environmental sound (arXiv 2306.06546). Its authors warn that the true sampling rate of source data varies (Common Voice is commonly 8 to 16 kHz), and that models trained on such data often fail to reconstruct content above the data's average true rate. Stable Audio Open generates 44.1 kHz stereo, yet its authors state it cannot produce intelligible speech or singing, so a high rate does not make a model a TTS system (arXiv 2407.14358).

Vocoder. HiFi-GAN pairs a multi-period discriminator with the multi-scale discriminator from MelGAN, plus feature-matching and mel L1 losses (arXiv 2010.05646). It reports MOS 4.36 for its V1 model against 4.45 for ground truth on a single-speaker set. BigVGAN adds the periodic Snake activation, an anti-aliased up-sample, activate, down-sample step with low-pass filters, and a multi-resolution spectrogram discriminator in place of the multi-scale one (arXiv 2206.04658). It is trained on LibriTTS at 24 kHz with a 100-band log mel and hop 256, scales to 112M parameters, and the authors claim it generalises to unseen speakers, languages and recording conditions. In F5-TTS, swapping Vocos for BigVGAN lowered WER from 1.83 to 1.62 at 32 NFE on Seed-TTS test-en (arXiv 2410.06885). That is an intelligibility measure, not a direct audio-quality score.

Codec bitrate and losses. DAC reaches about 90x compression of 44.1 kHz audio at 8 kbps, using nine 10-bit codebooks at an 86 Hz frame rate (arXiv 2306.06546). Its ViSQOL score rises from 3.76 at 1.78 kbps to 4.18 at 8 kbps, against 3.13 for EnCodec at 12 kbps. Its listening test finds it well above EnCodec at every bitrate, yet still below the reference. Naive quantizer dropout hurt full-bandwidth quality, so DAC applies it with probability 0.5. The loss mix is adversarial (1.0), feature matching (2.0) and multi-scale mel L1 (15.0). Mimi shows how much the losses matter: training with only adversarial and feature losses lifted MUSHRA from 58.8 to 81.0 at 1.1 kbps, while objective metrics degraded (arXiv 2410.00037). Its split RVQ raised MUSHRA from 57.8 to 64.0 and worsened ABX from 6.5% to 8.1%.

Super-resolution as a post step. AudioSR is a diffusion-based model that upsamples input of 2 to 16 kHz bandwidth to 48 kHz sampling (arXiv 2309.07314). Moshi's fine-tuning data, Fisher, is 8 kHz telephone audio, which the authors upsample to 24 kHz with AudioSR (arXiv 2410.00037). Inference A generative upsampler invents plausible high frequencies; it does not recover them.

Measuring it. MUSHRA (DAC, Mimi) and ViSQOL (DAC) are used for codecs. UTMOS comes from a MOS-prediction system for the VoiceMOS Challenge 2022 (arXiv 2204.02152), and F5-TTS reports it as an objective naturalness metric next to CMOS and SMOS. UTMOS is not fidelity to a reference: F5-TTS scores 3.72 against ground truth 3.53 on Seed-TTS test-en, but 3.89 against 4.09 on LibriSpeech-PC (arXiv 2410.06885).

Fidelity checklist

Sources did not give a confirmed figure for loudness normalisation or PESQ, so they are left out.

Continuous autoregression and next-token diffusion

A smaller, newer branch drops discrete tokens. An AR model predicts continuous latents, and a small diffusion or flow head samples each next frame or patch.

Inference The two lists below are design analysis, not results from the papers.

Streaming and low-latency generation

Streaming TTS needs a causal, low-frame-rate codec on the AR side, and a chunkable, few-step decoder on the acoustic side.

First-packet numbers

Latency levers

MeanFlow's headline FID 3.43 is an ImageNet one-step result (arXiv 2505.13447). IntMeanFlow reports 1-NFE token-to-spectrogram and 3-NFE text-to-spectrogram inference for speech (arXiv 2510.07979).