Speech understanding

Speech input for LLMs

Three ways to feed speech to an LLM: continuous encoder embeddings, discrete semantic tokens, or codec tokens. Most models use the first.

Per-model paths

Why 12.5 Hz converges

Whisper emits 50 Hz frames. Voxtral notes that 30 minutes of audio at 50 Hz is a sequence of 90k tokens, which is why it adds a 4x downsampling adapter (arXiv 2507.13264). Moshi's Mimi codec also runs at 12.5 Hz (arXiv 2410.00037).

Trade-offs

• Continuous: richest acoustic and paralinguistic signal, easy to train as ASR, but the output side needs a separate path.
• Discrete: one vocabulary for input and output, enables self-supervised next-token training, loses detail.
• Codec: the model hears and speaks the same tokens, but multi-codebook streams cost more per frame. Inference This suits duplex.
• Inference Downsampling to 10-12.5 Hz is the main sequence-length lever. It costs fine acoustic detail, which motivates hybrids like Kimi-Audio.

ASR with speech LLMs: offline and streaming

ASR is one more speech-to-text task on the encoder, projector and LLM stack. The design questions are how much to train, and how to stream.

Training recipes

• Freezing the LLM is the direct guard against forgetting. Granite-speech states that in text mode it calls the underlying LLM without LoRA, which preserves its text capabilities.
• Canary-Qwen's LLM mode takes a text transcript only, not raw audio (NVIDIA model card).

Streaming approaches

Inference Chunk size and lookahead trade latency against WER.

Multilingual scale

Omnilingual ASR (Meta) covers over 1,600 languages, in models from 300M to 7B parameters (arXiv 2511.09690). Qwen3-ASR covers 52 languages and dialects (arXiv 2601.21337).

Leaderboard state

The Open ASR Leaderboard compares 86 systems on 12 datasets, with English short- and long-form and multilingual tracks (arXiv 2510.06961). Its authors observe that Conformer encoders paired with Transformer decoders reach the best average WER, while CTC and TDT decoders give the best RTFx.

Open question and scope

• Not covered here: transducer plus LLM hybrids and CTC-prefix joint decoding.

How encoder audio and ASR text meet inside the LLM

A speech LLM must decide where audio positions and text positions sit in one sequence. That choice sets when text can be emitted, whether the model streams, and what alignment the training data needs. This section lays out six layouts from the papers; encoder rates are in Speech input, the Whisper front end is in Whisper, and the full streaming-ASR taxonomy is in Streaming ASR families.

Six layouts

1. Prefix (soft prompt). [audio e1 … eT] <asr prompt> → text tokens Qwen2-Audio models each text token given the previous text and the encoded audio, $P(x_t \mid x_{<t}, \mathrm{Encoder}(a))$, with the objective stated over the next text token (Section 2; arXiv 2407.10759). Its encoder is initialised from Whisper-large-v3, and a stride-2 pooling makes each frame about 40 ms (arXiv 2407.10759). The official repo prompt for the pretrained model is <|audio_bos|><|AUDIO|><|audio_eos|>Generate the caption in English:, and its chat example puts the audio item before the text item (QwenLM/Qwen2-Audio). Voxtral passes encoder output through an MLP adapter that downsamples 4x (50 to 12.5 Hz), and its decoder predicts text conditioned on the audio and text inputs (arXiv 2507.13264). Its pretraining uses an audio segment followed by its transcript (<repeat>), or alternating audio and text segments (<next>), and a special token selects transcribe mode with no text prompt (arXiv 2507.13264). Neither paper states whether the loss is masked to text positions, and neither describes the audio positions beyond conditioning. Inference The <|AUDIO|> placeholder suggests encoder frames replace those positions in the input embeddings.

2. Chunk-interleaved streaming. [audio chunk 1] [text 1] $ [audio chunk 2] [text 2] $ … SpeechLLM-XL splits the audio into 1.28 s chunks and trains the LLM to emit each chunk's text plus an end-of-chunk token $ (Sections III and IV-A; arXiv 2410.03752). The target transcript is segmented per chunk, with a $ after each chunk's tokens, and the audio encodings are interleaved with text embeddings by audio-text alignment (Section III, Fig. 1; arXiv 2410.03752). When reference alignments are missing, CTC forced alignment between audio encodings and text tokens gives each token's end time (Section III; arXiv 2410.03752). Self-attention caches the previous $b$ chunks, and at inference each chunk's text is generated until $ is predicted, then $ is dropped from the transcript (arXiv 2410.03752).

3. Per-frame wait or blank. e1 BLANK e2 w w BLANK e3 BLANK … In Speech ReaLLM an input embedding arrives every 240 ms, the model appends it and keeps predicting tokens until it predicts BLANK (Section 2, Algorithm 1; arXiv 2406.09569). A frame can therefore yield zero, one or several words. Training targets come from fixed alignments produced by an external CTC "alignment-teacher", with cross-entropy loss; the paper found it better to give the encoder 960 ms of right context than to delay the labels (Sections 3 and 4.2; arXiv 2406.09569).

4. Transducer with an LLM predictor. LLM(text history) + encoder frame → joint network → token or blank Transducer-Llama uses the LLM as the non-blank predictor at decoding; encoder output enters through the joint network, not the LLM input (Sections II and III; arXiv 2412.16464). The non-blank distribution combines both scores:

Here $P_b$ is the blank probability, $P_{ac}$ comes from the acoustic encoder and $P_{ilm}$ from the internal LM, which the LLM replaces after RNN-T training (Sections II and III-C; arXiv 2412.16464). The LLM is frozen apart from new embedding and output layers for the ASR vocabulary (Section III-A; arXiv 2412.16464). The paper does not state the predictor's input. Inference As in any transducer predictor, it sees the previous non-blank labels, not audio.

5. Time-aligned parallel streams. step t: emb(text token) + emb(audio tokens) → one vector → text out Moshi gives each stream its own embedding table and sums them: the Temporal Transformer takes "the sum of K learnt embedding tables" for the previous step, with $K = 2Q + 1 = 17$ streams for $Q = 8$ (Section 3.4; arXiv 2410.00037). One stream holds the aligned text, with PAD between words and EPAD before each word, and PAD is about 65% of tokens in English conversation (Section 3.4.4; arXiv 2410.00037). Steps are 80 ms (12.5 Hz). DSM does the same, "the embeddings are summed before going into the backbone" (Section 3; arXiv 2509.08753). For ASR it teacher-forces the audio tokens, computes loss on the text stream including PAD and WORD, and delays the text by a default 2.5 s (Sections 3.2 and 4; arXiv 2509.08753). More in Inner Monologue and who emits what per frame.

6. Hybrid continuous plus discrete input. per 12.5 Hz step: whisper_feature(t) + emb(semantic_token(t)) Kimi-Audio downsamples Whisper features from 50 to 12.5 Hz with an adaptor, then adds them to the semantic-token embeddings as the audio LLM input (Sections 2.2 and 4.1; arXiv 2504.18425). Confirmed The paper is inconsistent: its Introduction says it concatenates the two (arXiv 2504.18425). Inference The method sections are more specific, so treat it as addition.

Comparison

Position encoding for audio

Qwen3-Omni uses TM-RoPE, which splits the rotary embedding into temporal, height and width components (24, 20 and 20 rotary angles) (Section 2.3; arXiv 2509.17765). For text, all three components share one position ID, which makes it equal to 1D RoPE. For audio, each temporal ID spans 80 ms, and the AuT encoder emits 12.5 Hz tokens (Sections 2.2 and 2.3; arXiv 2509.17765). Each modality starts at one plus the largest position ID of the previous one, and audio and video align through absolute-time IDs (arXiv 2509.17765).

Inference The Qwen2-Audio and Voxtral papers state no separate audio position scheme, so I assume ordinary sequential positions there. In layout 5 a position is a time step by construction, since all streams sum into one backbone input per step (arXiv 2509.08753).

Where ASR text leaves the speech model

Supporting facts: SpeechLLM-XL drops $ from the final transcript (arXiv 2410.03752), and Speech ReaLLM's response after each embedding is often empty (arXiv 2406.09569). Streaming ASR trade-offs are in Streaming approaches and Look-ahead.

Whisper and the Whisper encoder

Whisper is an encoder-decoder Transformer trained on 680,000 hours of weakly supervised audio. Its encoder became the default front end for speech LLMs. Source: arXiv 2212.04356 unless stated.

Architecture

• Confirmed Input is 16 kHz audio turned into an 80-channel log-magnitude mel spectrogram (25 ms window, 10 ms stride), in 30 s segments.
• Confirmed The stem is two conv layers (width 3, GELU); the second has stride 2, so one encoder frame covers 20 ms.
• Derived: the 10 ms hop gives 100 mel frames per second (100 Hz). The stride-2 conv halves that to one frame per 20 ms (50 Hz). A 30 s window is therefore 30 s / 10 ms = 3,000 mel frames and 3,000 / 2 = 1,500 encoder frames. Short audio is padded to 30 s, so compute is paid for the full window.

Rule (derived arithmetic):

• large-v3: same architecture, 128 mel bins instead of 80, and a Cantonese token. Reported OpenAI reports a 10% to 20% error reduction over large-v2 (OpenAI model card).
• large-v3-turbo: a fine-tuned, pruned large-v3 with the decoder cut from 32 to 4 layers (OpenAI model card). The card lists it at 809M parameters.

Multitask token format

The decoder prompt carries the task as tokens: start-of-transcript, language, task (transcribe or translate), then timestamps or no-timestamps. One model therefore does ASR, X-to-English translation and language ID. Timestamps are quantised to 20 ms.

Training data and recipe

• Filtering: drop machine-generated transcripts (all caps or all lower, no punctuation). Compare audio language ID (VoxLingua107-trained) with CLD2 on the transcript and drop mismatches; English transcripts of other-language audio are kept as translation data.
• Optimiser: AdamW, gradient-norm clipping, linear decay after 2,048 warmup steps, batch 256, FP16 with activation checkpointing, 2 to 3 passes over the data.
• Augmentation: none for the first models. Large-v2 trained 2.5x more epochs with SpecAugment, stochastic depth and BPE dropout.

Long-form decoding

Whisper slides a 30 s window over long audio and decodes with heuristics (paper section 4.5).

• Beam search with beam 5.
• Temperature fallback from 0, in steps of 0.2 up to 1.0, when average log-probability is below -1 or the gzip compression ratio exceeds 2.4.
• Condition on the previous window's text only when temperature is below 0.5.
• No-speech threshold 0.6 to skip silent windows.

Limitations

• The model card warns that predictions may include text that was not spoken (hallucination) and that the sequence-to-sequence architecture is prone to repetitive text. It also states that the models cannot be used for real-time transcription out of the box.
• The fixed 30 s window, non-causal encoder and autoregressive beam decoder make streaming hard.

The encoder as a speech-LLM front end

Inference Three issues recur. Padding to 30 s wastes compute on short turns. 50 Hz is too long a sequence for an LLM, so every system adds an adapter (50 Hz / 4 = 12.5 Hz). Bidirectional attention blocks streaming, so teams either make the encoder causal (GLM-4-Voice) or replace it (AuT).

Making the Whisper encoder stream

Whisper does not stream for three reasons.

• The encoder takes a fixed 30 s window (arXiv 2212.04356), and Voxtral describes it as a fixed receptive field of 30 s (arXiv 2507.13264).
• Streaming wrappers pad every buffer to the full context length, so each partial result pays for 30 s (arXiv 2506.12154).
• Encoder attention is bidirectional, so every frame sees later audio (GLM-4-Voice replaces it with block-causal attention, arXiv 2412.02612). The model card states the models cannot do real-time transcription out of the box.

Four families fix this, with different cost.

• Causal fine-tuning. CarelessWhisper fine-tunes Whisper with LoRA so each encoder frame attends only to its own chunk and earlier chunks, and it caches encoder keys and values instead of re-encoding padded buffers (arXiv 2508.12301). It trains on 960 h of LibriSpeech with chunk sizes of 40 to 300 ms. The U2-style variant puts a CTC head on the Whisper encoder, trains with dynamic chunk masks (0.1 to 1.0 s) on inputs shorter than 30 s, and rescores with the original decoder (arXiv 2506.12154). GLM-4-Voice makes the encoder causal inside its tokenizer: causal convolution before the Transformer and block-causal attention, then pooling and vector quantisation give 12.5 Hz tokens (arXiv 2412.02612).
• Replace the encoder. Qwen3-Omni trains AuT (Audio Transformer) from scratch on 20 million hours of supervised audio, with 8x Conv2D downsampling to 12.5 Hz and block-wise window attention for real-time prefill caching (arXiv 2509.17765). Qwen3-ASR reuses AuT with a dynamic flash-attention window of 1 s to 8 s, so one model serves streaming (short chunks) and offline (long queries) (arXiv 2601.21337).
• Wrap the offline model. whisper_streaming re-runs Whisper on a growing buffer and commits words once consecutive runs agree (LocalAgreement), at 3.3 s latency (arXiv 2307.14743). Simul-Whisper uses the cross-attention alignment to guide decoding on chunks, with a truncation detector for cut words (arXiv 2406.10052). The U2 authors note that wrappers reprocess the same audio and leave a training-inference mismatch (arXiv 2506.12154).

Streaming audio encoders beyond Whisper

Most streaming encoders do one of two things: restrict attention to a causal or windowed context, or cache past activations.

• Qwen3-ASR numbers: the paper reports 2.69 average WER offline vs 3.33 streaming for the 1.7B model, and 3.48 vs 4.40 for 0.6B, with streaming run at a 2 s chunk size (arXiv 2601.21337). It also reports an average time to first token as low as 92 ms for the 0.6B model.
• DSM-ASR: feeds Mimi tokens (12.5 Hz) and delays the text stream by a default 2.5 s, with timestamp precision of 80 ms (arXiv 2509.08753).
• Cache-aware Conformer: with a regular look-ahead, latency compounds with depth (1,360 ms for 17 layers and one token of look-ahead); chunk-aware look-ahead does not. On LibriSpeech test-other with an RNNT decoder, the cache-aware model reaches 6.3% WER at 1,360 ms, against 11.3% for buffered inference at 2,000 ms (arXiv 2312.17279).
• The model card lists a 120M-parameter cache-aware FastConformer-RNNT with 17 layers and latency of 80 to 160 ms.

Ecosystem

Interview insights

Look-ahead (right context) in streaming encoders

Look-ahead, also called right context, is how much future audio an encoder may see before it commits to an output for the current frame. Zero look-ahead means each output depends only on the past, so it can be emitted as soon as its frame arrives. N frames of look-ahead means each output waits for N more frames, which helps resolve ambiguous sounds and word boundaries but adds latency.

Reading "0LA" and "12LA"

Inference Shorthand such as 0LA and 12LA names a streaming configuration by its look-ahead in encoder frames: 0 frames or 12 frames. What 12 frames costs depends on the frame duration after subsampling, and on whether the look-ahead is applied per layer or per chunk (below).

For comparison, Speech ReaLLM uses 240 ms input frames with 960 ms of encoder right context (arXiv 2406.09569).

The latency and accuracy trade-off

NVIDIA's cache-aware streaming FastConformer paper measures the trade-off directly (arXiv 2312.17279). On LibriSpeech test-other with a hybrid model and chunk-aware look-ahead, the authors report:

Zero look-ahead costs about 50% relative WER against the 1,360 ms setting (derived, RNNT), and most of the gain arrives within a few hundred milliseconds. In the same paper, buffered (non-cache-aware) streaming reaches only 11.3% WER at 2,000 ms with RNNT, worse than cache-aware streaming with zero look-ahead.

Regular against chunk-aware look-ahead

How look-ahead is implemented matters as much as how much is allowed (arXiv 2312.17279).

So "12LA" is cheap or expensive depending on whether it means 12 frames once at the chunk level, or 12 frames per layer that compound through the stack. Inference

One model, several look-aheads

The same paper trains a single model with multiple look-ahead settings and compares it to separate single-setting models on LibriSpeech test-other (arXiv 2312.17279). With RNNT, the multi-look-ahead model reports 6.2%, 5.5% and 5.2% WER at 40, 240 and 520 ms, against 6.4%, 5.9% and 5.4% for single-look-ahead models. That lets a product pick the latency at inference time.

NVIDIA's Parakeet realtime end-of-utterance model card lists a cache-aware streaming FastConformer-RNNT with 80 to 160 ms latency.