How does speech recognition process filler words like 'um' and 'uh'?

Speech recognition systems handle filler words like “um” and “uh” through a combination of acoustic modeling, language modeling, and post-processing rules. These systems are typically trained on large datasets that include spontaneous speech, which naturally contains disfluencies. During processing, the acoustic model identifies the sounds associated with fillers, while the language model assesses their likelihood in a given context. For example, if a pause or a low-confidence phonetic segment is detected, the system might classify it as a filler based on patterns learned during training. However, whether the filler is included in the final output depends on the application’s requirements—some systems retain them for accuracy, while others filter them out.

The technical process involves multiple stages. First, the raw audio is converted into phonemes (distinct units of sound) using acoustic models. Filler words often have unique phonetic characteristics, such as elongated vowels or low-energy pauses, which the model can detect. Next, language models predict the probability of specific words appearing in sequence. Since “um” and “uh” are common in informal speech, the language model might assign them a higher probability in contexts like pauses between clauses. However, many systems apply confidence thresholds: if a detected segment has low confidence as a meaningful word, it’s flagged as a filler. For instance, a voice assistant like Alexa might suppress fillers to isolate actionable commands, while a transcription service could retain them for verbatim accuracy.

Developers can influence how fillers are handled based on the use case. APIs like Google’s Speech-to-Text or AWS Transcribe often provide options to enable “profanity filtering” or “disfluency removal,” which indirectly target fillers. For custom systems, post-processing rules—such as regex patterns to remove known filler words—can be added after transcription. However, this requires balancing accuracy: over-aggressive filtering might delete legitimate words (e.g., “um” in a medical term like “umbilical”). Testing with diverse datasets, including spontaneous speech samples, is critical. For example, a telehealth app might prioritize retaining fillers to capture patient nuances, while a meeting summarization tool might discard them for brevity. The key is aligning the system’s behavior with the end user’s needs.