What methods are used to measure intelligibility in TTS outputs?

To measure intelligibility in text-to-speech (TTS) outputs, developers use subjective evaluations, objective metrics, and hybrid approaches. Each method addresses different aspects of how clearly and accurately synthesized speech is perceived by listeners or analyzed by systems. These techniques help identify issues like mispronunciations, unnatural pacing, or audio artifacts that reduce understandability.

Subjective evaluations rely on human listeners to rate or transcribe TTS outputs. Common tests include the Diagnostic Rhyme Test (DRT), where listeners distinguish between similar-sounding words (e.g., “bat” vs. “pat”), and the Mean Opinion Score (MOS), which rates speech quality on a scale (e.g., 1–5). Crowdsourcing platforms like Amazon Mechanical Turk are often used to gather large listener samples efficiently. However, these tests require careful design to avoid bias, such as using randomized prompts or filtering low-quality responses. For example, a developer might ask 100 participants to transcribe 20 TTS-generated sentences, then calculate the percentage of correctly understood words. While subjective methods are time-consuming, they provide direct insight into human perception, which automated tools might miss.

Objective metrics use algorithms to quantify intelligibility without human input. The Word Error Rate (WER) compares automatic speech recognition (ASR) transcriptions of TTS outputs to the original text, flagging mismatches. A WER below 10% is often considered acceptable. Tools like Mozilla DeepSpeech or Whisper can automate this process. Another metric, the Speech Transmission Index (STI), analyzes acoustic clarity by measuring how well the signal preserves frequency bands. Developers might also inspect phonetic alignment—checking if synthesized phonemes (e.g., /k/ in “cat”) match expected durations. However, objective methods have limitations: ASR systems may struggle with accented speech, and acoustic metrics like STI don’t account for contextual language understanding.

Hybrid approaches combine subjective and objective data. For instance, a developer could use WER to flag problematic sentences, then run targeted human evaluations on those samples. Tools like Google’s TTS Evaluation Toolkit integrate ASR-based metrics with prosody analysis (e.g., pitch contours) to identify unnatural intonation. Another example is training machine learning models to predict MOS scores using acoustic features (e.g., mel-frequency cepstral coefficients) from TTS outputs. These hybrid methods balance scalability and accuracy, enabling iterative testing during TTS model training. For example, a team might automate WER checks in their CI/CD pipeline while reserving monthly human evaluations for critical updates. This layered approach ensures both technical rigor and alignment with user experience.