Icon Legend

This session is not in your schedule.

This session is in your schedule. Click again to remove it.

Biomedical Imaging and Instrumentation

Biomedical Imaging and Instrumentation - Poster Session B

Poster G4 - Evaluating Speech Intelligibility and Inter-Talker Variability of Synthesized Voices in Speech-Shaped Noise

Thursday, October 24, 2024

2:45 PM - 3:45 PM EST

Location: Exhibit Hall E, F & G

Co-Author(s)

YY

Ye Yang, n/a

PhD Candidate
Department of Biomedical Engineering, University of California, Irvine, United States

Presenting Author(s)

Dathan Nguyen

Undergraduate
Department of Biomedical Engineering, University of California, Irvine
ELK GROVE, California, United States

Co-Author(s)

KC

Katherine Chen, n/a

Undergraduate
Department of Biomedical Engineering, University of California, Irvine, United States

Primary Investigator(s)

FZ

Fan-Gang Zeng, n/a

Principal Investigator
Cognitive Sciences and Otolaryngology – Head and Neck Surgery, University of California, Irvine, United States

Introduction: Text-To-Speech (TTS), or synthesized speech has made significant progress in the last 70 years, surpassing the intelligibility of human speech. Commercial platforms including Amazon, Microsoft Azure, and Google offer high-quality synthesized speech to the public. A large challenge that has emerged is TTS's ability to remain highly intelligible in noisy environments. For natural human speech, some voices are easier to understand than others in the same noisy environments. This intelligibility difference is called inter-talker variability and has been studied for decades. However, whether synthesized voices exhibit the same inter-talker variability as human voices is unknown. This study aims to bridge this gap by evaluating both the subjective and objective intelligibility of synthesized speech in speech-shaped noise. The research also explores the capabilities of modern Automatic Speech Recognition (ASR) systems in capturing these variabilities and their potential to surpass human listeners in recognizing speech in noise.

Materials and

Methods: This study tested 250 sentences from the Hearing In Noise Test (HINT), organized into 25 lists of ten sentences each. Recordings were made by four human talkers (two females, two males) in a double-wall sound room in the Hearing and Speech Lab, University of California, Irvine using a Shure SM58 microphone at a 48000 Hz sampling rate, later downsampled to 16000 Hz. Additionally, synthesized speech was generated using twelve voices (six females, six males) from Amazon Polly, Microsoft Azure Text-To-Speech, and Google Text-To-Speech.

Twelve native English speakers with normal hearing thresholds participated in subjective listening tests. They listened to the sentences in both quiet and noisy environments (-5 dB SNR). Each participant typed the words they understood, and their responses were automatically checked for spelling accuracy.

For objective evaluation, five ASR systems were used: Amazon Transcribe, Microsoft Azure Speech-To-Text, Google Speech-To-Text, wav2vec2, and Whisper. The intelligibility of the synthesized and human speech was also assessed using five established objective intelligibility metrics: short-time objective intelligibility (STOI), weighted spectro-temporal modulation index (wSTMI), spectro-temporal glimpsing index (STGI), envelope-regression speech transmission index (ER-STI), and hearing-aid speech perception index version 2 (HASPI-v2). The ASR systems and objective metrics provided a comprehensive evaluation framework for comparing the intelligibility of synthesized and human voices in challenging acoustic environments.

Results, Conclusions, and Discussions: It was found that synthesized voices showed similar inter-talker variability as human voices, and some synthesized voices were more intelligible than human voices in the same noise condition. It was also found that modern Automatic Speech Recognition (ASR) systems captured this inter-talker variability as effectively as human listeners and two ASR systems recognized 10-percentage point more words than human listeners. Not only did this study find that modern artificial systems are generating and recognizing speech better than humans in certain situations, but also provides a framework to evaluate the intrinsic clarity of different voices, which helps to select or synthesize more intelligible speech in challenging acoustic environments.

Acknowledgements (Optional): Special thanks to the Hearing and Speech Lab at UC Irvine for allowing us to use their facilities.