Shvach Shprach: An analysis of automatic speech recognition on low-resource languages

Abstract

Natural language processing (NLP) is a field of artificial intelligence that is focused on training computers to understand human language. By analyzing copious amounts of data, computers are trained to recognize patterns that can be represented numerically as vectors. Through this, computers are able to complete many language related tasks. One such task is automatic speech recognition (ASR), a process in which audio can be converted into text. Like all NLP tasks, the performance of an ASR model is heavily dependent on the quality of data the model is trained on. Whisper is an ASR model created by OpenAI. Though Whisper works very well when transcribing English audio, its accuracy plummets when transcribing languages that are less represented in the training data, also known as low-resource languages. For example, in its current state, Whisper is highly inaccurate when used to transcribe Torah lectures given in Yeshivish English, a sociolect of English spoken by American Orthodox Jews. • This paper sets out to provide the background knowledge necessary to understand how ASR models work, with a focus on OpenAI’s Whisper model. Additionally, it includes a thorough analysis of Whisper’s performance on low-resource languages through experimentation with transcriptions of Rabbi Aryeh Lebowitz’s “Ten Minute Halacha” lecture series. Finally, this paper explores different technologies and techniques that can be used to improve Whisper’s performance.