Speech separation for monolingual and multilingual cocktail party scenarios

In everyday speech communication, humans face noisy multi-speaker soundscapes with overlapping sound sources, typically described as the cocktail party problem. Without much effort, humans can focus their attention on a specific voice or sound source while fading out the remaining voices or sound sources, referred to as selective auditory attention. For the development of future hearing aids and sophisticated algorithms for speech-based human-computer interaction, it is of great interest to equip machines with the same ability.

The research field dedicated to the development of machine learning algorithms that embed this human ability goes by the name of speech separation. The underlying scientific problem is to develop algorithms that work in the wide range of everyday cocktail party scenarios just as the selective auditory attention ability of humans.

This cumulative dissertation concerns deep neural network based single-channel speech separation and addresses five scientific problems (P): Target speaker absence and monologues (P1) deals with attended speakers in cocktail party scenarios who stop speaking for a while or hold monologues. Reliable target speaker reference (P2) formulates methods to enhance the reliability and stability of speech separation algorithms controlled by brain signals. Speech mode generalization (P3) has the goal to develop algorithms that work for multiple speech modes, such as normal and whispered speech. Cross-language generalization (P4) aims to develop algorithms that work for several seen and unseen languages. The multilingual cocktail party problem (P5) considers a cocktail party problem in which multiple languages are spoken and develops language-based speech separation algorithms. The contributions of this dissertation to solving P1-P5 mark a step towards the overall goal of equipping machines with a human-like ability of selective auditory attention that works reliably in real-world cocktail party scenarios.