EEG-based Auditory Attention Tracking in Dynamic Multi-Talker Scenes
Date Issued
2026-08-14
Author(s)
Supervisors
Li, Haizhou
Referees
Nakamura, Satoshi
Abstract
Humans can selectively attend to one speaker in the presence of competing voices and background noise, a phenomenon known as the cocktail-party effect (Cherry, 1953; Bronkhorst, 2000). Translating this ability into assistive listening technology remains difficult because an audio device must infer which source the listener intends to follow, often under short decision windows, multiple competing talkers, reduced sensor layouts, and changing spatial configurations (O’Sullivan et al., 2015; Van Eyndhoven et al., 2017). EEG-based Auditory Attention Decoding (AAD) addresses this problem by estimating the listener’s attentional focus from neural activity and is therefore an important research direction for future neuro-steered listening systems (Crosse et al., 2016; Ciccarelli et al., 2019; Geirnaert, Vandecappelle, et al., 2021). In this dissertation, such systems are treated as the motivating application context. The work develops EEG-based attention decoding and tracking methods, but it does not claim a clinically validated hearing-aid system, an embedded commercial hearable prototype, or validation with hearing-impaired users wearing hearing devices.
The scientific problem underlying this dissertation is how to move EEG-based AAD from controlled proof-of-concept settings toward deployment-relevant research conditions. Prior work has reported promising results, yet many studies still rely on relatively static scenes, permissive window-level evaluation, subject-specific calibration, scalp-mounted sensors, or discrete attended-speaker labels (Ivucic, Pahuja, et al., 2024; Debener et al., 2015; Bednar and Lalor, 2020). Practical neuro-steered listening research requires a broader formulation: decoders should work with short windows without overstating real-time performance, should be evaluated under leakage-controlled trial- and subject-independent protocols, should remain robust when sensor layouts are reduced toward ear-centered recordings, and should extend from static speaker selection to continuous tracking of attended spatial trajectories.
This cumulative dissertation addresses this problem through five connected research directions. First, it investigates short-window and reduced-montage binary AAD (bAAD) using cross-hemispheric attention in XAnet (Pahuja et al., 2023a). Second, it studies leakage-controlled evaluation for windowed EEG decoding and shows that cross-validation and segmentation choices can bias reported AAD performance when temporal dependence is not respected (Ivucic, Pahuja, et al., 2024; Pahuja et al., 2024). Third, it improves subject-independent AAD through physiologically motivated data augmentation and geometry-aware graph modeling (Pahuja et al., 2023b; Ivucic, Pahuja, et al., 2025). Fourth, it extends AAD to wearable multi-talker decoding (mAAD) with ear-EEG, where graph and cross-attention models capture intra-ear and inter-ear structure, and where DemoSAA demonstrates how four-speaker ear-EEG decoding can be organized in an interactive research pipeline (Pahuja et al., 2025c; Pahuja et al., 2025a). Fifth, it reframes AAD in moving-speaker scenes as tracking AAD (tAAD), where the target is not only a discrete attended talker but a time-varying attended azimuth trajectory estimated under causal and leakage-controlled conditions (Pahuja et al., 2025d; Pahuja et al., 2025b).
Together, the results show that progress in EEG-based AAD depends on the joint design of model architecture, sensing strategy, task formulation, and evaluation protocol. Cross-attention and graph-based inductive bias improve short-window robustness; blocked evaluation gives more credible estimates of trial and subject transfer; augmentation and geometry-aware learning reduce subject-transfer degradation; ear-EEG can support multi-class attention decoding despite sparse sensing; and temporal tracking models enable reconstruction of attended spatial trajectories from EEG in dynamic scenes. The overall contribution is a dissertation-level argument that reliable neuro-steered listening research cannot emerge from stronger neural networks alone, but requires structured modeling combined with leakage-free, deployment-aware evaluation.
The scientific problem underlying this dissertation is how to move EEG-based AAD from controlled proof-of-concept settings toward deployment-relevant research conditions. Prior work has reported promising results, yet many studies still rely on relatively static scenes, permissive window-level evaluation, subject-specific calibration, scalp-mounted sensors, or discrete attended-speaker labels (Ivucic, Pahuja, et al., 2024; Debener et al., 2015; Bednar and Lalor, 2020). Practical neuro-steered listening research requires a broader formulation: decoders should work with short windows without overstating real-time performance, should be evaluated under leakage-controlled trial- and subject-independent protocols, should remain robust when sensor layouts are reduced toward ear-centered recordings, and should extend from static speaker selection to continuous tracking of attended spatial trajectories.
This cumulative dissertation addresses this problem through five connected research directions. First, it investigates short-window and reduced-montage binary AAD (bAAD) using cross-hemispheric attention in XAnet (Pahuja et al., 2023a). Second, it studies leakage-controlled evaluation for windowed EEG decoding and shows that cross-validation and segmentation choices can bias reported AAD performance when temporal dependence is not respected (Ivucic, Pahuja, et al., 2024; Pahuja et al., 2024). Third, it improves subject-independent AAD through physiologically motivated data augmentation and geometry-aware graph modeling (Pahuja et al., 2023b; Ivucic, Pahuja, et al., 2025). Fourth, it extends AAD to wearable multi-talker decoding (mAAD) with ear-EEG, where graph and cross-attention models capture intra-ear and inter-ear structure, and where DemoSAA demonstrates how four-speaker ear-EEG decoding can be organized in an interactive research pipeline (Pahuja et al., 2025c; Pahuja et al., 2025a). Fifth, it reframes AAD in moving-speaker scenes as tracking AAD (tAAD), where the target is not only a discrete attended talker but a time-varying attended azimuth trajectory estimated under causal and leakage-controlled conditions (Pahuja et al., 2025d; Pahuja et al., 2025b).
Together, the results show that progress in EEG-based AAD depends on the joint design of model architecture, sensing strategy, task formulation, and evaluation protocol. Cross-attention and graph-based inductive bias improve short-window robustness; blocked evaluation gives more credible estimates of trial and subject transfer; augmentation and geometry-aware learning reduce subject-transfer degradation; ear-EEG can support multi-class attention decoding despite sparse sensing; and temporal tracking models enable reconstruction of attended spatial trajectories from EEG in dynamic scenes. The overall contribution is a dissertation-level argument that reliable neuro-steered listening research cannot emerge from stronger neural networks alone, but requires structured modeling combined with leakage-free, deployment-aware evaluation.
Subjects
EEG
;
Auditory Attention Decoding
;
Auditory Attention Tracking
;
Ear-EEG
;
Neuro-steered Listening
Institution
Institute
Type
Dissertation
License
Alle Rechte vorbehalten
Language
English
File(s)![Thumbnail Image]()
Loading...
Name
EEG-based Auditory Attention Tracking in Dynamic Multi-Talker Scenes.pdf
Size
39.86 MB
Format
Adobe PDF
Checksum
(MD5):2833a9b7d52f14b221d1164d638bb8b1