Sensory Substitution in Human-Computer Interaction: Bridging Computer Science, Design, and Cognitive Neuroscience

Sensory substitution devices (SSDs) are non-invasive human-machine interfaces that convey information from one sensory modality through another and have been shown to induce neuroplasticity. While Human-computer interaction (HCI) extensively explores different ways to engage the senses, the potential of neuroplasticity for the development of novel technologies has been largely overlooked. This work proposes using SSDs to give a framework for sensory technologies in HCI that alter human perception.

For this purpose, the work in this dissertation explores whether SSDs can be used to develop new forms of human-computer interaction, and how. Sensory perception is conceptualized as a programmable system that can be developed and enhanced, and SSDs as input devices for the programmable brain.

To ground this concept, a neuroplasticity-based design space for SSDs is proposed, constructed from literature reviews spanning neuroscience and HCI. The space categorizes devices along dimensions relating to their function as systems that shape perception by inducing neuroplasticity, including the substituted and substituting senses, level of augmentation from restorative to novel, nature of phenomenological experience, degree of perceptual learning, sensory masking, and embodiment proximity. This framework allows us to recognize gaps where the potential of SSDs has not yet been realized.

Four research questions are then addressed through twelve publications. On extending sensory perception, ThermalSense employs visual-to-auditory sensory substitution to convey thermal properties through auditory cues, extending the visual perceptual range to thermal information beyond its natural capacities, while Spatial Haptics enables distal object localization through vibrotactile feedback that imitates interaural level differences, placing “ears in the hands”. On achieving deferred phenomenology, where the sensory experience shifts toward the substituted sense, the Topo-Speech system demonstrates that combining sensory substitution with symbolic data shortens the path to proficiency, and a depth perception study further investigates the role of cross-modal correspondences in designing effective mappings for SSDs. Previously overlooked in traditional SSD research, the work demonstrates sensory substitution with the inner, interoceptive senses, demonstrating that internal bodily signals such as respiration and heartbeat can be externalized through exteroceptive modalities to enhance interoceptive abilities. The concept of sensory mediated interaction is proposed, with breathing-based immersive interactions, respiration-synchronized robotic embodiment, and an enriched embodiment environment designed for a cancer treatment facility providing initial evidence that SSDs can mediate effects on cognitive and affective functions beyond perception itself.

Together, these findings answer the overarching question in a bounded but affirmative way. SSDs can be leveraged to restore, enhance, and extend perception, to substitute information using interoceptive senses, and to mediate effects on embodiment and psychological state. The dissertation contributes design guidelines for perception-altering SSDs and proposes NeuroHCI as a framing for the developing research field at the intersection of neuroscience and human-computer interaction.