The role of visual grounding in visual question answering generalization and shortcut learning

Abstract

We set out on this thesis’ journey with a conceptual idea of what Visual Grounding’s role and impact in VQA should be, but soon learn that - apart from the interference of shortcut learning with these expectations - there are also other fundamental limitations of contemporary Visual Grounding research that impede an in-depth analysis, namely a lack of well-defined procedures to measure Visual Grounding in VQA and properly evaluate its impact, We address these limitations with our contributions and in the process gain a much clearer understanding about why the impact of Visual Grounding on VQA performance has been difficult to grasp and how its role can be better highlighted with appropriately designed evaluation scenarios. Finally, in the last chapter, this thesis culminates in the definition of a theoretical model (VGR) that clearly describes the role of Visual Grounding in the context of VQA generalization and shortcut learning, thereby marking the end of our journey.