Prospective perception through cognitive emulation for robot manipulation tasks: "Perceiving like humans do"

This thesis argues that bottom-up theories of perception suffer from the high semantic entropy arising from the severe spatial, temporal, and informational limitations of sensory input during everyday manipulation tasks. However, by emulating the “dark matter” of perception — including intent, functionality, utility, causality, and physis — and integrating it with sparse sensory data, robotic perception can achieve a causal, transparent, and computationally efficient ability to anticipate and explain relevant events in such tasks.

To this end, the thesis introduces Probabilistic Embodied Scene Grammars (PESGs) to formalize this perceptual “dark matter.” It also presents a generator and a parser to respectively anticipate and explain event-centric scenes. The approach is demonstrated in complex real-world scenarios, including household tasks such as pancake making in kitchen environments, shopping tasks in supermarkets, and sterility testing tasks in medical laboratories.