Steps Towards Generalized Manipulation Action Plans - Tackling Mixing Task

In the rapidly evolving field of household robotics, the ability to autonomously execute complex tasks like "Serve Me Breakfast" introduces considerable technical challenges, particularly in the precise and adaptive handling of mixing tasks that involve substances with diverse densities and viscosities. This study advances the field of cognitive robotics by delving into the intricacies of everyday, complex, and highly variable activities within household settings. These scenarios demand that robots manage multiple, interconnected actions—a concept known as schemas. We present a holistic strategy to enhance household robots with sophisticated task execution capabilities, utilizing the Cognitive Robotic Abstract Machine (CRAM) framework for strategic planning and execution. Our method begins with an in-depth analysis of human behaviour to develop a theoretical model that guides the creation of adaptable and comprehensible action plans for robots. A crucial element of our approach involves using Narrative Enabled Episodic Memories (NEEMs), which capture detailed records of task executions to aid performance analysis and experiential learning. We propose incorporating additional criteria based on events recorded in the NEEMs to assess task success. For instance, a robot maintaining a steady grip on the whisk while interacting with a fluid suggests correctly executing mixing tasks. These criteria enable further evaluation of performance through simulations, despite potential limitations in simulation fidelity. This article explores the transition from human expertise to robotic execution of mixing tasks in household environments, the methodology for gathering and analysing NEEMs, and their prospective future applications.