Neurosymbolic Robot Programming: A Framework for AI-Enabled Programming of Robot Manipulation Tasks

Abstract

The vision of robots as intelligent assistants, capable of solving manipulation tasks in domains ranging from household assistance to industrial manufacturing, requires methods for humans to endow them with the cognitive and physical abilities to understand our intents and competently act in accordance with them. The need for capable robot behavior is accompanied by an equal need for control: Pervasive use of robots carries significant safety implications, implying a need for humans to understand robot behavior. This work introduces a neurosymbolic framework for robot programming that combines neural, subsymbolic representations that afford learning and first-order optimization with symbolic representations that afford human interaction and understanding. It introduces Neurosymbolic Robot Programs (NRPs), a dual robot program representation that associates a skill-based, symbolic robot program with a differentiable, predictive model of robot behavior. NRPs bridge the representational divide between symbolic and subsymbolic program representations and serve as a data structure for program synthesis and optimization algorithms that offer powerful AI assistance to human programmers, while ultimately leaving the human in control of robot behavior. This work introduces a family of first-order program optimization algorithms that optimize robot program parameters and low-level motion trajectories with respect to near-arbitrary task objectives and constraints. It also introduces a family of program synthesis systems that generate executable robot programs by leveraging structured representations of task and domain knowledge. Taken together, they form a neurosymbolic programming framework capable of addressing major challenges in programming robots to solve complex, real-world manipulation tasks. The framework and its components are evaluated on tasks ranging from retail and household fetch-and-place to industrial surface treatment and electronics assembly.