Natural Tasking of Robots Based on Human Interaction Cues

MIT Computer Science and Artificial Intelligence Laboratory
The Stata Center
32 Vassar Street
Cambridge, MA 02139

PI: Rodney A. Brooks

DARPA logo

[Project Overview], [Approach], [Research Questions], [Achieved Deliverables], [Future Deliverables], [People], [Publications]

Cog turns a crank
M4 robot head drawing
Kismet plays with a frog
Coco the gorilla robot


We propose developing the perceptual and intellectual abilities of robots so that in the field, war-fighters can interact with them in the same natural ways as they do with their human cohorts. To illustrate our goals and illuminate the technical problems that we must solve to achieve these goals, we will outline three scenarios:

  • Showing a robot how to open the gas tank of an unfamiliar captured enemy vehicle
  • Instructing a robot to carry out a reconnaissance mission ranging a few hundred meters from the command post within a strife-torn downtown urban environment
  • Instructing a dextrous forklift-like robot to load a truck by showing it how the particular bulk food sacks should be stacked together, one by one

Our approach is based on two key ideas; imitation and social interaction.

Imitation involves the robot watching and listening to a person perform some task and then equivalently executing it. From its observations the robot must extract which aspects of the person's motions and utterances are essential to actually carrying out the task, which are part of the instruction but not part of the actual task, and which are simply connective or coincidental.

Social interaction involves the robot engaging a person in the same dynamic two-way communication processes which two people could share. Each participant gives the other subconscious cues that carry messages such as "I understand that", "you're going too fast", "I don't know what you mean", "I already know that", "look at what's important", "no, it's more like this", "you don't understand it", and "now you get it!". The mechanisms for these signals are complex and often interrelated and involve such indications as gaze direction, eye contact, averting eye contact, nodding, body posture shifts, facial expressions, head motions, pre-linguistic verbalizations ("hmmm" or "uh-huh"), and codified verbalizations ("Sir!").

The principles of development, embodimentand integrationcontribute to our approach. The process of development wherein humans perform incrementally more difficult tasks in complex environments as they mature inspires a developmental methodology for our robots. Embodiment emphasizes human-like aspects of our robots' bodies. The integration of multiple sensory modalities, physical degrees of freedom and behavioral systems all a single robot to imitate and interact with humans in a more sophisticated manner.

home button
next button

[Project Overview], [Approach], [Research Questions], [Achieved Deliverables], [Future Deliverables], [People], [Publications]