Head/Eyes/Visual Routines

  Head/Eyes/Visual Routines

Head/Eye Orientation

These clips show Cog orienting its head and neck to a visual stimulus. The eyes are moving to look at moving objects in the field of view. Whenever the eyes move to their new target, the neck moves to point the head toward that same target. The first clip is from the new revision of Cog's head.

This orientation clip is from Cog's old head.

Saccadic Eye Movement

Human eyes move as the result of one of four mechanisms. Two of these mechanisms are under voluntary control (saccadic movements, and smooth pursuit movements), while two are under involuntary control (the vestibulo-occular reflex and non-stabilizing micromovements). The saccadic movements are high-speed movements that cause the eye to jump to a new location approximately three times per second This short clip shows the first tests of the new active vision system as it saccades to random positions. Three of these heads were developed, one for Cog, and two to serve as desktop development platforms. The eyes are travelling at about one-half their maximum velocity.

Saccade to Motion

In this clip, Cog has been programmed to attend to moving objects. This motion detection operates by subtracting consecutive images, and the using region growing to identify boundaries of moving objects. In this video clip, you can see the eyes saccade to the moving stuffed animal.

Smooth Pursuit Tracking

The second type of voluntary eye motion is smooth pursuit tracking. This clip shows Cog smoothly tracking a moving object that was placed in front of it. The tracking uses a correlation based metric to determine where the desired object has moved in the visual field.

Vestibulo-Occular Reflex

One of the involuntary eye movements is the vestibulo-occular reflex. This reflex serves to keep the eyes fixed on a target while the head moves (or is moved). In humans, this relfex is accomplished by two systems:a very tight feedback loop from the vestibular system to the eye muscles, which is active at high velocities, and a measurement of visual slip which is active at slow velocities. We have implemented the high velocity vestibular reflex on Cog. Using two rate gyroscopes, we can measure the angular velocity of the head and move the eyes to compensate for that motion. The clip below first shows the head being moved without the vestibular reflex. Notice that the eyes move back and forth with the head as it moves. The second part of the clip shows the head being moved with the vestibular reflex intact. Notice that instead of moving with the head, the eyes continue to point straight ahead regardless of how the head is moved.

Face Detection

One visual task that infants are very good at is face detection. The face detection routine shown here was developed based on the ratio template work of Pawan Sinha. This clip shows the output of the face detection module. On the right side of the clip is the live video stream. On the left side is the same image, but with detected faces outlined. A red outline indicates a better match than a green outline. Notice that the face detection software is not sensitive to face motion, as can be seen when the face is occluded by the circular mountain picture shown in this clip.

Eye Finding

We would like Cog to be able to tell if someone is making eye contact with it. Using the face detection routines described above, we first locate a face in the peripheral camera. Using a learned sensory-motor mapping, Cog moves its eyes to look at that person. We then can use a second learned sensory-motor mapping to extract an image of that person's eyes. This clip shows the image processing steps used to find eyes. The upper right shows the raw video image and the upper left shows the outlined face images. The lower left is the prefilter results, and the lower right shows the extracted image of the eye. The image of the eye is not stable because the person in the video is moving almost continuously.

Imitating Head Nods

One of the long range tasks that we would like the robot to be able to perform is to imitate gestures and motion. The following clips show a very simple example of imitation of head motion. The output from the face detection module is passed to a tracking module, which then characterizes head motions as being either horiztonal "no" motions or vertical "yes" motions. The first clip shows one of the small active vision development platforms imitating head motions. Notice that the head responds only to a head nodding; similar motions with non-face stimuli do not provoke a response.

The second clip shows Cog imitating the head motions of a person.

The third clip shows Cog imitating the head motions of a toy cow. The stuffed animal is detected as a face, and the robot responds to it in the same way that it responds to a person.

The fourth clip shows Cog imitating a second stuffed animal (Mickey Mouse). The robot only responds to the face of the toy, not to the motion of the toy.

Representatives of the press who are interested in acquiring further information about the Cog project should contact Elizabeth Thomson, thomson@mit.edu, from the MIT News Office,  http://web.mit.edu/newsoffice/www/ .