These clips show Cog orienting its head and neck to a visual stimulus. The eyes are
moving to look at moving objects in the field of view. Whenever the eyes move to their new
target, the neck moves to point the head toward that same target. The first clip is from
the new revision of Cog's head.
This orientation clip is from Cog's old head.
Human eyes move as the result of one of four mechanisms. Two of these mechanisms are
under voluntary control (saccadic movements, and smooth pursuit movements), while two are
under involuntary control (the vestibulo-occular reflex and non-stabilizing
micromovements). The saccadic movements are high-speed movements that cause the eye to
jump to a new location approximately three times per second This short clip shows the
first tests of the new active vision system as it saccades to random positions. Three of
these heads were developed, one for Cog, and two to serve as desktop development
platforms. The eyes are travelling at about one-half their maximum velocity.
Saccade to Motion
In this clip, Cog has been programmed to attend to moving objects. This motion
detection operates by subtracting consecutive images, and the using region growing to
identify boundaries of moving objects. In this video clip, you can see the eyes saccade to
the moving stuffed animal.
The second type of voluntary eye motion is smooth pursuit tracking. This clip shows Cog
smoothly tracking a moving object that was placed in front of it. The tracking uses a
correlation based metric to determine where the desired object has moved in the visual
One of the involuntary eye movements is the vestibulo-occular reflex. This reflex
serves to keep the eyes fixed on a target while the head moves (or is moved). In humans,
this relfex is accomplished by two systems:a very tight feedback loop from the vestibular
system to the eye muscles, which is active at high velocities, and a measurement of visual
slip which is active at slow velocities. We have implemented the high velocity vestibular
reflex on Cog. Using two rate gyroscopes, we can measure the angular velocity of the head
and move the eyes to compensate for that motion. The clip below first shows the head being
moved without the vestibular reflex. Notice that the eyes move back and forth with the
head as it moves. The second part of the clip shows the head being moved with the
vestibular reflex intact. Notice that instead of moving with the head, the eyes continue
to point straight ahead regardless of how the head is moved.
One visual task that infants are very good at is face detection. The face detection
routine shown here was developed based on the ratio template work of Pawan Sinha. This
clip shows the output of the face detection module. On the right side of the clip is the
live video stream. On the left side is the same image, but with detected faces outlined. A
red outline indicates a better match than a green outline. Notice that the face detection
software is not sensitive to face motion, as can be seen when the face is occluded by the
circular mountain picture shown in this clip.
We would like Cog to be able to tell if someone is making eye contact with it. Using
the face detection routines described above, we first locate a face in the peripheral
camera. Using a learned sensory-motor mapping, Cog moves its eyes to look at that person.
We then can use a second learned sensory-motor mapping to extract an image of that
person's eyes. This clip shows the image processing steps used to find eyes. The upper
right shows the raw video image and the upper left shows the outlined face images. The
lower left is the prefilter results, and the lower right shows the extracted image of the
eye. The image of the eye is not stable because the person in the video is moving almost
One of the long range tasks that we would like the robot to be able to perform is to
imitate gestures and motion. The following clips show a very simple example of imitation
of head motion. The output from the face detection module is passed to a tracking module,
which then characterizes head motions as being either horiztonal "no" motions or
vertical "yes" motions. The first clip shows one of the small active vision
development platforms imitating head motions. Notice that the head responds only to a head
nodding; similar motions with non-face stimuli do not provoke a response.
The second clip shows Cog imitating the head motions of a person.
The third clip shows Cog imitating the head motions of a toy cow. The stuffed animal is
detected as a face, and the robot responds to it in the same way that it responds to a
The fourth clip shows Cog imitating a second stuffed animal (Mickey Mouse). The robot
only responds to the face of the toy, not to the motion of the toy.