We have made rapid progress on a number of problems related to the goals of the Variable Viewpoint Reality project:

• We have developed a number of basic algorithms for 3D reconstruction. One approach is designed to work in real time on many cameras. Another is a bit slower, but is designed to yield higher quality results. A third attempts to find the arm, leg and body positions of a human being from one or multiple camera views.

• We have designed and setup a multiple camera systems for acquiring data in real-time. This system was designed to be flexible and to work indoors. Right now we have 12 cameras working in synchrony. We would like to setup more.

• We have acquired a great deal of multi-camera data. This is allowing us to test our algorithms and to develop new ideas.

• In collaboration with students working on another project we have been observing outdoor activities. This system provides coarse tracking information of multiple people and cars. The system can also recognize simple activities.

We believe that we have begun to make rapid progress in a number of related areas. In order to continue at this pace, and more importantly in order to build a more reliable and usable system, I would like to request a moderate increase of funding. With this support we will hire an engineer who would be responsible for software and hardware engineering. This would allow for a more rapid transition between research advances and working systems.

Timeline for the future of the VVR project:

Demonstrate real-time 3D reconstruction and visualization from 12 cameras distributed around an indoor 3 meter cube. This space will be large enough for a single person to move and take actions. (We believe that this system is the first step along the way toward a more general purpose, larger scale system.)

We will extend the August system to include additional functionality:

• Tracking of people using articulated body models.

• Improved texture mapping of body models.

• First results on action interpretation.

Demonstrate real-time 3D reconstruction from 20-30 cameras around a larger area (approximately 10 meter). In addition to its larger scale, this system will provide a number of additional facilities:

• Multi-person 3D reconstruction

• Tracking of identified people

• Finding articulated body models for people.

• An early demonstration of action interpretation

(At this point the NTT sponsored portion of the VVR project may come to an end. We would hope to continue, perhaps at a larger scale, with additional graduate students and additional hardware.)

Demonstrate a large scale system which would include additional cameras and additional capabilities. Much of the effort in this year will focus on integration of the above mentioned capabilities. Our research emphasis will shift toward the interpretation of action.

Demonstrate an outdoor system operating at an actual sporting event. I must emphasize that the main difficulty in achieving outdoor operation is engineering and cost. The current level of funding for this project allows me to support 3 students and to purchase a modest amount of computer hardware. To move this system outside would require the intervention of a number of skilled engineers and additional hardware. NTT could be helpful in both of these areas.