NTT-MIT: MIT9904-14 Image Based Synthetic Aperture Rendering

Research Projects

NTT-MIT Research Collaboration: a partnership in the future of communication and computation

Image Based Synthetic Aperture Rendering

MIT9904-14

Start date: 07/99

Leonard McMillan and Julie Dorsey
MIT LCS

Hiroshi Murase
NTT

Project summary

The goal of the Synthetic-Aperture Camera Project is to develop new image-based representations for use in computer graphics. Our image-based representations are easy to acquire, yet they have similar flexibility and greater realism than traditional computer graphics models.

Project description

Our representation uses a two-dimensional array of images to form a four-dimensional light field of any given scene. From this light field, we can compute novel new views of the captured scene with greater flexibility than a real camera and higher quality than a three-dimensional computer graphics model. Our work addresses the following problems:

Light Field Acquisition Devices
Light Field Processing and Rendering
Light Field Viewing Algorithms
Light Field Display Devices

The ultimate goal of this project is to construct a randomly accessible two-dimensional camera array capable of synthesizing novel views of dynamic scenes.

Research Areas

Synthetic Aperture Camera Array

We are designing a two-dimensional camera array for acquiring and processing light fields in real time. In fact, the entire camera array can be regarded as a dynamic light field data structure. The pixels of each camera in the array are randomly accessible by the host computer system, where the array is mapped as a block of memory. These dynamic light fields are accessed "on-the-fly" to generate the desired images. The memory-like interface used by our camera array significantly reduces latency in the image generation process and makes the most effective use of available bandwidth. The camera array will employ a modular design consisting of a motherboard, sensor pods that support a range of imagers, and a PCI interface to the host, optimized for high throughput.

Dynamically Reparameterized Light Fields

We are also developing new processing techniques for light fields. First, we have added the ability to vary the apparent focus and depth-of-field within a light field using intuitive camera-like controls such as a variable aperture and focus ring. However, unlike a tradition camera, we allow for more general and flexible focal surfaces than the typical focal plane. Our techniques are based on the dynamic reparameterization of the light-field data structure in order to optimize and exhibit greater control over the image reconstruction process. Our reparameterization methods operate independent of scene geometry; we do not need to recover actual or approximate geometry of the scene for focusing. Our techniques also allow for the use of multiple focal surfaces within a single image rendering.

Light Fields on the Cheap

We are also developing low-cost devices for acquiring light fields. Not only does this effort provide direct benefits for our research, but it also provides many new opportunities. The small size and portability of this system allows us to easily acquire outdoor light fields of natural scenes, and it is also currently our fastest acquisition device.

In order to minimize cost we chose to use an off-the-shelf flatbed scanner as the digitizing element, in combination with a plastic lens array. However, intrinsic properties of scanner limit their use for acquiring images. We have developed a combination of low-cost optics and correction software to overcome these limitations. Currently we compensate for the following: color balance, radial distortion, camera calibration, noise removal, and sub-image segmentation.

Passive Autostereoscopic Light Field Displays

We are also working on passive autostereoscopic devices for the direct viewing of light fields. Our display allows multiple viewers to see a light field stereoscopically from a wide viewing range and it requires no tracking. The key component of our viewing system is a prefabricated lens-array sheet with a thickness equal to the focal length of each lenslet. This allows us to make a viewer-dependent image with resolution equal to the size of each lenslet in the array. Currently, we are using a lens array in a hex pattern, where each lens has a diameter of approx. 0.1" and a focal length of 0.12". Under the entire sheet, we place a large composite image of smaller hexagonal image. Each small image is a picture of a light field taken from some point in space. When viewed under the lenses, a different pixel in the small image is seen for each view direction. In addition, each lens performs interpolation, due to the lens optics. When the entire array is viewed, each lens takes on the role of a single pixel in the final autostereoscopic image.

Demos, movies and other examples

Dynamically Reparameterized Light Fields Animations

MovingToysSnap.jpg (7204 bytes)
animations/MovingToys.avi (320x240 - 11.2M)

These animation show how we can freely move around our dynamically reparameterized light fields. Throughout the MovingToys animation, we change the focal plane, camera orientation, and camera position. In the MovingTree animation, we vary the position and orientation of the camera. Because there is only one focal plane, a considerable amount of ghosting appears in these animations. This ghosting can be eliminated by 1) using a focal surface that conforms to the geometry of the scene, 2) using multiple focal planes that pass through the key features of the scene, 3) using a larger aperture such that ghosting turns into blur, or 4) sampling the camera surface at a higher resolution (using more cameras). The black regions occur when there is no ray in available in the ray database.

MovingTreeSnap.jpg (5925 bytes)
animations/MovingTree.avi (320x240 - 4.62M)