Light Field Parameterization
We take a non-traditional approach to computer graphics modeling and rendering, in which a scene is represented by a collection of images rather than the geometry and surface properties used in typical computer graphics. Essentially, we treat a collection of images as a database of rays. New views can be constructed from this database on a ray-by-ray basis by selecting the closest ray to each desired ray.
Project Goals
Approach
Real-time Acquisition
Low Cost Acquisition
Display Technology
MIT9904-14 PIs: Prof. Leonard McMillan (MIT LCS), Prof. Julie Dorsey (MIT LCS), and Dr. Hiroshi Murase (NTT)
The primary focus of our research effort is to develop technology to create virtual experiences that will approach the fidelity of the real world. In the future, such technologies will have a dramatic impact on the way we work and play. They will enable new forms of commerce, bring together individuals separated by large distances, and provide us with new forms of entertainment.
Our dynamically reparameterized light field representation allows us to synthesize
images with photographic effects such as variable focus and depth-of-field. Depth-
of-field effects are created by varying the extent of the reconstruction filters used
on the camera surface.
A variable focal length can be simulated by varying the focal plane used in the
reconstruction process. In a synthetic aperture camera both the aperture and focal-
length settings can be varied from pixel to pixel. The allows effects that are
impossible with a traditional camera.
Ultimately we intend to
create a device for capturing
and processing dynamically
reparameterized light fields
in real-time. We call this
device a synthetic aperture
camera array. It is composed
of a two-dimensional array of
randomly accessible image
sensors that memory-mapped
in the address space of a host
processor. Such a system will
allow images to be synthesized
from a wide range of virtual camera positions in real-time. We plan to support
multiple simultaneous video streams to support stereoscopic display as well as
multiple viewers.
A high-level block diagram of our proposed system is shown above. The camera’s
host interface will be an industry standard personal computer bus. The camera array
will be constructed from modular sensor units mounted on a common motherboard.
The addressing of sensor modules will be interleaved in order to maximize
the communication bandwidth between the image sensors and the host
computer. Each sensor
pod contains a CMOS
image sensor, buffer
memory, and glue
logic. The multi-
frame buffer
memory is used for
two functions. It
is used to store
information for
noise cancellation,
and it allows the
host to access
image rays asynch-
ronous to the image
scanning process.
This modular
design approach
will allow us to upgrade to higher resolution sensors as they become available.
We have also prototyped two low-
cost devices for acquiring light fields.
We have developed two acquisition
systems for acquiring light fields of
static scenes. The first uses a robotic
XY-platform to move a digital camera.
This system allows us to explore the
trade-offs between camera spacing and
resolution in order to estimate the per-
formance of our camera array. This
system uses a precision image sensor,
precision optics, and a motion platform with a travel distance of approximately one
meter squares. It can acquire a 16 by 16 image light field in under 20 minutes, and
it cost approximately $10,000 US to construct. Our second system is based on an
off-the-shelf flat bed scanner, and an array of plastic lenses. We have modified the
scanner to operate off of battery power so that this system can be taken out into the
field to acquire images. Additional processing is required to correct for shortcoming
in the image sensor and low cost optics. None the less, the system can acquire an 8
by 12 image light field in under 3 minutes, and cost under $100.
We have also developed techniques for direct auto-
stereoscopic viewing of our light fields. These methods
are similar to various lenticular techniques for viewing
stereo images. Our synthetic aperture generation ap-
proach provides much greater flexibility than tradition optical approaches. In particular
it can overcome many limitations such a focus control and skewed frustums. We have
demonstrated viewers with true parallax (both horizontal and vertical), and variable
controlled focus. Our displays have nearly all of the desired properties of holograms,
yet they are true color and viewable
under normal lights. Furthermore,
the technology is easily adaptable
to the display of dynamic 3-D
images. Currently we are only
limited by the resolution of flat
panel displays.
The image on the left, when viewed through a hexagonal lens array, can be seen as a three-dimensional image of a flower. It can be simul-taneously seen by multiple viewers. It was computed from a dy-namically re-parameterized light
field, which allows us to
precisely control the focus at all viewing angles. The inset provides a magnified view of the image.
Back to first slide | View graphic version |