Example-Based Image Synthesis


Progress Report: November 1, 2001–December 31, 2001

William T. Freeman



Project Overview

The capacities of disk drives and computer memories dramatically increase over time. Current capacities make feasible a variety of memory-intensive solutions to hard engineering problems. In this research, we explore fast, memory-intensive algorithms for image and shape synthesis, aiming at two primary applications: video image super-resolution, and improving conventional computer vision shape estimates. The image super-resolution application involves using examples of high and low-resolution images to learn a set of possible mappings from low-resolution to high-resolution. The goal is to generate a plausible high-resolution video sequence from a low-resolution input sequence. The shape reconstruction application involves an analogous scenario: a mapping from low-resolution image to low-resolution shape is learned from an initial computer vision shape reconstruction. Then the example-based method extends the shape estimate to high-resolution, based on the high-resolution image data. A third application, animation synthesis, may be pursued if the appropriate student is found.


Progress Through December 2001

We joined the NTT/MIT collaboration project on November 1, 2001. Progress through December was: forming connections with the collaborators at NTT, bringing an MIT student up-to-speed on the existing super-resolution software and algorithms, and initial studies in extending that still-image work to video sequences.

An important piece of progress in early January was meeting with Dr. Takeshi Ogura, the head of Human Communication Laboratory of NTT, in which the NTT PI , Mr. Onozawa, works. We discussed our common goals and interests. I learned of other interests in common with Mr. Sato of Dr. Ogura’s group: shape reconstruction. Based on that discussion, I have extended the application goals of this project to include our common interest in shape reconstruction.


Research Plan for the Next Six Months

The plans for each of the two primary applications:

Video image super-resolution

--include temporal consistency constraints in the image synthesis, for the special case of unchanging images.

--include motion estimation, to allow temporal consistency constraints to apply to changing images.

--extending the existing still-frame super-resolution algorithm to temporal data in a way that increases the efficiency of the processing. For example, we may extend the notion of patches of image data to 3-dimensional patches of spatio-temporal image data.

Example-based shape reconstruction

I hired a very good MIT undergraduate to join the project. He will be studying the possibilities of example-based shape reconstruction part-time, initially, during the spring, then full-time over the summer. Our goals will be:

--Literature review of the related shape-from-shading and machine learning literature.

--experimentation with synthetically generated shapes and their images

--initial experimentation with example-based shape reconstruction improvement for a conventional computer vision stereo reconstruction algorithm.

It is also possible to use the example-based approach to synthesize human animation that responds to the motions of another in real time. If we find the appropriate student to work on that project, we will explore that aspect of the example-based paradigm. I have an interview next week with an MIT graduate student who may be interested in that work.