The Recognition of Material Properties for Vision and Video Communication

MIT2000-02

Progress Report: January 1, 2002—June 30, 2002

Edward Adelson

Project Overview

How can we tell that an object is shiny or translucent or metallic by looking at it? Humans do this effortlessly, but machines cannot. Materials are important to humans and will be important to robots and other machine vision systems. For example, if a domestic cleaning robot finds something white on the kitchen floor, it needs to know whether it is a pile of sugar (use a vacuum cleaner), a smear of cream cheese (use a sponge), or a crumpled paper towel (grasp with fingers).

An object’s appearance depends on its shape, on the optical properties of its surface (the reflectance), the surrounding distribution of light, and the viewing position. All of these causes are combined in a single image. In the case of a chrome-plated object, the image consists of a distorted picture of the world; thus the object looks different every time it is placed in a new setting. Somehow humans are good at determining the "chromeness" that is common to all the chrome images.

Progress Through June 2002

Reflectance Estimation

In the last six months, we developed a theoretical foundation describing the relationship between illumination statistics and the statistics of an image of a surface with a particular reflectance. This allows us to select statistics for reflectance classification, using the framework described in previous reports. We also continued our psychophysical experiments to determine which statistics humans use in order to estimate the reflectance of objects.

Recovering Intrinsic Images from Single Images

Estimating material properties is difficult because an image is the combination of the object’s material properties and other visual characteristics, such as the shape of the objects and their illumination. We are developing a system that can isolate two of these characteristics, the reflectance and shading of the objects in a scene. We use the term shading to refer to the interaction of the objects shape and the illumination of the scene.

Previously, we had developed a system to decompose a color image into an image representing the shading of the scene and an image representing the reflectance of the scene. In the last six months, we have extended that system to gray-scale images. This is necessary because color information alone is not sufficient to distinguish shading changes from reflectance changes. Figure 1 shows an example of such an image. Using color information alone, our system would incorrectly classify the mouth and eyes that have been painted on the pillow as shading. Figure 2 shows the results of just using color. However, by combining gray-scale and color information, the shading and reflectance images are recovered correctly, as shown in Figure 3.

The system works by distinguishing image derivatives caused by a reflectance change from those caused by shading. The shading and reflectance images can then be recovered from the classified derivatives. Derivatives in gray-scale images are classified as shading or reflectance derivatives by using filters that can discriminate patterns in the image that tend to be created by shading from patterns which tend to be created from reflectance changes. These filters capture the different statistics of shading and reflectance and are be found using a machine learning technique based on the AdaBoost algorithm. Figure 4 shows another example.

We have also developed an algorithm to improve the results in areas of the image where it is not clear whether the image changes are caused by a reflectance change or shading. This algorithm propagates information from areas where the correct answer is clear into areas where the correct answer is unclear.

Figure 1 - Both gray-scale and color information is necessary to separate shading changes from reflectance changes in this image.

Shading	Reflectance

Figure 2 - The shading and reflectance images generated by using color alone.

Shading Image	Reflectance Image

Figure 3 - The results obtained using both color and gray-scale information.

Original Image	Shading Image	Reflectance Image

Figure 4 – An example of our system. The image on the left is the input and the two images on the right are the output of the system.

Auditory Material Properties

We have been developing a system to classify materials from the sound of impact. In order to perform its goal the system extracts acoustic properties from the sound and uses these values to determine the values of material properties, such as internal friction. We have been working on the extraction of acoustic properties to determine which are the most relevant and reliable properties to be used in acoustic material classification tasks.

Research Plan for the Next Six Months

Currently, the system for obtaining shading and reflectance images from gray-scale images uses synthetic images to learn the best filters for classifying the image derivatives. In the next six months, we will develop means to train the classifier from real images. In addition, we will research how to isolate other intrinsic characteristics of images.

We will continue building tools for synthesizing sounds corresponding to standard shapes and improve our system for recognizing materials based on their sound.