Research Projects NTT-MIT Research Collaboration: a partnership in the future of communication and computation

The Recognition of Material Properties for Vision and Video Communication

MIT2000-02

Start date: 07/2000

Ted Adelson
MIT AI Lab

Shin'ya Nishida
NTT

Project summary


We are developing methods for recognizing materials in images, and studying how humans perform this task.

Project description


 

Material recognition is a central problem in perception, but it is very different from object recognition. A material like stainless steel or peanut butter has a certain ³look,² yet there is no way to match the image to a standard pattern (as there is with a face or a letter). Surprisingly little is known about how humans recognize materials, or how computers could be taught to recognize them. The problem has many applications. If a domestic robot is to clean the table, it should know the difference between peanut butter, jelly, and bread crumbs. Likewise an autonomous land vehicle should be able to recognize grass, snow, or mud. We are currently approaching the problem in a simplified way, by looking at smooth objects such as spheres, viewed in various lighting conditions. Concepts from texture perception and computer graphics are proving useful.


Demos, movies and other examples


The image of an object depends on a combination of 3-D shape, surface reflectance, illumination distribution, and viewpoint. Can all these factors be disentangled? In particular, how can we tell that this object has a chrome-like surface?
A chrome-plated sphere reflects a distorted picture of the surrounding environment. Every example looks different (at the pixel level) because the environment is different. Traditional template matching is useless for detecting ³chrome-ness.²
A chrome sphere shows us a sharp (but distorted) picture of the environment. A sand-blasted metal sphere shows us a blurry picture of the environment. A matte sphere shows us an extremely blurry picture of the environment. Itıs useful to think of the surface as doing a convolution of the environment map with a blurring function.
We can adapt texture analysis techniques to the problem of classifying materials.
By using two well-chosen statistical features, it is possible to classify spheres fairly well, even when the illumination environment is unknown. More features allow better performance.
The method can be extended to other smooth shapes with known geometry.

The principal investigators


Presentations and posters


Ron Dror, Edward Adelson, and Alan Willsky. The Second International Workshop on Statistical and Computational Theories of Vision at the International Conference on Computer Vision, Vancouver, CA, July 2001.

 

Publications


Ron O. Dror, Edward H. Adelson, and Alan S. Willsky. Surface Reflectance Estimation and Natural Illumination Statistics. Proceedings of the Second International Workshop on Statistical and Computational Theories of Vision at the International Conference on Computer Vision, Vancouver, July 2001.

Ron O. Dror, Edward H. Adelson, and Alan S. Willsky. Estimating surface reflectance properties from images under unknown illumination. Proceedings of the SPIE Volume 4299: Human Vision and Electronic Imaging VI, pp. 231-242. San Jose, January 2001.

Edward H. Adelson. On seeing stuff: the perception of materials by humans and machines. Proceedings of the SPIE Volume 4299: Human Vision and Electronic Imaging VI, pp. 1-12. San Jose, January 2001.

Ron O. Dror, Edward H. Adelson, and Alan S. Willsky. Statistics of Real-World Illumination. To appear in the Proceedings of the Conference on Computer Vision and Pattern Recognition, Hawaii, December 2001. DRAFT - Please do not distribute.

Roland W. Fleming, Ron O. Dror, and Edward H. Adelson. How do Humans Determine Reflectance Properties under Unknown Illumination? To appear in the Proceedings of the Workshop on Identifying Objects Across Variations in Lighting at the Conference on Computer Vision and Pattern Recognition, Hawaii, December 2001. DRAFT - Please do not distribute.

 

Proposals and progress reports


Proposals:

NTT Bi-Annual Progress Report, July to December 2000:

NTT Bi-Annual Progress Report, January to June 2001:

NTT Bi-Annual Progress Report, July to December 2001:

NTT Bi-Annual Progress Report, January to June 2002:

NTT Bi-Annual Progress Report, July to December 2002:

 

For more information