The past thirty years have seen significant progress in the field of computer graphics, particularly in the area of rendering. However, the creation of realistic models is nearly as tedious today as it was 30 years ago, and many types of complex materials simply cannot be represented with today's graphics systems. To address these problems, we are developing a new 3D modeling system based on the metaphor of sculpting real materials. We believe that by combining haptic output devices, stereoscopic displays, physically-based surface models, user-guided simulations, and newly developed surface representations it will be possible to approach the feel, naturalness, and flexibility of interacting with materials such as stone, wood, metal, and plastic. This research also serves as a platform for studying the next generation of user interfaces, sensory fusion, and material representations. This work should find application in a variety of fields ranging from computer-aided design to entertainment.

Highly detailed geometric models are necessary to satisfy a growing expectation for realism in computer graphics. Within traditional modeling systems, complex models are created by applying a variety of modeling operations such as CSG and freeform deformations to a vast array of geometric primitives. Intricate meshes are also obtained by scanning physical objects using range scanning systems. A notable property of the new acquisition techniques is their ability to capture fine surface detail. These developments have made multi-million polygon models widely available and offer new opportunities to modelers and animators in the CAD and entertainment industries.

The goal of this work is to develop a new data structure, the volumetric surface, which captures attractive properties of both surfaces and volumes, and offers a convenient way of representing material properties of complex models. More specifically, the new approach retains the efficient sampling offered by surfaces, but also supports powerful volumetric operators, such as interactive operations for adding and removing material. Properties of materials, such as brittleness and distance to the surface, control the way materials respond to these operators, providing a higher level alternative to existing material descriptions in interactive modeling systems. Finally, a haptic interface provides a natural way to interact with these material representations.

In the past six months, we have expanded the volumetric surface data structure to accommodate several high-speed physically-based simulations. Realism is enhanced by a wide variety of material properties that affect the simulation parameters. New volumetric properties can be modified to vary the apparent softness and malleability of materials and to provide thresholds at which materials crack under excessive stress.

One library of functions allows for the simple removal of material from a model. Since each material has its own set of properties, some parts of the model appear to be more resilient than others. For instance, plaster feels softer and sculpts more easily than brick or mortar. Figure 1 shows an example of a complex façade composed of a variety of materials that have been modified using this library of sculpting operations.

A second library of functions relies on the use of finite element modeling to simulate malleable materials ranging from soft metals to wet mortar to gelatin. All of these functions are integrated with the haptic system to provide a flexible interface with realistic response forces that aid in the intuitive understanding of model structure. Figure 2 shows a brick wall that has been modified using this library of operations.

The allow the user to easily experiment with the rich library of material parameters, we have developed a simple scripting language that allows any model to be quickly loaded into the system with a customized set of materials properties.

Now that a strong foundation for physical modeling has been established, we plan to expand the library of modeling functions to include tools for localized melting and weathering, as well as more traditional modeling tools for drilling, chiseling, or painting. Using the established simulation framework, we will be able to model gravitational forces and to simulate the effects of collisions between the various components of a models. We also plan to expand the list of supported material properties to accommodate the representation of more complex materials and to allow materials to change behavior as they are heated or cooled.

Other future developments will focus on establishing multiple levels of detail for complex models. This will allow many of the physically-based simulations to run faster while providing a higher level of detail to support fine features such as bumps, scratches and small fractures.

Additional research will experiment with alternative display technologies, including stereoscopic plasma displays, which can provide the user with a high-resolution three-dimensional interface. Taken together, the haptic interface and three-dimensional display technologies can provide the user with a fully-immersive experience with intuitive real-world analogies.