A quick note I wanted to make is about the difference between Augmented Reality and Spatial Augmented Reality aka Projection Mapping. The first is concerned with inserting objects or information into a real context with whatever means necessary. The second is a neat subset in the whole mixed reality field concerned with augmenting real surfaces with new properties.

Optix-based SAR setup from the University of South Australia at ISMAR 2013

In Spatial Augmented Reality or SAR, the real scene has some diffuse dummy objects made from Styrofoam or similar onto which, broadly speaking, a new BRDF is projected. That’s right, SAR changes the appearance of real objects by computing a new image which is usually cast onto the object with a projector. Several problems have to be solved within SAR:

• Visibility is an issue when projecting images onto non-planar surfaces. Most SAR setups therefore use several projectors to cover most of the surfaces.
• Projecting images onto non-planar surfaces, for instance a diffuse ball which should appear glossy, requires the source image to be distorted so that after the projection real and virtual distortion cancel each other out. Here one will sooner rather than later run into the same parameterization issues as with texture mapping in general.
• Using several projectors to cover for the visibility problem introduces overlapping artifacts. The worst issues are visible in highly distorted areas. Even small differences in the calibration of the projectors will produce very bad moiré patterns.
• Since scene and projectors are tied together, it is now the user which has to be tracked instead of an anchor in the scene to simulate view-dependent effects such as glossy reflections.
• Projecting an image onto diffuse surfaces will simply by its nature diffusely reflect on its surrounding, automagically creating unwanted global illumination effects such as diffuse secondary bounces. These have to be countered in the simulation which produces the projected images. There’s a neat paper for that.

SAR is an interesting field when relighting real world objects to see them with different materials. In the video above for instance, one can preview a car in different colors before buying the real thing. A recent piece on Wired showed off an overhead projection into a sandbox tracked by a Kinect, where different heights in the sand will correspond to different layers of a landscape. Nevertheless, the next posts will not be on SAR, but I didn’t want to skip over this cool field.