Ph.D. Research

Open Light 3D Scanner

Figure: A sample scan of a bust.

With cameras and projectors continually decreasing in price, low-cost 3D scanners using off the shelf components are now within the reach of researchers and hobbyists with a modest budget. Douglas Lanman and Gabriel Taubin over at Brown University recently taught a course at SIGGRAPH 2009 and SIGGRAPH Asia 2009 on how to build your very own 3D scanner using Structured Light (SL).

This SL technique is derived from "A Novel Method for Structured Light System Calibration" by Zhang, S. & Huang, P. The SL implementation was also modified to work with a Point Grey Firefly MV camera in addition to standard web-cameras. There are still many components that need to be added onto the current implementation, such as basic smoothing and gap filling operations, global alignment, creating water tight meshes from the point cloud, and additional support for more cameras. Check out the LOP project below (one of many things we hope to add to the SL implementation). We will be releasing all of our updates and modifications to the code as an open source project! The link below points to the project which is currently hosted on Google Code.

Open Light

Master's Research

Spatial Augmented Reality Research

Projecting digital content onto physical 3D surfaces has generated a great deal of excitement in the new media, performing arts, marketing, gaming and research communities, to name a few. From projecting onto building facades in brilliant Son et. Lumiere (sound and light) shows to bringing everyday objects to life, people have been expanding the use of projectors beyond traditional flat display surfaces. This medium, formally known to researchers as Spatially Augmented Reality (SAR) and dubbed Projection Mapping by many in the art community, uses projectors as dynamic light sources. With an off the shelf camera and a projector, we can animate surfaces and transform the way we perceive and interact with our environment. Listed below are several projects that demonstrate our initial research efforts and applications in SAR.
Augmented Engineering
Working with Brett Jones


Build Your World and Play in It: Interacting with Surface Particles on Complex Objects

Figure: Playing miniature golf on a set of wooden blocks.

Paper to appear in ISMAR 2010: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality

Abstract:
"We explore interacting with everyday objects by representing content as interactive surface particles. Users can build their own physical world, map virtual content onto their physical construction and play directly with the surface using a stylus. A surface particle representation allows programmed content to be created independent of the display object and to be reused on many surfaces. We demonstrated this idea through a projector-camera system that acquires the object geometry and enables direct interaction through an IR tracked stylus. We present three motivating example applications, each displayed on three example surfaces. We discuss a set of interaction techniques that show possible avenues for structuring interaction on complicated everyday objects, such as Surface Adaptive GUIs for menu selection. Through a preliminary informal evaluation and interviews with end users, we demonstrate the potential of interacting with surface particles and identify improvements necessary to make this interaction practical on everyday surfaces."

Video
SIE.pdf

Interactive Dance Performance

Figure: Dancer influences the behavior of surface particle sprites.

This live performance uses data generated from accelerometers embedded in dancers to drive the behavior of interactive surface particles. With the gestural recognition expertise of Mary Pietrowicz, we were able to map dance gestures to unique sprite movements. For example, when a dancer starts jumping, the sprites begin to increase their movement and their size. The live performance uses the Surface Interaction Engine (published at ISMAR 2010) which includes an integrated structured light scanner and IR tracking.

Video

Locally Optimal Projection

Figure: Locally Optimal Projection

With structured light and many other 3D scanners, you can get noisy point sets with artifacts that are slightly off the surface. We recently re-implemented the Locally Optimal Projection operator (LOP) published at SIGGRAPH 2007 and SIGGRAPH Asia 2009 for surface approximation from point-set data and got good results applying it to the scans from our SL implementation. There are some nice properties of the operator that provide us with a good approximation to the surface even with noise and outliers.

The idea is, given an unorganized point set P, LOP defines a set of projected points that approximately represents the surface. Here is a quick snapshot of the results applied to a noisy sphere point set:


Video

The Projection Mapping Toolkit

Figure: Grid lines projected on a physical box.

The Projection Mapping Toolkit was created to help understand how to improve the user experience of SAR content authoring and the overall user interaction of SAR applications, particularly for new media and performance artists. While researchers such as Ramesh Raskar, Oliver Bimber and J.C. Lee among many otheres have demonstrated amazing techniques in projector calibration and image projection, there has been little work that explores the bottlenecks encountered when this technology is utilized by its end users. Traditional video editing and computer graphics workflows are not designed for animated texturing and easily matching physical and digital models. Lookout for a video in the near future that demonstrates the PMT's underlying technology including an interactive preview plug-in for Maya that allows users to dynamically change how animated textures are mapped to the physical surface. The PMT also uses the Panda3D Game Engine for playing the animated video textures on the display surface and the calibration process is the same as the one in Ramesh Raskar's Shader Lamps.

Astral Convertible

Figure: Dancers performing around the Astral Towers.

The UIUC Dance Department was recently tasked with reimagining Astral Convertible, a famous dance performance originally choreographed by Trisha Brown in 1989. The title of the dance is both a reference to the car parts used in the construction of the original set and a play on the name of a furniture store in New York. The original set pieces included self-powered light and sound tower units. These towers were redesigned for the Feb 2010 piece and outfitted with sensors to pick up light reflected from the dancer's costumers to trigger lighting and sound changes. Motion sensors embedded in the dancer's costumers were used to communicate movement data with several remote servers. These servers used machine learning algorithms to identify dance gestures as well as patterns such as synchronicity or disparateness.

With the help of Alex Betts from the National Center for Supercomputing Applications, we created graphical projections mapped to the tower set pieces. The projections would change and reflect to the dancers movements and patterns creating an interactive theatrical environment that responds to a complex network of actions from the performers.

The Magic Flute

Figure: A digital virtual character projected on a physical bust.

The Magic Flute is an opera in two acts composed in 1791 by Wolfgang A. Mozart that includes both singing and spoken dialogue. A particularly demanding movement in the piece, "The Vengence of Hell Boils in my Heart" sung by The Queen of the Night character, reaches a high F6 and is considered rare in opera making it difficult to cast students in the local performance at the University of Illinois. As a result, we were contacted by production members of the opera and asked to help create an interactive virtual character to portray The Queen of the Night. The video link below shows the projected digital character on a physcial bust singing along to a 1984 version of the performance.
Video

Automatic Projector Calibration

Figure: A series of Gray-code structured light patterns are projected onto the box.

This technique is explained extensively by Johnny Chung Lee , et. al, and is a re-implementation of Automatic Projector Calibration with Embedded Light Sensors. The video demonstrates how we calibrate a digital model of a box to the a physical box using embedded light sensors. The projector displays a series of Gray-code binary patterns, which photosensors embedded into the box receive and decode. With this information, we can determine the positions of the sensors in the projectors frame of reference. With enough sensors, we can determine the parameters of the projector (intrinsics) along with the rotation and orientation (extrinsics) of the box. A 3D game engine is used to render a 3D model of the box using the intrinsic and extrinsic parameters of the projector. An animated texture is applied to the 3D model and warped so that it accurately fits to the physical projection surface.
Video

Undergraduate Research

UbiFriends

UbiFriends is a scalable mobile social networking application that leverages aggregate location data to infer social ties between users. UbiFriends approximates social distance by leveraging the fact that users who spend time together or frequent the same locations are more likely to be friends. Traditional social networking sites such as Facebook and MySpace utilize friendship suggestion tools to recommend existing and potential social ties between users. Friendships are suggested based upon correlations between the user's profiles, similarities in position within the network and a number of other factors. While physical proximity is not a replacement for more complex social distance calculations, UbiFriends demonstrates that location data can be used as a major component in successful network analysis.

Groupware in Creativity

Creativity as modeled using the recluse genius paradigm fails to recognize the necessity and value of the collaborative aspects involved. Enhancing the creative output of teams will become more and more necessary as the complexity of problems increases; this is especially true within the design domain. This work explores the group processes and accompanying bottlenecks related to design work. Individuals and teams with experience in collocated design settings were studied to extract principles and best practices for group design work. Based on the findings, a set of design implications were extracted and IdeaSpaces, an early prototype of a collaborative system was built to implement those conclusions. The tool attempts to model the highly effective ways that design teams work in a collocated environment while also addressing some of the bottlenecks they face.
GroupwareCreativity.pdf

ClubDev

A new interaction framework for collaborating in multiple display environments (MDEs). The framework allows users to share task information across displays via off-the-shelf applications, to perform near simultaneous input between applications for focused problem solving, mediate which applications are shared and when, and to place information on shared displays for discussion and reflection. This project was done in collaboration with Jacob Biehl, William Baker, and Brian Bailey.
ClubDev.pdf