Light as Mouse

Human Scale Natural Interaction

Experiment Two Light as Mouse

This is experiment two out of the human-scaled natural interaction series that explores the light and gesture as a navigational mechanism.

As we were exploring human scale input, initially focusing on panning and zooming using the body, we began identifying a need for an even richer interaction. After being inspired by the evocative images made by the light writing technique, we were looking for something that extended the effect and that was capable of capturing multi-dimensional interaction in physical space. After exploring a few directions we decided to give the computer eyes, so that it can see the world the way we do; multi-dimensionally. Surprisingly, we found our solution on the shelf at Best Buy, in a cheap web-cam.

Test 1: Drawing in space

In this first test we wanted to focus on a gesture to draw in space using a single input mechanism. To accomplish this we used a small light in a darkened room that the computer would recognize. As we gestured and drew in the air the effect we got was like waving around a sparkler on the Fourth of July, or long exposure photography of lights.

Our minds immediately began racing with many possible extensions of this. We sketched hardware models for adding stereoscopy, in order to achieve not only the effect of 3D space which occurs when you render from 3D to 2D, but actual z-depth which could be used later to produce our drawings or gestures from any angle or (we imagined) some holographic display. We thought this might be accomplished either with multiple cameras or compositions of mirrors splitting the image for a single camera.

We encountered a number of limitations with the light-based technique which brought us to discussing alternative methods. For example, the low-light requirement would clearly exclude application in most environments, outdoors during daylight hours, or when light is needed for any other task. In our own use, we found that reflective surfaces and lightly colored clothing were problematic. Although we were able to reduce these problems by using diffused blue light in our controlled environment, this method clearly doesn’t lend itself to practical use. We look to infrared as a solution to this. Another limitation that we encountered with light was that it requires a direct line of sight between the sensor and the device. You have to face the camera and perform as one on stage, rather than focusing your attention in the space exclusively. Perhaps this is natural enough, as humans are familiar with the idea of other-sightedness and concerning oneself with what it sees might help to personify the computer. However, one potential solution to this that came up as an alternative was radio-wave.

Test 2: Manipulating in space

To really appreciate what we had accomplished in Test 1 we needed to do more than draw in human scale. For this second test we wanted to focus on using gesture to manipulate the objects in space that we drew. To accomplish this we needed a second hand held light that we use to move and scale our drawings similar to iPhone’s pinch and stretch gesture, only in human scale.

While this was merely a proof of concept, or icing on the cake, the implications are valuable. Sight enabled technology has the ability to deal with any number of focal points, can simultaneously receive input from multiple people independently, or combine these to form gestures in a particular context.

Here, too, our minds began to race with possibilities, spring-boarding from our earlier concepts. Assuming we arrive at a method for drawing three dimensionally in space, we will need an equally natural method for manipulating and viewing our work. This brings to mind some combination of Star Trek’s holodeck (or the white room where Neo suits up before reentering the Matrix) and microfilm as a navigation model. This seems like an ideal way to explore models of our world and universe, as well as theoretical systems and metaphysical constructs.

Envisioning real-world applications

Although the concept we ultimately arrived at is still a ways off, we envision a digital whiteboard that engulfs the space all around us. We would need some new display technology, approaching science fiction ideas of holography, in addition to the exercises we’ve explored and described here. Rather than shaking our heads and abandoning such things as fantasy, we have strived and grasped at achieving whatever we can of this. By doing so we learned more about the reality of such things better than we could have otherwise known. With products like the Nintendo Wii and Apple iPhone taking the market by storm and the impending release of even more fantastic products like Microsoft Surface, technology companies need to start thinking about the changes that are coming to the human/computer experience; ones which promise brighter, more natural interaction where the paradigm isn’t so much device or object centered, but at human scale, in the space around us.