[tweetmeme]This is the third installment of my blog on 3d. The first part outlined the gist of the story, The second part was about the aesthetics of 3d. This installment deals with the use of 3d to convey a ‘conceptual model’ for a user interface. In this article I’ll identify different approaches to building 3d user interfaces and ask the question are they useful?, usable?, and/or desirable?. I’ve provided tons of examples partly to make the point there is a ton of activity in this space but not a great deal of discussion around it and of course a terrible lack of tools to create it. But of course the real reason I’m writing this is ‘Just saying ‘Z-space’ raises your IQ 10 points.’
When talking about 3d conceptual models of different user experiences the litmus test of ‘usefulness’ is that the third dimension the ‘Z’ is conveying some additional data or meaning that a 2d representation of the same thing would not have. Presumably this additional information is of significant value, and relevant in helping people understand it.
When it comes to ‘usability’ of 3d UX, it’s a big topic. When using 2d input devices, the design choice of free-form cameras v constrained cameras can have a big effect on the usability of 3d user interfaces. By free-form, I refer to giving the user the controls to point the camera in any direction. This is useful in 3d environments (interiors), but comes with the baggage that the user must now construct a mental map of where everything in the space is relative to any other point in space.
Constrained looking of the camera on the other hand limits the ability for the user to look in certain directions. This ‘Cone’ of visibility usually has the advantage of keeping the user focused where they need to be and stops them becoming disoriented. This may not sound significant but the difference between these approaches makes a massive difference to ‘usability’ of 3d interfaces.
Similarly enabling the user control of free or constrained movement of the camera in space has a big impact on usability too. Here’s some useful advice:
If you give the user free form ability to point the camera in any direction – think twice. If you decide its for the best then constrain their movement of the camera through space, the easiest constraint is just giving them X and Y movement which is familiar to all users. If you give them the ability to move the camera through Z space then constrain where they can point the camera.
One more: If you’re building an interactive 3d thing don’t choose ‘both constraints’, (i.e.: no ability to move the camera or move the viewpoint) this defeats the entire point and btw you should be in the movie industry.
If you are working with exteriors looking at objects from the outside, its sometimes useful to constrain the cameras target to the center of the object and then giving the user the freedom to ‘orbit’ the object as they see fit. This is often used in car configuration UI’s. Being able to zoom in (rather than moving the camera) gives the user the ability to see more detail without becoming disoriented. Other common approaches here are to allow the camera to move in and out of close ups, and when in the close-up providing a button to escape from this mode.
This is called putting the camera on ‘rails’ for transitions from one place in space to another allows the user to specify a different item to orbit around, perhaps a different car in the showroom. Even if you have rails, and constraints in full effect, giving the users camera ‘a few degrees’ of freedom is a nice touch often this is accomplished by mapping the position of the cursor in the scene to move the camera a few degrees as it moves around the screen.
This portfolio site shows good use of animation around a simple hierarchy. The camera utilizes limited degrees of freedom for camera movement, but keeps the impression of control over the user. 2d text instructional cues on the 3d objects also help the usability.
Full six degrees of freedom, (up, down, left, right, Forward, backwards, pitch left/right, yaw left right,, rotate left, rotate right) is rarely useful, turning off as many of these as possible generally improves usability.
That’s not to disregard ‘desirability’, I said about aesthetics in the previous article, sometimes just because it looks great is enough to make a design sufficiently novel, different, interesting, at least in the short term. Desirability in a crowded market can be the stand out best place to innovate.
2d Cards in 3d space
Take an inventory of all the ‘digital’ things we have in our lives and the list can get pretty long, for example; Songs, documents, spreadsheets, messages, photos, movies, contacts, conversations, bookmarked locations on the Internet, blogs, appointments, applications, places of interest, lists of things to do. Before our digital lifestyles these things had physical precedents, type is 2d, illustrations and photos are 2d, audio and video is represented by 2d box covers, calendars, books, business cards essentially are just collections of 2d sheets.
No surprise then that our first category of 3d UI’s simply put these 2d digital things onto ‘cards’ and then put the cards into some kind of metaphorical, logical or ‘real’ spatial environment. The cards themselves add no dimensional data but presumably their position in three-dimensional space conveys some additional meaning.
I’m going to call this 2d ‘Cards’ UI and there is allot of it out there. Cards don’t literally have to look like cards they simply are using 2d text or images to convey the content.
I’ve broken it down into four reasonably distinct uses of cards, although this taxonomy is just for discussion rather than an attempt to be authoritative on the topic:
- Cards in metaphorical space
- Cards in logical space
- Cards on geometric space
- Cards in ‘real’ space
The second category of three-dimensional UI I’ve seen uses data that is naturally 3d, for example; models of physical objects, buildings, maps, bodies, space, environments are of course the simplest to understand 3d data types. They are relatively abundant on the web and already part of our mainstream digital lives. Second life, Virtual and Google Earth, Google Street view, and almost all video games use 3d effectively in this capacity.
Here the creative discussion is about creating immersive environments that need to be explored by the user because the data is naturally spatial and best understood from multiple perspectives and vantage points, or the creative team want ‘exploration’ to be part of the core reason for the user experience.
I’ll call this 3d ‘objects’ UI and like above I think there’s at least three important variants of it.
- 3d objects in Metaphorical or virtual space
- 3d objects in logical space
- 3d objects in real space (Augmented reality)
2d Cards
1. 2d Cards in metaphorical space
The basic idea in this example from Microsoft Research (1997) was testing people’s ability to organize and remember activities when mapped to a metaphorical physical environment. The idea was to tap into our spatial memory and perception as the ‘added value of 3d’. Skewing the windows enabled you to see more windows at once, but this two can be done effectively in 2d without the ugly skewing distortion that’s occurring below.
Variants of this approach have continued for years substituting walls and corridors for cities, art galleries, hillsides, abstract environments of all kinds you name it. As the years have gone by the windows look better, the environments are cooler, but fundamentally this isn’t going anywhere.
Put yourself in the user shoes for a moment and consider about the amount of ‘organization’ it takes to create a custom environment where activities or windows or collections of digital things are positioned in an environment. Then think about the extra time, (and controls) it takes for me to navigate around in this space and basically the UI is starting to obstruct the user’s primary task, its getting in the way of whatever I was trying to do in the first place.
This organizational burden, place on the user demands they pay attention to the positioning of various things in space. Some research studies have proven that user recall has improved when things where spatially presented in environments like the one above, however perhaps they ignore the added time it takes for people to position, navigate and conceive the appropriate organization of these things.
The logical conclusion of this approach 3dna’s Desktop application for Windows gives you a complete spatial environment filled with lots of 2d pictures as links to various windows you have open, organized into metaphorical objects and posters.
The marketing banter for this company talks about the ‘Fun’ of it all, which I suppose if you like this kind of thing is about the only thing you can say about it.
Useful – So as far as ‘usefulness’ this approach is fraught with the organizational overhead management, and raises questions about multi-tasking and task switching that are irreconcilable with physical space. (Alt tab, or Expose are fine ways to move from task to task, walking across a room gets kind of old once the metaphor is established.). Other questions mostly around the scalability of solutions like this (how many windows can it handle before the metaphor breaks down).
Usable – Usability here is a mixed bag, full camera movement systems, and manipulating things around in space obviously put a burden on the usability of these systems, but at least the metaphors are ‘understandable’ which gives them familiarity to less experience users.
Desirable – Best case the desirability of this technique is limited to niche appeal, technology history is littered with companies big and small (Microsoft Bob, General Magic) who attempted to extend the computing metaphor into a more literal spatial environment. If you follow this approach no doubt a small community of hardcore fans will love you playing house, but mainstream consumers don’t’ really need this.
2. 2d Cards in logical space
Three dimensionally sorted stacks of 2d cards have quickly become commonplace in adding 3d to our user experiences, on desktop, mobile and console devices. Can you count the number of carousels, walls, curved walls, piles, boxes that have shown up in our UI’s recently?
A toolkit above from Fotoviewer provides a variety of different card arrangements for images. All these owe a great deal to ‘Coverflow’ ,which was originally developed by an artist, Andrew Coulter Enright and was purchased by apple in 2006.
Bing’s new visual search using shelves
Clever use of slight overlap and perspective convergence gives us the layout benefits of getting more items on the screen, than a standard 2d Grid. This is straight forward basic use of 3d, no conceptual headaches, simple to understand, takes advantage of 3d layout, and is visually ‘novel’ something Microsoft needs to help it convince users its different/better than Google. Also in the same vein is Search me. And Safari’s 4 Top site view. Also worth a mention is Cool Iris; and Sun’s dead Looking Glass project if your interested in the history.
AT&T’s Pogo Browser
2d Cards, showing search history. Safari 4 uses Coverflow for the same functionality.
Useful – So is this Useful? They can be more efficient in showing more content than 2d views if careful attention is paid to layout. The third dimension for these cards is used to convey no additional information than a 2d sorted list, but distance away from the camera typically conveys some kind of sort order. On balance somewhat useful for interaction designers where visual searching through limited numbers of things is required.
Usable – From a usable perspective, these systems typically don’t rely on any camera movement beyond what we are already familiar with in 2d UI, (Zooming, X and Y translation.). So there’s no inherent complexity being added to these kinds of user experiences.
Desirable – From a desirability standpoint, the swift proliferation and success of Coverflow, carousels and systems like Cool iris is hard to deny in making our commonplace UI mechanisms slightly slicker.
There’s a community of information designers and developers out there working in scientific fields and on complex data visualization. They are building tools that enable very large data sets to be processed and visualized in flexible ways.
Some of the work of this group is fascinating; ground breaking stuff, and visually amazing. Every once in a while someone dabbles with 3d to ‘help’ visualize the complexity. The results are almost always spectacular failure of clarity. The raises an important issue about 3d UX of all kinds and that is, static representations of these networks are truly an unfair way to judge them. Unlike static 2d posters, which are designed, for comprehension from a static point in space and time, 3d becomes infinitely more valuable when it uses time (animation) and interactive movement around the 3d space to understand its topology.
Useful – Generally, Usefulness, I’d argue that these super complex visualizations of nodes, networks, social hierarchies and systems are ‘best case’ marginally valuable to a small group of IT professionals or social scientists.
Usable – From the perspective of Usability, they are disasters for all the reasons I outlined in the camera notes at the beginning – they literally require full freedom camera control. (Ed. common I’m being provocative here… someone send me a good example please).
Desirable – There is some aesthetic desirability to this approach for designers. I can’t help seeing these kinds of things and like a moth to flame, drawn in by the promise of ‘understanding’ data in a more visceral and immersive level. I think however on balance most consumers would look at these kinds of experiences and blanche.
3. 2d Cards on geometric space
Somewhere between, cards in metaphorical space and logical space is putting cards on geometric primitives which are not really 3d objects because there’s no added information to the additional dimensions this brings. Personally I hope to never see another 3d browser application that maps webpage’s or content onto cubes.
I’ve seen so many examples of this recently, There’s really nothing to say here but ‘stop it’. Why are you doing this???
There’s always some exception’s however. Here’s Absolut world a Paper vision extravaganza this physics powered UX, shows a dynamic site that has great music/art direction. Opening the cube’s is a really nice touch revealing nice shiny horizon/dioramas. Because the geometric cubes are objects not environments this approach to 3d UX is easy to understand and lots of fun to play with. I could even see this inspiring more productivity related scenarios.
Apple filed a bunch of patents with these schematics showing a 3d desktop last year. This patent is a bit mysterious but you can see again the idea that you are looking at the inside of a box with a floor, walls, ceiling and background.. 3d geometric primitives are shown on the tray at the bottom, utilizing ‘stacks’ of 2d content. Apple has been aggressive adopter of 3d with Time Machine, various dock visuals, Cover Flow and even specific spinner controls in the iPhone widget library. I doubt this design will see the light of day as described.
Bump top – 3d desktop
Really how much time do you spend organizing files on your desktop? The answer: lots more if you have something fun like Bump top to waste your time. The genius work here, is in the gestures, this product should have remained the cool demo it was four years ago, or it should have been a product 10 years ago, back when the desktop really was still a relevant place for innovation. Physics integration is nice but re-creating the imperfections of the real world (messy collisions) I can do without. Apple invented Piles, back in the early 90’s. Which is the coolest aspect of this whole product.
Useful – Is 2d content in Geometric space a useful approach? The ability to contain things and organize groups of things into relatively abstract containers like cubes is somewhat useful although frankly arbitrary. (The Absolut example is perhaps the best example here). Angled panes on walls reduce the real estate required for items but can create ugly skewing artifacts.
Usable – If the Camera remains static relative to the objects this approach is a simple extension of 2d techniques. The moment the camera starts navigating around or inside the geometric primitive all bets are off for people maintaining a useful mental model of the space.
Desirable – I’m not sure about the Desirable? Personally I don’t like the aesthetic of boxy environments too much. Time will tell if Bump top reaches any kind of commercial acceptance similar to the response it garnered from the audience at TED.
4. 2d Cards in ‘real’ space
Potentially one of the least explored ideas is putting 2d cards into real environments. ‘Acrossair’ has a nice demonstration of a way finding application for the New York Subway system that overlays cards (and arrows) onto the iPhone camera output. It uses compass and GPS information to determine where the camera is looking.
This is likely to be most explosive area of growth especially given the excitement around augmented reality applications. Take the amount of data that has real world data and its not hard to see applications of this everywhere from advertising, shopping, way finding, social networking to real estate.
One additional mention for Photosynth which takes 2d photos of things and creates a navigable 3d space. I have to say like everyone else when I first saw this I was amazed more by the ‘deep zoom’ than the stitching element between the photographs which sort of ‘recreates’ a three dimensional browsable space.
3d objects
1. 3d objects in metaphorical or virtual space
Obviously this is the realm of maps and games but its worth calling out a few of the many examples that continue to shape our digital experience.
Mapping applications like Google’s streetview or Bing maps are well on the way to providing a photo realistic version of our world that even 10 years ago would have been considered science fiction. The latest engines are doing automatic placement of 3d vegetation based on infra-red satellite images, actual weather information in the sky and perspective correction on the sides of buildings. This opens up the opportunity for virtual tourism. Almost full camera control is offered to the user that makes the systems clumsy to use from a humans eye line perspective. Today the content in the mapping application is mostly 3d buildings and vegetation but expect 2d content (call-outs, advertising, etc) to show up in the near future.
There have been and continue to be lots of attempts to do virtual shopping malls. Reading the blurb on these company’s sites is a little depressing. The unbridled marketing fueled enthusiasm they have for what they think is a new breakthrough in shopping experiences. In essence these sights are just fancy link catalogs.
This represents the next generation of retail, through a virtual environment; it’s original, it’s exciting and it has amazing potential, … The 3D visualization technology means consumers can enjoy the experience of shopping at any time of day or night.
Another attempt to make the web 3d, this one goes slightly beyond shopping and at least has a slightly distinct visual style. Really though how does this scale? I cant really imagine a mall that’s got hundreds of stores without them requiring, maps, directories and search which of course begs the question why bother with the whole spatial metaphor of a mall in the first place.
The R.U.S.E demo shown at E3 is a nice demonstration of multi-touch and 3d gaming, seriously watch the interactions in the demo and you’ll see an expert at work. This demo jockey is using a palette of learned gestures, from two-finger drag to two-finger spread to one finger arcing to change the camera perspective. Selection of objects is done with a two-finger poke drawing a selection rectangle. I’ve yet to play a RTS game on a multi-touch surface, its sure to be tiring, but this feels like the perfect use of all those gestures we’ve been seeing on surface tables for a couple of years now. Games are obviously fine with pushing the ‘usability’ of these systems, as learning them is part of the conquering the challenge of the mission.
Above, T.A.T’s recent ‘conceptual’ exploration of a 3d mobile phone UI. The UI is made to look physical (80’s plastic toy buttons and carousels). The camera moves around a ‘scene’ of widgets and various pivots are shown working in a kind of inspector carousel with filters. These guys have built their own rendering engine and tools pipeline to be able to do this all in real time and the output looks great especially for a mobile platform. The novelty of this makes the reaction of most technology people ‘wow’ but this is a deeply impractical UI.
When you watch this movie notice how the demo jockey has curved his finger to move the carousel… (cheater). This illustrates one of the fundamental issues with all these dimensional presentations of content and that is our input systems are still 2d. Fingers and mice don’t really map that well to manipulating 3d objects in space and until someone invests some kind of magic haptic feedback its never going to be easy.
Useful – Perhaps more than other examples this is highly dependent on the users, tasks and context. Generally 3d maps with 3d content (arrows, points of interest, landmarks) are obvious cases where this works perfectly.
I’d don’t think pursuing 3d objects as UI controls is a very good way to go (e.g. the plastic carousel in the TAT image above). The physical world is such a constraining starting point, but in limited contexts, (like the I-phone picker control that uses a 3d curved appearance whilst actually rendering all the text in 2d.) it might work as an affordance rather than a metaphor.
Usable – Again highly dependent on the model used, but this style suggests camera movement is a must.
Desirable – There are plenty of contexts where building a 3d world with 3d objects makes perfect sense, and has some aesthetic appeal in making it ‘beautiful’ and idealized. The dioramas style from the aesthetics post is an ideal style for this kind of thing.
3d objects in logical space
Typically the best applications of this are seen in the scientific visualization community where 3d objects are taken out of their context to aid in understanding them. This example from the cover of Tufte’s book, Envisioning information is a great example of providing just enough extra information in the abstract to aid in understanding the phenomena being described.
The same egregious use of overly complex data visualization from the 2d cards example also continues in 3d space. This tool isn’t exactly built for mainstream users but all the same super complex systems visualization continues to befuddle comprehension. (How many variables can a human process simultaneously?).
Useful – Useful, this can be the only way to convey the underlying structure of complex machinery, internal systems, biological systems, and hidden natural systems.) So yes its very useful for the scientific, engineering, educational and medical markets.
Check out systems like Catia for the complex engineering 3d software.
3d objects in real environments
Camera driven Augmented reality, the Wii, to Sony’s Eye toy, to Microsoft’s project Natal), Apple’s Iphone, Google’s G1 are using mostly camera’s to enable 3d input for the first time. Real 3d inputs involve moving a device like the Wii-mote or you’re phone or your arm around in X,Y and Z space and mapping real physical spatial inputs directly to outputs. Some aspect of this is Natural user interaction is destined to become part of our daily interactions with technology over the next few short years.
Total immersion did a demo 4 years ago that still is very impressive and blew my socks off at the time. Its taken a few years but we are now starting to see an explosion of toolkits that enable live video feed image processing to do motion tracking, even on relatively low powered devices. The result is 3d objects bursting out of video screens all over the web.
Sony at this years E3 showed Eye Pet which shows a 3d critter working with a PS3 and a Sony eye toy (web cam). There’s tons of interesting aspects to this its almost unbelievable. The camera is able to sense the ground plane and actually map the little pet character correctly into the scene.
This great project from some students showing off a pretty compelling game using a TEGRA development kit (more on this in the next article.). This is the first game I’ve seen using augmented reality through a mobile device, obviously it’s just the beginning.
Useful – There are so many potential applications of this from serious everyday productivity to entertainment to social that its hard to not see this as the most compelling use of 3d to date.
Usable – The obvious benefit of mobile systems for AR, is the camera being on the back of the device rendering the screen as a direct looking glass onto the real world, sort of ‘direct viewport manipulation’. Laptop or web cam form factors put the cameras facing the viewer, which inverts your inputs. Note the kid’s eye line is off center in the Eye toy video meaning they are looking at a TV to see the image we are seeing. This presents some input problems. No tactile feedback is another problem in this space that makes gesturing with a 3d device hard to judge.
Desirable – This is sufficiently magical that it stands as one of the longest standing most over-hyped technologies out there (along with VR). With 3d input devices and 3d outputs possible with head tracking there’s going to be some incredible things enabled that will challenge all of what we think we understand about U.X design. Who needs the full emersion geek headgear for that?
Final thoughts and tips
- Some arguments can be made for 3d around efficiency of layout, overlapping content can take up less real estate, but that only works in the most camera constrained cases.
- Research suggests that people can utilize spatial memory in the recall of objects placed in spaces but this can be true in 2d too, and the extra burden of users have to learn the physical layout of a space and manipulating objects in that space takes away more than it adds.
- The screen and mice are both 2d devices. Multi-touch gives us a few extra degrees of control but its still best as a 2d input system.
- 3d works best when combined with animation and camera movement. This typically makes it ‘4d’ because the expectation is that greater comprehension of the UX comes through exposure and exploration of it over time.
- A general rule of good information design is ‘reduction’ removing the un-necessary in favor of enhancing the important. 3d by nature can make that reduction task more complex and thus 3d designs tend to be more ‘complex’ visually. (Especially if they utilize the effects that make 3d worthwhile in the first place, lighting, shadows, material qualities etc).
- 3d organizational systems and user interfaces work well when they map to ‘simple’ metaphors but don’t take the metaphors too literally. Coverflow has reached widespread adoption because the underlying idea is incredibly simple ‘a stack of albums’. I don’t need camera movement or 3d input technologies to go next/back. Carousel’s borrow from the physical idea of a lazy-Susan device and similarly don’t need much in the way of explanation or flashy camera movements to work.
- If you are using 2d inputs, consider not giving the user any real freedom to move the camera, or the target of the camera, other than pan, track and zoom.
Summary
I think there’s a palpable hunger for the next novelty in the consumer technology space. 3d graphics and effects are for the time being a source of that visual novelty, and will help new products differentiate themselves in very competitive markets. This more than any other factor is the central reason to use it. But do so with care, so many bad experiences have been created that a backlash against 3d is likely if we continue to abuse it.
Mainstream adoption of 3d inputs sensor technologies, (including augmented reality) and eventually 3d output technologies will make a huge difference to adoption of 3d as a useful and practical interface experience. But don’t ask me to put on special glasses or headgear.
I’m still hopeful that someone will come up with a holistic consumer friendly 3d user experience that actually works, and is more useful, usable and desirable than any 2d counterpart across a wide range of activities. It’s conceptual model will be instantly familiar, natural but not kitsch or cheesy with overuse of tired metaphors. It’s clever integration of spatial information will give me new and important insights into practical tasks and activities and its use of 3d aesthetics (lighting, physics, subtle animation and materials) will engage me emotionally the way great design always has. Here’s hoping.
The next and final part of this blog series is on the tools and technology part of the story, what technologies exist, do they scale to different platforms, what tools make it easy for creative’s of all kinds to push the status quo.