Finally the final part of the 3d blog. As I mentioned in the previous posts on 3d, there are very poor tools for creating great looking 3d user experiences. This article will not concern itself with the aesthetics of 3d or the usefulness, utility or desirability of 3d UI’s, for discussions on that see here.
First off I’ll do a quick overview of the basic tools for creating 3d assets before diving into the tools for creating interactive 3d for different kinds of user experiences. This is a high level guide to the topic and in no way attempts to be exhaustive, just hopefully helpful.
Tools for digital illustration - Just creating great looking 3d.
The biggest tool provider for anykind of 3d anything is Autodesk. They are the million pound gorilla when it comes to 3d tools. Over the last few years they have swallowed up what used to be the top 3 tools in the 3d world, Autodesk SoftImage (formally XSI), Autodesk 3ds MAX and Autodesk MAYA - all previously rivals (stunning the DOJ where asleep at the wheel for those purchases) . These are considered the mega premium tools and have almost religious followings in the 3d community from game makers to film makers but to be honest from my perspective as a 15,000 foot generalist they are more or less identical in terms of capabilities. They are also ‘huge’ from a time investment perspective as they combine multiple sub tools, for animation, modeling and rigging, physics, creating shaders, lighting and rendering. Whilst they are widely used and regarded by moviemakers and game-makers there are plenty of much less expensive alternatives that can do 90% of the same basic things.
Sooner or later you will want to visit Turbosquid - it’s a huge 3d web marketplace for models, shaders, plug in tools and tutorials some of which are free.
COLLADA™ Is an open source intermediate file format that defines an XML-based schema to make it easy to transport 3D assets between applications - something that you will quickly discover is a god send. This includes geometry, materials, textures, lights, camera, animations, instantiations, and scenes. COLLADA also includes skinning and morphing for character animation. Shader effects for several shader languages as well as some physics support.
Once you’ve created or bought your 3d models, products like Autodesk Showcase include a fancy real-time 3d engine which allows you to move around high end renderings of industrial design models in real time, complete with soft shadows and high end shaders. (Car interior image below is showcase)
This tool is primarily intended for doing ‘pre-visualization’ and design review of industrial design forms using the power of the high end video cards and is a glimpse at what’s possible with the latest rendering engines. They also include the ability to play with libraries of pre made common materials, and environment lighting and immediately see the result, which has historically been a massive time waste with traditional renderer’s.
Hypershot Bunkspeed, is a quasi-competitor to Showcase allowing you to do semi-real time high dynamic range lighting, (HDRI). HDRI is a technique for doing image based lighting, which is how you get something to look photo real. The product is mesmerizing, super easy to use, and really useful for illustration. Simply drag the high end shaders onto the model, pick your image based lighting and (image below) is what you get. The soft shadows and ray tracing capabilities make the output look spectacular.
There is also a plugin for Google’s Sketchup tool that uses Bunkspeed. Sketchup is the world’s easiest 3d box-modeling tool, ideal for quick 3d sketches of physical relationships; bunkspeed kind of makes anything look hot with the right HDRI image choice.
RTT is a technology company making another one of these real time 3d rendering Showcase like tools. I find this type of product interesting because of how they clearly demonstrate what is possible with real time, high-end rendering. RTT also offers sever based rendering solutions which can deliver rendered frames to less capable hardware in semi-real time. This allows car configuration UI’s to be done remote from the client. (TT image below is real time)
3d Real time UI Rendering API’s
OpenGL® ES is a low-level API for advanced graphics using well-defined subset of OpenGL. Open GL has largely become the standard API for doing high end 3d outside of the PC and XBOX platforms which are dominated by Microsoft’s competing Direct X technology. GL-ES provides a low-level applications programming interface (API) between software applications and hardware or software graphics engines. Apple developers have the option to use open GL ES on the I-phone and now the I-PAD for example.
The wizzy bookshelf spin into the bookstore effect from the IPAD launch video, another example of 3d metaphors successfully working to add some bling to a product announcement.
These API’s make it easier for developers (mostly games developers) to write code that is slightly abstracted from the exact hardware it will run on, making it more compatible with a wider audience.
WEBGL - WebGL is a cross-platform, royalty-free, open web standard for a low-level 3D graphics API based on OpenGL ES 2.0, exposed through the HTML5 Canvas element. The basic premise is that most browser makers (except IE perhaps) are working together to enable basic support for this as part of the HTML 5 specification, which is still a year or two away from prime time. These open standards would go along way to making 3d UI not require a specialized engine plugin (like Flash or Anark); many people are hyped up about the potential for this to finally let the 3d cat out of the proverbial bag.
Nvidia’s TEGRA 3d UI
This 3d UI (carousel image shown in the article header) was shown in working from June 2008, and generated significant excitement that a 3d touch based UI that looked reasonably hot would actually become a commercial reality. Turns out this UI will not apparently see the light of day according to this report. I believe the UI below was authored using the now extinct tool ANARK game face which was bought from Anark after having been a tool for implementing 3d UI’s in games for a several years.
Tools for building 3d UX Today.
Today as a designer wanting to create a real time 3d interactive experience your choice is pretty much limited to Adobe Flash with one of the various plug-ins, like Papervision or Away 3d. Inherently Flash is not a great 3d platform basically because the first thing you should know about 3D and Flash is that Flash doesn’t really support 3d. What Flash does know is how to display vector shapes on the screen and how to calculate math expressions. With that, 3D can be faked by doing really sophisticated mathematical transforms. Generally this approach is ok for minimal 3d transitions and simple geometric models but as soon as the going gets rough (say you need high polygon models or shaders) designers resort to non real time 3d and interactive pre-rendered 3d Flash movies. Still, Flash 10 or one of the plugins mentioned above is really still the de facto option for developing interactive 3d sites today albeit they are often less than spectacular and don’t really take advantage of any of your specific GPU or video card hardware (features like anti-aliasing and texture smoothing are missing). One of the worst qualities of flash based 3d is the menacing ‘jaggies’ of aliased fonts and geometry, and the ‘sparkly’ effect that is created with non smoothed textures. The creative industry is obviously heavily invested in Flash, so if Adobe keeps plugging away it’s likely 3d will become easier and better through their classic and venerable authoring suite.
Electric Rain are a small software company who make Swift 3d which is a well respected modeling and animation program for Flash. It takes 3d rendered geometry and does something like the ‘live trace’ feature in illustrator, converting it into less memory consuming 2d vectors. For canned 3d effects it’s an easy and well travelled solution for designers.
Scaleform are a company plugged into the Games community with a tool that enables them to support Flash movies inside proprietary games engines, a great deal of modern games use this technology which gives them the, coding, production and animation benefits of Flash inside their rendering engines, thus making the production of sophisticated menu systems, Hud’s, (text, maps, equipment status, health etc) or the configuration UI’s much easier.
Expression Blend is a UI building tool produced by Microsoft which can create 2d or 3d interfaces for Microsoft’s client application engine WPF, or its online cousin Silverlight. Expression Blend is ‘good’ for building 2d UI, however when it comes to 3d we are back in the dark ages. It can display and manipulate 3d models with simple mapped textures but every time we’ve tried projects utilizing 3d - we have quickly run into performance challenges (even on high end hardware). There’s no support for any sophisticated effects, shaders, anything more than the most fundamental lighting, no decent animation support. There are some 3rd party tools to make life easy on WPF. Another Electric Rain product (I’ve not used) ZAM 3D is a 3D XAML Tool for Microsoft Windows Application Development. It provides developers and designers with a quick and easy solution for creating 3D elements for Microsoft Expression Blend and Visual Studio projects. It also acts as a 3ds or dxf to XAML converter.
It seems increasingly clear to me that Microsoft has de-invested in WPF, and I’ve given up hope that it would become a decent way to create a 3d UI. In all the hype about WPF at its launch, there was some good demos and excitement, yet here we are 4 years on and really there are no decent product examples of WPF applications, and very few announcements from Microsoft about its roadmap.
There are a few other choices out there like Anark, which is a long in the tooth tool from yesteryear which looked very promising as a way to author 3d user experiences about 10 years ago, but today seems to have been abandoned by its parent company as a super niche tool. We’ve built some incredible 3d demo’s in Anark but the tool seems to have not received a significant update in a long while and it’s expensive.
ARToolKit is a Cross-platform Library for the creation of augmented reality applications, developed by Hirokazu Kato in 1999 and was released by the University of Washington HIT Lab. Currently it is maintained as an open source project hosted on Source Forgewith commercial licenses available from ARToolWorks
ATOMIC Authoring Tool is a Cross-platform Authoring Tool software, for Augmented Reality Applications, which is a Front end for the ARToolKit library. It was developed for non-programmers, to create small and simple, Augmented Reality applications, and released under the GNU GPL License. I’ve not used it and can’t speak to its usefulness.
The Astonishing Tribe, also known as TAT have a 3d development environment I’m curious about, its called Kastor, and I think works with TAT Cascades which is a UI framework. Some of the demo’s mentioned in my previous post’s on 3d were probably developed using this tool set. The Red-dish demo (below) they did last year utilized OPEN GL ES 2.0 shaders and anti aliasing functionality.
Kanzi is a suite of tools and technology for 3d UX development. The Kanzi SDK tool is designed to be used with OPEN GL ES on mobile devices. Its Rendering engine Kanzi Engine is supported by a range of silicon manufacturers. I’ve not used this tool in a professional context yet, but the examples on the site suggest a mature product with many of the capabilities for building good looking, hi quality 3d. At the time of writing this, I just recieved an email from a Finnish company called Rightware that has just acquired Kanzi from Futuremark.
Images above from Kanzi demos above.
I’ve purposefully left out discussing the merits game engine technologies like Unreal, ID- Tech and Valve’s Source engine, Microsoft’s XNA and many others which are less ‘tools’ and more like ‘starter kits’ for the development of 3d games, they handle low level features but they are really only for those conversant in C# or lower level languages and not really optimized for building user experiences or more general products.
I’m very interested if you know of other commercial tools for building UX and will update this blog with suggestions and comments when I receive them.
Tools for building 3d UX tomorrow.
At heart the people using UX building tools and contemplating 3d UI concepts are desperately in need of tools that allow them to ‘play’ in high performance real-time 3d quickly and easily. The lack of these tools, I’d argue is the main adoption blocker for seeing more high quality 3d user experiences. Occasionally I see something really impressive in 3d that makes it more clear what we are striving towards in terms of output from such tools. At the Intel booth at CES 2010 was a very impressive dual touch wall (Image below). This was not created (to my knowledge) with a cutting edge 3d tool but it demonstrates the kind of output UX designers like ourselves at Artefact are striving/desiring to create.
Check out a video of the animation on Youtube. This would be impossible to build in any tool that I’m currently aware of and had to be coded from scratch.
Watch this video of someone creating a “little big planet” level. This is a tool that comes with the game for players to create additional content and share with the community. You use a PS3 controller and a character to create levels within the game. Note in the video that there is no difference between building the level and being in it which makes the authoring process very experimental. Objects are ‘physically’ enabled by default. There is minimal difference between the “authoring process” and the “experiencing it” process. This tool is actually very simple to learn, super complex to master. It’s also a modern work of art.
This tool represents a kind of design target for me, today it feels especially for UX like we are down where C++ developers where years ago, down in the weeds creating everything from scratch, specifying every possible variable and property. The desire is to have a palette of starter objects, 3d controls, shaders, lights, behaviors and effects that are a starting point to start sketching great user experiences. When my sketch work is complete, allow me to dive in and get busy scripting custom behaviors, defining custom shader’s and geometry.
The complexity of 3d tools makes production a very deliberate, detailed and labor intensive process. As a result designers are limited in their ability to experiment and iterate in building 3d user experiences. So my ideal tool combines lots of ‘starter objects’ with the ‘stateless’ UI demonstrated by Little Big Planet and Autodesk’s Showcase application. With such a tool I think we’d be pretty close to the kind of tool that would take the world by storm. (Quietly I’m hoping someone sends me a link) Contact me if you want Artefact to design such a tool, right now that would be a dream project.
I got worked up in a tizzy recently and had to have a drink with international sex symbol Dennis Wixon. I told Dennis that some of our lab work (which includes protocol analysis and RITE method) can feel like administrative busy work, especially if the person doing that work is an outsider to the project team. The role becomes meeting organizer. Organize the meeting of target users and stakeholders. It’s possible to be brought in to projects without an understanding of the business goals, the target customers, the design requirements, the purpose of the product, and the design decisions that need to be made. Not knowing these things, doesn’t set up the person doing this work to make very good design recommendations. Do the designers want an outside perspective? Not really. They have enough opinions from others. Do they want a meeting organizer? Yes, but is there more?
Dennis reminded me that facilitation skills are essential. I didn’t like the answer at first, but he’s right. There are many stakeholders involved in problem identification and generating solutions. There are designers, janitors, product managers, cooks, engineers, testers, marketers, executives, accountant’s, receptionists, lawyers and their lawyers. Researchers can be good people to facilitate the formation of stakeholder’s interpretation and conclusions of user’s behavior. Researcher can know which verbal reports are useful and useless pieces of data. The good researchers know the important differences between a sequence of thoughts generated by users solving problems and social verbalizations used to commuicate through descriptions, justification and rationalization. It is the researcher’s job to get stakeholders looking at the best data. Lead them through their own analysis and interpretation of observations. Influence their conclusions. Faciliate the process of making good product design decisions.
I have not thought much about what it takes to be a good facilitator. Experimental psychologist, sociologist, anthropologists and other social scientists that research people aren’t necessarily educated in the art of facilitation any more than designers, engineers, marketers and business managers. Researchers that know how to facilitate a group of stakeholders to make the best decisions from good data will be highly regarded. It’s the researcher’s facilitation skills that can make the difference when dropped in to projects to coordinate the meetings between target users and stakeholders.
One of the more obvious trends at this year’s CES was 3d TV, all the display manufacturers had major installations showing off their 3d HDTV offerings. The CE industry seems to have collectively decided that 3d is the next big thing and that consumers will go for it in the home. The mood at CES from the attendees I encountered however seemed overwhelmingly skeptical about what seems to be a supply side (industry) push of a new technology that is just not ready for primetime. There is an obvious need for the CE industry to repeat the success its had in recent years getting consumers to buy into big flat screen HDTV’s and they are obviously trying to get us to all upgrade again in the next few.
The skeptical arguments I heard landed in four camps, the effectiveness of the technology, (is the effect cool enough?) the content pipeline (Is there enough 3d content? ) , the cost of new glasses and displays?, and finally the glasses make me look like a ‘dork’ factor. (I must look like a right plonker wearing the oversized polarizing glasses over top of my existing glasses).
Almost all the demo’s I saw currently utilize the approach that requires polarized glasses to see the intended 3d effect. Let me tell you if you’ve not seen it, the effect of the 3d on these big displays with glasses is pretty compelling - not as dramatic as the first time you saw full HD, or say the transition from Color to black and white. But with good authoring the effect can be pretty cool and it’s not hard to see its significantly better than HD as long as you are wearing the glasses.
The Demo’s I saw of non glasses 3d relies on lenticular lenses overlays and multiple image perspectives being rendered at the same time for you to see the desired effect. They have a ‘sweet spot’ of convergence where your eyes are far enough apart to see 2 slightly different perspective images and the effect is cool if you are in the ‘zone’. If you are not in the zone you need to move your head from side to side to see the different perspectives. One technical problem with this technique is the need to render several different perspectives of the same image. This is fine for CG 3d game content, but 3d cameras with more than 2 lenses don’t exist to my knowledge, not to mention there are no standards or formats to support transmission of multi-perspective video content.
The lack of 3d content problem is non-trivial issue but something the industry from broadcasters to camera makers and TV manufacturers are probably prepared to invest several years to fix, (the way HD content eventually became commonplace).
Toshiba’s alternative approach is to convert 2d content into 3d on the TV in real time utilizing its just announced ‘Cell TV’. The Cell TV has ten times the processing power of most personal computers. All the computing power is used to convert 2d HD video into 3d on the fly which is a pretty impressive feat but starts to raise the obvious question of cost. As an aside the Toshiba Cell demo had so much computing power that they also used a magic wand controller (like the wiimote) for selection and manipulation of their 3d UI. The UI (below) was a vision of 3d UI hell, a nested sphere of spheres with no order, rhyme or reason. (See my 3d UI blogs)
Sony was demoing some 3d Playstation games with a PS3 firmware update they said would be available soon, games are an obvious way in for the likes of Sony to create a compelling amount of content with relative ease. The driving game I played was pretty cool the 3d seemed to work really well with a 3rd person camera perspective as good 3d requires, decent foreground, mid and long distance content to ‘pop’.
As far as cost goes, on top of the $2-3000 or so you may spend on a 3-D set, you’ll have to pay extra to buy or rent 3-D movies. Apparently Sony plans to start selling sets labeled “3-D ready,” meaning they can be upgraded to 3-D later with some additional costs and accessories. I don’t know about you but I’m happy to have just fairly recently upgraded to a nice HDTV so I’m going to be in the market for a newer better one for five or so years. I also have not heard anyone rule out the possibility of having the 3d effect simply be a standard HD video signal from an external box which could obviously reduce the cost of the upgrade to 3d significantly.
According to the latest data, around 126 million citizens of the United States of America currently wear eyeglasses. Since there are around 300 million people living in the United States, that’s almost half of the population. Looking like a Dork with two pairs of glasses on aside, customers of 3d who currently wear glasses will need to make a big behavioral shift (swapping glasses or adding an extra set) which I believe they simply not do - the effect is neat but not that good. Maybe optically correct 3d polarized glasses will become the norm at the eye doctor?,( let me hear a collective sigh of skepticism. ). Also for some of my colleagues at the show, the glasses seemed like a source of much frustration claiming they were causing strain and would lead to exhaustion. I think an additional negative of the glasses is the scenario of gathering a group of folks around to watch the ball game, with everyone having to wear the glasses or else what they see is a blurry unwatchable nightmare. Talk about a nerd fest.
I for one don’t think glasses based 3d HD TV’s will become broadly adopted. These devices will become an option for the high end technophiles who have to have the latest thing and perhaps loner gaming enthusiasts but broad based adoption will take many years as current flat screens are too good to warrant the upgrade for most people.
I think there’s enough technical and infrastructure challenges that perhaps 3-5 years from now we may start to see some major commercial adoption of non glasses 3d HD. Eventually I expect that 3d cameras like Microsoft’s Natal will track our gaze with sufficient resolution that a more convincing 3d can be created ‘in software’ with no glasses or requirement for new display devices. Until then 3d displays are and glasses are not for me.
The following is a result of a dialog between Gabriel Biller and I, Kevin Wong, who are both researchers at Artefact. Mobile manufacturers have put into people’s hands a device that unlocks information and connects people with each other. There is one question though, and that is:
what about connecting our phones to everything else?
Our intention is to share our thoughts and understanding of the marketplace for the sake of advancing conversation towards a connected world for everyday people. It all started after an IM chat the other day…
Kevin: Hey, Gabe. You know what’s annoying? Gabe: What is it this time? Kevin: Last week, I was at my sister’s. And she was showing me photos of my niece’s recital from her PC upstairs, using Apple TV connected to her big plasma. And I was, like, oh my god, I need to show you some photos of Lauren from the last time I was here that I haven’t showed you yet. They’re on my iPhone. Gabe: And? Kevin: Well, let’s just say there was no obvious, quick, and easy way for me to do that. I couldn’t do it, and I’m usually pretty good with these things. You’d think an Apple TV and an iPhone would easily allow that kind of sharing. Gabe: Hmmm… that is one rotten apple. Kevin: Why do they even call it a phone anymore anyway? Gabe: Good point. It’s basically barely even a phone anymore… at least, not primarily. It’s a mini computer, my Internet-enabled sidekick, my trusted friend… they should call it an “also-a-phone.” Kevin: An AAP? Ha! An AAP loaded with apps.
So, what’s the problem?
Despite the digitization of many devices and appliances in our everyday lives, very few of these devices can communicate with each other. The Internet-enabled smartphone, with its array of communications antennae, is poised to enable us unprecedented ability to monitor, manage, and control devices and systems that are communications-enabled. But if you look around today, we just aren’t there yet.
First, I’m still not connected to everything I want to be connected to.
Secondly, the things a phone can do are primarily passive. I can get information, and yes I can send information and do creative things like take photos and videos, but I can’t always take action with it – at least, not on other things.
For instance, without the resources and brainpower of Bill Gates,
Why can’t I check if my lights are off and my doors are locked when I go on vacation? And if they are on, shouldn’t I be able to switch them on/off from anywhere?
Why can’t I start my morning from bed by getting the coffee maker going, turning on lights, warming up the car, etc.? I can check my emails, the weather and my schedule, but I can’t do anything else to make my morning more productive.
Why can’t I carry my medical records with me and share them with my doctor? I can search symptoms, directions to the doctors office, but I can’t transfer my information from one doctor to the next.
Why can’t I print something from my phone to a nearby printer? Or just show some photos or videos on my phone on any big TV or monitor I get close to (without it being a big hassle)? I can view photos on the tiny screen and even Word documents, but I can’t use the TV that’s just sitting there to show photos for everyone else to see.
The following is an overview of the current state of mobility where, in the end, we’ll highlight what we believe are the important things for mobile manufacturers to consider as they evolve their strategies for improved user experiences for their customers.
So why the phone?
The most portable, personal and cherished digital object in our lives is the phone. It is more powerful and more capable of completing tasks similar to what you would expect from a full sized computer only a couple years ago. People are relying on it more to find, capture, collect and share their personal and professional experiences. Palm released the Pre last summer in June, a sliding touch-enabled phone that contains a full QWERTY keyboard, the power to multitask with multiple applications at once, and even cuts cheese! Android and Apple are bringing tens of thousands of applications to people everyday letting them know that, “there’s an app for that.” The phone, as Alexander Graham Bell originally conceived it to be, was an “electrical speech machine” for people to communicate with other people from remote distances. It has now evolved into a computer, a platform, that allows people to communicate with other people and machines with their voices, text, video, photos, their personal data, tweets, and whatever else the imagination holds. More importantly, phones have empowered people with control over many things, literally at their fingertips.
We can’t simply think of the phone as a phone anymore. Instead, we must continue with imaging the phone as a portable entry point into our surrounding digital world.
How people are using phones now
Let’s take a second and look at what’s going on right now, and what we might see happening in the near future. Today, the average smartphone user is doing the following activities with his/her device:
phone calls
texting/SMS
emailing
web browsing
game playing
watching videos, even recording videos
taking, sharing, and viewing photos
social networking
mobile banking
buying music, videos, TV shows, books, etc.
personal organization and management tasks (calendars, notes, etc.)
But, on the horizon are many other interactions that have either been demonstrated already or will soon become possible as well – possibly even mainstream:
While our phones continue to bridge our communication, information, and entertainment needs, we’re beginning to see some attention applied to interactions that allow people to have direct control over physical things around them. The exciting part is that as more of these things become digital, the more people can control them from a single point of access.
More things the phone should be able to talk to
How many appliances do you have currently in your home? How many electronics do you have in your home? Done counting? Gabe has 16 total and I have 14, respectively (of course, we are technophilic Seattleites). Technology is all around us, and it will continue to become more pervasive as time goes on. According to some futurists, using Moore’s Law and observed changes in purchasing power and technology spending, the number of “impact of computing” devices (loosely defined as those pieces of hardware where much more powerful/capacious versions will be available for the same price within 2 years) in an average household has been growing exponentially.
The average household in the current decade (the 2000s) will have had anywhere between 6-12 different digital, electronic devices. In the 1990s, that average was only 3-4 devices. Remember your first cell phone? In the 1980s, the average was 1-2 devices. Remember your first PC? In addition, consumers are buying more and more of these devices as their lifestyles have become increasingly digital. Have you ever seen the “what’s in your bag?” group on Flickr? And that’s just bags, not entire homes. Here’s mine. It’s predicted that in the next decade, the average home will have 15-30 impact of computing devices; in the 2020s, the average home will have as many as 40-80! Our world is becoming more digital by the moment and soon enough, these devices will all have computer chips and access to wireless networks just like our computers today.
More phones available to talk to these things
More people are going to have this control. You probably already know the numbers, but for everyone else, let’s break it down:
In the past 3 years, the smartphone category has shown a compound annual growth rate (CAGR) of 24.3%.
Smartphones as a percentage of all mobile phone shipments was previously 7.6% in Q4 2006 but jumped to 10.1% in Q1 2007. The size climbed steadily to 15.1% in Q1 2009 with only a slight drop in Q2 2009 to 14.9%. (sources: Canalys and Strategy Analytics)
While analysts have a range of predictions on how big it will be in the next couple of years, one thing is similar: they predict there are going to be at least 300 million units out the door by 2013. As it stands, there are currently 131-192 millions shipments worldwide (sources: Gartner and iSuppli). Gartner is most optimistic in that they believe 525 million units in will be shipped by 2012; Parks Associates says over 300 million in 2013 and iSuppli says 400-450 million by 2013.
Basically put, the smartphone market is growing and there are going to be more people using them.
Current attempts to ‘Connect the World’
The pieces seem like they are there. Powerful computers in our pockets, increasing number of devices that can talk to each other, and an expanding network of untethered connectivity. Yet, still, much to our dismay and frustration, very few of these devices and appliances communicate well with each other outside of our home theaters. To date, we’ve primarily been cobbling together a hodgepodge of point solutions to little problems and marveling that it actually works. This approach won’t work in the long run. The initial excitement of what can be done will soon fade away, and people will realize that how it is accomplished is awkward, convoluted, poorly designed, and incomplete. Fragmented solutions and poor user experience leaving us still wanting.
Unsurprisingly, Apple has been the most successful with their approach to offering a seamless digital lifestyle that is simultaneously desirable to consumers, technologically elegant and sound, and that has translated into financial success through new business models. A handful of scenarios that is possible from their approach might sound something like this:
Running a business presentation in Keynote on a MacBook via the Keynote Presenter application on your iPhone
Watching TV, listening to music, or watching local or streaming video content from your couch using Apple TV and the Remote application on your iPhone
Driving a car using the iDrive application available in the Application Store
Fighting global warming by warning you when you’ve reached your monthly electricity budget via Tendril’s application and allowing you to throttle your usage
Connecting to medical accessories through a standardized framework that senses, captures and controls health information
Everyone has their reasons for wanting these applications and frameworks at their fingertips. Apple has demonstrated a vision that could bring us closer to a tangible, connected world.
As for the rest of the mobile handset manufacturers?
Research in Motion (RIM)
Not made by RIM, but Unify4Life is an application that you can install on your BlackBerry giving you the ability to flip channels on your TV, change the volume of your stereo, and even open your garage door. It’s been well-received so far, but the experience isn’t completely seamless. In order to control your entertainment system, Unify4Life requires proprietary add-on device that sends an infra-red signal to your DVD player or TV. The same goes for your garage door opener. Your BlackBerry isn’t completely free to control your world just yet, but someone is starting it. RIM?
Motorola
Motorola is in disaster-recovery mode following 2+ years of poor performance. Once the #2 handset maker in the world (22.4% global market share in Q4 2006, plummeting to just 5.8% share in Q2 2009), they’ve re-strategized and re-focused (less is more, bet on Android) in the hopes of regaining some past glory. The new Cliq smartphone is their first Android handset, with many more likely to follow. But, what if anything, are they thinking about doing beyond catching up? They have broadband cable modems and digital cable boxes in millions of households too, but have they done anything to make their handsets talk to those devices? Yes and no. They recently announced a new Android based set top box in Japan that connects your media in home from your phone. It also plays the role of your typical media center such as playing DVDs and potentially allowing you to surf the web. Other than that, there are no signs of bridging other devices in the home in any other way, even under their own brand.
Sony
The recently released Sony Aino is offered as a companion phone to the Playstation 3, incorporating the PSP’s Remote Play capability allowing controls to the menu system on the device. In addition, attaching a Sony PlayTV to your Playstation 3 turns it into a DVR which can be controlled by the same Remote Play application. But, you can’t mow down soldiers in Call of Duty with your phone just yet (not that you’d want to).
Nokia
They hold the largest share in both the feature phone and smartphone categories, not to mention they are projected to have the largest share of mobile OS’s worldwide (Symbian). While they continue to exploit the bottom of the pyramid, capturing global market share (from 30.7% in Q3 2004 to 40.5% in Q2 2009) and global brand recognition, they have shown some signs of weakness in the smartphone category. Their early advantage and dominance has fallen slightly in the past 2 years from 50.8% share in Q2 2007 to around 45% in Q2 2009, as RIM and Apple have emerged as the strongest followers. Nokia has its Ovi Store (ho hum), and they just released a netbook, but how else can you integrate a Nokia phone into your life?
A new future
There are still a lot of questions out there: What business models should companies adopt? And the best suite of technology solutions necessary to do it?
But there are things that we believe in:
We believe open standards, an open developer platforms and more research should be done to continually understand how our digital lives are interwoven with a growing set of digital devices and appliances.
We also believe there are a lot of smart people already thinking about these problems and have some ideas. But the fact remains the same, we aren’t there yet.
There are also things that we know:
The market has had over 24% year-over-year growth in smartphone adoption since mid-2006, with forecasts expected to continue at 20%+ annual growth for the next few years
Over 50% of the world has a mobile phone (4.1 billion+ total subscriptions)
Appliance manufacturer Whirlpool will manufacture over 1 million “smart” dryers by 2011 (25% of their total dryer production). GE is also doing lots of work on making our appliances smarter, and $3.9 billion in stimulus grants from the Department of Energy is helping
People are highly mobile, and the things they interact with are in separate locations, or in the cloud. The phones that are with them all the time are the most convenient and readily available access points to their digital lives
A pair of pants typically has only 4 pockets. There is only so much technology you can fit in those jeans. Not to mention the current trend of skinny jeans
So let’s imagine…
Let’s imagine this new ‘also-a-phone’ as a remote control. Having a “remote control for life” can do so much more for you as an individual and for your family. We’re talking about smarter, more responsible living. Mobile manufacturers should advocate and enable people to do more than just “connect.” Use connectivity to help people do something good!
For example: today, understanding our energy and environmental “footprint” and the impact of changing our behaviors is still a difficult and complex undertaking. We don’t have the tools or infrastructure for understanding how our energy usage is divided among all the things in our homes and world that require energy; all we have is a monthly bill with a total number.
In the future, as more and more of the energy-consuming devices we use in our lives become capable of communicating with one another and the infrastructure (the “smart grid”) for understanding energy supply, cost, and usage is rolled out and matured, we should be able to use our smartphones as the central device for monitoring, understanding, and acting upon this aspect of responsible living.
No other device or tool has the ubiquity and power to transform billions of individual, small actions into world-changing, collective actions by regular people to inform, modify, and correct our behaviors related to energy consumption.
And that’s just one example.
In the future, mobile phone manufacturers should design the handsets, software, and services to work cooperatively with the other devices and appliances in our lives so that the “phone” can evolve into more than just the information-communication-productivity-and-entertainment device it is now. The phone of the future can also become the “remote control” for responsible living.
Oh yeah, and maybe in the future, I can show those photos to my sister and my parents on the big screen, instead of on my phone.
Keep the conversation going
If you like the idea of your phone giving you access to control the things around you, we’d love to hear about it! Use the hashtag #remoteArtefact on Twitter and share your ideas, stories or thoughts. Tell us your ideas, stories of when your phone could have saved the day or other considerations we haven’t touched on yet. We’ll keep exploring the possibilities on our end here. Thanks!
This is the third installment of my blog on 3d. The first part outlined the gist of the story, The second part was about the aesthetics of 3d. This installment deals with the use of 3d to convey a ‘conceptual model’ for a user interface. In this article I’ll identify different approaches to building 3d user interfaces and ask the question are they useful?, usable?, and/or desirable?. I’ve provided tons of examples partly to make the point there is a ton of activity in this space but not a great deal of discussion around it and of course a terrible lack of tools to create it. But of course the real reason I’m writing this is ‘Just saying ‘Z-space’ raises your IQ 10 points.’
When talking about 3d conceptual models of different user experiences the litmus test of ‘usefulness’ is that the third dimension the ‘Z’ is conveying some additional data or meaning that a 2d representation of the same thing would not have. Presumably this additional information is of significant value, and relevant in helping people understand it.
When it comes to ‘usability’ of 3d UX, it’s a big topic. When using 2d input devices, the design choice of free-form cameras v constrained cameras can have a big effect on the usability of 3d user interfaces. By free-form, I refer to giving the user the controls to point the camera in any direction. This is useful in 3d environments (interiors), but comes with the baggage that the user must now construct a mental map of where everything in the space is relative to any other point in space.
Constrained looking of the camera on the other hand limits the ability for the user to look in certain directions. This ‘Cone’ of visibility usually has the advantage of keeping the user focused where they need to be and stops them becoming disoriented. This may not sound significant but the difference between these approaches makes a massive difference to ‘usability’ of 3d interfaces.
Similarly enabling the user control of free or constrained movement of the camera in space has a big impact on usability too. Here’s some useful advice:
If you give the user free form ability to point the camera in any direction – think twice. If you decide its for the best then constrain their movement of the camera through space, the easiest constraint is just giving them X and Y movement which is familiar to all users. If you give them the ability to move the camera through Z space then constrain where they can point the camera.
One more: If you’re building an interactive 3d thing don’t choose ‘both constraints’, (i.e.: no ability to move the camera or move the viewpoint) this defeats the entire point and btw you should be in the movie industry.
If you are working with exteriors looking at objects from the outside, its sometimes useful to constrain the cameras target to the center of the object and then giving the user the freedom to ‘orbit’ the object as they see fit. This is often used in car configuration UI’s. Being able to zoom in (rather than moving the camera) gives the user the ability to see more detail without becoming disoriented. Other common approaches here are to allow the camera to move in and out of close ups, and when in the close-up providing a button to escape from this mode.
This is called putting the camera on ‘rails’ for transitions from one place in space to another allows the user to specify a different item to orbit around, perhaps a different car in the showroom. Even if you have rails, and constraints in full effect, giving the users camera ‘a few degrees’ of freedom is a nice touch often this is accomplished by mapping the position of the cursor in the scene to move the camera a few degrees as it moves around the screen.
This portfolio site shows good use of animation around a simple hierarchy. The camera utilizes limited degrees of freedom for camera movement, but keeps the impression of control over the user. 2d text instructional cues on the 3d objects also help the usability.
Full six degrees of freedom, (up, down, left, right, Forward, backwards, pitch left/right, yaw left right,, rotate left, rotate right) is rarely useful, turning off as many of these as possible generally improves usability.
That’s not to disregard ‘desirability’, I said about aesthetics in the previous article, sometimes just because it looks great is enough to make a design sufficiently novel, different, interesting, at least in the short term. Desirability in a crowded market can be the stand out best place to innovate.
2d Cards in 3d space
Take an inventory of all the ‘digital’ things we have in our lives and the list can get pretty long, for example; Songs, documents, spreadsheets, messages, photos, movies, contacts, conversations, bookmarked locations on the Internet, blogs, appointments, applications, places of interest, lists of things to do. Before our digital lifestyles these things had physical precedents, type is 2d, illustrations and photos are 2d, audio and video is represented by 2d box covers, calendars, books, business cards essentially are just collections of 2d sheets.
No surprise then that our first category of 3d UI’s simply put these 2d digital things onto ‘cards’ and then put the cards into some kind of metaphorical, logical or ‘real’ spatial environment. The cards themselves add no dimensional data but presumably their position in three-dimensional space conveys some additional meaning.
I’m going to call this 2d ‘Cards’ UI and there is allot of it out there. Cards don’t literally have to look like cards they simply are using 2d text or images to convey the content.
I’ve broken it down into four reasonably distinct uses of cards, although this taxonomy is just for discussion rather than an attempt to be authoritative on the topic:
Cards in metaphorical space
Cards in logical space
Cards on geometric space
Cards in ‘real’ space
The second category of three-dimensional UI I’ve seen uses data that is naturally 3d, for example; models of physical objects, buildings, maps, bodies, space, environments are of course the simplest to understand 3d data types. They are relatively abundant on the web and already part of our mainstream digital lives. Second life, Virtual and Google Earth, Google Street view, and almost all video games use 3d effectively in this capacity.
Here the creative discussion is about creating immersive environments that need to be explored by the user because the data is naturally spatial and best understood from multiple perspectives and vantage points, or the creative team want ‘exploration’ to be part of the core reason for the user experience.
I’ll call this 3d ‘objects’ UI and like above I think there’s at least three important variants of it.
3d objects in Metaphorical or virtual space
3d objects in logical space
3d objects in real space (Augmented reality)
2d Cards
1. 2d Cards in metaphorical space
The basic idea in this example from Microsoft Research (1997) was testing people’s ability to organize and remember activities when mapped to a metaphorical physical environment. The idea was to tap into our spatial memory and perception as the ‘added value of 3d’. Skewing the windows enabled you to see more windows at once, but this two can be done effectively in 2d without the ugly skewing distortion that’s occurring below.
Variants of this approach have continued for years substituting walls and corridors for cities, art galleries, hillsides, abstract environments of all kinds you name it. As the years have gone by the windows look better, the environments are cooler, but fundamentally this isn’t going anywhere.
Put yourself in the user shoes for a moment and consider about the amount of ‘organization’ it takes to create a custom environment where activities or windows or collections of digital things are positioned in an environment. Then think about the extra time, (and controls) it takes for me to navigate around in this space and basically the UI is starting to obstruct the user’s primary task, its getting in the way of whatever I was trying to do in the first place.
This organizational burden, place on the user demands they pay attention to the positioning of various things in space. Some research studies have proven that user recall has improved when things where spatially presented in environments like the one above, however perhaps they ignore the added time it takes for people to position, navigate and conceive the appropriate organization of these things.
The logical conclusion of this approach 3dna’s Desktop application for Windows gives you a complete spatial environment filled with lots of 2d pictures as links to various windows you have open, organized into metaphorical objects and posters.
The marketing banter for this company talks about the ‘Fun’ of it all, which I suppose if you like this kind of thing is about the only thing you can say about it.
Useful – So as far as ‘usefulness’ this approach is fraught with the organizational overhead management, and raises questions about multi-tasking and task switching that are irreconcilable with physical space. (Alt tab, or Expose are fine ways to move from task to task, walking across a room gets kind of old once the metaphor is established.). Other questions mostly around the scalability of solutions like this (how many windows can it handle before the metaphor breaks down).
Usable – Usability here is a mixed bag, full camera movement systems, and manipulating things around in space obviously put a burden on the usability of these systems, but at least the metaphors are ‘understandable’ which gives them familiarity to less experience users.
Desirable – Best case the desirability of this technique is limited to niche appeal, technology history is littered with companies big and small (Microsoft Bob, General Magic) who attempted to extend the computing metaphor into a more literal spatial environment. If you follow this approach no doubt a small community of hardcore fans will love you playing house, but mainstream consumers don’t’ really need this.
2. 2d Cards in logical space
Three dimensionally sorted stacks of 2d cards have quickly become commonplace in adding 3d to our user experiences, on desktop, mobile and console devices. Can you count the number of carousels, walls, curved walls, piles, boxes that have shown up in our UI’s recently?
A toolkit above from Fotoviewer provides a variety of different card arrangements for images. All these owe a great deal to ‘Coverflow’ ,which was originally developed by an artist, Andrew Coulter Enright and was purchased by apple in 2006.
Bing’s new visual search using shelves
Clever use of slight overlap and perspective convergence gives us the layout benefits of getting more items on the screen, than a standard 2d Grid. This is straight forward basic use of 3d, no conceptual headaches, simple to understand, takes advantage of 3d layout, and is visually ‘novel’ something Microsoft needs to help it convince users its different/better than Google. Also in the same vein is Search me. And Safari’s 4 Top site view. Also worth a mention is Cool Iris; and Sun’s dead Looking Glass project if your interested in the history.
AT&T’s Pogo Browser
2d Cards, showing search history. Safari 4 uses Coverflow for the same functionality.
Useful – So is this Useful? They can be more efficient in showing more content than 2d views if careful attention is paid to layout. The third dimension for these cards is used to convey no additional information than a 2d sorted list, but distance away from the camera typically conveys some kind of sort order. On balance somewhat useful for interaction designers where visual searching through limited numbers of things is required.
Usable – From a usable perspective, these systems typically don’t rely on any camera movement beyond what we are already familiar with in 2d UI, (Zooming, X and Y translation.). So there’s no inherent complexity being added to these kinds of user experiences.
Desirable – From a desirability standpoint, the swift proliferation and success of Coverflow, carousels and systems like Cool iris is hard to deny in making our commonplace UI mechanisms slightly slicker.
There’s a community of information designers and developers out there working in scientific fields and on complex data visualization. They are building tools that enable very large data sets to be processed and visualized in flexible ways.
Some of the work of this group is fascinating; ground breaking stuff, and visually amazing. Every once in a while someone dabbles with 3d to ‘help’ visualize the complexity. The results are almost always spectacular failure of clarity. The raises an important issue about 3d UX of all kinds and that is, static representations of these networks are truly an unfair way to judge them. Unlike static 2d posters, which are designed, for comprehension from a static point in space and time, 3d becomes infinitely more valuable when it uses time (animation) and interactive movement around the 3d space to understand its topology.
Useful – Generally, Usefulness, I’d argue that these super complex visualizations of nodes, networks, social hierarchies and systems are ‘best case’ marginally valuable to a small group of IT professionals or social scientists.
Usable – From the perspective of Usability, they are disasters for all the reasons I outlined in the camera notes at the beginning – they literally require full freedom camera control. (Ed. common I’m being provocative here… someone send me a good example please).
Desirable – There is some aesthetic desirability to this approach for designers. I can’t help seeing these kinds of things and like a moth to flame, drawn in by the promise of ‘understanding’ data in a more visceral and immersive level. I think however on balance most consumers would look at these kinds of experiences and blanche.
3. 2d Cards on geometric space
Somewhere between, cards in metaphorical space and logical space is putting cards on geometric primitives which are not really 3d objects because there’s no added information to the additional dimensions this brings. Personally I hope to never see another 3d browser application that maps webpage’s or content onto cubes.
I’ve seen so many examples of this recently, There’s really nothing to say here but ‘stop it’. Why are you doing this???
There’s always some exception’s however. Here’s Absolut world a Paper vision extravaganza this physics powered UX, shows a dynamic site that has great music/art direction. Opening the cube’s is a really nice touch revealing nice shiny horizon/dioramas. Because the geometric cubes are objects not environments this approach to 3d UX is easy to understand and lots of fun to play with. I could even see this inspiring more productivity related scenarios.
Apple filed a bunch of patents with these schematics showing a 3d desktop last year. This patent is a bit mysterious but you can see again the idea that you are looking at the inside of a box with a floor, walls, ceiling and background.. 3d geometric primitives are shown on the tray at the bottom, utilizing ‘stacks’ of 2d content. Apple has been aggressive adopter of 3d with Time Machine, various dock visuals, Cover Flow and even specific spinner controls in the iPhone widget library. I doubt this design will see the light of day as described.
Bump top - 3d desktop
Really how much time do you spend organizing files on your desktop? The answer: lots more if you have something fun like Bump top to waste your time. The genius work here, is in the gestures, this product should have remained the cool demo it was four years ago, or it should have been a product 10 years ago, back when the desktop really was still a relevant place for innovation. Physics integration is nice but re-creating the imperfections of the real world (messy collisions) I can do without. Apple invented Piles, back in the early 90’s. Which is the coolest aspect of this whole product.
Useful – Is 2d content in Geometric space a useful approach? The ability to contain things and organize groups of things into relatively abstract containers like cubes is somewhat useful although frankly arbitrary. (The Absolut example is perhaps the best example here). Angled panes on walls reduce the real estate required for items but can create ugly skewing artifacts.
Usable – If the Camera remains static relative to the objects this approach is a simple extension of 2d techniques. The moment the camera starts navigating around or inside the geometric primitive all bets are off for people maintaining a useful mental model of the space.
Desirable – I’m not sure about the Desirable? Personally I don’t like the aesthetic of boxy environments too much. Time will tell if Bump top reaches any kind of commercial acceptance similar to the response it garnered from the audience at TED.
4. 2d Cards in ‘real’ space
Potentially one of the least explored ideas is putting 2d cards into real environments. ‘Acrossair’ has a nice demonstration of a way finding application for the New York Subway system that overlays cards (and arrows) onto the iPhone camera output. It uses compass and GPS information to determine where the camera is looking.
This is likely to be most explosive area of growth especially given the excitement around augmented reality applications. Take the amount of data that has real world data and its not hard to see applications of this everywhere from advertising, shopping, way finding, social networking to real estate.
One additional mention for Photosynth which takes 2d photos of things and creates a navigable 3d space. I have to say like everyone else when I first saw this I was amazed more by the ‘deep zoom’ than the stitching element between the photographs which sort of ‘recreates’ a three dimensional browsable space.
3d objects
1. 3d objects in metaphorical or virtual space
Obviously this is the realm of maps and games but its worth calling out a few of the many examples that continue to shape our digital experience.
Mapping applications like Google’s streetview or Bing maps are well on the way to providing a photo realistic version of our world that even 10 years ago would have been considered science fiction. The latest engines are doing automatic placement of 3d vegetation based on infra-red satellite images, actual weather information in the sky and perspective correction on the sides of buildings. This opens up the opportunity for virtual tourism. Almost full camera control is offered to the user that makes the systems clumsy to use from a humans eye line perspective. Today the content in the mapping application is mostly 3d buildings and vegetation but expect 2d content (call-outs, advertising, etc) to show up in the near future.
There have been and continue to be lots of attempts to do virtual shopping malls. Reading the blurb on these company’s sites is a little depressing. The unbridled marketing fueled enthusiasm they have for what they think is a new breakthrough in shopping experiences. In essence these sights are just fancy link catalogs.
This represents the next generation of retail, through a virtual environment; it’s original, it’s exciting and it has amazing potential, … The 3D visualization technology means consumers can enjoy the experience of shopping at any time of day or night.
Another attempt to make the web 3d, this one goes slightly beyond shopping and at least has a slightly distinct visual style. Really though how does this scale? I cant really imagine a mall that’s got hundreds of stores without them requiring, maps, directories and search which of course begs the question why bother with the whole spatial metaphor of a mall in the first place.
The R.U.S.E demo shown at E3 is a nice demonstration of multi-touch and 3d gaming, seriously watch the interactions in the demo and you’ll see an expert at work. This demo jockey is using a palette of learned gestures, from two-finger drag to two-finger spread to one finger arcing to change the camera perspective. Selection of objects is done with a two-finger poke drawing a selection rectangle. I’ve yet to play a RTS game on a multi-touch surface, its sure to be tiring, but this feels like the perfect use of all those gestures we’ve been seeing on surface tables for a couple of years now. Games are obviously fine with pushing the ‘usability’ of these systems, as learning them is part of the conquering the challenge of the mission.
Above, T.A.T’s recent ‘conceptual’ exploration of a 3d mobile phone UI. The UI is made to look physical (80’s plastic toy buttons and carousels). The camera moves around a ‘scene’ of widgets and various pivots are shown working in a kind of inspector carousel with filters. These guys have built their own rendering engine and tools pipeline to be able to do this all in real time and the output looks great especially for a mobile platform. The novelty of this makes the reaction of most technology people ‘wow’ but this is a deeply impractical UI.
When you watch this movie notice how the demo jockey has curved his finger to move the carousel… (cheater). This illustrates one of the fundamental issues with all these dimensional presentations of content and that is our input systems are still 2d. Fingers and mice don’t really map that well to manipulating 3d objects in space and until someone invests some kind of magic haptic feedback its never going to be easy.
Useful – Perhaps more than other examples this is highly dependent on the users, tasks and context. Generally 3d maps with 3d content (arrows, points of interest, landmarks) are obvious cases where this works perfectly.
I’d don’t think pursuing 3d objects as UI controls is a very good way to go (e.g. the plastic carousel in the TAT image above). The physical world is such a constraining starting point, but in limited contexts, (like the I-phone picker control that uses a 3d curved appearance whilst actually rendering all the text in 2d.) it might work as an affordance rather than a metaphor.
Usable – Again highly dependent on the model used, but this style suggests camera movement is a must.
Desirable – There are plenty of contexts where building a 3d world with 3d objects makes perfect sense, and has some aesthetic appeal in making it ‘beautiful’ and idealized. The dioramas style from the aesthetics post is an ideal style for this kind of thing.
3d objects in logical space
Typically the best applications of this are seen in the scientific visualization community where 3d objects are taken out of their context to aid in understanding them. This example from the cover of Tufte’s book, Envisioning information is a great example of providing just enough extra information in the abstract to aid in understanding the phenomena being described.
The same egregious use of overly complex data visualization from the 2d cards example also continues in 3d space. This tool isn’t exactly built for mainstream users but all the same super complex systems visualization continues to befuddle comprehension. (How many variables can a human process simultaneously?).
Useful – Useful, this can be the only way to convey the underlying structure of complex machinery, internal systems, biological systems, and hidden natural systems.) So yes its very useful for the scientific, engineering, educational and medical markets.
Check out systems like Catia for the complex engineering 3d software.
3d objects in real environments
Camera driven Augmented reality, the Wii, to Sony’s Eye toy, to Microsoft’s project Natal), Apple’s Iphone, Google’s G1 are using mostly camera’s to enable 3d input for the first time. Real 3d inputs involve moving a device like the Wii-mote or you’re phone or your arm around in X,Y and Z space and mapping real physical spatial inputs directly to outputs. Some aspect of this is Natural user interaction is destined to become part of our daily interactions with technology over the next few short years.
Total immersion did a demo 4 years ago that still is very impressive and blew my socks off at the time. Its taken a few years but we are now starting to see an explosion of toolkits that enable live video feed image processing to do motion tracking, even on relatively low powered devices. The result is 3d objects bursting out of video screens all over the web.
Sony at this years E3 showed Eye Pet which shows a 3d critter working with a PS3 and a Sony eye toy (web cam). There’s tons of interesting aspects to this its almost unbelievable. The camera is able to sense the ground plane and actually map the little pet character correctly into the scene.
This great project from some students showing off a pretty compelling game using a TEGRA development kit (more on this in the next article.). This is the first game I’ve seen using augmented reality through a mobile device, obviously it’s just the beginning.
Useful - There are so many potential applications of this from serious everyday productivity to entertainment to social that its hard to not see this as the most compelling use of 3d to date.
Usable - The obvious benefit of mobile systems for AR, is the camera being on the back of the device rendering the screen as a direct looking glass onto the real world, sort of ‘direct viewport manipulation’. Laptop or web cam form factors put the cameras facing the viewer, which inverts your inputs. Note the kid’s eye line is off center in the Eye toy video meaning they are looking at a TV to see the image we are seeing. This presents some input problems. No tactile feedback is another problem in this space that makes gesturing with a 3d device hard to judge.
Desirable – This is sufficiently magical that it stands as one of the longest standing most over-hyped technologies out there (along with VR). With 3d input devices and 3d outputs possible with head tracking there’s going to be some incredible things enabled that will challenge all of what we think we understand about U.X design. Who needs the full emersion geek headgear for that?
Final thoughts and tips
Some arguments can be made for 3d around efficiency of layout, overlapping content can take up less real estate, but that only works in the most camera constrained cases.
Research suggests that people can utilize spatial memory in the recall of objects placed in spaces but this can be true in 2d too, and the extra burden of users have to learn the physical layout of a space and manipulating objects in that space takes away more than it adds.
The screen and mice are both 2d devices. Multi-touch gives us a few extra degrees of control but its still best as a 2d input system.
3d works best when combined with animation and camera movement. This typically makes it ‘4d’ because the expectation is that greater comprehension of the UX comes through exposure and exploration of it over time.
A general rule of good information design is ‘reduction’ removing the un-necessary in favor of enhancing the important. 3d by nature can make that reduction task more complex and thus 3d designs tend to be more ‘complex’ visually. (Especially if they utilize the effects that make 3d worthwhile in the first place, lighting, shadows, material qualities etc).
3d organizational systems and user interfaces work well when they map to ‘simple’ metaphors but don’t take the metaphors too literally. Coverflow has reached widespread adoption because the underlying idea is incredibly simple ‘a stack of albums’. I don’t need camera movement or 3d input technologies to go next/back. Carousel’s borrow from the physical idea of a lazy-Susan device and similarly don’t need much in the way of explanation or flashy camera movements to work.
If you are using 2d inputs, consider not giving the user any real freedom to move the camera, or the target of the camera, other than pan, track and zoom.
Summary
I think there’s a palpable hunger for the next novelty in the consumer technology space. 3d graphics and effects are for the time being a source of that visual novelty, and will help new products differentiate themselves in very competitive markets. This more than any other factor is the central reason to use it. But do so with care, so many bad experiences have been created that a backlash against 3d is likely if we continue to abuse it.
Mainstream adoption of 3d inputs sensor technologies, (including augmented reality) and eventually 3d output technologies will make a huge difference to adoption of 3d as a useful and practical interface experience. But don’t ask me to put on special glasses or headgear.
I’m still hopeful that someone will come up with a holistic consumer friendly 3d user experience that actually works, and is more useful, usable and desirable than any 2d counterpart across a wide range of activities. It’s conceptual model will be instantly familiar, natural but not kitsch or cheesy with overuse of tired metaphors. It’s clever integration of spatial information will give me new and important insights into practical tasks and activities and its use of 3d aesthetics (lighting, physics, subtle animation and materials) will engage me emotionally the way great design always has. Here’s hoping.
The next and final part of this blog series is on the tools and technology part of the story, what technologies exist, do they scale to different platforms, what tools make it easy for creative’s of all kinds to push the status quo.
You’re buying a home and it’s the biggest investment of your life. Bigger than your car, your student loan, and even your collection of Fabergé eggs. Which house, where, and just how much you are willing to pay… it’s a complex battleground of both rational and emotional thoughts running around inside your head. And… as if that weren’t enough, it’s something you very well might be doing with another person, making the process even more complicated. As a result, you’ll spend countless hours, weeks and months wading through all kinds of information both online and off, and doing your own convoluted (though of course completely logical) version of managing and analyzing all that information.
Information availability has exploded in the real estate industry (as it has in all areas) in recent history and, as such, it has become increasingly harder to find, store, organize, share and analyze that information.
In our recent project with Pacific Northwest real estate brokerage John L. Scott, and sponsored in part by Microsoft, we tackled problems surrounding information management for home buyers and sellers. From that we will discuss 3 important lessons:
Access to large amounts of information is changing the roles and relationships of agents, brokers, buyers and sellers
There are significant challenges with supporting opinion-based data in real estate transactions
There is a need for better filtering and analysis tools
Access to large amounts of information is changing the roles and relationships of agents, brokers, buyers and sellers
These days, people have access to all kinds of information from many different sources. However, this doesn’t always (or even usually) mean that the information is relevant, makes sense, or is accurate/trustworthy. In our project with John L. Scott, we first set out to understand how this ‘information overload’ is impacting the roles and responsibilities of buyers, sellers, agents and brokers. To do so, we conducted one-on-one interviews with people from each of the domains. With buyers, the focus was to understand what information they were looking for, how they used it, and the sources they used to obtain it. With sellers, we were interested in what kinds of tools and support they were looking for in pricing, marketing and tracking the sale of their property. Lastly, for brokers and agents, discussions centered around their willingness and preparedness to address the changing expectations of their clients.
What we found was that both buyers and sellers are doing more of the information-gathering (and even more of the marketing in the case of sellers) themselves. It’s not surprising that they are feeling empowered with their new found knowledge and begin to practice their rights to make their own decisions. With more information, sellers and buyers are becoming more informed about the process, getting detailed property information and doing this all at their own pace.
As a result, agents are having to work harder and in new ways to maintaining their expert go-to relationships with their clients. Their value is shifting from that of information gatekeeper to one of trusted expert collaborator. As such, the tools need to change to support the expectations of the buyers and sellers and bring forward the value of the agents and brokers.
Encouraging agents to collaborate with sellers as consultants
Experience is still the best tool for understanding the data and the implications with regard to the market. This is good news for agents, but they will need to adjust how they approach and communicate this value to their clients. Collaboration is going to be a very important theme in the future of this industry.
Sam Chenaur, platform Strategy Adviser at Microsoft discusses the importance of data visualization and presentation of market data in an interview with Pat Giles, Vice President of Marketing and Information Technology at John L. Scott Real Estate.
Everyone wants to sell their home quickly and still get the best deal, which requires understanding market conditions, identifying and understanding competing homes, and marketing a house well. However, many people don’t know what their homes are worth—or think it’s worth more than the market does. By enabling sellers to monitor competing homes and the number of online showings on our Web site, we’re helping them to better understand the market and what their home is worth.
There are significant challenges with supporting opinion-based data in real estate transactions
Understanding the opinions of others can be a very helpful thing when it comes to shopping. We rely on other people’s experiences and judgments to predict how happy we will be with a purchase. Companies like Amazon and Netflix have recognized how enabling their customers to be completely candid about their products can increase customer trust. And now, even companies like Dell are getting on board. However unlike Amazon, Netflix or Dell, the ‘products’ we are talking about with real estate are all one-of-a-kind, bringing on an entirely new level of complexity.
While buyers of course want to know absolutely everything about a house, both good and bad, they aren’t really motivated to share any information they acquire. Other buyers are, in essence, their ‘competition’, while sellers are responsible for assessing the value. Purchasing a house is much more like purchasing something on eBay… you wouldn’t expect someone who is planning to snipe that Fabergé egg they’ve been looking for to complete their collection to post wonderful comments about it, verifying its authenticity and value for everyone to see.
There are also issues surrounding the broadcasting of information from sellers themselves. While they are the ones with the most information about a given home and all of its positives and negatives, as you would expect, they are motivated to highlight the good (legally-required disclosures, of course, require them to report some of the bad). Now, that’s not really any different than any company selling you a product, but when you mix extreme emotions with marketing, things can get complicated.
Which brings us to the agents, who, like their seller clients, are interested in ensuring that a home is portrayed in its best light, but they are also concerned with keeping their clients’ expectations in check… so feedback from buyers and other agents (their primary source of feedback currently), can actually be a great way to do that… as long as they are there to filter that feedback.
As you can see, there’s a tug of war in multiple directions that John L. Scott has to manage. With customer expectations for transparency on the rise, there is a balance that needs to be achieved, and each party needs to have their own views of the same information.
There is a need for better filtering tools to address the new abundance of information
It’d be easy to say that there’s an information overload problem, but it’s inevitable that more information will become available online as more data is associated with every object that touches the Web. Information about photos has gone from dimensions and date taken to ratings, geotags, number of favorites and annotations. Homes alike will continue to amass more metadata as they become social. So the success of an information rich application must rely on robust and personalized filtering tools.
Solution
John L. Scott saw an opportunity to provide buyers and sellers with new tools to help them with the issues mentioned above. Our solution was a WPF application and a complementary website designed to help buyers, sellers and agents through the process. These two components of the experience bring together information from several online sources such as the John L. Scott website, market data sources and other real estate services. With a more unified experience, John L. Scott will help foster a collaborative relationship between the sellers, buyers and agents.
Dynamic Data property finder
Home buyers/sellers can use live dynamic data to dial in exactly the right criteria for properties they are interested in. Adjusting one set of criteria (such as price range) shows them just how many properties are available in the categories of additional criteria such as bedrooms, square footage, etc. This allows them to filter to a smaller, but more relevant set of results.
Analytics
Home Sellers have access to tools that track property activity such as page and photo views. Agents can add comments to their client’s data to help them understand the data’s implications.
Buying/Selling Team
Buyers or sellers can involve friends & family as part of their ‘team’ to help them find market a home.
Activity Feed
Buyer and sellers can get up-to-date information such as property recommendations as well as communications with their team and agent all in one tool.
Landmarks
Buyers can see landmarks & add their own (such as their work, school, day care, etc.). Homes they are looking at will then display distances & commute times to those locations.
Save and Categorize homes
Buyers can save and categorize homes they are interested in (or not) into groupings that they define themselves.
In this first installment I’m going to address the ‘where are we’ question, but mostly from the perspective of visual language, rather than interaction design, I’ll deal with that in the next installment.
When interaction and visual designers take the time to understand and master the new visual techniques at their disposal with 3d UI, (cameras, lighting and shadows, materials and physics.) really exciting and delightful things can emerge. We’ve been collecting a tiny fraction of the great work out there and see various trends emerging with designers getting creative with the extra dimension. What’s important about these examples is that it becomes clear that 3d has broken through as a viable graphic styling language and is being adopted widely in marketing, promotion and cutting edge user experience design. We shamelessly often look to emerging visual trends for ways to make stimulate UX thinking beyond standard flat 2d GUI controls when its appropriate.
I trained in Graphic Information design before my Interaction design training so there’s a part of me, (the Tufte indoctrinated part) that wants to boil everything down to its clearest, purist form to make it more understandable, communicative and effective. However when creating ‘experiences’ for consumer software applications we are often tasked with more than just simply communication. We are asked to move the needle on ‘excitement’, ‘animation’, branding language and many other variables. 3d is certainly not going to be a panacea for interaction designers or visual designers but when done right some beautiful and uniquely digital styles can be very effective.
I’ve identified three distinct visual trends we’ve noticed with 3d, I call them “Chunky Fonts”, “Shiny horizons” and “Diorama’s”.
Visual Trend #1. “Chunky fonts”
Chunky fonts is a style that uses 3d typography as the primary visual ‘content’. I’ve seen some great examples of this all over the place, ( here and here and here are some great inspiration). In making the fonts work visually, it helps to chamfer the edges of the extruded font to get a nice edge light of the shapes themselves. Lighting is key to making the font work in 3d, so designers resort to something called Ambient Occlusion rendering which gathers shadows in every nook and cranny of the font. (more on this later). High end shaders (reflections and sub surface scattering) and 3 point lighting helps the front face of the font the visually dominant component. Making things appear to be physical and touchable objects, is part of the appeal of 3d and gaining in importance with the rise of more natural user interfaces.
Zune Marketing getting in on the action.
Visual Trend #2. “Shiny Horizons”
This second style has been in vogue in the software industry for a little too long in my personal opinion. Perhaps it started with Coverflow or the Matrix scene where they order ‘guns’ in the preloader environment.
You know it when you see it, a see of white or dark on an infinite horizon plane with objects on it that has a reflection directly beneath the objects. This has been used ‘everywhere’ from Mac OS taskbar where everything is mega glossy, to the Xbox 360 NDE.
It has gotten so commonplace that PowerPoint 12 included the ‘automatic’ floating reflections as a canned styling effect a couple of years ago. Still whilst it’s a bit passé its still works and is (to these eyes) undeniably attractive. I think the reason it works so well visually is that with the ground plane establishes immediately a sense of physicality and dimension which makes content even just 2d cards in space, look ‘special’, touchable.
Visual Trend #3. “Dioramas”
Dioramas are like highly crafted little ‘sets’ of models staged to look hyper-real or cute and cuddly, physical, dimensional and realistic right down to the tiniest details. I’ve seen this style become a fixture in marketing materials recently, especially motion graphics. One common aesthetic technique is to use a collage style of as part of this, found objects, paper stencils, even hand-drawn assets, this nicely takes the geeky edge off the sometimes excessively garish shiny metallic computer graphics.
Still one of the most impressive things I’ve seen accomplished in real time, Ecozoo (Papervision 3d.) The reason it works so well, (apart from amazing art direction) is nice ‘baked’ lighting in the textures, excellent camera work which explains the concept, great depth established with blurred foreground and background elements. I know enough about 3d to be blown away that this ever got done, I’d have to imagine the designer was a person who’d worked in the games industry, it goes to show that very little geometry (Even 50 polygons) can look awesome with great (albeit not real time) lighting.
Well if you think, the EcoZoo is cool, Little big planet takes it up about a dozen notches in terms of complexity and execution. Little big planet is just a visual feast of a game for the PS3. The little ‘sack’ guy plays in a wonderland of found objects, which behave with amazing physics. This game has won every art award the game industry has to throw at it.
Lots of little physical objects in little dioramas, in Sprint’s – ‘what’s happening now’ campaign.
GE’s eco-magination website using amazing 3d rendered illustrations to convey some nice information graphics. These photo-real little sets are lovingly animated and super cute.
These trends are important, regardless of the fact that the rendering for many of them is not realtime.
If you want to start building beautiful real-time 3d what are the areas of knowledge you are going to have to invest in and get really good at.?
Cameras
So it’s almost so obvious that its easy to overlook one of the things about 3d vs 2d that’s completely different is the ‘camera’. There’s obviously an ability in 2d design to move the objects in the view around, but only in 3d can you also move around the objects. This small detail conceptually makes a big difference to interactive 3d design and can potentially get us into a world of trouble with producing a simple to understand, easy to use interactive 3d navigational model. (More on this in Blog 4).
Top tips, First if you want to get really good at this, you should invest in a cinematography book (which will help with the lighting topic too) Film and Photography craft skills are incredibly relevant to many aspects of 3d CG. Some simple interactive techniques that we’ve seen work well are really subtle camera movement ‘breathing’ and ‘panning/tracking’ are two simple techniques to make a static screen state become more interesting. Breathing is simply moving the camera, the camera’s target, or animating the field of view of the camera in a simple loop. This only needs to be a tiny amount to make things appear more dynamic and interesting visually. It can have secondary benefits, for keeping the screen from ‘burn in’ and inviting the possibility of interaction. Panning and tracking just a little bit the camera in response to finger or mouse or sensor inputs also succeeds in quickly creating a delightful sense of direct manipulation.
Generally speaking in interactive work ‘on rails’ camera movements (key framed movements of the camera to absolute coordinates) are a mixed blessing. They can ‘simplify’ 3d interactions considerably but at the cost of creating something that can feel less dynamic and interactive as a whole.
Lighting and Shadows
Of all the tricks to master in designing for 3d, this is the one topic you have to really understand, and it’s a ‘deep’ topic. (invest in a good CG lighting book). Even simple cubes in 3d can become really amazing when you have really good lighting and shadows. Essentially, most existing real time 3d technologies can’t do anything like the above image in real-time, the image above probably took about a gazillion hours to render. But ‘wait’ you say –“I’ve played Unreal tournament, that’s real-time and they have amazing lighting”. They do indeed, but they have a ‘lighting engine’ that renders the lighting information back onto the textures during the art production process (this is called ‘baking’ the textures), so the lighting component is not done live and no performance penalty is paid. It’s very effective to bake the lighting as long as you don’t move the objects or change the lights, everything is fine. So I could get close to the above image in real-time with baked textures on a bunch of cubes and a ground plane, but I couldn’t move any of the objects or the lights.
This ‘baking-the-lights cheat’ is actually just one of about a 100 cheats like this that real time 3d demands you learn to achieve a good-looking result. Game engines, (typically the most advanced real-time 3d technology out there) use hundreds of different cheats like this, carefully chosen to suit the needs of the content. Big outdoor games have very different rendering engines to driving games which are quite different to fighting games, which are completely different to first person corridor shooting games all have tons of tradeoff’s made to enhance quality and boost performance.
Some tips on lighting… I briefly mentioned in the ‘Chunky Fonts’ trend, 3-point lighting for maximizing the dimensionality of geometry in your environment. Tutorials abound on 3-point lighting because conceptually it’s the same as lighting for normal photography and film so if you understand the concepts of Key, Fill and edge lighting you are set. With real time lighting today however there’s no real notion of lights being anything more than color being added in variable amounts to the shading of a surface, real time lights in most real time 3d engines don’t cast shadows - Its widely considered to computationally intense, but ‘there are cheats’ for this that are highly effective.
Materials and Shaders
Next to lighting, and intimately tied to lighting is working with ‘materials’. What you need to understand is that real time materials are more complex than simple textures. Textures are images that are mapped onto geometry. In some real-time engines out there you can add multiple kinds of textures to the same geometry and tell the rendering engine to use the texture image to do special things, make it bumpy, shiny, transparent etc. A simple multi-texture example would be to make a alpha blended mask which when applied as a ‘specular’ map would make some areas of the object look more shiny than others. Its super time consuming to create good looking multi-textured geometry and for real time its more complicated in that there are few standards I’m aware of for exporting all that nice setup work you do in the 3d program into a real time engine.
Real time shaders (and there are many different kinds of shaders, vertex, geometry and pixel are the most common) do funky math at render time to make stuff render differently. A complex shader could for example make 3d model render in what appears to be in a 2d illustration (Toon shaders). Shaders are a deep science but in real time computer graphics engines (again mostly games) there are almost always custom shaders written to render aspects of a scene. From a Art direction perspective scene shaders are how you can tweak the overall aesthetic of your CG, There’s a ton of subtle shaders at work in this image below including the way sunlight bouncing of some materials tends to ‘bloom’ and blow out the contrast in an photographic image.
If you are doing really high end looking 3d work you going to need to hire someone from the games industry (a shader artist, or technical artist) who understands how to make great shaders. The good/bad news is that to my knowledge there are no tools or engines commercially available to do this kind of work for real time UI contexts. I’ll take up this topic in the final part of this blog, because there needs to be!
Effects
Aesthetic effects like depth of field, blur, bloom glow are all examples of sophisticated shaders, most of these are the subtle secret sauce that will make 3d come alive in real time. Depth of filed effects in an interactive context can provide users with a sense for what literally has ‘focus’. Expect to see lots of moderately excessive use of these effects in the coming years.
Physics
- A good example of the ‘chunky fonts’ trend, with baked lighting on parts of the geometry, with Physics applied in real time.
Physics engines simulate what happens when multiple objects hit each other are tied together, spring loaded, attached to one-another etc. They simulate mass, velocity, friction, inertia, gravity and other forces so that dynamic interactions between objects and environments can be created. This dynamism is really well suited to some kinds of interactive applications (certainly entertainment) but can be also applied to more subtle elements of the user experience especially with regards to animation. Take the nice example of Apple’s iphone UI where you drag a list beyond the final entry in the list and you reveal darkness, when you release the list it springs back into the correct position. This creates a nice ‘secondary’ animation effect that reinforces in the users mind that you’ve reached the end of a list. In 3d this can be extended so that objects that are attached to other objects (say Place of interest icons on a map) can jiggle, (as if attached to a springy suction cup) when the map is manipulated this secondary motion perhaps would add a sense of playfulness to a more natural direct manipulation experience.
Watch this to see the full range of state of the art real-time physics effects (cloth, springs, collisions, explosions etc). There are lots of real time open source physics engines again not much in the way of a great commercial available solution.
Anti-Aliasing
So aliasing is the little jagged lines that appear to stair step on diagonal edges of polygons, anti-aliasing is the smoothing of the Jaggies. Of all the factors about real-time 3d, this one can be the most distasteful aesthetically. Most modern video cards can overcome this but normally the anti-aliasing (smoothing) has to be manually turned on by the user (you need the correct driver installed and go into the control panel and futz with settings to make this happen- frankly it may as well not exist at all). One advantage of high dpi screens (especially mobile devices) is that the pixel dots are so close together that optically you’re eye doesn’t notice un-anti-aliased edges as much.
Production Pipeline
Working in real time 3d is a tools workflow headache. Geometry, Lighting, Shaders, animation, physics can bend your mind around a whole set of new production challenges that are immensely time consuming. Most of the 3d tools I know are squirrelly feature-bloated monsters… wildly esoteric from a UI perspective and designed for people who seem understand what the correct value to type into the ‘light bounce co-efficient’ dialog. These ‘fictional’ users also seem to have infinite amounts of time to spend playing with the tools.
Best advice if you are embarking on a ‘real-time’ 3d interactive extravaganza, hire someone who’s set up complex production pipelines, these people are called Technical artists in the games industry and movie special effects studios. If you don’t have that luxury give yourself ‘triple’ the amount of time you’d think it would normally take you to accomplish something in 2d. The mantra should be realtime 3d stands for 3 x duration and 3x difficulty and 3x demanding.
Summary
So the ‘technology’ to do high quality real-time 3d with all sorts of incredible lighting and shading effects, does exist (we see it daily when we look at games) but today there isn’t a good ‘engine’ and perhaps more importantly ‘toolset’ for doing this for building UI. So even if you understand and have mastered all the topics above, you are basically SOL when it comes to making it dynamic and interactive. Today’s hardware on most PC’s is so powerful that ‘anything’ is possible from a creative perspective, any of the images in the blog could be real time rendered if there was a general purpose UI engine available to do that. That is part of the reason I’m writing this slightly bloated blog article. In my 3final blog post in this series I’ll explain a little bit of what I want someone to build to make it easy for an army of interactive and interaction designers to get busy defining some amazing next generation user experiences.
The next blog post is about why some conceptual, organizational and navigational models work and some don’t and the rise of augmented Reality. The final part of this series will be more about tools and technology engines available today with a call to action about what would be the ideal UI rendering engine?
Contact me if you want Artefact to design such a tool, right now that would be a dream project.
by 
by 
