What the Operating Room Could Teach Us About VR
If you spend any time following tech (and even if you don’t), you are likely aware of augmented or mixed reality and virtual reality. Immersing yourself in an artificial world, or teleporting to a different physical location, is indeed a pretty mind-blowing experience, and it’s only natural that gaming and entertainment are some of the first (and most visible) applications. But while AR, MR and VR seem new, and while many industries are still trying to wrap their heads around how to respond to this new technology, Intuitive Surgical has been pioneering its application in a highly specialized environment for well over a decade.
Intuitive Surgical is the global technology leader in minimally invasive robotically assisted surgery. Rather than make a large incision to expose internal anatomy, procedures performed by surgeons with Intuitive’s da Vinci Surgical System involve creating small incisions, like keyholes, through which several instruments, including a 3D camera, are inserted. Seated at the Surgeon Console and while viewing a live hi-res 3D video feed from inside the patient’s body, the surgeon manipulates the system’s master controls, which seamlessly translate the surgeon’s hand, wrist and finger movements into precise, real-time movements of the instruments attached to the robotic arms.
While VR and robotic assisted surgery may seem completely different, they share a core challenge – the need to maintain a connection between the real and the virtual worlds. And while in VR for entertainment that connection is desirable, in the O.R. that connection is vital. The reason for that challenge is a built-in tension: the more deeply you are immersed in one context (whether the virtual context in VR, or a surgical view inside a patient’s body) and the more natural the interactions, the harder it is to maintain the connection to the broader physical world. You need to stay focused on the job at hand (e.g. completing a complex procedure with a surgical robot or battling a robot army), and staying connected to the broader context (e.g. the condition of the patient and orchestrating the actions of others in the O.R. or making sure you don’t knock over the furniture in your room).
The da Vinci Xi Surgical System showing (left to right) the Vision Cart, Patient Cart, and Surgeon Console. (Photo credit)
We worked to resolve some of this tension in our exploration into the future of VR. But we have also encountered it when collaborating on specific, tangible projects with Intuitive Surgical. In our experience, there are several key principles that help manage this tension in the O.R. environment. With a few slight modifications to account for the different context, they can easily be applied to VR:
Keep a lifeline to maintain a connection between the virtual and the physical reality:
While a person is immersed in the virtual world, they must also maintain a connection to the broader real world context. Two questions can help you decide how to manage this tension: 1) what is the intent of the experience? and 2) what is the mechanism that ultimately establishes this connection?
In the case of robotic-assisted surgery, the environment is inherently collaborative and the intent is to maintain broader and persistent awareness of context to ensure the surgeon remains connected to the surgical assistant, anesthesiologist and others in the O.R. – all in consideration and service of the patient on the table. In this instance, opening up an audio channel to preserve the ambient awareness of sounds (beeps and bings) in the O.R. and to encourage communication between the surgeon and the rest of the team is an effective mechanism to maintain a connection without disrupting the primary task at hand.
In VR gaming, the objective may be to make the experience as immersive as possible (it’s just you at home with your VR gear). Here you may want to limit any lifeline to the real world to the bare minimum and have it emerge only at key points to prevent the user from harm, for instance. The mechanism in this case could be indicating wireframe outlines of the physical space of the room or objects in the room to help bridge the gap.
HTC Vive / Chaperone reveals a blue mesh in the user’s virtual space to notify them that they are in close proximity to a physical barrier. (Photo Credit)
Use contrast to distinguish between experience and UI controls:
The more convincing our ability to create realistic objects and environments, the more important it is that we appropriately distinguish between what is real and what is virtual. You can use contrast to help make the distinction:
In the surgical case, in an environment that uses a high-fidelity live 3D video feed of the abdominal cavity to immerse the surgeon in a procedure, simple graphic elements are an appropriate way to present UI. Simple shapes, primary colors, and planar surfaces are all used to effectively distinguish the environment (e.g. anatomy & surgical tools) from the UI (controls & settings).
In a high-fidelity, photo-realistic virtual environment – whether this be VR gaming or VR storytelling – contrast between geometrically complex and realistically rendered environments, creatures or objects and simple wireframes or primitive shapes is an effective way to differentiate the real from the virtual. For instance, a two-dimensional mesh plane or cube can help delineate the confines of a room. Floating planes or spheres rendered in a color that is alien in the VR environment may be a good way to present a UI component. You can also use proximity and persistence: e.g. a changing environment ahead of the user and in the distance (e.g. in a first-person shooter game), combined with UI components nearby – especially if those are persistent (i.e. remain in the same location).
Panasonic’s VR Cabin Configurator nicely differentiates UI from environment (Photo credit)
Be subtle and give space to manage the transition between the virtual and physical worlds:
The more immersive the environment, and the more deeply engaged someone is in what they are doing, the more jarring and intrusive any interruption. Being subtle and giving space or time for someone to reorient / transition between the virtual reality and the physical one will yield a better experience. Prompts that slowly emerge or gradually increase in intensity give the person time to adjust or reorient and are less intrusive.
In the case of the da Vinci System, visual prompts and alerts appear in the periphery rather than the center of the surgeon’s field of view. They are clear without being intrusive. Since the surgeon has a persistent audio feed, changes in that feed, like increased urgency in someone’s voice, or the change in tone, tempo or volume of an alert, are other effective ways to transition between virtual and physical.
Walking up to a person immersed in a VR game and tapping them on the shoulder is shocking (often for both people), but much less so if the player is aware that someone else is in the room and approaching. Similarly, an outline of a wall or piece of furniture gradually emerging is less disruptive than if it suddenly appeared at the last moment. Spatialized (3D) audio could also be used to indicate position – whether that be a person approaching behind the player, or whether they may be getting too close to a wall to their left. Light could also be used to indicate another person’s presence (imagine someone opening a door to a brightly lit corridor behind the player).
Be transparent to people in the physical space about the context of the person in the virtual one:
In the same way that it is beneficial for the person immersed in the virtual reality to be aware of others in the physical space, it is also important for people in the physical reality to have an awareness of the context of the person in virtual reality.
Surgical teams operate in a highly synchronized manner in the O.R. Awareness of each other’s actions and being able to anticipate next steps are key to efficient and safe procedures. Displaying the surgeon’s view on a monitor (and in many cases more than one) helps the team maintain awareness of the surgeon’s actions. This helps the team distinguish between the parts of the procedure that are routine versus the parts that are highly complex and require particular concentration, and helps avoid an ill-timed interruption. When a surgeon using a da Vinci System requests that an instrument be swapped or replaced, it natural for them to point at or move that instrument. But their intent isn’t obvious to others in the O.R. unless what the surgeon is looking at appears on a monitor that everyone can see.
Similarly, showing what someone is experiencing in VR helps bystanders be aware and determine if, when and how to engage. In the home having the user play in front of a video screen that displays the VR environment is an effective way to do this, and audio can also play a role. Many games are designed with a spectator mode which allows non-players to immerse themselves in the action and follow along.
Mirroring what a VR player sees on a monitor provides helpful context to others in the room (Photo credit)
This is trickier in the case of mobile VR – when an external screen or peripheral speakers aren’t available – especially in a social or public environment. VR is new enough that we have not yet established social norms. It took a while for us to adjust to publicly using headsets for our mobile phones, and at least there we had the distinct advantage of being able to make eye contact. Our Shadow and Light project explored this challenge and offers insights.
These challenges become even more complex when we start thinking about augmented or mixed reality as the ambiguity is greater. Whereas it is obvious that a person wearing a VR headset is experiencing something in the virtual world – and likely detached from the real world, in AR the user has digital content added on top of the physical world that is entirely invisible to the bystander. This asymmetry between the AR user’s experience and the bystander, combined with the lack of transparency, creates a wide range of problems that require an entirely new series of articles to be examined.
As technology advances and our ability to create ever more convincing virtual experiences increases, navigating the transitions between those worlds is shaping up to be a very important design challenge. These four principles, drawn from the world of surgical robotics, can act as useful guidelines to help us navigate this challenging and exciting opportunity.