Coming soon to a hospital near you: Holograms

• Head-mounted devices that depend on gaze, gestures and voice commands are the future of human-machine interfaces

• Mixed reality technology is shaking up medical training and surgery

• Developing countries have the opportunity to leapfrog highly regulated and risk-averse economies in terms of deploying mixed reality technology in the clinical setting

Humans are visual creatures. With nearly a third of the billions of neurons in our brains dedicated to the processing of visual information, our perception of reality is strongly determined by what we see. Digital technologies such as virtual and augmented reality (VR and AR) are thus looking to leverage our sense of sight to deliver alternate realities, bringing a new dimension to the old adage, ‘seeing is believing’.

Depending on the extent to which the real world is tuned out, VR and AR can be further subdivided into a spectrum of mixed reality (MR) applications. In addition to the gaming industry, the medical profession is also keen to adopt MR technology to assist clinicians in their work and improve healthcare.

“We believe that mixed reality is an entirely new platform that may redefine the computing experience and healthcare,” said Dr Simon Kos, Chief Medical Officer and Senior Director of Worldwide Health at Microsoft, who was speaking at the Asia Pacific MedTech Forum in Singapore on 7 November 2017.

Gazing into the future

Microsoft’s flagship MR device—the HoloLens—is already being deployed in hospitals around the world, said Dr Kos. Unlike a VR headset that completely immerses the user in a different world, the HoloLens allows the user to view holograms overlaid onto real physical space. This is made possible by the three computer chips that are embedded within the HoloLens’ frame—a central processing unit, a graphics processor and a holographic processing unit. A suite of sensors ranging from gyroscopes to ambient light detectors are also used to position holograms over a three-dimensional area.

As impressive as the hardware may be, Dr Kos highlighted that the most challenging part of making MR devices practical in a clinical setting was getting the human-machine interface right. Without a mouse, keyboard, or touchscreen, how does one interact with elements in the virtual world?

“[The HoloLens] relies heavily on gaze,” Dr Kos explained. “The device knows where you’re looking, and it will pre-process frames on either side, so if you happen to look around, it will spatially map the area and smoothly transition over.”

The device also responds to gesture and voice commands. A built-in computer algorithm allows the user to call out menus and interact with them by simply clicking in the air.

“When we think about what the future of computing look like… this is where we’re headed towards—more natural interactions with intelligent devices,” he said.

A mixed bag of applications

MR devices like the HoloLens have myriad uses in clinics and hospitals, starting with the education of medical students, said Dr Kos. Whereas in the past, human anatomy was taught using textbooks and plastic mannequins, the deployment of MR technology in the pedagogy of medicine now allows students to go beyond just studying the inert, two-dimensional structure of organs.

“We are able to now visualise a world in three dimensions, the way that it’s meant to be,” said Dr Kos. “Medical education has become highly interactive, highly immersive and even collaborative.”

Beyond the classroom, MR devices are already making an impact in surgical wards. Typically, before a patient undergoes surgery, some form of medical imaging has already been performed. Surgeons can now use those images to create three-dimensional holograms, viewed through the MR device, that help them plan and even rehearse the entire procedure beforehand.

“A radiologist and a neurosurgeon in New York are even overlaying, in real time, CT and MRI images onto the patient, while they’re doing neurosurgery, to triangulate exactly where the pathology is,” said Dr Kos, emphasising how radically MR technology is changing healthcare.

Getting a grip on reality

As with most new technologies, MR devices in the medical arena are still rather pricey. This might raise concerns that only hospitals in more affluent, developed countries will have access to this technology.

On the contrary, Dr Kos noted that the divide between developed and developing countries in terms of the adoption of MR technology has been relatively small.

“Ironically, developed countries that have highly regulated medical ecosystems and are risk-averse are slowing down the adoption process in their hospitals,” he said. “Meanwhile, in some areas like India, Brazil and Southeast Asia, we’re seeing some really exciting examples of this technology being put into practice.”

Dr Kos expects that the production costs of MR devices for medical applications will come down in the next few years, making them even more affordable to clinicians worldwide. At the same time, devices will become lighter for enhanced comfort, and design improvements will allow an even more seamless MR experience.

“This is an area that’s taking off, and there is no perfect device. There are so many diverse use cases that the only thing you have is the right device for what you choose to be doing at that point in time, and that is going to change over time,” he said. But one thing seems certain—MR technology has a foothold in our reality, and it’s not letting go anytime soon.