Skip to main content

The future of surgery: AR, VR, and virtual learning will upend modern medicine

surgeons wear Hololens 2 headsets in the OR
Augmented reality is already making inroads in operating rooms — and the future of this tech is transformative. Microsoft

Technology is reshaping every aspect of our lives. Once a week in The Future Of, we examine innovations in important fields, from farming to transportation, and what they will mean in the years and decades to come. 

The case was complicated: Shoulder arthroplasty, to deal with an advanced case of arthritis affecting the patient’s glenoid — the ball part of the ball-and-socket joint in the shoulder. To handle the case most effectively, the surgeon wanted assistance from the best. But the best was physically half a world away. What to do?

From his operating theater in France, orthopedic surgeon Thomas Gregory slipped on a Microsoft Hololens 2 headset and dialed up three colleagues in Brazil, Belgium, and South Africa. They walked through holograms of the patient and collectively talked through the surgery; peering along as Gregory opened the patient’s shoulder joint, Stephen Roche, Bruno Gobbato, and Jean Florin Ciornohac suggested different clamps and alternate pathways and observed Gregory’s technique. Together, they turned a bit of surgery into a technology showcase.

This surgery wasn’t some science fiction film, however, nor was it the dream of a PR spinmaster. It actually took place just a few weeks ago, in Avicenne AP-HP Hospital in Bobigny, France — and it’s only the beginning.

With colleagues from half a world away watching through his eyes, Thomas Gregory performs orthopedic surgery.

Today: AR surgery is already here

“Hololens is like a smartphone for surgery, it’s the performance and information tool that moves surgery into a new era,” Gregory explained, during a panel he organized with Microsoft to spread the word about the power of mixed reality for medicine. Much of what a surgeon does involves mental gymnastics, he said: Keeping mental pictures of 2D charts and X-rays while walking through steps in one’s head and watching out for red flags.

“It’s almost a GPS for the surgeon.”

Augmented reality can take some of the burden out of that — it’s more than just a fun game for Pokemon fans, in other words. AR can very effectively help doctors plan out a procedure and execute it as efficiently as possible.

“It’s almost a GPS for the surgeon,” said Tom McGuiness, executive vice president of healthcare for Microsoft. Igor Sauer, head of experimental surgery at Charité Hospital in Germany, is another enthusiastic supporter of the tech. A traditional surgeon keeps 3D images in her head, and compares them mentally to the real person lying in front of her, Sauer explained. That doctor has a clear need for image guidance, at the same time that she needs an unobstructed view of the work before her. AR can offer exactly that — not tomorrow or next year but right now.

There are other benefits as well, explained Massimo Chessa, cardiologist interventionist with San Donato Hospital in Italy. Images are far clearer than words, and help bridge language barriers you probably didn’t know were there.

“Surgeons speak a different language, a true 3D language,” Chessa said. “because they’re used to working with 3D structures. But I’m a cardiologist. I was trained as a 2D cardiologist, then I became a 3D interventionist.” Mixed reality allows him to speak the same language as his supporting surgeons, improving the quality of patient care.

Indeed, Gregory believes in the tech so much that he’s built a facility within his hospital to allow surgeons from anywhere in the world to become familiar with holographic best practices and the power of mixed reality.

Tomorrow: Training future doctors with entirely new tools

Assisted surgery seems like the ultimate realization of this technology, but it’s just the first step. Any surgeon (except perhaps Dr. Strange) will tell you they appreciate an extra set of eyes during a particularly challenging surgery. But what about extra information? Sauer said he and his experimental surgery department are investigating ways to overlay things like anatomical depth information directly into the field of view, “to give the impression of shining a flashlight through the organ while examining it,” he said.

“Almost like having augmented eyes,” Gregory added.

Other intriguing advances lie in teaching, which necessitates observation. This is done traditionally in person: Think of the classic Rembrandt painting, “The Anatomy Lesson.” Sauer said he had his team have been working on volumetric recordings of surgery, allowing future students to not just study videotape but to walk around a three-dimensional virtual recreation of the operating theater itself.

But images alone aren’t sufficient training, especially for more specialized forms of surgery. You learn by doing, right? Virtual reality seems like an ideal platform to let a would-be surgeon hone his skills, but without a sense of touch, without being able to feel the scalpel penetrate the skin or the clink of a blade on bone, a piece of the puzzle is lost.

“It’s almost like having augmented eyes.”

Enter haptic feedback, the same tech that lets you feel the rumble of the road under your wheels in a driving game. FundamentalVR uses the advanced form of that — kinesthetic feedback, versus mere cutaneous haptics — to let a surgeon feel what it’s like to surgery within a submillimeter of accuracy, using a variety of off-the-shelf devices, from gloves to haptic arms.

“The precision we need in surgery is a different level than what you need to simulate picking up a ball or switching on a light,” CEO and Co-founder Richard Vincent told me. He too cited work on building a virtual version of the patient: Ideally, by the time a surgeon is ready to operate on the physical you, he’s cut the virtual version a dozen times. “If I can’t teach you what it feels like to do something wrong, and to do it right, how can you possibly transfer into the real world and do it?”

Further down the road, the intersection of cutting-edge technologies will enable a complete transformation in the literal cutting edge of the operating room.

“A.I. and extended reality could fundamentally change the way we perform certain procedures,” Sauer theorized. Imagine the possibilities of real-time spatial detection of organs, as artificial intelligence analyzes a patient on a table, highlighting the outline of the liver for a surgeon to ensure that an already delicate procedure is done as exactly as possible. Fundamental new forms of treatment could be possible. In the near future, doctors will be able to move around a scalable 3D, virtual version of the patient, studying a problem from all angles, before devising the least invasive, safest procedure possible. The actual surgery could be done by robots, or at least controlled, precise robotic arms. That’s way down the road, though … right?

“This of course is nothing we will do tomorrow,” Sauer said. “But maybe the day after.”

Editors' Recommendations

Jeremy Kaplan
As Editor in Chief, Jeremy Kaplan transformed Digital Trends from a niche publisher into one of the fastest growing…
Huawei finds its niche with the sporty Watch GT Runner
Huawei Watch GT Runner on wrist.

By focusing on a specific niche, Huawei may have found an audience for its latest smartwatch, the Watch GT Runner. It’s a spinoff of Huawei’s classier Watch GT 3 smartwatch, but as the new watch’s name suggests, it's targeted squarely at runners.

It’s a technically impressive bit of hardware, and the software is very good, so when you consider it as an alternative to other running watches, the Huawei idiosyncrasies that frustrate on the Watch GT 3 become less of a problem here. I've been trying it out and here are my thoughts.
A light touch
The Huawei Watch GT Runner is light -- just 51 grams with the very flexible silicone strap -- and that makes it comfortable to wear all day. The 46mm case is quite big, but at 11mm thick, it never feels that ungainly. For comparison, the new 47mm Garmin Fenix 7 weighs 79 grams and is nearly 15mm thick. The lightness comes from the polymer fiber case, which is given some visual appeal with a ceramic bezel and titanium crown. It’s also worth noting the huge amount of adjustment on the strap that allows it to be worn both under and over clothing.

Read more
Do you need a smart toothbrush?
Oral-B iO Series 9 Smart Toothbrush in hand

When will we come to a time when every accessory in our home is smart? Is there a limit to what should actually be smart? Let's talk about a category that may not be what you traditionally think of as a smart device -- the toothbrush.

I'm all for having smarter health products and having the best technology in our hygiene products (like bidets), but I've not put a lot of thought into how, or why, my toothbrush is smart. It's time to dive in and see if it's even worth it.

Read more
How Hawk-Eye cameras are making football fairer and faster than ever
Hawk-Eye goal line camera

Football can be a tough sport -- and nowhere more than at its most elite level where highly trained players compete for gridiron glory. There is a lot at stake, and a lot that can go wrong, too. From calls that are tough to make in real-time to the ever-present risk of season-ending injuries, you need a whole lot of eyes on the game to ensure that it runs smoothly.

Hawk-Eye is a company that's there to lend an automated assist. Used in an ever-growing number of sports, including the NFL, Hawk-Eye's tech consists of synchronized multi-angle cameras that can help track large numbers of data points on the sports field.

Read more