Skip to main content

The future of surgery: AR, VR, and virtual learning will upend modern medicine

surgeons wear Hololens 2 headsets in the OR
Augmented reality is already making inroads in operating rooms — and the future of this tech is transformative. Microsoft

Technology is reshaping every aspect of our lives. Once a week in The Future Of, we examine innovations in important fields, from farming to transportation, and what they will mean in the years and decades to come. 

The case was complicated: Shoulder arthroplasty, to deal with an advanced case of arthritis affecting the patient’s glenoid — the ball part of the ball-and-socket joint in the shoulder. To handle the case most effectively, the surgeon wanted assistance from the best. But the best was physically half a world away. What to do?

From his operating theater in France, orthopedic surgeon Thomas Gregory slipped on a Microsoft Hololens 2 headset and dialed up three colleagues in Brazil, Belgium, and South Africa. They walked through holograms of the patient and collectively talked through the surgery; peering along as Gregory opened the patient’s shoulder joint, Stephen Roche, Bruno Gobbato, and Jean Florin Ciornohac suggested different clamps and alternate pathways and observed Gregory’s technique. Together, they turned a bit of surgery into a technology showcase.

This surgery wasn’t some science fiction film, however, nor was it the dream of a PR spinmaster. It actually took place just a few weeks ago, in Avicenne AP-HP Hospital in Bobigny, France — and it’s only the beginning.

With colleagues from half a world away watching through his eyes, Thomas Gregory performs orthopedic surgery. Image used with permission by copyright holder

Today: AR surgery is already here

“Hololens is like a smartphone for surgery, it’s the performance and information tool that moves surgery into a new era,” Gregory explained, during a panel he organized with Microsoft to spread the word about the power of mixed reality for medicine. Much of what a surgeon does involves mental gymnastics, he said: Keeping mental pictures of 2D charts and X-rays while walking through steps in one’s head and watching out for red flags.

“It’s almost a GPS for the surgeon.”

Augmented reality can take some of the burden out of that — it’s more than just a fun game for Pokemon fans, in other words. AR can very effectively help doctors plan out a procedure and execute it as efficiently as possible.

“It’s almost a GPS for the surgeon,” said Tom McGuiness, executive vice president of healthcare for Microsoft. Igor Sauer, head of experimental surgery at Charité Hospital in Germany, is another enthusiastic supporter of the tech. A traditional surgeon keeps 3D images in her head, and compares them mentally to the real person lying in front of her, Sauer explained. That doctor has a clear need for image guidance, at the same time that she needs an unobstructed view of the work before her. AR can offer exactly that — not tomorrow or next year but right now.

There are other benefits as well, explained Massimo Chessa, cardiologist interventionist with San Donato Hospital in Italy. Images are far clearer than words, and help bridge language barriers you probably didn’t know were there.

“Surgeons speak a different language, a true 3D language,” Chessa said. “because they’re used to working with 3D structures. But I’m a cardiologist. I was trained as a 2D cardiologist, then I became a 3D interventionist.” Mixed reality allows him to speak the same language as his supporting surgeons, improving the quality of patient care.

Indeed, Gregory believes in the tech so much that he’s built a facility within his hospital to allow surgeons from anywhere in the world to become familiar with holographic best practices and the power of mixed reality.

Tomorrow: Training future doctors with entirely new tools

Assisted surgery seems like the ultimate realization of this technology, but it’s just the first step. Any surgeon (except perhaps Dr. Strange) will tell you they appreciate an extra set of eyes during a particularly challenging surgery. But what about extra information? Sauer said he and his experimental surgery department are investigating ways to overlay things like anatomical depth information directly into the field of view, “to give the impression of shining a flashlight through the organ while examining it,” he said.

“Almost like having augmented eyes,” Gregory added.

Other intriguing advances lie in teaching, which necessitates observation. This is done traditionally in person: Think of the classic Rembrandt painting, “The Anatomy Lesson.” Sauer said he had his team have been working on volumetric recordings of surgery, allowing future students to not just study videotape but to walk around a three-dimensional virtual recreation of the operating theater itself.

But images alone aren’t sufficient training, especially for more specialized forms of surgery. You learn by doing, right? Virtual reality seems like an ideal platform to let a would-be surgeon hone his skills, but without a sense of touch, without being able to feel the scalpel penetrate the skin or the clink of a blade on bone, a piece of the puzzle is lost.

“It’s almost like having augmented eyes.”

Enter haptic feedback, the same tech that lets you feel the rumble of the road under your wheels in a driving game. FundamentalVR uses the advanced form of that — kinesthetic feedback, versus mere cutaneous haptics — to let a surgeon feel what it’s like to surgery within a submillimeter of accuracy, using a variety of off-the-shelf devices, from gloves to haptic arms.

“The precision we need in surgery is a different level than what you need to simulate picking up a ball or switching on a light,” CEO and Co-founder Richard Vincent told me. He too cited work on building a virtual version of the patient: Ideally, by the time a surgeon is ready to operate on the physical you, he’s cut the virtual version a dozen times. “If I can’t teach you what it feels like to do something wrong, and to do it right, how can you possibly transfer into the real world and do it?”

Further down the road, the intersection of cutting-edge technologies will enable a complete transformation in the literal cutting edge of the operating room.

“A.I. and extended reality could fundamentally change the way we perform certain procedures,” Sauer theorized. Imagine the possibilities of real-time spatial detection of organs, as artificial intelligence analyzes a patient on a table, highlighting the outline of the liver for a surgeon to ensure that an already delicate procedure is done as exactly as possible. Fundamental new forms of treatment could be possible. In the near future, doctors will be able to move around a scalable 3D, virtual version of the patient, studying a problem from all angles, before devising the least invasive, safest procedure possible. The actual surgery could be done by robots, or at least controlled, precise robotic arms. That’s way down the road, though … right?

“This of course is nothing we will do tomorrow,” Sauer said. “But maybe the day after.”

Editors' Recommendations

Jeremy Kaplan
As Editor in Chief, Jeremy Kaplan transformed Digital Trends from a niche publisher into one of the fastest growing…
ClipDart is an on-demand barber app aimed at people of color
ClipDart founder, Kyle Parker.

It’s funny how we can take certain things for granted, like haircuts. Over the course of more than 50 years of living in different cities, different neighborhoods, or even visiting different countries, not once have I ever worried about whether I could find someone who could cut my hair the way I liked. Then again, I’m white.

But if you’re a person of color, it can be an entirely different experience. That’s what Kyle Parker discovered when he left his hometown of Chicago in 2013 to attend Grinnell College in Grinnell, Iowa, population 9,031. While 24% of Grinnell College’s students identify themselves as people of color, fewer than 10% of residents of the city of Grinnell would say the same of themselves.

Read more
Circular confirms its $259 smart ring is coming to the U.S.
best wearables of ces 2022 circular ring

The Circular smart ring is finally going to be available for pre-order on Sunday, February 27, via the Circular website and will cost $259. The wearable tech will be available for presale in European countries (France, Germany, the U.K., and Italy,) the United States, Australia, Hong Kong, and Singapore. Pre-orders will go live at 1:30 p.m. ET on Sunday, February 27. Those who pre-order the smart ring should expect delivery between April and June 2022, according to a Circular press release.

Circular doesn't clarify what ring sizes will be available when presales go live, however, the company has said that seven sizes for both men and women will be available. Digital Trends has reached out for clarification on the available sizes, and will update this article when we hear back. The Circular smart ring also comes in four different colors that can be switched out with replaceable outer shells: Black, rose gold, silver, and gold.

Read more
How to take an ECG with your Apple Watch and see irregular heart notifications
ecg app apple watch

The ECG app is one of the most vital features of the Apple Watch, allowing you to see an electrocardiogram of your heart whenever you want. Along with this, the Apple Watch can notify you of irregular heart rhythms.

Read more