Technology is reshaping every aspect of our lives. Once a week in The Future Of, we examine innovations in important fields, from farming to transportation, and what they will mean in the years and decades to come.
The case was complicated: Shoulder arthroplasty, to deal with an advanced case of arthritis affecting the patient’s glenoid — the ball part of the ball-and-socket joint in the shoulder. To handle the case most effectively, the surgeon wanted assistance from the best. But the best was physically half a world away. What to do?
From his operating theater in France, orthopedic surgeon Thomas Gregory slipped on a Microsoft Hololens 2 headset and dialed up three colleagues in Brazil, Belgium, and South Africa. They walked through holograms of the patient and collectively talked through the surgery; peering along as Gregory opened the patient’s shoulder joint, Stephen Roche, Bruno Gobbato, and Jean Florin Ciornohac suggested different clamps and alternate pathways and observed Gregory’s technique. Together, they turned a bit of surgery into a technology showcase.
This surgery wasn’t some science fiction film, however, nor was it the dream of a PR spinmaster. It actually took place just a few weeks ago, in Avicenne AP-HP Hospital in Bobigny, France — and it’s only the beginning.
Today: AR surgery is already here
“Hololens is like a smartphone for surgery, it’s the performance and information tool that moves surgery into a new era,” Gregory explained, during a panel he organized with Microsoft to spread the word about the power of mixed reality for medicine. Much of what a surgeon does involves mental gymnastics, he said: Keeping mental pictures of 2D charts and X-rays while walking through steps in one’s head and watching out for red flags.
“It’s almost a GPS for the surgeon.”
Augmented reality can take some of the burden out of that — it’s more than just a fun game for Pokemon fans, in other words. AR can very effectively help doctors plan out a procedure and execute it as efficiently as possible.
“It’s almost a GPS for the surgeon,” said Tom McGuiness, executive vice president of healthcare for Microsoft. Igor Sauer, head of experimental surgery at Charité Hospital in Germany, is another enthusiastic supporter of the tech. A traditional surgeon keeps 3D images in her head, and compares them mentally to the real person lying in front of her, Sauer explained. That doctor has a clear need for image guidance, at the same time that she needs an unobstructed view of the work before her. AR can offer exactly that — not tomorrow or next year but right now.
There are other benefits as well, explained Massimo Chessa, cardiologist interventionist with San Donato Hospital in Italy. Images are far clearer than words, and help bridge language barriers you probably didn’t know were there.
“Surgeons speak a different language, a true 3D language,” Chessa said. “because they’re used to working with 3D structures. But I’m a cardiologist. I was trained as a 2D cardiologist, then I became a 3D interventionist.” Mixed reality allows him to speak the same language as his supporting surgeons, improving the quality of patient care.
Indeed, Gregory believes in the tech so much that he’s built a facility within his hospital to allow surgeons from anywhere in the world to become familiar with holographic best practices and the power of mixed reality.
Tomorrow: Training future doctors with entirely new tools
Assisted surgery seems like the ultimate realization of this technology, but it’s just the first step. Any surgeon (except perhaps Dr. Strange) will tell you they appreciate an extra set of eyes during a particularly challenging surgery. But what about extra information? Sauer said he and his experimental surgery department are investigating ways to overlay things like anatomical depth information directly into the field of view, “to give the impression of shining a flashlight through the organ while examining it,” he said.
“Almost like having augmented eyes,” Gregory added.
Other intriguing advances lie in teaching, which necessitates observation. This is done traditionally in person: Think of the classic Rembrandt painting, “The Anatomy Lesson.” Sauer said he had his team have been working on volumetric recordings of surgery, allowing future students to not just study videotape but to walk around a three-dimensional virtual recreation of the operating theater itself.
But images alone aren’t sufficient training, especially for more specialized forms of surgery. You learn by doing, right? Virtual reality seems like an ideal platform to let a would-be surgeon hone his skills, but without a sense of touch, without being able to feel the scalpel penetrate the skin or the clink of a blade on bone, a piece of the puzzle is lost.
“It’s almost like having augmented eyes.”
Enter haptic feedback, the same tech that lets you feel the rumble of the road under your wheels in a driving game. FundamentalVR uses the advanced form of that — kinesthetic feedback, versus mere cutaneous haptics — to let a surgeon feel what it’s like to surgery within a submillimeter of accuracy, using a variety of off-the-shelf devices, from gloves to haptic arms.
“The precision we need in surgery is a different level than what you need to simulate picking up a ball or switching on a light,” CEO and Co-founder Richard Vincent told me. He too cited work on building a virtual version of the patient: Ideally, by the time a surgeon is ready to operate on the physical you, he’s cut the virtual version a dozen times. “If I can’t teach you what it feels like to do something wrong, and to do it right, how can you possibly transfer into the real world and do it?”
Further down the road, the intersection of cutting-edge technologies will enable a complete transformation in the literal cutting edge of the operating room.
“A.I. and extended reality could fundamentally change the way we perform certain procedures,” Sauer theorized. Imagine the possibilities of real-time spatial detection of organs, as artificial intelligence analyzes a patient on a table, highlighting the outline of the liver for a surgeon to ensure that an already delicate procedure is done as exactly as possible. Fundamental new forms of treatment could be possible. In the near future, doctors will be able to move around a scalable 3D, virtual version of the patient, studying a problem from all angles, before devising the least invasive, safest procedure possible. The actual surgery could be done by robots, or at least controlled, precise robotic arms. That’s way down the road, though … right?
“This of course is nothing we will do tomorrow,” Sauer said. “But maybe the day after.”
