Robot watches how-to videos and becomes an expert surgeon

(Credit: Johns Hopkins University)

BALTIMORE — How-to videos are great for humans looking to pick up some quick skills. Now, it turns out they’re also great for robots looking to become amazing surgeons. In a groundbreaking step towards robotic autonomy, researchers have developed an AI system that can carry out complex surgeries with the same precision as seasoned human doctors. The secret? Letting the robots learn by watching the pros do it first.

Traditionally, programming robots to perform even the simplest surgical maneuvers has required painstakingly hand-coding each individual movement. Now, the team from Johns Hopkins University, led by Axel Krieger, has cracked the code using a revolutionary technique called imitation learning.

“It’s really magical to have this model and all we do is feed it camera input and it can predict the robotic movements needed for surgery,” Krieger says in a media release. “We believe this marks a significant step forward toward a new frontier in medical robotics.”

The researchers trained their model on a trove of footage recorded by wrist-mounted cameras on the popular da Vinci Surgical System robots. With nearly 7,000 of these robots deployed worldwide, and over 50,000 surgeons trained on the platform, the team had a vast archive of surgical procedures to draw from.

“All we need is image input and then this AI system finds the right action,” explains lead author Ji Woong “Brian” Kim. “We find that even with a few hundred demos the model is able to learn the procedure and generalize new environments it hasn’t encountered.”

The model was able to master three fundamental surgical tasks: needle manipulation, tissue lifting, and suturing. In each case, the robotic performance was on par with human doctors.

“Here the model is so good learning things we haven’t taught it,” Krieger says. “Like if it drops the needle, it will automatically pick it up and continue. This isn’t something I taught it do.”

This breakthrough, presented at the Conference on Robot Learning in Munich, could pave the way for a future where robots can autonomously perform complex surgeries, reducing medical errors and achieving unprecedented precision.

“What is new here is we only have to collect imitation learning of different procedures, and we can train a robot to learn it in a couple days,” Krieger explains. “It allows us to accelerate to the goal of autonomy while reducing medical errors and achieving more accurate surgery.”

With this powerful imitation learning approach, the researchers are already working to train robots for full surgical procedures, not just individual tasks. The implications for the future of robotic medicine are nothing short of revolutionary.