Harnessing the power of AI to help revolutionize Olympic-level figure skating

American figure skater Andrew Torgashev was at an invitation-only camp organized by U.S. Figure Skating not long ago, a chance for elite athletes preparing for the high-level Grand Prix season to work out any flaws in their performances.

He tried a quad toe loop, a four-revolution jump where skaters launch from the back outside edge of their blade with the help of a toe pick from the other foot. The jump itself is the easiest of skating’s six primary jumps, but four revolutions ups the difficulty.

To the naked eye, Torgashev landed it perfectly. To the camera watching him, Torgashev landed a quarter-revolution short.

It was a small error, to be sure, but the kind that could make the difference in whether Torgashev medals in a sport where tenths and even hundredths of a point matter. And he knew instantly thanks to feedback he received from the camera – or, more accurately, the app it was running, designed by a pair of computer whiz kids with no background in skating, but who saw in the sport a chance to not only help athletes in training but perhaps someday assist in the actual scoring of competitions.

The app is called OOFSkate, and powered by AI technology it analyzes from a tablet or mobile phone a skater’s jump height, rotation speed, airtime and even landing quality. It provides skaters with feedback without having to wear sensors or other technology.

“Our vision for the system is to automate the technical calling of the sport,” said Jerry Lu, who along with his old college roomate, Jacob Blindenbach, have built out the system. “This manifests itself in a combination of using AI-assisted computer vision, but also the knowledge of figure skating, essentially taking out the stuff that should be judged without subjectivity.”

In other words, let humans judge the artistic side of the sport. Let computers handle the technical stuff.

“What are the things that go into a particular jump? We’re trying to measure those things as a semi-automated technical assistant,” Lu told The Associated Press in an interview ahead of this week’s Grand Prix Final, one of the premier events in the sport. “But also, it’s a coaching tool that teachers around the country can use to evaluate their own athletes.”

The system itself seems remarkably simple: It uses a phone or tablet camera to capture a skater in motion, then overlays the key points of a jump or spin – the idealized version of a given element – and records the metrics that technical panels typically use.

Instantly, a coach or judge can know whether a skater completed three full turns of a triple lutz, or landed on the correct blade edge for a salchow. They can know how high the skater jumped, which is one of the judging criteria, and how fast they were spinning.

It also helps the individual skater, who can marry the jump they have just done in practice against what they may have done in the past, or even compare the way they are performing any given element against the way the best have performed it.

In the case of U.S. Figure Skating, a team library will allow them to compare their skills to benchmark data collected over the years.

“So if I’m Andrew Torgashev and I’m at Champs Camp,” Lu said, “I can do a quad toe loop and can compare it against myself, along with all the other athletes that have executed this. ‘Am I spinning as well as Mikhail Shaidorov? Or a previous version of myself?'”

Lu and Blindenbach met while studying at the University of Virginia, where both were swimmers. They were drawn to ways emerging technology could help athletes in the water, such as accelerometers that better measure an athlete’s performance.

They went separate ways upon graduation, Lu to the MIT Sports Lab in Massachusetts and Blindenbach to Columbia in New York to specialize in artificial intelligence. But they stayed in touch, thinking up ways they could implement technology in other sports.

“We thought this would be a way to push the Olympic sports we love,” Blindenbach said.

It was NBC, which has the broadcast rights to the Summer and Winter Games in the U.S., that ultimately brought the pair into the figure skating world. The network asked them to conceive of some kind of technology that could help its analysts, former Olympians Tara Lipinski and Johnny Weir, with providing their commentary in real time.

U.S. Figure Skating got on board with the project, realizing its potential to help train its high-level athletes. Olympic skaters Jason Brown and Alysa Liu, along with her coach, Massimo Scali, have provided feedback. And almost weekly, Lu or Blindenbach head to the revered Skating Club of Boston, where they are able to put the latest iteration of their system to the test.

It could be a revolutionary moment in a sport that often changes at a glacial pace.

“At least from what I can tell, they all seem very open,” Lu said. “They love the sport, first of all. Whatever we can do to help the sport grow and become better, they would do. They are very inclusive of us being there, and shadowing them throughout the day – How do they train, how do they interact with coaches? It’s an artistic sport but it’s also an athletic sport.”

The name OOFSkate originally came from what skaters tend to say after seeing their feedback from a jump – as in, “Oof, that wasn’t very good!” Then, one of the folks at U.S. Figure Skating gave it a second meaning: “Obsessed over form.”

Indeed, the ability of OOFSkate to take the subjectivity out of judging form is intriguing, much like similar technology has taken the subjectivity out of line calls in tennis, and promises to take the same subjectivity out of balls and strikes in baseball.

“If someone under-rotates,” Blindenbach explained, “that should always be called. There shouldn’t be a missed call or a controversy because something doesn’t play out. Sometimes a position makes it hard (for a judge) to see if they’re on an edge or off an edge on a lutz or a flip. We hope that AI can make the sport more fair.”

Lu and Blindenbach are wary about looking too far ahead, though. Hawk-Eye, the system that uses high-speed cameras to track the flight of tennis balls and determine whether they land inbounds or out, was pioneered in the early 2000s, and just this past year did the All-England Club decide to replace human line judges with the system at Wimbledon.

When it comes to the Olympics, the fact that Omega is its official data provider provides another obstacle in implementation.

So given the slow pace of technological adoption, Lu and Blindenbach are focused for now on fine-tuning the system to help coaches, athletes and commentators better do their jobs, especially as many prepare for the Milan Cortina Olympics in February.

“We don’t want to step on toes,” Blindenbach said. “When you go fully AI and take the human out of the loop, people generally get mad, and the results are poor. We want to assist. If that’s jump height or rotation, or if someone under-rotates by a quarter, these are easy things to do with AI, relatively speaking, compared to trying to capture something artistic. That’s where we see ourselves.”