Tonight’s Bucks-Lakers NBA game will feature real-time data overlays

Second Spectrum is a machine learning company that has been quietly helping drive the NBA toward the fast-paced, three-point heavy experience that it has become today. The company uses cameras to track each game and algorithms to analyze every action on the floor. With partnerships with just about every team in the NBA, the company provides analytics that have helped identify inefficiencies in the game and shape the way teams play.

While one might imagine that Second Spectrum has every inch of the court covered in cameras, the company’s CEO Rajiv Maheswaran told Engadget, “The secret is really in the software.” The company has anywhere from eight to ten cameras in the rafters of every NBA arena. Most of the tracking work is done by algorithms and computer vision technology.

“What we have done is automate the identification of things like pick and roll, pick and roll defense, off-ball screens,” Maheswaran said. “We have AI-generated identification of all of these complex on-court actions.” That tracking information is turned into contextual data. It can calculate the likelihood of a player scoring or grabbing a rebound is, how open a player is and more.

Second Spectrum shares much of that information with teams in a database format, but the company has also turned it into visualizations that can be overlayed on top of game action. That started in part in partnership with the Los Angeles Clippers and resulted in the development of CourtVision — an augmented experience that takes pertinent information and turns it into a real-time visual overlay.

“We’ve had teams who have said, ‘We’d like to present information to our players and coaches this way,'” Maheswaran explained. “At the end of the day, teams watch a lot of video and players watch a lot of video. Augmentation is a way to make it not only more engaging but more informative.”

The visuals of CourtVision used by the Clippers will be on display on ESPN3 as Full Court Press and will give viewers at home multiple ways to watch the game depending on their interests.

First, there’s Coach Mode, which will draw diagrams of plays as they develop right on the court. It will also show when players are open identify switches on defense, among other areas of emphasis. “There’s a lot of things going on that your eye might not catch if you’re always watching the ball. Players are running interesting off-ball screens and doing interesting actions,” Maheswaran said. “The animations and graphics point your eyes to that.”

Finally, for the casual fan, there’s Mascot Mode, an NBA Jam-style presentation of the game. Three-pointers are greeted with a flaming “3” on screen, dunks are illustrated with lightning strikes. Maheswaran called it a celebration of the sport.

Some of Second Spectrum’s augmented experience has been used during broadcasts in the past, but typically on replays and highlights. The Bucks-Lakers game on ESPN3 will be the first time the technology will be on display in real-time for an entire broadcast and will mark the start down a path that Maheswaran believes will result in completely personalized broadcast experiences for viewers.

Maheswaran said getting to this point was “quite challenging” but said the latency for Full Court Press is indistinguishable from other streaming services. Reaching that milestone is a big one for Second Spectrum, as it opens up lots of opportunities. “Once we get here, we’ve done it,” the company’s CEO said. “From this point on, it’s only going to get better. The design is going to get better, the latency is going to get better, the number of options are going to get better.”