A reporter wears a “mixed reality” headset at the Raptors practice court at Scotiabank Arena in Toronto on Jan. 23, 2022, to test new augmented reality technology that puts the user in the middle of a virtual NBA basketball games. The SportsX project is a collaboration between MLSE Digital Labs (the technology and digital innovation arm of MLSE) and Amazon Web Services (AWS).MLSE/The Canadian Press
Imagine being on the court with the Toronto Raptors. Or watching an NHL game from the stands with real-time statistics literally in front of your nose.
Maple Leaf Sports and Entertainment is ahead of you, developing “mixed reality” technology to change how fans experience the action by integrating digital information with the user’s environment in real time.
MLSE Digital Labs, the sports conglomerate’s technology and digital innovation arm, and Amazon Web Services (AWS) have unveiled a new research and development program called SportsX. Its goal is to use technology – and game data provided by the NBA and NHL – to enhance the fan experience and give its teams a winning edge.
On Monday evening, media were given a hands-on look at the in-progress project at Scotiabank Arena – first in the Raptors’ old practice court attached to the arena and then in a suite as the Maple Leafs hosted the New York Islanders.
A cutting edge “mixed reality” headset helped unlock what was billed as an “immersive basketball experience” that turned the empty Raptors court into a game environment with life-size virtual players racing past or through you.
MLSE says the technology combines optical tracking body-pose data, 3D models and the Unity video game engine to deliver the experience to the headset.
In the arena suite, a table with a digital top – think of a vintage Pac-Man tabletop video game but with cutting-edge tech – showed data from the NHL game below. Moving poker chip-like icons with a photo of the player in question mirrored the action on the ice below, complete with puck.
A touch of a button and the table switches to real-time player statistics like distance skated, faceoffs won, hardest shot, total hits, maximum speed and total passes. Another touch of the button and the digital display offers a player’s heat map showing where they have been on the ice.
Putting on another augmented reality headset, attached to a hand-held app, and you see headshots of the players on the ice in your display as you watch the live action. Using a virtual pointer, you can choose a player and see his real-time stats come up in front of you as the game action unfolds.
The NHL provides the data, having microchipped the puck and player jerseys for NHL Edge, the league’s puck and player tracking technology system that has been deployed in all 32 NHL arenas. Some of that data is currently being used by the league’s broadcast partners, from skating speed to shot velocity.
But MLSE wants to bring it to your phone and its app sooner than later. And for Christian Magsisi, MLSE’s vice-president of venue and digital technology, it’s a matter of when not if such advances make it to your home.
Unlike the NHL data-tracking, the Raptors immersive experience uses cameras mounted in the arena to track movement. That data, which is being gathered at every NBA arena, is then processed and streamed to the headset.
“There’s a basketball game being played exactly where you’re standing right now. But not physically, virtually,” explained Magsisi.
Put on the headset and the virtual players emerge – playing the game full-tilt, running, passing and shooting around you or through you.
“We can quite literally recreate the game that happened last night or historically,” said Magsisi. “So imagine thinking about maybe a game you attended or a game that has some sort of significant relevance to you and being able to put on virtual reality goggles and being able to watch that courtside. Or even better than that actually, walk onto the court.”
The optical tracking data uses computer vision to capture human movement and identify the position of each joint and limb. It captures the data at some 60 frames per second or more with each frame containing the position of every player on the court including every joint/limb.
MLSE showcased it on a full-sized basketball court but says it can be used anywhere. The user can walk around the environment, look around and see things from the perspective of fans courtside, coaches on the bench or the players themselves on the floor.
At this stage, the players are generic – think realistic moving mannequins – but will become more lifelike facsimiles in the future once the red tape – re: player images and other issues – are resolved.
Game data is relayed to an AWS “outpost” at Scotiabank Arena, reducing delays, and then analyzed and distributed to the necessary applications.
The challenge now is to figure out how to best use it, with MLSE well on its way.
“The fact that the leagues now made this data available, I think that’s a huge win,” said Humza Teherany, MLSE’s chief technology and digital officer.
He expects other leagues to follow suit.
While the technology is eye-popping, it is still under development. Both MLSE projects started in July.
“We are testing technology that isn’t quite yet ready for production. But we want to show the fans now. We want fans to actually see it,” said Magsisi.
“We are working with some of the biggest tech companies in the world to advance this technology,” he added.
The headsets used by SportX aren’t cheap. But the tech is advancing and the hope is, in the near future, that they reduce in size and price.
How teams will mine such technology for their own uses has also yet to be fully determined. But Magsisi is confident they will find a use for it.
“If we can recreate a game, who knows what ideas our coaching staff, our players, our scouts, would come up with?” he said
But the ability to monitor movement on the NBA court could help in injury assessment and avoidance, while on-ice data can better help teams determine talent using souped-up analytics.