Microsoft powers a DJ’s live show with a Surface Book and Kinects

When it comes to live shows, the visuals are key to making a lasting impression. I’d even argue that what you see is just as important as the quality of the music. It is a performance, after all. Touring musicians employ all kinds of A/V gear in an attempt to offer a unique experience for concertgoers. For years, some acts have turned to Microsoft’s Kinect camera to capture movement live, translating that to graphics on a video display, among other things. To make the camera-based setup more portable, Microsoft teamed up with DJ Alison Wonderland to create a simplified Kinect-driven system that runs primarily on a Surface Book.

“Some of the effects are enhanced when she’s moving around, so she can make some of the effects bigger or more dramatic depending on how she’s moving in front of the Kinects.” Sorokas said. “The Kinect is also using sound, and some of the effects are also responding to the sound in the music she’s creating.” In fact, one Kinect is tasked with capturing Wonderland’s body movements while the other is focused on her hands, allowing her to manipulate the visuals, as Sorokas explained.

Her VJ also has a simple interface on the Surface Book that allows for quick adjustments and selections on the device’s touch-enabled display. In fact, SuperUber’s founder and CEO Russ Rive explained to me that all the computing power driving the visuals is coming from the Surface Book.

“We were surprised in the development process. We assumed we would use the Surface Book as, like, a MIDI controller interface,” Rive said. “Once we had it in there … it’s running the entire thing. It’s doing all the visual processing.” Sure, the project is using the top-of-the-line Surface Book configuration, but it’s impressive that the device is powerful enough to drive both the controller interface and the Resolume software in real time without any noticeable lag.

There are other computers at work here — for example, two Windows 10 Brix PCs connected to the Kinect cameras to analyze the gathered data and create grayscale-depth videos complete with skeleton coordinates. However, in terms of transforming what’s being collected onstage from the Kinects into visual elements, that’s all being done on the Surface Book. Rive explained that if just one Kinect camera was being used here, that pair of Brix PCs wouldn’t be necessary. Because you can only connect a single Kinect to a computer, the system demands them. Rive said those two machines are mostly for fine adjustments and calibration purposes for the Kinect cameras.

Rive also noted how important it was for the interface of the custom-built controller app to be as simple as possible on the Surface Book display. The environment of a live show can be chaotic and stressful, so Wonderland’s VJ needs to be able to make selections and adjustments without having to hunt around for what he’s after.

“Imagine you’re in the middle of a totally chaotic mosh pit, but there’s no panic when you look at the display,” he said. “You can’t press anything that’s wrong. Everything is bright and super simple to use.”

The app, which was written in C++ using openFrameworks, pulls in video from the Kinects that can be used in 10 different scenes. That collection of options is on the left side of the UI, with effects sliders down the middle and Kinect camera previews on the right. It looks simple, but everything the VJ might need at a moment’s notice is a tap away. Of course, the whole thing can be piped into more-robust Resolume software for more-detailed tweaks and layering. The fact that it works with a professional-grade app that the crew was already using meant there wasn’t too much of a learning curve either.

Wonderland admitted that she wasn’t familiar with Kinect before this project, but she explained how the setup expands the possibilities for her live show. “You could always film me dancing, but now you can film me dancing on fire,” she said. “Which is pretty damn cool.”

As mentioned earlier, this isn’t the first time Kinect has been used onstage during a live show. Nine Inch Nails put the cameras to work during its 2013 tour to track Trent Reznor’s movements and translate them into artsy visuals at different points in the show, for example. Microsoft’s Music x Technology initiative is no stranger to using the cameras either, employing them in projects with the likes of Washed Out and Neon Indian. However, the toolkit for Wonderland is more of a complete package. It’s something that could potentially be used by other artists in the future who need the visual element of their show to be more portable and fit a range of venues.

“It’s very plug-and-play — it’s not very difficult to set up at all,” Rive said. “Someone could easily set this up with one Kinect, plug it in and start playing around with it. We designed this for two Kinect cameras, but the backend has already been designed to accommodate as many as you want to use. It’s just a matter of adjusting the interface.”

He went on to give the example of a ring of a dozen or so Kinect cameras that could be used to create Matrix-like effects or something similar to a 3D scan. “There’s some really cool stuff that’s going to be done once we really push it,” Rive continued.

Wonderland echoed that sentiment. “Even at rehearsal today we were coming up with new ideas,” she explained. “I can’t wait to get out on tour and keep trying new things with the new music. There’s loads of stuff we could do. This is just the beginning.”

While Rive and the rest of the team are working on making those ideas a reality, you won’t have to wait long to see the collaboration put through its paces. Well, that’s if you’re planning to take a trip to Chicago. During Wonderland’s set at Lollapalooza this weekend, the Kinect-and-Surface-Book-powered system will be on display. For the rest of us, there’s a chance to catch the tech during her North American tour, which runs through October.

[Image credits: Jeremie Brillant and Microsoft]