The Untold Story Of Microsoft’s Surface Hub
Jeff Han’s fingers are dancing across an expansive, wall-mounted touch screen. Planet Earth spins in front of him in computer-generated form; he grabs it with both hands and starts to zoom in. He keeps going–Western Hemisphere, North America, United States, Pacific Northwest–until we’re finally staring at a prosaic industrial area alongside a highway.
“Here’s our building, right here,” he declares.
The conference room where this demo is going on is named after Bill Gates, so it shouldn’t come as a complete shock that we’re on Microsoft premises. But this isn’t corporate headquarters in Redmond, Washington. Han and I are 200 miles away, across the Oregon border in the Portland suburb of Wilsonville, inside a 4-acre structure flanked by a manufacturer of industrial shredders and a storage facility for boats and RVs.
Microsoft hasn’t played up the fact that it has a major operation in Wilsonville. Actually, it’s been downright stealthy about it. (The roadside signage directing visitors to the main entrance doesn’t even mention a company name.) But since March 2014, the building is where the company has been engineering the device Han has been showing me, the Surface Hub.
The company unveiled the device during its Windows 10 launch extravaganza in Redmond last January. It showed how the 84-inch all-in-one computer can be used for tasks such as brainstorming, videoconferencing, and PowerPoint presentations, all conducted via touch and pen input rather than mouse and keyboard.
Microsoft held back one of the most intriguing facts about this Windows 10 machine until now: It not only designed the Surface Hub but is about to begin manufacturing the thing itself, in 70,000 square feet of factory space in the Wilsonville building, steps away from where the hardware was engineered. “We don’t actually manufacture the LCD panel,” Han explains. “But that’s pretty much the only thing we don’t make here.” Two models, a $19,999 unit with an 84-inch display and a $6999 version with a 55-inch display, will roll off the assembly line.
Microsoft has done such a crafty job of keeping its Wilsonville plans on the QT that a September 2014 Oregonian article on the facility simply assumed that it was all but a given that the company would go offshore when it came time to build anything. “[M]ass production would almost certainly take place in a country with lower labor costs,” the story helpfully explained. “It’s that way with nearly all high-volume electronics manufacturing.”
For the 39-year-old Han, the impending release of the Surface Hub is the fullest expression yet of a mission he’s been pursuing for more than a decade. The world first took note of it in 2006, when he was a computer-science researcher giving a TED talk in Monterey, California, about an intuitive new computing interface called multi-touch. He was an exuberant, charismatic presenter–and still is, as I learned when he showed off the Surface Hub to me in Wilsonville.
In the pre-iPhone era of 2006, what he showed was mind-bending; the video version of his presentation became one of the first TED talks to go viral, and Han’s sheer enthusiasm made multi-touch feel like it might indeed be the next big thing. He then parlayed his fame into a startup, Perceptive Pixel (PPI), which sold pricey screens to everyone from the Department of Defense to Disney. Microsoft acquired the company in 2012.
Today, Han is general manager of Surface Hub. Without his research, vision, and persistence, the device might not exist. And yet at the same time, it’s also a profoundly Microsoftian product. PPI’s products were displays you plugged into a PC; the Surface Hub is a computer, running big-screen editions of Windows 10 and Office. It doubles as a videoconferencing system equipped with two high-resolution cameras and Microsoft’s Skype for Business service. There are even signs of Xbox influence, such as the noise-canceling four-element microphone, which originated as part of the game console’s Kinect sensor.
Unlike the other hardware device Microsoft unveiled at the January Windows 10 event–the mixed-reality HoloLens headset–the Surface Hub doesn’t have a whiff of science fiction about it. It’s about polish and technical sophistication more than raw futurism, and in a way, it feels like Microsoft tackling a bit of unfinished business.
“I’ve been working on product development around productivity forever,” says Mike Angiulo, who, as Microsoft’s corporate VP for hardware, is Han’s boss. (“Forever,” in this case, means since 1993.) “If I think about what’s changed since I’ve been there in terms of individuals doing work, your ability to create and communicate has gone up exponentially, with PCs and mobility and phones. But the conference rooms are exactly the same as they were when I started. They’re like time capsules. There’s a projector, there’s a whiteboard, there’s a conference phone.”
With the Surface Hub’s collaborative tools, Microsoft is making a hyper-ambitious attempt to yank those time capsules into the present day. Which is not anything like a guarantee that it will be successful. In 2002, for instance, the company launched Tablet PCs and said it expected them to displace conventional laptops within a half decade; they didn’t. And its first foray into multi-touch computing–2007’s original Surface, which built a computer into a table for use in retail environments–didn’t go anywhere.
Still, no matter how the market responds to the Surface Hub, the fact that Microsoft is behind it raises the stakes far beyond anything Han has attempted in the past. “This isn’t a trivial little experiment,” he says. “We’re committed.” He pauses. “We don’t do things small at Microsoft. We do things big.”
From TED to Bill, Steve, and Satya
For as long as he can remember, Han, a research scientist working out of New York University’s Courant Institute, has been fascinated by technology. He even doodles in right angles, rectangles, and squares–hieroglyphs that look almost like circuitry, a schematic of his unconscious. The son of middle-class Korean immigrants who emigrated to America in the 1970s to take over a Jewish deli in Queens, Han began taking apart the family TV, VCR, “anything that was blinking,” at the age of 5 (he still has a nasty scar courtesy of a hot soldering iron his little sister knocked onto his foot).
–From “Can’t Touch This,” a profile of Jeff Han by Adam Penenberg in the February 2007 issue of Fast Company
Han first noodled around with multi-touch interfaces–a concept dating back to the 1970s–in 2002, as a researcher at New York University. Then he put the project aside in favor of other experiments. (Though his name is practically synonymous with multi-touch, he’s also explored everything from autonomous navigation to eye tracking to motion capture.)
In 2005, he turned his attention back to multi-touch, and stuck with it. By February 2006, he was ready to show his work in progress at the TED conference, using a 36-inch screen set up like a drafting table, with cameras that tracked his finger movements. Viewed today, his demo remains engaging. But for a 2015 viewer, the most striking thing about it is the sound of attendees gasping and applauding at now-familiar gestures such as Han pinching a photo to resize it.
Han began his presentation by explaining that the interface he was about to show was ready to come out of the lab. He wasn’t exaggerating. Perceptive Pixel, he told me, “was founded shortly after the response I got from TED.” The startup’s goal was to turn the technology into useful commercial products.
However, the next meaningful moment in the popularization of multi-touch happened on January 9, 2007, when Steve Jobs announced the first iPhone at Macworld Expo in San Francisco. “We have invented a new technology called multi-touch,” he boasted, in one of the more reality-distorting things ever said during an Apple keynote. Han, by contrast, had been careful to share credit in his TED talk: “I’m not the only one doing it, there are a lot of people doing it.”
Among the other organizations that had been working on multi-touch was Microsoft. Four months after the iPhone reveal, it announced Surface, a Windows Vista computer built into a table with a 30-inch screen. The interface bore decided similarities to Han’s efforts. But instead of touting the table as a tool for workplace productivity, Microsoft said it was going to roll it out in public places such as hotels, casinos, and phone stores.
A press release quoted Microsoft CEO Steve Ballmer: “We see this as a multibillion dollar category, and we envision a time when surface computing technologies will be pervasive, from tabletops and counters to the hallway mirror. Surface is the first step in realizing that vision.” But the billions never rolled in. Microsoft gave the table a vote of no confidence in 2012 by stripping it of its name, redubbing it PixelSense and turning the Surface moniker over to a line of Windows 8 tablets. Samsung continues to sell a PixelSense-based table computer, but if you’ve never seen one in the wild, you’ve got plenty of company.
While the Surface table was failing to catch on, the iPhone was putting multi-touch into the hands of people around the world, transforming the phone business forever. Today, Han says that neither product led him to question whether he was on the right track. “I wanted to focus on upright productivity,” he explains. “Ever since TED, that’s what it was about for me.”
Perceptive Pixel’s earliest screens found an unexpected killer app during the 2008 presidential campaign. News broadcasters–most famously John King of CNN–used them to swoop around maps and tally up votes via software handcrafted for the purpose by PPI.
The company sold these screens for six figures, but better still, they served as potent free advertising. “John King knows every single one of those counties,” he says. “He’s not reading off the screen. He’s using this as a storytelling tool. At that time, whenever a customer saw that, they were like, ‘This is exactly what I want to do.’ This is what a teacher does. This is what a commander does in a mission briefing. This is what a surgeon does before they actually go into surgery.”
Han’s big-ticket high-end screens attracted attention, won awards, and generated revenue. But on some level, he remained unsatisfied. “It was good to put food on the table, but it was not going to be a huge, huge business,” he says. “We got a lot of criticism as a startup: ‘Great stuff, but when are you going to make it cheaper for the masses?’ That’s exactly why we started talking with Microsoft.”
Those talks included both Steve Ballmer and Microsoft’s cofounder and then-chairman, Bill Gates. “We started talking with them and realized they actually have the same vision,” Han says. “When I met Bill Gates, he literally said to me, ‘I want to see these things rain from the skies.’”
As for Ballmer, “he put one in his office right away. He was the first guy in the company to do it. He had a giant corner office. He took down the only whiteboard he had and put up the PPI, and said, ‘This is how we’re doing things.’”
When Perceptive Pixel became part of Microsoft in July 2012, it joined a company in more or less continuous flux. Ballmer was energetically trying to reposition the world’s most famous software developer by delving into the hardware business. A few weeks before announcing the PPI deal, the company had unveiled its Surface tablets, the first PCs ever to be sold under the Microsoft name.
At first, PPI was part of Microsoft’s Office division, placing it as close as possible to the nerve center of the company’s productivity efforts. A year later, Ballmer instigated a sweeping reorganization that clustered Han’s team, more conventionally, alongside other hardware products such as Xbox and Surface tablets (and, eventually, the former Nokia phone business, which Microsoft acquired in 2014).
Only six weeks after initiating the reorg, Ballmer announced he would step down as CEO once his successor had been found. The search dragged into February 2014, when longtime Microsoft executive Satya Nadella got the gig.
Nadella moved swiftly to rejigger how the company articulated its take on its own future, dropping Ballmer’s “devices and services” mantra in favor of emphasizing mobility, the cloud, and productivity. Despite being the least portable hardware Microsoft has ever offered–the 84-inch model weighs 280 pounds–the Surface Hub project’s focus on Net-connected collaboration had a place in the new CEO’s vision, especially as he clarified that mobility was about more than phones and tablets. As he began putting it: “We want to build experiences that are about the mobility of the experience–not the mobility of the device.”
Today, Han says, “the management support for this is even bigger than it was ever before. That’s not a ding on Ballmer. Every time you have a regime change, you wonder. But this perfectly fits into Satya’s vision of cloud-connected mobile workflows.”
The personal touches that contributed to making Han’s 2006 TED demo so engaging have helped him find his way inside the Microsoft labyrinth. “Jeff’s a very talented, energetic guy,” says hardware VP Angiulo. “And one of the few people who have made the transition from being a founder–where he was running the Jeff Show–to now being an important part of a cross-company initiative. Skype is the one writing the Skype client. OneNote is writing the whiteboard. Windows has an entire team of people creating team Windows now, that’s designed for sharing.”
Han, Angiulo says, “shows up with intelligence—deep intelligence, he really knows what he’s talking about. And the passion for the product, and the vision. He gets people to want to work with him.”
Microsoft won’t disclose how many staffers it has working on the Surface Hub, but does allow that the number is in the hundreds. They include Perceptive Pixel crew members such as David Slobodin, who supervises hardware, and Microsoft veterans such as Hayete Gallot, who heads up business strategy and marketing. Stevie Bathiche, director of research for Microsoft’s Applied Sciences Group, is also involved; even Alex Kipman, best known for spearheading the Kinect and HoloLens, is contributing to the Windows 10 side of things.
State-of-the-Art Screen
Not everyone is sold on Han’s idea. Ben Shneiderman, a computer science professor at the University of Maryland and a founding director of the Human-Computer Interaction Lab, calls Han a “great showman” who has “opened the door to exciting possibilities.” But he doesn’t think Han’s technology would be suitable for a large-scale consumer product, nor as useful as a mouse on a large display. If you are standing in front of the screen, Shneiderman wonders, how would people behind you be able to see what you’re doing?
–From Fast Company‘s 2007 profile of Jeff Han
The device that Han, former Perceptive Pixel staffers, and new Microsoft colleagues ended up designing isn’t exactly in a category of one. InFocus’s Mondopad and Clary Icon’s OneScreen, for instance, combine large touch screens, videoconferencing, and Windows into conference-room collaboration tools similar, in principle, to the Surface Hub.
“I know it sounds arrogant, but we don’t have competitors,” says Han when I ask about such machines. “Sure, we’ve seen large touch screens before, but they’re in a different league, to be honest.”
Arrogant, maybe, but also a defensible stance. There’s a lot about the Surface Hub that’s new, starting with its touch-screen technology. Like an iPad or Surface tablet, it uses an optically bonded capacitive touch screen that gives it the same sort of buttery responsiveness and ink-on-paper clarity that smaller-screen devices achieve. The screen’s refresh rate of 120Hz–twice that of smaller devices–adds to the general smoothness of the visual experience.
The Surface Hub’s capacitive screen also works with a pressure-sensitive pen for notetaking, sketching, and other purposes. Though it’s not the only big screen to offer a pen, it uses Perceptive Pixel technology, which permits a single capacitive touch-sensor system to deal with both fingers and pens. Han says that PPI was one of only two companies to have worked this out; in May, Microsoft announced that it had acquired the other one, Israel-based N-Trig, whose technology is used in Surface tablets.
So much about the Surface Hub experience is familiar that it’s tempting to think of it as an overgrown Surface tablet–a perception that the “Surface” in its name doesn’t do anything to discourage. But the ways in which it differs from its dinkier counterparts are at least as important as the similarities. For one thing, it’s designed to be used by multiple people at once–which, especially with the sprawling 84-inch model, makes sense.
“Multi-touch was about breaking the tyranny of the one, to me, the single cursor. But to bring multi-touch to its fullest potential, we’re going multi-user,” Han says. “A lot of plumbing needs to change in order to really accommodate what it means for multiple people to be working together. And there are some hard problems in there.” For instance, the Surface Hub must be able to to register not up to 10 fingers at a time, but 20 or more–and to figure out if two people are working in different apps at the same time.
The Surface Hub’s approach to videoconferencing is also new. Instead of plunking a single camera above the screen like a tablet would do, it’s got two side-mounted 1080p ones which are angled to capture the entire room and are “located at eye level, not God level,” Han says. The device’s special version of Skype for Business intelligently cuts between the two cameras to provide what Microsoft hopes is an optimal view of what’s going on at any given moment.
Speaking of software, it isn’t a coincidence that the Surface Hub will reach its first paying customers a few weeks after Windows 10 is available in final form. The device and its operating system were built with each other in mind, and though the hardware can run the same Windows apps that will work on a PC, tablet, or phone, it adjusts the interface to make sense on a huge display. The Start menu, for example, sits in the middle of the screen, rather than popping up in the lower left-hand corner, and some controls get duplicated alongside the left- and right-hand edges.
The Mondopad and OneScreen, by contrast, both ship with Windows 7–an operating system that, though still beloved by risk-averse large enterprises, was never designed with gigantic screens or serious touch input in mind.
So much of the Surface Hub’s hardware and software is designed around the fact that’s meant to be used by multiple people in a collaborative environment that Han bristles at the notion that it’s merely a really nice PC, or even a PC at all. “People have tried PCs in conference rooms before,” he says. “They suck.”
The Surface Hub aims to eliminate typical conference-room PC hassles. Using Microsoft Exchange calendaring features, it automatically credentials meeting attendees so they don’t have to log in to get access to their files and the network. A motion sensor allows it to sit unobtrusively until someone approaches, whereupon the screen lights up. Remove the pen from its holder, and the device automatically goes into whiteboard mode. And when your meeting ends, it wipes away all evidence of the activities you conducted, so it’s ready for the next user.
Microsoft has also put a lot of thought into how the device works with the PCs, tablets, and phones that people bring into meetings. Using the Wi-Fi Alliance’s Miracast wireless technology, attendees can fling their gadgets’ displays onto the big screen. Unlike garden-variety Miracast, the Surface Hub’s version is bidirectional: If you use its pen to scribble notes on a PowerPoint slide, your annotations travel back to the version on a Windows 10 laptop.
The Surface Hub will do a lot out of the box, and will run standard Windows 10 apps. But it’s pretty clear that it will become much more useful if third-party developers start to build software with it in mind. And some of that software will need to cater to specific industries.
“We digitally model everything, but our interface is a mouse and keyboard, and that’s not necessarily the most intuitive way,” says John Cerone, director of digital design and construction at New York-based SHoP Architects, whose projects include Uber’s new headquarters. SHoP has used Perceptive Pixel displays and has just gotten in a preproduction Surface Hub. With the right applications, it sees the potential for Microsoft’s new device to be transformative. “Anytime we have the opportunity to touch or draw or pull or push a 3-D surface, we want to do that,” Cerone says.
Assembly Required
Jeff Han is helping me buckle my boots. They happen to be part of the head-to-toe clean-room “bunny” suit I’m donning so we can visit the factory floor where Microsoft assembles the Surface Hub’s touch screen. Properly suited up, we must pass through a tiny room where jets of air blast away any remaining detritus from the outside world.
Cleanliness counts in manufacturing of all sorts, but the size of the Surface Hub’s display ups the ante. “It’s exponentially harder the bigger it goes,” Han explains to me. “It’s like defects in semiconductors. It’s literally by surface area. If I have one flaw–one little speck of dust or one little bubble–I throw away the entire thing.”
Almost everything that’s tricky about making a Surface Hub involves the process of fusing touch sensors and glass into enormous touch screens. Though the capacitive technology is similar to that used in smaller devices, the enormity of the 55-inch and 84-inch displays necessitated a new approach to manufacturing, and reduced the appeal of outsourcing the job to a third party on another continent.
“We looked at the economics of East Asia and electronics manufacturing,” says Angiulo. “When you go through the math, it doesn’t pencil out. It favors things that are small and easy to ship, where the development processes and tools are a commodity. The machines that it takes to do that lamination? Those only exist in Wilsonville. There’s one set of them, and we designed them.”
Surface Hubs are anything but small and easy to ship, a point that is obvious in the first place, but even more so once you’ve seen them being built. The sensors–the layers that detect finger presses and pen input–are stored as enormous Saran Wrap-like rolls. They get adhered in two parts to the glass that protects the LCD. The process of sandwiching the enormous layers together involves massive robotic arms capable of hoisting screens though the air, conveyors of various designs, ovenlike chambers, and other custom machinery, all of which must do its job gingerly to avoid damaging the component it’s creating.
“You get the sense of scale of these things?” Han asks me as we peer at an 84-inch piece of glass having its touch sensor applied. “I look at this almost as a patient on a table.” At various other times on the tour, he compares the touch screens under assembly to automobiles, flypaper, pizza, and buttered pieces of toast. Like a proud father, he beams continuously, pausing to fawn over items as mundane as the packing materials that will protect Surface Hubs when they’re shipped out of the factory.
Observing Surface Hubs being put together is an unexpectedly multi-sensory experience. It’s not just the thrum of the robotic equipment. Even before you spot the bank of gleaming metal vessels full of bonding materials, you notice the air is pungent with an epoxy-esque odor. In another area, signs warn you not to look directly into the ultraviolet light that cures the screens until the sticky adhesive is no longer sticky.
Exotic though this world seems, it’s in the same building as the cubicles and conference rooms where the Surface Hub’s hardware team toils. (Software development happens in Redmond, in closer proximity to the folks responsible for Windows and other Microsoft software.) Perceptive Pixel has had a presence in the Portland area since 2009, drawing on the region’s rich supply of display engineering talent, a legacy dating back to the founding of lab equipment maker Tektronix in the 1940s.
“I don’t have to send my folks over to China, so they’re happier,” Han says. “It’s faster. There’s no language, time, or culture barrier to deal with. To have my engineers go down the hallway to talk to the guys in the manufacturing line and tune the recipe? That’s just incredible.”
What Next?
At the time I visited Microsoft’s Wilsonville factory, it was still building Surface Hubs in small quantities and gearing up for the mass-production effort necessary to get units to customers by September. When I ask Han what the future holds for the product line, once it becomes a shipping product, he refers obliquely to next-generation models the company is currently working on, but provides no details.
In Microsoft’s San Francisco office, his boss, Mike Angiulo, is more explicit about where the line could go over time. He gestures at the 84-inch screen, and then at a wall that’s mostly whiteboard (with a sign tacked up reminding people to erase it when they’re done). “By the time that’s mundane, this entire wall is going to be a screen,” he says. “And everything you touch on the whiteboard is going to be responsive and digitally synced to the phone. We have a vision for group productivity that extends beyond what you can do with this kind of a screen today.”
“I didn’t actually just announce those products,” he adds, just in case it wasn’t abundantly clear. “But you can imagine that if bigger is better, the future is about having all the spaces you work with when you’re around other people to be useful to you.”
Will Microsoft keep plugging away at making that dream into reality, especially if the Surface Hub isn’t an immediate breakout hit? “If anything, they’ve already exhibited a fair bit of patience,” says Forrester analyst J.P. Gownder, referencing the 2012 Perceptive Pixel acquisition and effort the company has put into making Windows 10 work well on a large display. “This is a good core-competency kind of place for them. I suspect this is going to be for the long haul, especially as they get customers.”
Much more patience will be required. The Wilsonville quarters is festooned with Surface Hubs in use–even hung in a break area near a pool table. But when I stroll around with Han and notice that the engineering team still writes on whiteboards and pins things up on bulletin boards, the day when such old-school tools are as archaic as a typewriter does not feel imminent. Even though someone has playfully scrawled, “This is obsolete technology” on one of the whiteboards.
In Adam Penenberg’s 2007 Fast Company profile, Han spoke of wanting to flit from technological challenge to technological challenge, like an entrepreneurial honeybee. Now he seems game to continue on with this quest. “This is one of the final frontiers of computing,” he tells me. “We’ve got the personal thing nailed pretty well, you know? The next undiscovered country is how we do things with multiple users, together.” Just as when he gave his TED talk in 2006, his enthusiasm for his work is powerful, infectious stuff. And with Microsoft’s help, we’re about to learn how much further he can take it.
[Photo & Video: courtesy of Microsoft]
(21)
