When vending machines attack (a university)

A university found that its internet was choking and stuttering. An investigation showed the on-campus vending machines were waging a quiet war, alternately throttling and completely shutting down the connection. The infected soda machines and 5,000 other botnet-infected IoT devices were making hundreds of DNS lookups every 15 minutes.

The story comes from Verizon’s recent Data Breach Digest, where the university is not named, obviously to protect the innocent. One security team member remarked, “This botnet spread from device to device by brute forcing default and weak passwords. Once the password was known, the malware had full control of the device and would check in with command infrastructure for updates and change the device’s password—locking us out of the 5,000 systems.”

Connected vending machines are part of the Internet of Things, which you may have heard of. This genre of internet-connected appliances and vehicles brings our lives a new kind of convenience, really great, the best convenience. If your idea of convenience is an unpredictable device that occasionally inconveniences you beyond comprehension.

According to the tech who discovered their uni’s soda machines had gone rogue, the botnet was hungry. It spent days and an untold amount of bandwidth searching for yummy “fish” dinners. They said in the report, “The name servers, responsible for Domain Name Service (DNS) lookups, were producing high-volume alerts and showed an abnormal number of sub-domains related to seafood.”

Everything from light bulbs to vending machines were hooked up to the university’s network in an effort to make management of all the connected devices efficient. “This was a mess,” the tech wrote. “Short of replacing every soda machine and lamp post, I was at a loss for how to remediate the situation.” That was when they reached out to the Verizon RISK Team, which the school had on retainer for network emergencies. In this instance, it was an “IoT outbreak.”

“Luckily for me,” the school’s tech wrote, “a less drastic option existed than replacing all the IoT devices on campus.” Whoever had infected the school’s devices had made a schoolboy error when it came to securing their malware’s own password. “Analysis of previous malware samples had shown that the control password, used to issue commands to infected systems, was also used as the newly updated device password.” They made a plan to intercept the clear text password for one of the compromised soda machines and then quickly perform a password change before the malware’s next update. This way, the university could retake control of its thousands of infected IoT devices.

“I instructed the network operations team to prepare to shut down all network access for our IoT segments once we had intercepted the malware password. Short lived as it was, the impact from severing all of our IoT devices from the internet during that brief period of time was noticeable across the campus—and we were determined never to have a repeat incident.”

It worked. One of the tech team’s developers wrote a script to log in, lock the botnet out with a new password, and clear all the devices of infection. “The whole process took a matter of minutes,” they wrote.

The university was lucky they had someone who knew what the hell was going on once they identified the problem. I wish there was a way to install this person in my home or my car. Because as our devices turn on us, we in turn have no one to turn to — and vending machines are now, officially, actually out to get us. Make no mistake, our new Skynet isn’t a cool film franchise or laughable allusion to the Terminator. It’s a place where phrases like “IoT security” cuddle up with phrases like “alternative facts” because both phrases have very distant relationships with reality.

That’s because the machines are really starting to act like they’re out to get us. Like the Volkswagen that “red screens” when you tap its in-dash screen too fast. Smart TVs that spy on users and sell their data to advertisers or get bricked with malware. Refrigerators that stop working until they’re updated. Pacemakers that rat you out to the cops. And, of course, DVRs and webcams that attack the internet and bring it to its knees.

It’s humiliating! We’re now lower on the food chain than the robots. We already have to prove to computers that we’re living breathing humans every day when we log in to anything with CAPTCHA. And by using anything with an algorithm, we’re teaching them how to outsmart us even more in the future.

The information security sector, to its credit, has been in a panic over this for quite some time. To the point that it’s becoming a tired trope for conference talks. Here in San Francisco, we just endured the RSA Conference. It’s a gathering of tens of thousands of infosec professionals from corporate and government security sectors. And it’s where you go if you want to see the same old, same old for security trends. I’ve come to think of it as a place where people simply shout to hear the echoes.

Some people there repeated what hackers have been warning everyone about for years, but no companies seem to care to listen. IoT security is really a problem.

There was a talk on solar panels being compromised and one on making IoT devices into attack vectors. Another speaker posited a vision of security updates for our new connected nightmare, while some heartened soul put forward a survival guide for the “IoThreats Era.” One talk finally named our collective demon in Protecting Consumers in the Age of IoT.

That while these people were preaching to the choir from a conference stage we found out an entire university had been taken down by connected vending machines should be a call to action. That is, if anyone was listening.