At about ten past seven on the morning of Friday, October 21, 2016, the East Coast of the United States woke up, reached for its phone, and discovered that the internet had been unplugged. Not their internet. The internet. Twitter would not load. Netflix would not load. Reddit, Spotify, PayPal, GitHub, Airbnb, the New York Times, the Wall Street Journal: all unreachable, all at once, on a Friday morning, which is statistically the worst possible time to ask several million people to be patient.
Here is the first strange thing, and it is the key to the whole day. Nothing was wrong with Twitter. Netflix's servers were fine. The New York Times could have printed money that morning if anyone had been able to read it. Every one of those companies had healthy machines sitting in healthy data centers, humming along, waiting for visitors who could no longer find the address. The buildings were standing. Somebody had burned the phone book.
How you lose an internet that is still there
The Domain Name System, DNS, is the internet's phone book, and it does exactly one job: it translates names people can remember into numbers machines can use. You type a name. Your computer asks a resolver where that name lives. The resolver asks around, gets back a number, and your computer goes there. The whole exchange takes milliseconds and happens so reliably that almost nobody on Earth thinks about it, in the same way almost nobody thinks about the floor.
Dyn was one of the companies that answered those questions for a large share of the biggest names on the internet. When you asked where Twitter lived, there was a decent chance Dyn's infrastructure was somewhere in the chain of the answer. This was a sensible arrangement right up until it was the only arrangement, which for many of Dyn's customers it was. They had one phone book. It was an excellent phone book. It was also, on that Friday, the only copy in the building.
A distributed denial of service attack on a DNS provider does not break the websites. It breaks the directions to the websites. Starting at about 7:10 AM Eastern, Dyn's managed DNS infrastructure began drowning in junk traffic, a flood of bogus requests so heavy that legitimate questions could not get through. Ask where Netflix lives, and your question joined a riot. The answer, if it came at all, came too late. To the person holding the phone, this is indistinguishable from Netflix being dead. Netflix was not dead. Netflix was unreachable, which from the outside of the system is the same funeral with a different cause of death.
Dyn's engineers mitigated the first wave and restored service by about 9:20 AM Eastern. The East Coast exhaled, made coffee, and started composing tweets about the outage, which is the traditional way an outage ends. Then, just before noon, the second wave arrived, more global this time, not content with the East Coast. Dyn fought it for just over an hour and restored service again at about 1:00 PM. A third wave came later in the day, and this one Dyn absorbed without customer impact, because by then the company had spent the whole day learning exactly how it was being hit. The scoreboard, if you kept one, read: attackers three swings, defenders two bloody recoveries and one clean block. The internet survived the day. But the interesting question was never how Dyn fought. It was what, exactly, had been doing the hitting.
The army was appliances
The traffic was coming, in significant part, from a botnet built on malware called Mirai, a conclusion Dyn reached with analysis help from Flashpoint and Akamai. Mirai did not infect computers in the way people pictured in 2016, with a user clicking the wrong attachment. Mirai infected things: security cameras, DVRs, home routers, baby monitors. The internet of things, conscripted.
And here is the detail that should be printed on a plaque somewhere. Mirai did not break into these devices. There was no exotic vulnerability, no zero day, no cinematic hacking montage. Mirai scanned the internet for devices with telnet open, and when it found one, it simply tried to log in, working through a built-in list of 62 factory default username and password combinations. Admin and admin. Root and the password printed in the manual of a camera made by a company that assumed, reasonably enough by the standards of the time, that nobody would ever point the thing at the open internet and leave the keys in it.
Sixty-two tries. That was the armory. If one of the 62 pairs worked, and on hundreds of thousands of devices one of them worked, the device quietly joined the botnet and began scanning for the next device, because Mirai was a worm as well as a weapon: every new recruit became a recruiter. Owners noticed nothing. A camera with Mirai on it still recorded the driveway. A DVR still recorded the game. The devices were not broken. They were moonlighting.
The numbers from that week deserve a careful sentence, because the first numbers were wrong in an interesting way. Dyn initially described tens of millions of discrete IP addresses associated with the attack. Five days later, in a statement on October 26, Dyn's Scott Hilton walked that back with unusual honesty: the attack's retry storm, the thunder of legitimate resolvers asking again and again because the first answer never came, had made the crowd look far bigger than it was. Dyn's revised estimate was up to about 100,000 malicious endpoints. One hundred thousand cameras and recorders. Estimates of the peak flood ran as high as about 1.2 terabits per second in contemporary reporting, a figure Dyn pointedly declined to confirm, and the honest way to say it is that the attack was enormous and its exact height was never certified. It did not need a certified height. It needed a Friday.
Pause on the composition of that army for a moment, because it is the funniest and bleakest part of the story. The most powerful attack fleet assembled on the internet up to that point was not a supercomputer. It was the world's neglected electronics drawer, weaponized. Devices their owners had never logged into, could not have logged into, and in many cases did not know possessed a login at all. A baby monitor does not look like critical infrastructure. A hundred thousand baby monitors, pointed at the same door at the same second, are a siege engine.
The feud at the bottom of it
Every army has a general, and the natural assumption in October 2016 was that this one belonged to a nation state, or at least to somebody with a manifesto. The truth took about fourteen months to unseal, and it was better than any manifesto.
In December 2017, the US Department of Justice unsealed guilty pleas from three young men: Paras Jha, 21, of New Jersey, a computer science student at Rutgers; Josiah White, 20, of Pennsylvania; and Dalton Norman, 21, of Louisiana. Jha and White were co-founders of a company called Protraf Solutions, which sold DDoS mitigation, protection against exactly the kind of attack Mirai performed, a business model with the structural elegance of selling umbrellas in a town where you also control the weather.
They had built Mirai for the most consequential arena in modern geopolitics: Minecraft server hosting. The market for hosting the block game was fiercely competitive, and a DDoS attack was the standard instrument of competition, used to knock rival servers offline so their players would wander over to yours. Mirai was a competitive advantage in an argument about a video game. It then escaped the argument. In September 2016, Mirai was used in a record attack on the security journalist Brian Krebs, measured at about 620 gigabits per second, and another enormous one on the French host OVH, whose target turned out to be a Minecraft server. On September 30, 2016, the source code was published on a hacker forum under the name Anna-senpai, which meant that from that day forward, anyone could build their own camera army. Three weeks later, Dyn got hit by Mirai-based botnets at historic scale.
Precision matters here, because it makes the story stranger, not weaker. The three creators were charged with creating and unleashing Mirai. They were not charged with personally directing the Dyn attack, and by the time Dyn went down, their creation belonged to anyone who could download it. They had not fired the shot that burned the phone book. They had invented the printing press for matches, published the blueprints, and left the warehouse door open.
The ending defied every convention of the genre. The three pleaded guilty and cooperated, extensively, in the word the prosecutors kept using. The FBI, facing three defendants who understood the botnet economy better than almost anyone in law enforcement, did the pragmatic thing and put them to work. In September 2018, each was sentenced to five years of probation, 2,500 hours of community service defined to include continued work with the FBI on cybercrime cases, and a shared $127,000 in restitution, alongside forfeited cryptocurrency. The young men who built the weapon that knocked over the internet's phone book were sentenced, in effect, to help guard the library. Prosecutors called their assistance exceptional and asked for the reduction themselves. It may be the only case in the canon where the punishment for breaking the internet was homework.
The lesson: count your phone books
Strip the comedy off and the Dyn outage is the cleanest lesson in concentrated dependency ever taught at scale.
First, the dependency nobody lists. Ask a company in 2016 to name its critical infrastructure and it would list its servers, its cloud regions, its payment processor. Almost none of them would have listed DNS, because DNS was not a thing they ran; it was a thing that was simply there, like gravity. Dyn's customers had outsourced the phone book so completely that many had a single provider and no plan for the day the provider could not answer. Their resilience was capped, to the decimal, by their most concentrated dependency, and the dependency was invisible precisely because it had never failed before. A backup that has never been tested is a rumor. A second DNS provider you have never switched to is a hope wearing a contract.
Second, defaults are decisions. Every camera in that botnet was a small, shipped, unmade decision: a manufacturer chose a password, a retailer sold the box, an owner plugged it in, and nobody in that chain owned the job of changing 62 well-known keys. Mirai's entire recruiting department was a list shorter than a grocery run. The fixes the industry eventually adopted, unique passwords per device, forced changes on first setup, laws in some places requiring exactly that, are all admissions of the same point: the default is the product, because the default is what ships to the world and stays there.
Third, and this is the one our own studio writes on the wall: your system is not the machines you control. Dyn's customers learned that their uptime was a property of a network they were renting pieces of, including the pieces they forgot they rented. We run a small production factory on the same principle, stated from the other end. Anything the work depends on gets a name, a second path, and a receipt proving the second path works, because the failure never announces which layer it plans to arrive through. On October 21, 2016, it arrived through the layer that answers the question "where is it," and it turned out that was the only layer there was.
There were no villains with a superweapon that day, and that is precisely why the day matters. There was a password list of 62 entries, a feud over a block game, a hundred thousand appliances nobody had ever introduced themselves to, and the largest names on the internet discovering, in public, on a Friday, that they all shared one phone book and had never made a copy.
Sources, for the record: Dyn's incident statements of October 2016, including Kyle York's post-incident statement and Scott Hilton's statement of October 26, 2016 (the 100,000 endpoint estimate, the retry storm correction, and the refusal to confirm the 1.2 Tbps estimates); Symantec Security Response analysis of Mirai (the 62 credential combinations, September and October 2016); US Department of Justice statements on the Mirai guilty pleas (December 2017) and the Alaska sentencing (September 2018); Krebs on Security reporting on the plea documents and the Minecraft hosting motive, January 2017 through December 2017; RIPE Labs' contemporaneous account of the attack timeline.
The internet was not defeated by a superweapon on October 21, 2016. It was defeated by the password "admin," a hundred thousand times, in the service of a Minecraft feud, and the only copy of the phone book burned while every building it described stood untouched and open for business.