The night the guard opened its eyes
On the night of July 29, 2017, at about nine o'clock, a team at Equifax's data center in Alpharetta, Georgia did a piece of routine maintenance. They uploaded 67 fresh SSL certificates to a box called an SSL Visibility appliance, a device whose entire job was to sit in front of Equifax's consumer dispute portal, decrypt the traffic flowing in and out, read it, and notice if anything about it looked like a crime in progress.
The device woke up, looked at the traffic for effectively the first time in a year and a half, and immediately saw a crime in progress.
Suspicious requests were running to an address in China. Responses were carrying out more data than a dispute portal has any business carrying. The portal was taken offline the next day, July 30. The chief executive was told on July 31. And the investigation that followed established that the attackers had been inside for 76 days, that they had run roughly 9,000 queries against 48 databases the portal was never supposed to reach, and that they had walked away with personal information on what would eventually be counted as 147.9 million people.
Here is the part that deserves to be printed on a poster. The appliance had not been broken. It had not been hacked, disabled, unplugged, or outsmarted. Its SSL certificate, the small dated credential that let it decrypt and inspect traffic, had expired on January 31, 2016. For 19 months the device sat in the rack, powered on, passing traffic through, inspecting none of it, because its permission slip had lapsed and nobody's job, apparently, was to check the permission slip. Nor was it an anomaly: the House Oversight Committee later found Equifax had more than 300 expired certificates on its books, including 79 involved in monitoring business-critical domains. Three hundred. The company did not have a certificate problem. It had a certificate system, running at scale.
And the device's default behavior deserves its own line on the poster. When the certificate expired, the appliance did not stop traffic or refuse to work. It let the traffic flow through uninspected. Engineers call this failing open. It is a defensible design choice for keeping a business running, right up until you remember that the business was holding Social Security numbers for half the adults in the United States, and the whole point of the box was to be the thing that notices. A smoke detector that decides, when its battery dies, to simply stop mentioning smoke is not a broken smoke detector. It is decor.
So the cold open of this breach is not a hacker movie scene. Nobody rappelled past a laser grid. The lasers were installed, aimed, and powered. They had been wearing a blindfold for 19 months, and the building had gotten so used to it that the blindfold had stopped appearing in anyone's mental picture of the room.
The patch that sat
The second failure is almost disappointingly ordinary, and it came first in time.
Apache Struts is a framework for building Java web applications, and on March 6, 2017, the Apache team disclosed CVE-2017-5638: a flaw in its Jakarta Multipart parser that let a remote attacker execute commands on a server by sending it a malformed Content-Type header. No login required. No subtlety required. Apache released fixed versions the same day, bulletin S2-045, and exploit code was circulating essentially immediately. This was not a whisper in a research paper. It was a fire alarm with the patch attached to it.
The alarm reached Equifax fast, which is what makes the rest of the timeline so hard to read. The Department of Homeland Security alerted the company on March 8. On March 9, Equifax's Global Threat and Vulnerability Management team emailed the alert to over 400 people, about 430 individuals and distribution lists, with an instruction that could not have been much clearer: if you run Apache Struts, patch it within 48 hours, because the flaw is critical and is already being exploited. On March 16 the team covered the same flaw at its monthly threat meeting and sent the slides to the same list.
Then the machinery of a large company did what large-company machinery does when a task has 430 owners. It is a law of organizational physics: an email sent to 430 people has been sent to no one, with excellent documentation. The FTC's later complaint puts its finger on the exact missing step. The security team ordered the patching and did not follow up to make sure the responsible people had actually done it. There was an order. There was a deadline. There was no verification that the world matched the order, and in the gap between the two, an entire credit bureau fit comfortably.
The one system that mattered most was ACIS, the Automated Consumer Interview System: Equifax's internet-facing portal where ordinary people went to dispute errors in their credit files. Objection, your honor, the name alone should have triggered a review. This was a custom-built legacy system with roots in the 1970s, still standing in 2017 with a door open to the public internet, running the vulnerable version of Struts. The attackers found it before Equifax's patch process did. In fact, according to GAO's reconstruction, unidentified actors were already probing Equifax servers for this exact vulnerability by March 10, one day after the internal email, and found the dispute portal sitting there unpatched.
Equifax did try to look. That is the detail that elevates this from negligence theater into something more instructive and more frightening. The company ran scans for the Struts flaw. The scans came back clean. The reason, recorded in the House report, is that the scan examined the root directory of the system and not the subdirectory where Apache Struts was actually listed. Read that the way an engineer reads it. The check ran. The check reported success. The check was pointed at the wrong place, and because the output of a mis-aimed check looks exactly like the output of a passed check, everyone downstream treated the clean result as a fact about the world instead of a fact about the scan. It is checking the lobby thoroughly, finding no intruders in the lobby, and filing a report that the building is secure, while the intruders are in the back office reading the filing cabinets.
Underneath the scan sat the quietest failure of the three: the inventory. Patching every copy of a piece of software requires knowing where every copy lives. Equifax's process leaned on people knowing what was running on their systems and saying so. An asset inventory that depends on memory and self-reporting is not an inventory. It is a rumor with a spreadsheet attached. ACIS was old, custom, half-forgotten, and exactly the kind of system attackers look for first.
76 days
The attackers came back on May 13, 2017, this time to stay.
They came in through the unpatched Struts flaw in ACIS and did the standard, unglamorous work of a competent intrusion. They planted web shells, small scripts that give an outsider remote control of a server; investigators counted about 30 installed over the course of the attack. They explored. And then they found the gift that turned a foothold in a dispute portal into the largest credit bureau breach in history: an unsecured file containing administrative usernames and passwords, stored in plain text.
Pause on that file, because it does more work in this story than the famous vulnerability did. The Struts flaw got the attackers through the front door of one aging portal. The plaintext credential file handed them the keys to the building. ACIS needed access to three databases to do its job. It was not walled off from the rest of the network, so with harvested credentials the attackers reached 48 other databases that had nothing to do with consumer disputes. Segmentation is the unglamorous discipline of assuming the front door will someday open for the wrong person and arranging the interior so that it does not matter very much. Equifax had arranged the interior so that it mattered completely.
Over 76 days the attackers ran approximately 9,000 queries. Most came back empty or useless to them; 265 returned personal information. They saved the results, compressed them, staged them where they could be reached, and pulled the data out in small pieces over encrypted connections, which is to say over exactly the kind of traffic the SSL Visibility appliance existed to examine. Every layer of this exfiltration ran through the blindfold. The data did not sneak out a side window. It walked down the main corridor, past the guard station, in a steady procession, for two and a half months, while the guard's eyes were closed by an expired credential and the guard station's default setting was to wave everyone through anyway.
Seventy-six days is long enough for a monitoring rotation, a patch cycle, and several thousand individual moments in which a functioning inspection device would have had something to say. The device said nothing, because it could see nothing. Its one genuinely impressive performance came on July 29, when the certificates were renewed for unrelated maintenance reasons and it spotted the exfiltration essentially on sight. The tool worked. The tool had always worked. It had been waiting, sightless, for somebody to hand it a current piece of paper, and the instant somebody did, a 76-day breach became visible in an evening. Security tooling you cannot see through is decoration, and Equifax's decoration turned out to be an excellent detective on the one night in 19 months it was allowed to open its eyes.
The side story, labeled as reporting
One coda belongs in the record with its label on, because it is journalism rather than an official finding, and because no honest telling of Equifax's 2017 can quite leave it out.
In September 2017, days after the U.S. breach was announced, Brian Krebs reported at KrebsOnSecurity that researchers at Hold Security had looked at an Equifax employee portal in Argentina, part of its Veraz operation, used for handling consumer disputes there. According to Krebs's reporting, the portal's administrative side was protected by the username admin and the password admin, and once inside, the researchers could see records for more than 100 employees and thousands of consumers, including Argentine national identity numbers. Equifax took the portal offline after Krebs contacted the company and said the Argentine system was not connected to the U.S. event. That is the right way to hold this story: reported, sourced, and separate from the breach itself. It is not evidence about how the U.S. intrusion happened. It is evidence about the environment it happened in, a patchwork of side systems where the distance between the security policy and the actual password on an actual door could be measured in a single guess. After ACIS, admin/admin does not read as a coincidence. It reads as a climate.
The fallout, and the price of a blindfold
Equifax announced the breach publicly on September 7, 2017, six weeks after discovering it, and put the number of affected U.S. consumers at 143 million. The number then did something numbers in breach announcements tend to do when the underlying inventory was assembled from memory: it grew. In October 2017 the company added 2.5 million people, taking the total to 145.5 million. On March 1, 2018, it added 2.4 million more. The final figure was 147.9 million U.S. consumers, roughly half the country, including, per the FTC, at least 147 million names and dates of birth, 145.5 million Social Security numbers, and 209,000 payment card numbers with expiration dates. The House Oversight Committee, rounding, called it 148 million and delivered its verdict in a sentence that has no comedy in it at all: the breach was entirely preventable.
The company's leadership departed in stages over the following weeks, as leadership tends to do when Congress starts using the word preventable. Then came the long tail: investigations by the FTC, the CFPB, and attorneys general across the country; a GAO reconstruction; a House report; a Senate report. The paper trail is one of the most complete autopsies in the industry's history, which is its own kind of indictment. Nothing about the failure was exotic. Every step of it was diagrammed by auditors using ordinary documents the company itself had produced.
On July 22, 2019, the bill arrived in its most quotable form. Equifax agreed to a global settlement of at least $575 million, and potentially up to $700 million. The structure matters more than the headline. Three hundred million dollars went into a fund for affected consumers, with up to $125 million more behind it if the first amount ran dry, taking the consumer fund to as much as $425 million. Another $175 million went to 48 states, the District of Columbia, and Puerto Rico. Another $100 million was a civil penalty to the CFPB. Beyond the money, Equifax accepted an information security program with annual risk assessments, third-party assessments every two years, and annual certifications from its own board, a regulator's way of saying: from now on, someone with a name signs a document saying the checks were done, so that next time there is a signature to ask about.
Divide the headline by the headcount and the settlement comes to under five dollars per affected person, which tells you less about what the data was worth than about how impossible it is to make 147.9 million people whole with a fund. The real cost is distributed across every one of those people, forever, in the form of a Social Security number that cannot be reissued like a password and a date of birth that cannot be changed like one, now permanently in circulation. A credit card can be canceled. A birthday is load-bearing. Fittingly, the one thing that expires, the certificate, was the thing nobody renewed, and the things that never expire, the stolen identifiers, are the ones everybody else is stuck renewing their vigilance over for life.
The lesson for any shop whose monitoring depends on things that expire
Strip the names off this story and it stops being about a credit bureau. It becomes a story about three quiet systems every organization runs, including ours: the patch system, the inventory system, and the monitoring system. Equifax's breach was not one failure. It was all three failing at once, quietly, in ways that produced reassuring output. The patch email reported a successful broadcast. The scan reported a clean system. The appliance reported nothing at all, which everyone had learned to read as health. Each system, asked how things were going, answered fine. The answers were the most dangerous artifacts in the building.
So the practical lessons, in the order an ordinary shop can actually use them:
First, everything that expires is a monitoring dependency, and it needs its own monitor. Certificates, tokens, licenses, credentials, the log pipeline's disk, the backup job's storage: if your ability to see depends on a dated object, its expiry date is a production alert with an owner and a lead time, not a calendar surprise. And the monitor of the monitor needs checking by a second, stupidly simple path, because a sophisticated observability stack will happily report green from inside the blindfold. The cheapest version is a standing question, asked mechanically and often: when did this system last actually see something? A detector that has detected nothing for 19 months is not having a quiet year. It needs its eyes examined. Silence from a sensor is a signal about the sensor before it is a signal about the world.
Second, a patch order is not a patch. Equifax had a policy, a deadline measured in hours, and an email list of 430 people, and what it lacked was the unglamorous closing step: proof, per system, that the fix was in, produced by a mechanism that checks the system rather than asks the owner. Closure by reply-all is how you get a 48-hour policy and a two-month exposure. The rule that survives contact with reality is the one our studio runs on, learned from much smaller disasters: a claim is not a fact, and work counts when a verifiable artifact exists, not when somebody says it is done. Knight Capital lost $440 million to a deployment that was seven-eighths verified. Equifax lost half the country's identifiers to a patch verified by a scan of the wrong directory. Same accident, different suit: the boundary between intended and actual, unmeasured.
Third, inventory is not paperwork; it is the index every other control searches. You cannot patch what you cannot list, you cannot segment what you cannot map, and you cannot count your victims if your asset list was a rumor, as Equifax demonstrated by revising its headcount upward twice across six months. The inventory has to be mechanical, generated from the systems themselves, and reconciled against what the network can actually see, because the systems it forgets are precisely the old, custom, internet-facing ones an attacker enumerates for free with a scanner on a Tuesday.
And one more, small enough to fit on the poster: decide, in advance and in writing, whether your safety devices fail open or fail closed, and argue about it on a calm day. Equifax's appliance failed open by default, and that default, chosen for some forgotten operational convenience, converted an expired certificate from an outage into an invisibility cloak. There are systems where failing open is right. A system whose only job is watching the door of the vault is not one of them.
None of this requires Equifax's budget, which is the bleakest part of the story. The patch was free, the scan could have been aimed one directory deeper, the certificate renewal was routine maintenance that eventually happened anyway for unrelated reasons, and the segmentation was a design decision, not a purchase. What failed was not spending. What failed was verification, the discipline of making systems prove, continuously, that the protection you believe you have is the protection that exists.
Our studio is a film operation, not a credit bureau, and our failures cost footage and deadlines rather than national identity theft. But the shape of the failure is identical at every scale, which is why this story belongs in our canon. We run fleets of workers whose results only count when artifacts land and verify. We treat a sensor that reports nothing as a suspect, not a success. We learned those rules from small, embarrassing versions of the same three quiet failures: the job that said it ran, the scan that checked the wrong place, the monitor that watched faithfully over a blindfold. Equifax is what those small failures grow up to be when the building holds 147.9 million people.
Sources, for the record: U.S. House Committee on Oversight and Government Reform, The Equifax Data Breach, Majority Staff Report, December 2018 (the certificate dating and 19 months without visibility, the 67-certificate renewal on July 29, 2017, the scan of the root directory rather than the subdirectory, the 76 days, approximately 9,000 queries, 48 databases, 265 results, 300-plus expired certificates, and the verdict that the breach was entirely preventable). U.S. Government Accountability Office, GAO-18-559, August 2018 (at least 145.5 million people affected, the March 10 probing, and the same certificate described as expired about 10 months before the breach occurred). Apache Security Bulletin S2-045 and the NVD record for CVE-2017-5638 (disclosure and patch, March 6, 2017). FTC press release and complaint, July 22, 2019 (the 48-hour patch order without follow-up, the data totals, and the settlement: at least $575 million, up to $700 million, structured as a consumer fund of $300 million plus up to $125 million, $175 million to the states, and $100 million to the CFPB). Equifax's public announcements of September 7, 2017 and March 1, 2018, as contemporaneously reported (143 million, then 147.9 million). The Argentina passage is KrebsOnSecurity reporting from September 2017 on research by Hold Security, labeled as reporting in the text.
The alarm worked perfectly, for 19 months, at the job of watching a blindfold, and on the one night somebody finally renewed a piece of paper, it caught the burglars still standing in the vault.