On January 31, 2017, GitLab's production database ran into trouble during a replication problem late at night. An engineer, working to fix it, opened a terminal on what was believed to be the secondary database server and ran a command to clear out its data directory so replication could start fresh. The terminal was connected to the primary. The command deleted roughly 300 gigabytes of live production data. Then the team discovered the part that turns an accident into a legend: of the several backup and replication methods they believed they had, most were not working, were not complete, were not restorable in the way they needed, or were snapshots of the wrong thing. The best available recovery source was a manually created backup that was hours old. GitLab stayed transparent in a way the industry still talks about: a public incident document, a livestreamed recovery effort, and a postmortem that named the failures plainly. They lost about six hours of data for a subset of users and spent many more hours restoring service.
If you have never typed the wrong thing into the wrong terminal while tired, you are either very new or very lucky. The engineer in this story is not the villain. The villain is a system in which one tired person, at the worst hour, with a production shell, could turn a mistaken belief about which machine they were on into irreversible deletion, and in which five layers of supposed protection turned out to be a single layer of hope wearing five hats.
Let us count the hats, because this is the useful part. There were database dumps that had been failing silently because of a version mismatch. There was replication, which faithfully replicated the deletion, because replication is not backup; it is a photocopier that also copies fires. There was cloud disk snapshotting that had not been enabled for that server. There was a backup to object storage that was an empty bucket because the tool had not been configured correctly. And there was the one manual backup, taken hours earlier by someone preparing for maintenance, which became the rope the whole company climbed out on. The incident report reads like a farce and lands like a sermon: a backup is not a file. A backup is a restore you have actually performed.
We keep a rule in our studio that sounds excessive until GitLab's night becomes your night: copy, hash verify, independent read, register, confirm recoverability, and only then reclaim the source. Six steps where one would be faster. The step everyone wants to skip is recoverability, because it is slow and because the copy "obviously" worked. GitLab had copies. What they lacked, until the worst possible moment, was evidence that the copies could become a database again.
The second lesson is about destructive commands and mistaken identity. Our factory moves and deletes data under a law that borders on paranoia: identity is proven by physical and logical identifiers before any destructive step, never by a hostname you think you remember or a drive letter that Windows assigned on a cheerful morning. Terminals should scream their identity. Prompts should carry the host and the stakes. Destructive operations should require the operator to prove, mechanically, that the target is the target. Not because operators are careless. Because operators are human, at 11 p.m., during an incident, exactly when care is scarcest and consequences are largest.
The third lesson is reputational, and GitLab earned their good ending. They told the truth in public, in real time, with the ugly parts left in. Users watched them work. The postmortem did not hide behind "a database issue." It listed the failed safeguards one by one. That transparency did not restore the lost data, but it restored something a quieter company would have spent: trust. In an industry that loves the phrase "postmortem culture," GitLab's January night remains the reference because they behaved like the incident belonged to their users too.
Our version of their rule is laminated over the storage desk: never delete the only authoritative copy blindly; never call something a backup until it has survived being read back by an independent path; never let a job that threatens the reserve run without admission checks. And when something does go wrong, the report names what failed, in order, without makeup.
A database was deleted by a person who thought they were somewhere else, and rescued by the one backup nobody had automated. The machines did exactly what they were told. As usual. The lesson is to make sure that what they are told is wrapped in enough verification that a single human mistake has to work much harder to become a company event.
Sources, for the record: GitLab's public postmortem of the January 31, 2017 production database incident documents the accidental deletion of about 300 GB of data, the state of the five backup and replication mechanisms, the roughly six hours of data loss for affected data types, and the recovery from a manually created backup. The incident was documented publicly by GitLab as it unfolded and in the postmortem published days later.