Build notes

Our studio learned to run on hashes the way you learn to run on anything: by being burned in its absence. The hash ends the argument before it starts.

WannaCry hit 200,000 computers in 150 countries, shut wards across the NHS, and was stopped by a $10.69 domain. The patch had existed for fifty-nine days.

On October 1, 2013 HealthCare.gov opened to about 4.7 million visitors and completed exactly six enrollments, because nobody owned the whole system.

SolarWinds shipped a backdoor to fewer than 18,000 customers inside clean updates, because SUNSPOT swapped one file only while the build was running.

On July 19, 2024, one trusted security update reached millions of Windows machines at once, and the world learned what an unlimited blast radius costs.

Eighteen thousand networks installed a signed SolarWinds update that carried a backdoor, because the signature proved who shipped it, not what was inside.

One mistyped input removed the servers under Amazon S3 in 2017, and the internet lost its index: the data was perfectly safe, and nobody could find a thing.

On September 20, 2016, a Mirai botnet of about 145,000 hacked cameras and DVRs hit KrebsOnSecurity at 623 Gbps, and the lasting defense was a password law.

This is not a metaphor stretched for comedy. It is a mapping precise enough to be useful, and once you see it, you cannot unsee why your standups work or fail.

Public Health England lost 15,841 COVID cases inside a legacy Excel format capped at 65,536 rows, and nobody counted what went in against what came out.

A $125 million spacecraft was destroyed by a units conversion: one team expressed force in pounds, the other expected newtons, for nine straight months.

MCAS listened to one angle of attack sensor and no other, with no vote and no cross-check: two 737 MAX crashes, 346 dead, and the fix that followed.

The calendar answers what was supposed to happen. It is silent on what happened: why our factory judges every standing job by its run record, not its schedule.

Mars Pathfinder kept resetting itself days after landing on July 4, 1997. The cause was priority inversion, and the cure had flown to Mars switched off.

On June 4, 1996, Ariane 5 veered off course and was destroyed thirty-seven seconds into its first flight: a 64-bit number squeezed into just 16 bits.

A lost response, a second click, two charges: why retries without an identity duplicate work, and the small index-card fix that makes asking twice safe.

The replacement phone caught fire on a parked airliner, Samsung shipped a second defective battery, and the United States banned the Note 7 by name.

When fire destroyed the OVHcloud data center in Strasbourg in March 2021, customers learned their backups had burned in the same building as their servers.

One tired engineer, one terminal on the wrong server, 300 gigabytes of production data gone: a backup is not a file, it is a restore you have performed.

One leap second in 2012 livelocked Linux servers worldwide, then broke Cloudflare in 2017. The timekeepers voted to abolish it. Time can go backwards.

Between 1985 and 1987, a radiation therapy machine called the Therac-25 delivered massive overdoses to at least six patients in the United States and Canada.

In 1988 a graduate student tried to count the internet. His worm infected about 6,000 machines, roughly one in ten, and forced the creation of CERT.

On a Friday in 2016, a hundred thousand cameras with factory passwords burned the internet phone book, and healthy sites became impossible to find at all.

On May 6, 2010, the Dow fell 998.5 points in minutes as a sell algorithm, hot potato volume, and stub quotes fed each other, then mostly recovered by 3 p.m.

A 43-second network partition split GitHub into two databases in October 2018, and choosing data integrity over usability turned it into a 24-hour recovery.

On October 4, 2021, Facebook withdrew the routes to its own DNS servers and vanished: connectivity is belief, and belief withdrawn cannot shout back.

Equifax had the patch, missed the system, scanned the wrong directory, and left its traffic inspector blind for 19 months on an expired certificate.

Air France Flight 447 stalled from 38,000 feet to the sea after ice blocked its pitot probes, the autopilot resigned, and its stall warning inverted.

The Y2K bug never bit because an army fixed it first, and the world called the prevention hype: why success erases the evidence that work was needed.

In March 2021 the Ever Given wedged sideways in the Suez Canal for six days, and a planet built for efficiency learned the price of having no margin.

A race condition silenced the alarms in Akron in 2003 and 55 million people lost power: a monitoring system must be built to notice its own death first.

On May 18, 2012 Facebook priced at $38 and printed at $42.05 half an hour late, after a validation loop left more than 30,000 orders stuck for hours.

In April 2018 TSB moved 1.3 billion records for 5.2 million customers in one weekend big bang, and the new platform locked customers out within minutes.

Southwest cancelled 16,900 flights at Christmas 2022 not because planes broke, but because its crew scheduling software lost track of who was where.

Knight died of an unverified boundary: between old and new, between intended and actual, between the deployment they planned and the deployment that existed.

Colonial Pipeline shut down 5,500 miles of pipeline in May 2021 after attackers entered through a legacy VPN account that had a password and no second factor.

On 15 January 1990, AT&T's long distance network spent nine hours healing itself to death: one mis-scoped break in 114 switches, spreading by recovery.

Twenty-three hours of silence from a healthy machine: our silent host postmortem, and why the runbook now starts with the radios, not the exotic causes.

During Apollo 11, the guidance computer raised five program alarms in four minutes, and a priority scheduler plus a handwritten list kept Eagle flying.

Episode 3 of The Long Ping is live: a three-hour-and-forty-three-minute film about how power grids fail, why they fail that way, and what every system learns.

Episode 2 of The Long Ping, chapters 1 to 10, live on YouTube and Rumble: the water wheel, the grid it built, and the long argument over centralised power.

Episode 1 of The Long Ping, live on YouTube with its five-part transcript on Substack: if everyone maintaining your world vanished tonight, what survives?

How a trades-company website became the proof of concept for a multi-tenant platform model, and what the transformation taught about configuration boundaries.

Stateless AI starts from zero every session. Systems that act in the real world need continuity: events, knowledge, evidence, and replay. Memory, not retrieval.

How agents run commands across machine boundaries without corruption, and why capability separation matters more than clever prompting. Silence is the danger.

Never claim what you cannot prove: how PING attaches evidence to claims, grades verification, and makes failures visible instead of silent. Show the derivation.