On Sunday 4 October 2020, Britain's official coronavirus dashboard did something dashboards are not supposed to do. It confessed.
The headline figure that day was 22,961 new cases, by far the largest daily number the country had yet recorded, and attached to it was a note, written in the flat prose of people who know the number is about to be on the news: the figures for 3 and 4 October included 15,841 additional cases with specimen dates between 25 September and 2 October, and were therefore "artificially high." Which is a careful, civil service way of saying that for eight days the national count of a lethal pandemic had been wrong in the direction that kills people, and that the missing cases had not merely been miscounted. They had never been handed to the contact tracers whose entire job was to find the people those cases had breathed on.
Nobody had hacked anything. No test had failed. No scientist had lied. The results existed, pixel perfect, in files sent in good order by the laboratories that ran the tests. They were lost in the last few feet of the pipeline, inside one of the most familiar pieces of software on earth, by a limit so old it predates the World Wide Web: a spreadsheet format that stops counting at 65,536 rows and, when it reaches the end, does not scream, does not warn, does not even shrug. It just stops writing things down.
This is the story of how a file format ate 15,841 cases of a plague, and why the most dangerous sentence in data engineering is "the file loaded fine."
The pipeline
To see where the cases went, you have to follow a swab.
By the autumn of 2020, England's testing system ran in pillars, and the pillar that mattered for community cases was Pillar 2: swabs taken at drive through sites, walk in centres, and home test kits, processed not by hospital laboratories but by commercial firms running enormous Lighthouse labs. These firms did their part well. A swab went in, a result came out, and the results were compiled into files and sent to Public Health England, the agency charged with turning raw lab output into the national picture.
The format the labs used was CSV: comma separated values, the cockroach of data formats, a plain text list that has survived every software fashion since the punched card's funeral and will survive several more. CSV has no opinions. It has no row limit worth the name. A CSV file can hold ten results or ten million; it is just lines of text, and it arrives complete or it does not arrive at all. As an input to a national disease surveillance system, it was a perfectly respectable choice.
Then PHE's systems took those files and did the thing that a thousand organisations do every day without thinking, the thing that feels so normal it does not register as a decision at all. An automated process pulled the lab data together into Excel templates, which were then loaded into the central systems feeding two consumers: the public dashboards, and NHS Test and Trace. One of those consumers produced charts. The other produced phone calls to frightened people telling them to stay home. Both were fed by the same templates.
And the templates, it emerged, were saved in XLS.
The ceiling
XLS is the original Excel file format, with a lineage running back to 1987. When it was designed, 65,536 rows was not a limitation. It was a boast. No human being in 1987 had 65,536 of anything to put in a spreadsheet, and if they had, they would have been gently advised to seek help. The number is 2 to the power of 16, the kind of round binary ceiling you choose when the machine in front of you counts memory in kilobytes and the ceiling sits so far above any conceivable workload that it functions, for all practical purposes, as infinity.
Microsoft replaced XLS with XLSX in 2007. The new format raised the ceiling sixteen fold, to 1,048,576 rows, and widened the sheet from 256 columns to 16,384. The old format did not die, because file formats never die; they linger in templates, in macros, in the automated process somebody built in a hurry and everybody since has been afraid to touch. PHE's process, by the BBC's account, had been built on the old template. Health Secretary Matt Hancock, standing in the Commons on the Monday after the story broke, used the phrase every engineer dreads hearing applied to their own work: a "legacy system." He added, with the air of a man reading out his own alibi, that a decision to replace it had been taken in July and the contracts were already awarded. The replacement was coming. The virus, unfortunately, had not waited for the procurement to complete.
Now, the ceiling alone does not explain 15,841. Sixty five thousand rows is a lot of rows. The multiplier was in the shape of the data. Each test result, once expanded into the template, occupied several rows, not one. Do the arithmetic the way the BBC laid it out and each template could hold roughly 1,400 cases. One thousand four hundred: that is the capacity of the vessel into which a nation poured the results of a testing programme running at hundreds of thousands of tests a day and climbing. Every time a template filled, the process did not halt. It did not raise an alarm, write a log line in red, or send anyone a confessional email. The remaining results were simply left off, the way the last guests at a party are left off a group photograph, except the photograph was the national pandemic response and the guests were infectious.
There is a footnote here that deserves to be a headline, because it is the most honest part of the whole affair. When the story broke, the first technical explanation to race around was about columns. Word went out that PHE had logged each case in a column and run out of columns, and this sounded plausible because Excel's column limits are small enough to bite. The Register, to its credit, printed the correction in the same article as the rumour: the column story wilted, and the row story replaced it. It matters because it shows how these failures actually present in the wild. The system gives you no error message to work from, so intelligent people invent one, and the first invented explanation is usually wrong, and while everyone is arguing about columns, the rows keep quietly ending at 65,536.
The silence
The feature of this failure that should be printed on mugs is that nothing, at any point, appeared to be wrong.
Every person who took a test got their result. PHE was emphatic about this in its statement, and it was true: if you tested positive, you were told, and you were told to self-isolate. The dashboards updated daily, with numbers that rose in a smooth, plausible curve. The contact tracers traced the cases they were given. Each part of the system worked, in the sense that a watch with one gear missing still ticks most of the time. The failure lived in the join between parts: in the difference between the number of results the laboratories sent and the number of cases the central system received. And nobody, anywhere in the chain, was counting both sides of that join.
The agency's own table, published with its apology, tells the story in numbers if you read it as a graph of a vessel filling up. On the first day of the window, 957 cases went missing. Then 744, then 757. Then a day with zero missing, a Sunday lull. Then the wave arrived: 1,415, then 3,049, then 4,133, then 4,786 in a single day. That is not a constant fault. That is a fault whose size is proportional to the load, because it is not a fault in a machine at all. It is a capacity masquerading as one. As England's second wave pushed more results through more templates, more templates hit their ceilings, and each ceiling ate everything above 65,536 rows without leaving a mark. By the last three days, three quarters of the entire loss was concentrated: 11,968 of the 15,841 cases. The system was not degrading. It was succeeding, at a fixed size, in a growing world.
It was found the way these things are usually found, not by monitoring but by a human noticing that the world and the spreadsheet had drifted apart: identified, in PHE's phrase, "overnight on Friday 2 October." The recovery was brisk and slightly medieval. All outstanding cases were transferred to the contact tracing system by 1am on the Saturday morning. The mitigation, in the agency's own words, was to split large files into smaller batches, which is the data engineering equivalent of carrying water in two buckets because the one bucket has a hole of a very specific size. And the public explanation, also in the agency's own words, was that "some files containing positive test results exceeded the maximum file size." File size. Not format, not rows, not the name of a 33 year old specification. If you want to know why the public record of this incident had to be assembled by the BBC rather than by the institution that owned the pipeline, that sentence is the whole answer. The institution described the symptom in the vaguest available units. The format that caused it was discovered by journalists.
The bill
What does a week of missing cases actually cost? The cases themselves were eventually traced; the tracers worked the weekend, call attempts per case were raised from 10 to 15, and the backlog was chased down. But contact tracing is a perishable good. Its value lives in the first days, when a contact can be found before they become a case. A call that arrives on day nine is epidemiology. A call that arrives on day two is prevention. The glitch converted prevention into epidemiology for 15,841 people and everyone they had been near. The Guardian estimated that as many as 50,000 potentially infectious people may initially have been missed by the tracers, a figure built from the contacts each case would have named.
Then the economists arrived, because economists cannot resist a natural experiment, and this one was perfect in the grimmest way. The missing cases had not been lost evenly. They were lost in proportion to how full each area's templates happened to be, which is to say, almost at random with respect to everything else. Two researchers, Thiemo Fetzer and Thomas Graeber, compared the places the spreadsheet had eaten with the places it had spared, and measured what happened next. What happened next was worse: more infections, higher positivity, more deaths, in the eaten places, in a pattern that tracked the glitch and not the underlying behaviour of the populations. Their conservative estimate, the one they were willing to defend with the cautious wording economists reserve for numbers they believe are understated, associates the eight day failure with more than 125,000 additional infections and over 1,500 additional deaths. The paper's title is a question: "Does Contact Tracing Work?" The spreadsheet answered it, at a cost no ethics board would ever have approved, by switching tracing off for a random sample of England and letting everyone watch the results.
Hold the two ends of that together. At one end, a file extension. At the other, a peer reviewed estimate of four figures of deaths. In between there is no villain, no sabotage, no moment where a person looked at a warning and clicked through it. There is a format limit chosen when floppy disks roamed the earth, a template nobody re-examined, and a join between two systems that nobody instrumented. Engineers have a saying for systems like this, usually delivered while staring into a pint: it worked in testing. It did. Testing happened at volumes the format could hold. Production happened at volumes a pandemic could hold. The gap between those two volumes is where the bodies are buried, in this case close to literally.
The lesson
The tidy moral, the one in the headlines, is "do not use Excel for national infrastructure," and it is wrong, or at least insufficient. Excel did exactly what its specification said it would do. It held 65,536 rows. The specification was not a secret; it was old enough to vote in two countries. The failure was not the tool. The failure was a chain of small, reasonable, unexamined decisions: a template inherited from an earlier system, a format nobody chose so much as failed to un-choose, and a pipeline with no reconciliation step, no place where a human or a script asked the only question that mattered: does the number that came out equal the number that went in?
That question is embarrassingly cheap. Rows in, rows out. Count them, compare them, and make the comparison loud. It is the data world's version of the count of sponges before and after surgery, and surgery learned to do it after losing the alternative argument. Every system that moves records between owners needs it, because the defining property of silent data loss is that both sides look healthy: the sender sent, the receiver received, and the difference fell on the floor in a format boundary, a truncation, a silent type conversion, a limit written into a spec before the engineers were born. The sender's dashboard is green. The receiver's dashboard is green. The missing records do not have a dashboard. Missing data is the only bug class whose primary symptom is that nothing appears to be a symptom.
Our own studio runs on a phrasing of this rule blunt enough to survive tiredness: a claim is not a fact, and a file that loaded is not a file that arrived. Every worker in our little factory hands over artifacts with counts attached, and the counts get checked by something that did not do the work, because the 15,841 taught anyone paying attention that the most expensive failures are the ones where every participant can honestly say their part ran fine. PHE's pipeline was, by that standard, a complete success at every individual step: the labs analysed, the files loaded, the dashboards published, the tracers traced. The truth lived only in the totals, and nobody owned the totals. Own the totals. That is the entire lesson, and England paid for it at a rate of roughly 2,000 cases a day.
There is a coda worth keeping. The system's defenders pointed out, correctly, that the cases were a fraction of the whole, that everyone's own result reached them, that the dashboards were backfilled within days and the scientific record corrected. All true, and all beside the point, because the point of a surveillance system is not to be mostly right in the archive. It is to be complete on the day, when completeness is the difference between a phone call that prevents a chain of infections and a footnote in someone else's apology. The spreadsheet did not fail Britain in the long run. It failed Britain on eight specific days, at the exact moment the second wave was gathering, in the one direction, undercounting, that makes a growing fire look smaller than it is. Formats do not care about timing. That is what engineers are for.
Sources, for the record: PHE's official statement on GOV.UK, 4 October 2020, gives the 15,841 figure, the 25 September to 2 October window, the 11,968 concentration in the final three days, the per day table, the 1am 3 October transfer to contact tracing, the "maximum file size" wording, and the file splitting mitigation. The mechanism (CSV inputs, automated Excel templates in the legacy XLS format, the 65,536 row limit, several rows per result, roughly 1,400 cases per template) is from BBC News reporting of 5 October 2020; The Register's 5 October account documents the early column limit explanation and its replacement by the row explanation. Matt Hancock's "legacy system" remarks and the July replacement decision are from his Commons statement of 5 October 2020 as reported by Digital Health. The dashboard figure of 22,961 cases on 4 October and its "artificially high" note are from the government's coronavirus dashboard as reported at the time. The estimate of up to 50,000 potentially infectious people missed is The Guardian's, 5 October 2020. The infection and death estimates are Fetzer and Graeber's, Warwick Economics working paper, November 2020, later in PNAS 2021, and are econometric estimates, stated by the authors as conservative.
A spreadsheet never tells you what it failed to write down; it just hands you a shorter list with the same confidence as the full one.