23:59:60
There is a second that does not exist on your clock, and it has a body count.
On June 30, 2012, the last minute of the day in Coordinated Universal Time had sixty-one seconds. The sequence ran 23:59:58, 23:59:59, 23:59:60, and only then midnight. That extra second, the twenty-fifth leap second since 1972, had been announced six months earlier by the International Earth Rotation and Reference Systems Service, the body that watches the planet spin and files the paperwork when the planet falls behind. Nobody could claim surprise. The notice went out in January. The second arrived on schedule.
Within minutes, some of the most visited sites on Earth started falling over. Reddit went down. Mozilla's infrastructure choked. LinkedIn, Yelp, Foursquare, StumbleUpon, and Gawker all reported trouble. In Australia, the Amadeus Altea reservation system was taken offline for about an hour, and staff at Qantas and Virgin Australia checked passengers in by hand, with pens, like it was 1975. One second, inserted on purpose by the people in charge of time itself, had done all of it.
No attacker did this. There was no exploit, no intrusion, no bad deploy that weekend. The planet's rotation had drifted a fraction of a second away from the atomic clocks, the timekeepers applied the standard correction, and the standard correction detonated a bug that had been sitting quietly in the Linux kernel for five years, waiting for the one code path almost nobody ever runs.
This is the story of that second, twice. Because it happened again four and a half years later, at a different company, in a different language, for the same underlying reason. And it is the story of how the world's timekeepers eventually looked at the wreckage and voted to stop doing the thing that caused it. The leap second is being abolished, not because astronomers stopped caring about the Earth's rotation, but because engineers got tired of being paged by it.
Why the planet needs a patch
The problem starts with the fact that there are two kinds of time, and they do not agree.
Atomic time is the tidy one. Since 1967, the second has been defined by the vibrations of a cesium atom, and atomic clocks keep it with a precision that borders on the theological. Time is a metronome. It does not speed up. It does not slow down.
The other kind of time is the Earth's rotation, and the Earth is a sloppy clock. Its spin is braked by the Moon's tides and nudged by earthquakes, winds, and ocean currents. A day is not exactly 86,400 atomic seconds. It is usually a touch more, and the touch accumulates. Left alone, atomic time and solar time would drift apart, a second here, a second there, until, over centuries, noon by the clock stopped meaning noon by the Sun.
Coordinated Universal Time, UTC, is the treaty between the two. It ticks at the atomic rate, but whenever the difference between UTC and the Earth's actual rotation angle, called UT1, approaches 0.9 seconds, the IERS orders a leap second: one extra second, inserted at the end of June or the end of December, announced about six months ahead in a document called Bulletin C. Twenty-seven of them have been inserted since 1972. Every single one has been positive, an addition. A negative leap second, a second removed, a minute with fifty-nine seconds, is theoretically possible. It has never happened, and no production system on Earth has ever lived through one.
Here is the detail that turns an astronomical curiosity into an engineering hazard. Most computer timekeeping does not actually represent the leap second. The POSIX clock, the one underneath nearly everything, counts seconds on the assumption that every day has exactly 86,400 of them. There is no 23:59:60 in that arithmetic. So when a leap second arrives, the operating system cannot show it. Instead, it cheats: it lets the leap second pass and then sets the system clock back by one second. From the software's point of view, the final second of the day happens twice. The timestamp repeats. Logs from that night carry the cheerful kernel message "Clock: inserting leap second 23:59:60 UTC," and then time politely rewinds and does the second again.
Every assumption your software makes about time is now on the table. That a timestamp identifies a unique moment: false, one just repeated. That subtracting two clock readings gives a duration that is at least zero: false, the second reading can be earlier than the first. That a timer set for half a second from now will fire in roughly half a second: hold that thought.
The bug that waited five years
The Linux kernel keeps more than one notion of now. Beyond the system time, the high-resolution timer subsystem, hrtimer, maintains its own time bases, per CPU, expressed as offsets from system time. When the system time changes, the offsets have to be adjusted to match, and there is a function whose entire job is that adjustment: clock_was_set(). The contract is simple. Change the time, then call clock_was_set(). As long as every time change is followed by that call, the two notions of now stay married.
In 2007, in the course of other work, the call to clock_was_set() was removed from the leap second handling code. Nobody noticed, because the leap second code almost never runs. After the leap second at the end of 2008, there was not another one for three and a half years, and in all that time nothing asked the clock and the timer bases the one question they disagreed on. The bug was not hiding. It was dormant: a loaded code path with no traffic.
On June 30, 2012, the traffic arrived. The leap second code did its job on the system clock: at the end of the extra second, it set the clock back one second. But with no clock_was_set() call, the hrtimer bases never got the memo. They continued running one second in the future relative to the corrected system time. One second does not sound like much until you consider what timers are for. Most timers in a running system are set for less than a second ahead: a thread sleeping briefly, a wait with a sub-second timeout, a retry that backs off for a few hundred milliseconds. Every one of those timers, judged against a clock that believed it was already a second later, was in the past. And a timer in the past fires immediately.
Firing immediately is survivable once. The poison was repetition. Many of these timers are recurring or are instantly re-armed by the code that set them: wait, wake, check, wait again. Each new wait was also in the past, so it also expired at once. The result was a loop: timers expiring, being reset, and expiring again, as fast as the machine could go. Processes were not deadlocked, exactly. They were livelocked, sprinting in place, burning entire CPUs on the urgent business of waking up to discover there was nothing to do yet. Load averages climbed to absurd heights. Machines that ran fine at single-digit CPU utilization went flat out. Some became so unresponsive they had to be rebooted. The emergency fix circulating that night: reset the date by hand, or reboot the box.
The blast radius followed the stack, not the company org chart. Anything big, Java-based, and timing-sensitive was in the blast zone. Mozilla reported its Hadoop and Elasticsearch clusters misbehaving, and its engineer Eric Ziegenhorn posted the diagnosis in real time: Java applications were not working, and it seemed to be related to the leap second happening that night. Reddit's Apache Cassandra nodes spun. Gawker's Tomcat servers struggled. Different products, different vendors, two shared ingredients: Linux underneath, and an absolute faith that timers behave. The kernel bug did not care what your application did. It cared what your application waited on, and everything waits on something.
The unglamorous hero of the night is John Stultz, a kernel developer who had spent years in the timekeeping subsystem, the least glamorous and most load-bearing corner of the codebase. Stultz tracked the failure to the missing clock_was_set() call, the one removed in 2007, and wrote the fix: call it again from the leap second code. Even that was not a one-liner in practice, because clock_was_set() has to update timer bases on every CPU, and the leap second code runs in a context where that work cannot be done directly. His patch detects that situation and defers the resynchronization to a workqueue, to do the job a moment later, safely. The fix for the rare path has to respect the rules of the hot path, or you have traded one disaster for another.
Sit with the shape of this failure, because it is the shape of most great outages. The defect was introduced by a reasonable change. It survived for years because the code path it lived on executes at most twice a year, and some years not at all. It was triggered simultaneously on every affected machine on the planet, at an instant announced six months in advance: no canary, no gradual rollout, no early warning from a smaller fleet. The leap second is a global, synchronized, perfectly scheduled change to the most fundamental input every program has, a deploy with a 100 percent blast radius and a rollback plan that consists of rebooting. Almost nobody tested for it, because testing a thing that happens for one second every few years feels, right up until midnight, like a waste of an afternoon.
The company that lied to its clocks, on purpose
One major operator watched the 2012 pile-up from a safe distance, and it is worth understanding why, because their trick is the most elegant hack in this entire story.
Google had been burned by leap second reasoning early, and by 2008 it had adopted a policy of refusing to let its clocks experience one. The technique is called the leap smear. Instead of stepping the clock back at midnight, Google's time servers spread the extra second across a twenty-hour window centered on the event, slowing every clock by roughly fourteen parts per million. Each individual second during the smear is imperceptibly longer than a true second. No timestamp repeats. No duration goes negative. No timer fires early, because as far as any Google machine is concerned, nothing unusual is happening at all: it is just a day on which time runs a hair slow, the way a mechanical watch might, and by the end of the window the full extra second has been absorbed and the clocks agree with civil time again.
The smear has one rule, and Google states it flatly: never mix smeared and unsmeared time sources. During the window, a smeared clock and a stepped clock disagree by up to a second, and a machine listening to both will chase its own tail. You pick one story about what time it is, and you tell it consistently.
Note what the smear really is: an admission. Google could not fix every program that assumed time only moves forward, so it fixed time instead, bending the input until the assumption became true.
The echo, four and a half years later
The last leap second, the twenty-seventh, was inserted at the end of December 31, 2016. At 00:00 UTC on January 1, 2017, deep inside Cloudflare's DNS infrastructure, a number went negative that should, at worst, have been zero.
Cloudflare runs its own recursive resolver software, RRDNS, written in Go. To choose among its upstream resolvers, RRDNS continuously measures how fast each one answers, timing a lookup the obvious way: take the time, do the lookup, take the time again, subtract. The difference is the round trip time. Then it smooths those measurements together and feeds them into a weighted selection algorithm, so the fastest resolvers get the most traffic. Sensible code. It ran correctly for years. It contained exactly one assumption, and Cloudflare's own postmortem states it with unusual candor: the root cause was "the belief that time cannot go backwards."
Go's time.Now() reads the wall clock, and the wall clock, at a leap second, steps back. A lookup that straddled the step produced a round trip of roughly negative one second: the answer, apparently, arrived a second before the question was asked, which would be a remarkable quality of service if it were real. One negative measurement was not enough to break anything, because the smoothing averaged it against healthy values. But measurements kept landing during the step, the smoothed value kept sinking, and eventually it crossed below zero. That negative number flowed into the weighted selection code, which handed it to Go's rand.Int63n(), a function that selects a random number below a limit and that panics, immediately and by design, when the limit it is given is negative. RRDNS panicked. The panic was caught, the CNAME lookup failed, and DNS resolutions for affected properties started returning errors.
The scale was modest, and Cloudflare said so: at peak, about 0.2 percent of its DNS queries and less than 1 percent of HTTP requests ran into errors, on a subset of machines across its 102 data centers. The most affected machines were patched within ninety minutes, and the fix was rolled out worldwide by 06:45 UTC. The fix itself has become a small legend, because it amounted to changing a check that asked whether a value was equal to zero into one that asked whether it was less than or equal to zero: one character, give or take, standing between a negative number and a function that cannot abide one. Cloudflare reviewed its codebase for other places where a time difference could leak below zero, and the industry took the hint. Go itself later added a monotonic reading to its time values, so that measuring a duration with the standard library stopped depending on the wall clock's mood.
Compare the two incidents and the pattern sharpens. In 2012 the assumption lived in the kernel, in a code path that ran twice a decade. In 2017 it lived in application code, in a subtraction so ordinary that no reviewer would ever flag it. In both cases the code was correct under every condition its authors had ever observed, and the condition that broke it was not exotic. It was published in an official bulletin, six months early, by an international standards body. The leap second is the most heavily announced outage in computing. It still gets people, because the assumption it violates is not written down anywhere in the code. It is written into the programmer's picture of the world: later minus earlier is a positive number. Except at 23:59:60.
The vote to kill the second
After 2017, the industry's patience ran out in public. Large operators who run the planet's clock infrastructure, Google and Meta prominent among them, began arguing that the leap second costs more than it buys: that keeping civil time within a second of the Earth's rotation is a nineteenth-century courtesy being paid for with twenty-first-century outages. The astronomers and national time laboratories had real counterarguments. Sever the link between clock time and the Sun and you have made a philosophical change to what a day means, one that compounds quietly for centuries. But the people who get paged won.
In November 2022, at the 27th General Conference on Weights and Measures in Versailles, the member states adopted Resolution 4, on the use and future development of UTC. Its operative decision is dryly worded and enormous: the maximum permitted difference between the Earth's rotation time and UTC will be increased, in or before 2035, to a value that keeps UTC continuous for at least a century. Translated out of standards language: no more leap seconds. The correction mechanism that ran from 1972 will be switched off, the gap between atomic time and solar time will be allowed to grow past the old 0.9 second tripwire, and the discrepancy will be somebody else's problem at a scale of, eventually, minutes, centuries from now. The twenty-seventh leap second, the one that panicked Cloudflare, stands as the last one ever inserted. The count is closed at twenty-seven.
There is a rich irony in the timeline. The leap second was created to keep clocks honest to the sky. It is being retired because it made clocks dishonest to computers, once too often, in front of everyone. And the negative leap second, the fifty-nine-second minute no system has ever met, will now in all likelihood never occur. Engineers spent decades fearing a second that arrives twice. They never had to face the one that never arrives at all.
Sources, for the record: LWN's technical post-mortem of the 2012 kernel bug and John Stultz's fix; The Register's report on the Amadeus, Qantas, Reddit, and LinkedIn outages; Cloudflare's own postmortem of the 2017 RRDNS failure; Google's description of the leap smear; the IERS leap second record; and Resolution 4 of the 27th CGPM (2022).
The lesson: time is a dependency, and nobody versions it
Strip the astronomy away and what remains is a dependency problem, the most familiar problem in our trade, wearing an unusual costume.
Every system you run depends on time the way it depends on the filesystem or the network, except that time is the one dependency with no version number, no staging environment, and a maintainer who is a rotating ball of rock with a molten core and no on-call rotation. You cannot pin it. You cannot roll it back. You cannot even observe it directly; you can only read clocks, which are rumors about time, maintained by software, subject to correction by other software. And twice a decade or so, the maintainers of the standard itself reach into your running production systems, everywhere on Earth at the same instant, and change the contract: this minute has sixty-one seconds, this timestamp happens twice, this duration may be negative. Every program that had quietly promoted "time moves forward" from an observation into an axiom found out which one it had been relying on.
The defenses are known, and they are embarrassingly concrete. Measure durations with a monotonic clock, never by subtracting wall clock readings; the wall clock is for display, the monotonic clock is for truth about intervals. Treat any duration that comes out negative as a data point about your clock, not as a number, and clamp it before it reaches code that cannot handle it, because somewhere downstream there is always a rand.Int63n() waiting. Test the paths that run once a year, or once a decade, as seriously as the hot paths, because rarity is not safety; rarity is just a bug with a long fuse and an alibi. And when a rare event is announced six months in advance by an international body, believe the announcement. The leap second is the only outage in history that arrives with a save-the-date card.
Our own studio runs on measured time more than most. We count our output in finished minutes per hour, which means our core metric is a duration divided by a duration, a number built entirely out of clock readings. If the hour can secretly contain 3,601 seconds, or a worker's start and finish stamps can straddle a step and produce a negative shift, the metric does not merely become imprecise. It becomes fiction with good formatting. So the rule here is the rule Cloudflare paid tuition to learn: any clock can lie about what time it is, but a duration you measured honestly cannot go below zero, and when one does, distrust the clock. Do not feed the lie into the machinery and let the machinery panic.
That is the real inheritance of June 30, 2012, and January 1, 2017. Not fear of a second. A discipline about assumptions. The systems that broke did not break because their engineers were careless. They broke because one true thing about the world, that the Earth does not rotate at a constant rate, had been abstracted away so completely that an entire civilization of software grew up believing the abstraction was the reality. The map said seconds only move forward. The territory, once every few years, inserted a second that arrived twice, and the territory always wins.
The most expensive second in history cost nothing to buy, was announced six months in advance, and broke everything it touched anyway.