On July 4, 1997, Mars Pathfinder fell out of the Martian sky wrapped in airbags, bounced across the surface of another planet like the world's most expensive beach ball, and came to rest upright and alive. Within days it had deployed Sojourner, the first wheeled vehicle ever to drive on another planet, and started sending home pictures that made the evening news everywhere on Earth. It was the triumph of the faster, better, cheaper era: a full Mars landing mission for a fraction of what the Viking landers had cost, executed by a team that had bet everything on doing more with less.
Then the spacecraft started rebooting itself.
Not crashing in flames, not dying. Rebooting. The computer would reset, reinitialize all of its hardware and software, recover whatever science data had already been collected, and then sit there, healthy and useless, for the rest of the day. The next day's activities would proceed, until, some days later, it happened again. A reset cost the mission a day of work on a spacecraft with a limited lifetime, and nobody on Earth could see why. There was no error message from Mars that said what was wrong. There was only a machine, on another planet, that kept deciding to start over.
It is worth being precise about whose computer this was, because the rover gets the headline. The resets lived in the lander, the stationary spacecraft that carried Sojourner to Mars and ran the whole surface operation: the radio, the camera, the weather station, the bus that moved data between instruments. When the lander's computer reset, everything stopped, rover included. The most famous software bug in spaceflight history was not in the little rover. It was in the mothership, in the plumbing, in a piece of operating system machinery so mundane that nobody had thought about it since long before launch.
The first suspect, reasonably enough, was the watchdog. Every spacecraft carries some mechanism whose job is to notice that the computer has hung and restart it, on the theory that a rebooted computer is better than a dead one. Pathfinder's version of this was woven into the software itself. Two tasks ran the information bus on a strict eight times a second rhythm. The bc_sched task set up each cycle's transactions and, critically, checked that the bc_dist task had finished distributing the previous cycle's data. Bc_dist had a hard deadline. If bc_sched woke and found bc_dist still unfinished, the spacecraft declared an error and reset itself, by design. So the resets were not malfunctions of the safety system. The safety system was working perfectly. It kept reporting, truthfully, that a critical job had failed to finish on time. The question was never whether the deadline had been missed. The question was how the third most important job on the spacecraft could be late at all, on a computer that had idle time to spare.
Picture a hospital with one door between the wards, and a rule that the person holding the door gets to finish walking through it before anyone else uses it. The weather task was the slowest errand in the building: a low priority chore that ambled up to the door a few times a day carrying a thermometer reading, opened it, and started through. The ambulance crew, bc_dist, came up behind it at a run, carrying the data that had to move on schedule or the whole hospital day was void. Correctly, the ambulance waited for the person in the doorway. That wait is normal and it is supposed to be brief. But the doorway stood in a busy corridor, and the corridor was full of shoppers: imaging work, image compression, communications, a crowd of medium priority tasks, all of them more important than the errand and less important than the ambulance. Every time the errand tried to take another step through the door, a shopper shouldered past it and it froze. The shoppers were not blocking the ambulance on purpose. They could not even see the ambulance. They were simply outranking the one task the ambulance was waiting for. The errand could not finish, so the lock was never released, so the ambulance never moved, and eventually the hospital administrator, bc_sched, looked at the clock, concluded the ambulance crew must be dead, and rebooted the entire hospital.
That is priority inversion, and it is a queue where the slowest errand holds the door against the ambulance while shoppers walk past. Nothing in the system was broken. Every task was doing exactly what its priority told it to do. The priorities themselves, composed together around one shared lock, produced an outcome none of them intended: the highest priority work in the system was effectively running at the priority of the lowest, plus shoppers.
Meanwhile, on Earth, the team did something that deserves to be as famous as the bug. They did not guess. Pathfinder's flight software carried a trace and log facility, built by one of the team's engineers and instrumented into the pipe, message queue, interrupt, and scheduling services by a Wind River engineer, which recorded what the software had been doing into ring buffers, continuously, from startup. It had been built to hunt a bug in an early port of the operating system, and it had flown to Mars switched on. Mike Jones's widely read account called the debugging tools fortuitously left in. Glenn Reeves, who led the software team, corrected the record with some force: the features were not fortuitously left enabled, they remained in the software by design, because the team believed in test what you fly and fly what you test. In the lab in Pasadena sat a replica of the spacecraft computer. The team ran the same set of activities that had preceded the failures on Mars, over and over, with the trace facility armed to freeze and dump its record the moment the deadline error fired. In less than 18 hours, they made the failure happen on Earth. The moment they could reproduce it, Reeves writes, the priority inversion was obvious.
The trace they captured reads like a security camera replay of a crime nobody committed. The ASI/MET task had called select(), which had called pipeIoctl(), which had called a routine named selNodeAdd(), and the weather task was preempted at the worst possible instant: in the middle of giving the mutex semaphore back, the release itself unfinished. Medium priority tasks ran. Then bc_dist woke up, tried to send the newest weather data through the same pipe mechanism, and blocked on the very semaphore whose release had been interrupted. More medium priority tasks ran, each one individually entitled to run ahead of the weather task, none of them aware they were standing in the doorway. When bc_sched woke to set up the next bus cycle and found bc_dist still incomplete, it declared the error and the replica reset, the same way the spacecraft had. Notice what the trace rules out: no task misbehaved, no code was corrupt, no hardware glitched. The recording showed a system executing its rules faithfully into a wall. That is what made the trace priceless. Without it, the team would have been arguing from telemetry and suspicion. With it, they were reading the incident like a transcript.
Here is the part that should be printed on a card and taped above every engineer's desk: the cure already existed, inside the operating system, on the spacecraft, the whole time. VxWorks mutexes could be created with priority inheritance switched on. With inheritance, the moment the ambulance blocks on the door, the errand in the doorway temporarily inherits the ambulance's priority. The shoppers can no longer shoulder past it, because for as long as it holds that lock, it outranks them. It hurries through, releases the door, drops back to being an errand, and the ambulance rolls. The fix for the most famous real time bug in history was not new code. It was one creation flag on a lock.
And the flag was off. It was off for an embarrassingly human chain of reasons. Wind River, the vendor, had deliberately left priority inheritance out of the default configuration for the select() service, for optimum performance. The JPL team had never consciously reviewed that default for this particular lock, even though elsewhere in the same flight software, where they had thought carefully about critical data structures, their semaphores did have inversion protection switched on. Reeves's own verdict is the honest one: "There was no conscious decision to not have the priority inversion enabled. We just missed it." Nobody chose the bug. A vendor chose a default, the default optimized for the wrong thing, and a busy team, racing to land on Mars, never got down the list to that line item. His summary of the whole affair deserves quoting exactly: "A good lesson when you fly COTS stuff - make sure you know how it works."
Changing the flag on Mars took more care than changing it on Earth would have, which is itself part of the lesson. The configuration lived in a global variable that governed how select() semaphores were created, so flipping it changed the behavior of every such semaphore created afterward, not just the one in the doorway. The team had to ask what that meant for the rest of the system, and whether performance would suffer, and whether select() itself behaved differently with inheritance on. Wind River analyzed it: the performance impact would be minimal, and behavior would not change so long as only one task ever waited on any given file descriptor, which was true in their system. They tested the system extensively on the ground. Then they sent the change to the spacecraft through the mission's specialized patch process, a careful mechanism that transmitted the differences between the software on board and the software they wanted, and modified the onboard copy after a great deal of validation. The popular version of the story, from Jones's account, has the engineers typing a short C program into a live interpreter aboard the spacecraft and changing the variable in memory. Reeves's account gives the patch process the credit, though he notes the command shell and the output stream stayed aboard and stayed useful for the entire mission. Either way, the essential fact stands: human beings diagnosed a timing bug in a computer on another planet, proved the diagnosis on a twin in a lab, and repaired the original in place, without ever touching it. The resets stopped.
Why had nobody caught it before launch? Partly because the bug is shy. It only appears when the weather task is collecting data at the same moment the medium priority load is heavy, a coincidence of timing that preflight testing, which concentrated on best case data rates, rarely produced. The team had in fact seen resets during testing before landing, and had tried to track them down, and could not make the problem repeat, which is the single most corrosive sentence in engineering, because an unreproducible failure begs to be deprioritized. And partly, Reeves argues, because their priorities really were right. Entry and landing got the attention, as they should have. And the system had been deliberately built to survive exactly this class of failure: resets were recoverable, collected data was safe, the spacecraft could even have run a sixty day mission on its own, rover deployment included, if the radio had died on landing day. The robustness that made the bug survivable is also what made it permissible to ship with it unfixed. Success then made it worse: the surface data rates and the volume of science activity on Mars were higher than anyone had planned for, so the shy bug found its crowd of shoppers far more often than the test program ever had. They had not tested, in Reeves's dry phrase, the better than we could have ever imagined case.
Our own studio runs a small fleet of software workers, and priority inversion shows up there wearing work clothes. A trivial chore picks up a shared resource, a file, a claim, a machine, and dawdles. A stream of medium urgency noise, all of it legitimately more important than the chore, keeps the chore from ever finishing. The one job with a real deadline waits behind the whole pile, and the supervisor's version of a reset is to declare the day lost and start over. The fix is the same fix, translated: whoever holds the scarce thing temporarily inherits the urgency of whoever is waiting on it, and everything else gets out of the way. The second lesson translates without any translation at all. When our workers fail somewhere we cannot see, the only thing that saves us is the trace we decided, in advance and on purpose, to leave running: the logs, the checkpoints, the receipts. Pathfinder's engineers did not get lucky. They had spent effort, years earlier, on instrumentation for a machine they would never be able to sit in front of, because they assumed production would someday surprise them. Production was on another planet, and it did.
So the two rules we take from Mars are these. First: enable the safety you already bought. Audit your vendors' defaults as if they were your own decisions, because in production they are. A protection that exists but is switched off is not a protection; it is a story you tell yourself about one. Second: leave yourself a way to debug in production, and leave it in by design, not by accident. Ship the trace. Keep the replica honest. Assume that the failure will be shy and the deadline hard and the distance, whatever your personal Mars is, too far to improvise across.
The cure for the most famous bug in spaceflight had flown to Mars with the spacecraft, switched off, inside the software, the whole time. Somebody just had to see the problem clearly enough, from a whole planet away, to turn it on.
Sources, for the record: Glenn E. Reeves, "What really happened on Mars?" (December 15, 1997), the authoritative account by the Mars Pathfinder flight software team lead, is the source for the reset mechanism, the less than 18 hour reproduction on the Earth based replica, the trace facility's design history, the global variable fix, the Wind River analysis, and the preflight history. Mike Jones, "What really happened on Mars?" (December 7, 1997), summarizing Wind River CTO David Wilner's keynote at the IEEE Real-Time Systems Symposium, is the source for the mutex creation parameter account and the C interpreter version of the fix; where the two differ, this essay follows Reeves. The Rapita Systems retrospective on the Pathfinder software confirms the scheduling and watchdog account.