Core dump epidemiology: fixing an 18-year-old bug

OpenAI engineers described how they investigated rare crashes in Rockset, a data service used in parts of ChatGPT infrastructure. The failures looked contradictory: some core dumps showed a return to an invalid address, while others suggested a stack-pointer shift that ordinary application code should not produce.

One symptom, two root causes

Rather than treating each crash as an isolated puzzle, the team compared a larger population of crash dumps to find patterns. The investigation ultimately uncovered two unrelated defects: silent hardware corruption on an Azure host and an 18-year-old race condition in GNU libunwind, a widely used open-source library. The coincidence made the incidents look like one impossible software bug.

The account is a useful example of “crash epidemiology”: collect enough comparable failures to distinguish a rare environmental fault from a software defect. It also shows why a plausible stack trace is not always a complete diagnosis, especially in low-level systems where hardware, libraries, and application code interact.

Practical takeaway

For infrastructure teams, the lesson is to preserve diagnostic data across incidents and compare failures systematically. A fix should be supported by reproducible evidence and checked against the broader incident population. This is an engineering case study from OpenAI, not evidence that the same failure pattern is common across cloud services.

Source: OpenAI’s engineering write-up.