We said in our About page that reliability is a feature, not a footnote. That only means something if we're honest when it slips. This is a walkthrough of an incident from earlier this year, what caused it, and what changed afterward.
What happened
A scheduled batch job that keeps the reporting layer in sync with live schedule data failed silently after a database migration changed a column type it depended on. Reporting kept serving data — just increasingly stale data — for roughly four hours before anyone noticed a dashboard number looked wrong.
Why it took four hours to catch
The honest answer: we didn't have an alert on sync freshness, only on whether the job had crashed. A job that runs "successfully" but silently processes nothing doesn't trip a failure alert — it just quietly stops being useful.
"A job that fails loudly is easy to catch. A job that succeeds at doing nothing is the dangerous one."
What we changed
- Added a freshness check that alerts if reporting data hasn't updated within its expected window, independent of whether the job "succeeded."
- Schema changes to shared tables now require a compatibility check against every consumer of that table before they can ship.
- Status updates during an active incident now post automatically to our status page rather than waiting on someone to remember.
Nobody enjoys writing these up, but a system this many teams rely on for daily operations only earns trust by being honest about the times it didn't hold up.