One Stale Index Hint Broke GitHub Pull Requests
Today's 25-minute outage was small, but its root cause says a lot about GitHub under AI-era load.
GitHub broke again today, and the outage itself is almost not the story. At 16:16 UTC, GitHub's status page flagged degraded performance for Issues and Pull Requests; within minutes users were hitting straight 500s on PR and issue pages, with Search wobbling too. By 16:41 it was resolved — 25 minutes end to end, fast even by good-year standards.
What makes it worth your attention is the root cause GitHub posted at 16:38: "A database index hint was referencing an index that had been removed by a recent migration, causing query failures for some users." That single sentence is a compact lesson in how large MySQL fleets fail, and a telling snapshot of where GitHub is right now — mid-migration, under unprecedented AI-driven load, shipping schema changes at speed.
Why a dropped index takes you down hard
Index hints — FORCE INDEX, USE INDEX and friends in MySQL — exist because at GitHub's scale you sometimes can't trust the optimizer. A query plan that's fine at a million rows can tip into a full table scan at a billion, so engineers pin the query to a known-good index and move on. It works, right up until it doesn't.
The failure mode is the nasty part. If the optimizer merely prefers an index that disappears, it re-plans and your query gets slower. But FORCE INDEX naming a nonexistent index isn't a degraded plan — MySQL rejects the query outright ("Key doesn't exist"). No fallback, no graceful degradation. Every request hitting that code path gets a hard error, which is exactly why users saw 500s on two of GitHub's most-trafficked page types rather than sluggish load times.
The irony is that GitHub practically wrote the book on safe online schema changes — gh-ost, their triggerless migration tool, is the industry standard for exactly this kind of operation. And the migration itself almost certainly ran cleanly. The bug lived in the coupling: an application-layer hint hardcoding a schema-layer assumption, with nothing in the deploy pipeline checking that the two still agreed. If your own codebase has FORCE INDEX strings in it, this is your reminder that every one of them is a latent runtime dependency on your schema. A CI check that greps hinted index names against the live schema — or against your migration files — is an afternoon of work and would have caught this class of failure entirely.
The context: a platform running hot
A 25-minute blip wouldn't merit a writeup in 2023. In 2026 it lands differently, because it's another tick in a pattern. The Register counted 26 GitHub incidents in April and 23 in May; unofficial trackers put observed availability in the high-80s percent over the trailing 90 days, while GitHub's official status metrics still read ~99.9% — a gap that says as much about what the status page measures as about uptime. Today's Hacker News thread was thin on sympathy and thick on Forgejo migration anecdotes, which tells you where developer sentiment has drifted.
Two forces are colliding. First, GitHub is executing an accelerated migration of its Rails monolith and Git infrastructure onto Azure — roughly 40% of monolith traffic and 30% of Git traffic as of June, up from single digits in February. That kind of replatforming means schema churn, dual-write plumbing, and a lot of migrations landing quickly. Today's dropped index didn't happen in a vacuum; it happened in an environment where the database layer is being reshaped aggressively.
Second, the load curve went vertical. GitHub's own leadership confirmed the platform is processing around 275 million commits per week — a pace of 14 billion for the year, versus roughly 1 billion for all of 2025. Pull requests opened by AI coding agents reportedly jumped from about 4 million in September 2025 to over 17 million by March. Agents don't behave like humans: they hammer PR and issue endpoints in tight loops, retry aggressively, and never sleep. The two services that failed today are precisely the ones agentic workflows lean on hardest. That's likely coincidence in this specific incident — a stale hint fails regardless of traffic — but it explains why hints and hand-tuned query plans proliferate in the first place, and why the blast radius of any PR/Issues failure keeps growing.
What to actually do about it
My read: this isn't a platform in collapse, but it is a platform whose error budget you can no longer ignore. GitHub's SVP of engineering says the team is "making structural changes that permanently remove failure modes," and the trajectory — Azure capacity, reported burst offload of Actions runners to AWS — suggests the capacity story improves over the next year. But migration-era turbulence is measured in quarters, not weeks. Plan for double-digit incidents a month through at least the rest of 2026.
Practically, that means treating GitHub like any other third-party dependency with a real failure rate:
- Your code is fine; your workflow isn't. Git is distributed — clones and pushes to a second remote (
git remote set-url --add --push) give you continuity for free. PRs, Issues, and Actions are centralized SaaS with no local fallback. - Don't let Actions be the only path to production. If your deploy gate is a GitHub-hosted workflow, a PR-page outage can freeze releases. Keep a documented break-glass deploy that runs from a laptop or an internal runner.
- Export what you can't re-derive. Issues and PR discussions are the least portable data you keep on GitHub. A nightly API export to object storage is cheap insurance, whether or not you ever touch Forgejo.
And take the database lesson home: audit your index hints, tie them to your migrations, and prefer fixing the optimizer's inputs — statistics, histograms, query shape — over pinning plans forever. GitHub just demonstrated, on the world's most-watched status page, what the alternative costs.
Sources & further reading
- Incident with Pull Requests and Issues — githubstatus.com
- Wednesday, August 12: GitHub, Incident with Pull Requests and Issues — news.ycombinator.com
- GitHub outages persist as AI coding drives traffic surge — theregister.com
- GitHub's AI Agent Problem: 17 Million PRs, Five Outages, and a Kill Switch — danilchenko.dev
Emeka has spent over a decade tracking threat actors, vulnerability disclosures, and the evolving landscape of application security, bringing a sharp continent-spanning perspective to his reporting. He's known for translating dense CVE advisories into clear, actionable context that developers and security teams alike actually read.
Discussion 5
stale index hints are why i avoid managed databases that hide schema changes. you're always one deploy away from this.
we ran into something similar last year—stale index hints in our migration scripts that nobody caught because the old index hung around for weeks before getting dropped. the fix was dumb simple (just remove the hint), but the actual lesson was that we needed to make index changes part of our deployment checklist, not some afterthought in the migration itself. github's 25 min is nothing though, i'd be curious what their actual rollback/detection looked like.
yeah exactly—the real problem is the gap between when you *think* you've cleaned something up and when it actually matters. we had a similar mess where a force index hint stayed in production code for like three deploys after the index existed nowhere, just quietly failing on certain query patterns nobody tested. the checklist idea is solid but honestly we should've caught it in code review; stale hints should fail loud in staging if you're dropping the underlying index, not silently break in prod.
the brutal part is that code review catches stale hints *if* someone's actually reading the migration carefully against the queries that reference it. at scale, that's just not happening. we'd have caught this in postgres because you can't hint an index that doesn't exist—the query just fails at parse time instead of silently degrading under load. mysql's graceful fallback here is what bit them.
postgres's fail-fast behavior here is definitely cleaner, but i'd push back slightly—mysql's graceful degradation is the real trap. the hint becomes a performance suggestion rather than a hard requirement, so it silently uses a worse plan and nobody notices until you're under peak load and the margin evaporates. by then you're in incident mode trying to figure out which migration broke what. the fix is tedious: either strict validation of hints at deploy time, or just... don't use them across schema changes. easier said than done when you've got years of accumulated query tuning.
