{
"vendor": "Python Package Index",
"slug": "python-package-index",
"platform": "statuspage",
"status_url": "https://status.python.org",
"last_checked": "2026-09-16T12:28:20Z",
"last_state": "ok",
"history_backfilled": true,
"first_watched": "2026-09-04T07:06:16Z",
"incidents": [
{
"body": "The incident has been resolved.\n\n## Incident Report: Cluster-wide service outage\n\n**Duration**: ~36 minutes \\(20:04\u201320:40 UTC\\)\n\n**Impact**: Some PSF-hosted services were unavailable, including [python.org](http://python.org), [us.pycon.org](http://us.pycon.org), PyPI stats, [bugs.python.org](http://bugs.python.org), and related services. \n\nWhat was unaffected was our other cluster that manages [PyPI.org](http://PyPI.org) among other services related to PyPI.\n\n**Root Cause**: During local development of kubernetes workloads locally there was an incorrect context switch to one of our production clusters.\n\nThe scale-down commands ran against the production cluster instead of the local environment, iterating through all deployments and setting them to zero replicas. which created cascading failures.\n\n**Recovery**: Services were restored with the help of Ee Durbin by bringing up infrastructure in dependency order, original replica counts were recovered from Kubernetes event history.\n\nAction items:\n\n* Separate kubeconfig files for production vs local, rather than relying on context switching\n* Research adding admission control or policies to prevent bulk scale-to-zero operations\n* Document the infrastructure dependency chain and recovery runbook for future incidents\n\nJacob Coffee, PSF Infrastructure  Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-26T23:26:50.000Z",
"resolved_inferred": false,
"started_at": "2026-02-26T20:04:21.943Z",
"state": "postmortem",
"title": "Python Infrastructure Outage",
"updated_at": "2026-02-26T23:27:22.819Z",
"url": "https://stspg.io/7slh65j2y14b"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-19T23:59:14.881Z",
"resolved_inferred": false,
"started_at": "2026-02-19T22:46:46.414Z",
"state": "resolved",
"title": "API Traffic experiencing intermittent failures",
"updated_at": "2026-02-19T23:59:15.180Z",
"url": "https://stspg.io/zhcdc6gwlhrs"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-01-07T23:00:57.952Z",
"resolved_inferred": false,
"started_at": "2026-01-07T22:42:33.653Z",
"state": "resolved",
"title": "PyPI Uploads failing",
"updated_at": "2026-01-07T23:00:59.678Z",
"url": "https://stspg.io/x0hv9ys48d17"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-12T16:26:16.252Z",
"resolved_inferred": false,
"started_at": "2025-11-12T15:46:54.533Z",
"state": "resolved",
"title": "Slow updates to PyPI pages/API responses",
"updated_at": "2025-11-12T16:26:16.864Z",
"url": "https://stspg.io/rjh2ytldcr0k"
}
]
}