{
"vendor": "1Password",
"slug": "1password",
"platform": "statuspage",
"status_url": "https://status.1password.com",
"last_checked": "2026-09-16T12:28:20Z",
"last_state": "ok",
"history_backfilled": true,
"first_watched": "2026-09-04T07:06:16Z",
"incidents": [
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-10T12:15:40Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-09-09T17:11:08.191-04:00",
"resolved_inferred": false,
"started_at": "2026-09-09T16:43:50.413-04:00",
"state": "resolved",
"title": "Device Trust Service Disruption",
"updated_at": "2026-09-09T17:11:08.205-04:00",
"url": "https://stspg.io/9sjwgt4v11kr"
},
{
"body": "A fix has been deployed and this issue is now resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-22T15:20:54.949-04:00",
"resolved_inferred": false,
"started_at": "2026-07-22T13:55:18.026-04:00",
"state": "resolved",
"title": "Account governance functions are unavailable in SaaS Manager",
"updated_at": "2026-07-22T15:20:54.965-04:00",
"url": "https://stspg.io/9bt7rtjljklp"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-17T14:57:38.340-04:00",
"resolved_inferred": false,
"started_at": "2026-07-17T14:52:15.282-04:00",
"state": "resolved",
"title": "SaaS Manager Workflows are not running as expected",
"updated_at": "2026-07-17T14:57:38.358-04:00",
"url": "https://stspg.io/662s09h6z852"
},
{
"body": "A fix has been deployed and this issue is now resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-13T15:57:14.635-04:00",
"resolved_inferred": false,
"started_at": "2026-07-13T14:59:33.940-04:00",
"state": "resolved",
"title": "Degraded user-related operations for individual and family accounts",
"updated_at": "2026-07-13T15:57:14.657-04:00",
"url": "https://stspg.io/nvvxqlclh4gh"
},
{
"body": "A fix for this issue has been implemented, and our teams are monitoring. During the outage, GitHub Actions might have failed to pull the latest CLI version, causing the pipeline to fail.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-13T10:30:00.000-04:00",
"resolved_inferred": false,
"started_at": "2026-07-13T10:30:00.000-04:00",
"state": "resolved",
"title": "Password Manager Update Server Outage",
"updated_at": "2026-07-13T11:47:57.101-04:00",
"url": "https://stspg.io/shy2jfrdnzfq"
},
{
"body": "# Incident Postmortem - Device Trust Authentication Outage\n\n**Date of Incident:** 2026-07-02  \n**Time of Incident \\(UTC\\):** 19:28 - 20:28  \n**Service\\(s\\) Affected:** This incident affected: USA/Global \\(Device Trust Authentication Service, Device Trust Checks Server, Device Trust Portal, Device Trust API\\) and Europe \\(Device Trust Authentication Service, Device Trust Checks Server, Device Trust Portal, Device Trust API\\).  \n**Impact Duration:** 1 Hour\n\n## Summary\n\nOn July 2, 2026, 1Password Device Trust experienced an outage lasting approximately one hour, beginning at 19:28 UTC. During this window, all Device Trust services were unavailable and users were unable to authenticate. We have identified the root cause: an internal configuration change to our cloud load balancing infrastructure was applied more broadly than intended which temporarily made a required endpoint unreachable. This was not a security incident and did not involve any third party. The configuration has been corrected, service has been fully restored, and we have implemented additional safeguards to help prevent this type of issue from recurring.\n\n## Impact on Customers\n\n* **All Device Trust services were completely unavailable during the 60-minute outage window. Users were unable to authenticate to any Device Trust service, including the authentication portal \\(**[**auth.kolide.com**](http://auth.kolide.com)**\\), admin console \\(**[**app.kolide.com**](http://app.kolide.com)**\\), API endpoints, and device server.**\n* **Number of Affected Customers \\(approximate\\):** 100% of Device Trust customers\n* **Geographic Regions Affected \\(if applicable\\):** US and EU\n\n## What Happened?\n\nOn July 2, 2026, A configuration change intended to improve the security configuration of an internal service was applied more broadly than intended, affecting a shared network routing component relied on by Device Trust. The change replaced the underlying network infrastructure supporting Device Trust. However, the traffic routing configuration continued to reference the previous infrastructure instead of the newly provisioned resources. As a result, Device Trust authentication endpoints became unreachable. Reverting the change restored connectivity and service.\n\n* **Timeline of Events \\(UTC\\):**\n\n    * 19:28: Network configuration change applied; Network Load Balancers responsible for routing traffic for Device Trust services became unreachable\n    * 19:30: Automated monitoring detects the outage and triggers alerts\n    * 19:33: Incident is declared at [status.1password.com](http://status.1password.com)\n    * 20:03: Root cause is identified as a load balancer configuration issue\n    * 20:24: Load balancer is recreated with correct configuration\n    * 20:26 Recovery is confirmed for [app.kolide.com](http://app.kolide.com)\n    * 20:28 Recovery is confirmed for [auth.kolide.com](http://auth.kolide.com)\n    \n* **Contributing Factors \\(if any\\):**\n\n    * **Shared component scope not visible during review or validation.** The network infrastructure class affected by the change was shared across multiple services, but this dependency was not documented or surfaced in the change review process. As a result, the pre-production validation, while performed in a lower environment before production rollout, did not cover all dependent services, and the impact on Device Trust was not identified before the change reached production.\n    * **Concurrent diagnostic signals slowed root cause identification.** At the time the outage began, unrelated certificate renewal errors surfaced alongside the network connectivity failure. Investigating both signals simultaneously delayed identification of the network configuration change as the root cause.\n    \n\n## How Was It Resolved?\n\n* **Resolution Steps:** Once the source of the outage was identified, engineers reverted the configuration change. The affected network infrastructure was recreated with its original, publicly accessible configuration, restoring routing to all Device Trust authentication endpoints. Service recovery was confirmed sequentially across affected services before the incident was closed.\n* **Verification of Resolution:** Following the revert, engineers verified that all Device Trust authentication endpoints were reachable and accepting traffic. Customer confirmation of resolution was subsequently received through the support team.\n\n## What We Are Doing to Prevent Future Incidents\n\n* **Require explicit approval for destructive infrastructure changes.** We are implementing safeguards that require manual review and approval before changes that would destroy and recreate critical network infrastructure can be applied. This creates a forcing function to catch unintended scope before changes reach production.\n* **Improve scope visibility for shared infrastructure components.** We are documenting the dependencies between shared infrastructure components and the services that rely on them, and integrating this information into our change review process. Future changes to shared components will require an explicit assessment of downstream impact before deployment.\n* **Strengthen pre-production validation across shared dependencies.** We are updating our change promotion process to require validation against all services that share affected components \u2014 not only the intended target \u2014 before changes are promoted to production.\n* **Reduce single points of failure in traffic routing.** We are evaluating architectural changes to introduce redundancy into Device Trust's traffic routing configuration, so that a failure in any single component results in degraded service rather than a complete outage.\n\n## Next Steps and Communication\n\n* No action is required from our customers at this time.\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-02T16:30:48.837-04:00",
"resolved_inferred": false,
"started_at": "2026-07-02T15:44:00.651-04:00",
"state": "postmortem",
"title": "Device Trust Outage",
"updated_at": "2026-07-10T21:11:02.513-04:00",
"url": "https://stspg.io/3vbh7xm7z6zv"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-30T10:09:25.287-04:00",
"resolved_inferred": false,
"started_at": "2026-06-29T18:08:19.241-04:00",
"state": "resolved",
"title": "Provisioned users with SSO enforcement encounter a 403 error when accepting their invitation",
"updated_at": "2026-06-30T10:09:25.302-04:00",
"url": "https://stspg.io/fr0bj8tl8vb4"
},
{
"body": "The issue is now resolved, and access to 1Password services should be fully restored.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-24T15:01:46.148-04:00",
"resolved_inferred": false,
"started_at": "2026-06-24T13:15:10.565-04:00",
"state": "resolved",
"title": "Restoring Access with a Recovery Code Is Failing for Some Users",
"updated_at": "2026-06-24T15:01:46.167-04:00",
"url": "https://stspg.io/4l9c631tvtk5"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-18T14:50:42.250-04:00",
"resolved_inferred": false,
"started_at": "2026-06-18T13:54:11.964-04:00",
"state": "resolved",
"title": "Sync Issues Affecting Legacy 1Password 7 Clients",
"updated_at": "2026-06-18T14:50:42.263-04:00",
"url": "https://stspg.io/0tw56x8hwf47"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-17T12:00:17.093-04:00",
"resolved_inferred": false,
"started_at": "2026-06-16T12:24:03.697-04:00",
"state": "resolved",
"title": "Sign in events are not appearing and admins are unable to migrate to hosted provisioning",
"updated_at": "2026-06-17T12:00:17.108-04:00",
"url": "https://stspg.io/xm7kvknkzx3p"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-12T19:18:57.981-04:00",
"resolved_inferred": false,
"started_at": "2026-06-11T17:49:05.374-04:00",
"state": "resolved",
"title": "Device Trust Authentication Service Experiencing Timeout Errors",
"updated_at": "2026-06-12T19:18:58.001-04:00",
"url": "https://stspg.io/3h5wz8m1ryh4"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-05-16T16:36:03.709-04:00",
"resolved_inferred": false,
"started_at": "2026-05-16T11:51:33.535-04:00",
"state": "resolved",
"title": "SaaS Manager performance is degraded",
"updated_at": "2026-05-16T16:36:03.725-04:00",
"url": "https://stspg.io/k22qf9c71xw0"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-05-04T14:34:49.601-04:00",
"resolved_inferred": false,
"started_at": "2026-05-04T13:59:28.385-04:00",
"state": "resolved",
"title": "SaaS Manager workflows and integrations are failing",
"updated_at": "2026-05-04T14:34:49.617-04:00",
"url": "https://stspg.io/nknq7xrbw3tl"
},
{
"body": "# Incident Postmortem - Login and Sync Issues\n\n**Date of Incident:** 2026-04-27  \n**Time of Incident \\(UTC\\):** 14:15 - 15:41  \n**Service\\(s\\) Affected:** All web APIs  \n**Impact Duration:** Approximately 86 minutes\n\n## Summary\n\nOn April 27, 2026, [1Password.com](http://1password.com/) experienced a period of elevated error and full service unavailability in our USA/Global region. A defect deployed in application scaling logic caused a cascading failure during peak usage. This resulted in a period of service unavailability for customers in the USA/Global region.\n\nThis was not a security incident, and there was no loss of data.\n\n## Impact on Customers\n\n* **Sign-in and account access:** Customers in the USA/Global region experienced errors or were unable to sign in or access their 1Password accounts between approximately 14:15 and 15:41 UTC on April 27, 2026.\n* **All 1Password clients:** The disruption applied to all client types, including the web app, CLI, browser extensions, and desktop and mobile applications, for customers connecting through the USA/Global region.\n* **Geographic regions affected:** Only customers on the USA/Global region were affected. Other regions were not impacted.\n\n## What Happened?\n\nWe deployed a change to how our services signal their readiness to serve requests, which contained a latent defect.\n\n* **Timeline of events:**\n\n    * **2026-04-27 14:15 UTC** - Load increase triggers issues.\n    * **2026-04-27 14:29 UTC** - Incident declared; response team mobilized.\n    * **2026-04-27 15:41 UTC** - Issue identified and fix deployed.\n    * **2026-04-28 18:06 UTC** - Incident formally resolved following monitoring.\n    \n\n## How Was It Resolved?\n\nOur on-call engineering team identified the scaling logic as the cause of the cascading failure and disabled it via a configuration change. Once disabled, capacity was restored, and traffic recovered. A subsequent configuration change was made to prevent this logic from being re-enabled in production environments.\n\n## What We Are Doing to Prevent Future Incidents\n\n* **Immediate fix:** We disabled the defective health check logic and locked the configuration to prevent it from being re-enabled in production.\n* **Retrospective:** We are conducting a full retrospective with the incident responders to identify further improvements to our systems and our response processes.\n\n## Next Steps and Communication\n\n* No action is required from our customers at this time.\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your patience and understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-28T14:00:34.801-04:00",
"resolved_inferred": false,
"started_at": "2026-04-27T10:54:26.804-04:00",
"state": "postmortem",
"title": "Login and Sync Issues",
"updated_at": "2026-05-12T13:18:52.353-04:00",
"url": "https://stspg.io/rl7gr1h78lyk"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-22T12:06:19.982-04:00",
"resolved_inferred": false,
"started_at": "2026-04-22T11:18:52.323-04:00",
"state": "resolved",
"title": "Sign-in issues affecting 1Password SaaS Manager",
"updated_at": "2026-04-22T12:06:19.999-04:00",
"url": "https://stspg.io/mwg0pldwp18t"
},
{
"body": "# 2026-04-14 SEV1 Incident Postmortem - Hosted Service in the US experiencing high latency and errors\n\n# Incident Postmortem - Elevated Application Error\n\n**Date of Incident:** 2026-04-14  \n**Time of Incident \\(UTC\\):** 17:25 - 18:00  \n**Service\\(s\\) Affected:** Sign in, Admin console  \n**Impact Duration:** Approximately 35 minutes\n\n## Summary\n\nOn April 14, 2026, [1Password.com](http://1Password.com) experienced elevated error rates due to an increase in traffic to our USA/Global infrastructure. Requests arrived more quickly than they could be serviced in our USA/global region. Customers experienced errors and high latency.\n\nThis was not a security incident, and there was no loss of data.\n\n## Impact on Customers\n\n* **Sign-in and account access:**\u00a0Some customers in the USA/Global region experienced errors when attempting to sign in or access their 1Password accounts between 17:25 and 18:00 UTC on April 14, 2026.\n* **All 1Password clients:**\u00a0The disruption applied to all client types, including the web app, browser extensions, and desktop and mobile applications, for customers connecting through the USA/Global region.\n* **Number of affected requests:**\u00a0Approximately 2.95% of requests over the incident window to [1Password.com](http://1Password.com) \\(USA/Global\\) during the incident window resulted in errors or timeouts, though for some brief spikes up to 45% of requests failed.\n* **Geographic regions affected:**\u00a0Customers on our USA/Global only, other regions were not affected.\n\n## What Happened?\n\nAn unusually high volume of traffic exceeded the capacity of our controls, and triggered latency and errors in the underlying applications.\n\n* **Timeline of events:**\n\n    * **2026-04-14 17:25**\u00a0- Start of incident.\n    * **2026-04-14 17:36**\u00a0- Issue identified.\n    * **2026-04-14 17:51**\u00a0- Fix implemented.\n    * **2026-04-14 18:01** - Recovery complete.\n    \n* **Root cause analysis:** An increase in traffic exceeded the capacity of our traffic management layer, causing degraded service for customers in the USA/Global region.\n\n## How Was It Resolved?\n\n* Established runbooks for load shedding and system upgrades mitigated the issue.\n\n## What We Are Doing to Prevent Future Incidents\n\n* **Resolution:** We made two key changes to prevent recurrence:\n\n    1. **Increased capacity of internal systems**\n    2. **Upgraded traffic controls at the network edge**\n    \n* **Verification:** Since implementing these changes, we have observed several additional instances of unusually high traffic matching this incident. In each case, the new controls handled the traffic automatically with no impact to customers or our systems.\n\n## Next Steps and Communication\n\n* No action is required from our customers at this time.\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-14T21:54:58.647-04:00",
"resolved_inferred": false,
"started_at": "2026-04-14T13:25:14.000-04:00",
"state": "postmortem",
"title": "1Password Hosted Service in the US experiencing high latency and errors",
"updated_at": "2026-05-06T11:34:07.043-04:00",
"url": "https://stspg.io/649xbzq5nnrk"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-22T11:17:50.561-04:00",
"resolved_inferred": false,
"started_at": "2026-04-13T13:24:27.560-04:00",
"state": "resolved",
"title": "SSO sign-in failing on some attempts for some users",
"updated_at": "2026-04-22T11:17:50.580-04:00",
"url": "https://stspg.io/4d2lgv6ywgyq"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-02T10:23:16.557-04:00",
"resolved_inferred": false,
"started_at": "2026-04-01T10:13:38.187-04:00",
"state": "resolved",
"title": "Teams accounts are unable to access Developer Environments",
"updated_at": "2026-04-02T10:23:16.577-04:00",
"url": "https://stspg.io/3xjqwqtjmvxf"
},
{
"body": "A fix has been deployed and this issue is now resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-31T20:43:02.537-04:00",
"resolved_inferred": false,
"started_at": "2026-03-31T17:28:13.895-04:00",
"state": "resolved",
"title": "Individual and Family accounts are unable to access Developer Environments",
"updated_at": "2026-03-31T20:43:02.555-04:00",
"url": "https://stspg.io/lkvq1qb9gtr5"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-16T13:43:02.427-04:00",
"resolved_inferred": false,
"started_at": "2026-03-16T13:11:54.093-04:00",
"state": "resolved",
"title": "Feature Availability Issue Affecting Some Customers",
"updated_at": "2026-03-16T13:43:02.447-04:00",
"url": "https://stspg.io/lvgtnys2tyyy"
},
{
"body": "Users were unable to retrieve documents across multiple platforms. This included degraded performance or failures when attempting to access documents via the CLI as well as through other interfaces.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-12T11:00:00.000-04:00",
"resolved_inferred": false,
"started_at": "2026-03-12T11:00:00.000-04:00",
"state": "resolved",
"title": "Degraded performance for retrieving documents",
"updated_at": "2026-03-12T13:53:51.123-04:00",
"url": "https://stspg.io/nbfkf03w1mhn"
},
{
"body": "The scheduled maintenance is now complete",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "maintenance",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-11T03:39:58.741-04:00",
"resolved_inferred": false,
"started_at": "2026-03-10T16:35:10.567-04:00",
"state": "resolved",
"title": "Scheduled Support Maintenance \u2013 Response Delays Expected",
"updated_at": "2026-03-11T03:39:58.757-04:00",
"url": "https://stspg.io/77vcmyvqhnyp"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-01-30T13:09:06.557-05:00",
"resolved_inferred": false,
"started_at": "2026-01-30T10:35:45.063-05:00",
"state": "resolved",
"title": "Intermittent Performance Issues",
"updated_at": "2026-01-30T13:09:06.574-05:00",
"url": "https://stspg.io/dh6xfrrg25lm"
},
{
"body": "The fix for the identified 2FA issue has been resolved as of version 8.12.4. The Sign-In attempts policy will become available in the coming weeks.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-03T10:52:27.515-05:00",
"resolved_inferred": false,
"started_at": "2026-01-23T15:33:54.000-05:00",
"state": "resolved",
"title": "Sign-in failure in some cases due to 2FA prompt not appearing",
"updated_at": "2026-03-03T10:52:27.530-05:00",
"url": "https://stspg.io/964rrz5cq8py"
},
{
"body": "# Incident Postmortem - 1Password Browser Extension Code Syntax Rendering Issue\n\n**Customer impact began \\(stable rollout start\\):** 2025-12-09\n\n**Investigation Started:** 2025-12-17\n\n**Incident Declared \\(UTC\\):** 2025-12-30 13:13\n\n**Fixed Release First Available:** 2026-01-01\n\n**Fixed Release Fully Available and Verified:** 2026-01-05\n\n**Incident Marked Resolved \\(UTC\\):** 2026-01-05 02:15\n\n**Service\\(s\\) Affected:** 1Password browser extension\n\n## Summary\n\nThe 1Password browser extension, which works by injecting code into web pages, inadvertently included code from PrismJS, a third party dependency, breaking syntax highlighting on some websites that display code blocks. The issue was reported in beta in early December, escalated after additional customer reports and a report from an external partner, and required releasing a stable update to remove the problematic dependency chain. This issue affected page rendering only and did not expose vault data or credentials.\n\n## Impact on Customers\n\nCustomers experienced broken code-block syntax highlighting on websites with `<code>` HTML elements while using the 1Password browser extension version 8.11.22 across all major browsers.\n\n* **Code snippet rendering issue:** Syntax highlighting for code blocks was broken on sites that display code snippet; impacted sites included developer documentation pages, technical forums, and blogs with code snippets.\n* **Browser scope:** Reported in Chromium-based browsers initially, and confirmed to affect all major browsers.\n* **Customer reports:** Started in beta with a single report on December 3, 2025, with additional reports in stable December 15\u201316, 2025. From December 9, 2025 through January 3, 2026 we received 55 unique customer reports. The issue was later observed in stable and amplified via social media.\n\n## What Happened?\n\nThe injected content script in the 1Password browser extension was able to include UI-related dependencies from other parts of the 1Password codebase in a context where they should not exist. This happened due to a small change that accidentally pulled in additional libraries caused by insufficient restrictions/guardrails on what dependencies could be imported into injected scripts.\n\n* **Detection and Escalation:**\n\n    On December 3, 2025, we received a beta report that syntax highlighting was broken when the 1Password extension was enabled. That report was incorrectly tagged and did not reach the owning team for timely triage, so we did not recognize it as a potential release blocker before version 8.11.22 rolled out to stable on December 9, 2025.\n\n\n\n    On December 17, 2025, an external partner reported the issue affecting stable. That report was routed correctly, we connected it to the earlier beta issue, and we began investigation and remediation.\n\n\n* **Timeline of Events \\(UTC\\):**\n\n    * 2025-12-03: Internal ticket created after beta report of broken syntax highlighting.\n    * 2025-12-09: Bug released to stable version 8.11.22.\n    * 2025-12-15 to 2025-12-16: Three additional customers reported the issue \\(not yet routed to the owning team\\).\n    * 2025-12-17: An external partner reported the issue affecting stable via a shared Slack channel; the Filling & Saving team began investigation.\n    * 2025-12-22: Fix merged \\(script pruning applied broadly to remove PrismJS from injected scripts\\).\n    * 2025-12-30: Reports surfaced that social media users observed the issue; stable update coordination began.\n    * 2025-12-31: Releases published across all browser platforms.\n    * 2026-01-01: Release approved by web stores.\n    \n* **Root Cause Analysis:** We did not have an enforced dependency boundary for injected content scripts, which allowed unexpected UI and runtime dependencies to be bundled into the page-injected context.\n* **Trigger:** A new import introduced an indirect dependency chain from an injected script to a UI module and ultimately to PrismJS.\n* **Contributing Factors \\(if any\\):**\n\n    * Mis-triage of beta report\n    * Missing side-effect regression tests\n    \n\n## How Was It Resolved?\n\nWe removed PrismJS from the scripts the extension injects into web pages by eliminating the import chain that pulled it into the injected bundle. We then shipped an updated extension across all supported browsers.\n\n* **Mitigation Steps:**\n\n    * Coordinated releasing a stable update once stable impact and broader visibility were confirmed.\n    \n* **Resolution Steps:**\n\n    * Fix merged on 2025-12-22.\n    * Fix was applied to the previous stable release.\n    * New beta/stable builds created and published across platforms.\n    * Fix approved by web stores as of 2026-01-01.\n    * Releases published across nightly, beta, and stable.\n    * Because extension updates roll out via browser web stores and auto-update schedules, we consider the fix fully available once all stores had approved the release and we verified the issue could no longer be reproduced on known affected sites.\n    \n* **Verification of Resolution:**\n\n    * We tested the new builds on known affected websites, validating that code block formatting is working as expected.\n    * We confirmed that the browser extension build no longer contained the PrismJS library.\n    \n\n## What We Are Doing to Prevent Future Incidents\n\n* Added lint rules to block disallowed imports into injected scripts.\n* Added internal documentation describing how to avoid dangerous imports and the potential impacts.\n* Ensure new lint rules to prevent dangerous imports are blocking in CI pipelines.\n* Add an automated test to internal testing sites to ensure injected content scripts don\u2019t introduce side effects.\n* Implement automated rules \\+ a triage SLA so new beta issues are reviewed by the owning team before stable release decisions.\n* Ensure bundle size changes are blocking changes, enforced by our automated build systems as part of the code review process.\n\n## Next Steps and Communication\n\n* Most browsers will auto-update extensions via their web stores. If the issue persists, customers should manually check for updates and verify they are on the fixed extension versions \\(8.11.27 and newer\\). Customers can verify their version in their browser\u2019s extension/add-ons settings \\(1Password extension version **8.11.27** or newer\\).\n* We will continue monitoring for regressions and follow up on guardrails \\(blocking lint rules, bundle-size checks, and automated website-side-effect tests\\).\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-01-23T15:04:42.406-05:00",
"resolved_inferred": false,
"started_at": "2026-01-23T15:04:42.360-05:00",
"state": "postmortem",
"title": "1Password Browser Extension Code Syntax Rendering Issue",
"updated_at": "2026-01-23T15:10:25.700-05:00",
"url": "https://stspg.io/zljt84cgsdvm"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-01-12T17:58:31.086-05:00",
"resolved_inferred": false,
"started_at": "2026-01-12T12:24:08.874-05:00",
"state": "resolved",
"title": "Intermittent issues with some forms on 1password.com",
"updated_at": "2026-01-12T17:58:31.103-05:00",
"url": "https://stspg.io/kx1ttvv0l3d5"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-12-17T10:08:48.151-05:00",
"resolved_inferred": false,
"started_at": "2025-12-17T09:35:23.396-05:00",
"state": "resolved",
"title": "Intermittent sign in issues on 1Password.com",
"updated_at": "2025-12-17T10:08:48.165-05:00",
"url": "https://stspg.io/hf6f83j5ww2m"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-30T16:52:23.620-05:00",
"resolved_inferred": false,
"started_at": "2025-11-30T15:35:48.095-05:00",
"state": "resolved",
"title": "1Password.com Website Access Degradation",
"updated_at": "2025-11-30T16:52:23.636-05:00",
"url": "https://stspg.io/tqfpfvx0jgtr"
},
{
"body": "# Incident Postmortem - Cloud Services Degraded\n\n**Date of Incident:** 2025-11-18 \n\n**Time of Incident \\(UTC\\):** 5:03pm UTC - 6:05pm UTC \n\n**Service\\(s\\) Affected:** SSO, sign in, sign up, CLI, web interface, access to vault content and other items, admin console, MFA\n\n**Impact Duration:** ~60 mins\n\n## Summary\n\nOn November 18, 2025, at 5:03 PM UTC, 1Password experienced degraded and temporarily unavailable cloud services for customers in the US region. The issue was caused by database resource exhaustion, causing operations to fail and connections to be rejected. This was not a security incident and no customer data was impacted. The issue was resolved by resizing the database to restore normal performance and ensure additional capacity for future growth.\n\n## Impact on Customers\n\n* **Single Sign-on \\(SSO\\), Multi-factor Authentication \\(MFA\\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to log in.\n* **Browser Extension:** Users who needed to authenticate via the web interface were unable to unlock their vaults.\n* **Web Interface, Administration**: Customers were unable to log in, sign-ups failed, syncing between devices was not functioning, access to vault and other items were unavailable and the admin console was not reachable.\n* **API Access:** CLI users and API requests received timeout errors and slow responses.\n* **Number of Affected Customers \\(approximate\\):** All customers utilizing cloud interfaces and APIs in the affected region for the duration of the incident.\n* **Geographic Regions Affected \\(if applicable\\):** US/Global.\n\n## What Happened?\n\n* **Timeline of Events \\(UTC\\):**\n\n    * 2025-11-18 4:59pm: Automated monitoring detects increased errors\n    * 2025-11-18 5:03pm: Team began investigating\n    * 2025-11-18 5:09pm: Servers restarted, service is still degraded\n    * 2025-11-18 5:23pm: Public status page updated to Investigating and services Degraded\n    * 2025-11-18 5:25pm: Servers scaled down to reduce database load\n    * 2025-11-18 5:41pm: Database instance size upgrade started\n    * 2025-11-18 5:44pm: Potentially problematic cron job disabled\n    * 2025-11-18 5:56pm: Services slowly started to scale up\n    * 2025-11-18 5:57pm: Services started to come back as the database instance resize completes\n    * 2025-11-18 6:05pm: Incident marked as Identified\n    * 2025-11-18 6:05pm: Team continues to monitor, performance has returned to normal levels\n    * 2025-11-18 6:23pm: Incident marked as Monitoring and services Operational\n    * 2025-11-18 7:16pm: Incident marked as resolved\n    \n* **Root Cause Analysis:** The refactor of a feature increased the impact of a poorly performing query that had previously gone undetected. The result was the exponential increase in resource consumption for the main database. Once resources were fully exhausted, the service rejected connections and all requests failed.\n* **Contributing Factors \\(if any\\):**\n\n    * Non-performant queries\n    * Database under-provisioned\n    \n\n## How Was It Resolved?\n\n* **Mitigation Steps:**\n\n    * Background services were halted to reduce load on the database.\n    * Application servers were scaled down to further reduce load.\n    \n* **Resolution Steps:** Increasing the database instance size resolved the issue.\n* **Verification of Resolution:** Monitoring metrics were closely observed to ensure error rates returned to normal and database performance had stabilized.\n\n## What We Are Doing to Prevent Future Incidents\n\n* **Improving monitoring:** We are updating our monitoring systems to better detect database issues like this before impacting customers.\n* **Improve database performance:** We are refactoring the responsible query to improve performance and reduce load, and tuning the background service to prevent resource contention.\n\n## Next Steps and Communication\n\nNo action is required from our customers at this time.\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-18T14:15:40.905-05:00",
"resolved_inferred": false,
"started_at": "2025-11-18T12:22:41.032-05:00",
"state": "postmortem",
"title": "Service Degradation Affecting Sign In, SSO, MFA",
"updated_at": "2025-11-21T14:06:58.051-05:00",
"url": "https://stspg.io/298y7wsr39w5"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-18T10:47:43.501-05:00",
"resolved_inferred": false,
"started_at": "2025-11-18T07:06:21.480-05:00",
"state": "resolved",
"title": "1password.com Website Access Degradation",
"updated_at": "2025-11-18T10:47:43.517-05:00",
"url": "https://stspg.io/x8xvlkzn838r"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-18T10:37:24.918-05:00",
"resolved_inferred": false,
"started_at": "2025-11-13T10:57:39.653-05:00",
"state": "resolved",
"title": "Item Usage Metrics Not Updating in Real Time",
"updated_at": "2025-11-18T10:37:24.939-05:00",
"url": "https://stspg.io/r67drjxy3d6d"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-06T17:04:08.478-05:00",
"resolved_inferred": false,
"started_at": "2025-11-06T15:16:11.532-05:00",
"state": "resolved",
"title": "Elevated Errors and Latency Affecting Authentication",
"updated_at": "2025-11-06T17:04:08.497-05:00",
"url": "https://stspg.io/qd2bs7mq59vz"
},
{
"body": "The upstream provider has reported full service recovery and Trelica is fully operational. This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-30T10:11:06.156-04:00",
"resolved_inferred": false,
"started_at": "2025-10-29T14:20:52.280-04:00",
"state": "resolved",
"title": "Some Trelica users experiencing issues",
"updated_at": "2025-10-30T10:11:06.172-04:00",
"url": "https://stspg.io/d7y80y65xpxn"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-28T21:37:45.856-04:00",
"resolved_inferred": false,
"started_at": "2025-10-28T16:41:50.485-04:00",
"state": "resolved",
"title": "Some MSP users are unable to access Managed Companies",
"updated_at": "2025-10-28T21:37:45.871-04:00",
"url": "https://stspg.io/xklbpzylhk4s"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-24T14:08:14.942-04:00",
"resolved_inferred": false,
"started_at": "2025-10-24T13:04:32.048-04:00",
"state": "resolved",
"title": "Degraded Performance When Sharing Items",
"updated_at": "2025-10-24T14:08:14.959-04:00",
"url": "https://stspg.io/ypld8fy54pgz"
},
{
"body": "# Incident Postmortem - 1Password Cloud Services Degraded\n\n* **Date of Incident:** 2025-10-20 \n* **Time of Incident \\(UTC\\):** 07:26:00 - 20:55:00 \n* **Service\\(s\\) Affected:** [1Password.com](http://1password.com/) website, Sign in, Access to passwords and other items \n* **Impact Duration:** 13 hours, 29 minutes\n\n## Summary\n\nOn October 20, 2025 at 07:26:00 UTC, [1Password.com](http://1password.com/) faced intermittent latency, authentication failures, and degraded service availability due to a major outage at [AWS in the us-east-1 region](https://aws.amazon.com/message/101925/). This was not a security incident and no customer data was affected.\n\nAs a result, the 1Password server-side application experienced degradation or intermittent failures, affecting up to 50% of traffic in the US region. Complete service restoration occurred in conjunction with AWS\u2019s final mitigations around 18:30 UTC.\n\n## Impact on Customers\n\nAll US customers accessing 1Password cloud services experienced intermittent latency, authentication failures, and degraded availability on [1Password.com](http://1password.com/).\n\n* **File Share:** Sharing of passwords via links could intermittently fail\n* **Login:** Users logging into vaults experienced timeout errors and slow responses\n* **Web Access:** Users accessing their vault through the web interface experienced timeout errors and slow responses\n* **API Access:** CLI users and API requests received timeout errors and slow responses\n\n## What Happened?\n\nAt 07:11:00 UTC, AWS began experiencing DNS resolution failures in the us-east-1 region, initially affecting DynamoDB and rapidly cascading to multiple AWS services.\u00a0 1Password monitoring detected impact at 07:26:00 UTC when monitoring alerts fired for inability to scale up clusters, and an incident was declared.\n\n1Password immediately deployed mitigations inside our infrastructure to ensure there was adequate compute capacity to serve our US-based users, which included pausing deployments and scaling down any services not critical to key functionality for our users.\n\n**Timeline of Events \\(UTC\\):**\n\n* 06:55:05 - 1Password monitoring triggers warning for unavailable Pods in Deployment \\(caused by inability to obtain AWS IAM credentials\\)\n* 07:03:06 - 1Password monitoring alerts for 5xx errors on auth start endpoint \\(caused by inability to obtain AWS IAM credentials\\) - pages authentication team, but alert recovers within minutes\n* 07:26:00 - 1Password monitoring alerts for inability to scale clusters, engineers begin investigating, Incident declared\n* 07:26:41- AWS confirms elevated error rates across multiple services\n* 07:49:06 - 1Password monitoring alerts for 5xx errors on auth start endpoint \\(caused by inability to obtain AWS IAM credentials\\)\n* 07:51:09 - AWS identifies DNS as the root cause, begins mitigation\n* 08:02:13 - 1Password suspends auto-scaling tooling to retain existing capacity\n* 09:27:33 - AWS reports significant recovery signs\n* 10:35:37 - AWS declares DNS issue fully mitigated, services recovering\n* 14:14:00-15:43:00 - AWS announced full recovery across all services; throttles EC2 launches\n* 16:42:49 - 1Password tooling and users start reporting 503s and inability to login due to volume of traffic\n* 16:50:00 - 1Password services restarted to reset and flush connections, prioritizing post-recovery traffic.\n* 20:53:00 - AWS resolves their incident\n* 20:55:00 - 1Password engineers overscale deployments for stability and overnight observation\n* Oct 21, 2025 - Incident resolved after confirmation of complete upstream recovery\n\n## How Was It Resolved?\n\n* **Mitigation Steps:** 1Password paused deployments and auto-management of cluster capacity to ensure enough capacity was available to serve users through peak access times. As demand outstripped available capacity, 1Password engineering reset the circuit breaker to allow additional connections to the service.\n* **Resolution Steps:** AWS announced system restoration and a reduction in throttling of EC2 API calls. To ensure sufficient capacity for peak traffic, 1Password engineers updated the required number of pods for core services the following business day and resumed auto-management of cluster capacity tooling. The following day, 1Password engineers resumed verification of the health of the systems, deployments, and auto-scaling of the services.\n* **Verification of Resolution:** Engineers observed monitoring systems and cluster management tooling logs to ensure system health.\n\n## **Root Cause Analysis**\n\n* **Root Cause Analysis:** The failures in AWS's internal network affected multiple AWS product APIs. This disruption directly impacted 1Password\u2019s ability to scale up infrastructure, deploy applications, and retrieve configuration data.\n* **Contributing Factors \\(if any\\):**\n\n    * Third-party incident response services and paging services were affected by the AWS incident, which complicated communications.\n    * Upstream customer IDPs were affected by the AWS outage, and returned errors that resulted in authentication failures.\n    \n\n## What We Are Doing to Prevent Future Incidents\n\n* **Improve Incident Response:** Create additional backup protocols for when our incident response tooling is unavailable.\n* **Improve multi-service outage response:** Create strong break-glass runbooks in the event of a multi-service cloud provider outage.\n\n## Next Steps and Communication\n\nNo action is required from our customers at this time.\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-21T09:51:06.701-04:00",
"resolved_inferred": false,
"started_at": "2025-10-20T05:05:23.290-04:00",
"state": "postmortem",
"title": "Intermittent performance issues with 1Password.com",
"updated_at": "2025-10-27T18:09:59.970-04:00",
"url": "https://stspg.io/2thnqbfzb7k7"
},
{
"body": "# Incident Postmortem - Degraded performance when accessing 1Password\n\n**Date of Incident:** 2025-09-26  \n**Time of Incident:** 4:20pm UTC - 5:39pm UTC  \n**Service\\(s\\) Affected:**  SSO, Web Sign In, Sign Up, Web Interface, CLI  \n**Impact Duration:** ~60 minutes\n\n## Summary\n\nOn September 26, 2025 at 4:20 UTC 1Password\u2019s web interface and APIs experienced degraded performance for all customers in the US region. This was not a result of a security incident and customer data was not affected.\n\n## Impact on Customers\n\nDuring the duration of the incident:\n\n* **Web interface, Administration:** Customers experienced delays when accessing the 1Password web interface.\n* **Single Sign-on \\(SSO\\), Multi-factor Authentication \\(MFA\\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to login.\n* **Command Line Interface \\(CLI\\):** CLI users faced increased latency and timeouts when attempting to access our web APIs.\n* **Browser Extension:** Users requiring web interface authentication experienced delays or failures.\n* **Number of Affected Customers \\(approximate\\):** ~30%\n* **Geographic Regions Affected:** [1password.com](http://1password.com) \\(US/Global\\)\n\n## What Happened?\n\nAt 4:20PM UTC and 5 PM UTC There were traffic bursts which caused extra load on one of our caches. This cache was under-provisioned to handle that spike of activity, which resulted in it exhausting available CPU. This caused cascading errors/latency which manifested in slow and failed requests.\n\n* **Timeline of Events \\(UTC\\):**\n\n    * 2025-09-26 4:20pm: Spike in customer traffic began\n    * 2025-09-26 4:29pm: Automated monitoring detects increased errors and latency\n    * 2025-09-26 4:35pm: The team activates our incident protocol and begins investigation\n    * 2025-09-26 4:58pm: The team decides to restart application servers\n    * 2025-09-26 5:00pm: The servers have been restarted, service is still degraded, as a second traffic burst begins\n    * 2025-09-26 5:18pm: Service starts to improve\n    * 2025-09-26 5:25pm: The team detects increased load for the second time\n    * 2025-09-26 5:33pm: The team restarts application servers again\n    * 2025-09-26 5:39pm: Service is back to normal, team continues to investigate\n    * 2025-09-26 7:26pm: Team has found the issue, and proceeds to upgrade cache instance size\n    * 2025-09-26 7:49pm: Cache upgrade completed successfully\n    * 2025-09-26 7:50pm: Team continues to monitor, performance has returned to nominal levels\n    * 2025-09-26 8:24pm: Incident is marked as resolved\n    \n* **Root Cause Analysis:**\n\n    A code library installed in July introduced latency issues for cache connections. Authentication operations weren't properly rate-limited, allowing large traffic influxes. During peak traffic periods, the cache infrastructure was operating near maximum CPU capacity. The incident occurred when a burst of authentication traffic pushed the cache CPU utilization to 100%. The increased latency and CPU usage together directly caused the incident.\n\n\n* **Contributing Factors:**\n\n    * Latency increase due to cache library version upgrade\n    * Inadequate rate limiting allowed traffic bursts to go unchecked\n    * Cache instance size is under-provisioned\n    \n\n## How Was It Resolved?\n\n* **Mitigation Steps:** Restarting application servers temporarily mitigated the latency and errors, but the problems returned when traffic spiked again.\n* **Resolution Steps:** Increasing the instance size for the cache resolved the issue.\n* **Verification of Resolution:** The incident team tested the upgrade in a staging deployment before executing it in production. They then monitored metrics to confirm the system returned to normal levels.\n\n## What We Are Doing to Prevent Future Incidents\n\n* **Improve capacity planning for cache:** We will ensure our internal infrastructure is properly sized to handle current traffic volumes and accommodate future growth. We'll implement regular resource evaluations to maintain adequate capacity as our traffic increases. We will also implement proactive alerting systems that notify our teams when resource utilization approaches critical thresholds.\n* **Update library to a more performant version:** We will upgrade our caching library to the latest stable version to eliminate the current latency issues.\n* **Improve rate limiting for operations that triggered the traffic burst:** Enhancing our rate limiting system will significantly improve our ability to handle future traffic bursts.\n* **Timeline for Implementation:** Observability improvements have already been implemented, and we will complete the remaining work by the end of Q1, 2026.\n\n## Next Steps and Communication\n\nNo action is required from our customers at this time.\n\n\u200c\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-26T16:24:58.720-04:00",
"resolved_inferred": false,
"started_at": "2025-09-26T12:51:38.625-04:00",
"state": "postmortem",
"title": "Degraded performance when accessing 1Password",
"updated_at": "2025-10-03T15:28:47.239-04:00",
"url": "https://stspg.io/0bw2wt81svs7"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-25T13:22:01.362-04:00",
"resolved_inferred": false,
"started_at": "2025-09-25T11:43:45.006-04:00",
"state": "resolved",
"title": "Some users are unable to interact with the admin console",
"updated_at": "2025-09-25T13:22:01.381-04:00",
"url": "https://stspg.io/vxmrb8r6166s"
},
{
"body": "# Incident Postmortem - Some customers are unable to interact with the admin console\n\n**Date of Incident:** 2025-09-24 \n\n**Time of Incident \\(UTC\\):** 02:27 - 17:16 \n\n**Service\\(s\\) Affected:** Admin console, Sign in \n\n**Impact Duration:** 36:49\n\n## Summary\n\nSome customers with certain account configurations were placed on a blocklist and presented with a 403 error page after accessing the admin console.\n\n## Impact on Customers\n\n* **Admin console:** Affected customers were presented with a 403 error page whenever they tried to interact with any of the admin console pages.\n* **Log in:** Affected customers were also unable to log in to the application.\n* **Number of Affected Customers \\(approximate\\):** 515\n* **Geographic Regions Affected \\(if applicable\\):** All regions\n\n## What Happened?\n\n* **Timeline of Events \\(UTC\\):**\n\n    * Sep 24th 2:27am: Spike in application monitoring alerted engineers to increased rates of IP blocking\n    * Sep 24th 3:00am: Cause identified as a change to requests in the application, which had been partially rolled out via a feature flag.\n    * Sep 24th 4:03am: The feature flag was enabled to all customers which reduced the spike, but IP blocks continued throughout the day.\n    * Sep 24th 10:03pm: Merged an application change to revert the change to prevent the issue reoccurring.\n    * Sep 25th 5:03pm: The change was deployed with scheduled application release, error rate dropped off shortly after.\n    \n* **Root Cause Analysis:** The issue was caused by GET requests to the Users API exceeding the URL length limit due to a recent change to append a list of UUIDs to the request parameters to resolve customer reported performance issues.\n* **Contributing factors:**\n\n    * Requests were switched from GET to POST to prevent requests from exceeding the URL limit, however an issue with the feature flag configuration was causing UUIDs to be sent with the GET endpoint.\n    * An underlying issue with the feature flag not resolving as expected in the application.\n    \n\n## How Was It Resolved?\n\n* **Mitigation Steps:** Customers were manually removed from the blocklist at multiple points in time as we evaluated the root cause and worked to patch the root issue.\n* **Resolution Steps:** The issue was mitigated by removing UUIDs at the API level if a GET request is used. Additional logging has been added to identify the root cause of the feature flag configuration issue.\n* **Verification of Resolution:** We monitored our server logs to ensure that we did not observe any additional GET requests to the affected URL.\n\n## What We Are Doing to Prevent Future Incidents\n\n* **Audit additional admin console API requests:** We\u2019re performing a sweep of admin console API requests to ensure the utilization of POST requests with highly parameterized URLs.\n* **Remove the feature flag misconfiguration:** We\u2019re correcting the way the feature flag is configured to ensure consistent outcomes.\n\n## Next Steps and Communication\n\nNo action is required from our customers at this time.\n\n\u200c\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-24T00:10:05.235-04:00",
"resolved_inferred": false,
"started_at": "2025-09-23T23:00:42.664-04:00",
"state": "postmortem",
"title": "Some users are unable to interact with the admin console",
"updated_at": "2025-10-08T17:56:48.851-04:00",
"url": "https://stspg.io/wg5wdhg3xtd6"
},
{
"body": "# **Provisioning Invites could not be accepted**\n\n**Date of Incident:** 2025-09-23   \n**Time of Incident \\(UTC\\):** 17:18 - 00:46   \n**Service\\(s\\) Affected:** Sign Up   \n**Impact Duration:** 7h 28m\n\n## **Summary**\n\nFor 7 hours and 28 minutes, 1Password Provisioning invites could not be accepted, presenting to the user as an invite expiry. Invites could not be accepted due to a web browser routing defect that was not caught during development, review, or release. First identified by customer reports approximately two and a half hours after release, the issue was escalated to development teams and an incident was immediately called. The root cause was identified as a defect introduced by a web client modification, and a fix was created, tested, and released. By 00:46 UTC, the fix was deployed to all environments and service was fully restored.\n\n## **Impact on Customers**\n\n* **Sign-up:** Provisioning invites could not be accepted.\n* **Number of Affected Customers \\(approximate\\):** 100% of provisioning invites could not be accepted\n* **Customer-facing impact**: Users clicking their invite links encountered a misleading \u2018Invite Expired\u2019 message.\n* **Geographic Regions Affected:** 1Password USA/Canada/EU/Enterprise\n\n## **What Happened?**\n\nA change to the web client contained a router defect that incorrectly rendered provisioning invites as expired. Users were presented with an error message that erroneously stated the invite was expired. The change responsible for introducing the defect was able to be released because it was not captured under automatic change notification rules, was lacking automated test coverage, and was not included in the set of manual tests.\n\n* **Timeline of Events \\(UTC\\):**\n\n    * 17:18: 1Password Release containing defect\n    * 19:53 \\(2 hours, 35 minutes later\\) First customer report\n    * 20:49 \\(56 minutes later\\) Escalation to developer teams\n    * 21:01: \\(12 minutes later\\) Incident called\n    * 22:02: \\(1 hour, 1 minute later\\) Root cause identified\n    * 22:29: \\(27 minutes later\\) Fix created and testing initiated\n    * 23:49: \\(1 hour, 30 minutes later\\) Fix merged\n    * 00:46: \\(57 minutes later\\) Fix released and service fully restored\n    \n* **Root Cause Analysis:** A change modified the order in which key provisioning web routes were rendered. As a result, the route handling provisioning invitations failed to use the correct query parameters and the invite rendered as expired.\n* **Contributing Factors:** Automated tests on this endpoint do not exist. Manual testing missed testing the Provisioning routes. The modified code was not covered by automatic change notification rules to notify the Provisioning team. An existing bug that can fail the resending of invites was an initial red herring during the investigation.\n\n## **How Was It Resolved?**\n\n* **Resolution Steps:** The defect in the 1Password web client was corrected so provisioning invites would render correctly.\n* **Verification of Resolution:** 1Password engineering tested the changes and validated that the functionality was restored, as well as verifying that requests for the affected endpoints were successful after the fix was deployed.\n\n## **What We Are Doing to Prevent Future Incidents**\n\n* **Improve automated tests:** We are enhancing our automated tests for the Provisioning Invite routes.\n* **Expand automatic change notifications:** Expanding coverage of automatic change notification rules for areas of code owned by the Provisioning team.\n\n## **Next Steps and Communication**\n\n* No action is required from our customers at this time. Existing invites do not need to be resent and may be accepted.\n* If you are still experiencing issues, please contact our support team at [support@1password.com](mailto:support@1password.com).  \n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-23T22:07:03.833-04:00",
"resolved_inferred": false,
"started_at": "2025-09-23T17:24:26.152-04:00",
"state": "postmortem",
"title": "Provisioning invite links are not working",
"updated_at": "2025-09-26T12:43:29.975-04:00",
"url": "https://stspg.io/m49bw2vmbhb6"
},
{
"body": "We have confirmed resolution of this issue and this incident is now resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-19T14:17:59.181-04:00",
"resolved_inferred": false,
"started_at": "2025-09-18T16:19:16.670-04:00",
"state": "resolved",
"title": "Email invites to some users are not being received for some customers",
"updated_at": "2025-09-19T14:17:59.202-04:00",
"url": "https://stspg.io/r11j4hk7p898"
},
{
"body": "Our provider has deployed a fix. Error rates have returned to normal. This incident is now resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-18T13:08:47.709-04:00",
"resolved_inferred": false,
"started_at": "2025-09-18T10:48:41.437-04:00",
"state": "resolved",
"title": "Contact and Sales Forms Returning Errors to Users",
"updated_at": "2025-09-18T13:08:47.726-04:00",
"url": "https://stspg.io/vvvmctx88rsm"
},
{
"body": "# Performance Degradation\n\n**Date of Incident:** 2025-09-03  \n**Time of Incident \\(UTC\\):** 11:06 - 12:07  \n**Service\\(s\\) Affected:** All APIs  \n**Impact Duration:** 61 minutes\n\n## Summary\n\nFor 61 minutes on the morning of September 3rd, 2025, all 1Password APIs in the US/Global environment had degraded performance or returned an error for approximately 20% of requests. 92% of the impact was  mitigated within 13 minutes at 11:19 by automation scaling up infrastructure. By 12:06 a manual restart of the remaining infrastructure completed mitigation. A permanent fix was implemented and deployed to prevent the issue from reoccurring.\n\n## Impact on Customers\n\n* **APIs:** High latency, or a 500 Internal Server Error.\n* **Number of Affected Customers:** 20% of all requests returned errors for 13 minutes, 1% thereafter.\n* **Geographic Regions Affected \\(if applicable\\):** 1Password USA/Global\n\n## What Happened?\n\n* **Timeline of Events \\(UTC\\):**\n\n    * 11:05: A customer started a stream of an unusually high volume of requests to an API with sub-optimal performance.\n    * 11:06: Some servers started consuming abnormally high memory, causing slow response times and high error rates.\n    * 11:19: Automation scaled up infrastructure to service additional load\n    * 11:30: Increased errors trigger escalation, on-call engineer begins investigation\n    * 11:51: Engineers declare an incident and alert response teams\n    * 12:02: Response team begins restarting affected servers.\n    * 12:07: All servers completed restarts, and error rates returned to normal levels\n    \n* **Root Cause Analysis:** A poorly performing cache operation was triggered repeatedly in a short period of time across multiple servers, leading directly to greatly delayed responses.\n\n## How Was It Resolved?\n\n* **Mitigation Steps:** Automatic instance scaling restored over 98% of operational capacity after 13 minutes. Full capacity was restored through manual intervention\n* **Resolution Steps:** We refactored the poorly performing query.\n* **Verification of Resolution:** We tested the affected API to confirm refactoring of query produced the desired performance improvement. We deployed the fix and monitored it for 24 hours to assert the issue was resolved.\n\n## What We Are Doing to Prevent Future Incidents\n\n* We are auditing services for sub-optimal query performance.\n\n## Next Steps and Communication\n\n* No action is required from our customers at this time.\n\n\u200c\n\nWe are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding.\n\nSincerely,\n\nThe 1Password Team",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-03T07:00:00.000-04:00",
"resolved_inferred": false,
"started_at": "2025-09-03T07:00:00.000-04:00",
"state": "postmortem",
"title": "Elevated API Errors",
"updated_at": "2025-09-10T19:12:54.901-04:00",
"url": "https://stspg.io/h7tdy5z31n3b"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-08-18T18:53:58.892-04:00",
"resolved_inferred": false,
"started_at": "2025-08-18T17:36:12.219-04:00",
"state": "resolved",
"title": "Events API Unavailable in the EU",
"updated_at": "2025-08-18T18:53:58.909-04:00",
"url": "https://stspg.io/qyxyl0ck4q4b"
}
]
}