https://approjects-vendor-status-watch.static.hf.space/v/upstash.htmlIncidents from Upstash's public status page, polled daily.Wed, 16 Sep 2026 12:28:20 +0000https://stspg.io/yfkkt5363477upstash:2026-09-09T12:00:00.000ZWed, 09 Sep 2026 12:00:00 +0000Upstash Vector experienced a brief DNS resolution issue in the US East (us-east-1) region lasting approximately 20 minutes. During this period, new DNS resolution attempts may have failed. Existing connections and clients with cached DNS records were not affected. The issue has been resolved, and DNS resolution is operating normally.https://stspg.io/dmdmfrm8lyczupstash:2026-09-03T08:00:00.000ZThu, 03 Sep 2026 08:00:00 +0000We experienced a brief issue affecting access to the Upstash Console. Our team quickly identified the cause and resolved the issue within minutes. The console is now operating normally.https://stspg.io/69bd84bqrpcqupstash:2026-08-28T22:20:00.000ZFri, 28 Aug 2026 22:20:00 +0000Between 22:20 and 22:33 UTC, QStash clients in us-east-1 experienced disruptions related to message persistence. The team applied a fix, and service has been restored.https://stspg.io/0l5sf44ng7bnupstash:2026-07-22T08:50:12.940ZWed, 22 Jul 2026 08:50:12 +0000This incident has been resolved.https://stspg.io/9qvjxdfprs57upstash:2026-07-16T06:30:00.000ZThu, 16 Jul 2026 06:30:00 +0000Partial outage due to http parsing errors. Problem was resolved couple minutes later. If problem continues, refreshing the DNS cache is recommended.https://stspg.io/9gynqrnvcpyjupstash:2026-06-25T15:23:44.611ZThu, 25 Jun 2026 15:23:44 +0000This incident has been resolved, we will publish RCA soon.https://stspg.io/r5k8bdfqm35cupstash:2026-06-23T16:22:25.859ZTue, 23 Jun 2026 16:22:25 +0000### **Summary** On June 23, 2026, QStash users in the EU-CENTRAL-1 region experienced elevated latency and degraded performance. The issue was caused by a regression introduced in a recent enhancement to Flow Control scheduling logic. The change increased resource consumption under load, leading to reduced performance on affected shards. The incident was resolved by rolling back to the previous stable version. ### **Impact** * Service: QStash \(EU-CENTRAL-1\) * Start: 2026-06-23 16:22 UTC * Resolved: 2026-06-23 17:14 UTC * Duration: ~52 minutes * Impact: Increased latency and degraded performance for workloads routed to the affected instance in the EU. Multiple customers in the region experienced slower request processing during the incident window. ### **Root Cause** We recently deplhttps://stspg.io/y6kf1kvffvjjupstash:2026-06-23T06:05:57.207ZTue, 23 Jun 2026 06:05:57 +0000Between 01:18 and 04:03 UTC, customers were unable to access running boxes or perform create, read, update, or delete operations. Running containers were not affected and continued operating throughout. The issue has been fully resolved and all box operations are functioning normally. We are taking all necessary measures to prevent this from happening again. We apologize for the disruption.https://stspg.io/5m39sbnstm49upstash:2026-06-19T09:11:00.000ZFri, 19 Jun 2026 09:11:00 +0000## What happened A configuration change applied to QStash's EU networking layer introduced an outbound connectivity fault. Between **09:11 and 09:45 UTC**, a subset of QStash instances had degraded egress connectivity, and some dispatched messages may have failed on their initial attempt. A related side effect caused some servers to egress from IP addresses outside QStash's advertised outbound range until **11:24 UTC**; during that window, endpoints enforcing QStash IP allowlists may have rejected requests from those servers. QStash's automatic retries delivered many affected messages on subsequent attempts once connectivity was restored, and QStash is now operating normally. ## Customer impact Impact was limited to the EU region and to two effects within the window above: * **Deliverhttps://stspg.io/w6vy8lf5kbyxupstash:2026-06-04T10:10:36.337ZThu, 04 Jun 2026 10:10:36 +0000This incident has been resolved.https://stspg.io/d5zkphtnpb9jupstash:2026-05-29T20:18:02.657ZFri, 29 May 2026 20:18:02 +0000The issue affecting some Upstash Vector indexes in the US-EAST-1 region has been resolved. Our team investigated the incident and identified the conditions that were contributing to elevated memory pressure on the affected servers. We mitigated those conditions by reducing memory utilization on the impacted nodes, rebalancing affected workloads where needed, and increasing available headroom capacity across the region. All affected indexes should now be operating normally, and based on the mitigations applied, we do not expect this issue to recur. We apologize for any inconvenience this may have caused.https://stspg.io/fcgvdbq1n5d9upstash:2026-05-22T23:27:48.418ZFri, 22 May 2026 23:27:48 +0000This incident has been resolved.https://stspg.io/qyl44kv5lw09upstash:2026-05-14T15:47:23.000ZThu, 14 May 2026 15:47:23 +0000Earlier today, unexpected load on our proxies caused intermittent connection issues for Upstash Redis in the following regions: us-east-1, us-west-1, ap-southeast-2, and ap-south-1. During this period, some clients may have seen connection timeouts or elevated error rates when reaching their databases. Our team identified the issue quickly and applied workarounds to relieve pressure on the affected proxies. Connection health has since been restored and we've been monitoring the regions to confirm everything is stable. All systems are now operating normally. We appreciate your patience and apologize for any disruption this may have caused.https://stspg.io/v8n17s5pgq0yupstash:2026-05-12T09:49:32.897ZTue, 12 May 2026 09:49:32 +0000On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruhttps://stspg.io/xc9fkbk20d2wupstash:2026-05-11T15:05:43.087ZMon, 11 May 2026 15:05:43 +0000On May 12th and 13th at various times, a subset of Upstash Redis instances on [Fly.io](http://Fly.io) experienced intermittent hangs and elevated error rates. The Redis process would stall inside a logging syscall — alive but not making progress — which made the issue hard to spot from our usual telemetry. After investigating with Fly's team, we identified the root cause as a bad interaction between a recent guest kernel update on Fly's newer machines and an upstream Cloud Hypervisor bug \([cloud-hypervisor#7672](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7672)\) affecting log writes from inside the VM. We mitigated by disabling the affected logging paths, and Fly has since rolled out a hypervisor-side patch, fully resolving the issue. No data was lost. Sorry for the disruhttps://stspg.io/zh00zgv2ks17upstash:2026-05-08T09:46:14.134ZFri, 08 May 2026 09:46:14 +0000**Root Cause Analysis** On **April 24**, we deployed a more optimized scheduler implementation in the **US East \(N. Virginia\)** region. On **May 8**, a user who had active schedules deleted their account. Under normal behavior, scheduled tasks associated with a deleted account should wake up, detect that the account no longer exists, and exit after performing cleanup. Due to a bug introduced in the new scheduler implementation, this code path did not return early as intended. Execution continued and resulted in a nil pointer dereference. A second issue then amplified the impact. When a panic occurs in the scheduler, it is designed to be recovered, logged, and isolated so that the process remains healthy. Because of another bug in the panic recovery path, the panic was not propehttps://stspg.io/qbgh8qr9p3cjupstash:2026-05-05T14:24:43.025ZTue, 05 May 2026 14:24:43 +0000# **Incident Postmortem: Scheduled Jobs Inconsistency in US Region** On May 1, 2026, we experienced an incident affecting a subset of schedules in the US region following a recent infrastructure update. The issue has been resolved, and all affected schedules have been restored. ## **Summary** As part of an ongoing scalability improvement, we recently updated scheduling infrastructure in the US region to a new architecture. During this transition, a legacy execution path remained in the codebase as a fallback mechanism. On May 1, a bug caused the system to revert to the legacy path. This resulted in inconsistent state between the old and new scheduling systems for some users. ## **Impact** The incident affected a limited number of users in the US region. **Most users were not affecthttps://stspg.io/jhtjz9c6723cupstash:2026-04-02T18:45:17.073ZThu, 02 Apr 2026 18:45:17 +0000Replication complete, incident resolved.https://stspg.io/wjqvlzqpdspsupstash:2026-03-27T15:30:00.000ZFri, 27 Mar 2026 15:30:00 +0000Due to a race condition in a process that attaches static IPs to nodes, some of the IPs in the dns were detached from the nodes, causing timeouts.https://stspg.io/fstdc8k6z87lupstash:2026-03-06T13:50:17.434ZFri, 06 Mar 2026 13:50:17 +0000On March 6, between approximately 13:44–14:07 UTC, some databases experienced elevated latency and connection errors in the Tokyo \(ap-northeast-1\) region. The issue was caused by a sudden spike in traffic that significantly increased network utilization and connection load on a subset of nodes. Our team mitigated the incident by scaling up capacity in the region and redistributing load across additional nodes. Service recovered once the additional capacity was brought online. Resolution We have increased the number of machines in the Tokyo region to provide additional headroom and reduce the likelihood of similar incidents during traffic spikes. Next Steps We are continuing to review capacity safeguards and connection-handling limits to improve resilience against sudden traffic surhttps://stspg.io/cfyy1vkxqjj6upstash:2026-01-23T15:30:00.000ZFri, 23 Jan 2026 15:30:00 +0000We identified the cause of elevated latency impacting some databases in us-east-1 region between 15:30–15:35 UTC as a sudden surge of connection attempts that hit OS-level connection limits on our proxy layer. This resulted in slower new connection establishment and increased latency for some requests. Databases were not impacted. We are implementing additional proxy-level metrics and safeguards to detect and manage similar edge cases earlier.https://stspg.io/53w7vln17ypdupstash:2025-12-10T11:50:47.166ZWed, 10 Dec 2025 11:50:47 +0000Issue has been identified and replicas were successfully reconnected.https://stspg.io/tf2f3cc3vnj4upstash:2025-12-05T09:03:56.241ZFri, 05 Dec 2025 09:03:56 +0000This incident has been resolved.https://stspg.io/5j0xwwz0gn3dupstash:2025-12-01T07:00:00.000ZMon, 01 Dec 2025 07:00:00 +0000We identified and fixed a bug that could cause messages with Flow Control enabled to be delayed longer than their configured delay, resulting in unexpectedly long pending times. The fix is in place and the issue should not recur. If you’re still seeing unusually long-delayed messages, please contact support@upstash.com and we can help with remediation.https://stspg.io/8twfwt7qbdynupstash:2025-10-20T14:52:07.077ZMon, 20 Oct 2025 14:52:07 +0000A fix has been deployed as a workaround so that our systems are not affected from the ongoing incident of the cloud providerhttps://stspg.io/blt426dr5x5bupstash:2025-10-20T07:00:00.000ZMon, 20 Oct 2025 07:00:00 +0000As a side effect of an incident on the underlying cloud provider, Upstash Console has had availability issues between 07:00UTC and 09:23UTC. Only Upstash Console is impacted, Upstash products remained operational.https://stspg.io/gn5kd8bwmxdpupstash:2025-10-10T15:46:00.000ZFri, 10 Oct 2025 15:46:00 +0000Between 15:46–15:55 UTC, some client connection attempts to databases in us-east-1 timed out due to unexpected high load on a server. The node was recovered at 15:50 UTC, and the updated DNS record propagated by 15:55 UTC. Services are operating normally.https://stspg.io/rt0hggtwj47zupstash:2025-09-09T12:00:00.000ZTue, 09 Sep 2025 12:00:00 +0000Impact: A subset of clients connecting through the eu-central-1 region experienced increased error rates and timeouts when accessing certain databases. Clients in us-west-2 were also briefly affected. The issue was limited in scope and did not impact other regions. Root Cause: During an ongoing migration to improve database routing reliability, a configuration step was applied inconsistently across regions. Resolution: Our monitoring alerted us within minutes, and the migration was promptly rolled back for the affected regions. Service definitions were restored, and normal database connectivity resumed by 15:08 UTC. Next Steps: We are reviewing our migration process to ensure consistency across all regions and adding additional safeguards to prevent similar issues in the future.https://stspg.io/s4f8dn5zlxwbupstash:2025-08-18T12:37:02.000ZMon, 18 Aug 2025 12:37:02 +0000Between 12:37 UTC and 12:50 UTC, an overload in the connection proxying system resulted in instability for a subset of databases. The root cause was identified as a misconfiguration in the routing rules, which caused certain requests to experience timeouts during the initial phase of a gradual deployment. Upon detection, the deployment was immediately rolled back, restoring normal service. We are reviewing our deployment and configuration validation processes to prevent similar issues in the future.