https://approjects-vendor-status-watch.static.hf.space/v/coveralls.htmlIncidents from Coveralls's public status page, polled daily.Wed, 16 Sep 2026 12:28:20 +0000https://stspg.io/nd9v2n7g91wccoveralls:2026-08-11T12:02:12.967-07:00Tue, 11 Aug 2026 19:02:12 +0000This incident has been resolved.https://stspg.io/fddc34gx1s03coveralls:2026-07-15T10:28:59.136-07:00Wed, 15 Jul 2026 17:28:59 +0000This issue was resolved over the weekend. We will continue to monitor for elevated latency in the modified queues.https://stspg.io/tyhgk4l60pfpcoveralls:2026-07-10T08:00:47.000-07:00Fri, 10 Jul 2026 15:00:47 +0000This incident has been resolved but we believe it triggered a worsened incident overnight Sun night/Mon morning (US PDT) which has just been resolved. To be confirmed by full RCA, we believe a deluge of large repo uploads tied up individual web servers that handle frontline requests. Each web server is able to recover on its own, and did, but as volume increased all servers were eventually affected, only allowing short windows where requests could get through—rejecting most requests with 504 errors. We'll post a post-mortem when we understand more about what happened and how to prevent it going forward.https://stspg.io/rsxh917p1sk6coveralls:2026-07-07T09:43:22.969-07:00Tue, 07 Jul 2026 16:43:22 +0000Increased latency for large repos has been resolved for the general public. We will be clearing three outlier repos overnight, which should be fully cleared by tomorrow AM.https://stspg.io/84xn0rdzf008coveralls:2026-05-18T09:41:15.110-07:00Mon, 18 May 2026 16:41:15 +0000We are closing this incident. Our main background processing queue for larger repos has had no backups in 48 hrs. We will continue monitoring for recurrence.https://stspg.io/gspzxkkdmphccoveralls:2026-05-10T07:44:16.282-07:00Sun, 10 May 2026 14:44:16 +0000This incident has been resolved.https://stspg.io/1sfctfdlmr9ccoveralls:2026-02-24T14:52:29.054-08:00Tue, 24 Feb 2026 22:52:29 +0000# Suspended for Not Paying—While Paying _Latest update: Tuesday, Mar 24_ [This post-mortem is also available as a PDF](https://s3.amazonaws.com/assets.coveralls.io/statuspage/incidents/20260224/postmortem/Coveralls-Post-Mortem-Service-Outage-Feb-2026.pdf). ## Overview On February 24, our hosting provider suspended our account. Our service was down for 68 hours. The stated reason was past-due charges. When we reached our account manager on the day of suspension, he reviewed our account and told us what he saw: an "account restriction" that, in his words, "shouldn't have happened." Bizarrely, his review turned up an internal note claiming we'd made _no payments in six months_. That was demonstrably false and we provided immediate evidence to prove it. We did have an outstanding balancehttps://stspg.io/gn1j89710t5hcoveralls:2026-02-02T09:33:51.977-08:00Mon, 02 Feb 2026 17:33:51 +0000Just a note to address the gap in our Coverage Calculation Background Job Dequeue Time Graph today, MON, FEB 2 from 00:15:00 PST to 07:35:00 PST: Coveralls experienced no disruption in service at this time. Instead, a deployment issue cause the cron job that reports the metric to fail until it was resolved at 07:35:00 PST.https://stspg.io/02l24lf4x109coveralls:2025-12-19T06:34:44.392-08:00Fri, 19 Dec 2025 14:34:44 +0000We had to pause some queues to recover performance for new jobs, but will clear those as soon as we've recovered normal build times for new jobs (ETA: 15-min). If you are a user in APAC or EU, you may have had your jobs paused. One repo in particular has represented 90% overnight workload. We will reach out to that user.https://stspg.io/7hg0tqpk0948coveralls:2025-12-18T06:51:18.919-08:00Thu, 18 Dec 2025 14:51:18 +0000Build times are normal for all new builds. Background queues have been cleared of 99% of backlogged jobs; the only ones that remain are for large repos (>5K source files), which should clear in the next 30-45 minutes depending on size. No further effects on latency expected. Closing this incident.https://stspg.io/79r8vvnz4l3mcoveralls:2025-11-17T08:35:03.565-08:00Mon, 17 Nov 2025 16:35:03 +0000This incident has been resolved. Build times for all new builds is normal across the board. We are still clearing a backlog of background jobs, which should be clear in the next 15-20-min. If you are having issues with a slow or stuck build, feel free to reach out to us at support@coveralls.io. These steps will save time: 1) Mention this incident: https://status.coveralls.io/incidents/hkqt790213m5 2) Share your Coveralls Build URL (from your CI build log), or your CI build number.https://stspg.io/j214kcyh341bcoveralls:2025-11-10T08:14:00.004-08:00Mon, 10 Nov 2025 16:14:00 +0000We have attempted to improve latency this morning for new jobs from users in EU and US time zones, which has meant offloading older jobs to specialized queues with scaled up resources, which, at this point, have already drained. System-wide, latency continues improving for all users and should reach normal in 15-30 minutes for most users. If you are still experiencing elevated latency after that, or have any jobs from CI builds run in the last 24 hrs that have not completed, please reach out to us at support@coveralls.io and we'll investigate to determine whether your builds were caught up in this incident, or if they have a different cause. NOTE: Missing data points in the "DEQUEUE" Graph on our main Status Page does not indicate that processing stopped during that period, just that thhttps://stspg.io/b71rcnbr99wpcoveralls:2025-10-24T11:35:11.000-07:00Fri, 24 Oct 2025 18:35:11 +0000We are closing this incident having received no further reports of 504 errors today. We will continue to monitor for them.https://stspg.io/3424pqnymqnpcoveralls:2025-10-13T11:02:40.474-07:00Mon, 13 Oct 2025 18:02:40 +0000We have received no further reports of 504 timeout errors on coverage report uploads today, but we continue to monitor and will continue trying to improve our mitigations.https://stspg.io/fjs06cmd8l4dcoveralls:2025-10-10T08:23:27.445-07:00Fri, 10 Oct 2025 15:23:27 +0000This incident is resolved. We have applied an additional layer of monitoring that should help us catch these cases earlier.https://stspg.io/km3p38qs5n00coveralls:2025-09-26T08:28:03.000-07:00Fri, 26 Sep 2025 15:28:03 +0000We are closing this incident after recent mitigations and a weekend without any reports. We continue to implement mitigations and infrastructure changes we believe will further reduce incidents of this error type.https://stspg.io/lhyfrgjdj1djcoveralls:2025-09-03T11:00:31.000-07:00Wed, 03 Sep 2025 18:00:31 +0000**This is a postmortem on this specific issue:** Intermittent 500 Errors on Coverage Uploads **Summary** Between September 20–24, some customers experienced intermittent `500 Internal Server Error` responses during coverage uploads \(`POST /api/v1/jobs`\). The issue was initially hard to diagnose because: * Failures did not surface reliably in our error tracker \(BugSnag\). * They appeared to affect only some requests, some customers. **Impact** * Some coverage uploads failed to process, causing build reporting delays or gaps. * Frequency was low enough to appear intermittent, which delayed detection and resolution. **Timeline** * **Sep 20–23**: First customer reports of intermittent 500s. Initial theories involved a regression in a recent release of our coverage-reporter integratihttps://stspg.io/4jcpgc1b166lcoveralls:2025-08-20T08:48:22.100-07:00Wed, 20 Aug 2025 15:48:22 +0000We believe this issue has been resolved for now. The underlying cause still appears to be large spikes in incoming Web traffic from other outlier repositories that we have not yet identified or not yet paused. **Interim Solution**: To reduce the risk of recurrence, we have applied _temporary load balancer adjustments_ that change how requests are distributed, which should _lower_—if not _eliminate_—the frequency of **503** “**This website is under heavy load**” **errors**. **Permanent Solution**: We are also designing a permanent solution to _rate-limit abnormal request patterns_. This will require coordination at the policy/SLA level before it can be fully implemented. In the meantime, we will continue to closely monitor traffic and use targeted load balancer and web server configurhttps://stspg.io/3srf5qdsc4s1coveralls:2025-08-19T08:21:44.595-07:00Tue, 19 Aug 2025 15:21:44 +0000### Postmortem: Reports of “Website under heavy load” errors We experienced multiple intermittent errors over the past several days before we were able to identify the true root cause and resolve the issue. **Root Cause** The errors were caused by a single outlier repository generating extremely high-volume requests \(750–1,800\+ coverage report uploads per build\). Combined with the default “sticky request” behavior in Passenger Enterprise \(which routes repeat requests from the same IP to the same HTTP server\), this overwhelmed individual servers. Once a server’s request queue was exhausted, subsequent requests returned a `503` error with the message: _“This website is under heavy load.”_ Although each server was able to process individual requests within normal timeframes, the conhttps://stspg.io/xyv9nm9f08k9coveralls:2025-08-18T12:25:56.000-07:00Mon, 18 Aug 2025 19:25:56 +0000A postmortem for this incident and its related incidents has been posted [here](https://status.coveralls.io/incidents/wqbsxnzv0jsf): * [**Postmortem: Reports of “Website under heavy load” errors**](https://status.coveralls.io/incidents/wqbsxnzv0jsf) **Related incidents** 1. **Aug 13**: [Intermittent request rejections](https://status.coveralls.io/incidents/1n7plxrj8j44) 2. **Aug 14**: [Service unavailable with HTML error page or 500 errors](https://status.coveralls.io/incidents/v5mcbrsbhgt4) 3. **Aug 18**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/fr6sp5kyn128) 4. **Aug 19 \(Today\)**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/wqbsxnzv0jsf)https://stspg.io/h69wjdsj5hsrcoveralls:2025-08-14T07:35:14.384-07:00Thu, 14 Aug 2025 14:35:14 +0000A postmortem for this incident and its related incidents has been posted [here](https://status.coveralls.io/incidents/wqbsxnzv0jsf): * [**Postmortem: Reports of “Website under heavy load” errors**](https://status.coveralls.io/incidents/wqbsxnzv0jsf) **Related incidents** 1. **Aug 13**: [Intermittent request rejections](https://status.coveralls.io/incidents/1n7plxrj8j44) 2. **Aug 14**: [Service unavailable with HTML error page or 500 errors](https://status.coveralls.io/incidents/v5mcbrsbhgt4) 3. **Aug 18**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/fr6sp5kyn128) 4. **Aug 19 \(Today\)**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/wqbsxnzv0jsf) ‌ **Previous postmortem, prior to final resolution:** We rehttps://stspg.io/qbz08fdtjbjtcoveralls:2025-08-13T10:14:07.058-07:00Wed, 13 Aug 2025 17:14:07 +0000A postmortem for this incident and its related incidents has been posted [here](https://status.coveralls.io/incidents/wqbsxnzv0jsf): * [**Postmortem: Reports of “Website under heavy load” errors**](https://status.coveralls.io/incidents/wqbsxnzv0jsf) **Related incidents** 1. **Aug 13**: [Intermittent request rejections](https://status.coveralls.io/incidents/1n7plxrj8j44) 2. **Aug 14**: [Service unavailable with HTML error page or 500 errors](https://status.coveralls.io/incidents/v5mcbrsbhgt4) 3. **Aug 18**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/fr6sp5kyn128) 4. **Aug 19 \(Today\)**: [Reports of "Website under heavy load" errors](https://status.coveralls.io/incidents/wqbsxnzv0jsf)