{
"vendor": "Balena",
"slug": "balena",
"platform": "statuspage",
"status_url": "https://status.balena.io",
"last_checked": "2026-09-16T12:28:20Z",
"last_state": "ok",
"history_backfilled": true,
"first_watched": "2026-09-04T07:06:16Z",
"incidents": [
{
"body": "Unusual load on the proxy, caused some degraded performances on the device URL. \nEverything is now working properly.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-27T18:41:36.497Z",
"resolved_inferred": false,
"started_at": "2026-07-27T16:49:05.052Z",
"state": "resolved",
"title": "Elevated Error on Public Device Url",
"updated_at": "2026-07-27T18:41:41.209Z",
"url": "https://stspg.io/68jqp7m96k5w"
},
{
"body": "A routine version upgrade to a backend cache changed how the API handled its request queues. Under unusually heavy request load, those queues filled up and the API started dropping traffic, returning timeouts and errors.\n\nOver the course of the incident, affected users saw elevated API error rates that intermittently disrupted dashboard logins, API access, and device connectivity.\n\nWe mitigated it by rate-limiting the abnormal traffic at our edge and rolling the cache back to the previous version, which restored normal queue behaviour and recovered the platform.\n\nTo avoid a repeat, we're improving rate limiting so a single heavy consumer can't degrade service for others, and exploring autoscaling and load testing so infrastructure changes are validated under realistic load before they reach production.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-06T14:37:47.181Z",
"resolved_inferred": false,
"started_at": "2026-07-06T13:47:23.022Z",
"state": "postmortem",
"title": "Elevated API Errors",
"updated_at": "2026-07-20T18:50:15.847Z",
"url": "https://stspg.io/cp8l2tp32btz"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-07-09T18:24:27.299Z",
"resolved_inferred": false,
"started_at": "2026-07-02T14:03:50.645Z",
"state": "resolved",
"title": "Elevated API Errors",
"updated_at": "2026-07-09T18:24:27.313Z",
"url": "https://stspg.io/bd8n10fr2bgw"
},
{
"body": "A vulnerability mitigation update required replacing compute cluster nodes, which, when applied, rolled back due to a timeout. This caused some workloads \\(API, delta, builder\\) to become temporarily unavailable and triggering some undesired secondary effects, including ungracefully terminating a few long lived instances servicing VPN connections. While the rest of the services came back reasonably quickly within a minute or so, it too around one and a quarter hours to re-establish VPN tunnels.\n\nA scheduled maintenance will be posted later to perform this update during a planned outage window.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-05-05T21:03:33.607Z",
"resolved_inferred": false,
"started_at": "2026-05-05T18:12:25.509Z",
"state": "postmortem",
"title": "Elevated API Errors",
"updated_at": "2026-05-05T21:23:32.368Z",
"url": "https://stspg.io/t1ymgw8km62t"
},
{
"body": "Starting around March 11, some cloud builds began failing intermittently with no such image errors. The failures were non-deterministic and affected all architectures. At peak, some users saw around 50% failure rates.\n\nWe identified and fixed several bugs in the builder's image garbage collector that caused it to over-count freed disk space and run too aggressively, eventually deleting images that in-progress builds still needed. Fixes were deployed between March 19 and April 14, with build failure rates dropping to near-zero after the final deploy.\n\nWe're continuing to monitor and working on additional safeguards to prevent the garbage collector from targeting images that active builds depend on.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-21T16:30:32.566Z",
"resolved_inferred": false,
"started_at": "2026-03-31T12:55:25.446Z",
"state": "postmortem",
"title": "Elevated GIT/Application Builder Errors",
"updated_at": "2026-04-21T17:06:38.486Z",
"url": "https://stspg.io/nkpt9dcys9ng"
},
{
"body": "We experienced degraded API performance due to an internal configuration change that unintentionally increased system load, resulting in slower response times and reduced request capacity. Our team identified the issue and rolled back the change to restore stability. A root cause fix has been implemented and deployed aswell. The system has now fully recovered, and services are operating normally.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-25T14:07:04.631Z",
"resolved_inferred": false,
"started_at": "2026-03-24T12:43:08.142Z",
"state": "resolved",
"title": "Elevated API Errors",
"updated_at": "2026-03-25T14:07:04.647Z",
"url": "https://stspg.io/s8bkp2tw5vx1"
},
{
"body": "Between March 11 and March 25, some cloud builds experienced intermittent failures with \"no such image\" errors. The issue was non-deterministic and did not affect all builds.\n\nWe've identified a likely contributing factor and deployed mitigations that have stabilized build reliability. We're continuing to investigate the underlying cause to prevent recurrence.\n\nIf you experienced build failures during this window, re-running your build should succeed. We appreciate your patience while we worked through this, and we apologize for the disruption.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-25T13:18:51.711Z",
"resolved_inferred": false,
"started_at": "2026-03-23T16:57:53.653Z",
"state": "postmortem",
"title": "Builder Degraded performance",
"updated_at": "2026-03-25T15:38:27.899Z",
"url": "https://stspg.io/z889t6dhkljg"
},
{
"body": "We identified an issue in Dashboard v32.2.0, released on March 2, 2026, where opening the dashboard via a direct link to certain pages \\(such as billing or other account management pages\\) could result in being unexpectedly redirected to the fleets overview. This was caused by a race condition in our access control logic that made a routing decision before all authorization data had finished loading.\n\nThe issue was resolved on March 3, 2026 with a fix that ensures the dashboard waits for all access information to be available before determining whether a user can view a page.\n\nWe understand this was frustrating, particularly for users trying to manage billing or account settings via bookmarked or shared links. We apologize for the disruption and are adding test coverage for direct-link navigation to prevent similar regressions in the future.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-03T20:17:41.098Z",
"resolved_inferred": false,
"started_at": "2026-03-03T17:44:25.283Z",
"state": "postmortem",
"title": "Elevated Dashboard Errors",
"updated_at": "2026-03-09T13:01:41.046Z",
"url": "https://stspg.io/wflk617tf0yj"
},
{
"body": "On February 26, 2026 at approximately 19:00 UTC, users began experiencing failures when connecting to their devices via balena device ssh. The command returned a generic \"host error\" with no further detail. The issue affected all CLI-based SSH access through the balena proxy.\n\nWeb terminal access and balena device tunnel remained functional throughout the incident and were available as workarounds. Devices themselves were online and healthy; the failure occurred at the authentication stage between the proxy and the device.\n\n**Root cause**\n\nThe balena proxy runs several internal processes. As a resource-saving measure, the service that handles web terminal connections had been disabled in the proxy pods dedicated to CLI SSH, since those pods do not serve web terminal sessions. What was not recognized at the time was that this service also carried the responsibility for loading SSH identity keys into the shared SSH agent used by the proxy. With it disabled, the proxy pods had no keys loaded and could not authenticate with devices, causing every SSH connection attempt to fail at the authentication stage.\n\n**Resolution**\n\nThe service that handles web terminal connections was temporarily re-enabled in the SSH proxy pods while a permanent fix was being developed, restoring SSH key availability and resolving the authentication failures. A permanent fix has since been deployed.\n\n**Follow-up**\n\nWe have decoupled the SSH key loading from the service that handles web terminal connections so that it is an independent initialization process in the proxy. We also added additional monitoring to ensure that SSH access through the proxy is adequately checked.\n\nWe sincerely apologize for the impact this caused. We are committed to improving our processes to prevent issues like these.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-27T03:39:08.670Z",
"resolved_inferred": false,
"started_at": "2026-02-26T19:20:35.588Z",
"state": "postmortem",
"title": "Elevated Device SSH Errors",
"updated_at": "2026-03-17T12:03:47.570Z",
"url": "https://stspg.io/n404yktrjs0w"
},
{
"body": "On Feb 24, 2026 around ~17:20 UTC, a routine infrastructure deployment caused intermittent availability issues with Device URLs and web terminal access. Devices remained online and functional throughout, and CLI-based SSH access was unaffected.\n\nThe issue was caused by a configuration change that intentionally disabled several internal services no longer required by our proxy infrastructure. However, these services were still associated with pod health checks. A misconfigured override mechanism applied this change to production before it had passed through all required release gate checks, which would have caught the failing health checks.\n\nThe issue was identified quickly through automated monitoring and service was restored manually while a permanent fix was deployed. We have since corrected the underlying configuration override mechanism and are adding additional monitoring coverage to catch similar issues before they reach production.\n\nWe apologize for the disruption and thank you for your patience.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-24T20:54:26.375Z",
"resolved_inferred": false,
"started_at": "2026-02-24T17:27:22.514Z",
"state": "postmortem",
"title": "Elevated Device SSH Errors",
"updated_at": "2026-02-26T13:46:10.769Z",
"url": "https://stspg.io/1yy5hzvbtc40"
},
{
"body": "v2 delta generation service experienced failures from ~21:15 UTC Feb 9 to ~10:00 UTC Feb 10, 2026, due to a missing configuration dependency during a logic change.\n\n**Impact:**\n\n* v2 delta generation requests failed to complete\n* No data loss or security impact\n\n**Root Cause:** Recent logic changes were deployed without the required accompanying configuration update, preventing the service from completing v2 delta requests.\n\n**Resolution:** The logic changes were rolled back, restoring the service to its previous stable state.\n\n**Follow-up Actions:**\n\n* Prepare and deploy the permanent fix\n\nWe apologize for the disruption and any inconvenience this caused. We are committed to improving our processes to prevent similar issues in the future.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-11T00:02:30.160Z",
"resolved_inferred": false,
"started_at": "2026-02-10T09:33:42.929Z",
"state": "postmortem",
"title": "Elevated Delta Errors",
"updated_at": "2026-02-11T00:35:37.184Z",
"url": "https://stspg.io/yxn2lz6gz60x"
},
{
"body": "Balena devices were unable to connect to Cloudlink on February 10, 2026, from approximately 02:26 GMT to 07:11 GMT due to an expired server certificate. Devices that were already connected to Cloudlink were unaffected unless the connection was terminated.\n\n**Root Cause:** The Cloudlink servers were using an expired certificate that was due for replacement.  Consequently, incoming Cloudlink connections failed with a certificate verification error.\n\n**Resolution:** The certificate has been replaced, and Cloudlink servers were restarted to use the new certificate. Balena devices are expected to reconnect to Cloudlink within a few minutes after being disconnected due to the restart.\n\n**Follow-up Actions:**\n\n* Expand certificate expiry monitoring coverage to include all active certificates\n* Automate the certificate renewal process for Cloudlink\n\nWe apologize for any disruption this caused and appreciate your patience as we continue improving our processes and operations.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-10T07:56:34.794Z",
"resolved_inferred": false,
"started_at": "2026-02-10T04:05:10.780Z",
"state": "postmortem",
"title": "Elevated Cloudlink Errors",
"updated_at": "2026-02-10T12:04:00.448Z",
"url": "https://stspg.io/5c61796hff90"
},
{
"body": "Delta generation service experienced degraded performance from ~0900 UTC Feb 4 to ~1700 UTC Feb 5, 2026, due to 32-bit address space limitations affecting large image delta processing.\n\n**Impact:**\n\n* Delta generation timeout rate increased significantly for large container images\n* Users experienced slower image deployments and potential timeouts\n* No data loss or security impact\n\n**Root Cause:** Worker auto-discovery began routing some large delta generation tasks to 32-bit Docker sockets, causing memory constraint failures.\n\n**Resolution:** Corrected worker selection routing to only use 64-bit sockets for delta generation.\n\n**Follow-up Actions:**\n\n* Improve monitoring of delta generation performance and failure rates\n* Add alerting for threshold violations to detect similar issues faster\n\nWe apologize for the service interruption and any inconvenience this caused to your deployments. We are committed to improving our monitoring and alerting to prevent similar issues in the future.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-06T13:53:02.752Z",
"resolved_inferred": false,
"started_at": "2026-02-04T20:43:19.771Z",
"state": "postmortem",
"title": "Elevated GIT/Application Builder Errors",
"updated_at": "2026-02-09T15:15:01.992Z",
"url": "https://stspg.io/z3jzy1kyfsch"
},
{
"body": "### What Happened\n\nOn January 28, our API experienced degraded performance due to a cascading pod failure triggered by infrastructure inconsistency.\n\n### Root Cause\n\nWe unexpectedly had one node running on an older generation CPU, which caused uneven load distribution. When that node's pod failed a health check under normal traffic spikes, it triggered a cascading failure across other pods \\(a \"thundering herd\" effect\\).\n\n### Resolution\n\nWe resolved the incident by:\n\n* Scaling up the number of API pods to restore capacity during recovery\n* Removing the infrastructure inconsistency that caused the initial failure\n* Adjusting health check configurations to prevent similar cascading failures\n\nWe apologize for any disruption this caused and appreciate your patience as we continue improving our platform's resilience.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-01-28T22:06:19.336Z",
"resolved_inferred": false,
"started_at": "2026-01-28T18:54:18.743Z",
"state": "postmortem",
"title": "Elevated API Errors",
"updated_at": "2026-01-29T21:49:19.855Z",
"url": "https://stspg.io/kr137qqn1n5h"
},
{
"body": "Between 00:35 GMT and 01:00 GMT on January 28, 2026, users may have experienced intermittent connectivity issues with the balenaCloud API. Consequently, users may have had difficulty using the balenaCloud dashboard, and some devices also faced delays in reporting their current state to the backend.\n\n**Root Cause:** The incident was triggered during a planned configuration update to our environment. This update caused a brief, simultaneous restart of several core service components. As these services resumed, a high volume of concurrent device Cloudlink reconnections created a temporary surge in traffic. This surge exceeded the immediate processing capacity of our API backend, leading to increased latency and a temporary reduction in available service instances as they struggled to clear the request queue.\n\n**Resolution:** Our engineering team monitored the API as it processed the initial backlog. Service stabilized as the surge of reconnection requests subsided and API instances returned to healthy operating levels. Normal service was fully restored by 01:00 GMT.\n\n**Follow-up Actions:** We are reviewing the API deployment configuration and performance thresholds to better handle rapid spikes in traffic.\n\nWe sincerely apologize for the disruption to your workflow and appreciate your continued patience as we improve the resilience of our platform.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-01-28T01:00:00.000Z",
"resolved_inferred": false,
"started_at": "2026-01-28T01:00:00.000Z",
"state": "postmortem",
"title": "Partial API Outage",
"updated_at": "2026-02-26T13:36:16.177Z",
"url": "https://stspg.io/l3qpv6nvkn0b"
},
{
"body": "Insufficient AWS compute capacity overloaded the remaining nodes. This high load caused readiness probes to fail, triggering API restarts that created a feedback loop of increasing pressure.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-12-24T09:38:25.418Z",
"resolved_inferred": false,
"started_at": "2025-12-23T16:32:02.019Z",
"state": "resolved",
"title": "Degraded Performance",
"updated_at": "2025-12-24T09:38:25.434Z",
"url": "https://stspg.io/92xzdkjfyx06"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-12-05T09:37:17.743Z",
"resolved_inferred": false,
"started_at": "2025-12-05T09:05:53.807Z",
"state": "resolved",
"title": "An upstream provider outage is affecting connectivity to balenaCloud services",
"updated_at": "2025-12-05T09:37:17.762Z",
"url": "https://stspg.io/ztn70qkg57j3"
},
{
"body": "### Summary\n\nOn November 12, 2025, we experienced an incident causing increased build failures. This was traced to a significant network issue upstream of our hosting provider, which was outside their direct control. Build success rates returned to normal as of 17:15 UTC, and we formally closed the incident at 20:00 UTC after a period of extended monitoring.\n\n### What Happened\n\nA network issue between upstream service providers resulted in severe packet loss. This network degradation disrupted connections to our remote builder workers, causing an increase in builds failing with connection timeout errors \\(e.g., `ETIMEDOUT`\\).\n\n### Our Response & Mitigation\n\n* While the upstream network issue was being addressed, we brought additional builder workers online in unaffected regions to successfully route and process builds.\n* We were in communication with our hosting provider, who was monitoring the external network problem.\n\n### Resolution\n\n* **Recovery \\(17:15 UTC\\):** The upstream network provider resolved the issue, and we observed build failure rates and connection quality return to normal levels.\n* **Incident Closed \\(20:00 UTC\\):** After an extended period of monitoring to ensure all systems remained stable, we formally closed the incident.\n\nWe apologize for the disruption this caused.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-12T19:58:15.735Z",
"resolved_inferred": false,
"started_at": "2025-11-12T09:13:13.927Z",
"state": "postmortem",
"title": "Builder partially degraded service",
"updated_at": "2025-11-12T20:09:08.320Z",
"url": "https://stspg.io/99fbcq1zkjnl"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-20T23:03:43.933Z",
"resolved_inferred": false,
"started_at": "2025-10-20T08:45:35.303Z",
"state": "resolved",
"title": "Elevated GIT/Application Builder Errors",
"updated_at": "2025-10-20T23:03:43.947Z",
"url": "https://stspg.io/3vrqwm48cqk3"
},
{
"body": "An update to our kube-system infrastructure resulted in a disruptive pod rollout that left some devices disconnected from Cloudlink for up to 10 minutes. We apologize for the interruption and future updates to this component will be handled within planned maintenance windows.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-01T13:30:00.000Z",
"resolved_inferred": false,
"started_at": "2025-10-01T13:30:00.000Z",
"state": "resolved",
"title": "Elevated Device URLs/VPN Errors",
"updated_at": "2025-10-01T14:51:42.598Z",
"url": "https://stspg.io/7xx354czl8y8"
},
{
"body": "On **September 30th**, following a production deployment of a core component, a critical authorization failure affected the `balena device tunnel` command, preventing users from establishing port tunnel connections.\n\n### Summary of the Incident\n\nA subtle bug was introduced several weeks ago in a **small database query change** within a component. This change passed our standard review process and all pre-deployment testing, including unit tests utilizing mocked API endpoints.\n\nBecause of other pending changes, this component update was not immediately deployed. When it was finally released to production on September 30th, the query exhibited an unexpected incompatibility with the **live production API environment**.\n\nThe failure was not immediately apparent through our primary monitoring, but once the authorization issue was identified, our team quickly found the flawed query, deployed a patched component, and restored full functionality.\n\n### Root Cause and Timeline\n\n* **Change Introduction:** A small change to a database query was merged several weeks ago.\n* **Failed Validation:** The change passed unit tests and code review but failed to correctly interact with the real-world production API due to a subtle environmental or data-specific condition.\n* **Deployment & Failure:** On **September 30th**, the component was deployed. The authorization failure for `balena device tunnel` was subsequently observed.\n* **Resolution:** The bug was quickly diagnosed, the query was patched, and a fixed component was deployed to production, resolving the incident.\n\n### Corrective Actions\n\nWe are taking immediate steps to prevent this type of failure from recurring:\n\n1. **Unit Test Realism:** We have **updated our unit test mocks** to more accurately reflect invalid or non-standard production responses, ensuring future query changes are validated against real-world failure modes.\n2. **End-to-End Test Scheduling:** We are prioritizing the development and scheduling of a new **end-to-end test** specifically dedicated to validating the full functionality of the `device tunnel` command in a production-like environment. This will catch integration errors sooner.\n\nWe apologize for the interruption this caused to your workflow. We are committed to using this incident to improve the robustness and reliability of our continuous deployment process.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-30T21:56:45.703Z",
"resolved_inferred": false,
"started_at": "2025-09-30T19:47:47.495Z",
"state": "postmortem",
"title": "Elevated Device VPN Tunnel Errors",
"updated_at": "2025-10-01T12:52:40.556Z",
"url": "https://stspg.io/w6zn70z5jp4d"
},
{
"body": "#### Impact\n\nUsers were unable to access their devices via SSH through the web-terminal or the CLI. The CLI would return an error:\n\n`user does not have permission to access device`\n\nAn error in our Renovate configuration allowed our automation system to merge and deploy an unintended backend component to production. \n\n#### Resolution\n\nOur team quickly identified the issue and:\n\n1. Immediately rolled back the component to the previous verified version\n2. Restored remote SSH access for all affected devices\n3. Corrected the Renovate bot configuration to prevent similar automatic deployments\n\n#### Response\n\n* **Enhanced deployment controls:** We've restored our automation configuration to ensure all components must pass manual review before production deployment\n* **Improved monitoring:** We're considering implementing additional alerts to catch similar issues faster\n* **Process review:** We're reviewing our automated deployment processes to identify other potential gaps\n\nWe apologize for the disruption and appreciate your patience as we resolved this issue. If you continue to experience any problems, please contact our support team.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-17T22:01:28.892Z",
"resolved_inferred": false,
"started_at": "2025-09-17T20:10:15.971Z",
"state": "postmortem",
"title": "Elevated Device URLs/VPN Errors",
"updated_at": "2025-09-18T14:18:16.948Z",
"url": "https://stspg.io/fn5s8f1bc5kn"
},
{
"body": "The core problem stemmed from processing large files in the build context, which led to increased memory usage and, at times, network latency between regions.\n\n### Summary of Events\n\n* **Investigation and Identification \\(September 16, 2025\\):** We initially identified the issue as excessive memory consumption on our builder services.\n* **Temporary Mitigation \\(September 23, 2025\\):** We detected that network latency, particularly with larger build contexts, was contributing to the problem. We deployed temporary servers to mitigate this regional impact.\n* **Resolution \\(September 25, 2025\\):** The issue was resolved by implementing higher memory allocations on the builder services to handle the larger build context files and buffer memory issues.\n\n### Permanent Fixes and Structural Changes\n\nTo ensure long-term stability and prevent recurrence, our team implemented a series of comprehensive structural improvements focused on reducing memory usage, improving stream handling, and modernizing our codebase:\n\n* **Improved Build Stream Handling:** The most significant fix involved updating the system to buffer individual task build streams via the filesystem rather than memory. This leverages disk space, which scales much better and more cheaply than memory, to massively reduce the RSS memory usage.\n* **Streamlining and Modernization:** We undertook a broad effort to update and optimize how data streams are processed, including switching to native Node.js stream implementations \\(`stream.promises.pipeline`, `Transform`, `Passthrough`\\) to replace older, archived dependencies, which reduces the likelihood of memory leaks and edge cases.\n* **Other Enhancements:** We introduced various performance tweaks, such as avoiding reprocessing contract headers, improving error forwarding for stream failures \\(fixing a memory leak\\), and using model info to improve typings across the system.\n\nWe realize the recent incidents can be very disruptive and understand your frustration. Each of these incidents as well as their cause and steps to resolution have been documented on our status page. You can view the recent ones here on the history page with links to the details about each event: [**https://status.balena.io/history**](https://status.balena.io/history).\n\nAs you may know, the recent frequency of incidents is historically unusual for us. Our DevOps team worked overtime to resolve the issues as they arose. Our general process is not only to resolve, but also involve the broader team to make careful structural changes as preventative measures. For instance, we are reducing our reliance on hosting partners that may suffer from regional network network issues.\n\nThank you for your patience.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-25T08:57:27.000Z",
"resolved_inferred": false,
"started_at": "2025-09-16T02:33:40.760Z",
"state": "postmortem",
"title": "Builder partially degraded service",
"updated_at": "2025-10-07T16:39:21.501Z",
"url": "https://stspg.io/r20kt9pxrg6y"
},
{
"body": "On Sept. 10th, around 13:30pm UTC, our alerting system reported intermittent elevated API errors. We quickly determined the cause of the incident to be an overly aggressive liveness probe rotating our API pods. A fix was deployed immediately.\n\nWhile the API was recovering, an automatic update of the Cloudlink pods occurred. As the API was slower to respond, device reconnections and SSH authentication were slower than usual, but steadily improving.\n\nThis apparent recovery was masking a different issue in the Cloudlink update itself\u2014an issue that was severely impacting container performance under high concurrency. This problem was only apparent at the scale of the production environment and was completely invisible at the lower scale of our development and testing environments.\n\nOnce the concurrency issue had been properly identified, we quickly reverted the update and Cloudlink returned to its expected performance level.\n\nIn the aftermath of this incident, we're making a few important changes in our Cloudlink testing, validation, and deployment protocols to better detect and automatically revert issues that would only happen in production.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-10T22:08:34.223Z",
"resolved_inferred": false,
"started_at": "2025-09-10T13:47:40.754Z",
"state": "postmortem",
"title": "Elevated API Errors",
"updated_at": "2025-09-19T09:38:00.818Z",
"url": "https://stspg.io/qv310jqz0lzm"
},
{
"body": "We experienced a temporary outage affecting ESR image downloads due to a recent system update. The issue was quickly identified and resolved by rolling back to a previous stable version while we address the underlying compatibility issue.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-08-27T20:45:04.280Z",
"resolved_inferred": false,
"started_at": "2025-08-27T16:48:35.907Z",
"state": "postmortem",
"title": "Elevated Dashboard Errors",
"updated_at": "2025-08-28T15:01:17.216Z",
"url": "https://stspg.io/zdb5z94v8g4x"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-08-19T11:29:10.026Z",
"resolved_inferred": false,
"started_at": "2025-08-19T11:20:51.617Z",
"state": "resolved",
"title": "Elevated API Errors",
"updated_at": "2025-08-19T11:29:10.044Z",
"url": "https://stspg.io/bcq092dbpjs9"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-08-14T15:01:04.397Z",
"resolved_inferred": false,
"started_at": "2025-08-14T13:52:15.670Z",
"state": "resolved",
"title": "Partial API Outage",
"updated_at": "2025-08-14T15:01:04.412Z",
"url": "https://stspg.io/zq9m2tzy8w2p"
}
]
}