{
"vendor": "ClearlyIP",
"slug": "clearlyip",
"platform": "statuspage",
"status_url": "https://status.clearlyip.com",
"last_checked": "2026-09-16T12:28:20Z",
"last_state": "ok",
"history_backfilled": true,
"first_watched": "2026-09-04T07:06:16Z",
"incidents": [
{
"body": "This issue has been resolved since 9:45 CDT",
"first_seen": "2026-09-12T12:30:15Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-09-11T10:22:23.622-05:00",
"resolved_inferred": false,
"started_at": "2026-09-11T09:30:38.000-05:00",
"state": "resolved",
"title": "Reports of outbound call failures with Clearly Cloud USA",
"updated_at": "2026-09-11T10:22:54.804-05:00",
"url": "https://stspg.io/j6fx9385zz09"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-09-03T06:54:31.987-05:00",
"resolved_inferred": false,
"started_at": "2026-09-01T18:00:42.000-05:00",
"state": "resolved",
"title": "Outbound Calls to a Small Percentage of Rate Centers are Failing",
"updated_at": "2026-09-03T06:54:32.001-05:00",
"url": "https://stspg.io/pf84l0ly544z"
},
{
"body": "Mail continues to flow normally and at expected deliverability rates without delays.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-08-27T09:18:14.123-05:00",
"resolved_inferred": false,
"started_at": "2026-08-25T15:51:54.961-05:00",
"state": "resolved",
"title": "Delayed Mail Delivery to Microsoft Mail Services",
"updated_at": "2026-08-27T09:18:14.140-05:00",
"url": "https://stspg.io/zpdhkswlqqd7"
},
{
"body": "Several reports of poor audio quality on inbound calls were received beginning around 6:30 AM CDT.\u00a0 Investigation revealed the issue was region-specific, involving an upstream carrier who was promptly notified.\u00a0 We observed the issue to be fully resolved sometime shortly after 9AM CDT.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-08-21T10:07:05.755-05:00",
"resolved_inferred": false,
"started_at": "2026-08-21T08:46:35.941-05:00",
"state": "postmortem",
"title": "Intermittent call quality issues with Inbound Calls",
"updated_at": "2026-09-15T15:14:05.283-05:00",
"url": "https://stspg.io/8rm0wz75j1hv"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-22T14:38:57.086-05:00",
"resolved_inferred": false,
"started_at": "2026-06-22T11:59:52.200-05:00",
"state": "resolved",
"title": "US East trunking intermittent call quality issues",
"updated_at": "2026-08-27T02:58:07.276-05:00",
"url": "https://stspg.io/y22gzntlflh4"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-15T12:48:29.140-05:00",
"resolved_inferred": false,
"started_at": "2026-06-15T07:53:33.039-05:00",
"state": "resolved",
"title": "CA-East trunks offline",
"updated_at": "2026-08-27T02:58:07.314-05:00",
"url": "https://stspg.io/j36gcrzzdjpw"
},
{
"body": "All services remain stable and operational, free from the previously experienced issues.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-09T20:16:23.023-05:00",
"resolved_inferred": false,
"started_at": "2026-06-09T14:22:22.652-05:00",
"state": "resolved",
"title": "Pulsar360 - Hosted PBX - Toronto Region - Connectivity Issues",
"updated_at": "2026-08-27T02:58:07.349-05:00",
"url": "https://stspg.io/dvlbl49rwr00"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-09T12:24:42.991-05:00",
"resolved_inferred": false,
"started_at": "2026-06-09T09:27:35.678-05:00",
"state": "resolved",
"title": "Issues with Chat/SMS on Clearly Cloud USA - w",
"updated_at": "2026-08-27T02:58:07.387-05:00",
"url": "https://stspg.io/s46gt539llrl"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "none",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-01T15:13:55.737-05:00",
"resolved_inferred": false,
"started_at": "2026-06-01T08:54:10.040-05:00",
"state": "resolved",
"title": "US-Central Connectivity Instability",
"updated_at": "2026-08-27T02:58:07.424-05:00",
"url": "https://stspg.io/bxc7fc2hwhj8"
},
{
"body": "**Summary / Impact**\n\nOn the afternoon of May 31st, 2026, 14IP's network operations group began to observe connectivity issues with resources located at its US-Central \\(MKE\\) data center.\u00a0 The team began working with their counterparts at the data center facility to understand the nature of the problem.\u00a0 It was quickly determined that a distributed denial of service \\(DDoS\\) attack targeted upstream was affecting all tenants of the data center.\n\nSignificant malicious traffic \\(exceeding 450gbps\\) was measured upstream during the peak of the attack which overwhelmed equipment as well as primary, secondary, and tertiary links.\u00a0 This traffic was mitigated through coordinated, escalating filtering with upstream transit providers, as well as internal engineering to stabilize routing.\n\nDuring the incident, connectivity to systems and services primarily located in this data center were impacted significantly for periods of time between approximately 2 PM ET and 9:30 PM ET.\u00a0 This included Clearly Cloud, ComXchange hosted, Xivea, Business Connect, EPlatform, and several other systems.\u00a0 \\(US-East trunking services continued to operate during the US-Central incident.\\)\n\n**Follow-Up**\n\nSeveral facility-level network and system improvements have been implemented following the incident which should reduce the risk of future occurrences.\u00a0 This includes the deployment of additional DDoS scrubbing capacity, improved granularity for detecting attacks, additional fiber capacity upstream, as well as core network enhancements.\n\n14IP plans additional improvements to its systems and operations to improve platform resilience and flexibility when responding to future incidents.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-06-01T06:54:41.806-05:00",
"resolved_inferred": false,
"started_at": "2026-05-31T13:03:07.711-05:00",
"state": "postmortem",
"title": "US-Central Datacenter Connectivity Issue",
"updated_at": "2026-08-27T02:58:07.460-05:00",
"url": "https://stspg.io/grwlk4qvrlc7"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-05-28T13:50:44.687-05:00",
"resolved_inferred": false,
"started_at": "2026-05-28T12:16:37.000-05:00",
"state": "resolved",
"title": "Upstream Canadian carrier down",
"updated_at": "2026-08-27T02:58:07.498-05:00",
"url": "https://stspg.io/4ns6qzrdj58l"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-04-27T13:51:31.249-05:00",
"resolved_inferred": false,
"started_at": "2026-04-27T13:26:07.040-05:00",
"state": "resolved",
"title": "Clearly Cloud USA Call Panel showing insecure",
"updated_at": "2026-08-27T02:58:07.568-05:00",
"url": "https://stspg.io/wsbkr7vs6kmn"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-05-08T14:22:04.831-05:00",
"resolved_inferred": false,
"started_at": "2026-04-03T15:32:25.076-05:00",
"state": "resolved",
"title": "Issues with Grandstream devices with XCast Labs",
"updated_at": "2026-08-27T02:58:07.533-05:00",
"url": "https://stspg.io/4tcwqvlyk2k6"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-19T10:38:19.292-05:00",
"resolved_inferred": false,
"started_at": "2026-03-19T09:00:51.000-05:00",
"state": "resolved",
"title": "Issues with portals and CDM",
"updated_at": "2026-08-27T02:58:07.602-05:00",
"url": "https://stspg.io/b4s8fm8szht2"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-03-13T20:19:37.705-05:00",
"resolved_inferred": false,
"started_at": "2026-03-13T18:52:01.952-05:00",
"state": "resolved",
"title": "Outbound calling issues for some clients",
"updated_at": "2026-08-27T02:58:07.636-05:00",
"url": "https://stspg.io/1zr9ypf6ntz6"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2026-02-19T11:03:38.602-06:00",
"resolved_inferred": false,
"started_at": "2026-02-19T08:54:55.585-06:00",
"state": "resolved",
"title": "Call Panel inoperative",
"updated_at": "2026-08-27T02:58:07.669-05:00",
"url": "https://stspg.io/18tkfwplp8pc"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-12-15T13:00:57.440-06:00",
"resolved_inferred": false,
"started_at": "2025-12-15T12:41:17.843-06:00",
"state": "resolved",
"title": "Clearly Cloud USA registration and calling issues",
"updated_at": "2026-08-27T02:58:07.703-05:00",
"url": "https://stspg.io/y702jyz7hmtj"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-12-04T13:24:07.672-06:00",
"resolved_inferred": false,
"started_at": "2025-12-04T12:57:22.233-06:00",
"state": "resolved",
"title": "Intermittent calling issues on PBXPro.net",
"updated_at": "2026-08-27T02:58:07.737-05:00",
"url": "https://stspg.io/h4vckjkwbqqz"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-12-02T13:46:11.981-06:00",
"resolved_inferred": false,
"started_at": "2025-12-02T10:45:56.217-06:00",
"state": "resolved",
"title": "Calling issues with Clearly Cloud USA",
"updated_at": "2026-08-27T02:58:07.770-05:00",
"url": "https://stspg.io/k08szf2m2st4"
},
{
"body": "SIP2 and Proxy55 are not responding.  Calls routing to failover trunks",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-11T05:15:37.000-06:00",
"resolved_inferred": false,
"started_at": "2025-11-11T05:15:37.000-06:00",
"state": "resolved",
"title": "SIP2 and Proxy55 offline",
"updated_at": "2026-08-27T02:58:07.805-05:00",
"url": "https://stspg.io/vy4smnp151l8"
},
{
"body": "At approximately 11:27AM ET on November 3, ClearlyIP's NOC team began to receive system alerts indicating excess session activity within its US-East trunking systems.  The team's investigation identified unauthenticated INVITE messages being received from unknown IPs as the source of the excess traffic, as well as a customer generating unexpected levels of outbound calls.  Steps were taken to mitigate the traffic's impact, including restarting affected services to clear sessions and restore normal status.  By approximately 11:55AM ET, the US-East situation was stabilized and session traffic had returned to normal levels.\n\nThe team has identified a number of improvements and traffic management strategies to help avoid similar incidents which will be implemented in future maintenance events.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-11-03T12:10:27.375-06:00",
"resolved_inferred": false,
"started_at": "2025-11-03T10:49:07.359-06:00",
"state": "postmortem",
"title": "Calling issues with US East trunks",
"updated_at": "2026-08-27T02:58:07.838-05:00",
"url": "https://stspg.io/4skwnghp351q"
},
{
"body": "At approximately 4:10 PM ET, ClearlyIP's monitoring systems detected registration issues within US-Central trunking.  Within a few minutes, it was determined that one of the two primary trunking servers was maintaining its active calls \\(2.us-central\\) and that its counterpart \\(1.us-central\\) had stalled and was not accepting traffic.\n\nIt was quickly determined that the issue stemmed from a database cluster deadlock within the US-Central footprint.  Upon analysis, the deadlock's cause was identified to be related to unsuccessful credential change activity.  A series of actions were taken during the next 30 minutes which restored 2.us-central to service.  Following that, maintenance was performed to resolve version differences on the database node not currently in production before restoring it \\(and 1.us-central\\) to normal service shortly after 6PM ET.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-30T16:13:01.287-05:00",
"resolved_inferred": false,
"started_at": "2025-10-30T15:21:04.448-05:00",
"state": "postmortem",
"title": "US Central trunks rejecting",
"updated_at": "2026-08-27T02:58:07.872-05:00",
"url": "https://stspg.io/c68fz5g7g736"
},
{
"body": "A scheduled data center facility network maintenance event occurred during the evening of 10/29, which caused a variety of anticipated monitoring alarms during its completion.  The data center facility provided expected updates on the maintenance activity and no issues were immediately noted following its conclusion.\n\nAt approximately 8AM ET on 10/30, a customer reported that trunk registrations to the ClearlyIP US-East trunking servers were failing, while US-Central trunks were operating normally.  The issue was escalated by support to engineering, as it indicated potential issues with inbound calling mapped to the US-East region and would also affect customers who hadn't configured redundancy with ClearlyIP US-Central trunking.  After analysis, some database connectivity issues within the affected data center were identified as likely to be responsible.\n\nThe engineering team conducted a full review of the affected equipment and began work to restore access to the unresponsive database servers.  This required a careful review of replication and synchronization states before steps to re-activate each of the affected servers were taken.  By approximately 8:45 AM ET, the team had determined data integrity was intact and started operations to restore synchronization between the US-East database notes.  At approximately 9 AM ET, US-East trunking servers were reopened to customer traffic.  \n\nSystems monitoring and review occurred immediately following US-East reactivation, and support teams were advised to monitor customer equipment for proper re-registration with US-East servers.  During final analysis, several process and monitoring improvements were identified for implementation which should help avoid similar monitoring ambiguity \\(and potential disruption\\) during service provider maintenance events.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-30T09:15:28.971-05:00",
"resolved_inferred": false,
"started_at": "2025-10-30T06:05:53.000-05:00",
"state": "postmortem",
"title": "Call failures for inbound",
"updated_at": "2026-08-27T02:58:07.907-05:00",
"url": "https://stspg.io/7tc0mfm13qqb"
},
{
"body": "At approximately 2:30 PM ET on October 29, alerts began to be received by ClearlyIP's Network Operations team indicating problems accessing the database cluster used by Clearly Cloud for most US-based customers.  This was interfering with call routing for a portion of customers and prevented access to Call Panel and other system components.  The team immediately began investigating and observed the problem related to a protective mechanism which had detected large amounts of database request traffic and proactively restarted services to prevent data loss or corruption.  The source of the requests was quickly identified, stopped, and determined to be legitimate.  The team then checked each server in the cluster before determining it was safe to restore normal service by approximately 2:45 PM ET.\n\nA postmortem analysis on the source of the database traffic identified it to be related to delivering an unusual backlog of SMS and email messages from the platform.  Temporary measures were taken to prevent a recurrence of the same issue while long-term improvements are scheduled.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-29T14:17:59.711-05:00",
"resolved_inferred": false,
"started_at": "2025-10-29T13:37:07.868-05:00",
"state": "postmortem",
"title": "Clearly Cloud USA down",
"updated_at": "2026-08-27T02:58:07.942-05:00",
"url": "https://stspg.io/qg1w2b4j5f3p"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "critical",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-29T13:37:40.331-05:00",
"resolved_inferred": false,
"started_at": "2025-10-29T11:51:46.952-05:00",
"state": "resolved",
"title": "Issue affecting sending/recieving SMS and email from Clearly Cloud USA",
"updated_at": "2026-08-27T02:58:07.996-05:00",
"url": "https://stspg.io/hhy07njfhgmw"
},
{
"body": "This incident has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-10-20T14:12:11.558-05:00",
"resolved_inferred": false,
"started_at": "2025-10-20T11:45:17.000-05:00",
"state": "resolved",
"title": "Intermittent audio issues on call",
"updated_at": "2026-08-27T02:58:08.050-05:00",
"url": "https://stspg.io/rzbvzpqxh38l"
},
{
"body": "Upstream vendor issue has been resolved.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-19T15:17:53.712-05:00",
"resolved_inferred": false,
"started_at": "2025-09-19T14:55:11.938-05:00",
"state": "resolved",
"title": "Upstream vendor impact on portals",
"updated_at": "2026-08-27T02:58:08.096-05:00",
"url": "https://stspg.io/ttchh7gkybb9"
},
{
"body": "On 9/16 at approximately 1:50 PM CDT, ClearlyIP began to receive indications of connectivity problems affecting the Pulsar360 Denver data center. Infrastructure and network teams were immediately notified and began troubleshooting. The team quickly ruled out internal infrastructure issues, confirming that links to all Pulsar360 equipment were working normally. Within 15 minutes, the team had identified the source of the issue as a regional Internet routing problem caused by actions of an unrelated third-party fiber provider.\n\nThe issue originated within a route \\(BGP\\) advertisement that erroneously included large sections of IP addresses--among them addresses used by Pulsar360's Denver location. That bad route information caused many ISPs to misdirect traffic away from the Denver data center location.\n\nClearlyIP's engineering team pursued parallel efforts to resolve the underlying issue and explore workarounds that could be executed more rapidly.  While contact was still being made with the third-party company through upstream contacts, a solution involving forced advertisement of smaller network segments \\(counteracting the large mis-advertisement\\) was tested successfully.  The approach was approved quickly and the team immediately began executing it for all of the Pulsar360 Denver network segments during the next 30 minutes, restoring connectivity.\n\nStable connectivity had been achieved across the entire Denver network approximately one hour after the incident had been reported.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "major",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-09-17T15:11:08.507-05:00",
"resolved_inferred": false,
"started_at": "2025-09-17T14:44:13.225-05:00",
"state": "postmortem",
"title": "Inbound and outbound calling issues and web assets down",
"updated_at": "2026-08-27T02:58:08.131-05:00",
"url": "https://stspg.io/3v92bgyhk9sq"
},
{
"body": "At approximately 6:18 PM ET, an internal alarm detected a component restart within the Clearly Cloud US platform.  The engineering team quickly determined that the component successfully failed over to a redundant system while restarting, but this resulted in a few moments of disruption for dialing and status notifications \\(BLF, MWI, etc.\\)  The issue was immediately escalated to key members of the infrastructure and software groups who were conferenced in to analyze the situation.  Approximately ten minutes later, another restart \\(and failover\\) occurred, causing another brief disruption to new calls and status updates.   \n\nThe team identified the issue to be triggered by data integrity problems with several contact records.  Research confirmed that this issue had never been encountered before, occurring while a routine system task purges expired contact records every 10 minutes.  Several solutions were explored, and ultimately a repair of the affected data table was performed at approximately 7:40 PM ET, resolving the underlying problem.  Additional cleanup tasks were implemented at approximately 8:00 PM as the escalation team monitored the platform.  After recording several task cycles without further incident, the team reviewed potential improvements and documented the incident before marking it resolved at approximately 8:15 PM ET.\n\nWhile every part of Clearly Cloud has redundancies and most of these fail over gracefully without impacting operations, the component in question requires a few moments for backup systems to take over.  During this timeframe--between a few seconds and a minute--established calls would be unaffected, but new calls may fail to complete and status messages for BLF, MWI, parking, and other notifications may fail to be sent.  Although the issue's impact was limited due to the time of day, multiple brief failovers in a short timeframe caused disruption, which was noticed by several users\n\nClearlyIP's engineering team will incorporate what was learned about this incident within upcoming platform improvements.",
"first_seen": "2026-09-04T07:06:16Z",
"impact": "minor",
"last_seen": "2026-09-16T12:28:20Z",
"resolved_at": "2025-08-13T20:51:30.522-05:00",
"resolved_inferred": false,
"started_at": "2025-08-13T17:20:58.000-05:00",
"state": "postmortem",
"title": "Intermittent calling issues with Clearly Cloud USA",
"updated_at": "2026-08-27T02:58:08.171-05:00",
"url": "https://stspg.io/wqx1sr60vxw9"
}
]
}