Incident Report for March 26th, 2025

Summary

Between 2025-03-25 22:42 and 2025-03-26 12:00 UTC, some Soracom IoT SIM subscriptions experienced issues where connections were unavailable or difficult to establish. In Australia, the issue continued until 2025-03-27 04:00.

Affected Services

The following subscriptions were affected during the above timeframe:

• plan01s
• plan01s - Low Data Volume
• planX1
• plan-NA1-package

Between 2025-03-25 22:42 and 2025-03-25 23:16 UTC, all new connections for the affected subscriptions failed. In addition, data communication failed for sessions routed through the Rendezvous Point in Frankfurt (Germany). For sessions already established through other Rendezvous Points, data communication was maintained as long as the sessions were not re-established or updated.

Between 2025-03-25 23:16 and 2025-03-26 12:00 UTC, new connections for the affected subscriptions continued to fail. For already-established sessions, data communication was maintained as long as the sessions were not re-established or updated.

Between 2025-03-26 12:00 and 2025-03-27 04:00 UTC, some new connections in Australia failed. For already-established sessions, data communication was maintained as long as the sessions were not re-established or updated.

Cause

Soracom IoT SIMs connect to the Soracom platform via each country’s mobile network operator (MNO) and our network partner. The connections between the network partner and Soracom are established via dedicated and redundant lines in an Active-Standby configuration.

At 2025-03-25 22:42 UTC, planned maintenance was conducted on one of these dedicated lines, specifically a link connecting a transit router that sits between the network partner and Soracom’s AWS infrastructure. During the maintenance, a latent misconfiguration in the transit router was exposed, preventing traffic from properly failing over to an alternate active path. As a result, communication between the network partner and Soracom’s AWS-hosted cellular core network was interrupted.

This led to a network-wide connectivity failure, where devices were unable to register with the network or establish data sessions throughout the maintenance window.

Although the maintenance itself was completed quickly, and the connection was restored at 23:16 UTC, the interruption had a cascading effect. Once connectivity resumed, a surge of signaling requests from IoT devices attempting to reconnect caused a signaling storm, which overwhelmed parts of the network partner’s infrastructure.

Additionally, some MNOs detected this spike in network activity and imposed temporary communication restrictions on Soracom traffic. As a result, even after the core connection was restored, data sessions continued to fail for some devices until those carrier-level restrictions were lifted, leading to a delayed recovery.

Resolution

Upon detecting abnormal communication behavior, Soracom promptly updated the Status Dashboard, initiated an internal investigation, and escalated the issue to the network partner responsible for the affected subscriptions. The connection between the network partner and Soracom was restored following the completion of the planned maintenance.

To manage the resulting traffic surge, Soracom implemented capacity control measures and attempted to mitigate the impact by switching to alternate dedicated line paths and applying load-reduction strategies. However, these actions did not lead to an immediate resolution.

Working closely with the network partner, we eventually identified specific infrastructure components that had become overloaded and were unable to handle the volume of signaling traffic. After implementing targeted mitigations, cellular session establishment began to recover at 2025-03-26 09:38 UTC. After monitoring the traffic trend, we announced the incident resolved at 12:00 UTC.

Following the initial recovery, Soracom collaborated with the network partner to address communication restrictions imposed by certain carriers in response to the earlier traffic spike. While most regions recovered shortly thereafter, recovery in Australia took longer, with full service restoration confirmed at 2025-03-27 04:00 UTC.

Future Actions

Remediation of the routing configuration issue identified as the root cause of this incident has been completed.

To prevent similar issues in the future, the following actions are being taken:

• Review and validate the configurations of other routers to ensure proper traffic flow during degraded operation in maintenance scenarios.
• Improve our operational processes for dedicated line maintenance and configuration changes on routers.

Additionally, Soracom is reviewing detailed information on the state of the network partner’s infrastructure during the incident, as well as the sequence of actions taken. Based on these findings, we will collaborate with the provider to strengthen the reliability of connectivity between the network partner and Soracom.

We are also working to improve coordination with the network partner and enhance internal response systems. A review of internal and external communication processes is underway to help minimize impact duration in the event of future incidents, and we will continue to focus on maintaining and improving overall service quality.

The following service incident occurred on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards on plan(s):

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Issue(s) Identified]
SIMs from the affected plan(s) were unable to connect to the network.

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42
• Start of Recovery Trend: 2025-03-26 09:38
• End of Incident: 2025-03-26 12:00

We sincerely apologize for any inconvenience caused by the extended service disruption. If your device connectivity was affected, please see our Connectivity Troubleshooting page for quick steps to get reconnected.

We are currently investigating the incident in detail and will provide further updates on this page during the week of March 31st. Should you require additional assistance, please submit a support ticket through our support site:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
We have confirmed that the network partner is taking effective actions, resulting in a recovery trend. Issues related to connectivity to some local carriers continue. We will continue to monitor the situation.

[Next Update]
As soon as the situation changes or at 2025-03-26 12:00 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 10:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 09:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 08:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 07:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 06:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 05:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 04:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is currently working with the network partner.

[Next Update]
As soon as the situation changes or at 2025-03-26 03:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network. We are particularly aware of the impact on the following subscriptions:

• plan01s
• plan01s - Low Data Volume
• plan-NA1
• planX1

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is actively investigating the issue.

[Next Update]
As soon as the situation changes or at 2025-03-26 02:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is actively investigating the issue.

[Next Update]
As soon as the situation changes or at 2025-03-26 01:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards

[Issue(s) Identified]
Cannot connect to the network

[Incident Timeline]
Times are in UTC:

• Start of Incident: 2025-03-25 22:42 UTC

[Current Status]
Our engineering team is actively investigating the issue.

[Next Update]
As soon as the situation changes or at 2025-03-26 00:30 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.

We are currently experiencing a service incident on the Soracom platform:

[Affected Service(s)]
Soracom Air for Cellular IoT SIM cards on plan(s):

[Issue(s) Identified]
Cannot connect to the network

[Incident Timeline]
TBD

[Current Status]
Our engineering team is actively investigating the issue.

[Next Update]
As soon as the situation changes or at 2025-03-26 00:00 UTC

We apologize for any inconvenience this may cause. If you need assistance, please contact our support team:
https://developers.soracom.io/en/docs/account/support/

Thank you for your understanding and cooperation.