BSS Application Service Disruption Due to Azure Load Balancer Routing Failure

Τhe BSS application on US region experienced a service disruption when the Azure Load Balancer failed to route incoming client traffic to the backend application servers. Initial investigation confirmed that the backend infrastructure and application services remained operational, indicating that the issue is isolated to the load balancing layer.

Based on the evidence collected, the incident appears to be caused by an issue within the Microsoft Azure managed infrastructure. A support case has been opened with Microsoft Azure, and the issue remains under investigation.

To minimize customer impact while awaiting a permanent resolution, traffic has been redirected using Azure Traffic Manager to bypass the affected load balancer and route requests directly to the application servers.

Impact

• BSS application accessibility was disrupted from July 23, 2026 14:49 UTC until July 23, 2026 16:25 UTC (1 hour, 35 minutes)
• Client requests were not forwarded from the Azure Load Balancer to the backend application servers.
• Users experienced service unavailability until the mitigation was implemented.
• Business operations relying on the BSS application were affected during the incident.

Root Cause

The preliminary investigation indicates that the Azure Load Balancer failed to forward incoming traffic to healthy backend servers despite the backend infrastructure remaining operational.

At this stage, the root cause is believed to reside within the Microsoft Azure managed networking or load balancing service. The final root cause will be confirmed upon completion of Microsoft’s investigation.

Mitigation

To restore service as quickly as possible, the engineering team implemented the following workaround:

• Verified the health of all backend application servers.
• Reconfigured Azure Traffic Manager to bypass the affected Azure Load Balancer.
• Routed client traffic directly to the backend application servers.
• Continued monitoring application availability and performance following the routing change.

This mitigation successfully restored service while reducing dependency on the affected load balancing component.

Corrective Actions

Completed

• Confirmed backend server health.
• Implemented routing through Azure Traffic Manager to bypass the affected load balancer.
• Opened and escalated a support case with Microsoft Azure.
• Continued monitoring of application health and traffic.

Pending

• Receive root cause analysis from Microsoft Azure.
• Implement Microsoft’s recommended corrective actions once available.
• Restore the standard traffic path through the Azure Load Balancer after the issue has been resolved and validated.
• Review the incident and identify any additional resilience improvements to reduce the impact of similar infrastructure failures in the future.

Current Status

The BSS application is currently operational using the temporary routing configuration via Azure Traffic Manager. Microsoft Azure Support is actively investigating the underlying load balancer issue. The temporary workaround will remain in place until Microsoft confirms that the issue has been resolved and the load balancer has been validated for normal operation.

Service is back up.

According to our monitoring system this service has become unresponsive, we’re investigating.