IMPACTED DATACENTER LOCATIONS: LHR2

TIME: December 4th, 2023 15:30 – 20:37 UTC

ISSUE SUMMARY: On December 4th, 2023, support was alerted that several customers experienced difficulty connecting to critical systems within their infrastructure. During the investigation process, the team was able to identify the impact as connection failures between the LHR2 Point of Presence and services in the London region of several cloud providers.

The connection failures appear to coincide with a routine maintenance window of the ISP Orange. The maintenance conducted was not expected to impact any services or systems, however, during the process the network path between several cloud providers was impacted resulting in degraded connectivity.

The Site Reliability team was able to work with the upstream provider to provide steps of how to identify and remediate the issue. The provider was able to execute the necessary steps to divert traffic away from the critical failure points which resulted in a complete remediation of the problems experienced. The provider is addressing all problematic devices to prevent any issues in the future.

ADDITIONAL INFORMATION: The team is taking steps to proactively identify these edge case upstream network issues in the future to prevent a recurrence.