New Horizon Hosting LLC - Network outage – Incident details

The cables have been replaced, and the network is traversing traffic as expected. We will continue to monitor logs over the next 24 hours for interface errors. As stated previously, we sincerely apologize for any inconvenience this incident has caused.

Our network team has concluded the investigation, and we have determined the root cause to be failing direct attached copper (DAC) cables between our spines and edge routers. Due to the severity of failing cables, the impact was felt more drastically throughout the network. As a result, we plan to replace the failing cables in the early morning of Wednesday, April 30th, at off-peak hours to minimize disruptions. We will be performing replacements of these DACs with backups from our storage at an off-peak time. This is scheduled to occur at 10:00 UTC on Wednesday, April 30th. We will provide an update when this is completed. We apologize for any inconvenience this incident has caused.

Our network team is investigating deeper, but we can confirm that one of our core spine switches flapped all LACP interfaces. Physical interfaces remained online.

The flapping of the LACP interfaces caused some connections between our top-of-rack switches and carriers to flap that were connected to said spine switch. More info will be posted soon.