New Horizon Hosting LLC - Network Issues – Incident details

Network Issues

Resolved
Major outage
Started 3 months agoLasted about 6 hours

Affected

Client/Billing Area

Major outage from 9:46 AM to 9:48 AM, Degraded performance from 9:48 AM to 12:51 PM, Major outage from 12:51 PM to 3:28 PM

Main Website

Major outage from 9:46 AM to 9:48 AM, Degraded performance from 9:48 AM to 12:51 PM, Major outage from 12:51 PM to 3:28 PM

Web

Major outage from 9:46 AM to 9:48 AM, Degraded performance from 9:48 AM to 12:51 PM, Major outage from 12:51 PM to 3:28 PM

US-CHI-W01 (Web Hosting)

Major outage from 9:46 AM to 9:48 AM, Degraded performance from 9:48 AM to 12:51 PM, Major outage from 12:51 PM to 3:28 PM

US-CHI-W02 (Web Builder)

Major outage from 9:46 AM to 9:48 AM, Degraded performance from 9:48 AM to 12:51 PM, Major outage from 12:51 PM to 3:28 PM

US-CHI-W03 (Web Radio Hosting)

Major outage from 9:46 AM to 9:48 AM, Degraded performance from 9:48 AM to 12:51 PM, Major outage from 12:51 PM to 3:28 PM

Updates
  • Resolved
    Resolved

    At 12:54 UTC, all affected services reported healthy operational status. Our network engineering team identified the root cause as a fault within one of our on-net Tier 1 carrier connections, specifically with Cogent. While BGP sessions to this carrier remained stable throughout the event, traffic traversing this link experienced degradation due to a carrier-side fault. During the investigation, we discovered that certain remote DDoS protection tunnels were affected by asymmetric routing caused by recent configuration changes implemented prior to this event. These changes introduced scenarios where inbound and outbound traffic were not consistently using the same carrier, resulting in portions of egress traffic being routed over the impaired Cogent link. This amplified the service impact and prevented our normal failover mechanisms from fully mitigating the issue. We are deploying updated configurations to enforce strict path symmetry and improve failover behaviour, ensuring that future single-carrier faults are properly isolated. We have notified Cogent and are awaiting their root cause analysis. Additional updates will be provided as new information becomes available.

  • Update
    Update

    These issues are widespread through the data centre and are actively been investigated by the data centre team,

  • Update
    Update

    The incident seems to have self-resolved for now, however we will keep looking into this to find the root cause

  • Investigating
    Investigating
    We are currently investigating this incident.