New Horizon Hosting LLC - Full network outage – Incident details

All systems operational

Full network outage

Resolved
Major outage
Started about 1 year agoLasted about 7 hours

Affected

Client/Billing Area

Major outage from 1:18 AM to 2:18 AM, Degraded performance from 2:18 AM to 4:00 AM, Major outage from 4:00 AM to 5:16 AM, Operational from 5:16 AM to 7:57 AM

Main Website

Major outage from 1:18 AM to 2:18 AM, Degraded performance from 2:18 AM to 4:00 AM, Major outage from 4:00 AM to 5:16 AM, Operational from 5:16 AM to 7:57 AM

Web

Major outage from 1:18 AM to 4:00 AM, Degraded performance from 2:18 AM to 4:00 AM, Major outage from 4:00 AM to 5:16 AM, Operational from 5:16 AM to 7:57 AM

US-CHI-W01 (Web Hosting)

Major outage from 1:18 AM to 5:16 AM, Operational from 5:16 AM to 7:57 AM

US-CHI-W02 (Web Builder)

Major outage from 1:18 AM to 5:16 AM, Operational from 5:16 AM to 7:57 AM

US-CHI-W03 (Web Radio Hosting)

Major outage from 1:18 AM to 2:18 AM, Degraded performance from 2:18 AM to 4:00 AM, Major outage from 4:00 AM to 5:16 AM, Operational from 5:16 AM to 7:57 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Update
    UTC
    Update

    We are continuing to investigate route instability isolated to our Chicago location. The issue is stemming from communication inconsistencies between our edge routers at the spine layer. This is resulting in frequent route flapping and intermittent disruptions in traffic flow.

    Our network engineering team is working to rebalance route propagation and mitigate further impact. We'll continue to provide additional updates as soon as possible.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident.

    However the following services seem to be online:

    • Budget Game

    • Gamepanel

    • Homepage

    • Billing panel

    • Web radio

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    We have identified the issue (dead router) and have replaced it and awaiting on things to head online again.

  • Investigating
    UTC
    Investigating

    We are currently investigating this issue regarding a full outage across all new horizon cloud llc services and platforms.