Railway - Elevated Packet Loss on Global Edge Network – Incident details

Elevated Packet Loss on Global Edge Network

ResolvedDegraded performance

Affected

Edge Network (Railway Metal)

Operational from 7:44 AM to 7:44 AM

US West (Metal / California, USA)

Operational from 7:44 AM to 7:44 AM

US East (Metal / Virginia, USA)

Operational from 7:44 AM to 7:44 AM

EU West (Metal / Amsterdam, Netherlands)

Operational from 7:44 AM to 7:44 AM

Southeast Asia (Metal / Singapore)

Operational from 7:44 AM to 7:44 AM

TCP Proxy (Metal)

Operational from 7:44 AM to 7:44 AM

  • ResolvedUTC
    Due to an erroneous PR that got past health checks, there was an elevated rate of dropped packets on the global edge network for about 20 minutes from 07:45 UTC to 08:05 UTC. Even though we stopped the rollout immediately after observing this failure, we estimate that about 10% of packets were dropped worldwide for this period.