Railway - Elevated Packet Loss on Global Edge Network – Incident details

All systems operational

Elevated Packet Loss on Global Edge Network

Resolved
Degraded performance
Started 6 days agoLasted less than a minute

Affected

Edge Network (Railway Metal)

Operational from 7:44 AM to 7:44 AM

US West (Metal / California, USA)

Operational from 7:44 AM to 7:44 AM

US East (Metal / Virginia, USA)

Operational from 7:44 AM to 7:44 AM

EU West (Metal / Amsterdam, Netherlands)

Operational from 7:44 AM to 7:44 AM

Southeast Asia (Metal / Singapore)

Operational from 7:44 AM to 7:44 AM

TCP Proxy (Metal)

Operational from 7:44 AM to 7:44 AM

Updates
  • Resolved
    Resolved
    Due to an erroneous PR that got past health checks, there was an elevated rate of dropped packets on the global edge network for about 20 minutes from 07:45 UTC to 08:05 UTC. Even though we stopped the rollout immediately after observing this failure, we estimate that about 10% of packets were dropped worldwide for this period.