Railway - Status

Recent notices

Resolved
July 25, 2024 at 10:31 PM
Resolved
July 25, 2024 at 10:31 PM
An update to our routing control plane incorrectly propagated a bad route for regional egress proxies, this led to the drop of static IP egress traffic from 21:28 UTC to approximately 22:00 UTC. Our static IP monitoring was configured globally in our largest region. We're adding regional static IP monitoring to catch this in staged rollouts (aka, much quicker).
Monitoring
July 25, 2024 at 10:12 PM
Monitoring
July 25, 2024 at 10:12 PM
We've reverted the routing config and static IPs traffic is flowing normally.
Identified
July 25, 2024 at 10:05 PM
Identified
July 25, 2024 at 10:05 PM
we've identified a configuration change in our network routing config that looks to have caused the problem. We're rolling out a fix globally.
Investigating
July 25, 2024 at 9:50 PM
Investigating
July 25, 2024 at 9:50 PM
We're investigating routing failures for deployments using static IPs for egress.

Resolved
July 23, 2024 at 10:06 PM
Resolved
July 23, 2024 at 10:06 PM
This incident has been resolved.
Monitoring
July 23, 2024 at 3:29 PM
Monitoring
July 23, 2024 at 3:29 PM
We are experiencing elevated pull times for images hosted in GitHub Container Registry (GHCR). This appears to be related to a third-party incident on GHCR's CDN (https://www.fastlystatus.com/incident/376717).

Resolved
July 22, 2024 at 10:08 PM
Resolved
July 22, 2024 at 10:08 PM
This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)
Additionally, we will be digging into the scheduling issue to prevent this in the future.
Monitoring
July 22, 2024 at 9:43 PM
Monitoring
July 22, 2024 at 9:43 PM
The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.
Identified
July 22, 2024 at 9:11 PM
Identified
July 22, 2024 at 9:11 PM
We've begun a failover process to recover the impacted host.
Investigating
July 22, 2024 at 9:05 PM
Investigating
July 22, 2024 at 9:05 PM
We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.