Railway - Notice history

The host has restarted and workloads should be coming back online. We are still monitoring this incident.

Our deployment pipeline throughput was impacted by a resource constraint on the database backing our Temporal cluster. We have increased cluster resources and upgraded the database; we have seen deployment throughput restored.

Queue backpresssure has significantly reduced, we have re-enabled non-pro deployments and are monitoring

Deployments are now processing as we're reducing the queue backpressure. We have halted some background operations to relieve deployment delays.

We continue to investigate the issue and have halted some active background operations to relieve the deployment delays

This incident has been resolved. All workloads have been successfully restored.

We are continuing to monitor, but have seen recovery.

We're seeing recovery across the fleet. We are continuing to monitor this incident.

Users may experience higher read/write latency on Railway Volumes. We are investigating this incident.

All volumes should be seeing recovery back to nominal latency. We are continuing to monitor the fleet latency.

Most volumes have recovered to nominal latency. We are continuing to investigate any remaining ones that have not recovered yet.

Some hosts are showing signs of recovery. We are continuing to investigate.

We have shipped a remedy for network discovery. We have re-enabled deployments. We implemented a fix and are currently monitoring the result.

During a routine update to BGP configuration for Railway's Anycast network 66.33.22.0/23, a small proportion (approximately 8%) of upstream ISPs failed to propagate a routing update - this led to a small amount of traffic to be dropped for a period less than 5 mins.

Once routes propagated, some ISPs were routed in a suboptimal manner leading to elevated latency for some users. We have reverted the routing configuration and are liaising with our vendors to root cause the issue. We will also be creating maintenance windows for future routing config changes and staging their rollout to minimize risk.

We are continuing to work on a fix for this incident.

This incident has been resolved.

We are seeing recovery and continuing to monitor the situation.

Attack targeting EU edge identified. Mitigation efforts underway.