Railway - Notice history

We have traced the issue to a guard that was trying to prevent users to deploying volumes on Railway Metal. We have since applied the fix and systems are working normally.

All proxies are back up in all regions. We are still monitoring as everything is coming back up.

Europe and US-East proxies are back up

Southeast Asia (Singapore) is back up

Most of the traffic appears to be back up, but we're still monitoring.

Our fix is rolling out and we are seeing recovery for some customers.

All applications on the impacted host have resumed.

The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

Additionally, we will be digging into the scheduling issue to prevent this in the future.

The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

We've begun a failover process to recover the impacted host.

Impacted regions have been rolled back and responses should be correct now.