Railway - Notice history

This incident has been resolved. We isolated the issue down to a Nixpacks version upgrade that caused a regression in image handling on our builders. We have reverted the Nixpacks version upgrade and are seeing nominal number of successful deploys.

A subset of deploys going through our image registry is failing. We've identified the issue and are rolling out a fix.

All proxies are back up in all regions. We are still monitoring as everything is coming back up.

Europe and US-East proxies are back up

Southeast Asia (Singapore) is back up

Most of the traffic appears to be back up, but we're still monitoring.

Our fix is rolling out and we are seeing recovery for some customers.

All applications on the impacted host have resumed.

The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

Additionally, we will be digging into the scheduling issue to prevent this in the future.

The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

We've begun a failover process to recover the impacted host.

Impacted regions have been rolled back and responses should be correct now.