Railway - A subset of our hosts are currently offline – Incident details

This incident has been resolved. We will be publishing a post-mortem of this incident after we conclude our investigations into the root cause.

Services for all regions with the exception of US West (us-west1 / Oregon, USA) have been restored. We are working on moving workloads off a single unhealthy host, and continuing to work on ensuring all impacted hosts that were restarted as a result of this incident are healthy.

All impacted hosts have been restarted, and we are currently working to ensure they are healthy. Services should be back online within 15 minutes of this update. Some services may encounter networking issues as the restart process completes.

Restarts have completed for most hosts. We are currently restarting the remaining hosts, and are working to ensure the restarted hosts are healthy.

Restarts for all impacted hosts are still in progress. We are seeing additional levels of service restoration and are working towards full service restoration.

The root cause of this incident is still under investigation.

We are seeing additional service restoration across Railway-hosted workloads. Hobby provisions have been re-enabled.

Restarts for all impacted hosts are currently in progress. We are seeing small amounts of service restoration at the moment, and will continue to work on full service restoration.

The root cause of this incident is still under investigation.

We are currently issuing a rolling restart of all impacted hosts. The root cause of the incident is still under investigation.

We are still investigating this incident. We have issued a restart on some hosts to attempt service recovery.

We are still investigating this incident.

We have isolated the majority of impact to the US West (us-west1 / Oregon, USA) region. Other regions are currently unstable for a subset of users. We're still investigating this incident.

We are still investigating this incident. Deployments for Hobby Plan users are temporarily disabled.