Railway - We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible. – Incident details

We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.

ResolvedPartial outage1 hour 2 minutes

Affected

Deployments (Legacy)

Partial outage from 9:05 PM to 10:08 PM

US East (GCP / Virginia, USA)

Partial outage from 9:05 PM to 10:08 PM

  • ResolvedUTC

    This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

    Additionally, we will be digging into the scheduling issue to prevent this in the future.

  • MonitoringUTC

    The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

  • IdentifiedUTC

    We've begun a failover process to recover the impacted host.

  • InvestigatingUTC
    We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.