Railway - Elevated error rates for deployments with volumes on Southeast Asia (Singapore) Metal region – Incident details

All systems operational

Elevated error rates for deployments with volumes on Southeast Asia (Singapore) Metal region

Resolved
Degraded performance
Started 5 days agoLasted about 2 hours

Affected

Deployments

Degraded performance from 11:42 AM to 1:58 PM

Southeast Asia Metal (asia-southeast2 / SIngapore)

Degraded performance from 11:42 AM to 1:58 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    All workloads with a volume on the impacted host is back online. All other workloads have been successfully moved to a different host. We are actively monitoring the recovery. If your deployment failed due to “Container failed to start" error, please attempt a re-deploy.

  • Identified
    Identified

    We have identified a single degraded host on Southeast Asia (Singapore) Metal. We are currently restoring active deployments without a volume on the host to a different host. Workloads with a volume located on the host may be unavailable during this period. Impacted users will see a deployment initiated by Railway as part of this restoration process.

  • Investigating
    Investigating

    We are currently investigating elevated error rates of deployments in Southeast Asia (Singapore) Metal. A small subset of services with volumes may be failing to deploy with the error message "Container failed to start"