Railway - Elevated deployment latency – Incident details

Elevated deployment latency

ResolvedPartial outage1 hour 29 minutes

Affected

Deployments (Railway Metal)

Partial outage from 4:26 PM to 5:55 PM

US West (Metal / California, USA)

Partial outage from 4:26 PM to 5:55 PM

US East (Metal / Virginia, USA)

Partial outage from 4:26 PM to 5:55 PM

EU West (Metal / Amsterdam, Netherlands)

Partial outage from 4:26 PM to 5:55 PM

Southeast Asia (Metal / SIngapore)

Partial outage from 4:26 PM to 5:55 PM

  • ResolvedUTC

    Our deployment pipeline throughput was impacted by a resource constraint on the database backing our Temporal cluster. We have increased cluster resources and upgraded the database; we have seen deployment throughput restored.

  • MonitoringUTC

    Queue backpresssure has significantly reduced, we have re-enabled non-pro deployments and are monitoring

  • IdentifiedUTC

    Deployments are now processing as we're reducing the queue backpressure. We have halted some background operations to relieve deployment delays.

  • UpdateUTC

    We continue to investigate the issue and have halted some active background operations to relieve the deployment delays

  • InvestigatingUTC

    We are investigating a number of deploys that are stuck initializing. Non-Pro deployments have been temporarily paused.