Railway - Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered – Incident details

Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered

ResolvedDegraded performance9 minutes

Affected

Builds (Legacy)

Degraded performance from 2:49 AM to 2:56 AM, Operational from 2:56 AM to 2:58 AM

Build Machines (GCP)

Degraded performance from 2:49 AM to 2:56 AM, Operational from 2:56 AM to 2:58 AM

  • ResolvedUTC
    Builds are going out nominally.
  • MonitoringUTC
    We have restored the build queue and are now monitoring the health of remaining builds.
  • InvestigatingUTC
    Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered