Railway - Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered – Incident details

All systems operational

Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered

Resolved
Degraded performance
Started almost 2 years agoLasted 10 minutes

Affected

Builds (Legacy)

Degraded performance from 2:49 AM to 2:56 AM, Operational from 2:56 AM to 2:58 AM

Build Machines (GCP)

Degraded performance from 2:49 AM to 2:56 AM, Operational from 2:56 AM to 2:58 AM

Updates
  • Resolved
    UTC
    Resolved
    Builds are going out nominally.
  • Monitoring
    UTC
    Monitoring
    We have restored the build queue and are now monitoring the health of remaining builds.
  • Investigating
    UTC
    Investigating
    Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered