Railway - Disabled Builds for Hobby – Incident details

All systems operational

Disabled Builds for Hobby

Resolved
Operational
Started about 2 years agoLasted about 10 hours

Affected

Builds (Legacy)

Degraded performance from 9:42 PM to 9:48 PM, Operational from 9:42 PM to 2:20 AM, Degraded performance from 2:20 AM to 7:17 AM

Build Machines (GCP)

Operational from 9:42 PM to 2:20 AM, Degraded performance from 2:20 AM to 7:17 AM

Image Registry (GCP)

Degraded performance from 9:42 PM to 9:48 PM, Operational from 9:48 PM to 2:20 AM, Degraded performance from 2:20 AM to 7:17 AM

External / Third-Party Integrations

Operational from 9:42 PM to 7:17 AM

GitHub

Operational from 9:42 PM to 7:17 AM

Registry - npm

Operational from 9:42 PM to 7:17 AM

Updates
  • Update
    UTC
    Update
    This incident has been resolved.
  • Resolved
    UTC
    Resolved

    Turns out we never really disabled hobby workloads. Anonymous provisions remain disabled

  • Update
    UTC
    Update

    We have found a few hot code paths that have contributed to past instability. Hobby builds remain disabled at this time as the team implements a long term fix.

  • Update
    UTC
    Update

    We have restored staged changes to all users. Hobby builds remain disabled while we implement a core fix.

  • Identified
    UTC
    Identified

    We have disabled builds from Hobby users once again to allow Pro organization builds to go through. While doing so, we are also going to disable the staged changes workflow to assist us in our investigation.

  • Monitoring
    UTC
    Monitoring

    We believe we've resolved the issue. Re-enabling hobby and monitoring

  • Investigating
    UTC
    Investigating

    We've seen a runup on hobby builds failing to push images and have disabled them. We're investigating