Railway - Deploys are taking longer than usual to go out – Incident details

Deploys are taking longer than usual to go out

Resolved
Degraded performance
Started about 1 year agoLasted about 3 hours

Affected

Deployments

Updates
  • Resolved
    Resolved

    We've fixed the issues that caused the recent deployment outages.

    On Friday we changed the way we interpolate environment variables and added a custom parser to allow for more complex behaviour. A bug in this parser caused it to occasionally lock up the workers that handle deployments - and this meant that we exhausted our deployment execution slots and slowed down builds. We have temporarily disabled several features (cron jobs and variable functions) while we were isolating the issue. We expect to re-enable these features early in the week. We are sorry for any inconvenience that these incidents caused.

  • Monitoring
    Monitoring

    Deployments are back to normal and we're continuing to monitor

  • Investigating
    Investigating

    Deploys are taking longer than usual to go out