Deploys are taking longer than usual to go out

Resolved
Degraded performance
Started 9 months ago Lasted about 3 hours

Affected

Builds & Deployments
Deployments
Updates
  • Resolved
    Resolved

    We've fixed the issues that caused the recent deployment outages.

    On Friday we changed the way we interpolate environment variables and added a custom parser to allow for more complex behaviour. A bug in this parser caused it to occasionally lock up the workers that handle deployments - and this meant that we exhausted our deployment execution slots and slowed down builds. We have temporarily disabled several features (cron jobs and variable functions) while we were isolating the issue. We expect to re-enable these features early in the week. We are sorry for any inconvenience that these incidents caused.

  • Monitoring
    Monitoring

    Deployments are back to normal and we're continuing to monitor

  • Investigating
    Investigating

    Deploys are taking longer than usual to go out