A percentage of deployments are failing to go out

Resolved
Degraded performance
Started 7 months ago Lasted about 1 hour

Affected

Deployments
Updates
  • Resolved
    Resolved

    When we cycled our regional builder fleet, the machine image they booted with had some outdated config. This had the unintended consequence of pushing all images from regional builders to the us-west1 registry which in turn caused deployments to any other region to fail.

    To ensure this doesn't happen again, we have removed the outdated config from our machine images and will be making the config path safer.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.