We are in the process of rolling out a mitigation for this incident.

We noticed a host become unresponsive to network pings

New builds are taking longer to process than usual

This is a retrospective incident: Healthchecks on user workloads were failing for the period between 7:10 and 7:40am UTC due to the implementation of an overrestrictive firewall rule. This caused new deployments on services with a healthcheck configured to fail due to a failed healthcheck. We reverted the rule, and healthchecks began operating normally.

This is a retrospective incident: A single host in the asia-southeast1 region was unavailable between 11:40 and 11:59am UTC due to host failure. We automatically failed-over to a new host and all workloads were brought back online.

Between 12:09 and 12:19 UTC our edge network processed a large amount of traffic. Requests for your application may have not completed.

At 7:45am UTC, a single host in asian-southeast1 encountered a hardware failure. An automatic failover was begun and all affected workloads were restored by 8am UTC. We're running final checks to ensure the everything is operational.

A failure in our internal incident tooling meant this incident wasn't posted to instatus until now.