Railway - Notice history

This incident has been resolved.

The affected host is back up and running. All deploys have been restarted.

deployments on the affected hosts are starting to come back up, we expect full resolution within the next 10 minutes.

This incident is now resolved. Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across our edge proxies in us-east and asia-southeast1 due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests originating from these regions failed / experienced higher latency during this time.

Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across all our edge proxies due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests failed / had higher latency during this time. Traffic has now recovered, and we're continuing to monitor.

This incident has been resolved.

The issue seems to have been abated. During our discovery, we saw timeouts to GitHub and a few other external services. Given we don't use any egress products other than Cloud NAT from Google, we've escalated this to Google and are awaiting a response

Moving back to monitoring as we've observing it again

We've identified the issue as a bottleneck in our service mesh under heavier-than-average load which causes higher-than-normal latency.

Recovered. Initial response appears to be a flood of deployments with lots of variables put temporary lag on the database, which propagated to the dashboard. We will monitor as we continue to dig in

The issue has been resolved.

The issue has been resolved.

We have pushed a fix and are now monitoring.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

This incident has been resolved.

The machine is currently back online and serving workloads once again. The Railway team will monitor this machine as workloads fully restore. We appreciate your patience.

The team is currently restarting the affected machine and bringing on workloads back in a staged manner, some workloads may continue to be unavailable.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

We implemented a fix and are currently monitoring the result.

We are currently investigating this incident.

This incident has been resolved.

We had a spike in deploy requests and a small delay for some deploys as they got backlogged.

We implemented a fix and are currently monitoring the result.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

This incident has been resolved. If you run into "Cannot create code snapshot" error, please issue a new build.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

We are in the process of rolling out a fix. It will be live in a few minutes.

When we cycled our regional builder fleet, the machine image they booted with had some outdated config. This had the unintended consequence of pushing all images from regional builders to the us-west1 registry which in turn caused deployments to any other region to fail.

To ensure this doesn't happen again, we have removed the outdated config from our machine images and will be making the config path safer.

We implemented a fix and are currently monitoring the result.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.