Railway - Notice history

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

We are currently investigating this incident.

This incident has been resolved.

This incident has been resolved. We've updated the incident title to match the full scope of the incident.

The rollback has been deployed and services are back in normal operation. DNS resolution for private networking was also impacted between 20:57:30 UTC and 21:05:00 UTC.

This incident has been resolved.

We identified a bottleneck in our log ingestion pipeline that delayed ingest by up to a minute. We've rolled out a fix and log should now be flowing as normal.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

The affected build node has been cycled. A critical daemon on the node crashed and did not correctly update the nodes healthcheck. Any work landing on that node was thus instantly failing. We're taking measures to fix the healthcheck to ensure only healthy build nodes take on work.

We've identified the affected builder instance and are cycling it out.

We've pushed a fix and builds are going through fine

We encountered an issue with config propagation to our builders causing deploys to not turn live. After deploying an update internally to make this configuration availible- queued deploys are now going through. No action is required on your part.

An issue related to a builder configuration is the culprit here. As a result, builds aren't able to progress because it's looking for configuration that isn't fully rolled out. We are kicking the tyres and deploying a fix. Apologies!

We've fixed the issues that caused the recent deployment outages.

On Friday we changed the way we interpolate environment variables and added a custom parser to allow for more complex behaviour. A bug in this parser caused it to occasionally lock up the workers that handle deployments - and this meant that we exhausted our deployment execution slots and slowed down builds. We have temporarily disabled several features (cron jobs and variable functions) while we were isolating the issue. We expect to re-enable these features early in the week. We are sorry for any inconvenience that these incidents caused.

Deployments are back to normal and we're continuing to monitor

Deploys are going out fine now

deploys are now going out as normal

We're rolling out a fix, expect a resolution in the next 10 minutes.

Builds are building but failing to deploy. We are investigating an issue with GCP networking

Deployments are going out again. We are monitoring the build backlog

deploys should be working fine now