Railway - Notice history

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

The affected build node has been cycled. A critical daemon on the node crashed and did not correctly update the nodes healthcheck. Any work landing on that node was thus instantly failing. We're taking measures to fix the healthcheck to ensure only healthy build nodes take on work.

We've identified the affected builder instance and are cycling it out.

We've pushed a fix and builds are going through fine

We encountered an issue with config propagation to our builders causing deploys to not turn live. After deploying an update internally to make this configuration availible- queued deploys are now going through. No action is required on your part.

An issue related to a builder configuration is the culprit here. As a result, builds aren't able to progress because it's looking for configuration that isn't fully rolled out. We are kicking the tyres and deploying a fix. Apologies!

We've fixed the issues that caused the recent deployment outages.

On Friday we changed the way we interpolate environment variables and added a custom parser to allow for more complex behaviour. A bug in this parser caused it to occasionally lock up the workers that handle deployments - and this meant that we exhausted our deployment execution slots and slowed down builds. We have temporarily disabled several features (cron jobs and variable functions) while we were isolating the issue. We expect to re-enable these features early in the week. We are sorry for any inconvenience that these incidents caused.

Deployments are back to normal and we're continuing to monitor

Deploys are going out fine now

deploys are now going out as normal

We're rolling out a fix, expect a resolution in the next 10 minutes.

Builds are building but failing to deploy. We are investigating an issue with GCP networking

Deployments are going out again. We are monitoring the build backlog

deploys should be working fine now

Incident resolved.

Login with GitHub should now be restored

Incident resolved.

More info: during a instance upgrade, as a result of Terraform drift, DNS was not able to update automatically, resulting in packets being dropped for about 8 minutes. While this is tolerable on the Hobby/Dev plan, for teams, this isn't expected (nor okay) and we will be putting into place drift protection to mitigate this in the future

Issue resolved

Affected hosts are back online and we are monitoring

This incident has been resolved.

Deployments are fully operational

Builds and deploys should be working as normal

We've identified the issue and are rolling out a fix

The issue has been fixed.

A fix has been issued.