Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

99.99% uptime
Aug 2023 · 99.99%Sep · 99.99%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023
100.0% uptime

DNS - Operational

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Traffic - Operational

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Central Station (station.railway.com) - Operational

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Railway Storage Buckets - Operational

100.0% uptime
Aug 2023 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2023
Sep 2023
Oct 2023

Notice history

View current status

Oct 2023

Volume and database provisions are failing
ResolvedPartial outage16 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

[US West] A small percentage of builds is failing to go out
ResolvedPartial outage9 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved. If you run into "Cannot create code snapshot" error, please issue a new build.

  • Monitoring
    UTC
    Monitoring

    One of our build hosts in US West is down. A small percentage of builds routed to that builder may be failing with "Cannot create code snapshot" errors.

    We've taken the broken builder out of rotation and is currently monitoring the results. Please try issuing a new build.

Volumes are failing to be created in uswest-1
Resolved14 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Identified
    UTC
    Identified

    We are in the process of rolling out a fix. It will be live in a few minutes.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

A percentage of deployments are failing to go out
ResolvedDegraded performance50 minutes
  • Resolved
    UTC
    Resolved

    When we cycled our regional builder fleet, the machine image they booted with had some outdated config. This had the unintended consequence of pushing all images from regional builders to the us-west1 registry which in turn caused deployments to any other region to fail.

    To ensure this doesn't happen again, we have removed the outdated config from our machine images and will be making the config path safer.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

Railway up is returning 504
ResolvedPartial outage12 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

Sep 2023

Deployments are failing to go out
ResolvedPartial outage11 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Update
    UTC
    Update

    We are currently investigating this incident.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

Builds are slow to go out
ResolvedDegraded performance26 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

A small subset of requests ingested at our us-east edge failed with a 503 error
ResolvedPartial outage
  • Resolved
    UTC
    Resolved

    We experienced a high volume of connections from AWS directed at a single tenant via our us-east edge between 19:55 UTC and 20:20UTC. This traffic caused resource exhaustion on our proxies and led to a small subset of connections being terminated. We have mitigated this temporarily and are provisioning more proxy nodes to handle the load in us-east.

Delayed route propagation and failing Private Network DNS lookups
ResolvedPartial outage12 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We've updated the incident title to match the full scope of the incident.

  • Monitoring
    UTC
    Monitoring

    The rollback has been deployed and services are back in normal operation. DNS resolution for private networking was also impacted between 20:57:30 UTC and 21:05:00 UTC.

  • Investigating
    UTC
    Investigating

    An fault in our routing control plane may cause domain updates for some deployments to be delayed. A rollback is in progress.

We're investigating reports of delayed deploy and build logs
ResolvedDegraded performance37 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We identified a bottleneck in our log ingestion pipeline that delayed ingest by up to a minute. We've rolled out a fix and log should now be flowing as normal.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

Aug 2023

A percentage of builds are failing to go through
ResolvedDegraded performance41 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

GitHub-triggered deploys may be delayed for some users due to an upstream issue
ResolvedDegraded performance4 hours 17 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    GitHub is currently experiencing issues with their web hooks. Refer to https://www.githubstatus.com/ for more details.

Some builds may be failing to go out
ResolvedDegraded performance26 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

We're investigating reports of build failures on a subset of builder hosts
ResolvedPartial outage12 minutes
  • Resolved
    UTC
    Resolved

    The affected build node has been cycled. A critical daemon on the node crashed and did not correctly update the nodes healthcheck. Any work landing on that node was thus instantly failing. We're taking measures to fix the healthcheck to ensure only healthy build nodes take on work.

  • Identified
    UTC
    Identified

    We've identified the affected builder instance and are cycling it out.

  • Investigating
    UTC
    Investigating

    We're investigating reports of build failures on a subset of builder hosts

A percentage of users are seeing an issue with builds during snapshot creation
ResolvedDegraded performance37 minutes
  • Resolved
    UTC
    Resolved

    We've pushed a fix and builds are going through fine

  • Investigating
    UTC
    Investigating

    A percentage of users are seeing an issue with builds during snapshot creation

Previous

Aug 2023 to Oct 2023

Next