Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
May 2024 · 100.0%Jun · 99.93%Jul · 99.99%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
100% - uptime

DNS - Operational

100% - uptime
May 2024 · 100.0%Jun · 99.63%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Traffic - Operational

100% - uptime
May 2024 · 100.0%Jun · 99.63%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Central Station (station.railway.com) - Operational

100% - uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Notice history

Jul 2024

a single host encountered a hardware failure and is restarting, a small portion of deploys in us-west may be unavailable
  • Resolved
    Resolved

    All applications on the impacted host have resumed.

  • Monitoring
    Monitoring

    The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

  • Investigating
    Investigating
    We are currently investigating this incident.
We're investigating routing failures for deployments using static IPs for egress.
  • Resolved
    Resolved
    An update to our routing control plane incorrectly propagated a bad route for regional egress proxies, this led to the drop of static IP egress traffic from 21:28 UTC to approximately 22:00 UTC. Our static IP monitoring was configured globally in our largest region. We're adding regional static IP monitoring to catch this in staged rollouts (aka, much quicker).
  • Monitoring
    Monitoring
    We've reverted the routing config and static IPs traffic is flowing normally.
  • Identified
    Identified
    we've identified a configuration change in our network routing config that looks to have caused the problem. We're rolling out a fix globally.
  • Investigating
    Investigating
    We're investigating routing failures for deployments using static IPs for egress.
Elevated image pull times for GitHub Container Registry
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are experiencing elevated pull times for images hosted in GitHub Container Registry (GHCR). This appears to be related to a third-party incident on GHCR's CDN (https://www.fastlystatus.com/incident/376717).

We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
  • Resolved
    Resolved

    This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

    Additionally, we will be digging into the scheduling issue to prevent this in the future.

  • Monitoring
    Monitoring

    The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

  • Identified
    Identified

    We've begun a failover process to recover the impacted host.

  • Investigating
    Investigating
    We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
Some requests from v2 Proxy in US-West and Asia returning incorrectly encoded responses
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Impacted regions have been rolled back and responses should be correct now.

  • Identified
    Identified

    The recent change is reverting.

Jun 2024

New volume provisioning in us-west region might be delayed
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

    We are adding capacity to the us-west region.

  • Investigating
    Investigating
    We are currently investigating this incident.
Dashboard loading slowly for some users
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Dashboard latency has recovered.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating
    We are currently investigating this incident.
Increased error rates for pulls from dockerhub
  • Resolved
    Resolved

    We've seen error rates drop since 22:30 UTC and docker have moved their incident status to monitoring. Resolving.

  • Identified
    Identified

    This has been confirmed as an upstream outage on DockerHub

  • Investigating
    Investigating

    We're seeing elevated errors for image pulls from dockerhub. Any deploys referencing dockerhub images or builds relying on dockerhub images as their base may fail or be delayed.

Traffic routed over US East to Railway workloads are currently down
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Traffic is now back online. All impacted users should have their services restored. We are seeing high load on our US East proxy and are currently adding further mitigations to prevent this from happening again.

  • Identified
    Identified

    We've isolated this incident to workloads on our new Edge Proxy only.

  • Investigating
    Investigating

    We are currently investigating this incident. Any users connecting to Railway-hosted applications going through our US East edge network is impacted.

Slightly delayed builds. We’re looking into it
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Builds are becoming fully operational again.

  • Investigating
    Investigating
    Slightly delayed builds. We’re looking into it

May 2024

we're investigating the failure of a single host in the us-west1 region
  • Resolved
    Resolved

    All workloads on the affected host have been fully restored.

  • Monitoring
    Monitoring

    All workloads on the affected host have been restored. We're continuing to monitor.

  • Identified
    Identified

    We're continuing to recover the failed host; the replacement is provisioning.

  • Investigating
    Investigating

    A single host in the us-west1 region has encountered a hardware failure. We're investigating and recovering the host.

Dashboard may be slow or failing to load for some users
  • Resolved
    Resolved

    This incident has been resolved. Our database was temporarily unreachable from a subset of our API backend hosts, resulting in slower load times and errors for some users. We have performed a rolling restart of those hosts and are seeing full recovery.

  • Monitoring
    Monitoring

    We have implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating
    We are currently investigating this incident.
Deployments are failing to go out in the Europe region
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    Deployments are failing to go out in the Europe region
Builds in us-west are stuck initializing
  • Resolved
    Resolved

    The incident has been resolved. Deploys are building normally now.

  • Monitoring
    Monitoring

    Some of our builders are back online and deployments are slowly progressing

  • Update
    Update

    We are still affected by the GCP outage and are monitoring it closely. https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#wVCaQTRV9VEch1ZTHVQn

  • Identified
    Identified

    GCP Virtual Private Cloud (VPC) is currently having an incident that is affecting our builders (https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#wVCaQTRV9VEch1ZTHVQn). We are working on mitigations

  • Investigating
    Investigating
    We are currently investigating this incident.
Builds may be delayed in EU West
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have identified and implemented a fix for this issue, and are continuing to monitor the results. Our builders in EU West was under-provisioned leading to slower builds. We have scaled it up, and builds should now be going through normally.

May 2024 to Jul 2024

Next