Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.86%May · 99.99%

Previous page

Next page

Mar 2025
Apr 2025
May 2025
100% - uptime

DNS - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.93%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Traffic - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 99.93%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Central Station (station.railway.com) - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Railway Storage Buckets - Operational

100% - uptime
Mar 2025 · 100.0%Apr · 100.0%May · 100.0%

Previous page

Next page

Mar 2025
Apr 2025
May 2025

Notice history

May 2025

we're temporarily pausing deployments while we investgate delays in network propagation
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We have shipped a remedy for network discovery. We have re-enabled deployments. We implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating
    we're temporarily pausing deployments while we investgate delays in network propagation
Elevated latency on Edge Network
  • Resolved
    UTC
    Resolved

    During a routine update to BGP configuration for Railway's Anycast network 66.33.22.0/23, a small proportion (approximately 8%) of upstream ISPs failed to propagate a routing update - this led to a small amount of traffic to be dropped for a period less than 5 mins.

    Once routes propagated, some ISPs were routed in a suboptimal manner leading to elevated latency for some users. We have reverted the routing configuration and are liaising with our vendors to root cause the issue. We will also be creating maintenance windows for future routing config changes and staging their rollout to minimize risk.

  • Monitoring
    UTC
    Monitoring

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    UTC
    Investigating

    As we're rolling out a routing reconfiguration of our metal edge, a small subset of users may notice latency issues while the new routing configuration is being propagated to all ISPs.

GCP edge network connectivity issues in the EU region
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We are seeing recovery and continuing to monitor the situation.

  • Identified
    UTC
    Identified

    Attack targeting EU edge identified. Mitigation efforts underway.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
[Retroactive] Magic Login Outage
  • Resolved
    UTC
    Resolved

    (This is a retroactive incident) magic login links failed to go out since this morning

Small % of Project 404's
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    We rolled out a change where we dropped a column unsafely. While the new deploy was rolling out (~5 minutes), old deploys, the ORM we used automatically queried the old field. As a result, some small % of users who hard refreshed (and had to load project state) may have seen a 404 briefly.

    We're rolling out a lint rule to catch this in the future. Apologies

Apr 2025

Elevated latency US East edge proxies
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We are seeing full recovery, and we are continuing to monitor.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Dashboard is experiencing degraded performance. You may notice occassional latency spikes or 503s
  • Resolved
    UTC
    Resolved

    This incident has been resolved. A full incident report is available at https://blog.railway.com/p/incident-report-april-30-2025.

  • Update
    UTC
    Update

    We have implemented a fix and are monitoring the results.

  • Monitoring
    UTC
    Monitoring

    We have implemented a fix and are monitoring the results.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this Issue.

  • Investigating
    UTC
    Investigating

    The dashboard is experiencing degraded performance. You may notice occasional latency spikes or 503s.

  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We have implemented a fix and are monitoring the results.

  • Update
    UTC
    Update

    We are continuing to work on a fix for this Issue.

  • Update
    UTC
    Update

    The Dashboard is currently inaccessible. We are continuing to work on a fix for the issue.

  • Update
    UTC
    Update

    We are continuing to work on a fix for the issue.

  • Update
    UTC
    Update

    We are continuing to work on a fix for the issue.

  • Update
    UTC
    Update

    We are continuing to work on a fix for the issue.

  • Update
    UTC
    Update
    Dashboard may be inaccessible for some users
  • Identified
    UTC
    Identified

    We have identified the issue and are working on a fix

  • Investigating
    UTC
    Investigating
    Dashboard is experiencing degraded performance. You may notice occassional latency spikes or 503s
US-East delayed placements
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We have fixed a bug preventing new deployments from landing on the newly provisioned hosts.

  • Investigating
    UTC
    Investigating

    We are continuing to investigate a subset of deploys that are delayed in the creating container step.

  • Monitoring
    UTC
    Monitoring

    We have finished adding capacity and are monitoring the process of deployments that were stuck in the queue.

  • Identified
    UTC
    Identified

    We are scaling up capacity in US-East

A single host on Railway GCP US-West has rebooted
  • Resolved
    UTC
    Resolved

    All applications are operational. A host was rebooted at 17:48 UTC. Most applications recovered by 18:06 UTC and ALL applications recovered by 18:13 UTC.

  • Monitoring
    UTC
    Monitoring

    Most applications are operational. We're monitoring for ALL applications to become operational before resolving.

  • Identified
    UTC
    Identified

    Affected deploys are coming back online. Impacted users may see their services re-deployed by Railway.

  • Investigating
    UTC
    Investigating

    We are failing over deploys from the host to a different host. Impacted users may see their services re-deployed by Railway.

A single host on Railway GCP US-West has rebooted
  • Resolved
    UTC
    Resolved

    All applications are operational. A host was rebooted at 13:53 UTC. Most applications recovered by 14:03 UTC and ALL applications recovered by 14:10 UTC.

  • Identified
    UTC
    Identified

    Affected deploys are coming back online. Impacted users may see their services re-deployed by Railway.

  • Investigating
    UTC
    Investigating

    We are failing over deploys from the host to a different host. Impacted users may see their services re-deployed by Railway.

Mar 2025

Some TCP proxy endpoints may be experiencing connectivity issues
  • Resolved
    UTC
    Resolved

    This incident has been resolved and we're seeing full recovery and have successfully rolled out the revert.

  • Identified
    UTC
    Identified

    We're reverting a change that has interrupted TCP proxy connectivity to a small subset of workloads still on Runtime V1. Workloads on Runtime V2 are unaffected.

  • Investigating
    UTC
    Investigating
    Some TCP proxy endpoints may be experiencing connectivity issues
Potential upstream issue causing delayed login via Magic emails. Login via GitHub may be used as a workaround.
  • Resolved
    UTC
    Resolved
    Login via Magic emails are being delivered timely
  • Investigating
    UTC
    Investigating
    Potential upstream issue causing delayed login via Magic emails. Login via GitHub may be used as a workaround.
A singular google cloud host is failing over
  • Resolved
    UTC
    Resolved
    Failover completed
  • Investigating
    UTC
    Investigating
    A singular google cloud host is failing over
A single host in US West was rebooted due to a kernel failure
  • Resolved
    UTC
    Resolved
    All workloads have now recovered
  • Monitoring
    UTC
    Monitoring
    We are monitoring as workloads come back online
  • Investigating
    UTC
    Investigating
    A single host in US West was rebooted due to a kernel failure
Potential upstream GitHub issue with "Failed to clone" for small subset of builds. We're looking into it
  • Resolved
    UTC
    Resolved
    We've deployed a permanent fix and consider the issue resolved.
  • Monitoring
    UTC
    Monitoring
    We have rolled out a temporary fix and are working on a permanent one.
  • Update
    UTC
    Update
    We continue to investigate the incident. It occurs on a very small % of builds and retries resolve
  • Investigating
    UTC
    Investigating
    Potential upstream GitHub issue with "Failed to clone" for small subset of builds. We're looking into it

Mar 2025 to May 2025

Next