Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Apr 2025 · 99.86%May · 99.99%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025
100% - uptime

DNS - Operational

100% - uptime
Apr 2025 · 99.93%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

Traffic - Operational

100% - uptime
Apr 2025 · 99.93%May · 100.0%Jun · 100.0%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

Central Station (station.railway.com) - Operational

100% - uptime
Apr 2025 · 100.0%May · 100.0%Jun · 99.97%

Previous page

Next page

Apr 2025
May 2025
Jun 2025

Notice history

Jun 2025

Slow builds
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Builds have been delayed over the last ~30 minutes. We're seeing recovery and are monitoring the incident.

Degraded read/write performance on Railway Volumes
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are continuing to monitor, but have seen recovery.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We're seeing recovery across the fleet. We are continuing to monitor this incident.

  • Update
    Update

    Users may experience higher read/write latency on Railway Volumes. We are investigating this incident.

  • Investigating
    Investigating
    We are currently investigating this incident.
Degraded read/write performance on Railway Volumes
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    All volumes should be seeing recovery back to nominal latency. We are continuing to monitor the fleet latency.

  • Update
    Update

    Most volumes have recovered to nominal latency. We are continuing to investigate any remaining ones that have not recovered yet.

  • Update
    Update

    Some hosts are showing signs of recovery. We are continuing to investigate.

  • Investigating
    Investigating
    Users may experience higher read/write latency on Railway Volumes. We are investigating this incident.
Elevated Packet Loss on Global Edge Network
  • Resolved
    Resolved
    Due to an erroneous PR that got past health checks, there was an elevated rate of dropped packets on the global edge network for about 20 minutes from 07:45 UTC to 08:05 UTC. Even though we stopped the rollout immediately after observing this failure, we estimate that about 10% of packets were dropped worldwide for this period.
GitHub Access is Down
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We have found that GitHub access is down. Deployments with GH as a source will not deploy. We are continuing to work on a fix for this incident.

May 2025

we're temporarily pausing deployments while we investgate delays in network propagation
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have shipped a remedy for network discovery. We have re-enabled deployments. We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating
    we're temporarily pausing deployments while we investgate delays in network propagation
Elevated latency on Edge Network
  • Resolved
    Resolved

    During a routine update to BGP configuration for Railway's Anycast network 66.33.22.0/23, a small proportion (approximately 8%) of upstream ISPs failed to propagate a routing update - this led to a small amount of traffic to be dropped for a period less than 5 mins.

    Once routes propagated, some ISPs were routed in a suboptimal manner leading to elevated latency for some users. We have reverted the routing configuration and are liaising with our vendors to root cause the issue. We will also be creating maintenance windows for future routing config changes and staging their rollout to minimize risk.

  • Monitoring
    Monitoring

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    As we're rolling out a routing reconfiguration of our metal edge, a small subset of users may notice latency issues while the new routing configuration is being propagated to all ISPs.

GCP edge network connectivity issues in the EU region
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are seeing recovery and continuing to monitor the situation.

  • Identified
    Identified

    Attack targeting EU edge identified. Mitigation efforts underway.

  • Investigating
    Investigating
    We are currently investigating this incident.
[Retroactive] Magic Login Outage
  • Resolved
    Resolved

    (This is a retroactive incident) magic login links failed to go out since this morning

Small % of Project 404's
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We rolled out a change where we dropped a column unsafely. While the new deploy was rolling out (~5 minutes), old deploys, the ORM we used automatically queried the old field. As a result, some small % of users who hard refreshed (and had to load project state) may have seen a 404 briefly.

    We're rolling out a lint rule to catch this in the future. Apologies

Apr 2025

Elevated latency US East edge proxies
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We are seeing full recovery, and we are continuing to monitor.

  • Investigating
    Investigating
    We are currently investigating this incident.
Dashboard is experiencing degraded performance. You may notice occassional latency spikes or 503s
  • Resolved
    Resolved

    This incident has been resolved. A full incident report is available at https://blog.railway.com/p/incident-report-april-30-2025.

  • Update
    Update

    We have implemented a fix and are monitoring the results.

  • Monitoring
    Monitoring

    We have implemented a fix and are monitoring the results.

  • Identified
    Identified

    We are continuing to work on a fix for this Issue.

  • Investigating
    Investigating

    The dashboard is experiencing degraded performance. You may notice occasional latency spikes or 503s.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have implemented a fix and are monitoring the results.

  • Update
    Update

    We are continuing to work on a fix for this Issue.

  • Update
    Update

    The Dashboard is currently inaccessible. We are continuing to work on a fix for the issue.

  • Update
    Update

    We are continuing to work on a fix for the issue.

  • Update
    Update

    We are continuing to work on a fix for the issue.

  • Update
    Update

    We are continuing to work on a fix for the issue.

  • Update
    Update
    Dashboard may be inaccessible for some users
  • Identified
    Identified

    We have identified the issue and are working on a fix

  • Investigating
    Investigating
    Dashboard is experiencing degraded performance. You may notice occassional latency spikes or 503s
US-East delayed placements
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We have fixed a bug preventing new deployments from landing on the newly provisioned hosts.

  • Investigating
    Investigating

    We are continuing to investigate a subset of deploys that are delayed in the creating container step.

  • Monitoring
    Monitoring

    We have finished adding capacity and are monitoring the process of deployments that were stuck in the queue.

  • Identified
    Identified

    We are scaling up capacity in US-East

A single host on Railway GCP US-West has rebooted
  • Resolved
    Resolved

    All applications are operational. A host was rebooted at 17:48 UTC. Most applications recovered by 18:06 UTC and ALL applications recovered by 18:13 UTC.

  • Monitoring
    Monitoring

    Most applications are operational. We're monitoring for ALL applications to become operational before resolving.

  • Identified
    Identified

    Affected deploys are coming back online. Impacted users may see their services re-deployed by Railway.

  • Investigating
    Investigating

    We are failing over deploys from the host to a different host. Impacted users may see their services re-deployed by Railway.

A single host on Railway GCP US-West has rebooted
  • Resolved
    Resolved

    All applications are operational. A host was rebooted at 13:53 UTC. Most applications recovered by 14:03 UTC and ALL applications recovered by 14:10 UTC.

  • Identified
    Identified

    Affected deploys are coming back online. Impacted users may see their services re-deployed by Railway.

  • Investigating
    Investigating

    We are failing over deploys from the host to a different host. Impacted users may see their services re-deployed by Railway.

Apr 2025 to Jun 2025

Next