Railway - Notice history

All systems operational

Dashboard - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

Builders - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.97%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Image Registry - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 99.98%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Deploy - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Third-Party: npm Registry - Operational

Third-Party: Github Integrations - Operational

Third Party: Docker Hub - Operational

Third-Party: GitHub Container Registry - Operational

100% - uptime

EU West (europe-west4 / Amsterdam, Netherlands) - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

US East (us-east4 / Virginia, USA) - Operational

100% - uptime
Dec 2024 · 99.95%Jan 2025 · 99.96%Feb · 99.99%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

US West (us-west1 / Oregon, USA) - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 99.98%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Southeast Asia (asia-southeast1 / Singapore) - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

US West (us-west1 / Oregon, USA) - Operational

100% - uptime
Dec 2024 · 99.95%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

US East (us-east4 / Virginia, USA) - Operational

100% - uptime
Dec 2024 · 99.95%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

EU West (europe-west4 / Amsterdam, Netherlands) - Operational

100% - uptime
Dec 2024 · 99.95%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Southeast Asia (asia-southeast1 / Singapore) - Operational

100% - uptime
Dec 2024 · 99.95%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

TCP Proxy - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

DNS - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Traffic - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

Logs - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Metrics - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 99.99%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025
100% - uptime

Payments - Operational

100% - uptime
Dec 2024 · 100.0%Jan 2025 · 100.0%Feb · 100.0%

Previous page

Next page

Dec 2024
Jan 2025
Feb 2025

Notice history

Feb 2025

Railway.com dashboard is down
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
A single Google Cloud host in US West was rebooted
  • Resolved
    Resolved
    All workloads are now back online
  • Investigating
    Investigating
    A single Google Cloud host in US West was rebooted
TCP Proxy Upgrades
  • Completed
    February 15, 2025 at 2:00 AM
    Completed
    February 15, 2025 at 2:00 AM
    Maintenance has completed successfully
  • In progress
    February 14, 2025 at 11:00 PM
    In progress
    February 14, 2025 at 11:00 PM
    Maintenance is now in progress
  • Update
    February 14, 2025 at 11:00 PM
    Planned
    February 14, 2025 at 11:00 PM

    We will be provisioning more domains for TCP Proxy allocations. During this time the new domain may not proxy traffic while the global TCP Proxy system picks up these changes. We expect this window to only be a few minutes.

    All existing TCP Proxy domains will continue to work. Connections may be closed due to reloads though and reconnecting will work. This is expected behaviour whenever we have pushed updates to this service in the past.

    These changes will allow us to live reload more configs in the future, minimizing the needs for "hard reloads" which may drop connections. We recommend people use the private network when possible, for which this is a non-issue.

  • Planned
    February 14, 2025 at 9:44 PM
    Planned
    February 14, 2025 at 9:44 PM

    A note; this will only, potentially, affect services which are dialing a Railway database from OUTSIDE Railway, via the TCP proxy (viaduct.rlwy.net, monorail.rwly.net, etc)

    Private network connections (myservice.railway.internal) will be unaffected

A single Google Cloud host in US West has restarted
  • Resolved
    Resolved
    All workloads are now back online
  • Monitoring
    Monitoring
    We are monitoring as workloads come back up
  • Investigating
    Investigating
    A single Google Cloud host in US West has restarted
A Google cloud host has restarted
  • Resolved
    Resolved
    Resolve
  • Monitoring
    Monitoring
    Failover complete
  • Investigating
    Investigating
    A Google cloud host has restarted

Jan 2025

We're investigating connection failures between the GCP and Metal interconnect in the us-east4 region
  • Resolved
    Resolved

    This incident has been resolved. We will be spinning up a secondary BGP Anycast route on metal to provide full dual redundancy, regardless of Google's failures.

  • Monitoring
    Monitoring
    Google Cloud failed over the interconnect, resulting in the primary failing over. The secondary did not have the correct config to propagate and as a result blackholed traffic for Hobby users into Metal US East. We have modified the config and are seeing recovery.
  • Update
    Update

    Redeploying in any other region will resolve the issue for affected users

  • Identified
    Identified

    Metal interconnect seems to have been taken down by Google Cloud. This incident affects any edge traffic to metal instances in us-east

  • Investigating
    Investigating

    We are currently investigating this incident

Elevated Latency on Europe West and APAC Edge Proxy
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    We have mitigated a DDoS attack between 1926 UTC ~ 1936 UTC. We are seeing full recovery and continuing to monitor the situation. This impacted only our Europe West and APAC edge proxies. Users connecting to Railway-hosted workloads from Europe and APAC may have experienced elevated latency during this time.

Intermittent Connection Issues for APAC Workloads
  • Resolved
    Resolved

    We have seen a full recovery, all connections are succeeding.

  • Monitoring
    Monitoring

    We are seeing connections recover, we are going to be monitoring closely.

  • Investigating
    Investigating

    We are experiencing elevated error rates and increased latency in our Asia region proxies. Users connecting to Railway-hosted workloads from Asia may experience some difficulties. We are investigating this incident.

  • Update
    Update
    This incident has been resolved.
  • Resolved
    Resolved

    All of our proxies in Asia have fully recovered, workload traffic is now nominal.

Builds are failing to deploy
  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating
    We are currently investigating this incident.
A single host in US West has restarted
  • Resolved
    Resolved

    All workloads on the host are now back online

  • Monitoring
    Monitoring

    We are actively monitoring workloads on this host as they come back online

  • Investigating
    Investigating
    We are currently investigating this incident.

Dec 2024

A single host in US East is unresponsive due to load on the box.
  • Resolved
    Resolved

    All applications on the affected host have been restored.

  • Identified
    Identified

    Host is being recovered by the team.

  • Investigating
    Investigating
    We are currently investigating this incident.
Services Outage
  • Resolved
    Resolved

    This incident has been resolved. Railway was down for a total of ~10 minutes. We will publish a full retroactive on this incident shortly.

  • Monitoring
    Monitoring

    Services should be fully restored and we'll continue to monitor.

  • Identified
    Identified

    Issue has been identified.

  • Investigating
    Investigating
    We are currently investigating this incident.
Dashboard Maintenance
  • Completed
    December 16, 2024 at 5:30 AM
    Completed
    December 16, 2024 at 5:30 AM
    Maintenance has completed successfully
  • In progress
    December 16, 2024 at 5:00 AM
    In progress
    December 16, 2024 at 5:00 AM
    Maintenance is now in progress
  • Planned
    December 16, 2024 at 5:00 AM
    Planned
    December 16, 2024 at 5:00 AM

    We will be upgrading our high availability database on Sunday at midnight EST. We do not expect any interruption of service. While we do not expect it, during this time there may be dashboard "blips" and potentially delayed builds.

Some builds relying on GitHub fail with connection error
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
Builds using NPM will fail during NPM’s scheduled maintenance.
  • Resolved
    Resolved
    NPM's maintenence window is completed.
  • Investigating
    Investigating
    Builds using NPM will fail during NPM’s scheduled maintenance.

Dec 2024 to Feb 2025

Next