All systems operational

Dashboard

99.80% uptime
Nov 2023 · 99.98%Dec · 100.0%Jan 2024 · 99.43%
Nov 202399.98% uptime
Dec 2023100.0% uptime
Jan 202499.43% uptime

Builds & Deployments

99.94% uptime

Builders

99.99% uptime
Nov 2023 · 99.97%Dec · 100.0%Jan 2024 · 100.0%
Nov 202399.97% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Image Registry

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Deployments

99.71% uptime
Nov 2023 · 99.93%Dec · 99.27%Jan 2024 · 99.94%
Nov 202399.93% uptime
Dec 202399.27% uptime
Jan 202499.94% uptime

Third-Party: npm Registry

Operational

Third-Party: Github Integrations

Operational

Observability

99.99% uptime

Logs

99.98% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 99.93%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 202499.93% uptime

Metrics

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Edge Network

100.0% uptime

US West (us-west1 / Oregon, USA)

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

US East (us-east4 / Virginia, USA)

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

EU West (europe-west4 / Amsterdam, Netherlands)

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Southeast Asia (asia-southeast1 / Singapore)

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

TCP Proxy

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Private Networking

100.0% uptime

DNS

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Traffic

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

External / Third-Party

100.0% uptime

Payments

100.0% uptime
Nov 2023 · 100.0%Dec · 100.0%Jan 2024 · 100.0%
Nov 2023100.0% uptime
Dec 2023100.0% uptime
Jan 2024100.0% uptime

Notice history

Jan 2024

Railway-hosted applications may be inaccessible for users connecting from Europe
  • Resolved
    Resolved
    Our mitigation efforts were successful and services have been restored for all users. We will publish a write-up of this incident shortly.
  • Monitoring
    Monitoring
    We have successfully rolled out a mitigation and are seeing service recovery for most users. We're continuing to monitor the situation and are working towards restoring services for all users.
  • Identified
    Update
    We are continuing to work on mitigating this attack.
  • Identified
    Update
    We are continuing to work on mitigating this attack.
  • Identified
    Update
    We apologize for the downtime and inconvenience this has caused. We are currently tweaking our mitigation strategies and will continue rolling out gradual service restoration to all impacted users.
  • Identified
    Update
    We're continuing to work on mitigating the DDoS attack. We have identified a potential mitigation strategy and will attempt rolling it out shortly.
  • Identified
    Identified
    We've identified a DDoS attack on our EU edge proxy hosts. We are working on mitigating and restoring service for all impacted users.
  • Investigating
    Investigating

    We rolled out a temporary fix that did not work. We are in the process of bringing our EU edge proxy hosts back online for further investigations. This incident only applies to a subset of requests originating from or routed through the EU. Applications are still serving fine globally.

  • Monitoring
    Monitoring
    NOTE: This only applies to a subset of requests originating from the EU. Applicaions are still serving fine globally. We've also rolled out a fix and requests are going through again
  • Investigating
    Investigating

    Our EU edge proxy is partially down. Railway-hosted applications may be inaccessible for users connecting from EU during this time.

Build logs from CLI returning a 403 but builds are still going out and logs are viewable in the dashboard.
  • Resolved
    Resolved

    Build and deploy logs are fully operational and streaming to the CLI again. These logs were always streaming in the dashboard.

  • Identified
    Identified

    We've identified a firewall rule blocking CLI from establishing WebSocket connections preventing build and deploy logs from streaming. Please visit your dashboard to view them for now while we roll out fix to firewall.

  • Investigating
    Investigating

    Build logs from CLI returning a 403 but builds are still going out and logs are viewable in the dashboard.

Dashboard and other Railway services may be down for some users
  • Resolved
    Resolved

    The dashboard and all impacted Railway services is back up. We suffered from another DDoS attack and successfully mitigated it.

  • Investigating
    Investigating

    Dashboard and other Railway services may be down for some users

API services are facing increased load causing certain requests to fail
  • Resolved
    Resolved

    Our services have recovered and all requests are being served properly.

  • Investigating
    Investigating

    API services are facing increased load causing certain requests to fail

Dashboard and other Railway services is failing to load for some users
  • Resolved
    Resolved

    This incident has been resolved.

    We had a series of targeted DDOS attacks causing heavy load on our systems and failures across the dashboard.

    We have since implemented and tested tertiary-level network jails to block and ban these attackers. Initially the bans were for shorter durations of time while we tested our systems and now that we have confidence in them, these bans should keep attackers out permanently.

  • Identified
    Identified

    We've identified the cause and are working towards restoring services for all users.

  • Investigating
    Investigating

    Dashboard and other Railway services is failing to load for some users

Dec 2023

A single host in the europe region has encountered a hardware error and is being failed over
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    The affected host is back up and running. All deploys have been restarted.

  • Identified
    Identified

    deployments on the affected hosts are starting to come back up, we expect full resolution within the next 10 minutes.

  • Investigating
    Investigating

    A single host in the europe region has encountered a hardware error - we're failing over that node. A small proportion of services in the europe region may be briefly unavailable.

Elevated request latency and error rates on edge network
  • Resolved
    Resolved

    This incident is now resolved. Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across our edge proxies in us-east and asia-southeast1 due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests originating from these regions failed / experienced higher latency during this time.

  • Monitoring
    Monitoring

    Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across all our edge proxies due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests failed / had higher latency during this time. Traffic has now recovered, and we're continuing to monitor.

  • Investigating
    Investigating

    Elevated request latency and error rates on edge network

Observed latency spikes on dashboard. Looking into it
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Update

    The issue seems to have been abated. During our discovery, we saw timeouts to GitHub and a few other external services. Given we don't use any egress products other than Cloud NAT from Google, we've escalated this to Google and are awaiting a response

  • Monitoring
    Monitoring

    Moving back to monitoring as we've observing it again

  • Resolved
    Resolved

    We've identified the issue as a bottleneck in our service mesh under heavier-than-average load which causes higher-than-normal latency.

  • Monitoring
    Monitoring

    Recovered. Initial response appears to be a flood of deployments with lots of variables put temporary lag on the database, which propagated to the dashboard. We will monitor as we continue to dig in

  • Investigating
    Investigating

    Observed latency spikes on dashboard. Looking into it

A subset of projects aren't loading in the dashboard
  • Resolved
    Update

    The issue has been resolved.

  • Resolved
    Resolved

    The issue has been resolved.

  • Monitoring
    Monitoring

    We have pushed a fix and are now monitoring.

  • Investigating
    Investigating

    A subset of projects aren't loading in the dashboard

Deploys failing to go out for some users & environment variables may not render correctly
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Update

    We implemented a fix and are currently monitoring the result.

  • Monitoring
    Monitoring

    We have rolled out a fix and are currently monitoring the results.

    Variables that were edited during the incident (within the past 1 hour) are currently being restored into their encrypted state.

Nov 2023

Workloads unavailable on Stacker 50
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Update

    The machine is currently back online and serving workloads once again. The Railway team will monitor this machine as workloads fully restore. We appreciate your patience.

  • Monitoring
    Monitoring

    The team is currently restarting the affected machine and bringing on workloads back in a staged manner, some workloads may continue to be unavailable.

  • Identified
    Identified

    Workloads on one machine hit a IO limit causing workloads to be unreachable for around 1 hour or so. We advise users to trigger a re-deployment if they haven't. For affected legacy plugins, migrating to a V2 database should resolve the issue.

A small percentage of provisions related to volumes are failing
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We're seeing a very small spike in provision errors for volumes in the US. Looking into it.

Deployments are slow to go out
  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

  • Resolved
    Resolved

    This incident has been resolved.

Deployments are failing to go out
  • Resolved
    Update

    This incident has been resolved.

  • Resolved
    Resolved

    We had a spike in deploy requests and a small delay for some deploys as they got backlogged.

  • Monitoring
    Monitoring

    We implemented a fix and are currently monitoring the result.

  • Investigating
    Investigating

    We are currently investigating this incident.

404 Flickers on Dashboard for some users
  • Resolved
    Resolved

    This incident has been resolved.

Nov 2023 to Jan 2024