Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

100% - uptime
Aug 2024 · 99.90%Sep · 99.68%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024
100% - uptime

DNS - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Traffic - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Central Station (station.railway.com) - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Railway Storage Buckets - Operational

100% - uptime
Aug 2024 · 100.0%Sep · 100.0%Oct · 100.0%

Previous page

Next page

Aug 2024
Sep 2024
Oct 2024

Notice history

View current status

Oct 2024

We are investigating an issue with the edge network proxy
Resolved58 minutes
  • Resolved
    UTC
    Resolved

    We have resolved this issue.

  • Monitoring
    UTC
    Monitoring

    A fix has been implemented and we're currently monitoring the edge network proxy.

  • Identified
    UTC
    Identified

    We have identified the issue as a DDoS attack. We're in the process of applying additional mitigations

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident.

A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
ResolvedPartial outage14 minutes
  • Resolved
    UTC
    Resolved
    The host has fully recovered and all impacted workloads are back online.
  • Investigating
    UTC
    Investigating
    A host in the us-west region experienced an unexpected restarts. We're recovering the last workloads on this host that failed to restore automatically.
[Retroactive] A single host in US East has restarted
Resolved
  • Resolved
    UTC
    Resolved

    (This is a retroactive incident) During this time we noticed a single host in the US East 1 region had restarted and we quickly worked to bring containers back online, users may have not been able to dial their deployments or connect to databases during the restart efforts. This incident has been resolved.

[Retroactive] Two hosts in EU West experienced degraded performance
Resolved
  • Resolved
    UTC
    Resolved

    This is a retroactive incident which is resolved. For the period of the incident (11:45-12:35 UTC), some workloads may have exhibited connection problems. The incident was due to resource exhaustion on the two hosts due to unaccounted resource thresholds. We have improved our monitoring to better respond to this type of incident and are working on preventative measures in our orchestration.

We have observed degraded performance of the proxy layer. Users may experience dropped connections. We are investigating.
ResolvedDegraded performance32 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We have taken action to mitigate the issue and are monitoring.
  • Investigating
    UTC
    Investigating

    We have observed degraded performance of the edge proxy layer. Users may experience dropped connections. We are investigating.

Sep 2024

[External] Dockerhub Image pull returns 401 Unauthorized
ResolvedDegraded performance6 hours 3 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    We are currently investigating this incident. For more info see Docker's status page: https://www.dockerstatus.com/

Failing to create volumes
Resolved19 minutes
  • Resolved
    UTC
    Resolved

    We have traced the issue to a guard that was trying to prevent users to deploying volumes on Railway Metal. We have since applied the fix and systems are working normally.

  • Investigating
    UTC
    Investigating
    Failing to create volumes
SEP25
Dockerhub is currently undergoing maintenance.
ResolvedMaintenance5 hours 2 minutes
  • Resolved
    UTC
    Resolved

    Dockerhub has completed their maintenance

  • Identified
    UTC
    Identified

    During this period, builds might fail, displaying a "401 Unauthorized" message in the logs.

    Please check Dockerhub's status page for more information.

    https://www.dockerstatus.com/

SEP19
Edge Network Hardware Upgrades
CompletedMaintenance3 hours 8 minutes
  • Completed
    September 19, 2024 at 9:08 PMUTC
    Completed
    September 19, 2024 at 9:08 PMUTC

    We have successfully upgraded our edge network hardware. You should not experience any further connection disconnects.

    Our monitors during this time show no degradation to TCP connections or HTTP requests failures while we performed this across our fleet but as mentioned, closing your connection to shutdown the machine was unavoidable. Your clients would reconnect and continue to work as expected.

  • Update
    September 19, 2024 at 8:17 PMUTC
    Update
    September 19, 2024 at 8:17 PMUTC

    We are upgrading our final set of machines ahead of schedule.

  • In progress
    September 19, 2024 at 6:43 PMUTC
    In progress
    September 19, 2024 at 6:43 PMUTC

    This upgrade is in progress.

  • Planned
    September 19, 2024 at 6:00 PMUTC
    Planned
    September 19, 2024 at 6:00 PMUTC

    We are upgrading the hardware for our edge network that powers TCP and HTTP proxying. During this time requests to boxes being upgraded will have 5 minutes to complete before being disconnected. These clients will be able to reconnect and will be routed to another box that will handle the traffic indefinitely.

Builds are currently in a queueing state. We're lookinig into it
ResolvedDegraded performance28 minutes
  • Resolved
    UTC
    Resolved
    Queue back to zero
  • Monitoring
    UTC
    Monitoring
    Queued backpressure is almost drained. Most queued deploys have been rolled out
  • Identified
    UTC
    Identified
    Deploys are going out and the queue is draining. We will move to monitoring when there are no queued builds
  • Investigating
    UTC
    Investigating
    Builds are currently in a queueing state. We're lookinig into it

Aug 2024

We're seeing degraded performance on our Asia proxy
ResolvedDegraded performance33 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We've restarted the impacted boxes and restored service. We are still monitoring
  • Investigating
    UTC
    Investigating
    We're seeing degraded performance on our Asia proxy
Asia edge proxy (v2) service degradation
Resolved
  • Resolved
    UTC
    Resolved

    This is a retroactive incident about severe service degradation that happened between 12:40 PM UTC and 13:10 PM UTC on our edge proxies in south-east Asia. We are conducting an investigation into what happened and will update this incident once it's complete.

Edge Proxy outage
ResolvedMajor outage1 hour 27 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    All proxies are back up in all regions. We are still monitoring as everything is coming back up.

  • Update
    UTC
    Update

    Europe and US-East proxies are back up

  • Update
    UTC
    Update

    Southeast Asia (Singapore) is back up

  • Monitoring
    UTC
    Monitoring

    Most of the traffic appears to be back up, but we're still monitoring.

  • Update
    UTC
    Update

    Our fix is rolling out and we are seeing recovery for some customers.

  • Identified
    UTC
    Identified

    Our new edge proxy is experiencing a major outage. We have identified the issue and are working on a fix

Some requests from our edge proxy in Europe may have experienced higher latency than normal
Resolved24 hours 30 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    Our proxy in Europe continued to process requests but from August 25th, 20:30 to August 26th, 15:00, HTTP latency may have been elevated as we dealt with a sudden increase in traffic from this region.

Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered
ResolvedDegraded performance9 minutes
  • Resolved
    UTC
    Resolved
    Builds are going out nominally.
  • Monitoring
    UTC
    Monitoring
    We have restored the build queue and are now monitoring the health of remaining builds.
  • Investigating
    UTC
    Investigating
    Builds are queuing up due to a bad rollout on the workers. We're reverting. Builds triggered in the last 10 minutes will start to go out without needing to be retriggered

Previous

Aug 2024 to Oct 2024

Next