Railway - Notice history

All systems operational

Dashboard (railway.com) - Operational

99.97% uptime
May 2024 · 100.0%Jun · 99.93%Jul · 99.99%

Previous page

Next page

May 2024
Jun 2024
Jul 2024
99.88% uptime

DNS - Operational

99.88% uptime
May 2024 · 100.0%Jun · 99.63%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Traffic - Operational

99.88% uptime
May 2024 · 100.0%Jun · 99.63%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Central Station (station.railway.com) - Operational

100.0% uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Railway Storage Buckets - Operational

100.0% uptime
May 2024 · 100.0%Jun · 100.0%Jul · 100.0%

Previous page

Next page

May 2024
Jun 2024
Jul 2024

Notice history

View current status

Jul 2024

a single host encountered a hardware failure and is restarting, a small portion of deploys in us-west may be unavailable
ResolvedPartial outage46 minutes
  • Resolved
    UTC
    Resolved

    All applications on the impacted host have resumed.

  • Monitoring
    UTC
    Monitoring

    The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
We're investigating routing failures for deployments using static IPs for egress.
ResolvedPartial outage40 minutes
  • Resolved
    UTC
    Resolved
    An update to our routing control plane incorrectly propagated a bad route for regional egress proxies, this led to the drop of static IP egress traffic from 21:28 UTC to approximately 22:00 UTC. Our static IP monitoring was configured globally in our largest region. We're adding regional static IP monitoring to catch this in staged rollouts (aka, much quicker).
  • Monitoring
    UTC
    Monitoring
    We've reverted the routing config and static IPs traffic is flowing normally.
  • Identified
    UTC
    Identified
    we've identified a configuration change in our network routing config that looks to have caused the problem. We're rolling out a fix globally.
  • Investigating
    UTC
    Investigating
    We're investigating routing failures for deployments using static IPs for egress.
Elevated image pull times for GitHub Container Registry
ResolvedDegraded performance6 hours 36 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    We are experiencing elevated pull times for images hosted in GitHub Container Registry (GHCR). This appears to be related to a third-party incident on GHCR's CDN (https://www.fastlystatus.com/incident/376717).

We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
ResolvedPartial outage1 hour 2 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

    Additionally, we will be digging into the scheduling issue to prevent this in the future.

  • Monitoring
    UTC
    Monitoring

    The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

  • Identified
    UTC
    Identified

    We've begun a failover process to recover the impacted host.

  • Investigating
    UTC
    Investigating
    We're investigating a single host failure in the us-east4 region, some deployments and volumes in the region may be inaccessible.
Some requests from v2 Proxy in US-West and Asia returning incorrectly encoded responses
ResolvedPartial outage17 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Impacted regions have been rolled back and responses should be correct now.

  • Identified
    UTC
    Identified

    The recent change is reverting.

Jun 2024

New volume provisioning in us-west region might be delayed
ResolvedDegraded performance2 hours 19 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We implemented a fix and are currently monitoring the result.

    We are adding capacity to the us-west region.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Dashboard loading slowly for some users
ResolvedDegraded performance50 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Dashboard latency has recovered.

  • Identified
    UTC
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Increased error rates for pulls from dockerhub
ResolvedMajor outage33 minutes
  • Resolved
    UTC
    Resolved

    We've seen error rates drop since 22:30 UTC and docker have moved their incident status to monitoring. Resolving.

  • Identified
    UTC
    Identified

    This has been confirmed as an upstream outage on DockerHub

  • Investigating
    UTC
    Investigating

    We're seeing elevated errors for image pulls from dockerhub. Any deploys referencing dockerhub images or builds relying on dockerhub images as their base may fail or be delayed.

Traffic routed over US East to Railway workloads are currently down
ResolvedPartial outage24 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved.

  • Monitoring
    UTC
    Monitoring

    Traffic is now back online. All impacted users should have their services restored. We are seeing high load on our US East proxy and are currently adding further mitigations to prevent this from happening again.

  • Identified
    UTC
    Identified

    We've isolated this incident to workloads on our new Edge Proxy only.

  • Investigating
    UTC
    Investigating

    We are currently investigating this incident. Any users connecting to Railway-hosted applications going through our US East edge network is impacted.

Slightly delayed builds. We’re looking into it
ResolvedDegraded performance1 hour 20 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Builds are becoming fully operational again.

  • Investigating
    UTC
    Investigating
    Slightly delayed builds. We’re looking into it

May 2024

we're investigating the failure of a single host in the us-west1 region
ResolvedPartial outage1 hour
  • Resolved
    UTC
    Resolved

    All workloads on the affected host have been fully restored.

  • Monitoring
    UTC
    Monitoring

    All workloads on the affected host have been restored. We're continuing to monitor.

  • Identified
    UTC
    Identified

    We're continuing to recover the failed host; the replacement is provisioning.

  • Investigating
    UTC
    Investigating

    A single host in the us-west1 region has encountered a hardware failure. We're investigating and recovering the host.

Dashboard may be slow or failing to load for some users
ResolvedDegraded performance21 minutes
  • Resolved
    UTC
    Resolved

    This incident has been resolved. Our database was temporarily unreachable from a subset of our API backend hosts, resulting in slower load times and errors for some users. We have performed a rolling restart of those hosts and are seeing full recovery.

  • Monitoring
    UTC
    Monitoring

    We have implemented a fix and are currently monitoring the result.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Deployments are failing to go out in the Europe region
ResolvedPartial outage20 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    UTC
    Investigating
    Deployments are failing to go out in the Europe region
Builds in us-west are stuck initializing
Resolved1 hour 10 minutes
  • Resolved
    UTC
    Resolved

    The incident has been resolved. Deploys are building normally now.

  • Monitoring
    UTC
    Monitoring

    Some of our builders are back online and deployments are slowly progressing

  • Update
    UTC
    Update

    We are still affected by the GCP outage and are monitoring it closely. https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#wVCaQTRV9VEch1ZTHVQn

  • Identified
    UTC
    Identified

    GCP Virtual Private Cloud (VPC) is currently having an incident that is affecting our builders (https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre#wVCaQTRV9VEch1ZTHVQn). We are working on mitigations

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Builds may be delayed in EU West
ResolvedDegraded performance35 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We have identified and implemented a fix for this issue, and are continuing to monitor the results. Our builders in EU West was under-provisioned leading to slower builds. We have scaled it up, and builds should now be going through normally.

Previous

May 2024 to Jul 2024

Next