Railway - Observed latency spikes on dashboard. Looking into it – Incident details

This incident has been resolved.

The issue seems to have been abated. During our discovery, we saw timeouts to GitHub and a few other external services. Given we don't use any egress products other than Cloud NAT from Google, we've escalated this to Google and are awaiting a response

Moving back to monitoring as we've observing it again

We've identified the issue as a bottleneck in our service mesh under heavier-than-average load which causes higher-than-normal latency.

Recovered. Initial response appears to be a flood of deployments with lots of variables put temporary lag on the database, which propagated to the dashboard. We will monitor as we continue to dig in