Devops

The site started throwing 502 Bad Gateway errors. Everything stopped. Restarting Gunicorn fixed it within seconds. Then it would happen again, at a completely random time, once every few days. This went on for about two weeks before we decided to properly dig in. That pattern is almost always a leak. In our case it was database connections. Infrastructure context RDS: db.m5.4xlarge, max_connections=5000, tcp_keepalives_idle=300s Gunicorn: 8 workers x 25 threads = 200 concurrent connections Celery: 3 instances (1 main worker at concurrency=35, 2 side workers at concurrency=25 each) = 85 worker threads total Total max DB connections across all processes: 285 Peak traffic (8am to 8pm SGT): ...

It was a Tuesday afternoon, around 3pm. We were deep into sprint planning when the Slack messages started coming in. “I can’t log in.” “The app is really slow.” “Is something wrong with Privyr?” The first instinct was to look for a simpler explanation. Maybe a flaky deploy. Maybe a user on a slow connection. Then more messages. Then a lot more. I tried to SSH into the box. It would not connect. CPU was pegged. I pulled up the metrics dashboards and stared at the graphs for a few seconds. Request counts were off the charts. Response times had collapsed. This was not a bug. ...