Restart nginx on every deploy to avoid stale upstream IPs

nginx resolves the api/web service names to a container IP once, at its own worker-process startup. docker compose up -d only recreates containers whose image/config changed, so nginx keeps proxying to the old, now-dead IP after a deploy rebuilds those containers - every request 502s, which shows up in the browser as a misleading CORS error since the bare 502 carries no Access-Control-Allow-Origin header. Confirmed live: this caused a real multi-hour production outage after the last two deploys. Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-08-06 07:07:55 -04:00
parent 4ee38b6241
commit 8ec7ca0881
2 changed files with 13 additions and 0 deletions
@@ -28,4 +28,14 @@ docker compose -f docker-compose.prod.yml build api worker beat web
 docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
 docker compose -f docker-compose.prod.yml up -d

+# nginx's proxy_pass resolves the api/web service names to a container IP
+# once, at its own worker-process startup - `up -d` above only recreates
+# the containers whose image/config actually changed, so nginx (unchanged)
+# keeps running with the OLD ip, now pointing at a dead container.
+# Confirmed live: this caused a real multi-hour outage (every request to
+# api.ciagent.org / ciagent.org 502'd) after the containers it proxies to
+# got rebuilt out from under it. `restart` forces new worker processes,
+# which re-resolve the current IPs via Docker's embedded DNS.
+docker compose -f docker-compose.prod.yml restart nginx
+
 echo "$(date -Iseconds) deploy complete: $REMOTE"