saksham/CIAgent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restart nginx on every deploy to avoid stale upstream IPs

Browse Source 
nginx resolves the api/web service names to a container IP once, at
its own worker-process startup. docker compose up -d only recreates
containers whose image/config changed, so nginx keeps proxying to the
old, now-dead IP after a deploy rebuilds those containers - every
request 502s, which shows up in the browser as a misleading CORS
error since the bare 502 carries no Access-Control-Allow-Origin
header. Confirmed live: this caused a real multi-hour production
outage after the last two deploys.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
saksham
2026-08-06 07:07:55 -04:00
co-authored by Claude Sonnet 5
parent 4ee38b6241
commit 8ec7ca0881
2 changed files with 13 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEPLOYMENT.md
+3
View File
@@ -127,8 +127,11 @@ git pull
docker compose -f docker-compose.prod.yml build api worker beat web
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml restart nginx
```
**The `restart nginx` step is not optional.** `up -d` only recreates the containers whose image/config changed - nginx's own image/config doesn't change on an app deploy, so it keeps running with the container IPs it resolved at its *own* last startup. Once `api`/`web` get rebuilt with new IPs, nginx keeps proxying to the old, now-dead ones until something forces it to re-resolve - every request 502s (and shows up in the browser as a misleading CORS error, since a bare 502 from nginx carries no `Access-Control-Allow-Origin` header). Confirmed live: this caused a real multi-hour outage. `scripts/auto-deploy.sh` includes this step automatically.
**Or automatically**: `scripts/auto-deploy.sh` runs exactly that sequence, gated on "is `origin/master` ahead of `HEAD`" so it's a no-op most runs. Install it as a systemd timer (polls every 2 minutes — deliberately polling, not a Gitea webhook, so there's no extra exposed service, no Docker-socket-in-a-container, and no shared secret to manage):
```bash
scripts/auto-deploy.sh
+10
View File
@@ -28,4 +28,14 @@ docker compose -f docker-compose.prod.yml build api worker beat web
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
# nginx's proxy_pass resolves the api/web service names to a container IP
# once, at its own worker-process startup - `up -d` above only recreates
# the containers whose image/config actually changed, so nginx (unchanged)
# keeps running with the OLD ip, now pointing at a dead container.
# Confirmed live: this caused a real multi-hour outage (every request to
# api.ciagent.org / ciagent.org 502'd) after the containers it proxies to
# got rebuilt out from under it. `restart` forces new worker processes,
# which re-resolve the current IPs via Docker's embedded DNS.
docker compose -f docker-compose.prod.yml restart nginx
echo "$(date -Iseconds) deploy complete: $REMOTE"