From 8ec7ca0881e73f77850ec09e87b31eb68cc22965 Mon Sep 17 00:00:00 2001 From: Saksham Das Date: Thu, 6 Aug 2026 07:07:55 -0400 Subject: [PATCH] Restart nginx on every deploy to avoid stale upstream IPs nginx resolves the api/web service names to a container IP once, at its own worker-process startup. docker compose up -d only recreates containers whose image/config changed, so nginx keeps proxying to the old, now-dead IP after a deploy rebuilds those containers - every request 502s, which shows up in the browser as a misleading CORS error since the bare 502 carries no Access-Control-Allow-Origin header. Confirmed live: this caused a real multi-hour production outage after the last two deploys. Co-Authored-By: Claude Sonnet 5 --- DEPLOYMENT.md | 3 +++ scripts/auto-deploy.sh | 10 ++++++++++ 2 files changed, 13 insertions(+) diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 75594f4..2500df3 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -127,8 +127,11 @@ git pull docker compose -f docker-compose.prod.yml build api worker beat web docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head docker compose -f docker-compose.prod.yml up -d +docker compose -f docker-compose.prod.yml restart nginx ``` +**The `restart nginx` step is not optional.** `up -d` only recreates the containers whose image/config changed - nginx's own image/config doesn't change on an app deploy, so it keeps running with the container IPs it resolved at its *own* last startup. Once `api`/`web` get rebuilt with new IPs, nginx keeps proxying to the old, now-dead ones until something forces it to re-resolve - every request 502s (and shows up in the browser as a misleading CORS error, since a bare 502 from nginx carries no `Access-Control-Allow-Origin` header). Confirmed live: this caused a real multi-hour outage. `scripts/auto-deploy.sh` includes this step automatically. + **Or automatically**: `scripts/auto-deploy.sh` runs exactly that sequence, gated on "is `origin/master` ahead of `HEAD`" so it's a no-op most runs. Install it as a systemd timer (polls every 2 minutes — deliberately polling, not a Gitea webhook, so there's no extra exposed service, no Docker-socket-in-a-container, and no shared secret to manage): ```bash diff --git a/scripts/auto-deploy.sh b/scripts/auto-deploy.sh index 5327f9d..3ee5df8 100644 --- a/scripts/auto-deploy.sh +++ b/scripts/auto-deploy.sh @@ -28,4 +28,14 @@ docker compose -f docker-compose.prod.yml build api worker beat web docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head docker compose -f docker-compose.prod.yml up -d +# nginx's proxy_pass resolves the api/web service names to a container IP +# once, at its own worker-process startup - `up -d` above only recreates +# the containers whose image/config actually changed, so nginx (unchanged) +# keeps running with the OLD ip, now pointing at a dead container. +# Confirmed live: this caused a real multi-hour outage (every request to +# api.ciagent.org / ciagent.org 502'd) after the containers it proxies to +# got rebuilt out from under it. `restart` forces new worker processes, +# which re-resolve the current IPs via Docker's embedded DNS. +docker compose -f docker-compose.prod.yml restart nginx + echo "$(date -Iseconds) deploy complete: $REMOTE"