diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index 75594f4..2500df3 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -127,8 +127,11 @@ git pull docker compose -f docker-compose.prod.yml build api worker beat web docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head docker compose -f docker-compose.prod.yml up -d +docker compose -f docker-compose.prod.yml restart nginx ``` +**The `restart nginx` step is not optional.** `up -d` only recreates the containers whose image/config changed - nginx's own image/config doesn't change on an app deploy, so it keeps running with the container IPs it resolved at its *own* last startup. Once `api`/`web` get rebuilt with new IPs, nginx keeps proxying to the old, now-dead ones until something forces it to re-resolve - every request 502s (and shows up in the browser as a misleading CORS error, since a bare 502 from nginx carries no `Access-Control-Allow-Origin` header). Confirmed live: this caused a real multi-hour outage. `scripts/auto-deploy.sh` includes this step automatically. + **Or automatically**: `scripts/auto-deploy.sh` runs exactly that sequence, gated on "is `origin/master` ahead of `HEAD`" so it's a no-op most runs. Install it as a systemd timer (polls every 2 minutes — deliberately polling, not a Gitea webhook, so there's no extra exposed service, no Docker-socket-in-a-container, and no shared secret to manage): ```bash diff --git a/scripts/auto-deploy.sh b/scripts/auto-deploy.sh index 5327f9d..3ee5df8 100644 --- a/scripts/auto-deploy.sh +++ b/scripts/auto-deploy.sh @@ -28,4 +28,14 @@ docker compose -f docker-compose.prod.yml build api worker beat web docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head docker compose -f docker-compose.prod.yml up -d +# nginx's proxy_pass resolves the api/web service names to a container IP +# once, at its own worker-process startup - `up -d` above only recreates +# the containers whose image/config actually changed, so nginx (unchanged) +# keeps running with the OLD ip, now pointing at a dead container. +# Confirmed live: this caused a real multi-hour outage (every request to +# api.ciagent.org / ciagent.org 502'd) after the containers it proxies to +# got rebuilt out from under it. `restart` forces new worker processes, +# which re-resolve the current IPs via Docker's embedded DNS. +docker compose -f docker-compose.prod.yml restart nginx + echo "$(date -Iseconds) deploy complete: $REMOTE"