sakshamandClaude Sonnet 5
8ec7ca0881
nginx resolves the api/web service names to a container IP once, at its own worker-process startup. docker compose up -d only recreates containers whose image/config changed, so nginx keeps proxying to the old, now-dead IP after a deploy rebuilds those containers - every request 502s, which shows up in the browser as a misleading CORS error since the bare 502 carries no Access-Control-Allow-Origin header. Confirmed live: this caused a real multi-hour production outage after the last two deploys. Co-Authored-By: Claude Sonnet 5 <[email protected]>
42 lines
1.7 KiB
Bash
42 lines
1.7 KiB
Bash
#!/usr/bin/env bash
|
|
# Polls the public CIAgent repo for new commits on master and redeploys if
|
|
# found. Runs as a systemd timer on the server (infrastructure/systemd/) -
|
|
# not a webhook, deliberately: a webhook receiver would need either the
|
|
# Docker socket mounted into a container reachable from a request handler,
|
|
# or a new exposed service/nginx route/shared secret to manage. Polling
|
|
# does the same job with none of that - it's just this same sequence run on
|
|
# a timer, on the host, as root, exactly like a manual deploy.
|
|
#
|
|
# The repo is public, so `git fetch` here needs no credentials.
|
|
|
|
set -euo pipefail
|
|
cd /opt/ci-agent
|
|
|
|
git fetch origin master -q
|
|
|
|
LOCAL=$(git rev-parse HEAD)
|
|
REMOTE=$(git rev-parse origin/master)
|
|
|
|
if [ "$LOCAL" = "$REMOTE" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
echo "$(date -Iseconds) deploying $REMOTE (was $LOCAL)"
|
|
|
|
git merge --ff-only origin/master
|
|
docker compose -f docker-compose.prod.yml build api worker beat web
|
|
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
|
|
docker compose -f docker-compose.prod.yml up -d
|
|
|
|
# nginx's proxy_pass resolves the api/web service names to a container IP
|
|
# once, at its own worker-process startup - `up -d` above only recreates
|
|
# the containers whose image/config actually changed, so nginx (unchanged)
|
|
# keeps running with the OLD ip, now pointing at a dead container.
|
|
# Confirmed live: this caused a real multi-hour outage (every request to
|
|
# api.ciagent.org / ciagent.org 502'd) after the containers it proxies to
|
|
# got rebuilt out from under it. `restart` forces new worker processes,
|
|
# which re-resolve the current IPs via Docker's embedded DNS.
|
|
docker compose -f docker-compose.prod.yml restart nginx
|
|
|
|
echo "$(date -Iseconds) deploy complete: $REMOTE"
|