CIAgent/DEPLOYMENT.md

Deployment

Production runbook for deploying this app to a single Ubuntu server behind Cloudflare, with a self-hosted Gitea instance for code hosting. Written for ciagent.org on Hetzner Cloud, but nothing here is Hetzner-specific beyond the firewall step.

Live since 2026-08-06 — auto-deploy via ci-agent-deploy.timer confirmed working end-to-end.

Architecture

One Docker host runs everything via docker-compose.prod.yml: Postgres, Redis, the API, a Celery worker, Celery beat, the Next.js frontend, Gitea, and an Nginx reverse proxy. Nginx is the only container with a host-published port (80/443); everything else is reached over the internal Compose network by service name. Three Cloudflare-proxied subdomains route to it:

Subdomain	Routes to	Purpose
ciagent.org	web:3000	Next.js frontend
api.ciagent.org	api:8000	FastAPI backend
git.ciagent.org	gitea:3000	Self-hosted git (public read, admin-only write)
db.ciagent.org	adminer:8080	Database viewer (any admin account, no separate password - see §7)

Cloudflare's proxy (orange-cloud DNS) hides the origin's real IP and absorbs generic bot/volumetric traffic; the app's own IP-based throttle/ban system (app/services/ip_throttle_service.py) handles the business-logic-specific abuse cases Cloudflare can't know about. Both need TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP set correctly or IP-based logic breaks — see KNOWN_LIMITATIONS.md.

1. Server prerequisites

• Ubuntu server with Docker Engine + the Compose plugin installed (docker compose version should work).
• A domain with its nameservers pointed at Cloudflare, and the zone added to a Cloudflare account.
• SSH access as a non-root user with Docker permissions (or root — your call, just don't run the app's own containers as root inside themselves, which they don't).

2. Cloudflare DNS + Origin CA certificate

1. Add four DNS A records, all proxied (orange cloud): ciagent.org, api.ciagent.org, git.ciagent.org, db.ciagent.org → the server's public IP.
2. Cloudflare dashboard → SSL/TLS → Origin Server → Create Certificate. Cover ciagent.org and *.ciagent.org (one cert for all four subdomains, including any added later), leave the default 15-year validity. Save the cert and private key.
3. On the server, create /etc/ci-agent/certs/ (outside the repo, never committed) and place the two files there as cloudflare-origin.pem and cloudflare-origin.key — this is exactly what docker-compose.prod.yml's nginx service mounts.
4. Cloudflare dashboard → SSL/TLS → set the encryption mode to Full (strict). Anything less either skips origin verification or falls back to plaintext HTTP to the origin.

3. Hetzner Cloud Firewall (or equivalent network-level firewall)

Restrict inbound to the origin so nothing can bypass Cloudflare's protection by hitting the server's real IP directly:

• Allow tcp/80 and tcp/443 only from Cloudflare's published IP ranges: https://www.cloudflare.com/ips-v4 and /ips-v6. Re-check these occasionally — they change rarely but do change.
• Allow tcp/22 (SSH) only from your own IP.
• Allow tcp/2222 (Gitea SSH push, optional — see §7) only from your own IP.
• Default-deny everything else inbound.

Prefer a network-level firewall (Hetzner Cloud Firewall) over host-only ufw — it blocks the packet before it reaches the box at all, so a host-firewall misconfiguration can't accidentally expose anything.

4. First boot

git clone <your git remote> ci-agent   # or `docker compose exec` your existing checkout
cd ci-agent
./scripts/bootstrap-env.sh

This creates .env from .env.example and generates real JWT_SECRET/API_KEY_ENCRYPTION_SECRET/POSTGRES_PASSWORD values. It deliberately leaves six things unset — see §8.

Then edit .env and set, at minimum:

APP_ENV=production
AUTH_MODE=jwt
FRONTEND_URL=https://ciagent.org
BACKEND_URL=https://api.ciagent.org
NEXT_PUBLIC_API_URL=https://api.ciagent.org
NEXT_PUBLIC_GIT_REPO_URL=https://git.ciagent.org/<your-admin-username>/ci-agent
NEXT_PUBLIC_DB_VIEWER_URL=https://db.ciagent.org
TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP
RESEND_API_KEY=<real Resend key>          # no admin-UI equivalent for this one

APP_ENV=production with AUTH_MODE=local will make the api container refuse to start (a deliberate crash-on-boot safety check, not a bug — see app/core/config.py's validator) — if you see that, it means AUTH_MODE wasn't actually changed to jwt.

Bring up the database first, run migrations as a one-shot step (not baked into the long-running service), then the rest of the stack:

docker compose -f docker-compose.prod.yml up -d postgres redis
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d

5. Create the Gitea admin account — do this before opening the firewall

Important, confirmed by testing: Gitea always allows the very first account to register through the web UI, regardless of DISABLE_REGISTRATION=true — this is intentional upstream behavior so there's a way to bootstrap an admin at all. INSTALL_LOCK=true only skips the database-setup wizard; it does not close this separate first-user loophole. If the firewall is already open and DNS is live when Gitea first boots, a stranger who reaches /user/sign_up before you do gets the admin account.

So: bring the stack up with the firewall still closed (or bring up everything except leave DNS unpointed / firewall rules not yet applied), then immediately:

docker compose -f docker-compose.prod.yml exec -u git gitea gitea admin user create \
  --username <you> --password '<a real password>' --email <you>@example.com --admin

Only open the firewall / point DNS at the box after this succeeds. Once any account exists, /user/sign_up correctly shows "Registration is disabled" with no working form (also confirmed by testing).

Then, as that admin, create the repo via the web UI (or git push to create it — Gitea supports push-to-create) and set it public.

6. Push access

Two ways to push, since Cloudflare's proxy only speaks HTTP(S) — raw SSH can't ride the orange-cloud proxy:

• HTTPS + personal access token (primary) — Gitea → Settings → Applications → generate a token, then git remote set-url origin https://<token>@git.ciagent.org/<you>/ci-agent.git. Rides the same Cloudflare-proxied 443 as normal traffic, no extra firewall port needed.
• SSH on port 2222 (optional, secondary) — [email protected]:2222/<you>/ci-agent.git, using the standard git SSH key flow. This bypasses Cloudflare entirely (firewalled to your own IP per §3), so it's a fallback for when you're the one connecting from a known IP, not a general-purpose access method.

Read-only clone/browse works for anyone, no account: https://git.ciagent.org/<you>/ci-agent.git or the web UI directly.

7. Database viewer (Adminer at db.ciagent.org)

Unlike Gitea's admin account, there's no manual credential-handoff step for this one — any account with is_admin=true on the app itself can use it immediately, once NEXT_PUBLIC_DB_VIEWER_URL is set (§4) and the site is redeployed. Settings → Database → "Open database viewer" mints a short-lived token from the admin's real, live-checked login, which db.ciagent.org's Nginx block (infrastructure/nginx/nginx.conf) exchanges for a signed session cookie via apps/api/app/api/v1/db_viewer.py — no separate password to generate, distribute, or rotate.

A couple of things worth knowing:

• Sessions last 60 minutes. Revoking someone's is_admin flag takes effect on their very next request through Nginx's auth_request check (it re-loads the user from the database each time), but an already-open Adminer tab isn't force-closed — it just stops being able to load anything new once that check runs again.
• Adminer's own Postgres login (username/password) is a second, independent layer past this gate — real DB credentials are still required to actually view or edit anything.
• An optional, commented-out IP-allowlist snippet is included in the db.ciagent.org Nginx block for admins with a static IP who want to require both the session and a matching source address — not enabled by default, since most admin connections don't have a stable IP to pin to.

8. Post-boot: configure provider API keys via the Settings UI, not .env

Where does the admin account come from? The very first account ever registered on the app (is_admin count is zero) is automatically promoted to admin - see auth_service.register. No manual DB access or setup script required, but it does mean the same race Gitea has in §5 applies here too: if the firewall is open and the site is reachable before you've registered your own account, a stranger who registers first becomes the admin instead of you. Register your own account immediately after the stack comes up, before opening the firewall/pointing DNS at it publicly. If the last admin's account is ever deleted, the next person to register becomes admin again - it never permanently locks the deployment out of admin access.

scripts/bootstrap-env.sh deliberately leaves these six blank. Sign in as the admin account and set them from the app itself:

• Settings → Your API keys: Anthropic, Brave Search, NinjaPear, USPTO — stored encrypted per-user (user_api_key_service.py), fall back to any global .env value if you ever set one, but there's no need to.
• Settings → Server secrets: Cloudflare Turnstile site key + secret — stored in the DB (system_secret_service.py), overrides .env at runtime, admin-only.

Also set LLM_PROVIDER=anthropic and SEARCH_PROVIDER=brave in .env (restart api/worker/beat after) once you've configured the corresponding keys above — provider selection is still a deployment-level .env setting, only the key values moved to the UI.

9. Updating the deployment

Manually:

git pull
docker compose -f docker-compose.prod.yml build api worker beat web
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml restart nginx

The restart nginx step is not optional. up -d only recreates the containers whose image/config changed - nginx's own image/config doesn't change on an app deploy, so it keeps running with the container IPs it resolved at its own last startup. Once api/web get rebuilt with new IPs, nginx keeps proxying to the old, now-dead ones until something forces it to re-resolve - every request 502s (and shows up in the browser as a misleading CORS error, since a bare 502 from nginx carries no Access-Control-Allow-Origin header). Confirmed live: this caused a real multi-hour outage. scripts/auto-deploy.sh includes this step automatically.

Or automatically: scripts/auto-deploy.sh runs exactly that sequence, gated on "is origin/master ahead of HEAD" so it's a no-op most runs. Install it as a systemd timer (polls every 2 minutes — deliberately polling, not a Gitea webhook, so there's no extra exposed service, no Docker-socket-in-a-container, and no shared secret to manage):

cp infrastructure/systemd/ci-agent-deploy.service infrastructure/systemd/ci-agent-deploy.timer /etc/systemd/system/
chmod +x scripts/auto-deploy.sh
systemctl daemon-reload
systemctl enable --now ci-agent-deploy.timer

Once running, pushing to master on Gitea is enough — the server picks it up within ~2 minutes, no manual SSH step needed. Check journalctl -u ci-agent-deploy.service to see deploy runs.

10. Backups

Nothing backs itself up by default. At minimum, a nightly cron job on the host:

docker compose -f docker-compose.prod.yml exec -T postgres \
  pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > /backups/ciagent-$(date +%F).sql.gz

docker run --rm -v ci-agent-prod_gitea-data:/data -v /backups:/backup alpine \
  tar czf /backup/gitea-$(date +%F).tar.gz -C /data .

Copy /backups off-box (a Hetzner Storage Box via rclone, or any object storage) — local-only backups don't survive a lost disk. Retain a sane number of days and prune older ones.

11. Operational notes

• beat must stay exactly one instance, always — duplicate scheduled monitoring runs otherwise. Don't --scale beat=2.
• Docker's log driver is capped per-service (max-size: 10m, max-file: 3 in docker-compose.prod.yml) — nothing else bounds log growth on the host.
• Postgres and Redis publish no host ports in prod (unlike the dev docker-compose.yml) — only reachable over the internal Compose network.
• The dev Dockerfiles/compose file (docker-compose.yml, infrastructure/docker/*.Dockerfile without .prod) are untouched by any of this and remain the local-development setup.