CIAgent/DEPLOYMENT.md

Deployment

Production runbook for deploying this app to a single Ubuntu server behind Cloudflare, with a self-hosted Gitea instance for code hosting. Written for ciagent.org on Hetzner Cloud, but nothing here is Hetzner-specific beyond the firewall step.

Architecture

One Docker host runs everything via docker-compose.prod.yml: Postgres, Redis, the API, a Celery worker, Celery beat, the Next.js frontend, Gitea, and an Nginx reverse proxy. Nginx is the only container with a host-published port (80/443); everything else is reached over the internal Compose network by service name. Three Cloudflare-proxied subdomains route to it:

Subdomain	Routes to	Purpose
ciagent.org	web:3000	Next.js frontend
api.ciagent.org	api:8000	FastAPI backend
git.ciagent.org	gitea:3000	Self-hosted git (public read, admin-only write)

Cloudflare's proxy (orange-cloud DNS) hides the origin's real IP and absorbs generic bot/volumetric traffic; the app's own IP-based throttle/ban system (app/services/ip_throttle_service.py) handles the business-logic-specific abuse cases Cloudflare can't know about. Both need TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP set correctly or IP-based logic breaks — see KNOWN_LIMITATIONS.md.

1. Server prerequisites

• Ubuntu server with Docker Engine + the Compose plugin installed (docker compose version should work).
• A domain with its nameservers pointed at Cloudflare, and the zone added to a Cloudflare account.
• SSH access as a non-root user with Docker permissions (or root — your call, just don't run the app's own containers as root inside themselves, which they don't).

2. Cloudflare DNS + Origin CA certificate

1. Add three DNS A records, all proxied (orange cloud): ciagent.org, api.ciagent.org, git.ciagent.org → the server's public IP.
2. Cloudflare dashboard → SSL/TLS → Origin Server → Create Certificate. Cover ciagent.org and *.ciagent.org (one cert for all three subdomains), leave the default 15-year validity. Save the cert and private key.
3. On the server, create /etc/ci-agent/certs/ (outside the repo, never committed) and place the two files there as cloudflare-origin.pem and cloudflare-origin.key — this is exactly what docker-compose.prod.yml's nginx service mounts.
4. Cloudflare dashboard → SSL/TLS → set the encryption mode to Full (strict). Anything less either skips origin verification or falls back to plaintext HTTP to the origin.

3. Hetzner Cloud Firewall (or equivalent network-level firewall)

Restrict inbound to the origin so nothing can bypass Cloudflare's protection by hitting the server's real IP directly:

• Allow tcp/80 and tcp/443 only from Cloudflare's published IP ranges: https://www.cloudflare.com/ips-v4 and /ips-v6. Re-check these occasionally — they change rarely but do change.
• Allow tcp/22 (SSH) only from your own IP.
• Allow tcp/2222 (Gitea SSH push, optional — see §7) only from your own IP.
• Default-deny everything else inbound.

Prefer a network-level firewall (Hetzner Cloud Firewall) over host-only ufw — it blocks the packet before it reaches the box at all, so a host-firewall misconfiguration can't accidentally expose anything.

4. First boot

git clone <your git remote> ci-agent   # or `docker compose exec` your existing checkout
cd ci-agent
./scripts/bootstrap-env.sh

This creates .env from .env.example and generates real JWT_SECRET/API_KEY_ENCRYPTION_SECRET/POSTGRES_PASSWORD values. It deliberately leaves six things unset — see §8.

Then edit .env and set, at minimum:

APP_ENV=production
AUTH_MODE=jwt
FRONTEND_URL=https://ciagent.org
BACKEND_URL=https://api.ciagent.org
NEXT_PUBLIC_API_URL=https://api.ciagent.org
NEXT_PUBLIC_GIT_REPO_URL=https://git.ciagent.org/<your-admin-username>/ci-agent
TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP
RESEND_API_KEY=<real Resend key>          # no admin-UI equivalent for this one

APP_ENV=production with AUTH_MODE=local will make the api container refuse to start (a deliberate crash-on-boot safety check, not a bug — see app/core/config.py's validator) — if you see that, it means AUTH_MODE wasn't actually changed to jwt.

Bring up the database first, run migrations as a one-shot step (not baked into the long-running service), then the rest of the stack:

docker compose -f docker-compose.prod.yml up -d postgres redis
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d

5. Create the Gitea admin account — do this before opening the firewall

Important, confirmed by testing: Gitea always allows the very first account to register through the web UI, regardless of DISABLE_REGISTRATION=true — this is intentional upstream behavior so there's a way to bootstrap an admin at all. INSTALL_LOCK=true only skips the database-setup wizard; it does not close this separate first-user loophole. If the firewall is already open and DNS is live when Gitea first boots, a stranger who reaches /user/sign_up before you do gets the admin account.

So: bring the stack up with the firewall still closed (or bring up everything except leave DNS unpointed / firewall rules not yet applied), then immediately:

docker compose -f docker-compose.prod.yml exec -u git gitea gitea admin user create \
  --username <you> --password '<a real password>' --email <you>@example.com --admin

Only open the firewall / point DNS at the box after this succeeds. Once any account exists, /user/sign_up correctly shows "Registration is disabled" with no working form (also confirmed by testing).

Then, as that admin, create the repo via the web UI (or git push to create it — Gitea supports push-to-create) and set it public.

6. Push access

Two ways to push, since Cloudflare's proxy only speaks HTTP(S) — raw SSH can't ride the orange-cloud proxy:

• HTTPS + personal access token (primary) — Gitea → Settings → Applications → generate a token, then git remote set-url origin https://<token>@git.ciagent.org/<you>/ci-agent.git. Rides the same Cloudflare-proxied 443 as normal traffic, no extra firewall port needed.
• SSH on port 2222 (optional, secondary) — [email protected]:2222/<you>/ci-agent.git, using the standard git SSH key flow. This bypasses Cloudflare entirely (firewalled to your own IP per §3), so it's a fallback for when you're the one connecting from a known IP, not a general-purpose access method.

Read-only clone/browse works for anyone, no account: https://git.ciagent.org/<you>/ci-agent.git or the web UI directly.

7. Post-boot: configure provider API keys via the Settings UI, not .env

scripts/bootstrap-env.sh deliberately leaves these six blank. Sign in as the admin account and set them from the app itself:

• Settings → Your API keys: Anthropic, Brave Search, NinjaPear, USPTO — stored encrypted per-user (user_api_key_service.py), fall back to any global .env value if you ever set one, but there's no need to.
• Settings → Server secrets: Cloudflare Turnstile site key + secret — stored in the DB (system_secret_service.py), overrides .env at runtime, admin-only.

Also set LLM_PROVIDER=anthropic and SEARCH_PROVIDER=brave in .env (restart api/worker/beat after) once you've configured the corresponding keys above — provider selection is still a deployment-level .env setting, only the key values moved to the UI.

8. Updating the deployment

Manually:

git pull
docker compose -f docker-compose.prod.yml build api worker beat web
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d

Or automatically: scripts/auto-deploy.sh runs exactly that sequence, gated on "is origin/master ahead of HEAD" so it's a no-op most runs. Install it as a systemd timer (polls every 2 minutes — deliberately polling, not a Gitea webhook, so there's no extra exposed service, no Docker-socket-in-a-container, and no shared secret to manage):

cp infrastructure/systemd/ci-agent-deploy.service infrastructure/systemd/ci-agent-deploy.timer /etc/systemd/system/
chmod +x scripts/auto-deploy.sh
systemctl daemon-reload
systemctl enable --now ci-agent-deploy.timer

Once running, pushing to master on Gitea is enough — the server picks it up within ~2 minutes, no manual SSH step needed. Check journalctl -u ci-agent-deploy.service to see deploy runs.

9. Backups

Nothing backs itself up by default. At minimum, a nightly cron job on the host:

docker compose -f docker-compose.prod.yml exec -T postgres \
  pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > /backups/ciagent-$(date +%F).sql.gz

docker run --rm -v ci-agent-prod_gitea-data:/data -v /backups:/backup alpine \
  tar czf /backup/gitea-$(date +%F).tar.gz -C /data .

Copy /backups off-box (a Hetzner Storage Box via rclone, or any object storage) — local-only backups don't survive a lost disk. Retain a sane number of days and prune older ones.

10. Operational notes

• beat must stay exactly one instance, always — duplicate scheduled monitoring runs otherwise. Don't --scale beat=2.
• Docker's log driver is capped per-service (max-size: 10m, max-file: 3 in docker-compose.prod.yml) — nothing else bounds log growth on the host.
• Postgres and Redis publish no host ports in prod (unlike the dev docker-compose.yml) — only reachable over the internal Compose network.
• The dev Dockerfiles/compose file (docker-compose.yml, infrastructure/docker/*.Dockerfile without .prod) are untouched by any of this and remain the local-development setup.