CIAgent/DEPLOYMENT.md

# Deployment

Production runbook for deploying this app to a single Ubuntu server behind Cloudflare, with a self-hosted Gitea instance for code hosting. Written for `ciagent.org` on Hetzner Cloud, but nothing here is Hetzner-specific beyond the firewall step.

_Live since 2026-08-06 — auto-deploy via `ci-agent-deploy.timer` confirmed working end-to-end._

## Architecture

One Docker host runs everything via `docker-compose.prod.yml`: Postgres, Redis, the API, a Celery worker, Celery beat, the Next.js frontend, Gitea, and an Nginx reverse proxy. Nginx is the only container with a host-published port (80/443); everything else is reached over the internal Compose network by service name. Three Cloudflare-proxied subdomains route to it:

| Subdomain | Routes to | Purpose |
|---|---|---|
| `ciagent.org` | `web:3000` | Next.js frontend |
| `api.ciagent.org` | `api:8000` | FastAPI backend |
| `git.ciagent.org` | `gitea:3000` | Self-hosted git (public read, admin-only write) |
| `db.ciagent.org` | `adminer:8080` | Database viewer (any admin account, no separate password - see §7) |

Cloudflare's proxy (orange-cloud DNS) hides the origin's real IP and absorbs generic bot/volumetric traffic; the app's own IP-based throttle/ban system (`app/services/ip_throttle_service.py`) handles the business-logic-specific abuse cases Cloudflare can't know about. Both need `TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP` set correctly or IP-based logic breaks — see `KNOWN_LIMITATIONS.md`.

## 1. Server prerequisites

- Ubuntu server with Docker Engine + the Compose plugin installed (`docker compose version` should work).
- A domain with its nameservers pointed at Cloudflare, and the zone added to a Cloudflare account.
- SSH access as a non-root user with Docker permissions (or root — your call, just don't run the app's own containers as root inside themselves, which they don't).

## 2. Cloudflare DNS + Origin CA certificate

1. Add four DNS **A records**, all proxied (orange cloud): `ciagent.org`, `api.ciagent.org`, `git.ciagent.org`, `db.ciagent.org` → the server's public IP.
2. Cloudflare dashboard → SSL/TLS → **Origin Server** → Create Certificate. Cover `ciagent.org` and `*.ciagent.org` (one cert for all four subdomains, including any added later), leave the default 15-year validity. Save the cert and private key.
3. On the server, create `/etc/ci-agent/certs/` (outside the repo, never committed) and place the two files there as `cloudflare-origin.pem` and `cloudflare-origin.key` — this is exactly what `docker-compose.prod.yml`'s `nginx` service mounts.
4. Cloudflare dashboard → SSL/TLS → set the encryption mode to **Full (strict)**. Anything less either skips origin verification or falls back to plaintext HTTP to the origin.

## 3. Hetzner Cloud Firewall (or equivalent network-level firewall)

Restrict inbound to the origin so nothing can bypass Cloudflare's protection by hitting the server's real IP directly:

- Allow tcp/80 and tcp/443 **only** from Cloudflare's published IP ranges: `https://www.cloudflare.com/ips-v4` and `/ips-v6`. Re-check these occasionally — they change rarely but do change.
- Allow tcp/22 (SSH) only from your own IP.
- Allow tcp/2222 (Gitea SSH push, optional — see §7) only from your own IP.
- Default-deny everything else inbound.

Prefer a network-level firewall (Hetzner Cloud Firewall) over host-only `ufw` — it blocks the packet before it reaches the box at all, so a host-firewall misconfiguration can't accidentally expose anything.

## 4. First boot

```bash
git clone <your git remote> ci-agent   # or `docker compose exec` your existing checkout
cd ci-agent
./scripts/bootstrap-env.sh
```

This creates `.env` from `.env.example` and generates real `JWT_SECRET`/`API_KEY_ENCRYPTION_SECRET`/`POSTGRES_PASSWORD` values. It deliberately leaves six things unset — see §8.

Then edit `.env` and set, at minimum:

```
APP_ENV=production
AUTH_MODE=jwt
FRONTEND_URL=https://ciagent.org
BACKEND_URL=https://api.ciagent.org
NEXT_PUBLIC_API_URL=https://api.ciagent.org
NEXT_PUBLIC_GIT_REPO_URL=https://git.ciagent.org/<your-admin-username>/ci-agent
NEXT_PUBLIC_DB_VIEWER_URL=https://db.ciagent.org
TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP
RESEND_API_KEY=<real Resend key>          # no admin-UI equivalent for this one
```

`APP_ENV=production` with `AUTH_MODE=local` will make the `api` container refuse to start (a deliberate crash-on-boot safety check, not a bug — see `app/core/config.py`'s validator) — if you see that, it means `AUTH_MODE` wasn't actually changed to `jwt`.

Bring up the database first, run migrations as a one-shot step (not baked into the long-running service), then the rest of the stack:

```bash
docker compose -f docker-compose.prod.yml up -d postgres redis
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
```

## 5. Create the Gitea admin account — do this before opening the firewall

**Important, confirmed by testing**: Gitea always allows the very first account to register through the web UI, *regardless* of `DISABLE_REGISTRATION=true` — this is intentional upstream behavior so there's a way to bootstrap an admin at all. `INSTALL_LOCK=true` only skips the database-setup wizard; it does not close this separate first-user loophole. If the firewall is already open and DNS is live when Gitea first boots, a stranger who reaches `/user/sign_up` before you do gets the admin account.

So: bring the stack up with the firewall still closed (or bring up everything except leave DNS unpointed / firewall rules not yet applied), then immediately:

```bash
docker compose -f docker-compose.prod.yml exec -u git gitea gitea admin user create \
  --username <you> --password '<a real password>' --email <you>@example.com --admin
```

Only open the firewall / point DNS at the box after this succeeds. Once any account exists, `/user/sign_up` correctly shows "Registration is disabled" with no working form (also confirmed by testing).

Then, as that admin, create the repo via the web UI (or `git push` to create it — Gitea supports push-to-create) and set it public.

## 6. Push access

Two ways to push, since Cloudflare's proxy only speaks HTTP(S) — raw SSH can't ride the orange-cloud proxy:

- **HTTPS + personal access token (primary)** — Gitea → Settings → Applications → generate a token, then `git remote set-url origin https://<token>@git.ciagent.org/<you>/ci-agent.git`. Rides the same Cloudflare-proxied 443 as normal traffic, no extra firewall port needed.
- **SSH on port 2222 (optional, secondary)** — `[email protected]:2222/<you>/ci-agent.git`, using the standard git SSH key flow. This bypasses Cloudflare entirely (firewalled to your own IP per §3), so it's a fallback for when you're the one connecting from a known IP, not a general-purpose access method.

Read-only clone/browse works for anyone, no account: `https://git.ciagent.org/<you>/ci-agent.git` or the web UI directly.

## 7. Database viewer (Adminer at db.ciagent.org)

Unlike Gitea's admin account, there's no manual credential-handoff step for this one — any account with `is_admin=true` on the app itself can use it immediately, once `NEXT_PUBLIC_DB_VIEWER_URL` is set (§4) and the site is redeployed. Settings → Database → "Open database viewer" mints a short-lived token from the admin's real, live-checked login, which `db.ciagent.org`'s Nginx block (`infrastructure/nginx/nginx.conf`) exchanges for a signed session cookie via `apps/api/app/api/v1/db_viewer.py` — no separate password to generate, distribute, or rotate.

A couple of things worth knowing:

- Sessions last 60 minutes. Revoking someone's `is_admin` flag takes effect on their very next request through Nginx's `auth_request` check (it re-loads the user from the database each time), but an already-open Adminer tab isn't force-closed — it just stops being able to load anything new once that check runs again.
- Adminer's own Postgres login (username/password) is a second, independent layer past this gate — real DB credentials are still required to actually view or edit anything.
- An optional, commented-out IP-allowlist snippet is included in the `db.ciagent.org` Nginx block for admins with a static IP who want to require both the session *and* a matching source address — not enabled by default, since most admin connections don't have a stable IP to pin to.

## 8. Post-boot: configure provider API keys via the Settings UI, not `.env`

**Where does the admin account come from?** The very first account ever registered on the app (`is_admin` count is zero) is automatically promoted to admin - see `auth_service.register`. No manual DB access or setup script required, but it does mean the same race Gitea has in §5 applies here too: if the firewall is open and the site is reachable before you've registered your own account, a stranger who registers first becomes the admin instead of you. Register your own account immediately after the stack comes up, before opening the firewall/pointing DNS at it publicly. If the last admin's account is ever deleted, the next person to register becomes admin again - it never permanently locks the deployment out of admin access.

`scripts/bootstrap-env.sh` deliberately leaves these six blank. Sign in as the admin account and set them from the app itself:

- **Settings → Your API keys**: Anthropic, Brave Search, NinjaPear, USPTO — stored encrypted per-user (`user_api_key_service.py`), fall back to any global `.env` value if you ever set one, but there's no need to.
- **Settings → Server secrets**: Cloudflare Turnstile site key + secret — stored in the DB (`system_secret_service.py`), overrides `.env` at runtime, admin-only.

Also set `LLM_PROVIDER=anthropic` and `SEARCH_PROVIDER=brave` in `.env` (restart `api`/`worker`/`beat` after) once you've configured the corresponding keys above — provider *selection* is still a deployment-level `.env` setting, only the key *values* moved to the UI.

## 9. Updating the deployment

Manually:

```bash
git pull
docker compose -f docker-compose.prod.yml build api worker beat web
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml restart nginx
```

**The `restart nginx` step is not optional.** `up -d` only recreates the containers whose image/config changed - nginx's own image/config doesn't change on an app deploy, so it keeps running with the container IPs it resolved at its *own* last startup. Once `api`/`web` get rebuilt with new IPs, nginx keeps proxying to the old, now-dead ones until something forces it to re-resolve - every request 502s (and shows up in the browser as a misleading CORS error, since a bare 502 from nginx carries no `Access-Control-Allow-Origin` header). Confirmed live: this caused a real multi-hour outage. `scripts/auto-deploy.sh` includes this step automatically.

**Or automatically**: `scripts/auto-deploy.sh` runs exactly that sequence, gated on "is `origin/master` ahead of `HEAD`" so it's a no-op most runs. Install it as a systemd timer (polls every 2 minutes — deliberately polling, not a Gitea webhook, so there's no extra exposed service, no Docker-socket-in-a-container, and no shared secret to manage):

```bash
cp infrastructure/systemd/ci-agent-deploy.service infrastructure/systemd/ci-agent-deploy.timer /etc/systemd/system/
chmod +x scripts/auto-deploy.sh
systemctl daemon-reload
systemctl enable --now ci-agent-deploy.timer
```

Once running, pushing to `master` on Gitea is enough — the server picks it up within ~2 minutes, no manual SSH step needed. Check `journalctl -u ci-agent-deploy.service` to see deploy runs.

## 10. Backups

Nothing backs itself up by default. At minimum, a nightly cron job on the host:

```bash
docker compose -f docker-compose.prod.yml exec -T postgres \
  pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > /backups/ciagent-$(date +%F).sql.gz

docker run --rm -v ci-agent-prod_gitea-data:/data -v /backups:/backup alpine \
  tar czf /backup/gitea-$(date +%F).tar.gz -C /data .
```

Copy `/backups` off-box (a Hetzner Storage Box via `rclone`, or any object storage) — local-only backups don't survive a lost disk. Retain a sane number of days and prune older ones.

## 11. Operational notes

- `beat` must stay exactly one instance, always — duplicate scheduled monitoring runs otherwise. Don't `--scale beat=2`.
- Docker's log driver is capped per-service (`max-size: 10m`, `max-file: 3` in `docker-compose.prod.yml`) — nothing else bounds log growth on the host.
- Postgres and Redis publish no host ports in prod (unlike the dev `docker-compose.yml`) — only reachable over the internal Compose network.
- The dev Dockerfiles/compose file (`docker-compose.yml`, `infrastructure/docker/*.Dockerfile` without `.prod`) are untouched by any of this and remain the local-development setup.