CIAgent/DEPLOYMENT.md

# Deployment

Production runbook for deploying this app to a single Ubuntu server behind Cloudflare, with a self-hosted Gitea instance for code hosting. Written for `ciagent.org` on Hetzner Cloud, but nothing here is Hetzner-specific beyond the firewall step.

## Architecture

One Docker host runs everything via `docker-compose.prod.yml`: Postgres, Redis, the API, a Celery worker, Celery beat, the Next.js frontend, Gitea, and an Nginx reverse proxy. Nginx is the only container with a host-published port (80/443); everything else is reached over the internal Compose network by service name. Three Cloudflare-proxied subdomains route to it:

| Subdomain | Routes to | Purpose |
|---|---|---|
| `ciagent.org` | `web:3000` | Next.js frontend |
| `api.ciagent.org` | `api:8000` | FastAPI backend |
| `git.ciagent.org` | `gitea:3000` | Self-hosted git (public read, admin-only write) |

Cloudflare's proxy (orange-cloud DNS) hides the origin's real IP and absorbs generic bot/volumetric traffic; the app's own IP-based throttle/ban system (`app/services/ip_throttle_service.py`) handles the business-logic-specific abuse cases Cloudflare can't know about. Both need `TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP` set correctly or IP-based logic breaks — see `KNOWN_LIMITATIONS.md`.

## 1. Server prerequisites

- Ubuntu server with Docker Engine + the Compose plugin installed (`docker compose version` should work).
- A domain with its nameservers pointed at Cloudflare, and the zone added to a Cloudflare account.
- SSH access as a non-root user with Docker permissions (or root — your call, just don't run the app's own containers as root inside themselves, which they don't).

## 2. Cloudflare DNS + Origin CA certificate

1. Add three DNS **A records**, all proxied (orange cloud): `ciagent.org`, `api.ciagent.org`, `git.ciagent.org` → the server's public IP.
2. Cloudflare dashboard → SSL/TLS → **Origin Server** → Create Certificate. Cover `ciagent.org` and `*.ciagent.org` (one cert for all three subdomains), leave the default 15-year validity. Save the cert and private key.
3. On the server, create `/etc/ci-agent/certs/` (outside the repo, never committed) and place the two files there as `cloudflare-origin.pem` and `cloudflare-origin.key` — this is exactly what `docker-compose.prod.yml`'s `nginx` service mounts.
4. Cloudflare dashboard → SSL/TLS → set the encryption mode to **Full (strict)**. Anything less either skips origin verification or falls back to plaintext HTTP to the origin.

## 3. Hetzner Cloud Firewall (or equivalent network-level firewall)

Restrict inbound to the origin so nothing can bypass Cloudflare's protection by hitting the server's real IP directly:

- Allow tcp/80 and tcp/443 **only** from Cloudflare's published IP ranges: `https://www.cloudflare.com/ips-v4` and `/ips-v6`. Re-check these occasionally — they change rarely but do change.
- Allow tcp/22 (SSH) only from your own IP.
- Allow tcp/2222 (Gitea SSH push, optional — see §7) only from your own IP.
- Default-deny everything else inbound.

Prefer a network-level firewall (Hetzner Cloud Firewall) over host-only `ufw` — it blocks the packet before it reaches the box at all, so a host-firewall misconfiguration can't accidentally expose anything.

## 4. First boot

```bash
git clone <your git remote> ci-agent   # or `docker compose exec` your existing checkout
cd ci-agent
./scripts/bootstrap-env.sh
```

This creates `.env` from `.env.example` and generates real `JWT_SECRET`/`API_KEY_ENCRYPTION_SECRET`/`POSTGRES_PASSWORD` values. It deliberately leaves six things unset — see §8.

Then edit `.env` and set, at minimum:

```
APP_ENV=production
AUTH_MODE=jwt
FRONTEND_URL=https://ciagent.org
BACKEND_URL=https://api.ciagent.org
NEXT_PUBLIC_API_URL=https://api.ciagent.org
NEXT_PUBLIC_GIT_REPO_URL=https://git.ciagent.org/<your-admin-username>/ci-agent
TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP
RESEND_API_KEY=<real Resend key>          # no admin-UI equivalent for this one
```

`APP_ENV=production` with `AUTH_MODE=local` will make the `api` container refuse to start (a deliberate crash-on-boot safety check, not a bug — see `app/core/config.py`'s validator) — if you see that, it means `AUTH_MODE` wasn't actually changed to `jwt`.

Bring up the database first, run migrations as a one-shot step (not baked into the long-running service), then the rest of the stack:

```bash
docker compose -f docker-compose.prod.yml up -d postgres redis
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
```

## 5. Create the Gitea admin account — do this before opening the firewall

**Important, confirmed by testing**: Gitea always allows the very first account to register through the web UI, *regardless* of `DISABLE_REGISTRATION=true` — this is intentional upstream behavior so there's a way to bootstrap an admin at all. `INSTALL_LOCK=true` only skips the database-setup wizard; it does not close this separate first-user loophole. If the firewall is already open and DNS is live when Gitea first boots, a stranger who reaches `/user/sign_up` before you do gets the admin account.

So: bring the stack up with the firewall still closed (or bring up everything except leave DNS unpointed / firewall rules not yet applied), then immediately:

```bash
docker compose -f docker-compose.prod.yml exec -u git gitea gitea admin user create \
  --username <you> --password '<a real password>' --email <you>@example.com --admin
```

Only open the firewall / point DNS at the box after this succeeds. Once any account exists, `/user/sign_up` correctly shows "Registration is disabled" with no working form (also confirmed by testing).

Then, as that admin, create the repo via the web UI (or `git push` to create it — Gitea supports push-to-create) and set it public.

## 6. Push access

Two ways to push, since Cloudflare's proxy only speaks HTTP(S) — raw SSH can't ride the orange-cloud proxy:

- **HTTPS + personal access token (primary)** — Gitea → Settings → Applications → generate a token, then `git remote set-url origin https://<token>@git.ciagent.org/<you>/ci-agent.git`. Rides the same Cloudflare-proxied 443 as normal traffic, no extra firewall port needed.
- **SSH on port 2222 (optional, secondary)** — `[email protected]:2222/<you>/ci-agent.git`, using the standard git SSH key flow. This bypasses Cloudflare entirely (firewalled to your own IP per §3), so it's a fallback for when you're the one connecting from a known IP, not a general-purpose access method.

Read-only clone/browse works for anyone, no account: `https://git.ciagent.org/<you>/ci-agent.git` or the web UI directly.

## 7. Post-boot: configure provider API keys via the Settings UI, not `.env`

`scripts/bootstrap-env.sh` deliberately leaves these six blank. Sign in as the admin account and set them from the app itself:

- **Settings → Your API keys**: Anthropic, Brave Search, NinjaPear, USPTO — stored encrypted per-user (`user_api_key_service.py`), fall back to any global `.env` value if you ever set one, but there's no need to.
- **Settings → Server secrets**: Cloudflare Turnstile site key + secret — stored in the DB (`system_secret_service.py`), overrides `.env` at runtime, admin-only.

Also set `LLM_PROVIDER=anthropic` and `SEARCH_PROVIDER=brave` in `.env` (restart `api`/`worker`/`beat` after) once you've configured the corresponding keys above — provider *selection* is still a deployment-level `.env` setting, only the key *values* moved to the UI.

## 8. Updating the deployment

```bash
git pull
docker compose -f docker-compose.prod.yml build api worker beat web
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.prod.yml up -d
```

## 9. Backups

Nothing backs itself up by default. At minimum, a nightly cron job on the host:

```bash
docker compose -f docker-compose.prod.yml exec -T postgres \
  pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > /backups/ciagent-$(date +%F).sql.gz

docker run --rm -v ci-agent-prod_gitea-data:/data -v /backups:/backup alpine \
  tar czf /backup/gitea-$(date +%F).tar.gz -C /data .
```

Copy `/backups` off-box (a Hetzner Storage Box via `rclone`, or any object storage) — local-only backups don't survive a lost disk. Retain a sane number of days and prune older ones.

## 10. Operational notes

- `beat` must stay exactly one instance, always — duplicate scheduled monitoring runs otherwise. Don't `--scale beat=2`.
- Docker's log driver is capped per-service (`max-size: 10m`, `max-file: 3` in `docker-compose.prod.yml`) — nothing else bounds log growth on the host.
- Postgres and Redis publish no host ports in prod (unlike the dev `docker-compose.yml`) — only reachable over the internal Compose network.
- The dev Dockerfiles/compose file (`docker-compose.yml`, `infrastructure/docker/*.Dockerfile` without `.prod`) are untouched by any of this and remain the local-development setup.