From 086ca1f13ffcd77809d69a661554a80f54473997 Mon Sep 17 00:00:00 2001 From: Saksham Das Date: Wed, 5 Aug 2026 12:59:41 -0400 Subject: [PATCH] Add production deployment tooling: Nginx, Gitea, prod Dockerfiles Sets up everything needed to deploy behind Cloudflare with a self-hosted git server: multi-stage prod Dockerfiles (non-root), docker-compose.prod.yml (Postgres/Redis with no host ports, Nginx reverse proxy, Gitea with public-read/admin-write access control), scripts/bootstrap-env.sh to auto-generate required secrets on first clone, and DEPLOYMENT.md covering the full runbook. Provider API keys (Anthropic/Brave/NinjaPear/USPTO/ Turnstile) are deliberately kept out of .env in favor of the existing DB-backed Settings UI, so the public repo stays safe to expose. Also fixes two bugs only surfaced by live-testing the prod stack: Celery beat couldn't write its schedule file as a non-root user, and Gitea's embedded SSH server conflicted with the base image's own sshd on port 22. Co-Authored-By: Claude Sonnet 5 --- .dockerignore | 45 ++++++ .env.example | 19 ++- .gitignore | 4 + DEPLOYMENT.md | 135 +++++++++++++++++ apps/api/celerybeat-schedule | Bin 4096 -> 0 bytes apps/web/app/page.tsx | 10 ++ apps/web/public/.gitkeep | 0 apps/web/tests/landing-page.test.tsx | 15 ++ docker-compose.prod.yml | 172 ++++++++++++++++++++++ docker-compose.yml | 3 + infrastructure/docker/api.Dockerfile.prod | 44 ++++++ infrastructure/docker/web.Dockerfile.prod | 39 +++++ infrastructure/nginx/nginx.conf | 82 +++++++++++ scripts/bootstrap-env.sh | 77 ++++++++++ 14 files changed, 644 insertions(+), 1 deletion(-) create mode 100644 .dockerignore create mode 100644 DEPLOYMENT.md delete mode 100644 apps/api/celerybeat-schedule create mode 100644 apps/web/public/.gitkeep create mode 100644 docker-compose.prod.yml create mode 100644 infrastructure/docker/api.Dockerfile.prod create mode 100644 infrastructure/docker/web.Dockerfile.prod create mode 100644 infrastructure/nginx/nginx.conf create mode 100644 scripts/bootstrap-env.sh diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..b07823b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,45 @@ +# Docker's ignore-pattern matching is NOT recursive-by-default the way +# .gitignore's is - a bare `node_modules/` only matches a top-level +# directory of that name, not apps/web/node_modules. Every pattern below +# is prefixed with **/ so it matches at any depth (confirmed by testing: +# without **/, `apps/web/node_modules` alone still uploaded ~600MB of +# build context that was never actually needed by any Dockerfile). + +**/.git/ +**/.env +**/.env.local +**/*.pem +**/*.key + +**/__pycache__/ +**/*.py[cod] +**/*.egg-info/ +**/.venv/ +**/venv/ +**/.pytest_cache/ +**/.ruff_cache/ +**/.mypy_cache/ +**/htmlcov/ +**/.coverage +**/*.db +**/*.sqlite3 + +**/node_modules/ +**/.next/ +**/out/ +**/dist/ +**/build/ +**/.turbo/ +**/coverage/ +**/*.tsbuildinfo +**/.eslintcache + +**/test-results/ +**/playwright-report/ +**/playwright/.cache/ + +**/.claude/ +**/.vscode/ +**/.idea/ +**/*.log +**/logs/ diff --git a/.env.example b/.env.example index b6d1106..56196ad 100644 --- a/.env.example +++ b/.env.example @@ -16,6 +16,10 @@ BACKEND_URL=http://localhost:8000 # so for LAN access this must be the host's LAN IP, not localhost (e.g. # http://192.168.1.190:8000). Leave unset for localhost-only access. NEXT_PUBLIC_API_URL=http://localhost:8000 +# "Git Repository" link on the landing page header, next to Sign in - only +# rendered when this is set (e.g. https://git.ciagent.org/you/ci-agent). +# Leave blank if you don't run a public git server for this deployment. +NEXT_PUBLIC_GIT_REPO_URL= # --- Reverse proxy (only relevant once deployed behind Cloudflare/Nginx) ----- # Empty = trust the direct connection for client-IP resolution (correct for @@ -40,9 +44,22 @@ AUTH_MODE=local JWT_SECRET=dev-only-change-me-32-characters-minimum JWT_ACCESS_TOKEN_MINUTES=15 JWT_REFRESH_TOKEN_DAYS=7 +# Encrypts per-user API keys / server secrets at rest (Fernet - must stay +# exactly 32 raw bytes, urlsafe-base64-encoded). The value below is a real, +# working dev-only key so local Docker Compose functions out of the box; +# scripts/bootstrap-env.sh regenerates a fresh one for any real deployment. +# Rotating this in a deployment that already has stored keys makes them +# permanently undecryptable - treat it like any other production secret. +API_KEY_ENCRYPTION_SECRET=_wYtsm3nJ070987snBFp2eWVI5pyC0H9gGFUb6Cy4cQ= # --- Database ---------------------------------------------------------------- -# Postgres (Docker Compose default): +# Postgres (Docker Compose default) - POSTGRES_USER/PASSWORD/DB feed both the +# `postgres` container's own credentials and DATABASE_URL below (see +# docker-compose.yml) - change the password here, not in DATABASE_URL +# directly, or they'll disagree. +POSTGRES_USER=ciagent +POSTGRES_PASSWORD=ciagent +POSTGRES_DB=ciagent DATABASE_URL=postgresql+psycopg://ciagent:ciagent@postgres:5432/ciagent # SQLite fallback for running the API without Docker: # DATABASE_URL=sqlite+aiosqlite:///./ciagent_dev.db diff --git a/.gitignore b/.gitignore index 094375a..aaaad17 100644 --- a/.gitignore +++ b/.gitignore @@ -16,8 +16,12 @@ venv/ htmlcov/ .coverage *.db +*.db-journal +*.db-shm +*.db-wal *.sqlite3 .fixture_state +celerybeat-schedule # --- Node / Next.js --- node_modules/ diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md new file mode 100644 index 0000000..0945b01 --- /dev/null +++ b/DEPLOYMENT.md @@ -0,0 +1,135 @@ +# Deployment + +Production runbook for deploying this app to a single Ubuntu server behind Cloudflare, with a self-hosted Gitea instance for code hosting. Written for `ciagent.org` on Hetzner Cloud, but nothing here is Hetzner-specific beyond the firewall step. + +## Architecture + +One Docker host runs everything via `docker-compose.prod.yml`: Postgres, Redis, the API, a Celery worker, Celery beat, the Next.js frontend, Gitea, and an Nginx reverse proxy. Nginx is the only container with a host-published port (80/443); everything else is reached over the internal Compose network by service name. Three Cloudflare-proxied subdomains route to it: + +| Subdomain | Routes to | Purpose | +|---|---|---| +| `ciagent.org` | `web:3000` | Next.js frontend | +| `api.ciagent.org` | `api:8000` | FastAPI backend | +| `git.ciagent.org` | `gitea:3000` | Self-hosted git (public read, admin-only write) | + +Cloudflare's proxy (orange-cloud DNS) hides the origin's real IP and absorbs generic bot/volumetric traffic; the app's own IP-based throttle/ban system (`app/services/ip_throttle_service.py`) handles the business-logic-specific abuse cases Cloudflare can't know about. Both need `TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP` set correctly or IP-based logic breaks — see `KNOWN_LIMITATIONS.md`. + +## 1. Server prerequisites + +- Ubuntu server with Docker Engine + the Compose plugin installed (`docker compose version` should work). +- A domain with its nameservers pointed at Cloudflare, and the zone added to a Cloudflare account. +- SSH access as a non-root user with Docker permissions (or root — your call, just don't run the app's own containers as root inside themselves, which they don't). + +## 2. Cloudflare DNS + Origin CA certificate + +1. Add three DNS **A records**, all proxied (orange cloud): `ciagent.org`, `api.ciagent.org`, `git.ciagent.org` → the server's public IP. +2. Cloudflare dashboard → SSL/TLS → **Origin Server** → Create Certificate. Cover `ciagent.org` and `*.ciagent.org` (one cert for all three subdomains), leave the default 15-year validity. Save the cert and private key. +3. On the server, create `/etc/ci-agent/certs/` (outside the repo, never committed) and place the two files there as `cloudflare-origin.pem` and `cloudflare-origin.key` — this is exactly what `docker-compose.prod.yml`'s `nginx` service mounts. +4. Cloudflare dashboard → SSL/TLS → set the encryption mode to **Full (strict)**. Anything less either skips origin verification or falls back to plaintext HTTP to the origin. + +## 3. Hetzner Cloud Firewall (or equivalent network-level firewall) + +Restrict inbound to the origin so nothing can bypass Cloudflare's protection by hitting the server's real IP directly: + +- Allow tcp/80 and tcp/443 **only** from Cloudflare's published IP ranges: `https://www.cloudflare.com/ips-v4` and `/ips-v6`. Re-check these occasionally — they change rarely but do change. +- Allow tcp/22 (SSH) only from your own IP. +- Allow tcp/2222 (Gitea SSH push, optional — see §7) only from your own IP. +- Default-deny everything else inbound. + +Prefer a network-level firewall (Hetzner Cloud Firewall) over host-only `ufw` — it blocks the packet before it reaches the box at all, so a host-firewall misconfiguration can't accidentally expose anything. + +## 4. First boot + +```bash +git clone ci-agent # or `docker compose exec` your existing checkout +cd ci-agent +./scripts/bootstrap-env.sh +``` + +This creates `.env` from `.env.example` and generates real `JWT_SECRET`/`API_KEY_ENCRYPTION_SECRET`/`POSTGRES_PASSWORD` values. It deliberately leaves six things unset — see §8. + +Then edit `.env` and set, at minimum: + +``` +APP_ENV=production +AUTH_MODE=jwt +FRONTEND_URL=https://ciagent.org +BACKEND_URL=https://api.ciagent.org +NEXT_PUBLIC_API_URL=https://api.ciagent.org +NEXT_PUBLIC_GIT_REPO_URL=https://git.ciagent.org//ci-agent +TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP +RESEND_API_KEY= # no admin-UI equivalent for this one +``` + +`APP_ENV=production` with `AUTH_MODE=local` will make the `api` container refuse to start (a deliberate crash-on-boot safety check, not a bug — see `app/core/config.py`'s validator) — if you see that, it means `AUTH_MODE` wasn't actually changed to `jwt`. + +Bring up the database first, run migrations as a one-shot step (not baked into the long-running service), then the rest of the stack: + +```bash +docker compose -f docker-compose.prod.yml up -d postgres redis +docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head +docker compose -f docker-compose.prod.yml up -d +``` + +## 5. Create the Gitea admin account — do this before opening the firewall + +**Important, confirmed by testing**: Gitea always allows the very first account to register through the web UI, *regardless* of `DISABLE_REGISTRATION=true` — this is intentional upstream behavior so there's a way to bootstrap an admin at all. `INSTALL_LOCK=true` only skips the database-setup wizard; it does not close this separate first-user loophole. If the firewall is already open and DNS is live when Gitea first boots, a stranger who reaches `/user/sign_up` before you do gets the admin account. + +So: bring the stack up with the firewall still closed (or bring up everything except leave DNS unpointed / firewall rules not yet applied), then immediately: + +```bash +docker compose -f docker-compose.prod.yml exec -u git gitea gitea admin user create \ + --username --password '' --email @example.com --admin +``` + +Only open the firewall / point DNS at the box after this succeeds. Once any account exists, `/user/sign_up` correctly shows "Registration is disabled" with no working form (also confirmed by testing). + +Then, as that admin, create the repo via the web UI (or `git push` to create it — Gitea supports push-to-create) and set it public. + +## 6. Push access + +Two ways to push, since Cloudflare's proxy only speaks HTTP(S) — raw SSH can't ride the orange-cloud proxy: + +- **HTTPS + personal access token (primary)** — Gitea → Settings → Applications → generate a token, then `git remote set-url origin https://@git.ciagent.org//ci-agent.git`. Rides the same Cloudflare-proxied 443 as normal traffic, no extra firewall port needed. +- **SSH on port 2222 (optional, secondary)** — `git@git.ciagent.org:2222//ci-agent.git`, using the standard git SSH key flow. This bypasses Cloudflare entirely (firewalled to your own IP per §3), so it's a fallback for when you're the one connecting from a known IP, not a general-purpose access method. + +Read-only clone/browse works for anyone, no account: `https://git.ciagent.org//ci-agent.git` or the web UI directly. + +## 7. Post-boot: configure provider API keys via the Settings UI, not `.env` + +`scripts/bootstrap-env.sh` deliberately leaves these six blank. Sign in as the admin account and set them from the app itself: + +- **Settings → Your API keys**: Anthropic, Brave Search, NinjaPear, USPTO — stored encrypted per-user (`user_api_key_service.py`), fall back to any global `.env` value if you ever set one, but there's no need to. +- **Settings → Server secrets**: Cloudflare Turnstile site key + secret — stored in the DB (`system_secret_service.py`), overrides `.env` at runtime, admin-only. + +Also set `LLM_PROVIDER=anthropic` and `SEARCH_PROVIDER=brave` in `.env` (restart `api`/`worker`/`beat` after) once you've configured the corresponding keys above — provider *selection* is still a deployment-level `.env` setting, only the key *values* moved to the UI. + +## 8. Updating the deployment + +```bash +git pull +docker compose -f docker-compose.prod.yml build api worker beat web +docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head +docker compose -f docker-compose.prod.yml up -d +``` + +## 9. Backups + +Nothing backs itself up by default. At minimum, a nightly cron job on the host: + +```bash +docker compose -f docker-compose.prod.yml exec -T postgres \ + pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > /backups/ciagent-$(date +%F).sql.gz + +docker run --rm -v ci-agent-prod_gitea-data:/data -v /backups:/backup alpine \ + tar czf /backup/gitea-$(date +%F).tar.gz -C /data . +``` + +Copy `/backups` off-box (a Hetzner Storage Box via `rclone`, or any object storage) — local-only backups don't survive a lost disk. Retain a sane number of days and prune older ones. + +## 10. Operational notes + +- `beat` must stay exactly one instance, always — duplicate scheduled monitoring runs otherwise. Don't `--scale beat=2`. +- Docker's log driver is capped per-service (`max-size: 10m`, `max-file: 3` in `docker-compose.prod.yml`) — nothing else bounds log growth on the host. +- Postgres and Redis publish no host ports in prod (unlike the dev `docker-compose.yml`) — only reachable over the internal Compose network. +- The dev Dockerfiles/compose file (`docker-compose.yml`, `infrastructure/docker/*.Dockerfile` without `.prod`) are untouched by any of this and remain the local-development setup. diff --git a/apps/api/celerybeat-schedule b/apps/api/celerybeat-schedule deleted file mode 100644 index b9e9814be42d3500a4f195478e221e0a48034abf..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 4096 zcmeHHO-~a+7@pECZGo*={6bL@1gYB2L8FPr#2EBo8i}zO6Ax^*+p)W$Oq<=Qv6v=3 zi6nDlxcDDTj6Xonc<~2-Cf6g<&;>xV zb0ciGvi-s~2LOgRY{o^5w?1iF!Gp^GdO(6v<*(`zj46MmSAv@I=c5w5RDLe5z`og; z3~1_@eJc~O$MWDm_ew}Ue^UMAIpu?xjIa29kMvi4UkSlhzE~yB`p%o&;pfp@_%eVn zaX~`Hg)FS`Gd7#Q+v6+CpT-U2G3rzK7P5GV_Y>2vPgRL!TwfROl()_D@b=ZzF)|0h zd9{y__$2Qeo5;C?yoOe1P~6A$oH90yhPLDjF^yb;m*GVQ$sCU=xdt?Re`^iuZ@hlq^swLzx+H0W_a%(V;Ov0$1Q(tcJ zTh-ViOW7ehVpb}UnBI&B{e>$cLM~jG CI Agent