Add production deployment tooling: Nginx, Gitea, prod Dockerfiles
Sets up everything needed to deploy behind Cloudflare with a self-hosted git server: multi-stage prod Dockerfiles (non-root), docker-compose.prod.yml (Postgres/Redis with no host ports, Nginx reverse proxy, Gitea with public-read/admin-write access control), scripts/bootstrap-env.sh to auto-generate required secrets on first clone, and DEPLOYMENT.md covering the full runbook. Provider API keys (Anthropic/Brave/NinjaPear/USPTO/ Turnstile) are deliberately kept out of .env in favor of the existing DB-backed Settings UI, so the public repo stays safe to expose. Also fixes two bugs only surfaced by live-testing the prod stack: Celery beat couldn't write its schedule file as a non-root user, and Gitea's embedded SSH server conflicted with the base image's own sshd on port 22. Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
@@ -0,0 +1,45 @@
|
|||||||
|
# Docker's ignore-pattern matching is NOT recursive-by-default the way
|
||||||
|
# .gitignore's is - a bare `node_modules/` only matches a top-level
|
||||||
|
# directory of that name, not apps/web/node_modules. Every pattern below
|
||||||
|
# is prefixed with **/ so it matches at any depth (confirmed by testing:
|
||||||
|
# without **/, `apps/web/node_modules` alone still uploaded ~600MB of
|
||||||
|
# build context that was never actually needed by any Dockerfile).
|
||||||
|
|
||||||
|
**/.git/
|
||||||
|
**/.env
|
||||||
|
**/.env.local
|
||||||
|
**/*.pem
|
||||||
|
**/*.key
|
||||||
|
|
||||||
|
**/__pycache__/
|
||||||
|
**/*.py[cod]
|
||||||
|
**/*.egg-info/
|
||||||
|
**/.venv/
|
||||||
|
**/venv/
|
||||||
|
**/.pytest_cache/
|
||||||
|
**/.ruff_cache/
|
||||||
|
**/.mypy_cache/
|
||||||
|
**/htmlcov/
|
||||||
|
**/.coverage
|
||||||
|
**/*.db
|
||||||
|
**/*.sqlite3
|
||||||
|
|
||||||
|
**/node_modules/
|
||||||
|
**/.next/
|
||||||
|
**/out/
|
||||||
|
**/dist/
|
||||||
|
**/build/
|
||||||
|
**/.turbo/
|
||||||
|
**/coverage/
|
||||||
|
**/*.tsbuildinfo
|
||||||
|
**/.eslintcache
|
||||||
|
|
||||||
|
**/test-results/
|
||||||
|
**/playwright-report/
|
||||||
|
**/playwright/.cache/
|
||||||
|
|
||||||
|
**/.claude/
|
||||||
|
**/.vscode/
|
||||||
|
**/.idea/
|
||||||
|
**/*.log
|
||||||
|
**/logs/
|
||||||
+18
-1
@@ -16,6 +16,10 @@ BACKEND_URL=http://localhost:8000
|
|||||||
# so for LAN access this must be the host's LAN IP, not localhost (e.g.
|
# so for LAN access this must be the host's LAN IP, not localhost (e.g.
|
||||||
# http://192.168.1.190:8000). Leave unset for localhost-only access.
|
# http://192.168.1.190:8000). Leave unset for localhost-only access.
|
||||||
NEXT_PUBLIC_API_URL=http://localhost:8000
|
NEXT_PUBLIC_API_URL=http://localhost:8000
|
||||||
|
# "Git Repository" link on the landing page header, next to Sign in - only
|
||||||
|
# rendered when this is set (e.g. https://git.ciagent.org/you/ci-agent).
|
||||||
|
# Leave blank if you don't run a public git server for this deployment.
|
||||||
|
NEXT_PUBLIC_GIT_REPO_URL=
|
||||||
|
|
||||||
# --- Reverse proxy (only relevant once deployed behind Cloudflare/Nginx) -----
|
# --- Reverse proxy (only relevant once deployed behind Cloudflare/Nginx) -----
|
||||||
# Empty = trust the direct connection for client-IP resolution (correct for
|
# Empty = trust the direct connection for client-IP resolution (correct for
|
||||||
@@ -40,9 +44,22 @@ AUTH_MODE=local
|
|||||||
JWT_SECRET=dev-only-change-me-32-characters-minimum
|
JWT_SECRET=dev-only-change-me-32-characters-minimum
|
||||||
JWT_ACCESS_TOKEN_MINUTES=15
|
JWT_ACCESS_TOKEN_MINUTES=15
|
||||||
JWT_REFRESH_TOKEN_DAYS=7
|
JWT_REFRESH_TOKEN_DAYS=7
|
||||||
|
# Encrypts per-user API keys / server secrets at rest (Fernet - must stay
|
||||||
|
# exactly 32 raw bytes, urlsafe-base64-encoded). The value below is a real,
|
||||||
|
# working dev-only key so local Docker Compose functions out of the box;
|
||||||
|
# scripts/bootstrap-env.sh regenerates a fresh one for any real deployment.
|
||||||
|
# Rotating this in a deployment that already has stored keys makes them
|
||||||
|
# permanently undecryptable - treat it like any other production secret.
|
||||||
|
API_KEY_ENCRYPTION_SECRET=_wYtsm3nJ070987snBFp2eWVI5pyC0H9gGFUb6Cy4cQ=
|
||||||
|
|
||||||
# --- Database ----------------------------------------------------------------
|
# --- Database ----------------------------------------------------------------
|
||||||
# Postgres (Docker Compose default):
|
# Postgres (Docker Compose default) - POSTGRES_USER/PASSWORD/DB feed both the
|
||||||
|
# `postgres` container's own credentials and DATABASE_URL below (see
|
||||||
|
# docker-compose.yml) - change the password here, not in DATABASE_URL
|
||||||
|
# directly, or they'll disagree.
|
||||||
|
POSTGRES_USER=ciagent
|
||||||
|
POSTGRES_PASSWORD=ciagent
|
||||||
|
POSTGRES_DB=ciagent
|
||||||
DATABASE_URL=postgresql+psycopg://ciagent:ciagent@postgres:5432/ciagent
|
DATABASE_URL=postgresql+psycopg://ciagent:ciagent@postgres:5432/ciagent
|
||||||
# SQLite fallback for running the API without Docker:
|
# SQLite fallback for running the API without Docker:
|
||||||
# DATABASE_URL=sqlite+aiosqlite:///./ciagent_dev.db
|
# DATABASE_URL=sqlite+aiosqlite:///./ciagent_dev.db
|
||||||
|
|||||||
@@ -16,8 +16,12 @@ venv/
|
|||||||
htmlcov/
|
htmlcov/
|
||||||
.coverage
|
.coverage
|
||||||
*.db
|
*.db
|
||||||
|
*.db-journal
|
||||||
|
*.db-shm
|
||||||
|
*.db-wal
|
||||||
*.sqlite3
|
*.sqlite3
|
||||||
.fixture_state
|
.fixture_state
|
||||||
|
celerybeat-schedule
|
||||||
|
|
||||||
# --- Node / Next.js ---
|
# --- Node / Next.js ---
|
||||||
node_modules/
|
node_modules/
|
||||||
|
|||||||
+135
@@ -0,0 +1,135 @@
|
|||||||
|
# Deployment
|
||||||
|
|
||||||
|
Production runbook for deploying this app to a single Ubuntu server behind Cloudflare, with a self-hosted Gitea instance for code hosting. Written for `ciagent.org` on Hetzner Cloud, but nothing here is Hetzner-specific beyond the firewall step.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
One Docker host runs everything via `docker-compose.prod.yml`: Postgres, Redis, the API, a Celery worker, Celery beat, the Next.js frontend, Gitea, and an Nginx reverse proxy. Nginx is the only container with a host-published port (80/443); everything else is reached over the internal Compose network by service name. Three Cloudflare-proxied subdomains route to it:
|
||||||
|
|
||||||
|
| Subdomain | Routes to | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| `ciagent.org` | `web:3000` | Next.js frontend |
|
||||||
|
| `api.ciagent.org` | `api:8000` | FastAPI backend |
|
||||||
|
| `git.ciagent.org` | `gitea:3000` | Self-hosted git (public read, admin-only write) |
|
||||||
|
|
||||||
|
Cloudflare's proxy (orange-cloud DNS) hides the origin's real IP and absorbs generic bot/volumetric traffic; the app's own IP-based throttle/ban system (`app/services/ip_throttle_service.py`) handles the business-logic-specific abuse cases Cloudflare can't know about. Both need `TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP` set correctly or IP-based logic breaks — see `KNOWN_LIMITATIONS.md`.
|
||||||
|
|
||||||
|
## 1. Server prerequisites
|
||||||
|
|
||||||
|
- Ubuntu server with Docker Engine + the Compose plugin installed (`docker compose version` should work).
|
||||||
|
- A domain with its nameservers pointed at Cloudflare, and the zone added to a Cloudflare account.
|
||||||
|
- SSH access as a non-root user with Docker permissions (or root — your call, just don't run the app's own containers as root inside themselves, which they don't).
|
||||||
|
|
||||||
|
## 2. Cloudflare DNS + Origin CA certificate
|
||||||
|
|
||||||
|
1. Add three DNS **A records**, all proxied (orange cloud): `ciagent.org`, `api.ciagent.org`, `git.ciagent.org` → the server's public IP.
|
||||||
|
2. Cloudflare dashboard → SSL/TLS → **Origin Server** → Create Certificate. Cover `ciagent.org` and `*.ciagent.org` (one cert for all three subdomains), leave the default 15-year validity. Save the cert and private key.
|
||||||
|
3. On the server, create `/etc/ci-agent/certs/` (outside the repo, never committed) and place the two files there as `cloudflare-origin.pem` and `cloudflare-origin.key` — this is exactly what `docker-compose.prod.yml`'s `nginx` service mounts.
|
||||||
|
4. Cloudflare dashboard → SSL/TLS → set the encryption mode to **Full (strict)**. Anything less either skips origin verification or falls back to plaintext HTTP to the origin.
|
||||||
|
|
||||||
|
## 3. Hetzner Cloud Firewall (or equivalent network-level firewall)
|
||||||
|
|
||||||
|
Restrict inbound to the origin so nothing can bypass Cloudflare's protection by hitting the server's real IP directly:
|
||||||
|
|
||||||
|
- Allow tcp/80 and tcp/443 **only** from Cloudflare's published IP ranges: `https://www.cloudflare.com/ips-v4` and `/ips-v6`. Re-check these occasionally — they change rarely but do change.
|
||||||
|
- Allow tcp/22 (SSH) only from your own IP.
|
||||||
|
- Allow tcp/2222 (Gitea SSH push, optional — see §7) only from your own IP.
|
||||||
|
- Default-deny everything else inbound.
|
||||||
|
|
||||||
|
Prefer a network-level firewall (Hetzner Cloud Firewall) over host-only `ufw` — it blocks the packet before it reaches the box at all, so a host-firewall misconfiguration can't accidentally expose anything.
|
||||||
|
|
||||||
|
## 4. First boot
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone <your git remote> ci-agent # or `docker compose exec` your existing checkout
|
||||||
|
cd ci-agent
|
||||||
|
./scripts/bootstrap-env.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates `.env` from `.env.example` and generates real `JWT_SECRET`/`API_KEY_ENCRYPTION_SECRET`/`POSTGRES_PASSWORD` values. It deliberately leaves six things unset — see §8.
|
||||||
|
|
||||||
|
Then edit `.env` and set, at minimum:
|
||||||
|
|
||||||
|
```
|
||||||
|
APP_ENV=production
|
||||||
|
AUTH_MODE=jwt
|
||||||
|
FRONTEND_URL=https://ciagent.org
|
||||||
|
BACKEND_URL=https://api.ciagent.org
|
||||||
|
NEXT_PUBLIC_API_URL=https://api.ciagent.org
|
||||||
|
NEXT_PUBLIC_GIT_REPO_URL=https://git.ciagent.org/<your-admin-username>/ci-agent
|
||||||
|
TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP
|
||||||
|
RESEND_API_KEY=<real Resend key> # no admin-UI equivalent for this one
|
||||||
|
```
|
||||||
|
|
||||||
|
`APP_ENV=production` with `AUTH_MODE=local` will make the `api` container refuse to start (a deliberate crash-on-boot safety check, not a bug — see `app/core/config.py`'s validator) — if you see that, it means `AUTH_MODE` wasn't actually changed to `jwt`.
|
||||||
|
|
||||||
|
Bring up the database first, run migrations as a one-shot step (not baked into the long-running service), then the rest of the stack:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml up -d postgres redis
|
||||||
|
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
|
||||||
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Create the Gitea admin account — do this before opening the firewall
|
||||||
|
|
||||||
|
**Important, confirmed by testing**: Gitea always allows the very first account to register through the web UI, *regardless* of `DISABLE_REGISTRATION=true` — this is intentional upstream behavior so there's a way to bootstrap an admin at all. `INSTALL_LOCK=true` only skips the database-setup wizard; it does not close this separate first-user loophole. If the firewall is already open and DNS is live when Gitea first boots, a stranger who reaches `/user/sign_up` before you do gets the admin account.
|
||||||
|
|
||||||
|
So: bring the stack up with the firewall still closed (or bring up everything except leave DNS unpointed / firewall rules not yet applied), then immediately:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml exec -u git gitea gitea admin user create \
|
||||||
|
--username <you> --password '<a real password>' --email <you>@example.com --admin
|
||||||
|
```
|
||||||
|
|
||||||
|
Only open the firewall / point DNS at the box after this succeeds. Once any account exists, `/user/sign_up` correctly shows "Registration is disabled" with no working form (also confirmed by testing).
|
||||||
|
|
||||||
|
Then, as that admin, create the repo via the web UI (or `git push` to create it — Gitea supports push-to-create) and set it public.
|
||||||
|
|
||||||
|
## 6. Push access
|
||||||
|
|
||||||
|
Two ways to push, since Cloudflare's proxy only speaks HTTP(S) — raw SSH can't ride the orange-cloud proxy:
|
||||||
|
|
||||||
|
- **HTTPS + personal access token (primary)** — Gitea → Settings → Applications → generate a token, then `git remote set-url origin https://<token>@git.ciagent.org/<you>/ci-agent.git`. Rides the same Cloudflare-proxied 443 as normal traffic, no extra firewall port needed.
|
||||||
|
- **SSH on port 2222 (optional, secondary)** — `[email protected]:2222/<you>/ci-agent.git`, using the standard git SSH key flow. This bypasses Cloudflare entirely (firewalled to your own IP per §3), so it's a fallback for when you're the one connecting from a known IP, not a general-purpose access method.
|
||||||
|
|
||||||
|
Read-only clone/browse works for anyone, no account: `https://git.ciagent.org/<you>/ci-agent.git` or the web UI directly.
|
||||||
|
|
||||||
|
## 7. Post-boot: configure provider API keys via the Settings UI, not `.env`
|
||||||
|
|
||||||
|
`scripts/bootstrap-env.sh` deliberately leaves these six blank. Sign in as the admin account and set them from the app itself:
|
||||||
|
|
||||||
|
- **Settings → Your API keys**: Anthropic, Brave Search, NinjaPear, USPTO — stored encrypted per-user (`user_api_key_service.py`), fall back to any global `.env` value if you ever set one, but there's no need to.
|
||||||
|
- **Settings → Server secrets**: Cloudflare Turnstile site key + secret — stored in the DB (`system_secret_service.py`), overrides `.env` at runtime, admin-only.
|
||||||
|
|
||||||
|
Also set `LLM_PROVIDER=anthropic` and `SEARCH_PROVIDER=brave` in `.env` (restart `api`/`worker`/`beat` after) once you've configured the corresponding keys above — provider *selection* is still a deployment-level `.env` setting, only the key *values* moved to the UI.
|
||||||
|
|
||||||
|
## 8. Updating the deployment
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git pull
|
||||||
|
docker compose -f docker-compose.prod.yml build api worker beat web
|
||||||
|
docker compose -f docker-compose.prod.yml run --rm api alembic upgrade head
|
||||||
|
docker compose -f docker-compose.prod.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. Backups
|
||||||
|
|
||||||
|
Nothing backs itself up by default. At minimum, a nightly cron job on the host:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f docker-compose.prod.yml exec -T postgres \
|
||||||
|
pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB" | gzip > /backups/ciagent-$(date +%F).sql.gz
|
||||||
|
|
||||||
|
docker run --rm -v ci-agent-prod_gitea-data:/data -v /backups:/backup alpine \
|
||||||
|
tar czf /backup/gitea-$(date +%F).tar.gz -C /data .
|
||||||
|
```
|
||||||
|
|
||||||
|
Copy `/backups` off-box (a Hetzner Storage Box via `rclone`, or any object storage) — local-only backups don't survive a lost disk. Retain a sane number of days and prune older ones.
|
||||||
|
|
||||||
|
## 10. Operational notes
|
||||||
|
|
||||||
|
- `beat` must stay exactly one instance, always — duplicate scheduled monitoring runs otherwise. Don't `--scale beat=2`.
|
||||||
|
- Docker's log driver is capped per-service (`max-size: 10m`, `max-file: 3` in `docker-compose.prod.yml`) — nothing else bounds log growth on the host.
|
||||||
|
- Postgres and Redis publish no host ports in prod (unlike the dev `docker-compose.yml`) — only reachable over the internal Compose network.
|
||||||
|
- The dev Dockerfiles/compose file (`docker-compose.yml`, `infrastructure/docker/*.Dockerfile` without `.prod`) are untouched by any of this and remain the local-development setup.
|
||||||
Binary file not shown.
@@ -48,6 +48,16 @@ export default function LandingPage() {
|
|||||||
<div className="mx-auto flex max-w-6xl items-center justify-between px-6 py-4">
|
<div className="mx-auto flex max-w-6xl items-center justify-between px-6 py-4">
|
||||||
<span className="text-lg font-semibold tracking-tight text-slate-900">CI Agent</span>
|
<span className="text-lg font-semibold tracking-tight text-slate-900">CI Agent</span>
|
||||||
<nav className="flex items-center gap-3">
|
<nav className="flex items-center gap-3">
|
||||||
|
{process.env.NEXT_PUBLIC_GIT_REPO_URL && (
|
||||||
|
<a
|
||||||
|
href={process.env.NEXT_PUBLIC_GIT_REPO_URL}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="rounded-md px-3 py-2 text-sm font-medium text-slate-700 hover:bg-slate-100"
|
||||||
|
>
|
||||||
|
Git Repository
|
||||||
|
</a>
|
||||||
|
)}
|
||||||
<Link
|
<Link
|
||||||
href="/login"
|
href="/login"
|
||||||
className="rounded-md px-3 py-2 text-sm font-medium text-slate-700 hover:bg-slate-100"
|
className="rounded-md px-3 py-2 text-sm font-medium text-slate-700 hover:bg-slate-100"
|
||||||
|
|||||||
@@ -67,4 +67,19 @@ describe("LandingPage", () => {
|
|||||||
);
|
);
|
||||||
expect(screen.queryByText(/local mode/i)).not.toBeInTheDocument();
|
expect(screen.queryByText(/local mode/i)).not.toBeInTheDocument();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("hides the Git Repository link when no NEXT_PUBLIC_GIT_REPO_URL is configured", () => {
|
||||||
|
renderWithQueryClient(<LandingPage />);
|
||||||
|
expect(screen.queryByRole("link", { name: /git repository/i })).not.toBeInTheDocument();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("shows a Git Repository link opening in a new tab when configured", () => {
|
||||||
|
vi.stubEnv("NEXT_PUBLIC_GIT_REPO_URL", "https://git.ciagent.org/admin/ci-agent");
|
||||||
|
renderWithQueryClient(<LandingPage />);
|
||||||
|
|
||||||
|
const link = screen.getByRole("link", { name: /git repository/i });
|
||||||
|
expect(link).toHaveAttribute("href", "https://git.ciagent.org/admin/ci-agent");
|
||||||
|
expect(link).toHaveAttribute("target", "_blank");
|
||||||
|
expect(link).toHaveAttribute("rel", "noopener noreferrer");
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,172 @@
|
|||||||
|
# Production stack for ciagent.org - a standalone file, not merged with
|
||||||
|
# docker-compose.yml (the dev file). The two differ enough (nginx + gitea
|
||||||
|
# added, Postgres/Redis no longer host-published, prod Dockerfile targets,
|
||||||
|
# ${POSTGRES_PASSWORD} instead of a hardcoded literal) that an
|
||||||
|
# override-merge risked someone running plain `docker compose up` on the
|
||||||
|
# box and silently getting the dev config instead. See DEPLOYMENT.md for
|
||||||
|
# the full first-boot runbook.
|
||||||
|
#
|
||||||
|
# Run `scripts/bootstrap-env.sh` first so POSTGRES_PASSWORD/JWT_SECRET/
|
||||||
|
# API_KEY_ENCRYPTION_SECRET are real values, not .env.example's dev
|
||||||
|
# placeholders, before bringing this up.
|
||||||
|
|
||||||
|
name: ci-agent-prod
|
||||||
|
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: ${POSTGRES_USER:-ciagent}
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD must be set - run scripts/bootstrap-env.sh}
|
||||||
|
POSTGRES_DB: ${POSTGRES_DB:-ciagent}
|
||||||
|
volumes:
|
||||||
|
- postgres-data:/var/lib/postgresql/data
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-ciagent} -d ${POSTGRES_DB:-ciagent}"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 10
|
||||||
|
logging: &default-logging
|
||||||
|
driver: json-file
|
||||||
|
options:
|
||||||
|
max-size: "10m"
|
||||||
|
max-file: "3"
|
||||||
|
|
||||||
|
redis:
|
||||||
|
image: redis:7-alpine
|
||||||
|
restart: unless-stopped
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "redis-cli", "ping"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 10
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
# No ports published to the host for api/worker/beat/web/gitea below -
|
||||||
|
# nginx is the only internet-facing service; everything else is reached
|
||||||
|
# over the internal Compose network by service name.
|
||||||
|
|
||||||
|
api:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: infrastructure/docker/api.Dockerfile.prod
|
||||||
|
restart: unless-stopped
|
||||||
|
env_file: .env
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-ciagent}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-ciagent}
|
||||||
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
worker:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: infrastructure/docker/api.Dockerfile.prod
|
||||||
|
restart: unless-stopped
|
||||||
|
env_file: .env
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-ciagent}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-ciagent}
|
||||||
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
command: celery -A app.tasks.celery_app worker --loglevel=INFO -Q default,collection,analysis,notifications,maintenance,enrichment
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
beat:
|
||||||
|
# Must stay exactly one instance, always - duplicate scheduled runs
|
||||||
|
# otherwise. Don't `docker compose up --scale beat=N`.
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: infrastructure/docker/api.Dockerfile.prod
|
||||||
|
restart: unless-stopped
|
||||||
|
env_file: .env
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgresql+psycopg://${POSTGRES_USER:-ciagent}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-ciagent}
|
||||||
|
REDIS_URL: redis://redis:6379/0
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
redis:
|
||||||
|
condition: service_healthy
|
||||||
|
command: celery -A app.tasks.celery_app beat --loglevel=INFO
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
web:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: infrastructure/docker/web.Dockerfile.prod
|
||||||
|
args:
|
||||||
|
NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL}
|
||||||
|
NEXT_PUBLIC_GIT_REPO_URL: ${NEXT_PUBLIC_GIT_REPO_URL:-}
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- api
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
gitea:
|
||||||
|
image: gitea/gitea:1
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
GITEA__database__DB_TYPE: sqlite3
|
||||||
|
GITEA__database__PATH: /data/gitea/gitea.db
|
||||||
|
GITEA__server__DOMAIN: git.ciagent.org
|
||||||
|
GITEA__server__ROOT_URL: https://git.ciagent.org/
|
||||||
|
GITEA__server__SSH_DOMAIN: git.ciagent.org
|
||||||
|
GITEA__server__SSH_PORT: 2222
|
||||||
|
# Deliberately NOT setting GITEA__server__START_SSH_SERVER=true - that
|
||||||
|
# flag is for the rootless image variant only. This standard image
|
||||||
|
# already bundles its own system sshd for SSH clone/push, configured
|
||||||
|
# via the plain (unprefixed) SSH_PORT below - setting START_SSH_SERVER
|
||||||
|
# here makes Gitea's own embedded SSH server also try to bind the same
|
||||||
|
# port the system sshd already holds, crash-looping the container
|
||||||
|
# (confirmed live: "listen tcp :22: bind: address already in use").
|
||||||
|
# The system sshd binds container-internal port 22 by default, which
|
||||||
|
# is exactly what the "2222:22" host port mapping below expects.
|
||||||
|
SSH_PORT: 2222
|
||||||
|
# No public sign-ups - only the CLI-created admin account exists.
|
||||||
|
GITEA__service__DISABLE_REGISTRATION: "true"
|
||||||
|
# Allows anonymous browsing - actual visibility still comes from
|
||||||
|
# each repo's own public/private flag, so this alone exposes
|
||||||
|
# nothing by itself. See DEPLOYMENT.md.
|
||||||
|
GITEA__service__REQUIRE_SIGNIN_VIEW: "false"
|
||||||
|
# Skips the web install wizard from the very first boot - never
|
||||||
|
# leaves a window where an unauthenticated visitor could complete
|
||||||
|
# first-time setup themselves.
|
||||||
|
GITEA__security__INSTALL_LOCK: "true"
|
||||||
|
volumes:
|
||||||
|
- gitea-data:/data
|
||||||
|
ports:
|
||||||
|
# SSH push (optional, secondary to HTTPS+token push) - Cloudflare's
|
||||||
|
# proxy only speaks HTTP(S), so this rides straight to the origin.
|
||||||
|
# Firewall this port to your own IP only (see DEPLOYMENT.md).
|
||||||
|
- "2222:22"
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
nginx:
|
||||||
|
image: nginx:1.27-alpine
|
||||||
|
restart: unless-stopped
|
||||||
|
depends_on:
|
||||||
|
- web
|
||||||
|
- api
|
||||||
|
- gitea
|
||||||
|
volumes:
|
||||||
|
- ./infrastructure/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||||
|
# Cloudflare Origin CA cert/key, generated once via the Cloudflare
|
||||||
|
# dashboard - kept outside the repo entirely, never committed.
|
||||||
|
- /etc/ci-agent/certs:/etc/nginx/certs:ro
|
||||||
|
ports:
|
||||||
|
- "80:80"
|
||||||
|
- "443:443"
|
||||||
|
logging: *default-logging
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
postgres-data:
|
||||||
|
gitea-data:
|
||||||
@@ -97,6 +97,9 @@ services:
|
|||||||
# (not the container) can reach, so a LAN client needs this set to
|
# (not the container) can reach, so a LAN client needs this set to
|
||||||
# the host machine's LAN IP, not localhost. See .env.example.
|
# the host machine's LAN IP, not localhost. See .env.example.
|
||||||
NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL:-http://localhost:8000}
|
NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL:-http://localhost:8000}
|
||||||
|
# "Git Repository" landing-page link - blank by default so it's
|
||||||
|
# hidden unless you actually run a git server for this deployment.
|
||||||
|
NEXT_PUBLIC_GIT_REPO_URL: ${NEXT_PUBLIC_GIT_REPO_URL:-}
|
||||||
ports:
|
ports:
|
||||||
- "3000:3000"
|
- "3000:3000"
|
||||||
volumes:
|
volumes:
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Production image for api/worker/beat - no bind mounts, no --reload, no
|
||||||
|
# dev-only dependencies. See infrastructure/docker/api.Dockerfile for the
|
||||||
|
# dev image (kept separate and untouched).
|
||||||
|
|
||||||
|
FROM python:3.12-slim AS builder
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PIP_NO_CACHE_DIR=1
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
build-essential \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
RUN python -m venv /venv
|
||||||
|
ENV PATH="/venv/bin:$PATH"
|
||||||
|
|
||||||
|
COPY apps/api /app
|
||||||
|
RUN pip install --upgrade pip && pip install .
|
||||||
|
|
||||||
|
|
||||||
|
FROM python:3.12-slim AS runtime
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1 \
|
||||||
|
PATH="/venv/bin:$PATH"
|
||||||
|
|
||||||
|
RUN useradd --create-home --uid 1000 appuser
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=builder /venv /venv
|
||||||
|
# --chown so appuser can actually write here - celery beat needs to write
|
||||||
|
# its schedule state file (celerybeat-schedule) into the working directory,
|
||||||
|
# and a plain COPY leaves everything root-owned even after USER switches
|
||||||
|
# the running process to appuser.
|
||||||
|
COPY --chown=appuser:appuser apps/api /app
|
||||||
|
|
||||||
|
USER appuser
|
||||||
|
|
||||||
|
EXPOSE 8000
|
||||||
|
|
||||||
|
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# Production image for the Next.js frontend - built with `next build`
|
||||||
|
# (output: "standalone" in next.config.js), served with `node server.js`,
|
||||||
|
# not `next dev`/`npm start`. See infrastructure/docker/web.Dockerfile for
|
||||||
|
# the dev image (kept separate and untouched).
|
||||||
|
|
||||||
|
FROM node:20-alpine AS deps
|
||||||
|
WORKDIR /app
|
||||||
|
COPY apps/web/package.json apps/web/package-lock.json* ./
|
||||||
|
RUN npm ci
|
||||||
|
|
||||||
|
|
||||||
|
FROM node:20-alpine AS builder
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=deps /app/node_modules ./node_modules
|
||||||
|
COPY apps/web ./
|
||||||
|
# Baked into the browser bundle at build time - must be the public URL the
|
||||||
|
# *browser* will use, not a Docker-internal service name. See .env.example.
|
||||||
|
ARG NEXT_PUBLIC_API_URL
|
||||||
|
ARG NEXT_PUBLIC_GIT_REPO_URL
|
||||||
|
ENV NEXT_PUBLIC_API_URL=${NEXT_PUBLIC_API_URL} \
|
||||||
|
NEXT_PUBLIC_GIT_REPO_URL=${NEXT_PUBLIC_GIT_REPO_URL}
|
||||||
|
RUN npm run build
|
||||||
|
|
||||||
|
|
||||||
|
FROM node:20-alpine AS runner
|
||||||
|
WORKDIR /app
|
||||||
|
ENV NODE_ENV=production
|
||||||
|
RUN addgroup --system --gid 1001 nodejs && adduser --system --uid 1001 nextjs
|
||||||
|
|
||||||
|
# output:"standalone" already traces only the files actually needed at
|
||||||
|
# runtime - no full node_modules copy required.
|
||||||
|
COPY --from=builder /app/.next/standalone ./
|
||||||
|
COPY --from=builder /app/.next/static ./.next/static
|
||||||
|
COPY --from=builder /app/public ./public
|
||||||
|
|
||||||
|
USER nextjs
|
||||||
|
EXPOSE 3000
|
||||||
|
|
||||||
|
CMD ["node", "server.js"]
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
# Reverse proxy for the three ciagent.org subdomains, sitting between
|
||||||
|
# Cloudflare (which terminates public-facing TLS and hides this origin's
|
||||||
|
# real IP) and the app's own containers. TLS here is a Cloudflare Origin CA
|
||||||
|
# certificate (Cloudflare dashboard -> SSL/TLS -> Origin Server -> Create
|
||||||
|
# Certificate; covers ciagent.org + *.ciagent.org, up to 15yr validity, only
|
||||||
|
# trusted by Cloudflare - no ACME/renewal machinery needed). Cloudflare SSL
|
||||||
|
# mode must be "Full (strict)" for this to be meaningful. See DEPLOYMENT.md.
|
||||||
|
#
|
||||||
|
# CF-Connecting-IP (the header app.core.security.get_client_ip reads once
|
||||||
|
# TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP is set) needs no special handling
|
||||||
|
# here - Nginx forwards any header it doesn't explicitly touch straight
|
||||||
|
# through to the upstream unmodified.
|
||||||
|
|
||||||
|
user nginx;
|
||||||
|
worker_processes auto;
|
||||||
|
|
||||||
|
events {
|
||||||
|
worker_connections 1024;
|
||||||
|
}
|
||||||
|
|
||||||
|
http {
|
||||||
|
include mime.types;
|
||||||
|
default_type application/octet-stream;
|
||||||
|
sendfile on;
|
||||||
|
server_tokens off;
|
||||||
|
|
||||||
|
# Bare :80 hits get redirected to :443 - Cloudflare already enforces
|
||||||
|
# HTTPS at the edge, but the origin shouldn't 400 a direct :80 probe.
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
server_name ciagent.org api.ciagent.org git.ciagent.org;
|
||||||
|
return 301 https://$host$request_uri;
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 443 ssl;
|
||||||
|
server_name ciagent.org;
|
||||||
|
|
||||||
|
ssl_certificate /etc/nginx/certs/cloudflare-origin.pem;
|
||||||
|
ssl_certificate_key /etc/nginx/certs/cloudflare-origin.key;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://web:3000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 443 ssl;
|
||||||
|
server_name api.ciagent.org;
|
||||||
|
|
||||||
|
ssl_certificate /etc/nginx/certs/cloudflare-origin.pem;
|
||||||
|
ssl_certificate_key /etc/nginx/certs/cloudflare-origin.key;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://api:8000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 443 ssl;
|
||||||
|
server_name git.ciagent.org;
|
||||||
|
|
||||||
|
ssl_certificate /etc/nginx/certs/cloudflare-origin.pem;
|
||||||
|
ssl_certificate_key /etc/nginx/certs/cloudflare-origin.key;
|
||||||
|
|
||||||
|
# Large git pushes (Gitea's HTTPS push path).
|
||||||
|
client_max_body_size 512m;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://gitea:3000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Prepares a real .env for a fresh clone with minimal manual setup:
|
||||||
|
# 1. Copies .env.example -> .env if .env doesn't exist yet.
|
||||||
|
# 2. Replaces the three "must be a real random secret, no safe shared
|
||||||
|
# default" values (JWT_SECRET, API_KEY_ENCRYPTION_SECRET,
|
||||||
|
# POSTGRES_PASSWORD) with freshly generated ones, but ONLY if they
|
||||||
|
# still equal the known .env.example placeholder or are blank -
|
||||||
|
# running this again after you've customized .env is a no-op.
|
||||||
|
#
|
||||||
|
# Deliberately does NOT touch DATABASE_URL: docker-compose.prod.yml builds
|
||||||
|
# it from POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DB directly (same as the
|
||||||
|
# dev compose file already does with its hardcoded values), so there's
|
||||||
|
# nothing to keep in sync by hand here.
|
||||||
|
#
|
||||||
|
# Deliberately does NOT set ANTHROPIC_API_KEY / BRAVE_SEARCH_API_KEY /
|
||||||
|
# NINJAPEAR_API_KEY / USPTO_API_KEY / TURNSTILE_SITE_KEY / TURNSTILE_SECRET -
|
||||||
|
# those are meant to be configured after first boot via the app's own
|
||||||
|
# Settings page ("Your API keys" / "Server secrets"), not .env. See
|
||||||
|
# KNOWN_LIMITATIONS.md.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
ENV_FILE=".env"
|
||||||
|
ENV_EXAMPLE=".env.example"
|
||||||
|
|
||||||
|
if [ ! -f "$ENV_FILE" ]; then
|
||||||
|
if [ ! -f "$ENV_EXAMPLE" ]; then
|
||||||
|
echo "error: $ENV_EXAMPLE not found" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
cp "$ENV_EXAMPLE" "$ENV_FILE"
|
||||||
|
echo "Created $ENV_FILE from $ENV_EXAMPLE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# $1 = var name, $2 = known placeholder value in .env.example, $3 = generator command
|
||||||
|
_replace_if_placeholder() {
|
||||||
|
local var_name="$1" placeholder="$2" generator="$3"
|
||||||
|
local current
|
||||||
|
current=$(grep -E "^${var_name}=" "$ENV_FILE" | head -n1 | cut -d= -f2-)
|
||||||
|
|
||||||
|
if [ -z "$current" ] || [ "$current" = "$placeholder" ]; then
|
||||||
|
local new_value
|
||||||
|
new_value=$(eval "$generator")
|
||||||
|
# Escape characters that are special to sed's replacement text.
|
||||||
|
local escaped
|
||||||
|
escaped=$(printf '%s' "$new_value" | sed -e 's/[\/&]/\\&/g')
|
||||||
|
sed -i "s/^${var_name}=.*/${var_name}=${escaped}/" "$ENV_FILE"
|
||||||
|
echo "Generated a new ${var_name}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
_replace_if_placeholder "JWT_SECRET" "dev-only-change-me-32-characters-minimum" \
|
||||||
|
"openssl rand -hex 32"
|
||||||
|
|
||||||
|
# Fernet requires exactly 32 raw bytes, urlsafe-base64-encoded - plain hex
|
||||||
|
# would pass silently until the first encrypt/decrypt call, then crash.
|
||||||
|
_replace_if_placeholder "API_KEY_ENCRYPTION_SECRET" "_wYtsm3nJ070987snBFp2eWVI5pyC0H9gGFUb6Cy4cQ=" \
|
||||||
|
"openssl rand -base64 32 | tr '+/' '-_'"
|
||||||
|
|
||||||
|
_replace_if_placeholder "POSTGRES_PASSWORD" "ciagent" \
|
||||||
|
"openssl rand -hex 32"
|
||||||
|
|
||||||
|
cat <<'EOF'
|
||||||
|
|
||||||
|
Done. Before running this for real, still set manually in .env:
|
||||||
|
- APP_ENV=production, AUTH_MODE=jwt (AUTH_MODE=local refuses to start
|
||||||
|
when APP_ENV=production - this is an intentional safety check)
|
||||||
|
- FRONTEND_URL / NEXT_PUBLIC_API_URL / BACKEND_URL for your real domain
|
||||||
|
- TRUSTED_PROXY_IP_HEADER=CF-Connecting-IP once behind Cloudflare
|
||||||
|
- RESEND_API_KEY (or SMTP_*) for real security emails - no admin UI for this one
|
||||||
|
|
||||||
|
Once the app is running, sign in as the admin account and set these from
|
||||||
|
the Settings page instead of .env:
|
||||||
|
- Your API keys: Anthropic, Brave Search, NinjaPear, USPTO
|
||||||
|
- Server secrets: Cloudflare Turnstile site key + secret
|
||||||
|
EOF
|
||||||
Reference in New Issue
Block a user