# Security

## Threat model summary

This app fetches user-submitted and discovered URLs from the public internet on a schedule, stores API keys for third-party providers, and sends emails/SMS. The main risks are: SSRF via URL fetching, credential leakage, cross-user data access, and abuse of the scraping/LLM/SMS pipeline as a cost or spam vector.

## SSRF prevention (`app/core/http.py::safe_fetch`)

All outbound collector/custom-URL requests go through one function. It:

- Only allows `http`/`https` schemes (no `file://`, `ftp://`, `gopher://`, etc.).
- Resolves DNS itself before connecting, and rejects the request if any resolved address is loopback, link-local, private (RFC1918/RFC4193), multicast, or a known cloud metadata address (`169.254.169.254`, etc.).
- Disables automatic redirect following in the HTTP client and instead re-validates each redirect target against the same rules before following it, up to a small max-hop count.
- Applies a per-request timeout and a per-domain rate limit / delay (`SCRAPER_DOMAIN_DELAY_SECONDS`).
- Sends a configurable, identifying `User-Agent` (`SCRAPER_USER_AGENT`) rather than impersonating a browser.

No collector or route handler is allowed to call `httpx`/`aiohttp` directly for a user- or discovery-supplied URL — code review should reject that pattern. The guard has zero exceptions: every resolved private/loopback/link-local/metadata address is rejected unconditionally, with no allowlist.

## Secrets

- All secrets are read from environment variables via `app/core/config.py`. Nothing is hard-coded.
- `.env` is git-ignored; `.env.example` ships with empty/placeholder values only.
- Secrets are never logged. The structured logger has a redaction filter for known secret-shaped keys (`*_key`, `*_secret`, `*_token`, `*password*`).
- The frontend never receives backend API keys; all third-party calls happen server-side.
- `NotificationDestination.destination_value` (email/phone) is treated as PII, not a secret, but is still excluded from audit logs beyond a masked form.

## AuthN/AuthZ

- Passwords hashed with Argon2 (`passlib[argon2]`), never stored or logged in plaintext.
- JWT access tokens are short-lived (default 15 min); refresh tokens are longer-lived, stored hashed, and rotated on use.
- Every service-layer query that loads a `Company`, `Report`, `Alert`, `Source`, or `NotificationDestination` filters by `owner_user_id` — enforced in the repository layer, not just the route layer, so a missed check in one route can't leak data.
- Auth endpoints are rate-limited (`slowapi`) to blunt credential stuffing. Per-IP request-rate limits also apply to every other mutation endpoint that creates data or costs money to call — company/source/notification-destination creation, report generation (LLM call), notification-destination test-send (real SMTP/Twilio call), and run-now — on top of (not instead of) the business-logic caps in "Cost/abuse controls" below. See `@limiter.limit(...)` usages across `app/api/v1/*.py`.
- `AUTH_MODE=local` is for local development only; the app refuses to start in this mode when `APP_ENV=production`.
- **Notification destinations are not verified.** `NotificationDestination.verified` exists on the model but nothing ever sets it to `true` - a destination is usable for real alert sends as soon as it's created, with no confirmation-code/link flow. Combined with SMS being real-money-per-message once `NOTIFICATION_SMS_ENABLED=true` and Twilio is configured, this means a user could point alerts at a phone number or inbox they don't control. Rate limiting on the `/test` endpoint (above) narrows the abuse window but doesn't close it; a verification flow is the real fix and is tracked in `KNOWN_LIMITATIONS.md` as unimplemented.

## Ethical/legal collection rules

Enforced by the collector layer, not left to convention:

1. `robots.txt` is fetched and honored before crawling a domain (website collector).
2. No collector attempts to bypass a CAPTCHA, login wall, paywall, or anti-bot control. If a source requires that, the `Source.status` is set to `AUTH_REQUIRED`/`BLOCKED_BY_POLICY` and the failure is recorded, never silently skipped or faked.
3. Only publicly reachable URLs are collected — `safe_fetch` also serves as the enforcement point for "no private/internal targets."
4. Per-domain delay + retry-with-backoff avoids hammering a source.
5. Content hashing avoids redundant re-fetches of unchanged pages.
6. A run that partially fails is marked `partial`, not silently reported as complete — the report generator is told which sources failed so it doesn't imply completeness it doesn't have.

## Input validation & injection

- All request/response bodies are Pydantic models with explicit types and length limits.
- SQLAlchemy ORM with parameter binding throughout — no raw string-interpolated SQL.
- LLM output is only ever deserialized into a strict Pydantic schema (`generate_structured`); free-text LLM output is never concatenated into a shell command, SQL string, or HTML without escaping, and is rendered in the frontend as text, not `dangerouslySetInnerHTML`.
- Request bodies are size-limited at the ASGI layer.

## Cost/abuse controls

Per-user limits (configurable, enforced in `services/`): max monitored companies, max manual "run now" triggers per day, max pages crawled per run (`MAX_PAGES_PER_DOMAIN`), max LLM tokens per request, max LLM retries, monthly SMS cap. Paid providers (Anthropic, Brave, Twilio) are opt-in via explicit env configuration and default to their Mock/Console counterparts, so a fresh checkout cannot incur cost by accident, and automated tests never hit a paid provider (they run under `LLM_PROVIDER=mock`, `SEARCH_PROVIDER=mock`, `NOTIFICATION` console-only fixtures).

## Observability & data retention

- Every HTTP request and every Celery task run carries a correlation id (`request_id` for HTTP, `task_id` for Celery, plus `run_id` for monitoring runs) bound into structlog's contextvars for the duration of the request/task, so every log line emitted while handling it can be grepped together - see `app/main.py::correlation_id_middleware` and the `bound_contextvars(...)` calls in `app/tasks/*.py`. The HTTP middleware also echoes the id back as an `X-Request-ID` response header, reusing an inbound one from a gateway if present.
- `DATA_RETENTION_DAYS` (default 365) is enforced by a daily Celery Beat task (`app.tasks.maintenance.purge_expired_data`, 3am UTC) that deletes `SourceDocument` rows - the raw collected text - older than the window. Only `SourceDocument` is in scope: nothing else in the schema has a foreign key onto it, so this can never cascade-delete a `Snapshot`, `DetectedChange`, `Alert`, or `Report` a user might still want to see (see the docstring on `SourceDocumentRepository.delete_older_than`).

## Reporting a vulnerability

This is a local/dev-stage project; if you find an issue, open an issue in the repository describing the problem and reproduction steps rather than exploiting it further.