CIAgent/SECURITY.md

Security

Threat model summary

This app fetches user-submitted and discovered URLs from the public internet on a schedule, stores API keys for third-party providers, and sends emails/SMS. The main risks are: SSRF via URL fetching, credential leakage, cross-user data access, and abuse of the scraping/LLM/SMS pipeline as a cost or spam vector.

SSRF prevention (app/core/http.py::safe_fetch)

All outbound collector/custom-URL requests go through one function. It:

• Only allows http/https schemes (no file://, ftp://, gopher://, etc.).
• Resolves DNS itself before connecting, and rejects the request if any resolved address is loopback, link-local, private (RFC1918/RFC4193), multicast, or a known cloud metadata address (169.254.169.254, etc.).
• Disables automatic redirect following in the HTTP client and instead re-validates each redirect target against the same rules before following it, up to a small max-hop count.
• Applies a per-request timeout and a per-domain rate limit / delay (SCRAPER_DOMAIN_DELAY_SECONDS).
• Sends a configurable, identifying User-Agent (SCRAPER_USER_AGENT) rather than impersonating a browser.

No collector or route handler is allowed to call httpx/aiohttp directly for a user- or discovery-supplied URL — code review should reject that pattern. The guard has zero exceptions: every resolved private/loopback/link-local/metadata address is rejected unconditionally, with no allowlist.

Secrets

• All secrets are read from environment variables via app/core/config.py. Nothing is hard-coded.
• .env is git-ignored; .env.example ships with empty/placeholder values only.
• Secrets are never logged. The structured logger has a redaction filter for known secret-shaped keys (*_key, *_secret, *_token, *password*).
• The frontend never receives backend API keys; all third-party calls happen server-side.
• NotificationDestination.destination_value (email/phone) is treated as PII, not a secret, but is still excluded from audit logs beyond a masked form.

AuthN/AuthZ

• Passwords hashed with Argon2 (passlib[argon2]), never stored or logged in plaintext.
• JWT access tokens are short-lived (default 15 min); refresh tokens are longer-lived, stored hashed, and rotated on use.
• Every service-layer query that loads a Company, Report, Alert, Source, or NotificationDestination filters by owner_user_id — enforced in the repository layer, not just the route layer, so a missed check in one route can't leak data.
• Auth endpoints are rate-limited (slowapi) to blunt credential stuffing. Per-IP request-rate limits also apply to every other mutation endpoint that creates data or costs money to call — company/source/notification-destination creation, report generation (LLM call), notification-destination test-send (real SMTP/Twilio call), and run-now — on top of (not instead of) the business-logic caps in "Cost/abuse controls" below. See @limiter.limit(...) usages across app/api/v1/*.py.
• AUTH_MODE=local is for local development only; the app refuses to start in this mode when APP_ENV=production.
• Notification destinations are not verified. NotificationDestination.verified exists on the model but nothing ever sets it to true - a destination is usable for real alert sends as soon as it's created, with no confirmation-code/link flow. Combined with SMS being real-money-per-message once NOTIFICATION_SMS_ENABLED=true and Twilio is configured, this means a user could point alerts at a phone number or inbox they don't control. Rate limiting on the /test endpoint (above) narrows the abuse window but doesn't close it; a verification flow is the real fix and is tracked in KNOWN_LIMITATIONS.md as unimplemented.

Ethical/legal collection rules

Enforced by the collector layer, not left to convention:

1. robots.txt is fetched and honored before crawling a domain (website collector).
2. No collector attempts to bypass a CAPTCHA, login wall, paywall, or anti-bot control. If a source requires that, the Source.status is set to AUTH_REQUIRED/BLOCKED_BY_POLICY and the failure is recorded, never silently skipped or faked.
3. Only publicly reachable URLs are collected — safe_fetch also serves as the enforcement point for "no private/internal targets."
4. Per-domain delay + retry-with-backoff avoids hammering a source.
5. Content hashing avoids redundant re-fetches of unchanged pages.
6. A run that partially fails is marked partial, not silently reported as complete — the report generator is told which sources failed so it doesn't imply completeness it doesn't have.

Input validation & injection

• All request/response bodies are Pydantic models with explicit types and length limits.
• SQLAlchemy ORM with parameter binding throughout — no raw string-interpolated SQL.
• LLM output is only ever deserialized into a strict Pydantic schema (generate_structured); free-text LLM output is never concatenated into a shell command, SQL string, or HTML without escaping, and is rendered in the frontend as text, not dangerouslySetInnerHTML.
• Request bodies are size-limited at the ASGI layer.

Cost/abuse controls

Per-user limits (configurable, enforced in services/): max monitored companies, max manual "run now" triggers per day, max pages crawled per run (MAX_PAGES_PER_DOMAIN), max LLM tokens per request, max LLM retries, monthly SMS cap. Paid providers (Anthropic, Brave, Twilio) are opt-in via explicit env configuration and default to their Mock/Console counterparts, so a fresh checkout cannot incur cost by accident, and automated tests never hit a paid provider (they run under LLM_PROVIDER=mock, SEARCH_PROVIDER=mock, NOTIFICATION console-only fixtures).

Observability & data retention

• Every HTTP request and every Celery task run carries a correlation id (request_id for HTTP, task_id for Celery, plus run_id for monitoring runs) bound into structlog's contextvars for the duration of the request/task, so every log line emitted while handling it can be grepped together - see app/main.py::correlation_id_middleware and the bound_contextvars(...) calls in app/tasks/*.py. The HTTP middleware also echoes the id back as an X-Request-ID response header, reusing an inbound one from a gateway if present.
• DATA_RETENTION_DAYS (default 365) is enforced by a daily Celery Beat task (app.tasks.maintenance.purge_expired_data, 3am UTC) that deletes SourceDocument rows - the raw collected text - older than the window. Only SourceDocument is in scope: nothing else in the schema has a foreign key onto it, so this can never cascade-delete a Snapshot, DetectedChange, Alert, or Report a user might still want to see (see the docstring on SourceDocumentRepository.delete_older_than).

Reporting a vulnerability

This is a local/dev-stage project; if you find an issue, open an issue in the repository describing the problem and reproduction steps rather than exploiting it further.