# Tasks

Legend: `[ ]` pending, `[x]` done, `[~]` partial/stubbed (see KNOWN_LIMITATIONS.md).

## Phase 1 — Foundation
- [x] Monorepo layout (`apps/web`, `apps/api`, `packages/shared`, `infrastructure`, `scripts`, `docs`)
- [x] PLAN.md, ARCHITECTURE.md, TASKS.md, SECURITY.md, README.md
- [x] `.env.example`, `.gitignore`
- [x] `docker-compose.yml` (postgres, redis, mailpit, api, worker, beat, web) — verified with `docker compose up --build`
- [x] FastAPI skeleton with `/health`, `/ready` (+ `/api/v1/system/status`) — pytest passing
- [x] Next.js skeleton with landing page shell — build/lint/typecheck/test passing
- [x] Backend lint/format (ruff, black) + pytest scaffolding
- [x] Frontend lint/format (eslint, prettier) + vitest scaffolding

## Phase 2 — Auth & Users
- [x] User model + Alembic migration (`users`, `refresh_tokens`)
- [x] `AuthProvider`-equivalent dependency (`get_current_user`): `LocalAuthProvider` and `JWTAuthProvider` behavior behind one seam
- [x] `/auth/register`, `/auth/login`, `/auth/refresh` (rotating), `/auth/logout`, `/auth/me`
- [x] Protected-route dependency (`get_current_user`, `require_admin`) — per-user isolation enforced at repository layer going forward
- [x] Auth rate limiting (slowapi; disabled under `APP_ENV=test`, verified by a dedicated test)
- [x] Frontend: login, register (RHF+Zod), local-mode banner, `useAuth` hooks, dashboard shell with auth guard — verified live against Dockerized API (Postgres) and 14 backend + 6 frontend tests passing

## Phase 3 — Company Management
- [x] Company, CompanyAlias, Competitor, MonitorConfiguration, NotificationDestination models + migrations
- [x] Companies CRUD + pause/resume endpoints (`/run` moves to Phase 5 alongside MonitoringRun/Celery)
- [x] Dashboard page (summary cards, schedule, system health, recent companies) — real data, no placeholders
- [x] Companies list page (table, pause/resume, delete with confirm)
- [x] Add-Company wizard (5 steps: company, focus templates, schedule, notifications+consent, review)
- [x] Company detail page (Overview + Configuration functional; Report/Alerts/Sources/History/Snapshots tabs show phase-appropriate empty states) — verified live end-to-end (create → dashboard → list → detail → configuration) against Dockerized Postgres API; 33 backend + 6 frontend tests passing

## Phase 4 — Collection Pipeline
- [x] `SourceCollector` protocol + `safe_fetch`/`fetch_with_retries` SSRF guard (DNS pre-resolution, redirect re-validation, domain rate limiting)
- [x] Website collector (sitemap + heuristic pages, robots.txt)
- [x] RSS/Atom collector
- [x] Custom URL collector
- [x] SEC EDGAR collector (CIK lookup + recent 10-K/10-Q/8-K metadata)
- [x] GitHub collector (org search + repo metadata)
- [x] Job posting collector (generic HTML link-heuristic extraction real; board-specific APIs stubbed, see KNOWN_LIMITATIONS.md)
- [x] Patent source interface (fixture adapter only, documented, never fabricates)
- [x] Review source interface (fixture adapter only, documented, never fabricates)
- [x] Extraction/normalization (trafilatura + BeautifulSoup fallback, whitespace normalization, content hashing, URL canonicalization, cross-page dedup)
- [x] Source, SourceDocument, Snapshot models + migration
- [x] Sources API (list/create/update/delete/test) with ownership isolation, wired into the company detail page's Sources tab
- [x] `collection_service` (discover_sources_for_company, collect_source) — 22 collector tests + 9 integration tests, all respx-mocked (no live network in the suite); verified live end-to-end against Dockerized Postgres + real `example.com` fetch

## Phase 5 — Background Processing
- [x] Celery app + queues (default/collection/analysis/notifications/maintenance) — bound explicitly via `celery_app.task(...)`, not `@shared_task` (see KNOWN_LIMITATIONS.md for why that distinction mattered)
- [x] Celery Beat dynamic schedule sync from `MonitorConfiguration` (`sync_schedules`, runs every minute, no static per-company entries needed)
- [x] MonitoringRun model + status lifecycle (queued → running → successful/partial/failed) + migration (incl. the deferred `snapshots.monitoring_run_id` FK from Phase 4)
- [x] Run-now endpoint → enqueue without disrupting schedule (`last_run` updates always; `next_run` only advances for scheduled-trigger runs) + idempotent (active run returned, not duplicated)
- [x] Retry/backoff policy (task-level `max_retries`; per-source failures caught and recorded without failing the whole run)
- [x] Run status polling in frontend (2s interval while queued/running) + Monitoring history tab + "Run now" wired on both list and detail pages
- [x] Verified live with the *real* worker/beat/Redis stack (not eager mode): task received, HTTP collection executed, run completed — confirmed via docker logs and API polling

## Phase 6 — Change Detection
- [x] Hash comparison layer (short-circuits before any diffing when unchanged)
- [x] Structured field diff layer (added/removed item sets between consecutive snapshots)
- [x] Bounded text diff + noise filters (timestamps/cookies/copyright/counters stripped before diffing; capped output size)
- [~] Semantic comparison layer — deferred to Phase 7 (needs the LLM provider interface); Layers 1-3 + deterministic scoring are sufficient to ship real detections now
- [x] Significance scoring (documented formula in `app/change_detection/scoring.py`, unit tested — 9 tests covering trust/corroboration/focus-match/repeat/diff-ratio scaling)
- [x] Severity classification (deterministic buckets + hard confidence floor for Critical, unit tested)
- [x] DetectedChange model + migration
- [x] Change-level dedup/cooldown (exact-repeat suppression within a 24h window; repeats that aren't exact still recorded but dampened) — Alert-level dedup/digest is Phase 8 scope
- [x] Wired into the monitoring run task; verified live end-to-end in Docker (real worker, real Postgres) — 34 new tests (scoring/structured-diff/text-diff/noise-filters/extractors/service integration), 130 total passing

## Phase 7 — LLM Analysis
- [x] `LLMProvider` interface, Mock/Anthropic/Ollama implementations (Anthropic via forced tool-use, Ollama via JSON mode; both have a bounded repair loop, tested with the SDK/HTTP layer mocked)
- [x] Task A: document relevance
- [x] Task B: fact/signal extraction
- [x] Task C: cross-source synthesis
- [x] Task D: report generation (evidence gathered from real DB rows, never invented)
- [x] Task E: change significance (narrative only — severity itself stays deterministic per ARCHITECTURE.md)
- [x] Task F: alert summarization
- [x] Report model (JSON + Markdown) + Report page UI (16 sections, copy/print/export-JSON, Markdown-with-sources toggle) — wired into monitoring runs (baseline on first evidence, update when a run detects a change) and verified live end-to-end in Docker (real Postgres, real report generated from real collected evidence)

## Phase 8 — Notifications
- [x] `NotificationProvider` interface: Console, SMTP (Mailpit, stdlib `smtplib` via `asyncio.to_thread`), Twilio SMS (plain REST API, no SDK dependency)
- [x] Alert model + NotificationDelivery model + migration
- [x] `alert_service.create_alert_for_change`: two independent thresholds by design — `MonitorConfiguration.severity_threshold` gates whether an Alert is created at all; each `NotificationDestination.minimum_severity` separately gates whether that destination is notified — wired into the monitoring run task right after change detection
- [x] Alerts API (list with company/severity/read/resolved filters, detail with delivery statuses, PATCH + mark-read/resolve actions) — ownership-scoped
- [x] `POST /notification-destinations/{id}/test` endpoint for a real send-path test (not just CRUD)
- [x] Alerts page (filters, unread/resolved indicators, mark read/resolve actions, detail view with delivery status per destination)
- [x] Settings page (notification destination management incl. per-destination minimum-severity/enabled/test/delete, system configuration incl. LLM/search provider, SMS-enabled flag, DB/Redis health)
- [x] 26 new backend tests (providers, alert_service threshold/dispatch/failure-recording, alerts API ownership+filters, notification-destination test endpoint, one full HTTP-driven E2E: two real monitoring runs → detected change → alert → email) — 175 total passing
- [x] Verified live in Docker: Alembic migration applied against real Postgres, worker/beat rebuilt and healthy, a real SMTP send round-tripped through the live Mailpit container, and both new frontend pages verified in-browser against seeded data
- [x] `TwilioSmsProvider` re-verified against a real Twilio account (user supplied real credentials): auth/transport/error-surfacing all confirmed correct end-to-end (a real `400` from `api.twilio.com` was correctly caught and displayed inline, not swallowed). Uncovered a real product-level blocker rather than a bug: Twilio trial accounts only permit predefined message templates, not the dynamically-generated alert text this app sends — see `KNOWN_LIMITATIONS.md`. Sending a real SMS alert end-to-end requires the user's Twilio account to be upgraded off trial first.
- [x] Added `TelnyxSmsProvider` (`app/notifications/telnyx_sms.py`) as a second SMS vendor option, selected via new `SMS_PROVIDER=twilio|telnyx` setting (`app/notifications/factory.py` now routes on it — same pattern as `LLM_PROVIDER`/`SEARCH_PROVIDER`); `SystemStatusResponse`/`/system/status` and the Settings page's system-configuration panel now show which SMS provider is active. 4 new backend tests (not-configured, real-send, API-error-with-Telnyx's-`errors[]`-shape, factory routing by `sms_provider`) — 210 backend tests total, 18 frontend tests unchanged.
- [x] `TelnyxSmsProvider` driven live against a real Telnyx account: a real `403` ("only pre-verified destinations allowed") was caught and displayed cleanly; after the user verified the destination number, a retry got a real `200 OK` from `api.telnyx.com` — but the text never arrived. Polling the message afterward revealed the real cause: `delivery_failed`, error `40010`, the sending number isn't 10DLC-registered (a US carrier requirement, industry-wide, not app-specific). No code fix possible — needs 10DLC/toll-free registration completed in the Telnyx portal. See `KNOWN_LIMITATIONS.md`.
- [x] Fixed a real gap found in the same pass: `alert_service.send_test_notification` didn't respect `NOTIFICATION_SMS_ENABLED`, so the "Test" button could fire a real SMS API call even with SMS globally disabled — only real alert dispatch checked the flag. Fixed so both paths gate identically; verified live (API logs show zero calls to `api.telnyx.com` after the fix). Settings page now shows an inline notice on the phone-number field explaining SMS is paused for 10DLC registration, not just that it's off. `NOTIFICATION_SMS_ENABLED` set back to `false` per the user's direction while registration is pending. 211 backend tests (up from 210), 18 frontend tests unchanged.

## Phase 9 — Fixture Demo + Tests
- [x] Acme Mobility Systems fixtures v1/v2 (about/products/careers/press/pricing) — real HTML under `apps/api/tests/fixtures/acme_mobility/`, versions differ by a leadership change, a price increase, a new job posting, and an expansion press release; `products` is unchanged v1→v2 to demonstrate the hash-based short-circuit
- [x] Dev-only fixture server (`app/dev/fixtures.py`, mounted only when `APP_ENV != production`) serving those pages over real HTTP, plus a narrow single-hostname SSRF allowlist (`Settings.demo_fixture_host`, unset by default, never honored in production) so the collectors can fetch them
- [x] `scripts/seed_acme_demo.py` (creates the demo company + 5 sources + notification destination, idempotent) and `scripts/switch_fixture.py` (v1/v2/status) + a matching dev-only "Acme Mobility demo" panel on the Settings page (only shown in local auth mode)
- [x] `tests/integration/test_acme_fixture_demo.py` — locks the exact demo scenario (real fixture files off disk, respx-mocked HTTP) into the automated suite: leadership/price/content changes detected correctly, `products` correctly produces no change
- [x] SSRF allowlist unit tests (`tests/unit/test_ssrf.py`) — exact-hostname-only bypass, production-mode ignores it entirely
- [x] Frontend component tests: `SeverityBadge`/`CompanyStatusBadge` rendering, `AlertsPage` (company-name resolution, severity-filter refetch)
- [x] Playwright E2E (`apps/web/e2e/acme-demo-flow.spec.ts`): full UI-driven workflow against the live Docker stack — add company via the wizard → add fixture sources → baseline run (v1) → switch fixture to v2 via the Settings panel → run again → real alert appears on the Alerts page → real email confirmed in Mailpit. Verified passing against the live stack, not just written.
- [x] Backend/frontend regression check: full suites re-run after every addition (179 backend, 14 frontend, all passing)

## Phase 10 — Hardening & Docs
- [x] Rate limiting beyond auth — `@limiter.limit(...)` added to company/source/notification-destination creation, notification-destination test-send, and report generation; verified live (22 rapid company-creation calls → 20×201 then 2×429)
- [x] Manual "run now" daily cap actually enforced — `MAX_MANUAL_RUNS_PER_DAY` was defined in `Settings` since Phase 1 but never wired up until now (`monitoring_service.enqueue_run_now`, new `RateLimitedError` → HTTP 429)
- [x] Structured logging + correlation IDs — `request_id` (HTTP, echoed as `X-Request-ID`) and `run_id`/`task_id` (Celery tasks) bound via structlog contextvars; fixed a real context-propagation gap in the threaded eager-mode fallback (`app/tasks/base.py`) along the way — verified live (header round-trips, task log line carries `task_id`)
- [x] Data retention job — `app.tasks.maintenance.purge_expired_data`, daily Celery Beat task, purges `SourceDocument` rows older than `DATA_RETENTION_DAYS`; scope deliberately limited to the one table with no incoming FK (see KNOWN_LIMITATIONS.md) — verified live via direct `.delay()` invocation against the real worker
- [x] SECURITY.md review pass — added the demo-fixture SSRF allowlist exception, the expanded rate-limiting surface, the notification-destination verification gap, and a new Observability & data retention section
- [x] README run/setup instructions finalized — fixed stale `python -m scripts.*` references from Phase 1 planning that never matched what got built; added the real Acme demo walkthrough and E2E instructions
- [x] `docs/FIREBASE_MIGRATION.md` — honest architecture-fit assessment (auth maps over cleanly; scheduling, cascading deletes, and cross-collection filtering do not), effort estimate, and an explicit recommendation against a full migration
- [x] `KNOWN_LIMITATIONS.md` final pass — Phase 10 section added; all prior phases' sections already up to date
- [x] Full regression check: 185 backend tests, 14 frontend tests, all passing after every Phase 10 change

## Phase 11 — Company Discovery & Provider Completion
- [x] `SearchProvider` interface (`app/search/base.py`) + `MockSearchProvider` (deterministic, honest about having no evidence) + `BraveSearchProvider` (`SEARCH_PROVIDER=brave`) + factory
- [x] Company-profile extraction LLM task (`app/prompts/company_profile.py`) — evidence-grounded (real search snippets + real fetched homepage text in, structured industry/country/region/headquarters/aliases/competitors/public_identifiers out), never asked to recall facts from training data
- [x] `discovery_service.discover_company_profile` — resolves official website, fetches it for real, runs a few targeted searches, calls the extraction task once, merges (user hints win over discovered values), and previews likely sources via the existing collectors' `.discover()` methods (no persistence)
- [x] `POST /api/v1/companies/discover` — rate-limited (`5/minute`), returns a `DiscoveredCompanyProfile`, writes nothing to the DB
- [x] `Company.headquarters` + `Company.public_identifiers` columns (migration `f01919a99ee9`) threaded through the repository/service/schema layers
- [x] `GeminiLLMProvider` (`app/analysis/llm/gemini_provider.py`) on the official `google-genai` SDK, native structured-output (`response_schema`) + the same repair-loop pattern as the Anthropic provider; `LLM_PROVIDER=gemini`
- [x] Add-Company wizard redesigned around discover → review: `STEPS = ["Discover", "Review", "Schedule", "Notifications", "Confirm"]`. Discover step only requires a company name (official website/focus/competitors/aliases are optional accuracy hints); Review step shows the discovered profile, fully editable, with a potential-sources list and a "sources consulted" transparency line
- [x] Company detail page's Details panel now shows `headquarters` and any `public_identifiers` (was previously captured by the wizard but had nowhere to display after creation — fixed during live verification)
- [x] 206 backend tests (up from 185), 18 frontend tests (up from 14) — new coverage for `SearchProvider` (mock + Brave respx-mocked), `discovery_service` (respx-mocked fetch + mock LLM, hint-overrides-discovery merge, plus a regression pair for the non-corporate-host resolution fix below), the `/companies/discover` endpoint, `GeminiLLMProvider` (mocked SDK client), and the wizard's discover→review flow
- [x] Verified live end-to-end against the Docker stack, first with mock providers (`api`/`worker`/`beat` rebuilt for the new `google-genai` dependency, Alembic migration applied to real Postgres, full wizard flow Discover → Review (edited fields survived) → Confirm → a real `Company` row created with `headquarters`/`public_identifiers` persisted; pre-existing "Run now" → lazy source-discovery pipeline confirmed unaffected), then re-verified with **real provider keys the user supplied**: `BraveSearchProvider` confirmed live (`api.search.brave.com`, all queries `200 OK`); `GeminiLLMProvider` uncovered and fixed a real bug (`public_identifiers: dict[str,str]` produced an `additionalProperties` JSON schema the Gemini Developer API rejects - changed to `list[PublicIdentifier]`), then hit an account-level `429 limit:0` unrelated to app code; user switched to `AnthropicLLMProvider`, which then verified the **entire pipeline live with real APIs**: real discovery (Brave + Claude) → real company creation → real `stripe.com` crawl on first run (23 items, 3/3 sources successful) → a real, evidence-grounded, correctly-hedged baseline report from `claude-sonnet-5`. Also found and fixed live: `_resolve_official_website` picking Brave's top-ranked Wikipedia result over the real corporate domain for well-known companies, which corrupted downstream source-preview URLs - see `KNOWN_LIMITATIONS.md`

## Phase 12 — UI Polish & Bug-Fix Round (post-live-testing feedback)
User drove the live app (real Brave + Anthropic keys) and reported 9 concrete issues. All addressed:
- [x] Company detail page's delete-confirmation control now animates in/out with a `grid-template-columns` + opacity transition (200ms, matching the app's existing `fade-in` timing) instead of an instant DOM swap that also instantly shifted the Pause/Run now buttons sideways
- [x] Add-Company wizard's "What do you want to know?" field now shows placeholder text summarizing the categories the app actually monitors for (products/pricing, leadership, hiring, financial signals, M&A, patents, expansion, regulatory/legal, competitor positioning) — was previously blank with no guidance
- [x] `CompanyProfileExtraction`/`DiscoveredCompanyProfile` gained a `description` field (`app/prompts/company_profile.py`, `app/schemas/discovery.py`, `app/services/discovery_service.py`) - the Review step's Description box was always blank before because discovery never produced one; `MockLLMProvider`'s heuristic derives it from the fetched homepage's first real sentence, real providers extract it from evidence like every other field
- [x] Wizard's Confirm step no longer flashes the full confirm view before redirecting - a new `isFinalizing` state swaps in a "Setting up monitoring for X…" spinner the instant "Create company" is clicked and stays there through the redirect (previously `createCompany.isPending` flipped false before `router.push` completed, causing a one-frame flash of the re-enabled button/full confirm content)
- [x] Company detail page's Details panel switched from side-by-side `dt`/`dd` (which produced a hanging-indent wrap for long values like a multi-clause Headquarters string) to a stacked label-above-value layout with `break-words`
- [x] Competitors on the company detail page are now clickable: hovering highlights them, and clicking navigates to that competitor's own company page if it's already monitored (case-insensitive name match against the user's company list) or to `/companies/new?name=<rival>` (wizard reads the `name` query param and pre-fills the Discover step) if not - verified live both ways (Stripe → PayPal went to the wizard pre-filled, then after creating PayPal for real, the same link went straight to its company page)
- [x] Investigated "16 points but basically empty" report finding for the live Stripe company - **not a bug**: the report was generated before any monitoring run had ever collected evidence (0 source documents, 0 detected changes), and the real report correctly refused to fabricate findings, explicitly stating "insufficient evidence" throughout rather than hallucinating - exactly the evidence-grounded behavior this app is designed around. Fixed the actual gap, which was a missing warning: the Latest Report tab now shows an inline notice when generating a report with zero monitoring runs, and Generate now/"Run now" ordering is explained rather than silently producing a thin report
- [x] Investigated empty Sources tab for the same company - also **not a bug**: no monitoring run had ever executed for that company, and `Source` rows are only ever created lazily on first collection (by design, since Phase 4/5). Fixed the gap: the empty-state message now explicitly says sources appear after the first "Run now" rather than a bare "No sources configured yet."
- [x] Built the previously-unimplemented Snapshots tab end-to-end: new `GET /companies/{id}/snapshots` endpoint (`app/api/v1/snapshots.py`, `app/services/snapshot_service.py`, `SnapshotRepository.list_for_company` added to `app/repositories/source_repository.py`, newest-first, capped at 50), `SnapshotResponse` schema, frontend `useSnapshots` hook + `api.listSnapshots`, and an expandable list UI (source name, type, timestamp, char count when collapsed; hash, full text summary, and pretty-printed structured summary when expanded) - verified live against real PayPal snapshots (real scraped homepage/careers/GitHub text, expand/collapse working)
- [x] 214 backend tests (up from 211: 3 new for the snapshots endpoint - empty list, newest-first ordering via a directly-inserted `Snapshot` row, ownership isolation), 18 frontend tests unchanged (existing wizard test extended to assert the description field pre-fills)
- [x] Verified live end-to-end in Docker with real Brave + Anthropic keys: ran the full wizard for PayPal (Stripe's competitor), confirmed `paypal.com` resolved directly (not Wikipedia, confirming the Phase 11 fix still holds), description field populated with real Claude-extracted text, a real monitoring run (50 items, 3/3 sources) populated the Sources and Snapshots tabs with real data, and the Stripe↔PayPal competitor cross-link resolved correctly both before and after PayPal existed as a monitored company

## Phase 13 — Second UI Polish & Bug-Fix Round (wizard nav, dedup, notification linking, report grounding)
User reported 4 more issues after driving the app further. All addressed:
- [x] Wizard's Back button on the Discover step no longer stays permanently disabled - `goBack()` now calls `router.back()` when `step === 0` instead of a no-op, so it correctly returns to whichever of the 3 entry points (Dashboard, Companies, or a competitor link) the user actually came from. Verified live via a real click-through chain (Companies → wizard → Back → landed back on `/companies`)
- [x] Add-Company wizard now detects likely duplicate company names client-side (`normalizeCompanyName`/`findPossibleDuplicate` in `apps/web/app/(app)/companies/new/page.tsx` - strips legal suffixes/punctuation, then exact/substring match against the user's existing companies) **before** spending a real search+LLM call on discovery, shows a "You might already be monitoring X" warning with a link to the existing company and a "Continue anyway" override, and re-checks automatically if the name is edited afterward. Backend now guarantees the persisted `name` is unique per user regardless (`company_service._unique_display_name`, mirroring the existing `_unique_slug` pattern) - "Stripe" → "Stripe (2)" → "Stripe (3)" on collision, filesystem-style. Verified live: typing "Stripe" warned correctly, no discovery API call fired until "Continue anyway", and the created company was actually named "Stripe (2)"
- [x] **Notification destinations are now linked to specific companies** instead of being flat per-user rows every destination implicitly applied to every company. New `notification_destination_companies` join table (migration `60a25ddfc6a3`, includes a data backfill+dedup step - see below), `NotificationDestinationRepository.list_for_company`/`link_company`/`find_by_value`/`delete_orphaned_for_user`, `POST /notification-destinations` now requires `company_ids: list[UUID]` (min 1) and reuses an existing destination by (user, type, value) instead of duplicating it - this is the actual fix for the wizard silently creating a fresh row per company even when the same email was already registered. `alert_service.create_alert_for_change` now dispatches via `list_for_company(company.id)` instead of `list_for_user`, so a destination only fires for companies it's actually linked to. Deleting a company now garbage-collects any destination left with zero remaining links (`company_service.delete_company` → `delete_orphaned_for_user`), with `Company.notification_links`/`NotificationDestination.company_links` given explicit ORM `cascade="all, delete-orphan"` since SQLite (used in dev/tests) doesn't enforce `ON DELETE CASCADE` without a pragma this app doesn't set - relying on the DB-level FK alone would've silently broken cleanup under SQLite while appearing to work on Postgres
- [x] The one-time migration backfills every existing destination onto every company the same user currently has (a no-op behavior change - it's exactly what already happened implicitly before the join table existed) and then deduplicates rows sharing the same (user, type, normalized value), keeping the earliest and deleting the rest (cascading their `NotificationDelivery` history, an acceptable one-time cleanup). Verified against the live Postgres DB: went from several duplicate `theminecraftboy...@gmail.com` rows down to exactly 4 unique destinations, with 48 backfilled links (4 destinations × 12 companies at migration time) - matching the pre-migration behavior exactly
- [x] Settings page redesigned: `AddDestinationForm` now has a required company multi-select (toggle-chip buttons, at least one required to submit); `DestinationRow` shows each unique destination once with a horizontally-scrollable row of clickable company chips (`overflow-x-auto`, hover-highlight, links to `/companies/{id}`) instead of no company visibility at all. Verified live: scrollWidth (1134px) exceeds clientWidth (384px) on the chip row, confirming it actually scrolls rather than wrapping/clipping; creating a destination linked to only one company and then deleting that company correctly removed the destination from the list (confirmed via direct API calls against the live stack)
- [x] Report generation now receives the company's discovered profile (`description`, `official_website`, `headquarters`, `country`, `region`, `public_identifiers`) as a `company_profile` evidence block (`app/prompts/report_generation.py`, threaded through from `report_service.py`), not just `source_documents`/`detected_changes` - this data is genuine evidence (fetched from the company's real website/search results at onboarding), just previously never wired into report generation, which is why reports generated before any monitoring run came back almost entirely "insufficient evidence" even when the company profile had real content. `MockLLMProvider._build_report` also rewritten to ground `company_overview`/`market_positioning` in profile fields. Verified live: generated a report for a brand-new "Airbnb" company with zero monitoring runs - `company_overview` and `market_positioning` came back as substantive, evidence-grounded paragraphs (real HQ, industry, named competitors, business model) instead of "insufficient evidence", while sections with genuinely no evidence (financials, hiring, leadership) still correctly said so
- [x] 223 backend tests (up from 214: company name-uniqueness ×3, notification-destination linking/dedup/GC ×6, mock report profile-grounding ×1, plus updates to existing destination/alert tests to pass `company_ids`), 19 frontend tests (up from 18: new duplicate-warning wizard test)
- [x] Verified live end-to-end against the Docker stack with real Brave + Anthropic keys and the live Postgres DB (migration applied, backfill/dedup confirmed via direct SQL) - see individual bullets above for what was checked

## Phase 14 — New Intelligence Sources + Per-Source Scheduling
User compared the live app against the original ChatGPT-authored planning document that inspired it and found real gaps - not a policy problem (bypassing paywalls/logins was explicitly rejected as out of scope again), just free/public sources that were never wired up, plus no way to check a fast-moving source (news) more often than a slow one (patents) within the same company.
- [x] Google News RSS auto-discovery - `RssCollector.discover()` (`app/collectors/rss.py`) was previously a stub returning `[]`; now builds `https://news.google.com/rss/search?q={company}&hl=en-US&gl=US&ceid=US:en` and registers it as a real discoverable source, reusing `RssCollector.collect()`'s already-real `feedparser` fetch/parse
- [x] Government contracts via USASpending.gov - new `SourceType.GOV_CONTRACT` + `GovContractCollector` (`app/collectors/gov_contracts.py`), a free/keyless `POST /api/v2/search/spending_by_award/`, structurally mirroring `SecEdgarCollector` (always offered, a private company just gets zero results, not an error)
- [x] Patents wired to USPTO's Open Data Portal - new `USPTO_API_KEY` setting; `PatentSourceCollector.collect()` now attempts a real `POST /api/v1/patent/applications/search` call when a key is configured, with the existing honest fixture/disabled fallback completely unchanged (byte-for-byte) for the default no-key case, so none of the 12 existing companies with a dormant `PatentSourceCollector` regressed
- [x] Per-source check-frequency scheduling - `Source` gained 4 nullable columns (`frequency_type`/`interval_minutes`/`cron_expression`/`next_check`, migration `06e7f03cea36`); `NULL` means "inherit the company's default cadence" (the default for every source, zero behavior change unless a source opts in). `sync_schedules` now enqueues a company if *either* its own `MonitorConfiguration.next_run` is due *or* any of its sources has an independently-due override (`SourceRepository.company_has_due_work`/`list_due_for_company`). A `SCHEDULED` run only collects the sources actually due; a `MANUAL` "Run now" still collects every active source regardless of cadence, unchanged. Frontend: a per-row "Check frequency" `Select` on the Sources tab (`apps/web/app/(app)/companies/[id]/page.tsx`), defaulting to "Same as company", wired through `PATCH /sources/{id}` (`SourceUpdate` gained the same 3 fields; `source_service.update_source` validates the schedule via the existing `validate_and_compute_next_run` and resets `next_check` to `None` so a changed/cleared override takes effect on the very next scheduler tick rather than waiting out the old cadence)
- [x] Explicitly dropped: Nubela/NinjaPear company-enrichment API (confirmed enterprise-only pricing, not accessible to a normal user - no code written); dedicated PRNewswire/BusinessWire/GlobeNewswire collectors (redundant with what Google News RSS already surfaces)
- [x] 240 backend tests (up from 223: 2 RSS discovery-URL tests, 4 gov-contracts collector tests, 5 patents-live-branch tests, 4 scheduler/per-source-due-ness integration tests, 2 source-update-API tests for the new scheduling fields), frontend `tsc`/`eslint`/vitest (19 tests) all clean with the new `SourceUpdatePayload`/`useUpdateSource` additions
- [x] Verified live end-to-end against the Docker stack: `api`/`worker`/`beat`/`web` rebuilt, migration `06e7f03cea36` applied to the real Postgres DB, a real "Run now" against Stripe auto-discovered and successfully collected from all 5 sources including the two new ones (`GET https://news.google.com/rss/search?q=Stripe...` → `200 OK`, `POST https://api.usaspending.gov/api/v2/search/spending_by_award/` → `200 OK`), and the new per-row frequency Select was exercised live in-browser (set "Stripe — Google News" to Daily, confirmed via a re-fetch from the API; cleared it back to "Same as company", confirmed that round-tripped too, both backed by real `PATCH /api/v1/sources/{id}` → `200 OK` calls in the API logs). Patents' real-API branch is unit-tested against a mocked HTTP call only - not live-verified, since it requires a user-supplied `USPTO_API_KEY` that hasn't been provided yet; the existing no-key fixture path was already covered by the pre-existing patents tests and is unaffected

## Phase 15 — NinjaPear Company Enrichment
Phase 14 dropped Nubela/NinjaPear as "enterprise-only" - the user found and paid for an individual $49/mo tier ($49-$1899/mo range) that Phase 14's research missed, then asked for it wired up. Scoped via `AskUserQuestion` before building: **all** endpoint categories including customer listing (not just company-level data), but **onboarding-only** timing (never a recurring per-cycle cost) - confirmed given the API bills real credits per field per request, unlike every other free/flat-rate provider in this app.
- [x] New `app/enrichment/` provider package (`base.py` Protocol + Pydantic result shapes, `mock.py` honest-empties default, `ninjapear.py` real per-endpoint `httpx` calls to `nubela.co`, `factory.py`), mirroring `app/search/`'s shape. New `NINJAPEAR_API_KEY`/`NINJAPEAR_MAX_LEADERSHIP_LOOKUPS` settings
- [x] New `CompanyEnrichment` model (1:1 with `Company`, migration `79e3aa041131`) - `status` (pending/partial/complete/failed), a single `data` JSON blob (employee count, leadership team, funding rounds, competitors-with-reasons, products, recent updates, customers), and an `errors` map so a partially-failed enrichment is never silently presented as complete
- [x] `app/services/enrichment_service.py` orchestrates ~6 independent per-endpoint calls plus capped per-leadership-member work-email/profile lookups (`NINJAPEAR_MAX_LEADERSHIP_LOOKUPS`, default 5) - one failed call is recorded in `errors` and never sinks the others, same principle as `tasks/collection.py`'s per-source loop. New `app/tasks/enrichment.py` Celery task (own `enrichment` queue, generous time limits given NinjaPear's documented up-to-5-minute endpoints)
- [x] `company_service.create_company` enqueues the task post-commit, **gated entirely on `NINJAPEAR_API_KEY` being set** - zero extra background-task volume for the overwhelming majority of users who haven't configured it, matching `PatentSourceCollector.discover()`'s "gate on the key, not the provider" precedent. A `status=pending` `CompanyEnrichment` row is created synchronously in the same transaction (not left implicit) so the frontend has something real to poll on
- [x] Report generation gains a `company_enrichment` evidence block (`app/prompts/report_generation.py`, threaded through `report_service.py`) right alongside the existing `company_profile` block - no new report schema needed, since funding/leadership/competitors/products/customers all map onto existing `ReportContent` sections (`financial_signals`, `leadership_changes`, `competitor_comparison`, `products_and_services`, `customer_sentiment`)
- [x] `GET /system/status` gains `ninjapear_configured`/`ninjapear_credit_balance` (a live, free credit-balance call), shown in Settings' System configuration panel - real-money cost visibility, same treatment the SMS provider status already gets
- [x] Frontend: new "Enrichment" tab on the company detail page (funding table, leadership list with resolved work-email/profile links, competitors-with-reasons kept visually separate from the user's own reviewed Competitors list, products, recent updates, customers, employee count), polling (`useCompany`'s `refetchInterval`) while `status === "pending"` so it updates itself once the background task finishes
- [x] Deliberately excluded: the "Similar People" endpoint (a role-anchored prospecting tool with no natural onboarding-time trigger), the Website Lookup endpoint (redundant - `official_website` already comes from Phase 11 discovery), and auto-merging NinjaPear's suggested competitors into the user-reviewed `Company.competitors` list (would silently mutate user-controlled data)
- [x] 261 backend tests (up from 240: 8 provider tests incl. mock honesty, 6 orchestration-service tests incl. the leadership cap and partial/failed status derivation, 3 Celery-task integration tests, 2 create-company enqueue-gating tests - the real regression guard, since every other test in the suite runs without a key configured and stays green throughout - 2 report-generation evidence-block tests), frontend `tsc`/eslint/vitest (19 tests) all clean
- [x] **Verified live** with the user's real `NINJAPEAR_API_KEY` and `USPTO_API_KEY` (both supplied after explicit go-ahead, since NinjaPear spends real credits per call). The initial schema (written from a JS-rendered docs page that couldn't be fully scraped) had real bugs the live pass caught: NinjaPear identifies companies by `website` only (no name-based lookup), `employee_count`/`industry`/funding amounts come back as raw numbers not strings, funding's `investors` are objects not plain strings, and most response field names differed from the initial guesses (`executives`, `total_funds_raised`, `x_profile_url`, etc. - full list in `KNOWN_LIMITATIONS.md`). All fixed and re-verified: a real company creation (Stripe) returned real leadership bios, work emails, X profiles, funding history, competitors-with-reasons, live blog updates, and customers for 34 real credits, landing on `status: partial` (one now-fixed bug, one legitimate 404 for a person not in NinjaPear's database) - and the credit-balance endpoint path/field was also wrong and fixed (`/api/v1/meta/credit-balance` → `credit_balance`, not `/company/credit-balance` → `balance`). USPTO's real branch (Phase 14, unverified until now) turned out to have its own live-only bugs too - a wrong sort-field path (`500` error), title/date fields nested one level deeper than assumed, and `404` "no matching records" being treated as a failure instead of an honest empty result - all fixed. 265 backend tests (up from 261: schema-corrected provider/service tests, a no-website-guard test, and a USPTO 404-as-empty-result test)
- [x] **USPTO company-name search follow-up**: confirmed (by inspecting a real response's full field list, for a query that *did* return 110k+ real results by inventor name) that USPTO's Patent Application Search has no queryable assignee/company field at all - not a bug to fix, a real constraint of that dataset. At the user's request, worked around it: `CompanyContext` gained `leadership_names` (`app/collectors/base.py`), threaded through from `CompanyEnrichment.data["leadership_team"]` in `collection_service.to_company_context` (new `_leadership_names` helper - DB-free collectors stay DB-free, the ORM→dataclass seam is the one place allowed to read it). `PatentSourceCollector.collect()` now searches USPTO by each of the company's leadership names (capped at 5, `_MAX_INVENTOR_SEARCHES`) instead of by company name, dedupes results by application number across names, and trust-scores every match at 0.5 with explicit "heuristic, not verified" labeling in the document content - a name match is a real signal, not proof of company ownership. Live-verified against Stripe: 19 real patent documents found via its executives' names, zero NinjaPear credits spent (USPTO is free). Self-heals over time: a company whose patents source was discovered before enrichment finished just returns empty until the next scheduled collection re-reads (now-populated) leadership names from the DB. 266 backend tests (up from 265: leadership-name search/dedup, empty-without-names no-network-call, and eager-load fixes to 3 pre-existing integration tests that constructed `Company` objects directly without loading the new `.enrichment` relationship)