CIAgent/TASKS.md

Tasks

Legend: [ ] pending, [x] done, [~] partial/stubbed (see KNOWN_LIMITATIONS.md).

Phase 1 — Foundation

• Monorepo layout (apps/web, apps/api, packages/shared, infrastructure, scripts, docs)
• PLAN.md, ARCHITECTURE.md, TASKS.md, SECURITY.md, README.md
• .env.example, .gitignore
• docker-compose.yml (postgres, redis, mailpit, api, worker, beat, web) — verified with docker compose up --build
• FastAPI skeleton with /health, /ready (+ /api/v1/system/status) — pytest passing
• Next.js skeleton with landing page shell — build/lint/typecheck/test passing
• Backend lint/format (ruff, black) + pytest scaffolding
• Frontend lint/format (eslint, prettier) + vitest scaffolding

Phase 2 — Auth & Users

• User model + Alembic migration (users, refresh_tokens)
• AuthProvider-equivalent dependency (get_current_user): LocalAuthProvider and JWTAuthProvider behavior behind one seam
• /auth/register, /auth/login, /auth/refresh (rotating), /auth/logout, /auth/me
• Protected-route dependency (get_current_user, require_admin) — per-user isolation enforced at repository layer going forward
• Auth rate limiting (slowapi; disabled under APP_ENV=test, verified by a dedicated test)
• Frontend: login, register (RHF+Zod), local-mode banner, useAuth hooks, dashboard shell with auth guard — verified live against Dockerized API (Postgres) and 14 backend + 6 frontend tests passing

Phase 3 — Company Management

• Company, CompanyAlias, Competitor, MonitorConfiguration, NotificationDestination models + migrations
• Companies CRUD + pause/resume endpoints (/run moves to Phase 5 alongside MonitoringRun/Celery)
• Dashboard page (summary cards, schedule, system health, recent companies) — real data, no placeholders
• Companies list page (table, pause/resume, delete with confirm)
• Add-Company wizard (5 steps: company, focus templates, schedule, notifications+consent, review)
• Company detail page (Overview + Configuration functional; Report/Alerts/Sources/History/Snapshots tabs show phase-appropriate empty states) — verified live end-to-end (create → dashboard → list → detail → configuration) against Dockerized Postgres API; 33 backend + 6 frontend tests passing

Phase 4 — Collection Pipeline

• SourceCollector protocol + safe_fetch/fetch_with_retries SSRF guard (DNS pre-resolution, redirect re-validation, domain rate limiting)
• Website collector (sitemap + heuristic pages, robots.txt)
• RSS/Atom collector
• Custom URL collector
• SEC EDGAR collector (CIK lookup + recent 10-K/10-Q/8-K metadata)
• GitHub collector (org search + repo metadata)
• Job posting collector (generic HTML link-heuristic extraction real; board-specific APIs stubbed, see KNOWN_LIMITATIONS.md)
• Patent source interface (fixture adapter only, documented, never fabricates)
• Review source interface (fixture adapter only, documented, never fabricates)
• Extraction/normalization (trafilatura + BeautifulSoup fallback, whitespace normalization, content hashing, URL canonicalization, cross-page dedup)
• Source, SourceDocument, Snapshot models + migration
• Sources API (list/create/update/delete/test) with ownership isolation, wired into the company detail page's Sources tab
• collection_service (discover_sources_for_company, collect_source) — 22 collector tests + 9 integration tests, all respx-mocked (no live network in the suite); verified live end-to-end against Dockerized Postgres + real example.com fetch

Phase 5 — Background Processing

• Celery app + queues (default/collection/analysis/notifications/maintenance) — bound explicitly via celery_app.task(...), not @shared_task (see KNOWN_LIMITATIONS.md for why that distinction mattered)
• Celery Beat dynamic schedule sync from MonitorConfiguration (sync_schedules, runs every minute, no static per-company entries needed)
• MonitoringRun model + status lifecycle (queued → running → successful/partial/failed) + migration (incl. the deferred snapshots.monitoring_run_id FK from Phase 4)
• Run-now endpoint → enqueue without disrupting schedule (last_run updates always; next_run only advances for scheduled-trigger runs) + idempotent (active run returned, not duplicated)
• Retry/backoff policy (task-level max_retries; per-source failures caught and recorded without failing the whole run)
• Run status polling in frontend (2s interval while queued/running) + Monitoring history tab + "Run now" wired on both list and detail pages
• Verified live with the real worker/beat/Redis stack (not eager mode): task received, HTTP collection executed, run completed — confirmed via docker logs and API polling

Phase 6 — Change Detection

• Hash comparison layer (short-circuits before any diffing when unchanged)
• Structured field diff layer (added/removed item sets between consecutive snapshots)
• Bounded text diff + noise filters (timestamps/cookies/copyright/counters stripped before diffing; capped output size)
• [~] Semantic comparison layer — deferred to Phase 7 (needs the LLM provider interface); Layers 1-3 + deterministic scoring are sufficient to ship real detections now
• Significance scoring (documented formula in app/change_detection/scoring.py, unit tested — 9 tests covering trust/corroboration/focus-match/repeat/diff-ratio scaling)
• Severity classification (deterministic buckets + hard confidence floor for Critical, unit tested)
• DetectedChange model + migration
• Change-level dedup/cooldown (exact-repeat suppression within a 24h window; repeats that aren't exact still recorded but dampened) — Alert-level dedup/digest is Phase 8 scope
• Wired into the monitoring run task; verified live end-to-end in Docker (real worker, real Postgres) — 34 new tests (scoring/structured-diff/text-diff/noise-filters/extractors/service integration), 130 total passing

Phase 7 — LLM Analysis

• LLMProvider interface, Mock/Anthropic/Ollama implementations (Anthropic via forced tool-use, Ollama via JSON mode; both have a bounded repair loop, tested with the SDK/HTTP layer mocked)
• Task A: document relevance
• Task B: fact/signal extraction
• Task C: cross-source synthesis
• Task D: report generation (evidence gathered from real DB rows, never invented)
• Task E: change significance (narrative only — severity itself stays deterministic per ARCHITECTURE.md)
• Task F: alert summarization
• Report model (JSON + Markdown) + Report page UI (16 sections, copy/print/export-JSON, Markdown-with-sources toggle) — wired into monitoring runs (baseline on first evidence, update when a run detects a change) and verified live end-to-end in Docker (real Postgres, real report generated from real collected evidence)

Phase 8 — Notifications

• NotificationProvider interface: Console, SMTP (Mailpit, stdlib smtplib via asyncio.to_thread), Twilio SMS (plain REST API, no SDK dependency)
• Alert model + NotificationDelivery model + migration
• alert_service.create_alert_for_change: two independent thresholds by design — MonitorConfiguration.severity_threshold gates whether an Alert is created at all; each NotificationDestination.minimum_severity separately gates whether that destination is notified — wired into the monitoring run task right after change detection
• Alerts API (list with company/severity/read/resolved filters, detail with delivery statuses, PATCH + mark-read/resolve actions) — ownership-scoped
• POST /notification-destinations/{id}/test endpoint for a real send-path test (not just CRUD)
• Alerts page (filters, unread/resolved indicators, mark read/resolve actions, detail view with delivery status per destination)
• Settings page (notification destination management incl. per-destination minimum-severity/enabled/test/delete, system configuration incl. LLM/search provider, SMS-enabled flag, DB/Redis health)
• 26 new backend tests (providers, alert_service threshold/dispatch/failure-recording, alerts API ownership+filters, notification-destination test endpoint, one full HTTP-driven E2E: two real monitoring runs → detected change → alert → email) — 175 total passing
• Verified live in Docker: Alembic migration applied against real Postgres, worker/beat rebuilt and healthy, a real SMTP send round-tripped through the live Mailpit container, and both new frontend pages verified in-browser against seeded data
• TwilioSmsProvider re-verified against a real Twilio account (user supplied real credentials): auth/transport/error-surfacing all confirmed correct end-to-end (a real 400 from api.twilio.com was correctly caught and displayed inline, not swallowed). Uncovered a real product-level blocker rather than a bug: Twilio trial accounts only permit predefined message templates, not the dynamically-generated alert text this app sends — see KNOWN_LIMITATIONS.md. Sending a real SMS alert end-to-end requires the user's Twilio account to be upgraded off trial first.
• Added TelnyxSmsProvider (app/notifications/telnyx_sms.py) as a second SMS vendor option, selected via new SMS_PROVIDER=twilio|telnyx setting (app/notifications/factory.py now routes on it — same pattern as LLM_PROVIDER/SEARCH_PROVIDER); SystemStatusResponse//system/status and the Settings page's system-configuration panel now show which SMS provider is active. 4 new backend tests (not-configured, real-send, API-error-with-Telnyx's-errors[]-shape, factory routing by sms_provider) — 210 backend tests total, 18 frontend tests unchanged.
• TelnyxSmsProvider driven live against a real Telnyx account: a real 403 ("only pre-verified destinations allowed") was caught and displayed cleanly; after the user verified the destination number, a retry got a real 200 OK from api.telnyx.com — but the text never arrived. Polling the message afterward revealed the real cause: delivery_failed, error 40010, the sending number isn't 10DLC-registered (a US carrier requirement, industry-wide, not app-specific). No code fix possible — needs 10DLC/toll-free registration completed in the Telnyx portal. See KNOWN_LIMITATIONS.md.
• Fixed a real gap found in the same pass: alert_service.send_test_notification didn't respect NOTIFICATION_SMS_ENABLED, so the "Test" button could fire a real SMS API call even with SMS globally disabled — only real alert dispatch checked the flag. Fixed so both paths gate identically; verified live (API logs show zero calls to api.telnyx.com after the fix). Settings page now shows an inline notice on the phone-number field explaining SMS is paused for 10DLC registration, not just that it's off. NOTIFICATION_SMS_ENABLED set back to false per the user's direction while registration is pending. 211 backend tests (up from 210), 18 frontend tests unchanged.

Phase 9 — Fixture Demo + Tests

• Acme Mobility Systems fixtures v1/v2 (about/products/careers/press/pricing) — real HTML under apps/api/tests/fixtures/acme_mobility/, versions differ by a leadership change, a price increase, a new job posting, and an expansion press release; products is unchanged v1→v2 to demonstrate the hash-based short-circuit
• Dev-only fixture server (app/dev/fixtures.py, mounted only when APP_ENV != production) serving those pages over real HTTP, plus a narrow single-hostname SSRF allowlist (Settings.demo_fixture_host, unset by default, never honored in production) so the collectors can fetch them
• scripts/seed_acme_demo.py (creates the demo company + 5 sources + notification destination, idempotent) and scripts/switch_fixture.py (v1/v2/status) + a matching dev-only "Acme Mobility demo" panel on the Settings page (only shown in local auth mode)
• tests/integration/test_acme_fixture_demo.py — locks the exact demo scenario (real fixture files off disk, respx-mocked HTTP) into the automated suite: leadership/price/content changes detected correctly, products correctly produces no change
• SSRF allowlist unit tests (tests/unit/test_ssrf.py) — exact-hostname-only bypass, production-mode ignores it entirely
• Frontend component tests: SeverityBadge/CompanyStatusBadge rendering, AlertsPage (company-name resolution, severity-filter refetch)
• Playwright E2E (apps/web/e2e/acme-demo-flow.spec.ts): full UI-driven workflow against the live Docker stack — add company via the wizard → add fixture sources → baseline run (v1) → switch fixture to v2 via the Settings panel → run again → real alert appears on the Alerts page → real email confirmed in Mailpit. Verified passing against the live stack, not just written.
• Backend/frontend regression check: full suites re-run after every addition (179 backend, 14 frontend, all passing)

Phase 10 — Hardening & Docs

• Rate limiting beyond auth — @limiter.limit(...) added to company/source/notification-destination creation, notification-destination test-send, and report generation; verified live (22 rapid company-creation calls → 20×201 then 2×429)
• Manual "run now" daily cap actually enforced — MAX_MANUAL_RUNS_PER_DAY was defined in Settings since Phase 1 but never wired up until now (monitoring_service.enqueue_run_now, new RateLimitedError → HTTP 429)
• Structured logging + correlation IDs — request_id (HTTP, echoed as X-Request-ID) and run_id/task_id (Celery tasks) bound via structlog contextvars; fixed a real context-propagation gap in the threaded eager-mode fallback (app/tasks/base.py) along the way — verified live (header round-trips, task log line carries task_id)
• Data retention job — app.tasks.maintenance.purge_expired_data, daily Celery Beat task, purges SourceDocument rows older than DATA_RETENTION_DAYS; scope deliberately limited to the one table with no incoming FK (see KNOWN_LIMITATIONS.md) — verified live via direct .delay() invocation against the real worker
• SECURITY.md review pass — added the demo-fixture SSRF allowlist exception, the expanded rate-limiting surface, the notification-destination verification gap, and a new Observability & data retention section
• README run/setup instructions finalized — fixed stale python -m scripts.* references from Phase 1 planning that never matched what got built; added the real Acme demo walkthrough and E2E instructions
• docs/FIREBASE_MIGRATION.md — honest architecture-fit assessment (auth maps over cleanly; scheduling, cascading deletes, and cross-collection filtering do not), effort estimate, and an explicit recommendation against a full migration
• KNOWN_LIMITATIONS.md final pass — Phase 10 section added; all prior phases' sections already up to date
• Full regression check: 185 backend tests, 14 frontend tests, all passing after every Phase 10 change

Phase 11 — Company Discovery & Provider Completion

• SearchProvider interface (app/search/base.py) + MockSearchProvider (deterministic, honest about having no evidence) + BraveSearchProvider (SEARCH_PROVIDER=brave) + factory
• Company-profile extraction LLM task (app/prompts/company_profile.py) — evidence-grounded (real search snippets + real fetched homepage text in, structured industry/country/region/headquarters/aliases/competitors/public_identifiers out), never asked to recall facts from training data
• discovery_service.discover_company_profile — resolves official website, fetches it for real, runs a few targeted searches, calls the extraction task once, merges (user hints win over discovered values), and previews likely sources via the existing collectors' .discover() methods (no persistence)
• POST /api/v1/companies/discover — rate-limited (5/minute), returns a DiscoveredCompanyProfile, writes nothing to the DB
• Company.headquarters + Company.public_identifiers columns (migration f01919a99ee9) threaded through the repository/service/schema layers
• GeminiLLMProvider (app/analysis/llm/gemini_provider.py) on the official google-genai SDK, native structured-output (response_schema) + the same repair-loop pattern as the Anthropic provider; LLM_PROVIDER=gemini
• Add-Company wizard redesigned around discover → review: STEPS = ["Discover", "Review", "Schedule", "Notifications", "Confirm"]. Discover step only requires a company name (official website/focus/competitors/aliases are optional accuracy hints); Review step shows the discovered profile, fully editable, with a potential-sources list and a "sources consulted" transparency line
• Company detail page's Details panel now shows headquarters and any public_identifiers (was previously captured by the wizard but had nowhere to display after creation — fixed during live verification)
• 206 backend tests (up from 185), 18 frontend tests (up from 14) — new coverage for SearchProvider (mock + Brave respx-mocked), discovery_service (respx-mocked fetch + mock LLM, hint-overrides-discovery merge, plus a regression pair for the non-corporate-host resolution fix below), the /companies/discover endpoint, GeminiLLMProvider (mocked SDK client), and the wizard's discover→review flow
• Verified live end-to-end against the Docker stack, first with mock providers (api/worker/beat rebuilt for the new google-genai dependency, Alembic migration applied to real Postgres, full wizard flow Discover → Review (edited fields survived) → Confirm → a real Company row created with headquarters/public_identifiers persisted; pre-existing "Run now" → lazy source-discovery pipeline confirmed unaffected), then re-verified with real provider keys the user supplied: BraveSearchProvider confirmed live (api.search.brave.com, all queries 200 OK); GeminiLLMProvider uncovered and fixed a real bug (public_identifiers: dict[str,str] produced an additionalProperties JSON schema the Gemini Developer API rejects - changed to list[PublicIdentifier]), then hit an account-level 429 limit:0 unrelated to app code; user switched to AnthropicLLMProvider, which then verified the entire pipeline live with real APIs: real discovery (Brave + Claude) → real company creation → real stripe.com crawl on first run (23 items, 3/3 sources successful) → a real, evidence-grounded, correctly-hedged baseline report from claude-sonnet-5. Also found and fixed live: _resolve_official_website picking Brave's top-ranked Wikipedia result over the real corporate domain for well-known companies, which corrupted downstream source-preview URLs - see KNOWN_LIMITATIONS.md

Phase 12 — UI Polish & Bug-Fix Round (post-live-testing feedback)

User drove the live app (real Brave + Anthropic keys) and reported 9 concrete issues. All addressed:

• Company detail page's delete-confirmation control now animates in/out with a grid-template-columns + opacity transition (200ms, matching the app's existing fade-in timing) instead of an instant DOM swap that also instantly shifted the Pause/Run now buttons sideways
• Add-Company wizard's "What do you want to know?" field now shows placeholder text summarizing the categories the app actually monitors for (products/pricing, leadership, hiring, financial signals, M&A, patents, expansion, regulatory/legal, competitor positioning) — was previously blank with no guidance
• CompanyProfileExtraction/DiscoveredCompanyProfile gained a description field (app/prompts/company_profile.py, app/schemas/discovery.py, app/services/discovery_service.py) - the Review step's Description box was always blank before because discovery never produced one; MockLLMProvider's heuristic derives it from the fetched homepage's first real sentence, real providers extract it from evidence like every other field
• Wizard's Confirm step no longer flashes the full confirm view before redirecting - a new isFinalizing state swaps in a "Setting up monitoring for X…" spinner the instant "Create company" is clicked and stays there through the redirect (previously createCompany.isPending flipped false before router.push completed, causing a one-frame flash of the re-enabled button/full confirm content)
• Company detail page's Details panel switched from side-by-side dt/dd (which produced a hanging-indent wrap for long values like a multi-clause Headquarters string) to a stacked label-above-value layout with break-words
• Competitors on the company detail page are now clickable: hovering highlights them, and clicking navigates to that competitor's own company page if it's already monitored (case-insensitive name match against the user's company list) or to /companies/new?name=<rival> (wizard reads the name query param and pre-fills the Discover step) if not - verified live both ways (Stripe → PayPal went to the wizard pre-filled, then after creating PayPal for real, the same link went straight to its company page)
• Investigated "16 points but basically empty" report finding for the live Stripe company - not a bug: the report was generated before any monitoring run had ever collected evidence (0 source documents, 0 detected changes), and the real report correctly refused to fabricate findings, explicitly stating "insufficient evidence" throughout rather than hallucinating - exactly the evidence-grounded behavior this app is designed around. Fixed the actual gap, which was a missing warning: the Latest Report tab now shows an inline notice when generating a report with zero monitoring runs, and Generate now/"Run now" ordering is explained rather than silently producing a thin report
• Investigated empty Sources tab for the same company - also not a bug: no monitoring run had ever executed for that company, and Source rows are only ever created lazily on first collection (by design, since Phase 4/5). Fixed the gap: the empty-state message now explicitly says sources appear after the first "Run now" rather than a bare "No sources configured yet."
• Built the previously-unimplemented Snapshots tab end-to-end: new GET /companies/{id}/snapshots endpoint (app/api/v1/snapshots.py, app/services/snapshot_service.py, SnapshotRepository.list_for_company added to app/repositories/source_repository.py, newest-first, capped at 50), SnapshotResponse schema, frontend useSnapshots hook + api.listSnapshots, and an expandable list UI (source name, type, timestamp, char count when collapsed; hash, full text summary, and pretty-printed structured summary when expanded) - verified live against real PayPal snapshots (real scraped homepage/careers/GitHub text, expand/collapse working)
• 214 backend tests (up from 211: 3 new for the snapshots endpoint - empty list, newest-first ordering via a directly-inserted Snapshot row, ownership isolation), 18 frontend tests unchanged (existing wizard test extended to assert the description field pre-fills)
• Verified live end-to-end in Docker with real Brave + Anthropic keys: ran the full wizard for PayPal (Stripe's competitor), confirmed paypal.com resolved directly (not Wikipedia, confirming the Phase 11 fix still holds), description field populated with real Claude-extracted text, a real monitoring run (50 items, 3/3 sources) populated the Sources and Snapshots tabs with real data, and the Stripe↔PayPal competitor cross-link resolved correctly both before and after PayPal existed as a monitored company

Phase 13 — Second UI Polish & Bug-Fix Round (wizard nav, dedup, notification linking, report grounding)

User reported 4 more issues after driving the app further. All addressed:

• Wizard's Back button on the Discover step no longer stays permanently disabled - goBack() now calls router.back() when step === 0 instead of a no-op, so it correctly returns to whichever of the 3 entry points (Dashboard, Companies, or a competitor link) the user actually came from. Verified live via a real click-through chain (Companies → wizard → Back → landed back on /companies)
• Add-Company wizard now detects likely duplicate company names client-side (normalizeCompanyName/findPossibleDuplicate in apps/web/app/(app)/companies/new/page.tsx - strips legal suffixes/punctuation, then exact/substring match against the user's existing companies) before spending a real search+LLM call on discovery, shows a "You might already be monitoring X" warning with a link to the existing company and a "Continue anyway" override, and re-checks automatically if the name is edited afterward. Backend now guarantees the persisted name is unique per user regardless (company_service._unique_display_name, mirroring the existing _unique_slug pattern) - "Stripe" → "Stripe (2)" → "Stripe (3)" on collision, filesystem-style. Verified live: typing "Stripe" warned correctly, no discovery API call fired until "Continue anyway", and the created company was actually named "Stripe (2)"
• Notification destinations are now linked to specific companies instead of being flat per-user rows every destination implicitly applied to every company. New notification_destination_companies join table (migration 60a25ddfc6a3, includes a data backfill+dedup step - see below), NotificationDestinationRepository.list_for_company/link_company/find_by_value/delete_orphaned_for_user, POST /notification-destinations now requires company_ids: list[UUID] (min 1) and reuses an existing destination by (user, type, value) instead of duplicating it - this is the actual fix for the wizard silently creating a fresh row per company even when the same email was already registered. alert_service.create_alert_for_change now dispatches via list_for_company(company.id) instead of list_for_user, so a destination only fires for companies it's actually linked to. Deleting a company now garbage-collects any destination left with zero remaining links (company_service.delete_company → delete_orphaned_for_user), with Company.notification_links/NotificationDestination.company_links given explicit ORM cascade="all, delete-orphan" since SQLite (used in dev/tests) doesn't enforce ON DELETE CASCADE without a pragma this app doesn't set - relying on the DB-level FK alone would've silently broken cleanup under SQLite while appearing to work on Postgres
• The one-time migration backfills every existing destination onto every company the same user currently has (a no-op behavior change - it's exactly what already happened implicitly before the join table existed) and then deduplicates rows sharing the same (user, type, normalized value), keeping the earliest and deleting the rest (cascading their NotificationDelivery history, an acceptable one-time cleanup). Verified against the live Postgres DB: went from several duplicate [email protected] rows down to exactly 4 unique destinations, with 48 backfilled links (4 destinations × 12 companies at migration time) - matching the pre-migration behavior exactly
• Settings page redesigned: AddDestinationForm now has a required company multi-select (toggle-chip buttons, at least one required to submit); DestinationRow shows each unique destination once with a horizontally-scrollable row of clickable company chips (overflow-x-auto, hover-highlight, links to /companies/{id}) instead of no company visibility at all. Verified live: scrollWidth (1134px) exceeds clientWidth (384px) on the chip row, confirming it actually scrolls rather than wrapping/clipping; creating a destination linked to only one company and then deleting that company correctly removed the destination from the list (confirmed via direct API calls against the live stack)
• Report generation now receives the company's discovered profile (description, official_website, headquarters, country, region, public_identifiers) as a company_profile evidence block (app/prompts/report_generation.py, threaded through from report_service.py), not just source_documents/detected_changes - this data is genuine evidence (fetched from the company's real website/search results at onboarding), just previously never wired into report generation, which is why reports generated before any monitoring run came back almost entirely "insufficient evidence" even when the company profile had real content. MockLLMProvider._build_report also rewritten to ground company_overview/market_positioning in profile fields. Verified live: generated a report for a brand-new "Airbnb" company with zero monitoring runs - company_overview and market_positioning came back as substantive, evidence-grounded paragraphs (real HQ, industry, named competitors, business model) instead of "insufficient evidence", while sections with genuinely no evidence (financials, hiring, leadership) still correctly said so
• 223 backend tests (up from 214: company name-uniqueness ×3, notification-destination linking/dedup/GC ×6, mock report profile-grounding ×1, plus updates to existing destination/alert tests to pass company_ids), 19 frontend tests (up from 18: new duplicate-warning wizard test)
• Verified live end-to-end against the Docker stack with real Brave + Anthropic keys and the live Postgres DB (migration applied, backfill/dedup confirmed via direct SQL) - see individual bullets above for what was checked

Phase 14 — New Intelligence Sources + Per-Source Scheduling

User compared the live app against the original ChatGPT-authored planning document that inspired it and found real gaps - not a policy problem (bypassing paywalls/logins was explicitly rejected as out of scope again), just free/public sources that were never wired up, plus no way to check a fast-moving source (news) more often than a slow one (patents) within the same company.

• Google News RSS auto-discovery - RssCollector.discover() (app/collectors/rss.py) was previously a stub returning []; now builds https://news.google.com/rss/search?q={company}&hl=en-US&gl=US&ceid=US:en and registers it as a real discoverable source, reusing RssCollector.collect()'s already-real feedparser fetch/parse
• Government contracts via USASpending.gov - new SourceType.GOV_CONTRACT + GovContractCollector (app/collectors/gov_contracts.py), a free/keyless POST /api/v2/search/spending_by_award/, structurally mirroring SecEdgarCollector (always offered, a private company just gets zero results, not an error)
• Patents wired to USPTO's Open Data Portal - new USPTO_API_KEY setting; PatentSourceCollector.collect() now attempts a real POST /api/v1/patent/applications/search call when a key is configured, with the existing honest fixture/disabled fallback completely unchanged (byte-for-byte) for the default no-key case, so none of the 12 existing companies with a dormant PatentSourceCollector regressed
• Per-source check-frequency scheduling - Source gained 4 nullable columns (frequency_type/interval_minutes/cron_expression/next_check, migration 06e7f03cea36); NULL means "inherit the company's default cadence" (the default for every source, zero behavior change unless a source opts in). sync_schedules now enqueues a company if either its own MonitorConfiguration.next_run is due or any of its sources has an independently-due override (SourceRepository.company_has_due_work/list_due_for_company). A SCHEDULED run only collects the sources actually due; a MANUAL "Run now" still collects every active source regardless of cadence, unchanged. Frontend: a per-row "Check frequency" Select on the Sources tab (apps/web/app/(app)/companies/[id]/page.tsx), defaulting to "Same as company", wired through PATCH /sources/{id} (SourceUpdate gained the same 3 fields; source_service.update_source validates the schedule via the existing validate_and_compute_next_run and resets next_check to None so a changed/cleared override takes effect on the very next scheduler tick rather than waiting out the old cadence)
• Explicitly dropped: Nubela/NinjaPear company-enrichment API (confirmed enterprise-only pricing, not accessible to a normal user - no code written); dedicated PRNewswire/BusinessWire/GlobeNewswire collectors (redundant with what Google News RSS already surfaces)
• 240 backend tests (up from 223: 2 RSS discovery-URL tests, 4 gov-contracts collector tests, 5 patents-live-branch tests, 4 scheduler/per-source-due-ness integration tests, 2 source-update-API tests for the new scheduling fields), frontend tsc/eslint/vitest (19 tests) all clean with the new SourceUpdatePayload/useUpdateSource additions
• Verified live end-to-end against the Docker stack: api/worker/beat/web rebuilt, migration 06e7f03cea36 applied to the real Postgres DB, a real "Run now" against Stripe auto-discovered and successfully collected from all 5 sources including the two new ones (GET https://news.google.com/rss/search?q=Stripe... → 200 OK, POST https://api.usaspending.gov/api/v2/search/spending_by_award/ → 200 OK), and the new per-row frequency Select was exercised live in-browser (set "Stripe — Google News" to Daily, confirmed via a re-fetch from the API; cleared it back to "Same as company", confirmed that round-tripped too, both backed by real PATCH /api/v1/sources/{id} → 200 OK calls in the API logs). Patents' real-API branch is unit-tested against a mocked HTTP call only - not live-verified, since it requires a user-supplied USPTO_API_KEY that hasn't been provided yet; the existing no-key fixture path was already covered by the pre-existing patents tests and is unaffected

Phase 15 — NinjaPear Company Enrichment

Phase 14 dropped Nubela/NinjaPear as "enterprise-only" - the user found and paid for an individual $49/mo tier ($49-$1899/mo range) that Phase 14's research missed, then asked for it wired up. Scoped via AskUserQuestion before building: all endpoint categories including customer listing (not just company-level data), but onboarding-only timing (never a recurring per-cycle cost) - confirmed given the API bills real credits per field per request, unlike every other free/flat-rate provider in this app.

• New app/enrichment/ provider package (base.py Protocol + Pydantic result shapes, mock.py honest-empties default, ninjapear.py real per-endpoint httpx calls to nubela.co, factory.py), mirroring app/search/'s shape. New NINJAPEAR_API_KEY/NINJAPEAR_MAX_LEADERSHIP_LOOKUPS settings
• New CompanyEnrichment model (1:1 with Company, migration 79e3aa041131) - status (pending/partial/complete/failed), a single data JSON blob (employee count, leadership team, funding rounds, competitors-with-reasons, products, recent updates, customers), and an errors map so a partially-failed enrichment is never silently presented as complete
• app/services/enrichment_service.py orchestrates ~6 independent per-endpoint calls plus capped per-leadership-member work-email/profile lookups (NINJAPEAR_MAX_LEADERSHIP_LOOKUPS, default 5) - one failed call is recorded in errors and never sinks the others, same principle as tasks/collection.py's per-source loop. New app/tasks/enrichment.py Celery task (own enrichment queue, generous time limits given NinjaPear's documented up-to-5-minute endpoints)
• company_service.create_company enqueues the task post-commit, gated entirely on NINJAPEAR_API_KEY being set - zero extra background-task volume for the overwhelming majority of users who haven't configured it, matching PatentSourceCollector.discover()'s "gate on the key, not the provider" precedent. A status=pending CompanyEnrichment row is created synchronously in the same transaction (not left implicit) so the frontend has something real to poll on
• Report generation gains a company_enrichment evidence block (app/prompts/report_generation.py, threaded through report_service.py) right alongside the existing company_profile block - no new report schema needed, since funding/leadership/competitors/products/customers all map onto existing ReportContent sections (financial_signals, leadership_changes, competitor_comparison, products_and_services, customer_sentiment)
• GET /system/status gains ninjapear_configured/ninjapear_credit_balance (a live, free credit-balance call), shown in Settings' System configuration panel - real-money cost visibility, same treatment the SMS provider status already gets
• Frontend: new "Enrichment" tab on the company detail page (funding table, leadership list with resolved work-email/profile links, competitors-with-reasons kept visually separate from the user's own reviewed Competitors list, products, recent updates, customers, employee count), polling (useCompany's refetchInterval) while status === "pending" so it updates itself once the background task finishes
• Deliberately excluded: the "Similar People" endpoint (a role-anchored prospecting tool with no natural onboarding-time trigger), the Website Lookup endpoint (redundant - official_website already comes from Phase 11 discovery), and auto-merging NinjaPear's suggested competitors into the user-reviewed Company.competitors list (would silently mutate user-controlled data)
• 261 backend tests (up from 240: 8 provider tests incl. mock honesty, 6 orchestration-service tests incl. the leadership cap and partial/failed status derivation, 3 Celery-task integration tests, 2 create-company enqueue-gating tests - the real regression guard, since every other test in the suite runs without a key configured and stays green throughout - 2 report-generation evidence-block tests), frontend tsc/eslint/vitest (19 tests) all clean
• Verified live with the user's real NINJAPEAR_API_KEY and USPTO_API_KEY (both supplied after explicit go-ahead, since NinjaPear spends real credits per call). The initial schema (written from a JS-rendered docs page that couldn't be fully scraped) had real bugs the live pass caught: NinjaPear identifies companies by website only (no name-based lookup), employee_count/industry/funding amounts come back as raw numbers not strings, funding's investors are objects not plain strings, and most response field names differed from the initial guesses (executives, total_funds_raised, x_profile_url, etc. - full list in KNOWN_LIMITATIONS.md). All fixed and re-verified: a real company creation (Stripe) returned real leadership bios, work emails, X profiles, funding history, competitors-with-reasons, live blog updates, and customers for 34 real credits, landing on status: partial (one now-fixed bug, one legitimate 404 for a person not in NinjaPear's database) - and the credit-balance endpoint path/field was also wrong and fixed (/api/v1/meta/credit-balance → credit_balance, not /company/credit-balance → balance). USPTO's real branch (Phase 14, unverified until now) turned out to have its own live-only bugs too - a wrong sort-field path (500 error), title/date fields nested one level deeper than assumed, and 404 "no matching records" being treated as a failure instead of an honest empty result - all fixed. 265 backend tests (up from 261: schema-corrected provider/service tests, a no-website-guard test, and a USPTO 404-as-empty-result test)
• USPTO company-name search follow-up: confirmed (by inspecting a real response's full field list, for a query that did return 110k+ real results by inventor name) that USPTO's Patent Application Search has no queryable assignee/company field at all - not a bug to fix, a real constraint of that dataset. At the user's request, worked around it: CompanyContext gained leadership_names (app/collectors/base.py), threaded through from CompanyEnrichment.data["leadership_team"] in collection_service.to_company_context (new _leadership_names helper - DB-free collectors stay DB-free, the ORM→dataclass seam is the one place allowed to read it). PatentSourceCollector.collect() now searches USPTO by each of the company's leadership names (capped at 5, _MAX_INVENTOR_SEARCHES) instead of by company name, dedupes results by application number across names, and trust-scores every match at 0.5 with explicit "heuristic, not verified" labeling in the document content - a name match is a real signal, not proof of company ownership. Live-verified against Stripe: 19 real patent documents found via its executives' names, zero NinjaPear credits spent (USPTO is free). Self-heals over time: a company whose patents source was discovered before enrichment finished just returns empty until the next scheduled collection re-reads (now-populated) leadership names from the DB. 266 backend tests (up from 265: leadership-name search/dedup, empty-without-names no-network-call, and eager-load fixes to 3 pre-existing integration tests that constructed Company objects directly without loading the new .enrichment relationship)