Backfill sparse report sections, add enrichment section refresh, and bootstrap first-admin

Reports: the LLM reliably used company_enrichment for prose fields but
inconsistently populated the parallel Finding-list/string-list fields from
the same evidence, even with progressively more explicit prompting. Add a
code-level backfill (products, recent developments, financial signals,
strategic initiatives, regulatory signals, risks/opportunities mirrored
from SWOT, unknowns, monitoring recommendations) that only ever fills in
what the model left empty, never overwrites what it produced.

Enrichment tab: reorder sections (Products/Recent updates before
Customers/Competitors) and add a per-section "Refresh" button that
re-fetches just one of NinjaPear's six independent per-company endpoints
when it came back empty - confirmed live that a data-coverage gap (e.g.
Amazon returning no products) is real provider behavior, not a bug.

Auth: the first account registered on a deployment with zero existing
admins is now auto-promoted to admin, closing the chicken-and-egg gap
where the only path to admin access was direct DB access. Self-heals if
the last admin ever deletes their account.

Also bumps nginx's proxy_read_timeout for api.ciagent.org to cover the
enrichment refresh's synchronous funding-endpoint call (up to 5 minutes
per NinjaPear's docs).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
2026-08-06 17:41:02 -04:00
co-authored by Claude Sonnet 5
parent 1be3e53584
commit 18305b545c
18 changed files with 1087 additions and 53 deletions
+4 -3
View File
@@ -26,7 +26,7 @@ This file is updated as each phase lands. It exists so nothing is silently claim
- Access/refresh tokens are stored in `localStorage` on the frontend for simplicity (see `apps/web/lib/api-client.ts`). This is a reasonable tradeoff for a local-first MVP but is XSS-exposed compared to httpOnly cookies; a production multi-user deployment should move to cookie-based storage with CSRF protection before going live publicly.
- There is no email verification, password reset, or account lockout after repeated failed logins yet (rate limiting mitigates brute force but doesn't lock the account). Password reset is called out in the spec as a "documented future capability" for the MVP.
- Admin role (`User.is_admin`) exists on the model and the local dev user is admin, but no admin-only endpoints exist yet to gate with `require_admin`.
- **Admin role (`User.is_admin`) now gates several real endpoints** (`require_admin` - Server secrets, IP bans, Database viewer, system logs, unban-request review) - this note originally said none existed yet, long since outdated. **The first account ever registered on a deployment with zero existing admins is now automatically promoted to admin** (`auth_service.register`, checked via `UserRepository.count_admins`) - otherwise the only path to admin access was direct DB manipulation, a real chicken-and-egg problem for anyone self-hosting from a clean clone (see DEPLOYMENT.md §8). Checked by admin *count*, not total user count, so it also self-heals if the last admin ever deletes their own account - the next registration becomes admin again rather than leaving the deployment permanently admin-less. A benign race is possible if two people register in the same instant on a brand-new, zero-admin deployment (both could become admin) - acceptable since it only ever matters once, before any real traffic exists, and DEPLOYMENT.md tells operators to register immediately after bringing the stack up, before opening the firewall.
## Collection pipeline (Phase 4)
@@ -57,6 +57,7 @@ This file is updated as each phase lands. It exists so nothing is silently claim
- **Tasks A (relevance), B (extraction), and C (synthesis) are implemented and tested but not yet wired into the collection/report pipeline** - Task D (report generation) currently consumes raw `SourceDocument`/`DetectedChange` rows directly rather than pre-filtering through Task A or pre-extracting via Task B/C. Wiring them in would improve report quality (e.g. filtering irrelevant documents before they reach the report prompt) but the report already works correctly without it, since `report_service.py`'s evidence-gathering only pulls documents/changes already scoped to the company.
- **A report is regenerated only on baseline (first evidence) or when a run detects an actual change** - not on every scheduled run, to keep LLM cost proportional to real activity rather than to schedule cadence. See `app/tasks/collection.py`.
- **No token/cost usage tracking or per-user LLM budget yet** - `LLM_MAX_TOKENS_PER_REQUEST` and `LLM_MAX_RETRIES` bound a single call, but there's no aggregate usage dashboard (spec section 30's "Display usage statistics in settings" is Phase 10/not-yet-done).
- **Report list/string fields are backfilled from `company_enrichment` when the LLM leaves them empty** - confirmed live across multiple real Anthropic calls (even with an explicit worked example in the prompt) that the model reliably uses enrichment data for prose fields (`company_overview`, `market_positioning`) but is inconsistent about mirroring the same data into list-shaped fields (`products_and_services`, `financial_signals`, `strategic_initiatives`, etc.). `report_service.py`'s `_backfill_from_enrichment`/`_mirror_swot_into_flat_lists`/`_backfill_reflective_sections` fill these in mechanically from `company_enrichment.products`/`recent_updates`/`funding`/`specialties`, from the model's own SWOT output, and from the actual presence/absence of evidence - never overwriting content the model did produce, and never inventing anything not already present in real evidence. **`key_inferred_projects` is the one field left with no such fallback** - it asks for specific, speculative in-progress projects, which has no honest mechanical source (unlike a specialty list or a funding round, there's no real "raw project" data to transcribe), so it stays genuinely empty unless the model or actual source documents/detected changes surface something.
## Notifications (Phase 8)
@@ -132,7 +133,7 @@ This file is updated as each phase lands. It exists so nothing is silently claim
- **The "Similar People" endpoint is not wired in at all** - it's a role/company-anchored prospecting tool ("find people like X at competitor Y") with no natural trigger at onboarding time, since no specific role is ever selected. Would need its own UI (e.g. "find people like this one" on a leadership-team entry) to make sense, not an automatic onboarding call.
- **NinjaPear's suggested competitors are never merged into `Company.competitors`** (the user-typed/reviewed list that drives Phase 12's competitor cross-linking) - shown only in the Enrichment tab, with their stated reasons, kept deliberately separate so a background API call can never silently change what the user explicitly reviewed. A "add as competitor" action would be a reasonable follow-up if this becomes a papercut.
- **Person-level lookups (work email, profile) are capped at `NINJAPEAR_MAX_LEADERSHIP_LOOKUPS` (default 5) per company**, applied to however many leadership members NinjaPear's Company Details call happens to return, in whatever order it returns them - there's no ranking by seniority/relevance before the cap is applied, so for a company with a large leadership team, which specific people get resolved is effectively arbitrary.
- **No re-enrichment mechanism** - by design (see Phase 15's Context in `TASKS.md`, onboarding-only was the explicit scoping decision to bound cost), but it does mean `CompanyEnrichment.data` can go stale indefinitely (a funding round happens, a leadership change occurs) with no way to refresh it short of manually deleting the row and re-triggering via the API/DB directly - there's no "re-enrich this company" button anywhere in the UI.
- **No automatic re-enrichment** - by design (see Phase 15's Context in `TASKS.md`, onboarding-only was the explicit scoping decision to bound cost), so `CompanyEnrichment.data` can go stale indefinitely (a funding round happens, a leadership change occurs) with no scheduled refresh. **A manual, per-section refresh does exist** (`POST /companies/{id}/enrichment/sections/{section}/refresh`, `enrichment_service.refresh_section`): each of NinjaPear's six per-company endpoints (details, funding, updates, competitors, products, customers) maps to its own UI section, and a "Refresh" button appears next to a section's empty-state message ("None found.") so a genuine provider-side data gap (confirmed live: NinjaPear's real API returns `products: []` for Amazon specifically, with no error - not a bug, just no product data on their end for that company) can be manually retried without re-running the whole enrichment or burning credits on the sections that already succeeded. Runs synchronously in the request (not queued via Celery like onboarding enrichment), since it's a single provider call rather than the full six-plus-leadership fan-out - nginx's `proxy_read_timeout` on `api.ciagent.org` is set to 320s to cover `funding`'s documented up-to-5-minute worst case.
- **`GET /system/status`'s live credit-balance check adds one outbound HTTPS call (to NinjaPear) to every load of that endpoint** whenever a key is configured - guarded so a failure there can never break the rest of system status (returns `null` for the balance, logs a warning), but it does mean the Settings page's load time now has a dependency on `nubela.co`'s availability/latency for that one field, not just the app's own DB/Redis health.
## Live logging, API-key visibility, and enrichment-tab formatting (Phase 17)
@@ -169,7 +170,7 @@ This file is updated as each phase lands. It exists so nothing is silently claim
- **The old admin-only, localhost-only "API keys" Settings box (`/system/api-keys`, Phase 17) has been removed entirely**, replaced by two independent mechanisms with a deliberately different visibility model each: per-user API keys (Anthropic/Brave/NinjaPear/USPTO - every user manages their own, no admin/localhost gate at all, since each user only ever sees their own value) and a new admin-only (but **not** localhost-gated) "Server secrets" box for values that are genuinely global rather than per-user - today just the Cloudflare Turnstile site key and secret. `SystemSecret` (`app/models/system_secret.py`) is a true singleton-per-key table (one row per `SystemSecretKey`, not scoped to any user), encrypted at rest the same way as `UserApiKey`, with the identical fallback-to-`.env`-when-unset pattern via `system_secret_service.get_effective_settings`.
- **The Turnstile site key is no longer delivered to the frontend via a `NEXT_PUBLIC_*` build-time env var.** It used to be baked into the Next.js bundle at container-build time (`NEXT_PUBLIC_TURNSTILE_SITE_KEY`), which meant an admin-updated value could never take effect without a full frontend rebuild - defeating the point of making it admin-editable. It's now served live by the already-public, unauthenticated `GET /system/status` endpoint (`turnstile_site_key`, resolved through the same effective-settings fallback), and `TurnstileWidget` takes it as a prop instead of reading `process.env` directly. This does mean the site key round-trips through one more network hop (an extra field on a call the login/register pages already make) rather than being inlined - a negligible cost for a value that isn't itself secret.
- **`system_secrets` rows are true global singletons with no natural per-test isolation** (unlike `user_api_keys`, which is naturally isolated by a randomized per-test `user_id`) - `test_system_secrets.py` has to explicitly clear the table before and after every test via an autouse fixture to avoid cross-test pollution within a full-suite run. Worth remembering if this pattern is extended to more server-wide keys later: any new key added to `SystemSecretKey` inherits the same test-isolation requirement.
- **The real `.env` `TURNSTILE_SITE_KEY`/`TURNSTILE_SECRET` values were copied into this DB-backed storage** (both now `configured: true` via the admin Settings box) rather than removed from `.env` outright - the user plans to purge the `.env` values manually once satisfied the DB-backed path works, so both currently agree and nothing changes behaviorally either way. `NEXT_PUBLIC_TURNSTILE_SITE_KEY` specifically *was* removed from `.env`/`.env.example`/`docker-compose.yml`, since nothing reads it anymore (see above) - that one wasn't a "purge later" judgment call, it was genuinely dead as soon as the frontend stopped reading it.
- **The real `.env` `TURNSTILE_SITE_KEY`/`TURNSTILE_SECRET`/`RESEND_API_KEY` values on the production deployment have since been moved into this DB-backed storage via the admin API and removed from `.env` outright** (all three now `configured: true`, set through `PUT /system/secrets/{key}` using the deployment's own admin account rather than hand-edited into `.env`) - the earlier state of this note (values copied into the DB but also left in `.env` "to purge later") has been resolved; `.env` is no longer a source of truth for these three at all on that deployment. `NEXT_PUBLIC_TURNSTILE_SITE_KEY` specifically *was already* removed from `.env`/`.env.example`/`docker-compose.yml` earlier, since nothing reads it anymore (see above).
## Account activity logging expansion: local-dev sign-ins, secret/key updates, known IPs