saksham
1a4c80958f
FastAPI + Celery + Next.js + Postgres/Redis app with company monitoring, source collection, LLM-based change analysis, enrichment, and account security (Turnstile, escalating lockout, email verification).
83 lines
11 KiB
Markdown
83 lines
11 KiB
Markdown
# Architecture
|
|
|
|
## Overview
|
|
|
|
```
|
|
apps/web (Next.js) <---REST/JSON---> apps/api (FastAPI) <---> PostgreSQL
|
|
| \
|
|
| ---> Redis (Celery broker + cache)
|
|
|
|
|
Celery worker(s) + Celery Beat
|
|
|
|
|
collectors -> LLM analysis -> change detection -> notifications
|
|
```
|
|
|
|
Everything runs via `docker-compose.yml`: `postgres`, `redis`, `api`, `worker`, `beat`, `web`. Each backend concern (`apps/api/app/*`) is organized by responsibility, not by HTTP route, so services can be reused from Celery tasks and from route handlers alike.
|
|
|
|
## Provider abstraction pattern
|
|
|
|
Every external dependency that costs money, requires a credential, or might later move to a different vendor (including a future Firebase/GCP migration) sits behind a small `Protocol` interface with a `Mock`/`Console` implementation that requires no configuration and is what the app uses by default:
|
|
|
|
| Concern | Interface | Implementations |
|
|
|---|---|---|
|
|
| Auth | `AuthProvider` (`app/auth/base.py`) | `LocalAuthProvider`, `JWTAuthProvider` |
|
|
| LLM | `LLMProvider` (`app/analysis/llm/base.py`) | `MockLLMProvider`, `AnthropicLLMProvider`, `OllamaLLMProvider`, `GeminiLLMProvider` |
|
|
| Search | `SearchProvider` (`app/search/base.py`) | `MockSearchProvider`, `BraveSearchProvider` |
|
|
| Notifications | `NotificationProvider` (`app/notifications/base.py`) | `ConsoleProvider`, `SmtpEmailProvider`, `TwilioSmsProvider` |
|
|
| Source collection | `SourceCollector` (`app/collectors/base.py`) | `WebsiteCollector`, `RssCollector` (incl. Google News auto-discovery), `CustomUrlCollector`, `SecEdgarCollector`, `GithubCollector`, `JobPostingCollector`, `GovContractCollector` (USASpending.gov, free/keyless), `PatentSourceCollector` (real USPTO ODP call when `USPTO_API_KEY` is set, fixture fallback otherwise), `ReviewSourceCollector` (fixture) |
|
|
| Company enrichment | `EnrichmentProvider` (`app/enrichment/base.py`) | `MockEnrichmentProvider`, `NinjaPearProvider` (nubela.co, paid/per-credit, real call when `NINJAPEAR_API_KEY` is set) |
|
|
|
|
Selection is via environment variables (`LLM_PROVIDER`, `SEARCH_PROVIDER`, `AUTH_MODE`, etc.), read once in `app/core/config.py` (a pydantic-settings `Settings` singleton) and resolved through small factory functions — never imported directly by call sites. This is what makes the Firebase/GCP migration path (see `docs/FIREBASE_MIGRATION.md`) tractable: swapping `AuthProvider` for a Firebase-backed one, or the DB session for a Firestore client, doesn't ripple through route handlers or Celery tasks.
|
|
|
|
## Data flow: monitoring run
|
|
|
|
1. `MonitorConfiguration.next_run` is reached, or an individual `Source` has its own faster `frequency_type` override that's independently due (Beat), or a user hits "Run Now" (API) → a `MonitoringRun` row is created (`status=queued`) and a `collection.run_monitoring` Celery task is enqueued with the run ID.
|
|
2. The task loads active `Source` rows for the company. For a `SCHEDULED` trigger, only sources actually due right now are collected (a source with no override rides the company's own cadence; one with an override uses its own `next_check`, via `SourceRepository.list_due_for_company`) - a `MANUAL` "Run now" always collects every active source regardless of individual cadence. For each, the task calls the matching `SourceCollector.collect()`, going through `app/core/http.safe_fetch()` (SSRF-guarded) for anything network-bound.
|
|
3. Each successful collection produces a `SourceDocument` (raw extracted text + metadata) and updates the run's `sources_successful` / `sources_failed` counters. A `Snapshot` (structured summary + hash) is derived per source.
|
|
4. `change_detection` compares the new `Snapshot` against the most recent prior one for the same source, layer by layer (hash → structured fields → bounded text diff → optional LLM semantic check), producing a `DetectedChange` with a `significance_score`, `confidence`, and `severity`.
|
|
5. `analysis` (LLM) generates/updates the `Report` for the company from the accumulated evidence (`SourceDocument`s + `DetectedChange`s), never inventing facts outside that evidence.
|
|
6. For each `DetectedChange` above the user's severity threshold, an `Alert` is created (after dedup/cooldown checks) and `notifications` delivers it to each enabled `NotificationDestination` *linked to that company* (via the `notification_destination_companies` join table - a destination can be shared across several companies, e.g. one email registered once but linked to every company the user monitors), recording a `NotificationDelivery`. A destination with zero remaining company links is garbage-collected when the last linking company is deleted.
|
|
7. The frontend polls `MonitoringRun` status and then renders the `Report`/`Alert` once available.
|
|
|
|
## Data flow: company discovery (onboarding)
|
|
|
|
`POST /api/v1/companies/discover` (`app/services/discovery_service.py`) turns a bare company name into a pre-filled, user-editable profile, and is deliberately kept separate from the two pipelines above:
|
|
|
|
- **`SearchProvider`** answers "where should I look" — it returns URLs/snippets, never facts about the company itself.
|
|
- **`SourceCollector`** (the same collectors the monitoring pipeline uses) answers "what's actually there" by fetching real pages.
|
|
- **`LLMProvider`** only ever *analyzes* evidence already fetched by the two above — it is never prompted with "tell me about company X" from its own training data, matching the evidence-grounded pattern every other analysis task in this app follows (`app/prompts/*`).
|
|
|
|
Steps: (1) resolve `official_website` from the user's hint or `search.search(f"{name} official website")`; (2) fetch that homepage via the existing `fetch_with_retries`/`extract_readable_text` helpers; (3) run a few targeted searches (headquarters, competitors, aliases) for more evidence snippets; (4) call `app/prompts/company_profile.py::extract_company_profile` once over all of it; (5) merge — any user-supplied hint wins over the discovered value; (6) call the `website`/`github`/`sec_edgar`/`job_posting` collectors' existing `.discover()` methods against a synthetic, unsaved `CompanyContext` to preview likely sources. **Nothing is persisted by this endpoint** — the wizard's Review step shows the result for editing, and only the final `POST /companies` call (unchanged) writes a `Company` row. Source rows themselves still come from the pre-existing lazy discovery gate in `tasks/collection.py` (first monitoring run, not onboarding) — discovery preview and source persistence intentionally stay two different code paths so a user can preview/abandon without any DB writes.
|
|
|
|
## Data flow: company enrichment (paid, onboarding-only)
|
|
|
|
`app/services/enrichment_service.py` is a third, deliberately separate data-gathering path from the two above, for a paid third-party vendor (NinjaPear/nubela.co) that bills per-field per-request:
|
|
|
|
- Only ever runs **once**, right after a `Company` row is actually committed in `company_service.create_company` — never on a recurring schedule, and never during the discovery *preview* step (which would spend real credits on companies a user might abandon before creating). The enqueue itself (`enrich_company.delay(...)`, routed to its own `enrichment` Celery queue) is skipped entirely unless `NINJAPEAR_API_KEY` is configured, so a user who never opts in gets zero extra background-task volume — same "gate the enqueue on the key, not just the provider" pattern as `PatentSourceCollector.discover()`.
|
|
- A `CompanyEnrichment` row (1:1 with `Company`, same shape as `MonitorConfiguration`) is created with `status=pending` synchronously, in the same transaction as company creation, *before* the Celery task even starts — this gives the frontend something real to poll on (`useCompany`'s `refetchInterval`), since without it "not yet enriched" and "never configured" would both look like `enrichment: null`.
|
|
- The task (`app/tasks/enrichment.py`) calls several independent NinjaPear endpoints (company details, funding, competitors, products, customer/updates listings, plus capped per-leadership-member work-email/profile lookups) through `EnrichmentProvider`; one failed call is recorded in `errors` and never sinks the others, same principle as `tasks/collection.py`'s per-source loop. Overall `status` becomes `complete`/`partial`/`failed` depending on how many of those independent calls actually succeeded.
|
|
- Once present, `CompanyEnrichment.data` feeds into report generation as a `company_enrichment` evidence block (`report_service.py`, `prompts/report_generation.py`) exactly like `company_profile` does — real fetched data, never invented, and skipped entirely (falls back to an honest empty block) if enrichment never ran, is still pending, or failed outright.
|
|
|
|
## Change significance scoring (documented, testable formula)
|
|
|
|
Implemented in `app/change_detection/scoring.py`. Each detected difference starts with a base weight from its category (e.g. `leadership_change=0.9`, `pricing_change=0.6`, `wording_change=0.15`), then is adjusted by:
|
|
|
|
```
|
|
significance = base_weight
|
|
* source_trust_score (0.3 - 1.0)
|
|
* min(1.0, independent_sources / 2) # corroboration bonus, caps at 2 sources
|
|
* focus_match_multiplier (1.3 if matches user's stated focus, else 1.0)
|
|
* recency_multiplier (1.0 if new, 0.5 if a repeat of a prior alert)
|
|
confidence = f(extraction_confidence, source_trust_score, corroboration)
|
|
```
|
|
|
|
`severity` is then a deterministic bucket over `significance * confidence` (see `SEVERITY_THRESHOLDS` in the same module), with a hard rule that `critical` requires `confidence >= 0.7` regardless of score — an uncorroborated single-source signal cannot be labeled Critical. Full unit tests live in `apps/api/tests/unit/test_scoring.py`.
|
|
|
|
## Directory layout
|
|
|
|
See `PLAN.md` for the top-level file tree. Within `apps/api/app`, packages are: `api/v1` (routers only — thin, no business logic), `auth`, `core` (config, logging, http/SSRF, security), `db` (session, base model), `models` (SQLAlchemy ORM), `schemas` (Pydantic request/response), `repositories` (query layer, one per aggregate), `services` (business logic orchestration), `collectors`, `analysis` (LLM provider + prompt tasks), `change_detection`, `notifications`, `tasks` (Celery task definitions, thin wrappers around `services`), `prompts` (prompt templates + response schemas, one module per analysis task).
|
|
|
|
## Why FastAPI routes stay thin
|
|
|
|
Route handlers only: parse/validate input (Pydantic), call one service method, map the result/exception to an HTTP response. All ownership checks, business rules, and orchestration live in `services/`, which is what both the API and Celery tasks call — this avoids duplicating authorization or business logic between the two entry points.
|