saksham
1a4c80958f
FastAPI + Celery + Next.js + Postgres/Redis app with company monitoring, source collection, LLM-based change analysis, enrichment, and account security (Turnstile, escalating lockout, email verification).
39 lines
7.5 KiB
Markdown
39 lines
7.5 KiB
Markdown
# Migrating to Firebase
|
||
|
||
This document exists because the spec asks for a documented migration path, not because migrating is recommended. **Recommendation: don't, unless a specific constraint (e.g. "must run on Firebase because the rest of the org's infrastructure does") forces it.** The current stack (Postgres + SQLAlchemy + Celery/Redis) is a better fit for this app's actual access patterns — relational joins across companies/sources/changes/alerts, scheduled background jobs, and multi-step transactional writes — than Firestore's document model and Cloud Functions' request-scoped execution model. The sections below are honest about where the fit is good and where it's actively painful, so the tradeoff is a real decision, not a checkbox.
|
||
|
||
## What would map over cleanly
|
||
|
||
| Current | Firebase equivalent | Notes |
|
||
|---|---|---|
|
||
| `AUTH_MODE=jwt` (email/password, Argon2, refresh tokens) | Firebase Authentication | Firebase Auth's email/password provider replaces `app/auth/jwt.py` and the `refresh_tokens` table almost directly. `get_current_user` becomes "verify the Firebase ID token"; the app stops minting its own JWTs. `AUTH_MODE=local` (fixed dev user) has no Firebase equivalent and would need a dev-only emulator user instead. |
|
||
| SMTP (Mailpit/real SMTP) | Keep as-is, or Firebase Extensions' "Trigger Email" | Firebase doesn't have a first-party SMTP-sending primitive; the official path is the Trigger Email extension (writes to a Firestore collection, an extension sends via SendGrid) or just keep calling SMTP directly from a Cloud Function. Either way, `app/notifications/smtp_email.py`'s logic barely changes. |
|
||
| Twilio SMS | Unchanged | Twilio is already a plain REST call (`app/notifications/twilio_sms.py`); nothing Firebase-specific here either way. |
|
||
| Static frontend hosting | Firebase Hosting | The Next.js app would need to run in static-export or server-mode-on-Cloud-Run form; Firebase Hosting's native Next.js integration (`firebase deploy` with `next build`) handles the common cases. |
|
||
| SSRF-safe fetch, source collectors, change-detection scoring | Unchanged | None of this touches the database engine or auth system; it moves to Cloud Functions/Cloud Run verbatim. |
|
||
|
||
## What gets meaningfully harder
|
||
|
||
- **Relational integrity and cascades.** Every `ondelete="CASCADE"` in `app/models/*.py` (company → sources → source_documents/snapshots → detected_changes → alerts → notification_deliveries) is enforced by Postgres today. Firestore has no foreign keys or cascading deletes — deleting a company would require a Cloud Function that recursively walks and deletes every dependent subcollection/document by hand, and getting that transactionally correct (so a crash mid-delete doesn't leave orphans) is real work that SQL gives you for free.
|
||
- **Multi-table joins.** `CompanyRepository._with_relations` eager-loads a company with its aliases, competitors, and monitor configuration in one query. The alerts list endpoint filters by company/severity/read/resolved with a single indexed SQL query. In Firestore, cross-collection filtering either means denormalizing (duplicating company name/severity onto every alert document, then keeping those copies in sync by hand) or running multiple round-trip queries and joining client-side. This app has a lot of "list X filtered by three unrelated fields across two tables" endpoints (alerts, sources, runs) that would each need its own denormalization strategy.
|
||
- **Scheduled + queued background work.** `Celery Beat` dynamically re-reads `MonitorConfiguration.next_run` every minute with no static per-company schedule entry (see ARCHITECTURE.md). Firebase's closest primitive, Cloud Scheduler + Cloud Functions, is fundamentally *static* — each scheduled job has a fixed cron expression baked in at deploy time. Supporting "company added with a custom schedule at runtime" would mean either (a) one Cloud Scheduler job per company (operationally ugly, has quotas), or (b) keeping a single fixed-interval poller function that queries Firestore for due companies, which is just reimplementing `sync_schedules` on top of a worse-suited datastore. The five-queue Celery setup (default/collection/analysis/notifications/maintenance) with per-queue concurrency also has no direct Cloud Functions equivalent — Cloud Tasks is the nearest analog but needs its own queue-per-purpose provisioning.
|
||
- **Long-running collection work.** A monitoring run fetches multiple sources sequentially with per-domain rate limiting (`SCRAPER_DOMAIN_DELAY_SECONDS`) and can take longer than a single request. Cloud Functions (2nd gen) caps execution time (up to 60 min on Cloud Run-backed functions, but 1st-gen HTTP functions cap at 9 min) — workable, but it changes the current "one task, several minutes, progress recorded incrementally on the run row" model into something that needs to be chunked or moved to Cloud Run directly (which is really just "run Celery/FastAPI on Cloud Run instead of Firebase" — at that point you've mostly kept the current architecture and only swapped the database and auth provider).
|
||
- **Complex evidence structures.** `DetectedChange.raw_diff` and `Report.structured_report` are large nested JSON blobs written and read atomically today via a normal Postgres row. Firestore documents cap at 1 MiB, which is generous for these but not unlimited, and Firestore's lack of a real JSON/JSONB query operator means "find all changes where raw_diff.structured_added contains X" (not currently needed, but the kind of query this schema invites) isn't a native capability the way a Postgres GIN index on JSONB is.
|
||
- **Transactions across the pipeline.** `alert_service.create_alert_for_change` creates an `Alert` row, then loops over notification destinations creating `NotificationDelivery` rows and dispatching sends, all inside one SQLAlchemy session/transaction that commits once at the end. Firestore transactions exist but are limited to 500 documents and have stricter contention semantics; this particular flow would likely still work, but it's not the drop-in swap it sounds like.
|
||
|
||
## If a partial migration is the actual goal
|
||
|
||
The realistic middle ground, if there's a hard requirement to touch Firebase without a full rewrite: **Firebase Authentication only**, keeping Postgres/Celery/FastAPI as-is on Cloud Run. Swap `app/auth/jwt.py` for Firebase Admin SDK token verification, drop the `users`/`refresh_tokens` tables' credential columns (keep `users` as a profile-only table keyed by Firebase UID), and everything else in this document becomes moot. That change is roughly a day of focused work; the full migration described above is a multi-week rewrite of the persistence and scheduling layers, not a port.
|
||
|
||
## Effort estimate (full migration)
|
||
|
||
Rough order of magnitude, assuming the person doing it already knows both stacks well:
|
||
|
||
- Auth swap: 1–2 days.
|
||
- Data model redesign + denormalization strategy for every filtered-list endpoint: 1–2 weeks.
|
||
- Scheduling/queue re-architecture (Cloud Scheduler + Cloud Tasks, or just Cloud Run + Celery which defeats the point): 1 week.
|
||
- Rewriting every repository (`app/repositories/*.py`) against the Firestore SDK, including cascade-delete logic currently free from Postgres: 1–2 weeks.
|
||
- Re-verifying the full change-detection/alerting pipeline against the new datastore's consistency model: 3–5 days.
|
||
|
||
Total: roughly 4–6 weeks for one engineer, plus the ongoing cost of Firestore's per-document-read/write pricing model at the query volume this app generates (every alert list, every dashboard load, every monitoring run touches many documents).
|