# Bulk Gmail Mailer — Production Build

## A. Environment & Scaffolding
- [x] A1. Install/verify PostgreSQL, Redis, Python deps (or document docker-compose path)
- [x] A2. Project layout, pyproject/requirements, .env.example, config module
- [x] A3. Database models (18 tables), migrations (Alembic), encryption layer (Fernet)
- [x] A4. Security primitives: password hashing, JWT, CSRF, rate limiting, audit log, upload validation

## B. Core Backend
- [x] B1. Auth: user registration/login, Google OAuth 2.0 start/callback (state verification), token storage encrypted
- [x] B2. Gmail accounts manager: connect/test/reauthorize/disconnect (revoke tokens), Clear All, per-account limits
- [x] B3. Gmail JSON/client-config import + validation (no secrets exposed), never plaintext
- [x] B4. Recipients: manual/CSV/TXT import, validation (syntax, dupes, suppression, bounces), preview, max 2000
- [x] B5. Subject lines manager (CRUD, search, import/export, Clear All)
- [x] B6. HTML templates manager (paste/upload/editor/preview/duplicate/import/Clear All) + sanitization
- [x] B7. Variable engine (all %vars%, custom CSV columns, unknown-var warnings)
- [x] B8. Invoice number generator (configurable format, crypto-strong, collision check, per-campaign unique)
- [x] B9. Random financial generators (RandomTotal, RandomShippingCharge, admin-configurable ranges)
- [x] B10. Attachment system (types, HTML→PDF via WeasyPrint, unique filenames, malware-scan hook)
- [x] B11. Company phone numbers manager (manual/CSV, selection rule) → %CompanyPhone%
- [x] B12. Suppression list + unsubscribe flow (public tokenized link + POST /unsubscribe)
- [x] B13. Campaign wizard (12 steps) + preview endpoint rendering real messages
- [x] B14. Template sequencing modes (repeat/stop/random), subject rotation, account rotation (sequential/weighted, cooldown, per-account limits, auto-pause)

## C. Sending Engine
- [x] C1. Celery app + queue config, per-recipient jobs, rate limiting, retry/backoff, DLQ, pause/resume/cancel
- [x] C2. Real Gmail API send (OAuth2, MIME construction, message ID/thread ID capture) + OAuth2 SMTP alternative
- [x] C3. Status model: queued/submitted/accepted/temp_failed/perm_failed/suppressed/bounced — no fake success
- [x] C4. SSE live progress stream + campaign monitor endpoints
- [x] C5. Send Test Email flow with provider-confirmed results
- [x] C6. Campaign logs + CSV export

## D. Frontend
- [x] D1. Layout, responsive CSS (13 nav pages), professional UI
- [x] D2. Pages: Dashboard, Gmail Accounts, Recipients, Subject Lines, HTML Templates, Saved Templates, Attachments, Company Phones, Campaigns (wizard+monitor), Send Test, Logs, Suppression, Settings
- [x] D3. Live progress via SSE, confirm dialogs for destructive ops (fixed route conflict: recipients/subjects/templates root GETs moved to /list)

## E. Tests & Docs
- [ ] E1. pytest suite (OAuth, tokens, CSV/TXT, validation, variables, invoice gen, uniqueness, PDF, MIME, retry, rotation, clear-all, etc.)
  - [x] test_variable_engine.py (21) — invoice format, collision, ranges, substitution, analysis
  - [x] test_recipient_service.py (35) — parse txt/csv, aliases, validation, 2000 cap
  - [x] test_message_builder.py (28) — MIME, attachments, List-Unsubscribe, PDF, sanitize
  - [x] test_security_rotation.py (51) — Fernet, bcrypt, JWT, classify_error, RotationAssigner, selectors
  - [x] test_gmail_service.py (28) — client json, state, auth URL, token exchange/refresh, SMTP XOAUTH2
  - [x] test_api_recipients_subjects_templates.py (41) — auth/CSRF/recipients/subjects/templates APIs
    - App fixes driven by tests: CSRF added to 25 endpoints (recipients/subjects/templates/logout), empty-subject validation, google_start honest 400, TXT import header-skip bug
  - [x] test_api_attachments_phones_suppression_settings_logs.py (56) \u2014 attachments (upload/scan/kinds/download/preview/convert-pdf/Clear All), phones, suppression, settings, logs/DLQ/audit/export
    - App fixes driven by tests: HTML attachment upload now allows %variables% (substituted per-recipient at send), convert-pdf substitutes variables with sample values before WeasyPrint render (real %PDF%), conftest truncate list had wrong table names (html_templates/suppression_list) leaking settings rows across tests
  - [x] test_api_accounts_campaigns_testsend_unsubscribe.py (52) \u2014 accounts (list/shape/patch/revoke/DELETE/clear-all/import-config/connect real URL/isolation), test-send (honest failure without creds, substitution pipeline, CSRF), unsubscribe (landing/confirm/suppression/EmailEvent/idempotent), campaigns (wizard/create/patch/delete/clear-all skipping running/validate-step/preview with unique invoice numbers/start materializes+queues with stagger/pause/resume/cancel/progress/monitor/export/SSE done-frame/isolation)
    - App fixes driven by tests: (8) account.oauth_token lazy-load raised MissingGreenlet on every /gmail/accounts listing with tokens \u2192 relationship now eager-loaded (lazy="joined"); (9) DELETE /gmail/accounts/{id} and /accounts/clear-all claimed "removed" but only revoked \u2192 now really delete rows (oauth_tokens cascade, send FKs SET NULL); (10) test-send _FakeRecipient lacked __table__ \u2192 AttributeError 500 \u2192 plain dict recipient; (11) test-send validated HTML before variable substitution, wrongly rejecting %Variables% bodies \u2192 validate after substitution (same advisory filter as attachments)
    - Test infra: starlette TestClient runs the whole ASGI app before returning a response, so endless SSE streams deadlock \u2014 SSE test uses a background thread flipping campaign status to completed mid-stream so the generator terminates itself (connected \u2192 progress \u2192 keepalives \u2192 done + payload asserts)
  - [x] test_campaign_engine.py (materialization, sequencing, uniqueness) — 21/21. Real app bug #12: due_date was today+due_days even with custom invoice_date → 2026 due on a 2031 invoice; fixed in BOTH campaign_engine.py + send_tasks.py (due = effective invoice_date + due_days, safe parse helper). Teardown ObjectInUse fixed: _async_materialize/_run_async now use PRIVATE NullPool async engines (app pool survived asyncio.run → blocked session DROP DATABASE). test-side: preview defaults to campaign preview_sample_size(5) → pass explicit preview_size=8. Also de-flaked test_variable_engine collision test (scripted RNG enumerating the 100-value space; rejection sampling had ~60% miss odds near exhaustion). Regression: 333 passing (281 + 52 API).
  - [x] test_send_one_task.py (retry/backoff/DLQ, statuses, mocked Gmail) — 20/20. Real app bugs found & fixed:
    - #13: quota/429 cooldown logic lived in the PERMANENT branch of _handle_send_error, but classify_error always classifies quota/429/rate-limit text as TEMPORARY (checked first) → cooldown was unreachable dead code. Moved the 30-min account cooldown into the temporary-error path (send_tasks.py).
    - #14: build_mime_message attached PDFs/files as siblings inside multipart/alternative (RFC 2046 violation — alternative must only hold the SAME content in different formats; mail clients could misrender/drop the attachment). Fixed: multipart/mixed[ alternative[text,html], attachment... ] structure (message_builder.py). Verified via real iter_attachments()/pdftotext.
    - #16: `float(campaign.send_interval_seconds or 1.0)` silently coerced an explicit 0 (disable rate limiting) to 1.0, in BOTH campaign_service.py (stagger) and send_tasks.py (per-account rate window). Fixed with `is not None` checks.
    - Test-infra bug (not app): test_security_rotation.py does a top-level `from app.tasks.send_tasks import classify_error`, executed at pytest COLLECTION time — before the session-scoped test_app fixture creates the test DB / reloads app.db.session. That cached app.tasks.send_tasks in sys.modules with SyncSession bound to the wrong engine, so send_one() couldn't see freshly-seeded rows ("campaign deleted") whenever both modules ran in the same session. Fixed by having the send_mod fixture depend on test_app and importlib.reload(app.tasks.send_tasks) before patching.
    - Also disposed app.db.session.sync_engine at module teardown (autouse) to prevent ObjectInUse on the session-scoped DROP DATABASE.
    - Full regression: 353 passing.
  - [x] test_sse_dashboard.py — 13/13. Covers /progress (draft zero-state, ownership 404, manual multi-status counters/percent), /recipients monitor (status filter, case-insensitive search, ownership, empty-before-start), and SSE /stream across ALL terminal states (completed already covered elsewhere; added cancelled, failed, no-premature-done-while-running, progress-payload shape parity with /progress, ownership 404). No new app bugs found — reused the background-thread-flips-status-mid-stream pattern for starlette TestClient's whole-app-to-completion SSE limitation. Full regression: 366 passing.
- [ ] E2. Integration test gated on real Gmail credentials (skips otherwise)
- [ ] E3. Dockerfile, docker-compose (api/worker/postgres/redis), gunicorn/uvicorn prod config, HTTPS notes
- [ ] E4. README: Google Cloud OAuth setup, Gmail API setup, Postgres/Redis setup, workflow, compliance notes

## F. Verification & Delivery
- [ ] F1. Run full test suite + boot the app against real Postgres/Redis; verify endpoints/UI
- [ ] F2. Package source (zip) and deliver with docs