MarketingTool/docs/OPERATIONS.md

Operations Runbook

Start, inspect, stop

From the repository root:

docker compose up --build -d
docker compose ps
docker compose logs --follow api web
docker compose down

The expected health endpoints are:

• API: GET http://localhost:8000/api/v1/health/live
• Web: GET http://localhost:8080/healthz

A service is ready only when Compose reports healthy; container running status alone is insufficient. Health checks call public liveness endpoints and must remain unauthenticated—do not add a session requirement to /api/v1/health/live or /healthz. The Compose environment explicitly carries AUTOMATED_OUTREACH_ENABLED=false as an operational safety setting.

Phase 3 workflow operations

Phase 3 is a human-operated prospect workflow. Operators manually create a business, add child intelligence/evidence observations with their provenance, review the detail page, add notes, and move the prospect through the permitted pipeline states. The API records the acting user and tenant on state-changing actions and exposes bounded activity/audit history where configured.

• Treat source/provenance fields as required lineage for manual evidence: retain the source reference or label, captured/observed time, and confidence/context supplied by the operator.
• Review pagination metadata and filters when investigating a list. Never infer that a page is the complete tenant dataset, and never use a UI filter as proof of authorization.
• Investigate a missing detail or child record as a possible tenant/parent scope issue before retrying with alternate IDs. Cross-tenant IDs are expected to return not found.
• Pipeline state is coordination metadata only. Suppressed records remain blocked, and no state enables outreach.
• Notes may contain sensitive information. Limit access and avoid copying secrets, credentials, or unnecessary personal data into notes or audit details.
• Audit/activity records are operational evidence of changes, not a replacement for a production-grade immutable audit service.

There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.

Phase 5 source operations boundary

Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.

Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.

dry_run must perform validation/planning only: no network I/O, external adapter side effects, prospect-fact writes, or outreach. CSV and manual references may be previewed or recorded as operator-supplied observations with source attribution and capture time. A preview is not an import, verification, or discovery result.

Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.

Phase 4 jobs and live logging

The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves queued → running → succeeded/failed/cancelled, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.

Creation must use a tenant-scoped idempotency key and request fingerprint. A repeated identical request returns the existing job/attempt; a conflicting payload is rejected. Cancellation is cooperative and race-safe, while retry is an explicit authorized new attempt linked to the original job and must not repeat completed side effects. Do not treat HTTP acceptance as completion, and do not reconstruct history from ephemeral container logs.

The MVP has no SSE handler, durable queue, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.

Configuration and deployment

Copy .env.example for local development. Production values must be supplied by the deployment environment, never committed. BOOTSTRAP_ADMIN_EMAIL and BOOTSTRAP_ADMIN_PASSWORD are optional API environment variables for first-run admin provisioning only; set them together through a secret store or protected deployment environment, remove them immediately after successful bootstrap, and rotate the password. Do not put real values in Compose files, CI variables visible to logs, images, or committed .env files.

For production, use Argon2id for password hashing and require MFA for administrator accounts. Configure TLS before enabling Secure session cookies. Local Compose uses HTTP, so browser testing of production Secure cookie behavior requires an HTTPS staging environment. Treat session cookies as bearer credentials: protect state-changing routes with CSRF controls, expire/revoke sessions, and never print cookie values in logs.

Before deployment:

1. Run docker compose config and review the rendered configuration; optional bootstrap values should be empty in CI and local validation.
2. Build from a reviewed commit and scan the resulting images.
3. Restrict host/network exposure at the ingress/firewall.
4. Verify both unauthenticated health checks and review logs for unexpected errors, cross-tenant errors, or sensitive data.
5. Exercise tenant-scoped list/detail/child routes with bounded pagination and filters, and verify that notes/pipeline changes appear in the intended tenant's audit trail only.
6. Record the image digest and configuration revision for rollback.

Data, backups, and retention

The canonical API runtime under apps/api uses the named Docker volume prospect-platform-api-data.

Any older files under infrastructure/docker/ are not referenced by Compose and are non-canonical; use apps/api and apps/web for all container changes. They can be removed during repository cleanup once no external tooling references them.

Inspect the volume with docker volume inspect prospect-platform-api-data; do not treat a local Docker volume as a backup.

For the current MVP there is no database migration or backup command. If runtime data is material, stop writes first and snapshot/copy the volume using an approved host backup process. Protect business, child intelligence, notes, provenance, and audit data with encryption and access controls, test a restore into an isolated environment, and document the result. Define retention/deletion rules that cover source references and notes as well as contact fields.

When jobs are introduced, backups and retention must include job definitions, idempotency records, terminal results/errors, and persisted sequence events. Verify that restoring a database preserves event ordering/cursors and does not cause a retried worker to repeat side effects. Define event redaction and retention separately from short-lived delivery connections.

Recommended starting policy for a future production data store:

• daily encrypted backups, with at least 30 days of retention;
• point-in-time recovery where supported;
• one offline or separately isolated copy;
• quarterly restore drills, plus a restore test after storage/provider changes;
• retention and deletion schedules aligned with the source/contact policy and applicable law.

Do not run docker compose down -v on a data-bearing environment: it removes the named volume.

Failure handling

• Unhealthy API: inspect docker compose logs api, verify port binding and resource availability, then restart with docker compose restart api if appropriate.
• Unhealthy web: inspect docker compose logs web; confirm port 8080 is available and the image contains /healthz.
• Missing or inconsistent detail: preserve the request identifiers/log context, verify the authenticated tenant and parent-child association, and do not retry by guessing another tenant's ID.
• Audit gap: stop the affected mutation workflow, preserve the database/log evidence, and investigate before allowing operators to rely on the history.
• Build failure: run docker compose build --no-cache from a reviewed checkout and check Docker daemon/network status.
• Unexpected outbound traffic: stop the stack, preserve logs/metadata, and investigate. The MVP has no outreach worker and must not send automated messages.

Production migration and scaling path

Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker operations, metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and scanners are possible future components—not implicit Compose dependencies and not implemented by this MVP. No automated discovery or outreach may be inferred from the scaling path.

Incident checklist

1. Record time, affected service, image/config revision, and observed health state.
2. Preserve relevant logs and audit records without exporting secrets or unnecessary contact data.
3. Stop or isolate the affected service if data loss, unauthorized access, SSRF, provenance tampering, or unexpected outreach is suspected.
4. Rotate exposed credentials through the secret manager.
5. Validate recovery with health checks and a targeted tenant-isolation/detail smoke test.
6. Document root cause, corrective action, and any retention/suppression or audit impact.