MarketingTool/docs/OPERATIONS.md

Operations Runbook

Start, inspect, stop

From the repository root:

docker compose up --build -d
docker compose ps
docker compose logs --follow api web
docker compose down

The expected health endpoints are:

• API: GET http://localhost:8000/api/v1/health/live
• Web: GET http://localhost:8080/healthz

A service is ready only when Compose reports healthy; container running status alone is insufficient. Health checks call public liveness endpoints and must remain unauthenticated—do not add a session requirement to /api/v1/health/live or /healthz. The Compose environment explicitly carries AUTOMATED_OUTREACH_ENABLED=false as an operational safety setting.

Phase 3 workflow operations

Phase 3 is a human-operated prospect workflow. Operators manually create a business, add child intelligence/evidence observations with their provenance, review the detail page, add notes, and move the prospect through the permitted pipeline states. The API records the acting user and tenant on state-changing actions and exposes bounded activity/audit history where configured.

• Treat source/provenance fields as required lineage for manual evidence: retain the source reference or label, captured/observed time, and confidence/context supplied by the operator.
• Review pagination metadata and filters when investigating a list. Never infer that a page is the complete tenant dataset, and never use a UI filter as proof of authorization.
• Investigate a missing detail or child record as a possible tenant/parent scope issue before retrying with alternate IDs. Cross-tenant IDs are expected to return not found.
• Pipeline state is coordination metadata only. Suppressed records remain blocked, and no state enables outreach.
• Notes may contain sensitive information. Limit access and avoid copying secrets, credentials, or unnecessary personal data into notes or audit details.
• Audit/activity records are operational evidence of changes, not a replacement for a production-grade immutable audit service.

There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.

Phase 5 source operations boundary

Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.

Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.

dry_run must perform validation/planning only: no network I/O, external adapter side effects, prospect-fact writes, or outreach. CSV and manual references may be previewed or recorded as operator-supplied observations with source attribution and capture time. A preview is not an import, verification, or discovery result.

Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.

Phase 6 normalization and deduplication operations

Normalization and duplicate review are data-integrity operations, not discovery. Record the normalization and algorithm versions with every derived SA phone/location value and suggestion. Verify that local South African phone forms are interpreted only with explicit +27 context, that original values remain available, and that ambiguous locations are flagged rather than guessed. Re-running the same input/version must produce the same canonical values, score, band, and reasons.

Operate the review queue with the fixed bands: >=0.90 strong suggestion, 0.75–0.8999 review suggestion, <0.75 no suggestion. No band authorizes a merge. A permitted human must choose the survivor and explicitly confirm; operators must be able to reject a suggestion. Before any merge, verify the tenant, permissions, conflict list, and that a complete pre-merge snapshot was durably written. Preserve every child, evidence/provenance item, source-record/raw lineage link, note, pipeline state, and audit entry; never resolve a conflict by deleting the losing source record.

After a merge, read back the survivor and audit trail, check child/evidence/source-record counts and provenance links, and record the snapshot ID. For reversal, use only the snapshot, verify the same tenant and authorization, restore the prior graph transactionally, and read back the restored records and audit event. If snapshot creation, preservation, or audit append fails, stop the mutation and investigate; do not retry blindly. Cross-tenant candidates must be absent, not merely hidden by the UI.

The current Compose/MVP runtime exposes the Phase 6 match-suggestion, explicit merge-confirmation, tenant-scoped merge-history, and reversal routes; no separate worker is required. It has basic canonicalization and exact duplicate handling plus pilot-grade fuzzy review. Monitor merge audit events and verify child/source-record counts after each operation. Do not schedule or report deduplication as automated discovery, enrichment, verification, or outreach.

Phase 7 domain-intelligence operations

Phase 7 is a review contract only; the current Compose stack has no DNS worker, PSL service, DNS cache, or availability provider. Do not add a resolver or provider by pointing a job at an arbitrary URL. Any future implementation must record a pinned PSL version, preserve raw and registrable-domain forms, and report unresolved reasons for public suffixes, unknown/private suffixes, malformed/single-label values, localhost, and IP literals. IDN/punycode handling and subdomain treatment must be deterministic.

Operate DNS as bounded, observable evidence collection. Surface not_checked, pending, resolved, nxdomain, no_data, timeout, servfail, blocked, and error; never translate an empty/error result into a business conclusion. Keep MX, NS, and TXT records separate and record resolver/source, observed time, TTL, freshness/staleness, truncation/partial state, and uncertainty/error reason. A cached answer is not a fresh check: caches must be bounded, tenant-isolated, keyed by normalized query/type/class and resolver policy, honor TTL, and be invalidated/re-evaluated when the PSL or resolver policy changes.

Review domain-to-business candidates separately from DNS results. Verify tenant scope before comparison, reject public-suffix-only and malformed/IP candidates, and flag shared, parked, wildcard, homograph, sibling-subdomain, and conflicting-evidence cases. Association confidence must show evidence, reasons, algorithm/version, freshness, and uncertainty; it is suggestion-only. Require an authorized human accept/reject decision and audit it. Never auto-attach a domain or treat DNS as proof of ownership.

Availability is unknown unless the API reports a result from an authorized provider. Before enabling one, verify current product/legal/security approval, terms owner, allowed tenant scope, request/concurrency and timeout limits, retention/deletion class, health/circuit state, and explicit operational enablement. nxdomain, no_data, timeout, stale cache, or provider error is not “available.” On provider outage, rate-limit, terms/approval expiry, circuit-open, or conflicting result, fail closed and report unknown/deferred; never purchase, reserve, contact, or retry through an unapproved provider.

Phase 8 website-scanning operations

Website scans are bounded evidence collection, not browser sessions. The API must accept only http/https, resolve and validate DNS/IP immediately before connection, block loopback/private/link-local/multicast/reserved/cloud-metadata destinations, and repeat those checks for every redirect. Review redirect chains for protocol or host-policy violations; DNS rebinding or an unsafe resolved address is a blocked scan, not a retry opportunity.

Monitor per-tenant and global scan counts, active concurrency, queue age, total/connect/read timeouts, response/decompressed bytes, redirect and crawl depth/link counts, cache hit/freshness, blocked destinations, DNS failures, partial results, and classification/error rates. Enforce hard budgets across redirects and discovered links. Classifications are conservative observations only: never submit forms, send cookies/credentials, execute JavaScript, or treat an HTTP 200/empty page as proof of a business fact. A scan that hits a limit is explicitly incomplete/unknown.

Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.

Phase 9 official-site contact extraction operations

Operate contact extraction as passive observation of an approved/public official-site origin, limited to bounded same-site contact/about pages. Before enabling it, verify tenant scope, official-site approval, extractor/policy version, retention class, suppression source, and configured hard limits for pages/URLs, redirects, bytes, time, candidates, and concurrency. Do not add arbitrary URLs or search results to the scope.

Review each result with its source/page URL and context, extraction method, observed time, confidence/reasons, syntax status, role/free-mail labels, and independent MX/DNS status/freshness. Treat not_checked, nxdomain, no_data, timeout, servfail, blocked, and error as uncertainty—not invalidity or non-deliverability. Syntax, role, free-mail, and MX labels never authorize contact. Values found in scripts/styles/assets/file names, examples/placeholders, tracking addresses, malformed schemes, or third-party content are false positives and must be excluded or quarantined.

Verify suppression matching before persistence, response, cache, export, or review-queue insertion. Suppressed contacts remain do not contact, regardless of later confidence, classification, syntax, MX, pipeline, or verification changes. Monitor extraction attempts, pages/bytes/candidates, limit hits, blocked destinations, parse errors, false-positive exclusions, suppression matches, cache freshness, retention/deletion jobs, and provenance/audit failures. On a suppression or provenance failure, stop the affected write path and investigate; do not retry blindly.

There is no SMTP probing, SMTP VRFY/EXPN, validation email, outreach worker, campaign queue, or follow-up action. Never contact a discovered address. If extraction is disabled, unapproved, out of budget, or uncertain, report deferred/blocked/unknown with the reason. Retain only the minimum value and lineage for the approved retention period; redact addresses and page content from routine logs.

Phase 4 jobs and live logging

The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves queued → running → succeeded/failed/cancelled, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.

Creation must use a tenant-scoped idempotency key and request fingerprint. A repeated identical request returns the existing job/attempt; a conflicting payload is rejected. Cancellation is cooperative and race-safe, while retry is an explicit authorized new attempt linked to the original job and must not repeat completed side effects. Do not treat HTTP acceptance as completion, and do not reconstruct history from ephemeral container logs.

There is no SSE handler, durable queue, scan worker/isolation boundary, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.

Configuration and deployment

Copy .env.example for local development. Production values must be supplied by the deployment environment, never committed. BOOTSTRAP_ADMIN_EMAIL and BOOTSTRAP_ADMIN_PASSWORD are optional API environment variables for first-run admin provisioning only; set them together through a secret store or protected deployment environment, remove them immediately after successful bootstrap, and rotate the password. Do not put real values in Compose files, CI variables visible to logs, images, or committed .env files.

For production, use Argon2id for password hashing and require MFA for administrator accounts. Configure TLS before enabling Secure session cookies. Local Compose uses HTTP, so browser testing of production Secure cookie behavior requires an HTTPS staging environment. Treat session cookies as bearer credentials: protect state-changing routes with CSRF controls, expire/revoke sessions, and never print cookie values in logs.

Before deployment:

1. Run docker compose config and review the rendered configuration; optional bootstrap values should be empty in CI and local validation.
2. Build from a reviewed commit and scan the resulting images.
3. Restrict host/network exposure at the ingress/firewall.
4. Verify both unauthenticated health checks and review logs for unexpected errors, cross-tenant errors, or sensitive data.
5. Exercise tenant-scoped list/detail/child routes with bounded pagination and filters, and verify that notes/pipeline changes appear in the intended tenant's audit trail only.
6. If Phase 9 is enabled, run official-site fixtures covering provenance/confidence, role and free-mail labels, syntax failures, every MX/DNS uncertainty state, false-positive exclusions, limit exhaustion, retention/deletion, and suppression-before-persistence/response/export. Confirm no SMTP or outreach network activity.
7. Record the image digest and configuration revision for rollback.

Data, backups, and retention

The canonical API runtime under apps/api uses the named Docker volume prospect-platform-api-data.

Any older files under infrastructure/docker/ are not referenced by Compose and are non-canonical; use apps/api and apps/web for all container changes. They can be removed during repository cleanup once no external tooling references them.

Inspect the volume with docker volume inspect prospect-platform-api-data; do not treat a local Docker volume as a backup.

For the current MVP there is no database migration or backup command. If runtime data is material, stop writes first and snapshot/copy the volume using an approved host backup process. Protect business, child intelligence, notes, provenance, and audit data with encryption and access controls, test a restore into an isolated environment, and document the result. Define retention/deletion rules that cover source references and notes as well as contact fields.

When jobs are introduced, backups and retention must include job definitions, idempotency records, terminal results/errors, and persisted sequence events. Verify that restoring a database preserves event ordering/cursors and does not cause a retried worker to repeat side effects. Define event redaction and retention separately from short-lived delivery connections.

Recommended starting policy for a future production data store:

• daily encrypted backups, with at least 30 days of retention;
• point-in-time recovery where supported;
• one offline or separately isolated copy;
• quarterly restore drills, plus a restore test after storage/provider changes;
• retention and deletion schedules aligned with the source/contact policy and applicable law.

Do not run docker compose down -v on a data-bearing environment: it removes the named volume.

Failure handling

• Unhealthy API: inspect docker compose logs api, verify port binding and resource availability, then restart with docker compose restart api if appropriate.
• Unhealthy web: inspect docker compose logs web; confirm port 8080 is available and the image contains /healthz.
• Missing or inconsistent detail: preserve the request identifiers/log context, verify the authenticated tenant and parent-child association, and do not retry by guessing another tenant's ID.
• Audit gap: stop the affected mutation workflow, preserve the database/log evidence, and investigate before allowing operators to rely on the history.
• Build failure: run docker compose build --no-cache from a reviewed checkout and check Docker daemon/network status.
• Unexpected outbound traffic: stop the stack, preserve logs/metadata, and investigate. The MVP has no outreach worker and must not send automated messages.

Production migration and scaling path

Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, domain observations, and scan history/cache, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker and scanner-isolation operations, bounded DNS/PSL/website processing, TTL/freshness-aware cache invalidation, SSRF/DNS-rebinding/redirect-chain tests, hard size/time/crawl budgets, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS/scanner status/error rates, cache freshness, blocked destinations, crawl-budget exhaustion, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and production scanners are possible future components—not implicit Compose dependencies. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.

Incident checklist

1. Record time, affected service, image/config revision, and observed health state.
2. Preserve relevant logs and audit records without exporting secrets or unnecessary contact data.
3. Stop or isolate the affected service if data loss, unauthorized access, SSRF, provenance tampering, or unexpected outreach is suspected.
4. Rotate exposed credentials through the secret manager.
5. Validate recovery with health checks and a targeted tenant-isolation/detail smoke test.
6. Document root cause, corrective action, and any retention/suppression or audit impact.