MarketingTool/docs/OPERATIONS.md

# Operations Runbook

## Start, inspect, stop

From the repository root:

```sh
docker compose up --build -d
docker compose ps
docker compose logs --follow api web
docker compose down
```

The expected health endpoints are:

- API: `GET http://localhost:8000/api/v1/health/live`
- Web: `GET http://localhost:8080/healthz`

A service is ready only when Compose reports `healthy`; container running status alone is insufficient. Health checks call public liveness endpoints and must remain unauthenticated—do not add a session requirement to `/api/v1/health/live` or `/healthz`. The Compose environment explicitly carries `AUTOMATED_OUTREACH_ENABLED=false` as an operational safety setting.

## Phase 3 workflow operations

Phase 3 is a human-operated prospect workflow. Operators manually create a business, add child intelligence/evidence observations with their provenance, review the detail page, add notes, and move the prospect through the permitted pipeline states. The API records the acting user and tenant on state-changing actions and exposes bounded activity/audit history where configured.

- Treat source/provenance fields as required lineage for manual evidence: retain the source reference or label, captured/observed time, and confidence/context supplied by the operator.
- Review pagination metadata and filters when investigating a list. Never infer that a page is the complete tenant dataset, and never use a UI filter as proof of authorization.
- Investigate a missing detail or child record as a possible tenant/parent scope issue before retrying with alternate IDs. Cross-tenant IDs are expected to return not found.
- Pipeline state is coordination metadata only. Suppressed records remain blocked, and no state enables outreach.
- Notes may contain sensitive information. Limit access and avoid copying secrets, credentials, or unnecessary personal data into notes or audit details.
- Audit/activity records are operational evidence of changes, not a replacement for a production-grade immutable audit service.

There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.

## Phase 5 source operations boundary

Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.

Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.

`dry_run` must perform validation/planning only: no network I/O, external adapter side effects, prospect-fact writes, or outreach. CSV and manual references may be previewed or recorded as operator-supplied observations with source attribution and capture time. A preview is not an import, verification, or discovery result.

Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.

## Phase 6 normalization and deduplication operations

Normalization and duplicate review are data-integrity operations, not discovery. Record the normalization and algorithm versions with every derived SA phone/location value and suggestion. Verify that local South African phone forms are interpreted only with explicit `+27` context, that original values remain available, and that ambiguous locations are flagged rather than guessed. Re-running the same input/version must produce the same canonical values, score, band, and reasons.

Operate the review queue with the fixed bands: `>=0.90` strong suggestion, `0.75–0.8999` review suggestion, `<0.75` no suggestion. No band authorizes a merge. A permitted human must choose the survivor and explicitly confirm; operators must be able to reject a suggestion. Before any merge, verify the tenant, permissions, conflict list, and that a complete pre-merge snapshot was durably written. Preserve every child, evidence/provenance item, source-record/raw lineage link, note, pipeline state, and audit entry; never resolve a conflict by deleting the losing source record.

After a merge, read back the survivor and audit trail, check child/evidence/source-record counts and provenance links, and record the snapshot ID. For reversal, use only the snapshot, verify the same tenant and authorization, restore the prior graph transactionally, and read back the restored records and audit event. If snapshot creation, preservation, or audit append fails, stop the mutation and investigate; do not retry blindly. Cross-tenant candidates must be absent, not merely hidden by the UI.

The current Compose/MVP runtime exposes the Phase 6 match-suggestion, explicit merge-confirmation, tenant-scoped merge-history, and reversal routes; no separate worker is required. It has basic canonicalization and exact duplicate handling plus pilot-grade fuzzy review. Monitor merge audit events and verify child/source-record counts after each operation. Do not schedule or report deduplication as automated discovery, enrichment, verification, or outreach.

## Phase 7 domain-intelligence operations

Phase 7 is a review contract only; the current Compose stack has no DNS worker, PSL service, DNS cache, or availability provider. Do not add a resolver or provider by pointing a job at an arbitrary URL. Any future implementation must record a pinned PSL version, preserve raw and registrable-domain forms, and report unresolved reasons for public suffixes, unknown/private suffixes, malformed/single-label values, localhost, and IP literals. IDN/punycode handling and subdomain treatment must be deterministic.

Operate DNS as bounded, observable evidence collection. Surface `not_checked`, `pending`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error`; never translate an empty/error result into a business conclusion. Keep MX, NS, and TXT records separate and record resolver/source, observed time, TTL, freshness/staleness, truncation/partial state, and uncertainty/error reason. A cached answer is not a fresh check: caches must be bounded, tenant-isolated, keyed by normalized query/type/class and resolver policy, honor TTL, and be invalidated/re-evaluated when the PSL or resolver policy changes.

Review domain-to-business candidates separately from DNS results. Verify tenant scope before comparison, reject public-suffix-only and malformed/IP candidates, and flag shared, parked, wildcard, homograph, sibling-subdomain, and conflicting-evidence cases. Association confidence must show evidence, reasons, algorithm/version, freshness, and uncertainty; it is suggestion-only. Require an authorized human accept/reject decision and audit it. Never auto-attach a domain or treat DNS as proof of ownership.

Availability is `unknown` unless the API reports a result from an authorized provider. Before enabling one, verify current product/legal/security approval, terms owner, allowed tenant scope, request/concurrency and timeout limits, retention/deletion class, health/circuit state, and explicit operational enablement. `nxdomain`, `no_data`, timeout, stale cache, or provider error is not “available.” On provider outage, rate-limit, terms/approval expiry, circuit-open, or conflicting result, fail closed and report unknown/deferred; never purchase, reserve, contact, or retry through an unapproved provider.

## Phase 8 website-scanning operations

Website scans are bounded evidence collection, not browser sessions. The API must accept only `http`/`https`, resolve and validate DNS/IP immediately before connection, block loopback/private/link-local/multicast/reserved/cloud-metadata destinations, and repeat those checks for every redirect. Review redirect chains for protocol or host-policy violations; DNS rebinding or an unsafe resolved address is a blocked scan, not a retry opportunity.

Monitor per-tenant and global scan counts, active concurrency, queue age, total/connect/read timeouts, response/decompressed bytes, redirect and crawl depth/link counts, cache hit/freshness, blocked destinations, DNS failures, partial results, and classification/error rates. Enforce hard budgets across redirects and discovered links. Classifications are conservative observations only: never submit forms, send cookies/credentials, execute JavaScript, or treat an HTTP 200/empty page as proof of a business fact. A scan that hits a limit is explicitly incomplete/unknown.

Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.

## Phase 9 official-site contact extraction operations

Operate contact extraction as passive observation of an approved/public official-site origin, limited to bounded same-site contact/about pages. Before enabling it, verify tenant scope, official-site approval, extractor/policy version, retention class, suppression source, and configured hard limits for pages/URLs, redirects, bytes, time, candidates, and concurrency. Do not add arbitrary URLs or search results to the scope.

Review each result with its source/page URL and context, extraction method, observed time, confidence/reasons, syntax status, role/free-mail labels, and independent MX/DNS status/freshness. Treat `not_checked`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error` as uncertainty—not invalidity or non-deliverability. Syntax, role, free-mail, and MX labels never authorize contact. Values found in scripts/styles/assets/file names, examples/placeholders, tracking addresses, malformed schemes, or third-party content are false positives and must be excluded or quarantined.

Verify suppression matching before persistence, response, cache, export, or review-queue insertion. Suppressed contacts remain **do not contact**, regardless of later confidence, classification, syntax, MX, pipeline, or verification changes. Monitor extraction attempts, pages/bytes/candidates, limit hits, blocked destinations, parse errors, false-positive exclusions, suppression matches, cache freshness, retention/deletion jobs, and provenance/audit failures. On a suppression or provenance failure, stop the affected write path and investigate; do not retry blindly.

There is no SMTP probing, SMTP `VRFY`/`EXPN`, validation email, outreach worker, campaign queue, or follow-up action. Never contact a discovered address. If extraction is disabled, unapproved, out of budget, or uncertain, report deferred/blocked/unknown with the reason. Retain only the minimum value and lineage for the approved retention period; redact addresses and page content from routine logs.

## Phase 10 scoring operations

Operate scoring as a versioned policy, not as a mutable numeric field. Before activating a rule set, verify its ID/version, owner/approval, weights, thresholds and priority bands, required evidence, freshness windows, suppression precedence, uncertainty behavior, rounding/tie-breaking, tenant scope, and rollback plan. Record the activation/configuration revision; never edit a rule set already used in production history.

Review score, priority band, and eligibility separately. A high-priority prospect can still be ineligible or unknown. Suppressed/do-not-contact records are hard blocked. Stale, expired, blocked, partial, missing, or uncertain required evidence must retain its state and reason and must not be silently treated as absent, negative, or current. Monitor counts by band and eligibility state, suppression matches, stale/uncertain outcomes, explanation failures, and unexpected score distribution changes.

Run recalculation only through an authenticated, tenant-scoped operation with an idempotency key or equivalent safe retry control. For each run record rule-set/algorithm versions, input snapshot or cutoff, actor/job, reason, start/end, processed/succeeded/failed counts, and partial status. Verify before/after score, band, eligibility, and explanation changes for representative records; read back the audit events. Do not report a request as complete merely because a job was accepted, and stop on tenant-scope, suppression, snapshot, or audit failures rather than retrying blindly.

For a rule or evidence-policy change, use a canary or bounded tenant batch, compare old/new explanations and eligibility, preserve the old version for reproducibility, and document rollback/recalculation scope. Ensure cached/list/detail projections do not mix rule versions. Retain and delete calculation inputs, explanations, and audit records under the approved data policy; do not put full contact values or sensitive evidence in routine logs.

The current Compose/MVP runtime remains pilot-only until durable rule-set storage/approval, scheduled recalculation with worker leases, complete audit/readback, and tenant-isolation and stale/uncertain regression checks are operationally verified.

## Phase 4 jobs and live logging

The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.

Creation must use a tenant-scoped idempotency key and request fingerprint. A repeated identical request returns the existing job/attempt; a conflicting payload is rejected. Cancellation is cooperative and race-safe, while retry is an explicit authorized new attempt linked to the original job and must not repeat completed side effects. Do not treat HTTP acceptance as completion, and do not reconstruct history from ephemeral container logs.

There is no SSE handler, durable queue, scan worker/isolation boundary, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.

## Phase 11 dashboard and review workflow operations

Operate saved filters as tenant-scoped configuration. Before exposing save/load/update/delete, verify the filter schema, server-side bounds, owner/share permissions, stable ordering, page-size maximum, retention/deletion behavior, and audit event format. A filter is not a permission grant; never troubleshoot a missing record by removing tenant or eligibility predicates.

The review queue must display the exact filter/predicate snapshot, whether each count is for the current page or the full matching set, and pagination/cursor state. Re-check tenant scope and current suppression/merge state when a row is opened or acted on. Suppressed/do-not-contact records remain blocked; merged/non-active records are not merge-eligible. A clickable count is safe only when it reuses the same bounded, tenant-scoped predicate; do not interpret stale, loading, or failed counts as zero.

For each bulk operation, verify the server-reported selection size and maximum, inspect the preview, obtain explicit operator confirmation, and ensure an idempotency key/correlation ID is recorded. The server must re-check permissions, tenant ownership, suppression, and merge eligibility at execution. Read back the per-record outcomes and audit event; investigate skipped/failed records and partial completion rather than retrying blindly. Never run an unbounded “all results” action from a count, and never describe bulk review as outreach or automatic deduplication.

Monitor saved-filter errors, queue count freshness, pagination/cursor failures, cross-tenant denials, suppression/eligibility skips, batch-limit violations, idempotency conflicts, partial bulk failures, audit append/readback failures, and merge snapshot/reversal outcomes. Preserve filter/selection snapshots or safe hashes and bounded totals in operational records, but redact secrets and unnecessary contact data. The current Compose/MVP runtime includes saved-filter creation/listing, a bounded review queue, clickable dashboard metadata, and explicit verify/reject/assign bulk review actions. It remains pilot-only: the remaining Phase 11 limitations are that update/delete saved-filter routes are not wired, queue counts lack complete matching-set/predicate snapshots, bulk operations have no preview/idempotency/per-record result contract, and the audit trail records a batch aggregate. Treat these as release blockers until the API/UI slices are hardened and verified.

## Phase 12 CRM operations

Operate CRM as human review and record-keeping, not outbound engagement. Before enabling the slice, verify the tenant/role matrix, canonical transition table (`new` → `contacted` → `qualified` → `proposal` → `negotiation` → `won`/`lost`, plus any explicitly configured paused/disqualified and reopen rules), append-only interaction policy, normalized outcome vocabulary, batch/report limits, suppression source, and retention class. Keep `AUTOMATED_OUTREACH_ENABLED=false` and verify there is no delivery provider, campaign queue, SMTP probe, or follow-up worker.

For pipeline changes, inspect the server response and audit event, including actor, before/after state, reason, timestamp, and correlation/idempotency ID. Reject direct jumps, edits to history, and actions on merged/inactive records. A same-state retry may be treated as idempotent; a reopen must be a new reasoned event. Adding an interaction does not advance a stage. For outcomes, use only `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, or `other`; preserve `other` as an explicit catch-all and treat `do_not_contact` as a separate immediate hard block.

Run reports with an explicit bounded date interval, `as_of`, timezone, filter snapshot, and requested metric semantics. Confirm whether totals are latest-state, event-time, distinct-business, page, or matching-set counts. Check freshness and partial/error status before distributing a report; never interpret a page count as a tenant total or a report as authorization. Report and export jobs must be tenant-scoped, idempotent where they have side effects, audited, and redacted.

Operate the suppression center as the final deny gate. Verify normalized email/domain/phone matching before CRM writes, responses, caches, exports, reports, and any queue. Investigate any record that is not visibly marked **Do not contact** after a match; stop the affected write/report path rather than retrying blindly. Unsuppression/removal requires an authorized reason and audit readback. Retain suppression provenance and history even when the underlying contact is deleted, subject to the approved legal/retention policy.

Monitor transition rejection and conflict rates, interaction/outcome write and correction failures, unknown outcomes, suppression matches and attempted bypasses, report freshness/partial failures, export denials, idempotency conflicts, cross-tenant denials, audit append/readback failures, and retention/deletion job results. Routine logs must contain no secrets, full contact values, or unnecessary free text. The current Compose/MVP remains pilot-only until durable CRM migrations, retention jobs, reproducible reports, integration tests, and recovery procedures are verified.

## Configuration and deployment

Copy `.env.example` for local development. Production values must be supplied by the deployment environment, never committed. `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` are optional API environment variables for first-run admin provisioning only; set them together through a secret store or protected deployment environment, remove them immediately after successful bootstrap, and rotate the password. Do not put real values in Compose files, CI variables visible to logs, images, or committed `.env` files.

For production, use Argon2id for password hashing and require MFA for administrator accounts. Configure TLS before enabling `Secure` session cookies. Local Compose uses HTTP, so browser testing of production `Secure` cookie behavior requires an HTTPS staging environment. Treat session cookies as bearer credentials: protect state-changing routes with CSRF controls, expire/revoke sessions, and never print cookie values in logs.

Before deployment:

1. Run `docker compose config` and review the rendered configuration; optional bootstrap values should be empty in CI and local validation.
2. Build from a reviewed commit and scan the resulting images.
3. Restrict host/network exposure at the ingress/firewall.
4. Verify both unauthenticated health checks and review logs for unexpected errors, cross-tenant errors, or sensitive data.
5. Exercise tenant-scoped list/detail/child routes with bounded pagination and filters, and verify that notes/pipeline changes appear in the intended tenant's audit trail only.
6. If Phase 9 is enabled, run official-site fixtures covering provenance/confidence, role and free-mail labels, syntax failures, every MX/DNS uncertainty state, false-positive exclusions, limit exhaustion, retention/deletion, and suppression-before-persistence/response/export. Confirm no SMTP or outreach network activity.
7. Record the image digest and configuration revision for rollback.

## Data, backups, and retention

The canonical API runtime under `apps/api` uses the named Docker volume `prospect-platform-api-data`.

Any older files under `infrastructure/docker/` are not referenced by Compose and are non-canonical; use `apps/api` and `apps/web` for all container changes. They can be removed during repository cleanup once no external tooling references them.

Inspect the volume with `docker volume inspect prospect-platform-api-data`; do not treat a local Docker volume as a backup.

For the current MVP there is no database migration or backup command. If runtime data is material, stop writes first and snapshot/copy the volume using an approved host backup process. Protect business, child intelligence, notes, provenance, and audit data with encryption and access controls, test a restore into an isolated environment, and document the result. Define retention/deletion rules that cover source references and notes as well as contact fields.

When jobs are introduced, backups and retention must include job definitions, idempotency records, terminal results/errors, and persisted sequence events. Verify that restoring a database preserves event ordering/cursors and does not cause a retried worker to repeat side effects. Define event redaction and retention separately from short-lived delivery connections.

Recommended starting policy for a future production data store:

- daily encrypted backups, with at least 30 days of retention;
- point-in-time recovery where supported;
- one offline or separately isolated copy;
- quarterly restore drills, plus a restore test after storage/provider changes;
- retention and deletion schedules aligned with the source/contact policy and applicable law.

Do not run `docker compose down -v` on a data-bearing environment: it removes the named volume.

## Failure handling

- **Unhealthy API:** inspect `docker compose logs api`, verify port binding and resource availability, then restart with `docker compose restart api` if appropriate.
- **Unhealthy web:** inspect `docker compose logs web`; confirm port `8080` is available and the image contains `/healthz`.
- **Missing or inconsistent detail:** preserve the request identifiers/log context, verify the authenticated tenant and parent-child association, and do not retry by guessing another tenant's ID.
- **Audit gap:** stop the affected mutation workflow, preserve the database/log evidence, and investigate before allowing operators to rely on the history.
- **Build failure:** run `docker compose build --no-cache` from a reviewed checkout and check Docker daemon/network status.
- **Unexpected outbound traffic:** stop the stack, preserve logs/metadata, and investigate. The MVP has no outreach worker and must not send automated messages.

## Production migration and scaling path

Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, domain observations, and scan history/cache, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker and scanner-isolation operations, bounded DNS/PSL/website processing, TTL/freshness-aware cache invalidation, SSRF/DNS-rebinding/redirect-chain tests, hard size/time/crawl budgets, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS/scanner status/error rates, cache freshness, blocked destinations, crawl-budget exhaustion, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and production scanners are possible future components—not implicit Compose dependencies. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.

## Phase 13 optional AI assistance operations

Keep AI disabled unless the provider registry, tenant scope, purpose, data class, redaction policy, retention class, rate/token/cost budgets, approval expiry, and operational enablement have been reviewed and recorded. The current Compose stack has no configured provider; do not enable one by adding an arbitrary URL or secret. Store credentials only in the deployment secret manager. A fallback must be pre-approved for the same purpose and input class and must inherit the primary provider's tenant, citation, redaction, retention, and authority constraints.

Before an AI request, verify the authenticated tenant and permission, active provider/policy version, suppression state, and bounded evidence selection. Minimize and redact inputs; exclude secrets, credentials, session data, raw page bodies, unrelated personal data, and unnecessary full contact values. Monitor request/response size, latency, provider health, fallback rate, quota/rate/cost usage, redaction failures, policy/approval denials, and `unknown`/`unavailable`/partial outcomes. Provider failure, stale or conflicting evidence, prompt-injection indicators, or hash/citation mismatch is a safe non-result—not a retry reason and never permission to present invented facts.

Review every suggestion as **AI draft — human review required**. Confirm citations resolve within the same tenant, the evidence hash still matches the cited snapshot, observed times/freshness are acceptable, suppression remains clear, and the provider/policy approval is current. An approval must be explicit, reasoned, audited, and read back; changed evidence or policy invalidates the proposal. Rejection, expiry, failed approval, and fallback events must remain auditable. Never report generation or approval request acceptance as persistence or completion.

The operational path must not alter pipeline/interactions/outcomes as if communication occurred, send outreach, probe SMTP, create campaigns, schedule follow-ups, merge records, acquire domains, or otherwise act autonomously in CRM. Suppressed/do-not-contact records remain visible for safety review and blocked from contact-related actions. Retain only the approved minimum AI lineage (redacted fingerprint, output, citations/hash, versions, approval and audit metadata); apply deletion/legal-hold rules to prompts, outputs, evidence snapshots, caches, and logs and verify deletion without removing required suppression/audit history.

On suspected provider misuse, data leakage, hallucinated/invented facts, prompt injection, cross-tenant exposure, unexpected outbound traffic, cost runaway, or suppression bypass: disable the AI capability/kill switch, stop affected jobs, preserve redacted evidence and audit metadata, revoke/rotate provider credentials, determine affected tenants and retention obligations, and require security/product/legal review before re-enabling. Production remains blocked until provider contracts/DPA, tenant-isolation, redaction, citation/hash, approval/rollback, evaluation, retention/deletion, monitoring, and recovery tests pass.

## Phase 14 draft-only outreach operations

Treat Phase 14 as preparation, never as delivery. Verify `AUTOMATED_OUTREACH_ENABLED=false` in the rendered Compose configuration and confirm there is no provider credential, send route, delivery worker, campaign scheduler, SMTP probe, validation-mail path, or follow-up job. A draft or approval request is not a send and must not be reported as completion.

Before any future enablement, record the provider allowlist entry, purpose/capability, tenant/data class, processing region/retention terms, timeout and payload bounds, per-tenant/global request/message/cost caps, health/circuit thresholds, approval owner/expiry, legal/consent policy version, and separate operational enablement. Load credentials only through the deployment secret manager; remove bootstrap values after provisioning and never print secrets in logs. A fallback is permitted only when separately approved for the same purpose and constraints.

For each draft, inspect tenant and recipient scope, normalized suppression/do-not-contact, consent or documented legal basis, jurisdiction/channel policy, evidence IDs/citations and exact snapshot hash, observed/freshness times, uncertainty/conflicts, provider/policy versions, and approval status/expiry. Suppression is the final deny gate. Require an authorized human to approve the unchanged draft version and evidence hash; edits, stale/changed evidence, expired approval, provider/policy changes, or uncertain gates require re-review. Read back the approval audit event. Never use a high score, pipeline state, public address, or AI citation as consent or permission to contact.

For any future side effect, require a tenant-scoped idempotency key and request fingerprint bound to draft/provider/policy versions. Verify exact replay returns the original result and conflicting reuse is rejected. Enforce caps before provider calls and across retries, fallbacks, workers, and tenants; use bounded backoff and circuit breaking. Monitor gate denials, suppression matches/bypass attempts, approval expiry, citation/hash mismatches, provider latency/errors/quota, cap consumption, retries, duplicate/conflicting keys, and audit append/readback failures. Record bounded per-item outcomes and never infer delivery from HTTP acceptance.

On suspected suppression bypass, invalid consent/legal basis, cross-tenant exposure, secret leakage, unexpected outbound traffic, provider misuse, cost runaway, citation/hash mismatch, or audit gap: disable the capability/kill switch, stop affected jobs, preserve redacted evidence and audit metadata, revoke/rotate credentials, identify affected tenants and legal-retention obligations, and require product/legal/security review before re-enabling. Production remains blocked until provider/DPA and legal review, durable approval/idempotency/audit, suppression synchronization, delivery feedback, retention/deletion/legal-hold verification, rollback, and no-send integration tests pass. The current Compose/MVP has none of these live outreach components.

## Incident checklist

1. Record time, affected service, image/config revision, and observed health state.
2. Preserve relevant logs and audit records without exporting secrets or unnecessary contact data.
3. Stop or isolate the affected service if data loss, unauthorized access, SSRF, provenance tampering, or unexpected outreach is suspected.
4. Rotate exposed credentials through the secret manager.
5. Validate recovery with health checks and a targeted tenant-isolation/detail smoke test.
6. Document root cause, corrective action, and any retention/suppression or audit impact.

## Phase 15 portable operations

See `docs/DEPLOYMENT.md` for the production-safe environment contract, `/api/v1/health/ready` readiness semantics, atomic SQLite backup/restore procedures, checksum/retention policy, non-destructive rollback guidance, monitoring references, and the systemd/Virtualmin-compatible service example. Run `scripts/healthcheck.sh` for an operator-safe readiness probe. Backups are host-side and must remain encrypted/off-host; never include `.env` or secret-manager material.

## Phase 15 production deployment runbook

### Prerequisites and Virtualmin layout

Use a dedicated, patched Linux VPS with Docker Engine and Compose v2, adequate disk/RAM/CPU, host firewalling, DNS control, HTTPS certificates with renewal monitoring, and an encrypted off-host backup destination. Virtualmin may host the domain and terminate TLS/reverse-proxy to Compose, but it does not replace Docker health checks, application authorization, backups, or monitoring. Keep the checkout and `.env` outside public web roots with restrictive permissions; expose only the reverse proxy publicly and keep the API binding private where the topology permits.

### Configuration and first bootstrap

```sh
cp .env.example .env
chmod 600 .env
docker compose config --quiet
docker compose up --build -d
docker compose ps
curl -fsS https://example.invalid/healthz
curl -fsS http://127.0.0.1:8000/api/v1/health/live
```

Replace the example hostname with the real HTTPS origin. Supply secrets through the protected deployment environment/secret manager, not shell history or committed files. Set `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` only for a fresh instance, confirm authenticated login, then remove both values and rotate the password. Verify the rendered config still contains `AUTOMATED_OUTREACH_ENABLED=false`; never override it as a routine deployment action.

### Readiness, monitoring, and release evidence

The API `/api/v1/health/live` is a liveness check and `/api/v1/health/ready` checks SQLite readiness; web `/healthz` is a liveness check. These endpoints do not prove backups, workers, migrations, or external dependencies, so readiness remains an operator gate: both services must report Compose `healthy`, HTTPS must reach the expected containers, authenticated tenant-scoped smoke tests must pass, and migration validation must be recorded. Monitor container health/restarts, CPU/RAM/disk and `/data` pressure, API latency/error rates, auth failures, backup age/failures, TLS expiry, and unexpected egress. Never log secrets, cookies, full contact values, or request bodies. Record the commit, image digests, rendered non-secret configuration fingerprint, schema/migration result, backup ID, and approver.

### Backup, restore, retention, and migration validation

The named volume `prospect-platform-api-data` is live state, not a backup. Before a release or schema change, quiesce writes, take an encrypted backup to an off-host/isolated destination, verify its checksum/manifest, and restore it into a disposable isolated volume. Run `PRAGMA integrity_check`, foreign-key checks, representative tenant-scoped API reads, row-count checks, and the API test suite against the restored copy. Record the result and retain the previous image/config. Apply documented retention to SQLite data, audit/source lineage, operational logs, and backup generations; honor legal holds and verify deletion jobs where present. Do not use `docker compose down -v` on a data-bearing environment.

There is currently no standalone migration or backup CLI. `schema.sql` is applied by the API startup and additive compatibility behavior is in application code; therefore every schema change requires a reviewed backup-first procedure and isolated restore test. Do not assume startup success means migration success. Stop and roll back the release if integrity, tenant isolation, health, or smoke validation fails.

### Rollback

1. Stop promotion and record symptoms, health, commit/image/config revisions, and backup ID.
2. Disable the affected Virtualmin route or put the site in maintenance mode; stop writes if data integrity is in doubt.
3. Re-deploy the previously verified image pair and exact configuration. Do not run an older binary against a schema it cannot read.
4. Re-run health, authenticated tenant-isolation smoke tests, and read-only integrity checks.
5. Restore the database only when the backup/schema compatibility is verified and an incident owner approves it; otherwise preserve the newer data and perform forward repair.
6. Re-enable traffic only after monitoring is green, then document root cause, retention/legal impact, and follow-up migration work.

### Explicit limitations

This repository does not provision Virtualmin/TLS/DNS, provide a dependency-aware readiness service beyond the API's SQLite check, durable migration runner, PITR, HA database, durable queue/worker leases, production egress proxy, or compliance-grade retention service. SQLite and the in-process worker are pilot-only. The release has no outbound provider/send path and must remain outbound-disabled by default. See `docs/RELEASE_CHECKLIST.md` for the short go/no-go gate.

## Phase 16 pilot benchmark operations

Run the Phase 16 benchmark only against the versioned, labeled synthetic fixture manifest described in `docs/BENCHMARKS.md`. Confirm that fixtures contain no live personal data, secrets, uncontrolled network responses, or production records. Keep expected labels separate from generated outputs, and record the manifest/hash, label-policy version, commit, clean/dirty state, algorithm and rule-set versions, runtime/dependency/image digests, non-secret configuration fingerprint, seed, locale/timezone, rounding/tie-breaking, host profile, command, and timestamps.

For each deterministic classifier/scorer, run every fixture at least twice in fresh processes. Compare class, score, band, eligibility, uncertainty reasons, explanation factors, and version metadata; any unexplained difference is a reproducibility failure. Report confusion matrices, support, precision and recall with zero-denominator `N/A`, plus unknown/blocked/partial/error and suppression outcomes separately. Do not tune thresholds, remove hard fixtures, or convert `N/A` into a pass after seeing results without publishing a new methodology version.

Measure latency after a documented warm-up, using monotonic timestamps and a fixed local, network-free workload unless separate approval exists. Separate cold, warm, bypass, and not-applicable cache states. Record every sample, including errors/timeouts, and report count, minimum, median, p95, p99, maximum, timeout/error rate, cache hits/misses, freshness/expiry, and invalidation behavior. State whether startup, serialization, database, and network time are included. Never present a cache hit as a fresh observation or infer production capacity/SLOs from pilot timings.

Before sign-off, evaluate the synthetic-only gates: 100% deterministic output equality; precision `>=0.90` and recall `>=0.85` only when positive support is at least 20; 100% safety preservation for suppressed, blocked, malformed, and explicitly unknown fixtures; 100% cross-tenant denial/non-disclosure probes; and complete latency sample/cache labels. Conduct blinded, seeded human review stratified by predicted class, uncertainty/suppression state, and FP/FN disagreement, reviewing at least 20 fixtures or 10% (whichever is larger, subject to the declared cap). Use pseudonymous reviewer IDs, reason codes, independent review for disagreements, and documented adjudication. A passed pilot gate is not production evidence.

Store the report and raw machine-readable samples with the pilot artifacts, including failed/incomplete runs and limitations. Do not place sensitive fixture values in routine logs. Phase 16 does not enable a worker, live source, or outreach path.

### Unresolved publication and deployment prerequisites

Do not report local benchmark or Compose success as deployment. Remote push remains blocked until repository authentication and intended remote/branch permission are supplied. Production deployment remains blocked until the Phase 15 operator prerequisites are verified: reviewed remote revision, Docker/Compose host access, protected deployment directory, secret injection, DNS/TLS/Virtualmin, encrypted off-host backup and restore evidence, monitoring, rollback owner, and operational approval. Record the blocker in the release record and keep the current outbound-disabled configuration.

## Phase 17 final acceptance and capacity smoke

The Phase 17 decision is **local acceptance PASS; production deployment BLOCKED and not attempted**. Evidence is collected at local commit `9622f769776637a40fdae797adccba91445cd351`; see `docs/FINAL_ACCEPTANCE.md` for the matrix and release record. The final collector passed 100 API tests, the no-latency Phase 16 benchmark, Python compilation, shell syntax, JSON validation, Compose config, Git state, and safety invariants.

The capacity smoke is intentionally deterministic and in-memory: 1,000 synthetic items with page size 100 returned 100 and `has_more=true`; a 5,000-item synthetic batch retained 100 and marked truncation. It verifies bounds only. Do not use it to size production hosts, infer throughput/concurrency, set SLOs, or claim durability/availability. A production capacity exercise requires an approved representative workload, host profile, observability, and durable database/worker design.

For reproducibility, retain the commit, fixture/manifest hash, algorithm and rule-set versions, runtime/dependency/image versions, non-secret configuration fingerprint, seed, locale/timezone, rounding/tie-breaking, command, host profile, timestamps, and raw results. Repeat deterministic fixtures in fresh processes and compare serialized output fields exactly. Keep latency/cache results separate from deterministic acceptance and label cache hits as non-fresh observations.

## Phase 17 release blockers and rollback decision

Remote publication remains blocked until repository authentication and intended remote/branch permission are available; no push was attempted. Production remains blocked until a reviewed remote revision, host and Docker/Compose access, protected deployment directory, secret manager, DNS/TLS/Virtualmin perimeter, encrypted off-host backup and restore drill, monitoring, rollback owner, and operational approval are verified. Real source, AI, DNS/availability, scanner, or outreach providers additionally require allowlisting, terms/DPA and legal review, data/retention policy, consent or lawful-basis approval where applicable, rate/cost/circuit controls, and explicit operational enablement. Keep outreach disabled.

**Rollback decision:** retain the local revision; there is no production deployment to undo. If a future promotion fails health, integrity, migration, tenant-isolation, or smoke checks, stop promotion/traffic and writes as needed, restore the previously verified compatible image/config first, and restore data only after compatibility and incident-owner approval. Re-run health, integrity, and authenticated isolation checks before reopening traffic. Never use `docker compose down -v` on a data-bearing environment.

These are separate states: a passing local acceptance run is not a remote push, and a remote push is not a production deployment.