add production readiness and recovery assets

docs/DEPLOYMENT.md

@@ -0,0 +1,50 @@
+# Portable deployment and recovery runbook (Phase 15)
+
+## Configuration
+
+Copy `.env.example` to an untracked deployment environment file. Production requires a secret-manager supplied `SESSION_SECRET` of at least 32 characters and refuses `AUTOMATED_OUTREACH_ENABLED=true`. Keep bootstrap credentials one-time only; remove and rotate them after provisioning. Never place secrets in images, Compose YAML, logs, backups, or public web roots.
+
+Validate before startup:
+
+```sh
+python3 -c 'from app.config import load_config; load_config()'
+docker compose config --quiet
+```
+
+## Health and readiness
+
+- `GET /api/v1/health/live` is process liveness and unauthenticated.
+- `GET /api/v1/health/ready` checks SQLite connectivity and returns HTTP 503 until ready.
+- `scripts/healthcheck.sh` checks readiness and the outreach safety flag.
+
+Use readiness for load balancers and container health checks; liveness is only for process supervision.
+
+## SQLite backup and restore
+
+Backups are host-side artifacts and never include `.env` or secret files. `backup_sqlite.sh` uses SQLite's online backup API for a consistent snapshot, writes with mode 0600 to a temporary file, atomically renames it, writes a SHA-256 sidecar, and retains only the newest configured count.
+
+```sh
+scripts/backup_sqlite.sh /var/lib/prospect-platform/prospects.db /var/backups/prospect-platform 30
+```
+
+Before restoring, stop application writes, verify the checksum sidecar, and use the explicit confirmation flag. The script first makes a pre-restore backup, then atomically replaces the target only after an integrity check:
+
+```sh
+scripts/restore_sqlite.sh /var/backups/prospect-platform/prospects-<timestamp>.db /var/lib/prospect-platform/prospects.db --confirm-restore
+```
+
+Validate backup directory permissions and keep copies encrypted/off-host according to the retention policy. Test restores in an isolated directory quarterly. Never use `docker compose down -v` on a data-bearing installation.
+
+## Rollback
+
+`scripts/rollback.sh` is deliberately non-destructive: it prints the approved image/tag or digest rollback procedure and executes no stop, delete, restore, or deployment action. Record old/new image digests, configuration revision, backup/checksum, and health/readiness evidence.
+
+## systemd / Virtualmin
+
+`systemd/prospect-api.service.example` is a least-privilege service example. Copy it to a reviewed systemd unit, create `/etc/prospect-platform/prospect.env` with mode 0600, use a dedicated user/data directory, and place TLS/reverse proxying in the Virtualmin-managed web tier. Do not put environment files under `public_html`.
+
+## Monitoring and incident response
+
+Monitor readiness failures, restart count, HTTP 5xx rate, SQLite backup age/checksum failures, disk usage, and unexpected outbound traffic. Alert when the latest backup is older than the agreed RPO or when a restore drill fails. Routine logs must not contain passwords, tokens, cookies, API keys, full contact values, or free-text notes.
+
+On incident: record image/config revision and health state; preserve redacted logs and audit evidence; isolate the service for data loss, unauthorized access, or unexpected outbound traffic; rotate secrets through the secret manager; validate restore/readiness and tenant-scoped reads; then document root cause and retention impact. Outreach remains disabled throughout.

docs/OPERATIONS.md

@@ -204,3 +204,50 @@ On suspected suppression bypass, invalid consent/legal basis, cross-tenant expos
 4. Rotate exposed credentials through the secret manager.
 5. Validate recovery with health checks and a targeted tenant-isolation/detail smoke test.
 6. Document root cause, corrective action, and any retention/suppression or audit impact.
+
+## Phase 15 portable operations
+
+See `docs/DEPLOYMENT.md` for the production-safe environment contract, `/api/v1/health/ready` readiness semantics, atomic SQLite backup/restore procedures, checksum/retention policy, non-destructive rollback guidance, monitoring references, and the systemd/Virtualmin-compatible service example. Run `scripts/healthcheck.sh` for an operator-safe readiness probe. Backups are host-side and must remain encrypted/off-host; never include `.env` or secret-manager material.
+
+## Phase 15 production deployment runbook
+
+### Prerequisites and Virtualmin layout
+
+Use a dedicated, patched Linux VPS with Docker Engine and Compose v2, adequate disk/RAM/CPU, host firewalling, DNS control, HTTPS certificates with renewal monitoring, and an encrypted off-host backup destination. Virtualmin may host the domain and terminate TLS/reverse-proxy to Compose, but it does not replace Docker health checks, application authorization, backups, or monitoring. Keep the checkout and `.env` outside public web roots with restrictive permissions; expose only the reverse proxy publicly and keep the API binding private where the topology permits.
+
+### Configuration and first bootstrap
+
+```sh
+cp .env.example .env
+chmod 600 .env
+docker compose config --quiet
+docker compose up --build -d
+docker compose ps
+curl -fsS https://example.invalid/healthz
+curl -fsS http://127.0.0.1:8000/api/v1/health/live
+```
+
+Replace the example hostname with the real HTTPS origin. Supply secrets through the protected deployment environment/secret manager, not shell history or committed files. Set `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` only for a fresh instance, confirm authenticated login, then remove both values and rotate the password. Verify the rendered config still contains `AUTOMATED_OUTREACH_ENABLED=false`; never override it as a routine deployment action.
+
+### Readiness, monitoring, and release evidence
+
+The API `/api/v1/health/live` is a liveness check and `/api/v1/health/ready` checks SQLite readiness; web `/healthz` is a liveness check. These endpoints do not prove backups, workers, migrations, or external dependencies, so readiness remains an operator gate: both services must report Compose `healthy`, HTTPS must reach the expected containers, authenticated tenant-scoped smoke tests must pass, and migration validation must be recorded. Monitor container health/restarts, CPU/RAM/disk and `/data` pressure, API latency/error rates, auth failures, backup age/failures, TLS expiry, and unexpected egress. Never log secrets, cookies, full contact values, or request bodies. Record the commit, image digests, rendered non-secret configuration fingerprint, schema/migration result, backup ID, and approver.
+
+### Backup, restore, retention, and migration validation
+
+The named volume `prospect-platform-api-data` is live state, not a backup. Before a release or schema change, quiesce writes, take an encrypted backup to an off-host/isolated destination, verify its checksum/manifest, and restore it into a disposable isolated volume. Run `PRAGMA integrity_check`, foreign-key checks, representative tenant-scoped API reads, row-count checks, and the API test suite against the restored copy. Record the result and retain the previous image/config. Apply documented retention to SQLite data, audit/source lineage, operational logs, and backup generations; honor legal holds and verify deletion jobs where present. Do not use `docker compose down -v` on a data-bearing environment.
+
+There is currently no standalone migration or backup CLI. `schema.sql` is applied by the API startup and additive compatibility behavior is in application code; therefore every schema change requires a reviewed backup-first procedure and isolated restore test. Do not assume startup success means migration success. Stop and roll back the release if integrity, tenant isolation, health, or smoke validation fails.
+
+### Rollback
+
+1. Stop promotion and record symptoms, health, commit/image/config revisions, and backup ID.
+2. Disable the affected Virtualmin route or put the site in maintenance mode; stop writes if data integrity is in doubt.
+3. Re-deploy the previously verified image pair and exact configuration. Do not run an older binary against a schema it cannot read.
+4. Re-run health, authenticated tenant-isolation smoke tests, and read-only integrity checks.
+5. Restore the database only when the backup/schema compatibility is verified and an incident owner approves it; otherwise preserve the newer data and perform forward repair.
+6. Re-enable traffic only after monitoring is green, then document root cause, retention/legal impact, and follow-up migration work.
+
+### Explicit limitations
+
+This repository does not provision Virtualmin/TLS/DNS, provide a dependency-aware readiness service beyond the API's SQLite check, durable migration runner, PITR, HA database, durable queue/worker leases, production egress proxy, or compliance-grade retention service. SQLite and the in-process worker are pilot-only. The release has no outbound provider/send path and must remain outbound-disabled by default. See `docs/RELEASE_CHECKLIST.md` for the short go/no-go gate.

docs/RELEASE_CHECKLIST.md

@@ -0,0 +1,32 @@
+# Phase 15 release checklist
+
+Use this checklist for a Virtualmin-managed VPS deployment. A checked item is evidence, not an assumption; record the owner, timestamp, commit, image digests, and backup ID in the release record.
+
+## Go/no-go prerequisites
+
+- [ ] Reviewed commit and release owner approved; previous image pair and configuration revision are retained for rollback.
+- [ ] Patched Linux host has Docker Engine and Compose v2, sufficient CPU/RAM/disk, restricted Docker access, host firewalling, and a protected deployment directory outside public web roots.
+- [ ] Virtualmin/DNS points to the host; HTTPS certificate, renewal monitoring, reverse proxy, and maintenance route are tested.
+- [ ] API exposure is restricted to the required private path/host; only the intended web entry point is public.
+- [ ] Production environment values are injected from a protected secret store/environment. `.env` is untracked, mode `0600`, and contains no committed or logged secrets.
+- [ ] Bootstrap admin values, if needed, are supplied together, used once, removed immediately, and the password is rotated. `AUTOMATED_OUTREACH_ENABLED=false` is verified in rendered Compose.
+- [ ] Encrypted off-host backup destination, retention schedule, legal-hold owner, monitoring destination, and incident/rollback owner are confirmed.
+
+## Validate, deploy, and verify
+
+- [ ] `docker compose config --quiet` passes; rendered configuration was reviewed without exposing secret values.
+- [ ] Images build from the reviewed commit, are scanned, and their digests are recorded.
+- [ ] Backup is taken before release/schema change; checksum/manifest is verified.
+- [ ] Backup restores into an isolated volume/environment; `PRAGMA integrity_check`, foreign-key checks, representative row counts, tenant-scoped reads, and API tests pass. Schema/migration result is recorded.
+- [ ] `docker compose up -d` completes and both services report `healthy`; running is not accepted as ready.
+- [ ] `curl -fsS https://<host>/healthz` and the API liveness/readiness endpoints pass (`/api/v1/health/live`, `/api/v1/health/ready`). These checks do not replace authenticated smoke tests.
+- [ ] Authenticated smoke tests cover login, tenant-scoped list/detail/child access, a safe mutation/audit readback, and cross-tenant non-disclosure.
+- [ ] Monitoring sees health/restarts, API errors/latency, disk and `/data` pressure, auth failures, backup age/failure, TLS expiry, migration failures, and unexpected egress without collecting secrets or full contact data.
+- [ ] No outbound provider/send/SMTP activity is present; unexpected egress is treated as an incident.
+
+## Retain and sign off
+
+- [ ] Release record contains commit, image digests, non-secret config fingerprint, schema/migration result, backup ID, test output, approver, and rollback decision.
+- [ ] Data, audit/source lineage, logs, and backup retention/deletion rules are applied; legal holds are preserved.
+- [ ] Rollback path was reviewed: restore the prior compatible image/config first, stop writes if needed, and restore data only after compatibility approval. Do not use `docker compose down -v` on a data-bearing host.
+- [ ] Known limitations are accepted explicitly: no Virtualmin/TLS/DNS provisioning, only SQLite-level readiness, no standalone migration/backup CLI, SQLite/in-process worker only, no PITR/HA, and no production egress isolation.

docs/SECURITY.md

@@ -162,3 +162,15 @@ Pin or review base-image and dependency updates, scan images before release, use
 - Production requires provider/DPA/legal review, tenant-isolation and citation/hash tests, redaction and prompt-injection/hallucination evaluations, human-review and rollback semantics, immutable/tamper-evident audit, cost/rate monitoring, incident kill switch, retention/deletion jobs, and durable worker/retry idempotency. The current Compose stack has no configured AI provider and is not production-ready for AI processing.
 
 Report vulnerabilities privately to the repository maintainers; do not include live credentials or personal data in an issue.
+
+## Phase 15 deployment security controls
+
+- **Portable Virtualmin boundary:** Virtualmin is an operator-managed perimeter, not an application dependency. Require Docker Engine/Compose v2 on a patched Linux host, a reviewed checkout with restricted ownership/permissions, host firewalling, DNS under the operator's control, HTTPS with renewal monitoring, and reverse-proxy rules that do not expose the database volume or arbitrary container ports. Keep the API private where possible and allow only the intended web/API paths.
+- **Secrets:** inject production values from a secret manager or protected deployment environment. `.env.example` is documentation only; never commit `.env`, passwords, tokens, provider credentials, certificates, or backup keys. Use bootstrap variables only once, remove them after provisioning, rotate the resulting credential, and prevent secrets from appearing in Compose output, process listings where feasible, logs, traces, metrics, browser storage, or error responses.
+- **Health versus readiness:** `/api/v1/health/live` and `/healthz` intentionally require no session and reveal only process health; `/api/v1/health/ready` additionally checks SQLite readiness. These endpoints are not proof of backup, worker, migration, or external-dependency readiness. Gate ingress on Compose health plus authenticated smoke tests and migration checks; do not expose tenant data through health responses.
+- **Data protection and retention:** encrypt backups in transit and at rest, restrict volume and backup access, use an off-host/isolated copy, and define retention separately for prospect/contact data, audit/source lineage, logs, and backups. Apply deletion and legal holds intentionally; preserve suppression/audit evidence when required. A Docker volume or host snapshot alone is not a verified backup.
+- **Migration and rollback:** the current image has no standalone migration runner. Back up and restore-test before schema changes, validate row counts/foreign keys/indexes/tenant predicates and representative API reads on an isolated copy, and record the schema/data validation result. Pin image digests and configuration, retain the previous release, and ensure rollback does not run a newer schema against an incompatible older binary. Restore data only through an approved, compatibility-checked procedure.
+- **Outbound deny-by-default:** `AUTOMATED_OUTREACH_ENABLED=false` is fixed in Compose and the current release has no send/provider/delivery path. Egress from the host/proxy should be restricted to documented needs; unexpected outbound traffic, SMTP, provider calls, or a newly introduced route is a security incident. Do not enable future outbound behavior without separate product/legal/security review, allowlisting, caps, suppression re-checks, audit, and a tested kill switch.
+- **Monitoring and incident evidence:** alert on unhealthy containers, restart loops, disk/volume pressure, backup age/failure, restore-test failure, TLS expiry, authentication/authorization failures, migration errors, unexpected egress, and log redaction failures. Monitoring must not collect secrets or full contact payloads. Preserve redacted logs, audit records, image/config digests, and affected-tenant scope during incidents.
+
+These controls describe deployment prerequisites and gates; they do not make SQLite, password fallback, HTTP local Compose, in-process workers, or the public liveness checks production-grade. Remaining gaps must be accepted explicitly or closed before production.