Portable deployment and recovery runbook (Phase 15)

Configuration

Copy .env.example to an untracked deployment environment file. Production requires a secret-manager supplied SESSION_SECRET of at least 32 characters and refuses AUTOMATED_OUTREACH_ENABLED=true. Keep bootstrap credentials one-time only; remove and rotate them after provisioning. Never place secrets in images, Compose YAML, logs, backups, or public web roots.

Validate before startup:

python3 -c 'from app.config import load_config; load_config()'
docker compose config --quiet

Health and readiness

GET /api/v1/health/live is process liveness and unauthenticated.
GET /api/v1/health/ready checks SQLite connectivity and returns HTTP 503 until ready.
scripts/healthcheck.sh checks readiness and the outreach safety flag.

Use readiness for load balancers and container health checks; liveness is only for process supervision.

SQLite backup and restore

Backups are host-side artifacts and never include .env or secret files. backup_sqlite.sh uses SQLite's online backup API for a consistent snapshot, writes with mode 0600 to a temporary file, atomically renames it, writes a SHA-256 sidecar, and retains only the newest configured count.

scripts/backup_sqlite.sh /var/lib/prospect-platform/prospects.db /var/backups/prospect-platform 30

Before restoring, stop application writes, verify the checksum sidecar, and use the explicit confirmation flag. The script first makes a pre-restore backup, then atomically replaces the target only after an integrity check:

scripts/restore_sqlite.sh /var/backups/prospect-platform/prospects-<timestamp>.db /var/lib/prospect-platform/prospects.db --confirm-restore

Validate backup directory permissions and keep copies encrypted/off-host according to the retention policy. Test restores in an isolated directory quarterly. Never use docker compose down -v on a data-bearing installation.

Rollback

scripts/rollback.sh is deliberately non-destructive: it prints the approved image/tag or digest rollback procedure and executes no stop, delete, restore, or deployment action. Record old/new image digests, configuration revision, backup/checksum, and health/readiness evidence.

systemd / Virtualmin

systemd/prospect-api.service.example is a least-privilege service example. Copy it to a reviewed systemd unit, create /etc/prospect-platform/prospect.env with mode 0600, use a dedicated user/data directory, and place TLS/reverse proxying in the Virtualmin-managed web tier. Do not put environment files under public_html.

Monitoring and incident response

Monitor readiness failures, restart count, HTTP 5xx rate, SQLite backup age/checksum failures, disk usage, and unexpected outbound traffic. Alert when the latest backup is older than the agreed RPO or when a restore drill fails. Routine logs must not contain passwords, tokens, cookies, API keys, full contact values, or free-text notes.

On incident: record image/config revision and health state; preserve redacted logs and audit evidence; isolate the service for data loss, unauthorized access, or unexpected outbound traffic; rotate secrets through the secret manager; validate restore/readiness and tenant-scoped reads; then document root cause and retention impact. Outreach remains disabled throughout.

3.3 KiB Raw Blame History