From 6b41d5b9ee032136e8d376fc5bb32acb7fbd6ca4 Mon Sep 17 00:00:00 2001 From: Marco0300 Date: Thu, 3 Sep 2026 12:38:27 +0200 Subject: [PATCH] add production readiness and recovery assets --- .env.example | 17 ++++--- README.md | 12 ++++- apps/api/README.md | 13 ++++- apps/api/app/config.py | 43 ++++++++++++++++ apps/api/app/main.py | 11 +++- apps/api/tests/test_phase15_ops.py | 73 +++++++++++++++++++++++++++ apps/web/Dockerfile | 4 ++ apps/web/README.md | 29 +++++++++-- apps/web/app.js | 3 +- apps/web/asset-manifest.json | 15 ++++++ apps/web/config.js | 5 ++ apps/web/error.html | 2 + apps/web/health.html | 2 + apps/web/index.html | 5 +- apps/web/scripts/smoke-deployment.mjs | 34 +++++++++++++ docker-compose.yml | 7 ++- docs/DEPLOYMENT.md | 50 ++++++++++++++++++ docs/OPERATIONS.md | 47 +++++++++++++++++ docs/RELEASE_CHECKLIST.md | 32 ++++++++++++ docs/SECURITY.md | 12 +++++ scripts/backup_sqlite.sh | 40 +++++++++++++++ scripts/healthcheck.sh | 18 +++++++ scripts/restore_sqlite.sh | 43 ++++++++++++++++ scripts/rollback.sh | 14 +++++ systemd/prospect-api.service.example | 27 ++++++++++ 25 files changed, 540 insertions(+), 18 deletions(-) create mode 100644 apps/api/app/config.py create mode 100644 apps/api/tests/test_phase15_ops.py create mode 100644 apps/web/asset-manifest.json create mode 100644 apps/web/config.js create mode 100644 apps/web/error.html create mode 100644 apps/web/health.html create mode 100644 apps/web/scripts/smoke-deployment.mjs create mode 100644 docs/DEPLOYMENT.md create mode 100644 docs/RELEASE_CHECKLIST.md create mode 100755 scripts/backup_sqlite.sh create mode 100755 scripts/healthcheck.sh create mode 100755 scripts/restore_sqlite.sh create mode 100755 scripts/rollback.sh create mode 100644 systemd/prospect-api.service.example diff --git a/.env.example b/.env.example index c833ae4..0005541 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,17 @@ -# Local, non-secret runtime configuration. Copy to .env; never commit real credentials. -APP_ENV=development +# Portable deployment example. Keep this file secret-free; supply SESSION_SECRET via a secret manager. +APP_ENV=production LOG_LEVEL=INFO API_PORT=8000 WEB_PORT=8080 -CORS_ORIGINS=http://localhost:8080 -# Optional first-run admin bootstrap. Set both only for a fresh deployment; -# remove them after the admin account has been created. Never commit credentials. +CORS_ORIGINS=https://your-approved-web-origin.example +DATA_DIR=/data +# Required in production; generate at least 32 random characters outside this file. +SESSION_SECRET= +# Optional first-run admin bootstrap. Remove both immediately after provisioning. BOOTSTRAP_ADMIN_EMAIL= BOOTSTRAP_ADMIN_PASSWORD= -# Deliberately fixed to false in compose for this MVP. +# Hard safety default; this release has no delivery capability. AUTOMATED_OUTREACH_ENABLED=false +# Backup operations (host-side, never mounted into the web container). +BACKUP_DIR=/var/backups/prospect-platform +BACKUP_RETENTION=30 diff --git a/README.md b/README.md index 9cea2a8..97b0fe5 100644 --- a/README.md +++ b/README.md @@ -217,4 +217,14 @@ Any future side-effecting operation must require a tenant-scoped idempotency key Phase 14 is preparation only and remains pilot-grade. The current Compose stack has no outreach provider, draft/send API, consent ledger, legal-policy engine, durable approval workflow, production secret manager, durable queue/worker, delivery telemetry, bounce/complaint handling, or compliance-grade audit/retention service. Before production, obtain jurisdiction-specific legal review and documented consent/legal-basis policy, implement provider contracts/DPA and secret isolation, durable transactional idempotency and audit, suppression synchronization, approval expiry/rollback, rate/cost controls, delivery feedback, incident kill switch, retention/deletion/legal-hold workflows, and end-to-end tenant-isolation and no-send tests. -See `apps/api/README.md`, `apps/web/README.md`, `docs/SECURITY.md`, and `docs/OPERATIONS.md` for details. +See `docs/DEPLOYMENT.md` for Phase 15 portable backup/restore, readiness, rollback, monitoring, and systemd/Virtualmin operations. No deployment is performed by this repository change. + +## Phase 15 production-readiness boundary + +Phase 15 documents a portable deployment pattern for a Virtualmin-managed VPS; it does not claim that this repository provisions Virtualmin, TLS, DNS, backups, or a production database. The supported baseline is Docker Engine plus the Compose v2 plugin on a Linux host, with Virtualmin (or another reverse proxy) terminating HTTPS and forwarding only to the published web/API ports. The operator owns firewalling, DNS, certificates, Docker access, host patching, resource capacity, and an off-host backup destination. + +Before deployment, verify the prerequisites in `docs/RELEASE_CHECKLIST.md`: a reviewed commit, Docker/Compose, DNS and TLS, a protected deployment directory, secret injection, backup destination, monitoring, and a tested rollback owner. Copy `.env.example` to an untracked `.env` only for non-secret defaults. Production secrets and bootstrap credentials must come from a secret manager or protected Virtualmin deployment environment; remove bootstrap values after first-run provisioning and never commit or print them. + +The API liveness endpoint (`/api/v1/health/live`) and web `/healthz` are public process checks; API `/api/v1/health/ready` additionally verifies SQLite readiness. They are suitable for Docker and monitoring but do not prove backups, tenant authorization, or external dependencies. Production ingress must not route traffic until both Compose services report `healthy` and the deployment smoke tests pass. The current image initializes SQLite from `schema.sql` and has no standalone migration runner. Treat schema changes as a versioned, backup-first migration: validate on a restored copy, record the schema/data checks, and keep the previous image/config available for rollback. + +The named Docker volume is not a backup. Stop or quiesce writes, create an encrypted off-host backup, verify it, and perform a restore drill before calling a deployment protected. Define retention for the database, audit/source lineage, logs, and backups; apply legal holds and deletion rules deliberately. Do not run `docker compose down -v` on a data-bearing host. Outbound traffic is deny-by-default for product behavior: `AUTOMATED_OUTREACH_ENABLED=false` is fixed in Compose and this release has no send/provider/worker path. Unexpected egress is an incident. SQLite, the in-process worker, HTTP-only local Compose, lack of durable migrations/PITR, and the limited SQLite-only readiness check are explicit limitations, not hidden guarantees. diff --git a/apps/api/README.md b/apps/api/README.md index f612d84..7bc5af9 100644 --- a/apps/api/README.md +++ b/apps/api/README.md @@ -57,6 +57,7 @@ Implementations should expose source/query/job state without leaking raw payload ### Health and workspace - `GET /api/v1/health/live` — unauthenticated liveness check. +- `GET /api/v1/health/ready` — unauthenticated readiness check; verifies SQLite connectivity and returns 503 when unavailable. - `GET /api/v1/auth/me` — current authenticated user and tenant. - `GET /api/v1/dashboard/summary` — tenant-scoped counts and score summary. @@ -188,7 +189,17 @@ Every draft must retain recipient/channel, bounded redacted content or a safe co Any future send or other side effect must require an `Idempotency-Key` scoped to tenant, operation, draft version, recipient, provider, and policy fingerprint. Exact retries return the original result; a different request under the same key is rejected. Enforce rate/message/cost caps before attempts and across retries, fallbacks, and workers; use bounded retry/backoff and circuit breaking. Audit draft/gate/approval/provider/cap/suppression events with safe per-item outcomes and redacted payloads. No route may infer completion from request acceptance. -The Phase 14 implementation remains documentation-only/pilot preparation: there is no draft persistence/API, consent ledger, legal-policy evaluator, configured provider, durable approval queue, delivery adapter, bounce/complaint feedback, secret manager, or production-grade audit/retention workflow in the current runtime. Production requires those components plus DPA/provider and jurisdictional legal review, suppression synchronization, kill switch, rollback/revocation, deletion/legal-hold verification, and integration tests proving no-send default, citation/hash binding, stale/uncertain gate failure, cap enforcement, idempotent replay/conflict rejection, approval expiry, and cross-tenant isolation. +The Phase 13 implementation remains documentation-only/pilot preparation: there is no draft persistence/API, consent ledger, legal-policy evaluator, configured provider, durable approval queue, delivery adapter, bounce/complaint feedback, secret manager, or production-grade audit/retention workflow in the current runtime. Production requires those components plus DPA/provider and jurisdictional legal review, suppression synchronization, kill switch, rollback/revocation, deletion/legal-hold verification, and integration tests proving no-send default, citation/hash binding, stale/uncertain gate failure, cap enforcement, idempotent replay/conflict rejection, approval expiry, and cross-tenant isolation. + +## Phase 15 deployment contract + +The API image is portable but intentionally small: it runs as a non-root user, writes only `/data`, and initializes the SQLite schema from the image-bundled `schema.sql`. It does not provision Virtualmin, DNS, TLS, a reverse proxy, a secret manager, a durable queue, or a production database. A Virtualmin deployment must provide Docker Engine with Compose v2, a protected checkout, host firewalling, HTTPS termination, a private deployment network, sufficient CPU/RAM/disk, and an encrypted off-host backup destination. Publish the API only where the reverse proxy requires it; preferably expose the web entry point publicly and keep the API port private to the host/network. + +Configuration is environment-only. `APP_ENV`, `LOG_LEVEL`, `CORS_ORIGINS`, and port values may be non-secret deployment settings. `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` are one-time provisioning inputs, must be supplied together through a protected environment/secret store, and must be removed and rotated immediately after bootstrap. Never accept provider credentials from clients or store secrets in source, images, Compose, committed `.env`, logs, traces, metrics, or audit payloads. `AUTOMATED_OUTREACH_ENABLED=false` is the enforced Compose default; no outbound provider or send route exists in this release. + +`GET /api/v1/health/live` is an unauthenticated liveness check and `GET /api/v1/health/ready` is an unauthenticated SQLite readiness check. Operators must gate traffic on Compose `healthy` status plus an authenticated smoke test and migration validation; do not interpret either response as proof that backups, tenant authorization, workers, or external dependencies are ready. + +The current runtime has no standalone migration, backup, restore, retention, or rollback command. Before changing an image or schema, quiesce writes, take and verify an encrypted backup, restore it into an isolated copy, run the API test suite and schema/data invariants, then deploy. Keep the previous image digest and configuration revision available; rollback must restore the application image/config first and only restore data when compatibility and operator approval are established. SQLite remains pilot-scale: no HA, PITR, durable worker lease/recovery, or online migration is implied. See `docs/OPERATIONS.md` and `docs/RELEASE_CHECKLIST.md` for the operator procedure. ## Remaining limitations and production migration work diff --git a/apps/api/app/config.py b/apps/api/app/config.py new file mode 100644 index 0000000..f3f7db7 --- /dev/null +++ b/apps/api/app/config.py @@ -0,0 +1,43 @@ +"""Environment configuration validation for portable deployments.""" +from dataclasses import dataclass +import os +from pathlib import Path + + +class ConfigError(ValueError): + """Raised when deployment configuration is unsafe or malformed.""" + + +@dataclass(frozen=True) +class Config: + app_env: str + data_dir: Path + session_secret: str + outreach_enabled: bool + log_level: str + + +def _env(values, key, default=""): + return str(values.get(key, default) or "").strip() + + +def load_config(values=None): + values = os.environ if values is None else values + app_env = _env(values, "APP_ENV", "development").lower() + if app_env not in {"development", "test", "staging", "production"}: + raise ConfigError("APP_ENV must be development, test, staging, or production") + data_dir = Path(_env(values, "DATA_DIR", ".") or ".").expanduser() + if not data_dir.is_absolute(): + data_dir = (Path.cwd() / data_dir).resolve() + if data_dir.exists() and not data_dir.is_dir(): + raise ConfigError("DATA_DIR must be a directory") + secret = _env(values, "SESSION_SECRET") + if app_env == "production" and (len(secret) < 32 or secret.lower() in {"change-me", "development", "dev"}): + raise ConfigError("SESSION_SECRET must be at least 32 characters in production") + outreach = _env(values, "AUTOMATED_OUTREACH_ENABLED", "false").lower() + if outreach not in {"", "0", "false", "no", "off"}: + raise ConfigError("automated outreach is disabled in this release") + log_level = _env(values, "LOG_LEVEL", "INFO").upper() + if log_level not in {"QUIET", "ERROR", "WARNING", "INFO", "DEBUG"}: + raise ConfigError("LOG_LEVEL is invalid") + return Config(app_env, data_dir, secret, False, log_level) diff --git a/apps/api/app/main.py b/apps/api/app/main.py index b8c5b3d..f7fb7a1 100644 --- a/apps/api/app/main.py +++ b/apps/api/app/main.py @@ -14,6 +14,7 @@ if __package__ in (None, ""): from app.contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS from app.scoring import DEFAULT_RULES, signals_for_business, evaluate_score, SCORE_VERSION from app.ai_assistance import generate as generate_ai, MAX_INPUT_ITEMS, MAX_OUTPUT_CHARS + from app.config import load_config else: from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses from .sources import adapter_for, contains_secret @@ -22,6 +23,7 @@ else: from .contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS from .scoring import DEFAULT_RULES, signals_for_business, evaluate_score, SCORE_VERSION from .ai_assistance import generate as generate_ai, MAX_INPUT_ITEMS, MAX_OUTPUT_CHARS +from .config import load_config ORGANIZATION_ID = "demo-tenant" SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql" SESSION_DAYS = 7 @@ -521,7 +523,13 @@ class ApiHandler(BaseHTTPRequestHandler): def do_GET(self): parsed=urlparse(self.path); path=parsed.path.rstrip("/") - if path=="/api/v1/health/live": return self.send_json(200,{"status":"ok","organization_id":ORGANIZATION_ID}) + if path=="/api/v1/health/live": return self.send_json(200,{"status":"ok","organization_id":ORGANIZATION_ID,"outreach_enabled":False}) + if path=="/api/v1/health/ready": + try: + check=connect(self.server.db_path); check.execute("SELECT 1"); check.close() + return self.send_json(200,{"status":"ok","ready":True,"database":"ok","outreach_enabled":False}) + except (sqlite3.Error, OSError): + return self.send_json(503,{"status":"not_ready","ready":False,"database":"error","outreach_enabled":False}) db=self.db() try: user=self.require_auth(db) @@ -1284,6 +1292,7 @@ def _job_worker(server): finally: db.close() def create_server(host="127.0.0.1",port=8000,db_path="prospects.db"): + load_config() server=ThreadingHTTPServer((host,port),ApiHandler);server.db_path=db_path;connect(db_path).close();server.job_stop=threading.Event();server.job_wakeup=threading.Event();server.job_thread=threading.Thread(target=_job_worker,args=(server,),daemon=True);server.job_thread.start() original_close=server.server_close def close(): diff --git a/apps/api/tests/test_phase15_ops.py b/apps/api/tests/test_phase15_ops.py new file mode 100644 index 0000000..e20d423 --- /dev/null +++ b/apps/api/tests/test_phase15_ops.py @@ -0,0 +1,73 @@ +import hashlib +import json +import os +import sqlite3 +import subprocess +import sys +import threading +import unittest +from http.client import HTTPConnection +from pathlib import Path +from tempfile import TemporaryDirectory + +from app.main import create_server +from app.config import ConfigError, load_config + +ROOT = Path(__file__).resolve().parents[3] +SCRIPTS = ROOT / "scripts" + + +class Phase15OpsTests(unittest.TestCase): + def test_config_defaults_are_safe_and_production_requires_explicit_secret(self): + with TemporaryDirectory() as tmp: + cfg = load_config({"DATA_DIR": tmp}) + self.assertFalse(cfg.outreach_enabled) + self.assertEqual(cfg.app_env, "development") + with self.assertRaises(ConfigError): + load_config({"APP_ENV": "production", "DATA_DIR": tmp}) + cfg = load_config({"APP_ENV": "production", "DATA_DIR": tmp, "SESSION_SECRET": "a" * 32}) + self.assertEqual(cfg.session_secret, "a" * 32) + + def test_health_and_readiness_report_database_state(self): + with TemporaryDirectory() as tmp: + db_path = str(Path(tmp) / "state.db") + server = create_server("127.0.0.1", 0, db_path) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + conn = HTTPConnection("127.0.0.1", server.server_port, timeout=3) + for path in ("/api/v1/health/live", "/api/v1/health/ready"): + conn.request("GET", path) + response = conn.getresponse() + payload = json.loads(response.read()) + self.assertEqual(response.status, 200) + self.assertEqual(payload["status"], "ok") + self.assertFalse(payload["outreach_enabled"]) + finally: + server.shutdown(); server.server_close(); thread.join(timeout=2) + + def test_backup_restore_integrity_and_checksum(self): + with TemporaryDirectory() as tmp: + root = Path(tmp); db = root / "prospects.db"; backups = root / "backups" + conn = sqlite3.connect(db); conn.execute("CREATE TABLE facts (value TEXT)"); conn.execute("INSERT INTO facts VALUES ('before')"); conn.commit(); conn.close() + result = subprocess.run([str(SCRIPTS / "backup_sqlite.sh"), str(db), str(backups), "2"], text=True, capture_output=True, check=True) + backup = Path(result.stdout.strip().splitlines()[-1]) + self.assertTrue(backup.exists()); self.assertTrue(Path(str(backup) + ".sha256").exists()) + self.assertEqual(hashlib.sha256(backup.read_bytes()).hexdigest(), Path(str(backup) + ".sha256").read_text().split()[0]) + conn = sqlite3.connect(db); conn.execute("UPDATE facts SET value='after'"); conn.commit(); conn.close() + subprocess.run([str(SCRIPTS / "restore_sqlite.sh"), str(backup), str(db), "--confirm-restore"], text=True, capture_output=True, check=True) + conn = sqlite3.connect(db); self.assertEqual(conn.execute("SELECT value FROM facts").fetchone()[0], "before"); conn.close() + self.assertTrue(any(p.name.startswith("pre-restore-") for p in backups.glob("*.db"))) + + def test_backup_output_does_not_contain_secret_values(self): + with TemporaryDirectory() as tmp: + root = Path(tmp); db = root / "prospects.db"; backups = root / "backups" + conn = sqlite3.connect(db); conn.execute("CREATE TABLE logs (message TEXT)"); conn.execute("INSERT INTO logs VALUES ('password=should-not-be-logged')"); conn.commit(); conn.close() + result = subprocess.run([str(SCRIPTS / "backup_sqlite.sh"), str(db), str(backups), "1"], text=True, capture_output=True, check=True) + combined = result.stdout + result.stderr + self.assertNotIn("should-not-be-logged", combined) + self.assertNotIn("password=", combined) + + +if __name__ == "__main__": + unittest.main() diff --git a/apps/web/Dockerfile b/apps/web/Dockerfile index 8e6cce0..9b719ff 100644 --- a/apps/web/Dockerfile +++ b/apps/web/Dockerfile @@ -3,8 +3,12 @@ RUN addgroup -S app && adduser -S -G app app WORKDIR /srv COPY index.html /srv/index.html COPY styles.css /srv/styles.css +COPY config.js /srv/config.js +COPY asset-manifest.json /srv/asset-manifest.json COPY app.js /srv/app.js COPY healthz /srv/healthz +COPY health.html /srv/health.html +COPY error.html /srv/error.html RUN chown -R app:app /srv USER app EXPOSE 8080 diff --git a/apps/web/README.md b/apps/web/README.md index acf08cd..126470d 100644 --- a/apps/web/README.md +++ b/apps/web/README.md @@ -4,14 +4,25 @@ Self-contained static frontend for the Prospect Platform API. There is no bundle ## Configure and run -The API base is configurable before `app.js` runs: +The public runtime configuration is loaded from `config.js` before `app.js`. It contains no credentials and may safely be replaced during deployment: -```html - - +```js +window.__PROSPECT_CONFIG__ = Object.freeze({ apiBase: 'https://api.example.invalid', assetVersion: 'phase-15' }); ``` -If not set, the UI uses `localStorage.prospect_api_base` when present; otherwise it targets the same origin. The dashboard requires an authenticated API session and shows the login screen until `/api/v1/auth/me` succeeds. +If `apiBase` is empty, the UI uses `window.API_BASE`, then `localStorage.prospect_api_base` when present, and otherwise targets the same origin. The dashboard requires an authenticated API session and shows the login screen until `/api/v1/auth/me` succeeds. Do not put tokens, passwords, or private keys in `config.js`. + +`asset-manifest.json` records the public entrypoints, cache-busting version, and SHA-256 digests for release verification. The HTML references the static assets with the `phase-15` version query string; update those references and regenerate the manifest when changing the release version. + +## Deployment readiness checks + +Serve this directory from the intended static-server root, then run: + +```sh +node scripts/smoke-deployment.mjs http://127.0.0.1:8080 +``` + +The smoke script checks HTTP delivery for the manifest-listed assets, health/error pages, `healthz`, expected markers, and common hardcoded-secret patterns. It validates static readiness only; it does not deploy the app or prove API/production availability. ## Phase 3 UI contract @@ -133,3 +144,11 @@ Every suggestion must display its evidence citations, tenant-scoped evidence IDs AI output must be visibly labeled **AI suggestion — human review required** and remain read-only until an authorized human explicitly approves it. Approval must show the proposed change, citations/hash, freshness, tenant scope, and safe reason; rejection and expiry must be available. The UI must require re-review when the evidence hash or policy version changes and must display partial/failed approval rather than implying persistence. Approval does not authorize contact or verification. No Phase 13 control may send email/SMS, probe SMTP, create a campaign, schedule follow-up, alter pipeline/interactions/outcomes as if communication occurred, merge records, acquire a domain, or perform autonomous CRM/outreach actions. The browser must not hide or export suppressed data as eligible, and exports/reports must retain safe AI provenance and redaction labels where applicable. Production remains limited until browser/API tests cover citations and hash mismatch, redaction, fallback boundaries, approval/rejection, stale/conflicting evidence, suppression precedence, tenant non-disclosure, and no-autonomy controls; the current Compose stack has no configured AI provider. + +## Phase 15 deployment and readiness + +The web image is a portable static server: it runs as a non-root user, serves only the files copied into `/srv`, and exposes `/healthz`. Virtualmin is responsible for DNS, HTTPS certificates, reverse-proxy routing, firewall rules, and any access control around the site. Set the API base deliberately for the deployed origin; do not put credentials or provider secrets in HTML, JavaScript, local storage, image layers, or `.env` files. `CORS_ORIGINS` must exactly match the approved HTTPS origin rather than a broad wildcard. + +`/healthz` is an unauthenticated process/liveness check. API `/api/v1/health/ready` checks SQLite readiness, but neither endpoint proves tenant authorization, backup validity, or external dependencies. Route traffic only after the web and API containers report `healthy`, the HTTPS proxy reaches the intended containers, and an authenticated browser/API smoke test succeeds. The browser must never be used to test or initiate outbound prospect/provider traffic; `AUTOMATED_OUTREACH_ENABLED=false` remains visible as the no-send default. + +For releases, validate the exact static image and API image together, capture image digests and configuration revision, and retain the prior pair for rollback. If a schema/data migration is involved, the API owner must complete backup/restore and migration validation before the web image is promoted. The current client has no service-worker cache or migration logic; stale browser tabs must be refreshed after a release, and Virtualmin/CDN caching must not serve an old API contract indefinitely. SQLite, HTTP-only local Compose, lack of a readiness endpoint, and lack of a production asset/CDN pipeline are explicit limitations. diff --git a/apps/web/app.js b/apps/web/app.js index 23f92e9..92497c0 100644 --- a/apps/web/app.js +++ b/apps/web/app.js @@ -1,7 +1,8 @@ /* ProspectOS frontend MVP. Configure before loading with window.API_BASE = 'http://127.0.0.1:8000'; */ (() => { 'use strict'; - const API_BASE = (window.API_BASE || localStorage.getItem('prospect_api_base') || '').replace(/\/$/, ''); + const publicConfig = window.__PROSPECT_CONFIG__ || {}; + const API_BASE = (publicConfig.apiBase || window.API_BASE || localStorage.getItem('prospect_api_base') || '').replace(/\/$/, ''); const endpoint = (path) => `${API_BASE}${path}`; let prospects = [], selectedId = null, selectedDetail = null, currentUser = null; let page = 1, pageSize = 10, hasNextPage = false, savedFilters = [], reviewQueue = [], selectedReviewIds = new Set(); diff --git a/apps/web/asset-manifest.json b/apps/web/asset-manifest.json new file mode 100644 index 0000000..07def99 --- /dev/null +++ b/apps/web/asset-manifest.json @@ -0,0 +1,15 @@ +{ + "schema": 1, + "version": "phase-15", + "entrypoints": ["config.js", "app.js", "styles.css"], + "publicAssets": ["index.html", "health.html", "error.html", "healthz"], + "integrity": { + "config.js": "sha256-20f3020432436dcccdbfc86fd56a6a6a49b71fc512a1e434187a5ddc134fda1c", + "app.js": "sha256-fc12f49012bb1329ffdd7bdcf655e9ab9097eaf1e0cc73cd0442b19fd57a0374", + "styles.css": "sha256-7340dccc648fa917fb497ceeba7286d9c4712b6552960de54cef759cdbda6de4", + "index.html": "sha256-37a9d5e2a81941c3c9f9bd3f42edf33a40565bf197cf675389d250dd60eb69db", + "health.html": "sha256-c352a6f37aa24628cfc8d5709a70ff2d94181d192ed5fe00916cca9378a61d81", + "error.html": "sha256-f3cc28d2dfc9e8af112d0257c6ee47b9e9146e5da8590a754dd589bf43702aaf", + "healthz": "sha256-dc51b8c96c2d745df3bd5590d990230a482fd247123599548e0632fdbf97fc22" + } +} diff --git a/apps/web/config.js b/apps/web/config.js new file mode 100644 index 0000000..a83ef1d --- /dev/null +++ b/apps/web/config.js @@ -0,0 +1,5 @@ +/* Public, non-secret runtime configuration. Replace this file at deploy time if needed. */ +window.__PROSPECT_CONFIG__ = Object.freeze({ + apiBase: '', + assetVersion: 'phase-15' +}); diff --git a/apps/web/error.html b/apps/web/error.html new file mode 100644 index 0000000..524b726 --- /dev/null +++ b/apps/web/error.html @@ -0,0 +1,2 @@ + +ProspectOS · Temporarily unavailable

ProspectOS frontend

Something went wrong

This page could not be loaded. Please try again or return to the workspace.

diff --git a/apps/web/health.html b/apps/web/health.html new file mode 100644 index 0000000..fd1cc76 --- /dev/null +++ b/apps/web/health.html @@ -0,0 +1,2 @@ + +ProspectOS · Healthy

ProspectOS frontend

Ready

Static asset delivery is available. API availability is checked separately by the application.

health.html

diff --git a/apps/web/index.html b/apps/web/index.html index 5cb7d1c..2d24091 100644 --- a/apps/web/index.html +++ b/apps/web/index.html @@ -5,7 +5,7 @@ ProspectOS · Pipeline intelligence - +
@@ -117,6 +117,7 @@ - + + diff --git a/apps/web/scripts/smoke-deployment.mjs b/apps/web/scripts/smoke-deployment.mjs new file mode 100644 index 0000000..d5ff793 --- /dev/null +++ b/apps/web/scripts/smoke-deployment.mjs @@ -0,0 +1,34 @@ +#!/usr/bin/env node +import { createHash } from 'node:crypto'; +import { readFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; + +const base = (process.argv[2] || 'http://127.0.0.1:8080').replace(/\/$/, ''); +const manifestPath = resolve(process.cwd(), 'asset-manifest.json'); +const manifest = JSON.parse(await readFile(manifestPath, 'utf8')); +const assets = [...new Set([...(manifest.entrypoints || []), ...(manifest.publicAssets || [])])]; +const failures = []; +const secretPattern = /(api[_-]?key|secret|token|password|private[_-]?key|authorization)\s*[:=]\s*["'][^"']+/i; + +async function check(path, expected = {}) { + const response = await fetch(`${base}/${path}`); + const body = await response.text(); + if (!response.ok) failures.push(`${path}: HTTP ${response.status}`); + if (expected.contentType && !response.headers.get('content-type')?.includes(expected.contentType)) failures.push(`${path}: unexpected content type`); + if (expected.marker && !body.includes(expected.marker)) failures.push(`${path}: missing marker ${expected.marker}`); + if (expected.integrity) { + const digest = `sha256-${createHash('sha256').update(body).digest('hex')}`; + if (digest !== expected.integrity) failures.push(`${path}: integrity mismatch`); + } + if (secretPattern.test(body)) failures.push(`${path}: possible hardcoded secret`); + console.log(`${response.ok ? 'PASS' : 'FAIL'} ${path} (${response.status})`); +} + +const expectations = { + 'index.html': { contentType: 'text/html', marker: 'ProspectOS' }, + 'health.html': { contentType: 'text/html', marker: 'Ready' }, + 'error.html': { contentType: 'text/html', marker: 'Something went wrong' }, + healthz: { marker: 'ok' } +}; +for (const asset of assets) await check(asset, { ...expectations[asset], integrity: manifest.integrity?.[asset] }); +if (failures.length) { console.error(`\n${failures.length} deployment smoke check(s) failed`); for (const failure of failures) console.error(`- ${failure}`); process.exitCode = 1; } else console.log(`\nDeployment smoke checks passed for ${base}`); diff --git a/docker-compose.yml b/docker-compose.yml index 34be2e5..10a650c 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -10,6 +10,7 @@ services: API_PORT: 8000 LOG_LEVEL: ${LOG_LEVEL:-INFO} CORS_ORIGINS: ${CORS_ORIGINS:-http://localhost:8080} + SESSION_SECRET: ${SESSION_SECRET:-} DATA_DIR: /data # Optional first-run admin bootstrap; leave unset after provisioning. BOOTSTRAP_ADMIN_EMAIL: ${BOOTSTRAP_ADMIN_EMAIL:-} @@ -20,6 +21,8 @@ services: volumes: - prospect_api_data:/data read_only: true + init: true + pids_limit: 128 tmpfs: - /tmp security_opt: @@ -27,7 +30,7 @@ services: cap_drop: - ALL healthcheck: - test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/v1/health/live', timeout=2)"] + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/v1/health/ready', timeout=2)"] interval: 10s timeout: 3s retries: 5 @@ -44,6 +47,8 @@ services: ports: - "${WEB_PORT:-8080}:8080" read_only: true + init: true + pids_limit: 128 tmpfs: - /tmp security_opt: diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md new file mode 100644 index 0000000..32ed9ab --- /dev/null +++ b/docs/DEPLOYMENT.md @@ -0,0 +1,50 @@ +# Portable deployment and recovery runbook (Phase 15) + +## Configuration + +Copy `.env.example` to an untracked deployment environment file. Production requires a secret-manager supplied `SESSION_SECRET` of at least 32 characters and refuses `AUTOMATED_OUTREACH_ENABLED=true`. Keep bootstrap credentials one-time only; remove and rotate them after provisioning. Never place secrets in images, Compose YAML, logs, backups, or public web roots. + +Validate before startup: + +```sh +python3 -c 'from app.config import load_config; load_config()' +docker compose config --quiet +``` + +## Health and readiness + +- `GET /api/v1/health/live` is process liveness and unauthenticated. +- `GET /api/v1/health/ready` checks SQLite connectivity and returns HTTP 503 until ready. +- `scripts/healthcheck.sh` checks readiness and the outreach safety flag. + +Use readiness for load balancers and container health checks; liveness is only for process supervision. + +## SQLite backup and restore + +Backups are host-side artifacts and never include `.env` or secret files. `backup_sqlite.sh` uses SQLite's online backup API for a consistent snapshot, writes with mode 0600 to a temporary file, atomically renames it, writes a SHA-256 sidecar, and retains only the newest configured count. + +```sh +scripts/backup_sqlite.sh /var/lib/prospect-platform/prospects.db /var/backups/prospect-platform 30 +``` + +Before restoring, stop application writes, verify the checksum sidecar, and use the explicit confirmation flag. The script first makes a pre-restore backup, then atomically replaces the target only after an integrity check: + +```sh +scripts/restore_sqlite.sh /var/backups/prospect-platform/prospects-.db /var/lib/prospect-platform/prospects.db --confirm-restore +``` + +Validate backup directory permissions and keep copies encrypted/off-host according to the retention policy. Test restores in an isolated directory quarterly. Never use `docker compose down -v` on a data-bearing installation. + +## Rollback + +`scripts/rollback.sh` is deliberately non-destructive: it prints the approved image/tag or digest rollback procedure and executes no stop, delete, restore, or deployment action. Record old/new image digests, configuration revision, backup/checksum, and health/readiness evidence. + +## systemd / Virtualmin + +`systemd/prospect-api.service.example` is a least-privilege service example. Copy it to a reviewed systemd unit, create `/etc/prospect-platform/prospect.env` with mode 0600, use a dedicated user/data directory, and place TLS/reverse proxying in the Virtualmin-managed web tier. Do not put environment files under `public_html`. + +## Monitoring and incident response + +Monitor readiness failures, restart count, HTTP 5xx rate, SQLite backup age/checksum failures, disk usage, and unexpected outbound traffic. Alert when the latest backup is older than the agreed RPO or when a restore drill fails. Routine logs must not contain passwords, tokens, cookies, API keys, full contact values, or free-text notes. + +On incident: record image/config revision and health state; preserve redacted logs and audit evidence; isolate the service for data loss, unauthorized access, or unexpected outbound traffic; rotate secrets through the secret manager; validate restore/readiness and tenant-scoped reads; then document root cause and retention impact. Outreach remains disabled throughout. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 9c6ac57..9287a6d 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -204,3 +204,50 @@ On suspected suppression bypass, invalid consent/legal basis, cross-tenant expos 4. Rotate exposed credentials through the secret manager. 5. Validate recovery with health checks and a targeted tenant-isolation/detail smoke test. 6. Document root cause, corrective action, and any retention/suppression or audit impact. + +## Phase 15 portable operations + +See `docs/DEPLOYMENT.md` for the production-safe environment contract, `/api/v1/health/ready` readiness semantics, atomic SQLite backup/restore procedures, checksum/retention policy, non-destructive rollback guidance, monitoring references, and the systemd/Virtualmin-compatible service example. Run `scripts/healthcheck.sh` for an operator-safe readiness probe. Backups are host-side and must remain encrypted/off-host; never include `.env` or secret-manager material. + +## Phase 15 production deployment runbook + +### Prerequisites and Virtualmin layout + +Use a dedicated, patched Linux VPS with Docker Engine and Compose v2, adequate disk/RAM/CPU, host firewalling, DNS control, HTTPS certificates with renewal monitoring, and an encrypted off-host backup destination. Virtualmin may host the domain and terminate TLS/reverse-proxy to Compose, but it does not replace Docker health checks, application authorization, backups, or monitoring. Keep the checkout and `.env` outside public web roots with restrictive permissions; expose only the reverse proxy publicly and keep the API binding private where the topology permits. + +### Configuration and first bootstrap + +```sh +cp .env.example .env +chmod 600 .env +docker compose config --quiet +docker compose up --build -d +docker compose ps +curl -fsS https://example.invalid/healthz +curl -fsS http://127.0.0.1:8000/api/v1/health/live +``` + +Replace the example hostname with the real HTTPS origin. Supply secrets through the protected deployment environment/secret manager, not shell history or committed files. Set `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` only for a fresh instance, confirm authenticated login, then remove both values and rotate the password. Verify the rendered config still contains `AUTOMATED_OUTREACH_ENABLED=false`; never override it as a routine deployment action. + +### Readiness, monitoring, and release evidence + +The API `/api/v1/health/live` is a liveness check and `/api/v1/health/ready` checks SQLite readiness; web `/healthz` is a liveness check. These endpoints do not prove backups, workers, migrations, or external dependencies, so readiness remains an operator gate: both services must report Compose `healthy`, HTTPS must reach the expected containers, authenticated tenant-scoped smoke tests must pass, and migration validation must be recorded. Monitor container health/restarts, CPU/RAM/disk and `/data` pressure, API latency/error rates, auth failures, backup age/failures, TLS expiry, and unexpected egress. Never log secrets, cookies, full contact values, or request bodies. Record the commit, image digests, rendered non-secret configuration fingerprint, schema/migration result, backup ID, and approver. + +### Backup, restore, retention, and migration validation + +The named volume `prospect-platform-api-data` is live state, not a backup. Before a release or schema change, quiesce writes, take an encrypted backup to an off-host/isolated destination, verify its checksum/manifest, and restore it into a disposable isolated volume. Run `PRAGMA integrity_check`, foreign-key checks, representative tenant-scoped API reads, row-count checks, and the API test suite against the restored copy. Record the result and retain the previous image/config. Apply documented retention to SQLite data, audit/source lineage, operational logs, and backup generations; honor legal holds and verify deletion jobs where present. Do not use `docker compose down -v` on a data-bearing environment. + +There is currently no standalone migration or backup CLI. `schema.sql` is applied by the API startup and additive compatibility behavior is in application code; therefore every schema change requires a reviewed backup-first procedure and isolated restore test. Do not assume startup success means migration success. Stop and roll back the release if integrity, tenant isolation, health, or smoke validation fails. + +### Rollback + +1. Stop promotion and record symptoms, health, commit/image/config revisions, and backup ID. +2. Disable the affected Virtualmin route or put the site in maintenance mode; stop writes if data integrity is in doubt. +3. Re-deploy the previously verified image pair and exact configuration. Do not run an older binary against a schema it cannot read. +4. Re-run health, authenticated tenant-isolation smoke tests, and read-only integrity checks. +5. Restore the database only when the backup/schema compatibility is verified and an incident owner approves it; otherwise preserve the newer data and perform forward repair. +6. Re-enable traffic only after monitoring is green, then document root cause, retention/legal impact, and follow-up migration work. + +### Explicit limitations + +This repository does not provision Virtualmin/TLS/DNS, provide a dependency-aware readiness service beyond the API's SQLite check, durable migration runner, PITR, HA database, durable queue/worker leases, production egress proxy, or compliance-grade retention service. SQLite and the in-process worker are pilot-only. The release has no outbound provider/send path and must remain outbound-disabled by default. See `docs/RELEASE_CHECKLIST.md` for the short go/no-go gate. diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md new file mode 100644 index 0000000..2274903 --- /dev/null +++ b/docs/RELEASE_CHECKLIST.md @@ -0,0 +1,32 @@ +# Phase 15 release checklist + +Use this checklist for a Virtualmin-managed VPS deployment. A checked item is evidence, not an assumption; record the owner, timestamp, commit, image digests, and backup ID in the release record. + +## Go/no-go prerequisites + +- [ ] Reviewed commit and release owner approved; previous image pair and configuration revision are retained for rollback. +- [ ] Patched Linux host has Docker Engine and Compose v2, sufficient CPU/RAM/disk, restricted Docker access, host firewalling, and a protected deployment directory outside public web roots. +- [ ] Virtualmin/DNS points to the host; HTTPS certificate, renewal monitoring, reverse proxy, and maintenance route are tested. +- [ ] API exposure is restricted to the required private path/host; only the intended web entry point is public. +- [ ] Production environment values are injected from a protected secret store/environment. `.env` is untracked, mode `0600`, and contains no committed or logged secrets. +- [ ] Bootstrap admin values, if needed, are supplied together, used once, removed immediately, and the password is rotated. `AUTOMATED_OUTREACH_ENABLED=false` is verified in rendered Compose. +- [ ] Encrypted off-host backup destination, retention schedule, legal-hold owner, monitoring destination, and incident/rollback owner are confirmed. + +## Validate, deploy, and verify + +- [ ] `docker compose config --quiet` passes; rendered configuration was reviewed without exposing secret values. +- [ ] Images build from the reviewed commit, are scanned, and their digests are recorded. +- [ ] Backup is taken before release/schema change; checksum/manifest is verified. +- [ ] Backup restores into an isolated volume/environment; `PRAGMA integrity_check`, foreign-key checks, representative row counts, tenant-scoped reads, and API tests pass. Schema/migration result is recorded. +- [ ] `docker compose up -d` completes and both services report `healthy`; running is not accepted as ready. +- [ ] `curl -fsS https:///healthz` and the API liveness/readiness endpoints pass (`/api/v1/health/live`, `/api/v1/health/ready`). These checks do not replace authenticated smoke tests. +- [ ] Authenticated smoke tests cover login, tenant-scoped list/detail/child access, a safe mutation/audit readback, and cross-tenant non-disclosure. +- [ ] Monitoring sees health/restarts, API errors/latency, disk and `/data` pressure, auth failures, backup age/failure, TLS expiry, migration failures, and unexpected egress without collecting secrets or full contact data. +- [ ] No outbound provider/send/SMTP activity is present; unexpected egress is treated as an incident. + +## Retain and sign off + +- [ ] Release record contains commit, image digests, non-secret config fingerprint, schema/migration result, backup ID, test output, approver, and rollback decision. +- [ ] Data, audit/source lineage, logs, and backup retention/deletion rules are applied; legal holds are preserved. +- [ ] Rollback path was reviewed: restore the prior compatible image/config first, stop writes if needed, and restore data only after compatibility approval. Do not use `docker compose down -v` on a data-bearing host. +- [ ] Known limitations are accepted explicitly: no Virtualmin/TLS/DNS provisioning, only SQLite-level readiness, no standalone migration/backup CLI, SQLite/in-process worker only, no PITR/HA, and no production egress isolation. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 603dd4f..7d03754 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -162,3 +162,15 @@ Pin or review base-image and dependency updates, scan images before release, use - Production requires provider/DPA/legal review, tenant-isolation and citation/hash tests, redaction and prompt-injection/hallucination evaluations, human-review and rollback semantics, immutable/tamper-evident audit, cost/rate monitoring, incident kill switch, retention/deletion jobs, and durable worker/retry idempotency. The current Compose stack has no configured AI provider and is not production-ready for AI processing. Report vulnerabilities privately to the repository maintainers; do not include live credentials or personal data in an issue. + +## Phase 15 deployment security controls + +- **Portable Virtualmin boundary:** Virtualmin is an operator-managed perimeter, not an application dependency. Require Docker Engine/Compose v2 on a patched Linux host, a reviewed checkout with restricted ownership/permissions, host firewalling, DNS under the operator's control, HTTPS with renewal monitoring, and reverse-proxy rules that do not expose the database volume or arbitrary container ports. Keep the API private where possible and allow only the intended web/API paths. +- **Secrets:** inject production values from a secret manager or protected deployment environment. `.env.example` is documentation only; never commit `.env`, passwords, tokens, provider credentials, certificates, or backup keys. Use bootstrap variables only once, remove them after provisioning, rotate the resulting credential, and prevent secrets from appearing in Compose output, process listings where feasible, logs, traces, metrics, browser storage, or error responses. +- **Health versus readiness:** `/api/v1/health/live` and `/healthz` intentionally require no session and reveal only process health; `/api/v1/health/ready` additionally checks SQLite readiness. These endpoints are not proof of backup, worker, migration, or external-dependency readiness. Gate ingress on Compose health plus authenticated smoke tests and migration checks; do not expose tenant data through health responses. +- **Data protection and retention:** encrypt backups in transit and at rest, restrict volume and backup access, use an off-host/isolated copy, and define retention separately for prospect/contact data, audit/source lineage, logs, and backups. Apply deletion and legal holds intentionally; preserve suppression/audit evidence when required. A Docker volume or host snapshot alone is not a verified backup. +- **Migration and rollback:** the current image has no standalone migration runner. Back up and restore-test before schema changes, validate row counts/foreign keys/indexes/tenant predicates and representative API reads on an isolated copy, and record the schema/data validation result. Pin image digests and configuration, retain the previous release, and ensure rollback does not run a newer schema against an incompatible older binary. Restore data only through an approved, compatibility-checked procedure. +- **Outbound deny-by-default:** `AUTOMATED_OUTREACH_ENABLED=false` is fixed in Compose and the current release has no send/provider/delivery path. Egress from the host/proxy should be restricted to documented needs; unexpected outbound traffic, SMTP, provider calls, or a newly introduced route is a security incident. Do not enable future outbound behavior without separate product/legal/security review, allowlisting, caps, suppression re-checks, audit, and a tested kill switch. +- **Monitoring and incident evidence:** alert on unhealthy containers, restart loops, disk/volume pressure, backup age/failure, restore-test failure, TLS expiry, authentication/authorization failures, migration errors, unexpected egress, and log redaction failures. Monitoring must not collect secrets or full contact payloads. Preserve redacted logs, audit records, image/config digests, and affected-tenant scope during incidents. + +These controls describe deployment prerequisites and gates; they do not make SQLite, password fallback, HTTP local Compose, in-process workers, or the public liveness checks production-grade. Remaining gaps must be accepted explicitly or closed before production. diff --git a/scripts/backup_sqlite.sh b/scripts/backup_sqlite.sh new file mode 100755 index 0000000..b1fe417 --- /dev/null +++ b/scripts/backup_sqlite.sh @@ -0,0 +1,40 @@ +#!/bin/sh +set -eu +umask 077 +DB_PATH=${1:-${PROSPECT_API_DB:-${DATA_DIR:-/data}/prospects.db}} +BACKUP_DIR=${2:-${BACKUP_DIR:-/var/backups/prospect-platform}} +RETENTION=${3:-${BACKUP_RETENTION:-30}} +case "$RETENTION" in ''|*[!0-9]*) echo 'retention must be a non-negative integer' >&2; exit 2;; esac +case "$BACKUP_DIR" in /*) ;; *) echo 'backup directory must be an absolute path' >&2; exit 2;; esac +case "$DB_PATH" in *.env|*secret*|*credentials*) echo 'refusing secret-like database path' >&2; exit 2;; esac +mkdir -p -- "$BACKUP_DIR" +python3 - "$DB_PATH" "$BACKUP_DIR" "$RETENTION" <<'PY' +import hashlib, os, sqlite3, sys, tempfile, time +from pathlib import Path +src, out_dir, retention = Path(sys.argv[1]).expanduser(), Path(sys.argv[2]).expanduser(), int(sys.argv[3]) +if not src.is_file() or not src.is_absolute(): raise SystemExit('database must be an existing absolute regular file') +out_dir = out_dir.resolve() +if src.resolve() == out_dir: raise SystemExit('backup directory must differ from database') +out_dir.mkdir(parents=True, exist_ok=True) +name = f"{os.environ.get('BACKUP_PREFIX', 'prospects')}-{time.strftime('%Y%m%dT%H%M%SZ', time.gmtime())}-{os.getpid()}" +final = out_dir / (name + '.db') +fd, temp_name = tempfile.mkstemp(prefix='.backup-', suffix='.tmp', dir=out_dir) +os.close(fd) +try: + with sqlite3.connect(f'file:{src}?mode=ro', uri=True) as source, sqlite3.connect(temp_name) as target: + source.backup(target) + target.execute('PRAGMA integrity_check') + target.commit() + with open(temp_name, 'rb') as handle: + os.fsync(handle.fileno()) + os.chmod(temp_name, 0o600); os.replace(temp_name, final) + digest = hashlib.sha256(final.read_bytes()).hexdigest() + sidecar = Path(str(final) + '.sha256') + sidecar.write_text(f'{digest} {final.name}\n', encoding='ascii'); os.chmod(sidecar, 0o600) + candidates = sorted([*out_dir.glob('prospects-*.db'), *out_dir.glob('pre-restore-*.db')], key=lambda p: p.stat().st_mtime, reverse=True) + for old in candidates[retention:]: + old.unlink(missing_ok=True); Path(str(old)+'.sha256').unlink(missing_ok=True) + print(final) +finally: + Path(temp_name).unlink(missing_ok=True) +PY diff --git a/scripts/healthcheck.sh b/scripts/healthcheck.sh new file mode 100755 index 0000000..c3a9243 --- /dev/null +++ b/scripts/healthcheck.sh @@ -0,0 +1,18 @@ +#!/bin/sh +set -eu +URL=${HEALTHCHECK_URL:-http://127.0.0.1:8000/api/v1/health/ready} +TIMEOUT=${HEALTHCHECK_TIMEOUT:-5} +python3 - "$URL" "$TIMEOUT" <<'PY' +import json, sys, urllib.request +url, timeout = sys.argv[1], float(sys.argv[2]) +try: + with urllib.request.urlopen(url, timeout=timeout) as response: + payload=json.loads(response.read()) + if response.status != 200 or payload.get('status') != 'ok' or payload.get('ready') is not True: + raise RuntimeError('service is not ready') + if payload.get('outreach_enabled') is not False: raise RuntimeError('outreach safety check failed') +except Exception as exc: + print(f'healthcheck failed: {type(exc).__name__}', file=sys.stderr) + raise SystemExit(1) +print('ok') +PY diff --git a/scripts/restore_sqlite.sh b/scripts/restore_sqlite.sh new file mode 100755 index 0000000..7e81d67 --- /dev/null +++ b/scripts/restore_sqlite.sh @@ -0,0 +1,43 @@ +#!/bin/sh +set -eu +umask 077 +if [ "${3:-}" != "--confirm-restore" ]; then + echo 'Refusing restore: pass --confirm-restore explicitly.' >&2 + exit 2 +fi +SOURCE=${1:-} +TARGET=${2:-${PROSPECT_API_DB:-${DATA_DIR:-/data}/prospects.db}} +BACKUP_DIR=${BACKUP_DIR:-$(dirname -- "$SOURCE")} +case "$SOURCE$TARGET" in *secret*|*credentials*|*.env*) echo 'refusing secret-like path' >&2; exit 2;; esac +[ -f "$SOURCE" ] || { echo 'restore source does not exist' >&2; exit 2; } +case "$TARGET" in /*) ;; *) echo 'restore target must be an absolute path' >&2; exit 2;; esac +mkdir -p -- "$BACKUP_DIR" +if [ -f "$TARGET" ]; then + pre_restore=$("$(dirname -- "$0")/backup_sqlite.sh" "$TARGET" "$BACKUP_DIR" 30) + pre_restore_db=$(printf '%s\n' "$pre_restore" | tail -n 1) + pre_restore_name="$BACKUP_DIR/pre-restore-$(date -u +%Y%m%dT%H%M%SZ)-$$.db" + mv -- "$pre_restore_db" "$pre_restore_name" + if [ -f "$pre_restore_db.sha256" ]; then mv -- "$pre_restore_db.sha256" "$pre_restore_name.sha256"; fi +fi +python3 - "$SOURCE" "$TARGET" <<'PY' +import hashlib, os, sqlite3, sys, tempfile +from pathlib import Path +source, target = Path(sys.argv[1]).resolve(), Path(sys.argv[2]).resolve() +if not source.is_file(): raise SystemExit('restore source must be a regular file') +sidecar = Path(str(source)+'.sha256') +if sidecar.exists(): + expected = sidecar.read_text(encoding='ascii').split()[0] + actual = hashlib.sha256(source.read_bytes()).hexdigest() + if expected != actual: raise SystemExit('restore checksum mismatch') +target.parent.mkdir(parents=True, exist_ok=True) +fd, tmp = tempfile.mkstemp(prefix='.restore-', suffix='.tmp', dir=target.parent); os.close(fd) +try: + with sqlite3.connect(f'file:{source}?mode=ro', uri=True) as src, sqlite3.connect(tmp) as dst: + src.backup(dst); result = dst.execute('PRAGMA integrity_check').fetchone()[0] + if result != 'ok': raise SystemExit('restored database integrity check failed') + dst.commit() + os.chmod(tmp, 0o600); os.replace(tmp, target) + print(target) +finally: + Path(tmp).unlink(missing_ok=True) +PY diff --git a/scripts/rollback.sh b/scripts/rollback.sh new file mode 100755 index 0000000..780c07e --- /dev/null +++ b/scripts/rollback.sh @@ -0,0 +1,14 @@ +#!/bin/sh +set -eu +cat >&2 <<'EOF' +Rollback is a documented, non-destructive procedure. No rollback command is executed. + +1. Identify the last known-good immutable image digest/tag and configuration revision. +2. Confirm the database backup is recent and run scripts/healthcheck.sh against the candidate. +3. Review the exact rendered config: docker compose config. +4. Change the deployment's image tag/digest in the deployment system (or pin IMAGE_TAG), then restart the service through the approved change process. +5. Verify /api/v1/health/ready, representative authenticated reads, logs, and outreach_enabled=false. +6. Record the rollback reason, old/new image digests, config revision, backup/checksum, and operator. + +This helper intentionally does not stop containers, delete images, restore databases, or alter production state. +EOF diff --git a/systemd/prospect-api.service.example b/systemd/prospect-api.service.example new file mode 100644 index 0000000..57bb1ed --- /dev/null +++ b/systemd/prospect-api.service.example @@ -0,0 +1,27 @@ +[Unit] +Description=Prospect Platform API (portable systemd/Virtualmin example) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=prospect +Group=prospect +WorkingDirectory=/srv/prospect-platform/apps/api +EnvironmentFile=-/etc/prospect-platform/prospect.env +ExecStart=/usr/bin/python3 /srv/prospect-platform/apps/api/app/main.py --host 127.0.0.1 --port 8000 --db /var/lib/prospect-platform/prospects.db +Restart=on-failure +RestartSec=5 +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/var/lib/prospect-platform +UMask=0077 + +[Install] +WantedBy=multi-user.target + +# Virtualmin: create the prospect user/domain, place the EnvironmentFile outside +# public_html, and use this unit (or its ExecStart/Restart settings) as the +# service command. Put TLS/reverse proxying in the Virtualmin-managed web tier.