add production readiness and recovery assets
This commit is contained in:
Marco0300
+12
-1
@@ -57,6 +57,7 @@ Implementations should expose source/query/job state without leaking raw payload
|
||||
### Health and workspace
|
||||
|
||||
- `GET /api/v1/health/live` — unauthenticated liveness check.
|
||||
- `GET /api/v1/health/ready` — unauthenticated readiness check; verifies SQLite connectivity and returns 503 when unavailable.
|
||||
- `GET /api/v1/auth/me` — current authenticated user and tenant.
|
||||
- `GET /api/v1/dashboard/summary` — tenant-scoped counts and score summary.
|
||||
|
||||
@@ -188,7 +189,17 @@ Every draft must retain recipient/channel, bounded redacted content or a safe co
|
||||
|
||||
Any future send or other side effect must require an `Idempotency-Key` scoped to tenant, operation, draft version, recipient, provider, and policy fingerprint. Exact retries return the original result; a different request under the same key is rejected. Enforce rate/message/cost caps before attempts and across retries, fallbacks, and workers; use bounded retry/backoff and circuit breaking. Audit draft/gate/approval/provider/cap/suppression events with safe per-item outcomes and redacted payloads. No route may infer completion from request acceptance.
|
||||
|
||||
The Phase 14 implementation remains documentation-only/pilot preparation: there is no draft persistence/API, consent ledger, legal-policy evaluator, configured provider, durable approval queue, delivery adapter, bounce/complaint feedback, secret manager, or production-grade audit/retention workflow in the current runtime. Production requires those components plus DPA/provider and jurisdictional legal review, suppression synchronization, kill switch, rollback/revocation, deletion/legal-hold verification, and integration tests proving no-send default, citation/hash binding, stale/uncertain gate failure, cap enforcement, idempotent replay/conflict rejection, approval expiry, and cross-tenant isolation.
|
||||
The Phase 13 implementation remains documentation-only/pilot preparation: there is no draft persistence/API, consent ledger, legal-policy evaluator, configured provider, durable approval queue, delivery adapter, bounce/complaint feedback, secret manager, or production-grade audit/retention workflow in the current runtime. Production requires those components plus DPA/provider and jurisdictional legal review, suppression synchronization, kill switch, rollback/revocation, deletion/legal-hold verification, and integration tests proving no-send default, citation/hash binding, stale/uncertain gate failure, cap enforcement, idempotent replay/conflict rejection, approval expiry, and cross-tenant isolation.
|
||||
|
||||
## Phase 15 deployment contract
|
||||
|
||||
The API image is portable but intentionally small: it runs as a non-root user, writes only `/data`, and initializes the SQLite schema from the image-bundled `schema.sql`. It does not provision Virtualmin, DNS, TLS, a reverse proxy, a secret manager, a durable queue, or a production database. A Virtualmin deployment must provide Docker Engine with Compose v2, a protected checkout, host firewalling, HTTPS termination, a private deployment network, sufficient CPU/RAM/disk, and an encrypted off-host backup destination. Publish the API only where the reverse proxy requires it; preferably expose the web entry point publicly and keep the API port private to the host/network.
|
||||
|
||||
Configuration is environment-only. `APP_ENV`, `LOG_LEVEL`, `CORS_ORIGINS`, and port values may be non-secret deployment settings. `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` are one-time provisioning inputs, must be supplied together through a protected environment/secret store, and must be removed and rotated immediately after bootstrap. Never accept provider credentials from clients or store secrets in source, images, Compose, committed `.env`, logs, traces, metrics, or audit payloads. `AUTOMATED_OUTREACH_ENABLED=false` is the enforced Compose default; no outbound provider or send route exists in this release.
|
||||
|
||||
`GET /api/v1/health/live` is an unauthenticated liveness check and `GET /api/v1/health/ready` is an unauthenticated SQLite readiness check. Operators must gate traffic on Compose `healthy` status plus an authenticated smoke test and migration validation; do not interpret either response as proof that backups, tenant authorization, workers, or external dependencies are ready.
|
||||
|
||||
The current runtime has no standalone migration, backup, restore, retention, or rollback command. Before changing an image or schema, quiesce writes, take and verify an encrypted backup, restore it into an isolated copy, run the API test suite and schema/data invariants, then deploy. Keep the previous image digest and configuration revision available; rollback must restore the application image/config first and only restore data when compatibility and operator approval are established. SQLite remains pilot-scale: no HA, PITR, durable worker lease/recovery, or online migration is implied. See `docs/OPERATIONS.md` and `docs/RELEASE_CHECKLIST.md` for the operator procedure.
|
||||
|
||||
## Remaining limitations and production migration work
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Environment configuration validation for portable deployments."""
|
||||
from dataclasses import dataclass
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ConfigError(ValueError):
|
||||
"""Raised when deployment configuration is unsafe or malformed."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Config:
|
||||
app_env: str
|
||||
data_dir: Path
|
||||
session_secret: str
|
||||
outreach_enabled: bool
|
||||
log_level: str
|
||||
|
||||
|
||||
def _env(values, key, default=""):
|
||||
return str(values.get(key, default) or "").strip()
|
||||
|
||||
|
||||
def load_config(values=None):
|
||||
values = os.environ if values is None else values
|
||||
app_env = _env(values, "APP_ENV", "development").lower()
|
||||
if app_env not in {"development", "test", "staging", "production"}:
|
||||
raise ConfigError("APP_ENV must be development, test, staging, or production")
|
||||
data_dir = Path(_env(values, "DATA_DIR", ".") or ".").expanduser()
|
||||
if not data_dir.is_absolute():
|
||||
data_dir = (Path.cwd() / data_dir).resolve()
|
||||
if data_dir.exists() and not data_dir.is_dir():
|
||||
raise ConfigError("DATA_DIR must be a directory")
|
||||
secret = _env(values, "SESSION_SECRET")
|
||||
if app_env == "production" and (len(secret) < 32 or secret.lower() in {"change-me", "development", "dev"}):
|
||||
raise ConfigError("SESSION_SECRET must be at least 32 characters in production")
|
||||
outreach = _env(values, "AUTOMATED_OUTREACH_ENABLED", "false").lower()
|
||||
if outreach not in {"", "0", "false", "no", "off"}:
|
||||
raise ConfigError("automated outreach is disabled in this release")
|
||||
log_level = _env(values, "LOG_LEVEL", "INFO").upper()
|
||||
if log_level not in {"QUIET", "ERROR", "WARNING", "INFO", "DEBUG"}:
|
||||
raise ConfigError("LOG_LEVEL is invalid")
|
||||
return Config(app_env, data_dir, secret, False, log_level)
|
||||
+10
-1
@@ -14,6 +14,7 @@ if __package__ in (None, ""):
|
||||
from app.contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS
|
||||
from app.scoring import DEFAULT_RULES, signals_for_business, evaluate_score, SCORE_VERSION
|
||||
from app.ai_assistance import generate as generate_ai, MAX_INPUT_ITEMS, MAX_OUTPUT_CHARS
|
||||
from app.config import load_config
|
||||
else:
|
||||
from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from .sources import adapter_for, contains_secret
|
||||
@@ -22,6 +23,7 @@ else:
|
||||
from .contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS
|
||||
from .scoring import DEFAULT_RULES, signals_for_business, evaluate_score, SCORE_VERSION
|
||||
from .ai_assistance import generate as generate_ai, MAX_INPUT_ITEMS, MAX_OUTPUT_CHARS
|
||||
from .config import load_config
|
||||
ORGANIZATION_ID = "demo-tenant"
|
||||
SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
|
||||
SESSION_DAYS = 7
|
||||
@@ -521,7 +523,13 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
|
||||
def do_GET(self):
|
||||
parsed=urlparse(self.path); path=parsed.path.rstrip("/")
|
||||
if path=="/api/v1/health/live": return self.send_json(200,{"status":"ok","organization_id":ORGANIZATION_ID})
|
||||
if path=="/api/v1/health/live": return self.send_json(200,{"status":"ok","organization_id":ORGANIZATION_ID,"outreach_enabled":False})
|
||||
if path=="/api/v1/health/ready":
|
||||
try:
|
||||
check=connect(self.server.db_path); check.execute("SELECT 1"); check.close()
|
||||
return self.send_json(200,{"status":"ok","ready":True,"database":"ok","outreach_enabled":False})
|
||||
except (sqlite3.Error, OSError):
|
||||
return self.send_json(503,{"status":"not_ready","ready":False,"database":"error","outreach_enabled":False})
|
||||
db=self.db()
|
||||
try:
|
||||
user=self.require_auth(db)
|
||||
@@ -1284,6 +1292,7 @@ def _job_worker(server):
|
||||
finally: db.close()
|
||||
|
||||
def create_server(host="127.0.0.1",port=8000,db_path="prospects.db"):
|
||||
load_config()
|
||||
server=ThreadingHTTPServer((host,port),ApiHandler);server.db_path=db_path;connect(db_path).close();server.job_stop=threading.Event();server.job_wakeup=threading.Event();server.job_thread=threading.Thread(target=_job_worker,args=(server,),daemon=True);server.job_thread.start()
|
||||
original_close=server.server_close
|
||||
def close():
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server
|
||||
from app.config import ConfigError, load_config
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
SCRIPTS = ROOT / "scripts"
|
||||
|
||||
|
||||
class Phase15OpsTests(unittest.TestCase):
|
||||
def test_config_defaults_are_safe_and_production_requires_explicit_secret(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
cfg = load_config({"DATA_DIR": tmp})
|
||||
self.assertFalse(cfg.outreach_enabled)
|
||||
self.assertEqual(cfg.app_env, "development")
|
||||
with self.assertRaises(ConfigError):
|
||||
load_config({"APP_ENV": "production", "DATA_DIR": tmp})
|
||||
cfg = load_config({"APP_ENV": "production", "DATA_DIR": tmp, "SESSION_SECRET": "a" * 32})
|
||||
self.assertEqual(cfg.session_secret, "a" * 32)
|
||||
|
||||
def test_health_and_readiness_report_database_state(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
db_path = str(Path(tmp) / "state.db")
|
||||
server = create_server("127.0.0.1", 0, db_path)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
conn = HTTPConnection("127.0.0.1", server.server_port, timeout=3)
|
||||
for path in ("/api/v1/health/live", "/api/v1/health/ready"):
|
||||
conn.request("GET", path)
|
||||
response = conn.getresponse()
|
||||
payload = json.loads(response.read())
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(payload["status"], "ok")
|
||||
self.assertFalse(payload["outreach_enabled"])
|
||||
finally:
|
||||
server.shutdown(); server.server_close(); thread.join(timeout=2)
|
||||
|
||||
def test_backup_restore_integrity_and_checksum(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp); db = root / "prospects.db"; backups = root / "backups"
|
||||
conn = sqlite3.connect(db); conn.execute("CREATE TABLE facts (value TEXT)"); conn.execute("INSERT INTO facts VALUES ('before')"); conn.commit(); conn.close()
|
||||
result = subprocess.run([str(SCRIPTS / "backup_sqlite.sh"), str(db), str(backups), "2"], text=True, capture_output=True, check=True)
|
||||
backup = Path(result.stdout.strip().splitlines()[-1])
|
||||
self.assertTrue(backup.exists()); self.assertTrue(Path(str(backup) + ".sha256").exists())
|
||||
self.assertEqual(hashlib.sha256(backup.read_bytes()).hexdigest(), Path(str(backup) + ".sha256").read_text().split()[0])
|
||||
conn = sqlite3.connect(db); conn.execute("UPDATE facts SET value='after'"); conn.commit(); conn.close()
|
||||
subprocess.run([str(SCRIPTS / "restore_sqlite.sh"), str(backup), str(db), "--confirm-restore"], text=True, capture_output=True, check=True)
|
||||
conn = sqlite3.connect(db); self.assertEqual(conn.execute("SELECT value FROM facts").fetchone()[0], "before"); conn.close()
|
||||
self.assertTrue(any(p.name.startswith("pre-restore-") for p in backups.glob("*.db")))
|
||||
|
||||
def test_backup_output_does_not_contain_secret_values(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp); db = root / "prospects.db"; backups = root / "backups"
|
||||
conn = sqlite3.connect(db); conn.execute("CREATE TABLE logs (message TEXT)"); conn.execute("INSERT INTO logs VALUES ('password=should-not-be-logged')"); conn.commit(); conn.close()
|
||||
result = subprocess.run([str(SCRIPTS / "backup_sqlite.sh"), str(db), str(backups), "1"], text=True, capture_output=True, check=True)
|
||||
combined = result.stdout + result.stderr
|
||||
self.assertNotIn("should-not-be-logged", combined)
|
||||
self.assertNotIn("password=", combined)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user