add deterministic pilot benchmarks

2026-09-03 12:50:00 +02:00
parent 6b41d5b9ee
commit 9622f76977
11 changed files with 763 additions and 0 deletions
@@ -174,3 +174,15 @@ Report vulnerabilities privately to the repository maintainers; do not include l
 - **Monitoring and incident evidence:** alert on unhealthy containers, restart loops, disk/volume pressure, backup age/failure, restore-test failure, TLS expiry, authentication/authorization failures, migration errors, unexpected egress, and log redaction failures. Monitoring must not collect secrets or full contact payloads. Preserve redacted logs, audit records, image/config digests, and affected-tenant scope during incidents.

 These controls describe deployment prerequisites and gates; they do not make SQLite, password fallback, HTTP local Compose, in-process workers, or the public liveness checks production-grade. Remaining gaps must be accepted explicitly or closed before production.
+
+## Phase 16 pilot benchmark security and evaluation controls
+
+- Benchmark only with labeled synthetic fixtures. Fixtures must exclude live personal/contact data, credentials, provider responses, uncontrolled network content, and production records. Give each fixture a stable ID, schema/label-policy version, expected class/safety state, rationale, and manifest hash; keep expected labels separate from implementation outputs to reduce leakage and silent relabeling.
+- Treat precision/recall as conditional evaluation metrics, not truth or authorization. Report `TP`, `FP`, `TN`, `FN`, support, formula, class averaging, threshold/tie-breaking/abstention rules, and `N/A` for zero denominators. Report unknown, blocked, partial, timeout, error, stale, conflicting, and suppressed states separately. Synthetic metrics cannot establish identity, ownership, consent, deliverability, fairness, generalization, security, or production performance.
+- Make classification/scoring reproducible from the commit, fixture hash, algorithm/rule-set versions, runtime/dependency/image digests, non-secret config fingerprint, seed, locale/timezone, rounding, tie-breaking, and host profile. Run fresh processes at least twice and compare deterministic class/score/band/eligibility/explanation/version fields exactly. Preserve failed and incomplete runs; do not tune away difficult fixtures or nondeterminism.
+- Measure latency with monotonic timestamps and include errors/timeouts. Report cold and warm cache distributions separately (count, min, median, p95, p99, max, error/timeout rate), cache key/version/freshness/expiry, hit/miss counts, and invalidation behavior. A cache hit is not a fresh security or business observation. Do not infer production SLOs, capacity, availability, or cost from local pilot measurements.
+- Use the synthetic-only acceptance gates from `docs/BENCHMARKS.md`: deterministic equality 100%; precision `>=0.90` and recall `>=0.85` only with positive support >=20; suppressed/blocked/malformed/unknown safety cases remain non-eligible/non-positive as applicable 100%; tenant-isolation probes deny or conceal cross-tenant access 100%; and no latency sample lacks a duration/cache-state label. These gates do not authorize production use.
+- Add blinded human review with a recorded seed and stratification by predicted class, uncertainty/suppression, and disagreement. Review at least 20 fixtures or 10% (whichever is larger, subject to a declared cap); use pseudonymous reviewer IDs, reason codes, independent review of disagreements, and documented adjudication. Human agreement and adjudication are pilot findings, not production accuracy claims.
+- Store raw samples and reports with minimum necessary data and access controls. Redact secrets, full contact values, and unnecessary fixture content from logs and audit records. Benchmark artifacts must not become an implicit source, evidence record, eligibility decision, cache authority, or outreach input.
+
+Phase 16 does not resolve publication or deployment security gates. Remote push remains blocked pending repository authentication and intended remote/branch permission. Production remains blocked pending the Phase 15 reviewed revision, host and Docker/Compose access, protected deployment directory, secret management, DNS/TLS/Virtualmin perimeter, encrypted off-host backup/restore evidence, monitoring, rollback ownership, and explicit operational approval. A local benchmark, Compose config pass, or local commit is not a remote publication or production deployment.