MarketingTool/docs/BENCHMARKS.md

# Phase 16 pilot benchmark methodology

Phase 16 defines a **pilot evaluation protocol**, not a production performance or quality claim. It evaluates the existing deterministic classification/scoring behavior and any explicitly named pilot path against labeled synthetic fixtures. It does not authorize live discovery, enrichment, acquisition, verification, or outreach, and it does not replace security, privacy, legal, or operational review.

## Scope and evaluation questions

The benchmark must answer only these bounded questions:

1. Given a fixed synthetic input and versioned configuration, does the classifier or scorer produce the expected class, score, band, eligibility, explanation, and uncertainty state?
2. How often do pilot outputs agree with the fixture labels under the declared policy?
3. What are the cold-cache and warm-cache latency distributions under a documented local workload?
4. Are results reproducible across repeated runs, processes, and supported Compose rebuilds?

A benchmark result is valid only for the fixture release, code revision, dependency/image digests, configuration fingerprint, host profile, and measurement procedure recorded with it. Do not extrapolate it to production traffic, real businesses, real people, live websites, external providers, or a different infrastructure profile.

## Labeled synthetic fixtures

Fixtures are synthetic and must contain no live personal data, credentials, provider responses, or uncontrolled network dependencies. Each fixture has a stable ID, schema version, input payload, expected label(s), expected uncertainty/suppression state where applicable, and a short rationale. Keep fixture inputs separate from expected outputs so the evaluator cannot silently derive labels from the implementation.

The fixture set should include balanced positive and negative examples plus deliberately difficult cases:

- positive, negative, unknown, blocked, partial, timeout, and error observations;
- boundary values around each score/band threshold;
- stale, conflicting, missing, suppressed, malformed, and duplicate inputs;
- adversarial-looking but harmless text, placeholder/example values, and irrelevant fields;
- repeated records and cacheable/non-cacheable cases;
- tenant-separation cases proving that identifiers, evidence, and cached results do not cross organizations.

Publish the fixture manifest and label policy with each benchmark run. Labels describe the expected outcome under the stated policy; they are not assertions that a real-world fact is true.

## Classification metrics and limitations

For a declared positive class, report a confusion matrix (`TP`, `FP`, `TN`, `FN`) and:

- **precision** = `TP / (TP + FP)`;
- **recall** = `TP / (TP + FN)`;
- support (number of labeled positive and negative fixtures);
- unknown/blocked/partial/error rates separately from positive/negative rates;
- per-class metrics when there is more than one class.

Use `N/A`, not zero, when a denominator is zero. Report micro/macro averaging choices for multiclass results and list the exact threshold, tie-breaking, abstention, and suppression rules used. A high precision or recall on synthetic fixtures does not demonstrate identity, ownership, consent, deliverability, safety, generalization, fairness, or production usefulness. Synthetic labels may be simpler, cleaner, or less representative than real inputs; class balance, fixture leakage, threshold tuning, and evaluator assumptions can materially change the result. Precision/recall also do not measure latency, calibration, operational failure modes, or tenant isolation.

## Reproducible classification and scoring

Every run records:

- repository commit and dirty/clean state;
- fixture manifest ID/hash and label-policy version;
- classifier/scorer algorithm and rule-set versions;
- Python/runtime and dependency versions plus Compose/image digests when Compose is used;
- explicit environment/configuration values, excluding secrets;
- random seed, locale, timezone, rounding mode, numeric precision, and tie-breaking rules;
- run command, host/CPU/RAM profile, start/end timestamps, and raw machine-readable results.

Run each fixture at least twice in fresh processes and compare serialized outputs, not only aggregate metrics. A reproducibility check passes only when class, score, band, eligibility, uncertainty reasons, explanation factors, and version metadata are identical. If a timestamp is intentionally dynamic, compare it through an explicitly documented normalization rule; never hide nondeterminism by rounding away output differences. Preserve failed and incomplete runs.

## Latency and cache measurements

Measure a fixed, documented workload with warm-up excluded from the reported sample. Record request/fixture ID, outcome, cache state (`cold`, `warm`, `bypass`, or `not_applicable`), start/end monotonic timestamps, duration, and failure/timeout reason. Report count, minimum, median, p95, p99, maximum, and timeout/error rate for cold and warm paths separately; do not report only an average. Use the same host, process model, concurrency, payload sizes, timeout budgets, and fixture order for comparable runs. State whether serialization, database access, container startup, and network calls are included. The default pilot benchmark is local and network-free; any network-backed experiment must be separately approved and named.

A cache hit must be labeled as a cache hit and must not be treated as a fresh observation. Record cache key policy, policy/scanner/rule versions, freshness/expiry, invalidation behavior, and hit/miss counts. Compare cold and warm measurements only when the cache contents and invalidation procedure are recorded. Do not infer production capacity, tail latency, cost, or availability from this pilot sample.

## Pilot acceptance thresholds

These are release gates for the **synthetic pilot evaluation only**; they are not service-level objectives or production claims:

| Gate | Pilot threshold | Required evidence |
| --- | --- | --- |
| Reproducibility | 100% byte-for-byte equality for deterministic output fields across repeated fresh-process runs | Per-fixture comparison report and manifest/config/version metadata |
| Positive classification precision | `>= 0.90` when positive support is at least 20; otherwise report `N/A` and do not pass the quality gate | Confusion matrix, support, threshold, and fixture manifest |
| Positive classification recall | `>= 0.85` when positive support is at least 20; otherwise report `N/A` and do not pass the quality gate | Confusion matrix, support, threshold, and fixture manifest |
| Safety abstention | 100% of labeled suppressed, blocked, malformed, or explicitly unknown cases remain non-eligible/non-positive as applicable | Per-fixture safety assertions and failure list |
| Tenant isolation | 100% of cross-tenant access/cache probes are denied or absent without revealing existence | Readback/probe evidence with tenant IDs redacted as needed |
| Latency reporting | 100% of samples have monotonic duration and cache-state labels; no missing/error samples are discarded | Raw sample file and summary statistics |

A failed or `N/A` gate is a pilot finding, not permission to tune labels or silently remove fixtures. Threshold changes require a new methodology/version and a fresh run. No benchmark may be signed off if fixtures, expected labels, failed samples, or configuration metadata are missing.

## Human review sampling

Automated metrics are supplemented by blinded human review. Select a reproducible random sample using the recorded seed, stratified at minimum by predicted class, uncertainty state, suppression state, and disagreement (`FP`/`FN`) status. Reviewers must see the bounded synthetic input and output explanation, but not the implementation label or aggregate result while labeling. Use at least 20 fixtures or 10% of the fixture set, whichever is larger, capped at the declared review budget; if fewer fixtures exist, review all and mark statistical power as limited.

Use two independent reviewers for the disagreement stratum and a random subset of the agreement stratum. Record reviewer IDs as pseudonymous codes, fixture ID, decision, reason code, time, and adjudication outcome. Do not turn reviewer agreement into a production accuracy claim. Disagreements, ambiguous labels, and unsafe outputs are findings requiring documented adjudication or fixture-policy revision before acceptance.

## Reporting and sign-off

The report includes methodology version, scope, fixture manifest/hash, run metadata, metrics, per-class support, raw latency/cache summaries, reproducibility result, human-review sample and adjudication, failed gates, limitations, and sign-off owner/date. Store reports with the pilot artifacts, not in application data or logs containing sensitive values.

Phase 16 is complete only when the report is reproducible locally, the acceptance gates are evaluated honestly, and all limitations are recorded. It does **not** mean the system is production-ready, scientifically validated, or safe for live prospect decisions.

## Deployment and remote-push prerequisites remain unresolved

Phase 16 documentation does not deploy the application or push Git history. Remote publication remains blocked until repository authentication and the intended remote/branch permissions are available. Deployment remains blocked until an operator supplies and verifies the Phase 15 prerequisites: reviewed remote revision, Docker/Compose host access, protected deployment directory, secrets, DNS/TLS/Virtualmin configuration, off-host encrypted backup destination, monitoring, rollback owner, and explicit operational approval. A local benchmark pass, local Compose validation, or local commit must not be reported as a remote push or production deployment.

## Suggested local evidence commands

From the repository root:

```sh
python3 -m unittest discover -v -s apps/api/tests -t apps/api
python3 -m compileall -q apps/api apps/web
docker compose config --quiet
git diff --check
```

Record command output and exit status with the benchmark report. These commands validate the local artifact; they do not provide production evidence or resolve the remote push/deployment prerequisites.