add policy-aware source adapter framework

2026-09-02 18:58:10 +02:00
parent cf034288e6
commit 46cc1f6182
13 changed files with 359 additions and 13 deletions
@@ -1,6 +1,6 @@
-# Prospect Platform API — Phase 4 boundary
+# Prospect Platform API — Phase 5 boundary

-Dependency-light JSON API for tenant-scoped, **manual** prospect workflows. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context, with the Phase 4 job/live-log contract described below. It never performs automated discovery, DNS/website scanning, or outreach.
+Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 5 source-ingestion contract. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.

 ## Run

@@ -34,6 +34,24 @@ The MVP job routes are `POST /api/v1/jobs`, `GET /api/v1/jobs`, `GET /api/v1/job

 The current runtime has no durable job queue and no Redis/Celery integration. Its in-process/SQLite worker is single-instance, non-durable across process loss, and unsuitable for horizontal scaling or guaranteed execution; it is pilot-only.

+### Phase 5 source adapter contract
+
+Phase 5 treats a source as a registered, reviewable capability rather than an arbitrary URL or scraper. Every adapter contract must identify:
+
+- a stable adapter/source ID and version, owner, permitted purpose, and terms/robots contact;
+- accepted discovery-query fields, result schema, provenance fields, and validation/error behavior;
+- request and concurrency rate limits, retry/backoff rules, timeout/size bounds, and a retention class for raw source records;
+- health signals and circuit-breaker states (`closed`, `open`, `half-open`), including fail-closed behavior when unhealthy; and
+- an explicit `dry_run` mode that validates and plans work without contacting a source or writing prospect facts.
+
+The initial safe adapters are `csv` and `manual_reference`. CSV input may be parsed and previewed; a manual reference records operator-supplied source identity, citation/reference, observed value, and timestamp. Neither adapter independently verifies a source or authorizes outreach. Raw source records should be retained immutably enough to reproduce the normalized result, with tenant/source/query identifiers, capture time, adapter version, and redaction/retention metadata; never retain secrets or unnecessary personal data.
+
+A source registry entry must include its terms owner, approval status/expiry, allowed tenants or scopes, rate-limit policy, retention class, and health/circuit policy. A discovery query is bounded and tenant-scoped, and its execution mode must be explicit (`dry_run` by default). A live network source is not implemented and must be rejected unless product, legal, and security approval is recorded and operations explicitly enables the registered adapter. No query, job acceptance, or successful parse may be described as network discovery.
+
+### Phase 5 source controls (contract, not current live routes)
+
+Implementations should expose source/query/job state without leaking raw payloads across tenants, including source approval, terms, rate-limit, retention, health, and circuit-open reason. On rate-limit, terms, approval, or circuit failure, return a safe non-live outcome and preserve an audit event; do not silently retry against another source. Dry-run must be side-effect-free with respect to external sources and prospect facts. These are Phase 5 design requirements; the current MVP has no network adapter or discovery endpoint.
+
 ### Health and workspace

 - `GET /api/v1/health/live` — unauthenticated liveness check.