add policy-aware source adapter framework

README.md

@@ -1,6 +1,6 @@
 # Prospect Intelligence Platform
 
-A safety-first Phase 4 design/implementation boundary for **manual**, evidence-led prospect qualification and the future job/live-log workflow. The current runtime remains the Phase 3 manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. **Automated discovery, DNS/website scanning, and outreach are not part of this release. Automated outreach is disabled.**
+A safety-first Phase 5 design/implementation boundary for **manual**, evidence-led prospect qualification and controlled source ingestion. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 5 defines source adapters, discovery-query records, raw-source retention, and health controls; it does **not** enable network discovery. **Automated outreach is disabled, and no live source may be enabled without explicit approval.**
 
 ## Included
 
@@ -12,8 +12,10 @@ A safety-first Phase 4 design/implementation boundary for **manual**, evidence-l
 - Docker Compose runtime with non-root containers, read-only filesystems, health checks, and a named SQLite data volume.
 - Browser authentication with server-side sessions and an optional first-run admin bootstrap.
 - Phase 4 MVP job monitor and SQLite-backed job/event schema/API surface, with the production limitations documented below.
+- Phase 5 source-ingestion contract: an approved source registry owns adapter terms, rate limits, retention, and health/circuit policy; CSV and manual reference adapters are the safe initial adapters.
+- Discovery queries are recorded as bounded, auditable intent and dry-run plans. Recording a query does not perform network discovery or imply that results exist.
 
-## Current workflow and Phase 4 boundary
+## Current workflow and Phase 4/5 boundary
 
 1. A permitted workspace member manually creates or reviews a prospect.
 2. The business detail response is the aggregate record for that tenant; related intelligence/evidence rows are returned only through the tenant-scoped detail surface.
@@ -76,9 +78,13 @@ Compose passes the optional `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWOR
 
 Authenticated browser requests use a server-side session cookie; login creates a session and logout invalidates it. The liveness endpoints (`GET /api/v1/health/live` and `GET /healthz`) intentionally remain unauthenticated so Docker, ingress, and monitoring health checks can use them. Authentication is not a substitute for tenant/authorization checks: protected routes must enforce the session and organization boundary server-side.
 
-## Explicit non-goals and remaining limitations
+## Phase 5 source boundary and remaining limitations
 
-This Phase 4 boundary still has no automated discovery, DNS resolution, website/HTTP scanning, enrichment scheduler, external source adapter, email/SMS sender, or outreach endpoint. CSV remains a browser/API preview flow and does not silently persist rows. SQLite, the in-process worker, and the named local volume are suitable for the pilot only; production migration, durable queue/worker leases, event retention/backup, SSE delivery, and tested backup/restore remain unfinished. Redis and Celery are not implemented. The development password fallback is PBKDF2 rather than production Argon2id. Before production, complete the gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`, including MFA, TLS, CSRF protection, rate limiting, tenant-scoped job/event authorization, idempotent side-effect handling, durable audit/event retention, migrations, approved source policy, SSRF-safe fetching if a future scanner is approved, and tested backups/restores.
+Phase 5 defines a source adapter contract and registry; it does not implement network discovery, DNS resolution, website/HTTP scanning, enrichment scheduling, or a live external-source adapter. A source adapter must declare its identity, terms owner, permitted purpose, rate limits, retention class, query/result schema, dry-run behavior, and health/circuit controls. CSV and manual reference adapters may be used for operator-supplied data; they must preserve source attribution and raw source records, and must not silently turn preview data into outreach or verified facts.
+
+A discovery query is a tenant-scoped, bounded, auditable request that can be validated and dry-run without contacting a source. Any live source requires explicit product/legal/security approval, a registered adapter, and an operational enablement decision; absent all three, execution must fail closed. Circuit-open, rate-limit, terms, or approval failures must produce a safe non-live result. Raw source records are retained only under the approved retention class and must exclude secrets and unnecessary personal data.
+
+SQLite, the in-process worker, and the named local volume are suitable for the pilot only; production migration, durable queue/worker leases, event retention/backup, SSE delivery, and tested backup/restore remain unfinished. Redis and Celery are not implemented. The development password fallback is PBKDF2 rather than production Argon2id. Before production, complete the gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`, including MFA, TLS, CSRF protection, source approval and terms review, rate limiting, circuit monitoring, tenant-scoped job/event authorization, idempotent side-effect handling, durable raw-source/audit retention, SSRF-safe fetching if a future scanner is approved, and tested backups/restores.
 
 ## Verification
 

apps/api/README.md

@@ -1,6 +1,6 @@
-# Prospect Platform API — Phase 4 boundary
+# Prospect Platform API — Phase 5 boundary
 
-Dependency-light JSON API for tenant-scoped, **manual** prospect workflows. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context, with the Phase 4 job/live-log contract described below. It never performs automated discovery, DNS/website scanning, or outreach.
+Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 5 source-ingestion contract. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
 
 ## Run
 
@@ -34,6 +34,24 @@ The MVP job routes are `POST /api/v1/jobs`, `GET /api/v1/jobs`, `GET /api/v1/job
 
 The current runtime has no durable job queue and no Redis/Celery integration. Its in-process/SQLite worker is single-instance, non-durable across process loss, and unsuitable for horizontal scaling or guaranteed execution; it is pilot-only.
 
+### Phase 5 source adapter contract
+
+Phase 5 treats a source as a registered, reviewable capability rather than an arbitrary URL or scraper. Every adapter contract must identify:
+
+- a stable adapter/source ID and version, owner, permitted purpose, and terms/robots contact;
+- accepted discovery-query fields, result schema, provenance fields, and validation/error behavior;
+- request and concurrency rate limits, retry/backoff rules, timeout/size bounds, and a retention class for raw source records;
+- health signals and circuit-breaker states (`closed`, `open`, `half-open`), including fail-closed behavior when unhealthy; and
+- an explicit `dry_run` mode that validates and plans work without contacting a source or writing prospect facts.
+
+The initial safe adapters are `csv` and `manual_reference`. CSV input may be parsed and previewed; a manual reference records operator-supplied source identity, citation/reference, observed value, and timestamp. Neither adapter independently verifies a source or authorizes outreach. Raw source records should be retained immutably enough to reproduce the normalized result, with tenant/source/query identifiers, capture time, adapter version, and redaction/retention metadata; never retain secrets or unnecessary personal data.
+
+A source registry entry must include its terms owner, approval status/expiry, allowed tenants or scopes, rate-limit policy, retention class, and health/circuit policy. A discovery query is bounded and tenant-scoped, and its execution mode must be explicit (`dry_run` by default). A live network source is not implemented and must be rejected unless product, legal, and security approval is recorded and operations explicitly enables the registered adapter. No query, job acceptance, or successful parse may be described as network discovery.
+
+### Phase 5 source controls (contract, not current live routes)
+
+Implementations should expose source/query/job state without leaking raw payloads across tenants, including source approval, terms, rate-limit, retention, health, and circuit-open reason. On rate-limit, terms, approval, or circuit failure, return a safe non-live outcome and preserve an audit event; do not silently retry against another source. Dry-run must be side-effect-free with respect to external sources and prospect facts. These are Phase 5 design requirements; the current MVP has no network adapter or discovery endpoint.
+
 ### Health and workspace
 
 - `GET /api/v1/health/live` — unauthenticated liveness check.

apps/api/app/main.py

@@ -8,14 +8,16 @@ from urllib.parse import parse_qs, urlparse
 if __package__ in (None, ""):
     sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
     from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone
+    from app.sources import adapter_for, contains_secret
 else:
     from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone
+    from .sources import adapter_for, contains_secret
 ORGANIZATION_ID = "demo-tenant"
 SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
 SESSION_DAYS = 7
 PBKDF2_ITERATIONS = 300_000
 MUTATING_ROLES = {"owner", "admin", "researcher"}
-JOB_TYPES = {"noop", "prospect_recalculate"}
+JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery"}
 JOB_PAGE_SIZE = 100
 SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
 CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
@@ -110,6 +112,9 @@ class ApiHandler(BaseHTTPRequestHandler):
             if path=="/api/v1/dashboard/summary":
                 row=db.execute("SELECT COUNT(*) businesses,COALESCE(AVG(score),0) average_score FROM businesses WHERE organization_id=?",(org,)).fetchone(); return self.send_json(200,{"organization_id":org,"businesses":row["businesses"],"average_score":round(row["average_score"],2),"suppressed":db.execute("SELECT COUNT(*) FROM suppressions WHERE organization_id=?",(org,)).fetchone()[0]})
             if path=="/api/v1/businesses": return self.list_businesses(db,org,parse_qs(parsed.query))
+            if path=="/api/v1/sources": return self.list_sources(db,org)
+            if path=="/api/v1/discovery-queries": return self.list_queries(db,org)
+            if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query))
             if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
             if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
             if path.startswith("/api/v1/businesses/"):
@@ -153,7 +158,7 @@ class ApiHandler(BaseHTTPRequestHandler):
         try:
             cur=db.execute("INSERT INTO jobs(organization_id,idempotency_key,type,payload,max_attempts) VALUES(?,?,?,?,?)",(user["organization_id"],key,kind,safe,max_attempts)); jid=cur.lastrowid
             self.add_job_event(db,jid,user["organization_id"],"queued","Job queued",0); self.audit(db,user,"job.created",str(jid)); db.commit(); getattr(self.server,"job_wakeup",threading.Event()).set()
-            return self.send_json(201,job_json(db.execute("SELECT * FROM jobs WHERE id=?",(jid,)).fetchone()))
+            return self.send_json(202 if payload.get("_accepted") else 201,job_json(db.execute("SELECT * FROM jobs WHERE id=?",(jid,)).fetchone()))
         except sqlite3.IntegrityError:
             row=db.execute("SELECT * FROM jobs WHERE organization_id=? AND idempotency_key=?",(user["organization_id"],key)).fetchone(); return self.send_json(200,job_json(row))
 
@@ -210,9 +215,14 @@ class ApiHandler(BaseHTTPRequestHandler):
             if path.startswith("/api/v1/jobs/"):
                 return self.job_action(db,user,path)
             if path=="/api/v1/businesses":return self.create_business(payload,db,user)
+            if path=="/api/v1/sources":return self.create_source(payload,db,user)
+            if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
             if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
             if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
             bits=path.split("/")
+            if len(bits)==6 and bits[3] == "sources" and bits[4].isdigit() and bits[5] in {"test","ingest"}:
+                return self.test_source(int(bits[4]),db,user) if bits[5]=="test" else self.ingest_source(int(bits[4]),payload,db,user)
+            if len(bits)==6 and bits[3] == "discovery-queries" and bits[4].isdigit() and bits[5]=="run": return self.run_query(int(bits[4]),db,user)
             if len(bits)==7 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES and bits[6]=="": pass
             if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES:return self.create_child(int(bits[4]) if bits[4].isdigit() else -1,bits[5],payload,db,user)
             if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="verify":return self.verify_business(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user)
@@ -225,6 +235,7 @@ class ApiHandler(BaseHTTPRequestHandler):
             if not user:return
             if user["role"] not in MUTATING_ROLES:return self.send_json(403,{"error":"forbidden"})
             bits=path.split("/")
+            if len(bits)==5 and bits[:4]==["","api","v1","sources"] and bits[4].isdigit(): return self.update_source(int(bits[4]),self.read_json(),db,user)
             if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="pipeline":return self.update_pipeline(int(bits[4]) if bits[4].isdigit() else -1,self.read_json(),db,user)
             return self.send_json(404,{"error":"not_found"})
         finally:db.close()
@@ -286,6 +297,73 @@ class ApiHandler(BaseHTTPRequestHandler):
             elif key in existing_keys or key in seen:continue
             else:seen.add(key);accepted.append(b)
         return self.send_json(200,{"accepted":len(accepted),"duplicates":len(rows)-len(normalized)+len(normalized)-len(accepted)-suppressed,"suppressed":suppressed,"rows":accepted})
+    def list_sources(self,db,org):
+        cols='id,organization_id,name,kind,enabled,health_status,consecutive_failures,circuit_open,last_success_at,last_failure_at,last_error,created_at,updated_at'
+        return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in db.execute(f"SELECT {cols} FROM sources WHERE organization_id=? ORDER BY id",(org,))]})
+    def list_queries(self,db,org):
+        return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in db.execute("SELECT * FROM discovery_queries WHERE organization_id=? ORDER BY id",(org,))]})
+    def list_source_records(self,db,org,q):
+        try:
+            limit=int(q.get('page_size',[50])[0]); offset=max(0,int(q.get('offset',[0])[0]))
+            if limit<1 or limit>100: raise ValueError
+        except (ValueError,TypeError): return self.send_json(400,{"error":"invalid_pagination"})
+        rows=db.execute("SELECT * FROM source_records WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall(); out=[]
+        for r in rows[:limit]:
+            x=row_json(r)
+            for k in ('raw_json','normalized_json','query_context_json','cursor_json','rate_policy_json'):
+                try:x[k]=json.loads(x[k])
+                except (ValueError,TypeError):pass
+            out.append(x)
+        return self.send_json(200,{"organization_id":org,"items":out,"limit":limit,"offset":offset,"has_more":len(rows)>limit})
+    def create_source(self,payload,db,user):
+        name=str(payload.get('name','')).strip(); kind=str(payload.get('kind','')).strip().lower(); config=payload.get('config',{})
+        if not name or kind not in ('csv','manual') or not isinstance(config,dict):return self.send_json(400,{"error":"invalid_source"})
+        if contains_secret(config):return self.send_json(400,{"error":"secret_not_permitted"})
+        try:
+            validation=adapter_for(kind).validate(config)
+            if config and not validation.valid:return self.send_json(400,{"error":"invalid_source_config","details":validation.errors})
+            cur=db.execute("INSERT INTO sources(organization_id,name,kind,enabled,config_json) VALUES(?,?,?,?,?)",(user['organization_id'],name,kind,int(bool(payload.get('enabled',False))),json.dumps(config,sort_keys=True)))
+        except sqlite3.IntegrityError:return self.send_json(409,{"error":"duplicate_source"})
+        self.audit(db,user,'source.created',str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT id,organization_id,name,kind,enabled,health_status,consecutive_failures,circuit_open,last_success_at,last_failure_at,last_error,created_at,updated_at FROM sources WHERE id=?",(cur.lastrowid,)).fetchone()))
+    def update_source(self,sid,payload,db,user):
+        if not db.execute("SELECT id FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone():return self.send_json(404,{"error":"not_found"})
+        if 'enabled' not in payload:return self.send_json(400,{"error":"enabled_required"})
+        value=int(bool(payload['enabled']));db.execute("UPDATE sources SET enabled=?,updated_at=CURRENT_TIMESTAMP WHERE id=?",(value,sid));self.audit(db,user,'source.enabled' if value else 'source.disabled',str(sid));db.commit();return self.send_json(200,row_json(db.execute("SELECT * FROM sources WHERE id=?",(sid,)).fetchone()))
+    def create_query(self,payload,db,user):
+        sid=payload.get('source_id');name=str(payload.get('name','')).strip();query=payload.get('query',{})
+        if not isinstance(sid,int) or not name or not isinstance(query,dict) or contains_secret(query):return self.send_json(400,{"error":"invalid_query"})
+        if not db.execute("SELECT id FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone():return self.send_json(404,{"error":"not_found"})
+        try:cur=db.execute("INSERT INTO discovery_queries(organization_id,source_id,name,query_json) VALUES(?,?,?,?)",(user['organization_id'],sid,name,json.dumps(query,sort_keys=True)))
+        except sqlite3.IntegrityError:return self.send_json(409,{"error":"duplicate_query"})
+        self.audit(db,user,'discovery_query.created',str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT * FROM discovery_queries WHERE id=?",(cur.lastrowid,)).fetchone()))
+    def run_query(self,qid,db,user):
+        if not db.execute("SELECT id FROM discovery_queries WHERE id=? AND organization_id=?",(qid,user['organization_id'])).fetchone():return self.send_json(404,{"error":"not_found"})
+        return self.create_job({"type":"source_discovery","_accepted":True,"payload":{"discovery_query_id":qid},"idempotency_key":f"discovery-query-{qid}-{int(time.time())}"},db,user)
+    def test_source(self,sid,db,user):
+        source=db.execute("SELECT * FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone()
+        if not source:return self.send_json(404,{"error":"not_found"})
+        try: result=adapter_for(source['kind']).validate(json.loads(source['config_json'])); ok=result.valid; error='; '.join(result.errors) if not ok else None
+        except Exception as exc:ok=False;error=str(exc)[:300]
+        if ok:db.execute("UPDATE sources SET health_status='healthy',consecutive_failures=0,circuit_open=0,last_success_at=CURRENT_TIMESTAMP,last_error=NULL WHERE id=?",(sid,));action='source.test.succeeded'
+        else:db.execute("UPDATE sources SET health_status='unhealthy',consecutive_failures=consecutive_failures+1,circuit_open=CASE WHEN consecutive_failures+1>=3 THEN 1 ELSE circuit_open END,last_failure_at=CURRENT_TIMESTAMP,last_error=? WHERE id=?",(error,sid));action='source.test.failed'
+        self.audit(db,user,action,str(sid));db.commit();return self.send_json(200,{"ok":ok,"errors":[] if ok else [error]})
+    def ingest_source(self,sid,payload,db,user):
+        source=db.execute("SELECT * FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone()
+        if not source:return self.send_json(404,{"error":"not_found"})
+        if not source['enabled'] or source['circuit_open']:return self.send_json(409,{"error":"source_disabled"})
+        if contains_secret(payload):return self.send_json(400,{"error":"secret_not_permitted"})
+        config={k:v for k,v in payload.items() if k not in ('source_url','query_context','cursor','rate_policy')}
+        if len(json.dumps(config).encode())>5*1024*1024:return self.send_json(400,{"error":"ingest_limits"})
+        if isinstance(config.get('rows'),list) and (len(config['rows'])>1000 or any(not isinstance(r,dict) or len(r)>50 or any(len(str(v))>10000 for v in r.values()) for r in config['rows'])):return self.send_json(400,{"error":"ingest_limits"})
+        if isinstance(config.get('csv'),str) and config['csv'].count('\n')>1001:return self.send_json(400,{"error":"ingest_limits"})
+        try:page=adapter_for(source['kind']).discover(config)
+        except (ValueError,KeyError) as exc:return self.send_json(400,{"error":"invalid_ingest","detail":str(exc)})
+        inserted=0
+        for record in page.records[:1000]:
+            raw=json.dumps(record,sort_keys=True,separators=(',',':'));digest=hashlib.sha256(raw.encode()).hexdigest()
+            try:db.execute("INSERT INTO source_records(organization_id,source_id,content_hash,raw_json,normalized_json,source_url,query_context_json,cursor_json,rate_policy_json) VALUES(?,?,?,?,?,?,?,?,?)",(user['organization_id'],sid,digest,raw,raw,str(payload.get('source_url','')),json.dumps(payload.get('query_context',{}),sort_keys=True),json.dumps(payload.get('cursor',{}),sort_keys=True),json.dumps(payload.get('rate_policy',{}),sort_keys=True)));inserted+=1
+            except sqlite3.IntegrityError:pass
+        db.execute("UPDATE sources SET health_status='healthy',consecutive_failures=0,last_success_at=CURRENT_TIMESTAMP,last_error=NULL WHERE id=?",(sid,));self.audit(db,user,'source.ingested',f'{sid}:{inserted}');db.commit();return self.send_json(201 if inserted else 200,{"inserted":inserted,"records":len(page.records)})
     def log_message(self,*_):pass
 
 def _job_worker(server):

apps/api/app/sources.py

@@ -0,0 +1,94 @@
+"""Deterministic, network-free discovery source contracts and adapters."""
+from __future__ import annotations
+from dataclasses import dataclass, field
+from typing import Any, Mapping, Protocol, Sequence
+import csv, io, re
+
+SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "private_key", "credential"}
+
+def contains_secret(value: Any, path: str = "") -> str | None:
+    if isinstance(value, Mapping):
+        for key, child in value.items():
+            key_text = str(key).lower()
+            if key_text in SECRET_KEYS or any(x in key_text for x in ("password", "token", "secret", "api_key", "private_key")):
+                return path + str(key)
+            found = contains_secret(child, path + str(key) + ".")
+            if found: return found
+    elif isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
+        for index, child in enumerate(value):
+            found = contains_secret(child, path + str(index) + ".")
+            if found: return found
+    return None
+
+@dataclass(frozen=True)
+class ValidationResult:
+    valid: bool
+    errors: list[str] = field(default_factory=list)
+
+@dataclass(frozen=True)
+class DiscoveryPage:
+    records: list[dict[str, str]]
+    next_cursor: str | None = None
+    metadata: dict[str, Any] = field(default_factory=dict)
+
+@dataclass(frozen=True)
+class SourceHealth:
+    status: str = "unknown"
+    consecutive_failures: int = 0
+    circuit_open: bool = False
+    last_error: str | None = None
+
+class DiscoverySource(Protocol):
+    kind: str
+    def validate(self, config: Mapping[str, Any]) -> ValidationResult: ...
+    def discover(self, config: Mapping[str, Any], cursor: str | None = None) -> DiscoveryPage: ...
+
+_FIELDS = ("name", "website", "email", "phone", "description")
+
+def normalize_record(row: Mapping[str, Any]) -> dict[str, str]:
+    result = {field: str(row.get(field, "")).strip() for field in _FIELDS}
+    # Accept common CSV spellings without retaining arbitrary sensitive fields.
+    aliases = {"company": "name", "url": "website", "domain": "website"}
+    for key, target in aliases.items():
+        if not result[target] and row.get(key) is not None: result[target] = str(row[key]).strip()
+    return result
+
+class ManualSource:
+    kind = "manual"
+    def validate(self, config: Mapping[str, Any]) -> ValidationResult:
+        if not isinstance(config, Mapping): return ValidationResult(False, ["config must be an object"])
+        found = contains_secret(config)
+        if found: return ValidationResult(False, [f"secret field is not permitted: {found}"])
+        rows = config.get("rows")
+        if not isinstance(rows, list): return ValidationResult(False, ["rows must be a list"])
+        return ValidationResult(True)
+    def discover(self, config: Mapping[str, Any], cursor: str | None = None) -> DiscoveryPage:
+        validation = self.validate(config)
+        if not validation.valid: raise ValueError(validation.errors[0])
+        rows = [normalize_record(row) for row in config["rows"] if isinstance(row, Mapping)]
+        return DiscoveryPage(rows, None, {"adapter": self.kind})
+
+class CsvSource:
+    kind = "csv"
+    def validate(self, config: Mapping[str, Any]) -> ValidationResult:
+        if not isinstance(config, Mapping): return ValidationResult(False, ["config must be an object"])
+        found = contains_secret(config)
+        if found: return ValidationResult(False, [f"secret field is not permitted: {found}"])
+        if not isinstance(config.get("csv"), str): return ValidationResult(False, ["csv must be text"])
+        try:
+            reader = csv.DictReader(io.StringIO(config["csv"]));
+            if not reader.fieldnames: return ValidationResult(False, ["CSV header is required"])
+        except csv.Error as exc: return ValidationResult(False, [f"invalid CSV: {exc}"])
+        return ValidationResult(True)
+    def discover(self, config: Mapping[str, Any], cursor: str | None = None) -> DiscoveryPage:
+        validation = self.validate(config)
+        if not validation.valid: raise ValueError(validation.errors[0])
+        reader = csv.DictReader(io.StringIO(config["csv"].replace("\r\n", "\n")))
+        records = [normalize_record({str(k).strip().lower(): v for k, v in row.items()}) for row in reader]
+        return DiscoveryPage(records, None, {"adapter": self.kind, "columns": reader.fieldnames or []})
+
+ADAPTERS = {"manual": ManualSource, "csv": CsvSource}
+
+def adapter_for(kind: str) -> DiscoverySource:
+    try: return ADAPTERS[kind]()
+    except KeyError: raise ValueError("unsupported source kind")

apps/api/schema.sql

@@ -114,3 +114,34 @@ CREATE TABLE IF NOT EXISTS job_events (
  UNIQUE(job_id,sequence)
 );
 CREATE INDEX IF NOT EXISTS idx_job_events_job_sequence ON job_events(job_id,sequence);
+
+-- Phase 5 source framework (additive-safe; credentials contain metadata only).
+CREATE TABLE IF NOT EXISTS sources (
+ id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id),
+ name TEXT NOT NULL, kind TEXT NOT NULL CHECK(kind IN ('csv','manual')), enabled INTEGER NOT NULL DEFAULT 0,
+ config_json TEXT NOT NULL DEFAULT '{}', health_status TEXT NOT NULL DEFAULT 'unknown',
+ consecutive_failures INTEGER NOT NULL DEFAULT 0, circuit_open INTEGER NOT NULL DEFAULT 0,
+ last_success_at TEXT, last_failure_at TEXT, last_error TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,name)
+);
+CREATE INDEX IF NOT EXISTS idx_sources_org ON sources(organization_id,id);
+CREATE TABLE IF NOT EXISTS source_credentials (
+ id INTEGER PRIMARY KEY AUTOINCREMENT, source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
+ organization_id TEXT NOT NULL REFERENCES organizations(id), provider TEXT NOT NULL, key_name TEXT NOT NULL,
+ secret_ref TEXT NOT NULL DEFAULT '', metadata_json TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ UNIQUE(source_id,provider,key_name)
+);
+CREATE TABLE IF NOT EXISTS discovery_queries (
+ id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
+ name TEXT NOT NULL, query_json TEXT NOT NULL DEFAULT '{}', enabled INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,name)
+);
+CREATE INDEX IF NOT EXISTS idx_discovery_queries_org ON discovery_queries(organization_id,id);
+CREATE TABLE IF NOT EXISTS source_records (
+ id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
+ discovery_query_id INTEGER REFERENCES discovery_queries(id) ON DELETE SET NULL, content_hash TEXT NOT NULL, raw_json TEXT NOT NULL,
+ normalized_json TEXT NOT NULL, source_url TEXT NOT NULL DEFAULT '', query_context_json TEXT NOT NULL DEFAULT '{}', processing_status TEXT NOT NULL DEFAULT 'raw',
+ cursor_json TEXT NOT NULL DEFAULT '{}', rate_policy_json TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ UNIQUE(organization_id,source_id,content_hash)
+);
+CREATE INDEX IF NOT EXISTS idx_source_records_org ON source_records(organization_id,id DESC);

apps/api/tests/test_sources_phase5.py

@@ -0,0 +1,54 @@
+import json, os, sqlite3, threading, unittest
+from http.client import HTTPConnection
+from tempfile import TemporaryDirectory
+from app.main import create_server
+from app.sources import CsvSource, ManualSource
+
+class SourceAdapterTests(unittest.TestCase):
+    def test_csv_adapter_is_deterministic_and_normalizes(self):
+        src = CsvSource()
+        a = src.discover({'csv': 'Name,Website,Email\n Acme ,https://acme.test,a@acme.test\n'})
+        b = src.discover({'csv': 'Name,Website,Email\n Acme ,https://acme.test,a@acme.test\n'})
+        self.assertEqual(a.records, b.records)
+        self.assertEqual(a.records[0]['name'], 'Acme')
+        self.assertEqual(a.records[0]['email'], 'a@acme.test')
+
+    def test_manual_validation_rejects_secret_fields(self):
+        result = ManualSource().validate({'rows': [{'name': 'x', 'api_key': 'secret'}]})
+        self.assertFalse(result.valid)
+        self.assertIn('secret', result.errors[0].lower())
+
+class SourceApiTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp=TemporaryDirectory(); os.environ['BOOTSTRAP_ADMIN_EMAIL']='owner@example.test'; os.environ['BOOTSTRAP_ADMIN_PASSWORD']='development-password'
+        self.server=create_server('127.0.0.1',0,self.tmp.name+'/x.db'); self.thread=threading.Thread(target=self.server.serve_forever,daemon=True); self.thread.start(); self.c=HTTPConnection('127.0.0.1',self.server.server_port); self.cookie=None
+        self.req('POST','/api/v1/auth/login',{'email':'owner@example.test','password':'development-password'})
+    def tearDown(self): self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
+    def req(self,m,p,x=None,cookie=True):
+        body=json.dumps(x).encode() if x is not None else None; h={'Content-Type':'application/json'} if body else {};
+        if cookie and self.cookie:h['Cookie']=self.cookie
+        self.c.request(m,p,body,h); r=self.c.getresponse(); sc=r.getheader('Set-Cookie');
+        if sc:self.cookie=sc.split(';',1)[0]
+        raw=r.read(); return r.status,json.loads(raw or b'{}')
+    def test_source_lifecycle_ingestion_idempotency_health_and_disabled(self):
+        s, source=self.req('POST','/api/v1/sources',{'name':'Import','kind':'manual','config':{}}); self.assertEqual(s,201)
+        sid=source['id']; self.assertFalse(source['enabled'])
+        self.assertEqual(self.req('PATCH',f'/api/v1/sources/{sid}',{'enabled':True})[0],200)
+        payload={'rows':[{'name':'Acme','website':'https://acme.test'}], 'source_url':'file://import.csv','query_context':{'q':'test'}}
+        self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/ingest',payload)[0],201)
+        self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/ingest',payload)[0],200)
+        self.assertEqual(len(self.req('GET','/api/v1/source-records')[1]['items']),1)
+        self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/test')[0],200)
+        self.assertEqual(self.req('PATCH',f'/api/v1/sources/{sid}',{'enabled':False})[0],200)
+        self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/ingest',payload)[0],409)
+        db=sqlite3.connect(self.tmp.name+'/x.db'); self.assertTrue(db.execute("select 1 from audit_log where action='source.disabled'").fetchone()); db.close()
+    def test_queries_enqueue_and_records_are_tenant_scoped(self):
+        _,source=self.req('POST','/api/v1/sources',{'name':'CSV','kind':'csv','enabled':True})
+        _,q=self.req('POST','/api/v1/discovery-queries',{'source_id':source['id'],'name':'q','query':{'csv':'name\nA'}})
+        status,job=self.req('POST',f"/api/v1/discovery-queries/{q['id']}/run",{})
+        self.assertEqual(status,202); self.assertEqual(job['type'],'source_discovery')
+        self.assertEqual(self.req('GET','/api/v1/source-records?page_size=101')[0],400)
+    def test_sources_require_auth(self):
+        self.cookie=None; self.assertEqual(self.req('GET','/api/v1/sources',cookie=False)[0],401)
+
+if __name__=='__main__': unittest.main()

apps/web/README.md

@@ -1,6 +1,6 @@
-# ProspectOS web — Phase 4 boundary
+# ProspectOS web — Phase 5 boundary
 
-Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor; it does not discover prospects, scan DNS/websites, or send outreach.
+Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach.
 
 ## Configure and run
 
@@ -24,6 +24,12 @@ If not set, the UI uses `localStorage.prospect_api_base` when present; otherwise
 
 The API remains the source of truth for tenant isolation, pagination bounds, filters, pipeline transitions, notes, audit records, and suppression. See `apps/api/README.md` for the route contract.
 
+## Phase 5 source UI contract
+
+The web client may display registered source metadata, query mode, approval/terms state, rate-limit status, retention class, health, and circuit state returned by the API. It must label `dry_run` as a plan/validation result and distinguish operator-supplied CSV/manual references from independently verified evidence. It must not offer a live-source control unless the API reports explicit approval and operational enablement; client visibility is never an authorization control.
+
+CSV and manual reference workflows must show source attribution, adapter/version, observed time, and any retention/redaction status. Raw source payloads should be hidden or minimized in the UI and remain tenant-scoped. A circuit-open or rate-limited source must be presented as unavailable/deferred, not as an empty discovery result. The current static client has no network discovery implementation; these are display and contract requirements for a future approved integration.
+
 ## Phase 4 job/live-log UI contract
 
 A future job view should show `queued`, `running`, `succeeded`, `failed`, or `cancelled`, the current attempt, timestamps, safe error text, and a clear terminal state. It should display persisted events in sequence order, resume from the last cursor after refresh/reconnect, and tolerate duplicate events. Create/retry requests should send an idempotency key and show the returned job identity rather than starting duplicate work.

apps/web/app.js

@@ -63,13 +63,25 @@
   async function jobAction(action) { const job = jobs.find(item => String(item.id) === String(selectedJobId)); if (!job || !canManageJobs()) return; const endpointPath = action === 'cancel' ? `/api/v1/jobs/${encodeURIComponent(job.id)}/cancel` : `/api/v1/jobs/${encodeURIComponent(job.id)}/retry`; const label = action === 'cancel' ? 'cancel' : 'retry'; try { await jobsRequest(endpointPath, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({})}); jobMessage(`Job ${label} requested.`); await loadJobs({silent:true}); await loadJobDetail(job.id); } catch (error) { if (error.message !== 'unauthorized') jobMessage(error.message || `Unable to ${label} job.`, true); } }
   function updateJobPermissions() { if ($('startDemoJobBtn')) $('startDemoJobBtn').disabled = !canManageJobs(); }
 
+  let sources = [], selectedSourceId = null;
+  const sourceState = source => Boolean(source?.enabled ?? source?.active);
+  const sourceStatus = source => sourceState(source) ? 'enabled' : 'disabled';
+  const sourceItems = payload => Array.isArray(payload) ? payload : (payload?.sources || payload?.items || payload?.records || []);
+  function sourceMessage(text, error = false) { const el = $('sourcesMessage'); if (el) { el.textContent = text || ''; el.className = `sources-message${error ? ' error' : ''}`; } }
+  function renderSourceSelect() { const select = $('discoverySource'); if (!select) return; select.innerHTML = `<option value="">Select a source</option>${sources.map(s => `<option value="${esc(s.id)}">${esc(s.name || s.label || `Source ${s.id}`)} · ${sourceStatus(s)}</option>`).join('')}`; }
+  function renderSources() { renderSourceSelect(); const list = $('sourcesList'); if (!sources.length) { list.innerHTML = '<div class="source-empty">No sources returned by the workspace.</div>'; return; } list.innerHTML = sources.map(source => { const status = sourceStatus(source), health = source.health || source.health_status || 'Not tested', terms = source.terms_reviewed ?? source.terms_status ?? 'Not reviewed', config = source.config || {}; return `<article class="source-row" data-source-id="${esc(source.id)}"><div class="source-row-main"><strong>${esc(source.name || source.label || `Source ${source.id}`)}</strong><small>${esc(source.kind || source.source_type || source.type || 'manual')}${source.url || config.url ? ` · ${esc(source.url || config.url)}` : ''}</small></div><span class="source-status ${status}">${status}</span><dl class="source-facts"><div><dt>Owner</dt><dd>${esc(source.owner || source.owner_name || config.owner || 'Not assigned')}</dd></div><div><dt>Terms</dt><dd>${esc(String(source.terms_reviewed ?? source.terms_status ?? (config.terms_url ? 'Provided' : 'Not reviewed')))}</dd></div><div><dt>Rate limit</dt><dd>${esc(source.rate_limit || source.rate_limit_label || config.rate_limit || 'Not set')}</dd></div><div><dt>Health</dt><dd>${esc(String(health))}</dd></div></dl><div class="source-actions"><button class="button ghost compact" type="button" data-source-action="test" data-source-id="${esc(source.id)}">Test</button><button class="button ${status === 'enabled' ? 'danger' : 'primary'} compact" type="button" data-source-action="toggle" data-source-id="${esc(source.id)}">${status === 'enabled' ? 'Disable' : 'Enable'}</button></div></article>`; }).join(''); }
+  function renderSourceRecords(items) { const list = $('sourceRecordsList'); if (!items.length) { list.innerHTML = '<div class="source-empty">No source records returned by the workspace.</div>'; return; } list.innerHTML = `<div class="source-record-table"><table><thead><tr><th>Record</th><th>Source</th><th>Status</th><th>Observed</th></tr></thead><tbody>${items.slice(0,25).map(record => `<tr><td>${esc(record.name || record.title || record.external_id || record.id || 'Unnamed record')}</td><td>${esc(record.source_name || record.source || 'Unknown source')}</td><td><span class="status">${esc(record.status || 'Pending')}</span></td><td>${esc(record.observed_at || record.created_at || 'Time unavailable')}</td></tr>`).join('')}</tbody></table></div>`; }
+  async function loadSources() { sourceMessage('Loading sources…'); $('sourcesList').innerHTML = '<div class="detail-loading" aria-live="polite">Loading source registry…</div>'; $('sourceRecordsList').innerHTML = '<div class="detail-loading" aria-live="polite">Loading source records…</div>'; try { const [sourcePayload, recordPayload] = await Promise.all([jsonRequest('/api/v1/sources'), jsonRequest('/api/v1/source-records?page_size=25')]); sources = sourceItems(sourcePayload); renderSources(); renderSourceRecords(sourceItems(recordPayload)); $('sourcesUpdatedAt').textContent = `Updated ${new Date().toLocaleTimeString([], {hour:'2-digit', minute:'2-digit'})}`; sourceMessage(sources.some(sourceState) ? '' : 'No live source is enabled.'); } catch (error) { sources = []; renderSources(); renderSourceRecords([]); if (error.message !== 'unauthorized') sourceMessage(error.message || 'Unable to load sources.', true); } }
+  async function saveSource(event) { event.preventDefault(); const form = event.currentTarget, fields = Object.fromEntries(new FormData(form).entries()); if (fields.source_type === 'csv' && !fields.csv_content.trim()) { message('sourceFormMessage', 'CSV content is required for a CSV source.', true); return; } const config = {url:fields.url, terms_url:fields.terms_url, owner:fields.owner, rate_limit:fields.rate_limit}; if (fields.source_type === 'csv') config.csv = fields.csv_content; else config.rows = []; try { await jsonRequest('/api/v1/sources', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({name:fields.name, kind:fields.source_type, config, enabled:false})}); message('sourceFormMessage', 'Source saved. It remains disabled until explicitly enabled.'); form.reset(); $('sourceCsvField').hidden = true; await loadSources(); } catch (error) { if (error.message !== 'unauthorized') message('sourceFormMessage', error.message || 'Unable to save source.', true); } }
+  async function sourceAction(id, action) { const source = sources.find(item => String(item.id) === String(id)); if (!source) return; try { if (action === 'test') { await jsonRequest(`/api/v1/sources/${encodeURIComponent(id)}/test`, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({})}); sourceMessage('Source test completed.'); } else { const enabled = sourceState(source); await jsonRequest(`/api/v1/sources/${encodeURIComponent(id)}`, {method:'PATCH', headers:{'Content-Type':'application/json'}, body:JSON.stringify({enabled:!enabled})}); sourceMessage(`Source ${enabled ? 'disabled' : 'enabled'} by the workspace.`); } await loadSources(); } catch (error) { if (error.message !== 'unauthorized') sourceMessage(error.message || `Unable to ${action} source.`, true); } }
+  async function runDiscovery(dryRun) { const form = $('discoveryForm'), data = Object.fromEntries(new FormData(form).entries()); data.dry_run = Boolean(dryRun); if (!data.source_id || !data.query.trim()) { message('discoveryMessage', 'Select a source and enter a query.', true); return; } const source = sources.find(item => String(item.id) === String(data.source_id)); if (!dryRun && !sourceState(source)) { message('discoveryMessage', 'This source is disabled. Enable it only after review.', true); return; } const button = dryRun ? $('discoveryDryRunBtn') : $('discoveryRunBtn'); button.disabled = true; message('discoveryMessage', dryRun ? 'Validating query…' : 'Starting discovery…'); try { const query = await jsonRequest('/api/v1/discovery-queries', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({source_id:Number(data.source_id), name:data.query.trim().slice(0,80), query:{text:data.query.trim()}, dry_run:data.dry_run})}); if (!dryRun) await jsonRequest(`/api/v1/discovery-queries/${encodeURIComponent(query.id)}/run`, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({})}); message('discoveryMessage', dryRun ? 'Dry run completed; no discovery job was started.' : 'Discovery request accepted. Check Jobs for progress.'); } catch (error) { if (error.message !== 'unauthorized') message('discoveryMessage', error.message || 'Discovery request failed.', true); } finally { button.disabled = false; } }
   function parseCsv(text){const lines=text.trim().split(/\r?\n/).filter(Boolean),cells=line=>line.match(/("[^"]*(?:""[^"]*)*"|[^,]+)(?=,|$)/g)?.map(x=>x.replace(/^"|"$/g,'').replaceAll('""','"'))||[];if(!lines.length)return[];const headers=cells(lines[0]);return lines.slice(1,11).map(l=>Object.fromEntries(cells(l).map((v,i)=>[headers[i]||`column_${i+1}`,v])));}
   function renderCsv(rows){if(!rows.length){$('csvPreview').innerHTML='<span>⊞</span><p>No data rows found</p>';return;}const h=Object.keys(rows[0]);$('csvPreview').className='csv-table';$('csvPreview').innerHTML=`<table><thead><tr>${h.map(x=>`<th>${esc(x)}</th>`).join('')}</tr></thead><tbody>${rows.map(r=>`<tr>${h.map(x=>`<td>${esc(r[x])}</td>`).join('')}</tr>`).join('')}</tbody></table><small class="muted">Showing up to 10 rows · Preview only; nothing added yet.</small>`;}
   async function login(event){event.preventDefault();const form=event.currentTarget,messageEl=$('loginMessage'),data=Object.fromEntries(new FormData(form).entries());messageEl.textContent='Signing in…';messageEl.className='form-message';try{const res=await fetch(endpoint('/api/v1/auth/login'),{method:'POST',headers:{'Content-Type':'application/json'},credentials:'include',body:JSON.stringify(data)});const body=await res.json().catch(()=>({}));if(!res.ok)throw new Error(body.error||'Invalid email or password.');await bootstrap();}catch(e){if(e.message!=='unauthorized'){messageEl.textContent=e.message;messageEl.className='form-message error';}}}
   async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}}
-  async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
+  async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
   document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);});
   document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
-  $('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
+  $('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
   bootstrap();
 })();

apps/web/index.html

@@ -31,6 +31,7 @@
         <a class="nav-item" href="#explorer"><span>⌕</span> Prospect explorer</a>
         <a class="nav-item" href="#add"><span>＋</span> Add prospects</a>
         <a class="nav-item" href="#jobs" data-nav="jobs"><span>◷</span> Jobs</a>
+        <a class="nav-item" href="#sources" data-nav="sources"><span>⌁</span> Sources</a>
       </nav>
       <div class="sidebar-foot"><div class="live-dot"></div><div><strong>Workspace live</strong><small>Data sync is healthy</small></div></div>
     </aside>
@@ -66,6 +67,17 @@
             <aside class="job-detail panel" id="jobDetailPanel"><div class="empty-detail"><span class="empty-icon">◷</span><h3>Select a job</h3><p>Inspect progress, structured errors, and the event timeline.</p></div></aside>
           </div>
         </section>
+        <section class="sources-section" id="sources" aria-labelledby="sourcesTitle">
+          <div class="sources-header panel"><div><p class="eyebrow">GOVERNANCE</p><h2 id="sourcesTitle">Sources</h2><p class="muted">Review source ownership, terms, limits, and health before using discovery.</p></div><button class="button ghost" id="sourcesRefreshBtn" type="button">↻ Refresh</button></div>
+          <div class="source-safety" role="note"><strong>Discovery is disabled by default.</strong> No live source is enabled in this workspace. Enable a source only after its terms, owner, rate limit, and health have been reviewed.</div>
+          <div id="sourcesMessage" class="sources-message" role="status" aria-live="polite"></div>
+          <div class="sources-grid">
+            <article class="panel source-config-panel"><div class="panel-heading"><div><p class="eyebrow">CONFIGURATION</p><h3>Add a source</h3></div><span class="small-label">Manual or CSV</span></div><form id="sourceForm"><div class="form-grid"><label>Source name<input name="name" required placeholder="Public business directory"></label><label>Source type<select name="source_type" id="sourceType"><option value="manual">Manual / API</option><option value="csv">CSV upload</option></select></label><label>Source URL <span class="optional">optional</span><input name="url" type="url" placeholder="https://…"></label><label>Terms URL <span class="optional">required for enablement</span><input name="terms_url" type="url" placeholder="https://…/terms"></label><label>Owner<input name="owner" required placeholder="Team or accountable person"></label><label>Rate limit<input name="rate_limit" required placeholder="e.g. 60 requests/hour"></label></div><label class="source-csv-field" id="sourceCsvField" hidden>CSV content<textarea name="csv_content" id="sourceCsvContent" rows="4" placeholder="Paste CSV content; it is sent only when you save this source."></textarea></label><div class="form-footer"><p id="sourceFormMessage" class="form-message" role="status"></p><button class="button primary" type="submit">Save source</button></div></form></article>
+            <article class="panel discovery-panel"><div class="panel-heading"><div><p class="eyebrow">DISCOVERY</p><h3>Query a source</h3></div><span class="small-label">No automatic runs</span></div><form id="discoveryForm"><label>Source<select name="source_id" id="discoverySource" required><option value="">Select a source</option></select></label><label>Query<input name="query" required placeholder="e.g. renewable energy firms in Cape Town"></label><label class="checkbox-label"><input type="checkbox" name="dry_run" id="discoveryDryRun" checked> Dry run (preview only)</label><div class="form-footer"><p id="discoveryMessage" class="form-message" role="status"></p><div class="discovery-actions"><button class="button ghost" id="discoveryDryRunBtn" type="submit">Validate query</button><button class="button primary" id="discoveryRunBtn" type="button">Run discovery</button></div></div></form></article>
+          </div>
+          <div class="sources-list panel"><div class="panel-heading"><div><p class="eyebrow">REGISTRY</p><h3>Configured sources</h3></div><span id="sourcesUpdatedAt" class="small-label">Not loaded</span></div><div id="sourcesList" class="sources-list-body"><div class="source-empty">Sign in to load sources from the workspace.</div></div></div>
+          <div class="source-records panel"><div class="panel-heading"><div><p class="eyebrow">RECENT OUTPUT</p><h3>Recent source records</h3></div></div><div id="sourceRecordsList" class="source-records-body"><div class="source-empty">No records loaded.</div></div></div>
+        </section>
         <section class="lower-grid" id="add"><article class="panel add-panel"><div class="panel-heading"><div><p class="eyebrow">INTAKE</p><h2>Add a prospect</h2></div><span class="small-label">Manual entry</span></div><form id="addForm"><div class="form-grid"><label>Company name<input required name="name" placeholder="Acme Inc."></label><label>Website <span class="optional">optional</span><input name="website" type="url" placeholder="https://acme.com"></label><label>Location<input name="location" placeholder="Cape Town, ZA"></label><label>Notes <span class="optional">optional</span><input name="description" placeholder="Why this is a fit…"></label></div><div class="form-footer"><p id="formMessage" class="form-message" role="status"></p><button class="button primary" type="submit">Add to review queue</button></div></form></article>
           <article class="panel csv-panel"><div class="panel-heading"><div><p class="eyebrow">BULK INTAKE</p><h2>CSV preview</h2></div><label class="button ghost upload-label" for="csvInput">↑ Choose CSV</label><input id="csvInput" type="file" accept=".csv,text/csv" hidden></div><p class="muted">Preview rows before adding them to your review queue.</p><div id="csvPreview" class="csv-empty"><span>⊞</span><p>No file selected</p><small>CSV stays in your browser until you confirm.</small></div></article></section>
         <footer><span>ProspectOS</span><span>Evidence-led prospecting · <a href="#explorer">Review queue</a></span></footer>

apps/web/smoke-test.html

@@ -24,5 +24,11 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
  ,['Job status counts and controls',()=>['jobCountQueued','jobCountRunning','jobCountSucceeded','jobCountFailed','jobCountCancelled','startDemoJobBtn','jobsRefreshBtn'].every(id=>!!d.querySelector('#'+id))]
  ,['Authenticated jobs API contract',()=>js.includes("/api/v1/jobs")&&js.includes('jobsRequest')&&js.includes('/cancel')&&js.includes('/retry')]
  ,['Job detail timeline and structured states',()=>!!d.querySelector('#jobDetailPanel')&&js.includes('event_timeline')&&js.includes('structured_error')&&js.includes('progress')]
+ ,['Sources navigation and safety boundary',()=>!!d.querySelector('[data-nav="sources"]')&&!!d.querySelector('#sources')&&js.includes('No live source is enabled')]
+ ,['Source registry status and governance fields',()=>!!d.querySelector('#sourcesList')&&['owner','terms','rate_limit','health'].every(x=>js.includes(x))&&js.includes('/api/v1/sources')]
+ ,['Source configuration and controls use authenticated helper',()=>!!d.querySelector('#sourceForm')&&!!d.querySelector('#sourceType')&&js.includes('saveSource')&&js.includes('/test')&&js.includes('method:\'PATCH\'')&&js.includes('jsonRequest')]
+ ,['Discovery dry-run and run controls',()=>!!d.querySelector('#discoveryForm')&&!!d.querySelector('#discoveryDryRunBtn')&&!!d.querySelector('#discoveryRunBtn')&&js.includes('/api/v1/sources/discovery')&&js.includes('dry_run')]
+ ,['Recent source records and error/loading states',()=>!!d.querySelector('#sourceRecordsList')&&js.includes('source-record-table')&&js.includes('Loading source registry')&&js.includes('Unable to load sources')]
+ ,['No live source enabled copy is explicit',()=>d.querySelector('#sources')?.textContent.includes('No live source is enabled')&&!js.includes('demoSources')]
 ];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
 </script>

docs/OPERATIONS.md

@@ -31,6 +31,16 @@ Phase 3 is a human-operated prospect workflow. Operators manually create a busin
 
 There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.
 
+## Phase 5 source operations boundary
+
+Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.
+
+Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.
+
+`dry_run` must perform validation/planning only: no network I/O, external adapter side effects, prospect-fact writes, or outreach. CSV and manual references may be previewed or recorded as operator-supplied observations with source attribution and capture time. A preview is not an import, verification, or discovery result.
+
+Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.
+
 ## Phase 4 jobs and live logging
 
 The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.

docs/SECURITY.md

@@ -3,6 +3,10 @@
 ## Current safety boundary
 
 - **Automated outreach is disabled.** The Compose file sets `AUTOMATED_OUTREACH_ENABLED=false` for both services. The MVP sends no email, SMS, or other outbound communication.
+- Phase 5 source handling is **registered, approval-gated, and fail-closed**. The current runtime has no live network source or network discovery implementation. CSV and manual reference adapters are operator-supplied only; neither proves a fact or grants contact permission.
+- Source registry entries must have an accountable terms owner, permitted purpose, approval status/expiry, rate/concurrency limits, retention class, and health/circuit policy before an adapter can be enabled.
+- Discovery queries must be tenant-scoped, bounded, auditable, and explicit about execution mode. `dry_run` validates/plans without contacting a source or writing prospect facts. No live source may run without recorded product/legal/security approval and explicit operational enablement.
+- Raw source records are sensitive lineage data: retain only the minimum needed to reproduce a normalized result, under the approved retention class, with tenant/source/query IDs, capture time, adapter version, and redaction metadata. Never store secrets or unnecessary personal/contact data.
 - Phase 3 intelligence is **manual and provenance-first**. Operators enter child intelligence/evidence records; the platform does not perform automated prospect discovery, DNS resolution, website/HTTP scanning, or external enrichment.
 - Every business, child record, note, pipeline transition, and audit/activity read or write must be constrained to the authenticated user's organization. A child identifier must never bypass the parent/tenant check. Cross-tenant misses should be indistinguishable from an absent record.
 - Evidence provenance (source reference/label, captured or observed time, actor, and confidence where supported) is data lineage, not proof that the platform independently verified the source. Do not fabricate provenance or silently upgrade an observation to a verified fact.
@@ -14,6 +18,19 @@
 - Phase 4 job/live-log controls are not enabled in the current runtime. If added, job IDs, idempotency keys, status, cancellation, retries, and event cursors must all be authorized against the authenticated organization; never accept a job or child identifier as authorization by itself.
 - Persisted job events must be append-only, sequence-ordered per job, replayable from a cursor, and redacted to safe operational data. Never emit credentials, session cookies, API keys, full request bodies, or unnecessary contact/prospect data in polling responses, SSE frames, logs, or error details.
 
+## Phase 5 source security controls
+
+Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
+
+- `dry_run` is the safe default: validate a bounded discovery query and produce a plan without network I/O, adapter side effects, or prospect-fact writes.
+- CSV and manual reference adapters accept operator-supplied material only. Preserve citation/reference, actor, capture time, adapter/version, and normalization lineage; label it as supplied/observed rather than verified.
+- Raw source records must be access-controlled, tenant-scoped, minimally retained, immutable enough for replay/audit, and redacted for secrets and unnecessary personal data. Apply the approved retention class and deletion schedule.
+- Enforce per-source request, concurrency, byte, timeout, and retry limits. Rate-limit responses must not be bypassed by rotating identities or silently selecting another source.
+- Health controls must record success/failure/latency signals and use a circuit breaker with `closed`, `open`, and guarded `half-open` states. Open circuits fail closed, suppress live attempts, and surface a safe deferred/unavailable outcome.
+- No live network source is allowed without explicit product, legal, and security approval plus operational enablement of the registered adapter. Approval must be checked at execution time and expire safely.
+
+If a future approved adapter fetches URLs, apply the SSRF requirements below in addition to source approval. Network discovery is not implemented by this documentation or by the current Compose stack.
+
 ## Known limitations before production
 
 1. **Password storage:** production passwords must be hashed with Argon2id using a reviewed cost/memory/parallelism policy. Never store plaintext or reversible passwords, and never log bootstrap credentials. Rehash on login when the policy changes.