add policy-aware source adapter framework
This commit is contained in:
Marco0300
+20
-2
@@ -1,6 +1,6 @@
|
||||
# Prospect Platform API — Phase 4 boundary
|
||||
# Prospect Platform API — Phase 5 boundary
|
||||
|
||||
Dependency-light JSON API for tenant-scoped, **manual** prospect workflows. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context, with the Phase 4 job/live-log contract described below. It never performs automated discovery, DNS/website scanning, or outreach.
|
||||
Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 5 source-ingestion contract. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
|
||||
|
||||
## Run
|
||||
|
||||
@@ -34,6 +34,24 @@ The MVP job routes are `POST /api/v1/jobs`, `GET /api/v1/jobs`, `GET /api/v1/job
|
||||
|
||||
The current runtime has no durable job queue and no Redis/Celery integration. Its in-process/SQLite worker is single-instance, non-durable across process loss, and unsuitable for horizontal scaling or guaranteed execution; it is pilot-only.
|
||||
|
||||
### Phase 5 source adapter contract
|
||||
|
||||
Phase 5 treats a source as a registered, reviewable capability rather than an arbitrary URL or scraper. Every adapter contract must identify:
|
||||
|
||||
- a stable adapter/source ID and version, owner, permitted purpose, and terms/robots contact;
|
||||
- accepted discovery-query fields, result schema, provenance fields, and validation/error behavior;
|
||||
- request and concurrency rate limits, retry/backoff rules, timeout/size bounds, and a retention class for raw source records;
|
||||
- health signals and circuit-breaker states (`closed`, `open`, `half-open`), including fail-closed behavior when unhealthy; and
|
||||
- an explicit `dry_run` mode that validates and plans work without contacting a source or writing prospect facts.
|
||||
|
||||
The initial safe adapters are `csv` and `manual_reference`. CSV input may be parsed and previewed; a manual reference records operator-supplied source identity, citation/reference, observed value, and timestamp. Neither adapter independently verifies a source or authorizes outreach. Raw source records should be retained immutably enough to reproduce the normalized result, with tenant/source/query identifiers, capture time, adapter version, and redaction/retention metadata; never retain secrets or unnecessary personal data.
|
||||
|
||||
A source registry entry must include its terms owner, approval status/expiry, allowed tenants or scopes, rate-limit policy, retention class, and health/circuit policy. A discovery query is bounded and tenant-scoped, and its execution mode must be explicit (`dry_run` by default). A live network source is not implemented and must be rejected unless product, legal, and security approval is recorded and operations explicitly enables the registered adapter. No query, job acceptance, or successful parse may be described as network discovery.
|
||||
|
||||
### Phase 5 source controls (contract, not current live routes)
|
||||
|
||||
Implementations should expose source/query/job state without leaking raw payloads across tenants, including source approval, terms, rate-limit, retention, health, and circuit-open reason. On rate-limit, terms, approval, or circuit failure, return a safe non-live outcome and preserve an audit event; do not silently retry against another source. Dry-run must be side-effect-free with respect to external sources and prospect facts. These are Phase 5 design requirements; the current MVP has no network adapter or discovery endpoint.
|
||||
|
||||
### Health and workspace
|
||||
|
||||
- `GET /api/v1/health/live` — unauthenticated liveness check.
|
||||
|
||||
+80
-2
@@ -8,14 +8,16 @@ from urllib.parse import parse_qs, urlparse
|
||||
if __package__ in (None, ""):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone
|
||||
from app.sources import adapter_for, contains_secret
|
||||
else:
|
||||
from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone
|
||||
from .sources import adapter_for, contains_secret
|
||||
ORGANIZATION_ID = "demo-tenant"
|
||||
SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
|
||||
SESSION_DAYS = 7
|
||||
PBKDF2_ITERATIONS = 300_000
|
||||
MUTATING_ROLES = {"owner", "admin", "researcher"}
|
||||
JOB_TYPES = {"noop", "prospect_recalculate"}
|
||||
JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery"}
|
||||
JOB_PAGE_SIZE = 100
|
||||
SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
|
||||
CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
|
||||
@@ -110,6 +112,9 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/dashboard/summary":
|
||||
row=db.execute("SELECT COUNT(*) businesses,COALESCE(AVG(score),0) average_score FROM businesses WHERE organization_id=?",(org,)).fetchone(); return self.send_json(200,{"organization_id":org,"businesses":row["businesses"],"average_score":round(row["average_score"],2),"suppressed":db.execute("SELECT COUNT(*) FROM suppressions WHERE organization_id=?",(org,)).fetchone()[0]})
|
||||
if path=="/api/v1/businesses": return self.list_businesses(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/sources": return self.list_sources(db,org)
|
||||
if path=="/api/v1/discovery-queries": return self.list_queries(db,org)
|
||||
if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/businesses/"):
|
||||
@@ -153,7 +158,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
try:
|
||||
cur=db.execute("INSERT INTO jobs(organization_id,idempotency_key,type,payload,max_attempts) VALUES(?,?,?,?,?)",(user["organization_id"],key,kind,safe,max_attempts)); jid=cur.lastrowid
|
||||
self.add_job_event(db,jid,user["organization_id"],"queued","Job queued",0); self.audit(db,user,"job.created",str(jid)); db.commit(); getattr(self.server,"job_wakeup",threading.Event()).set()
|
||||
return self.send_json(201,job_json(db.execute("SELECT * FROM jobs WHERE id=?",(jid,)).fetchone()))
|
||||
return self.send_json(202 if payload.get("_accepted") else 201,job_json(db.execute("SELECT * FROM jobs WHERE id=?",(jid,)).fetchone()))
|
||||
except sqlite3.IntegrityError:
|
||||
row=db.execute("SELECT * FROM jobs WHERE organization_id=? AND idempotency_key=?",(user["organization_id"],key)).fetchone(); return self.send_json(200,job_json(row))
|
||||
|
||||
@@ -210,9 +215,14 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path.startswith("/api/v1/jobs/"):
|
||||
return self.job_action(db,user,path)
|
||||
if path=="/api/v1/businesses":return self.create_business(payload,db,user)
|
||||
if path=="/api/v1/sources":return self.create_source(payload,db,user)
|
||||
if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
|
||||
if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
|
||||
if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
|
||||
bits=path.split("/")
|
||||
if len(bits)==6 and bits[3] == "sources" and bits[4].isdigit() and bits[5] in {"test","ingest"}:
|
||||
return self.test_source(int(bits[4]),db,user) if bits[5]=="test" else self.ingest_source(int(bits[4]),payload,db,user)
|
||||
if len(bits)==6 and bits[3] == "discovery-queries" and bits[4].isdigit() and bits[5]=="run": return self.run_query(int(bits[4]),db,user)
|
||||
if len(bits)==7 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES and bits[6]=="": pass
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES:return self.create_child(int(bits[4]) if bits[4].isdigit() else -1,bits[5],payload,db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="verify":return self.verify_business(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user)
|
||||
@@ -225,6 +235,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if not user:return
|
||||
if user["role"] not in MUTATING_ROLES:return self.send_json(403,{"error":"forbidden"})
|
||||
bits=path.split("/")
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","sources"] and bits[4].isdigit(): return self.update_source(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="pipeline":return self.update_pipeline(int(bits[4]) if bits[4].isdigit() else -1,self.read_json(),db,user)
|
||||
return self.send_json(404,{"error":"not_found"})
|
||||
finally:db.close()
|
||||
@@ -286,6 +297,73 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
elif key in existing_keys or key in seen:continue
|
||||
else:seen.add(key);accepted.append(b)
|
||||
return self.send_json(200,{"accepted":len(accepted),"duplicates":len(rows)-len(normalized)+len(normalized)-len(accepted)-suppressed,"suppressed":suppressed,"rows":accepted})
|
||||
def list_sources(self,db,org):
|
||||
cols='id,organization_id,name,kind,enabled,health_status,consecutive_failures,circuit_open,last_success_at,last_failure_at,last_error,created_at,updated_at'
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in db.execute(f"SELECT {cols} FROM sources WHERE organization_id=? ORDER BY id",(org,))]})
|
||||
def list_queries(self,db,org):
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in db.execute("SELECT * FROM discovery_queries WHERE organization_id=? ORDER BY id",(org,))]})
|
||||
def list_source_records(self,db,org,q):
|
||||
try:
|
||||
limit=int(q.get('page_size',[50])[0]); offset=max(0,int(q.get('offset',[0])[0]))
|
||||
if limit<1 or limit>100: raise ValueError
|
||||
except (ValueError,TypeError): return self.send_json(400,{"error":"invalid_pagination"})
|
||||
rows=db.execute("SELECT * FROM source_records WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall(); out=[]
|
||||
for r in rows[:limit]:
|
||||
x=row_json(r)
|
||||
for k in ('raw_json','normalized_json','query_context_json','cursor_json','rate_policy_json'):
|
||||
try:x[k]=json.loads(x[k])
|
||||
except (ValueError,TypeError):pass
|
||||
out.append(x)
|
||||
return self.send_json(200,{"organization_id":org,"items":out,"limit":limit,"offset":offset,"has_more":len(rows)>limit})
|
||||
def create_source(self,payload,db,user):
|
||||
name=str(payload.get('name','')).strip(); kind=str(payload.get('kind','')).strip().lower(); config=payload.get('config',{})
|
||||
if not name or kind not in ('csv','manual') or not isinstance(config,dict):return self.send_json(400,{"error":"invalid_source"})
|
||||
if contains_secret(config):return self.send_json(400,{"error":"secret_not_permitted"})
|
||||
try:
|
||||
validation=adapter_for(kind).validate(config)
|
||||
if config and not validation.valid:return self.send_json(400,{"error":"invalid_source_config","details":validation.errors})
|
||||
cur=db.execute("INSERT INTO sources(organization_id,name,kind,enabled,config_json) VALUES(?,?,?,?,?)",(user['organization_id'],name,kind,int(bool(payload.get('enabled',False))),json.dumps(config,sort_keys=True)))
|
||||
except sqlite3.IntegrityError:return self.send_json(409,{"error":"duplicate_source"})
|
||||
self.audit(db,user,'source.created',str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT id,organization_id,name,kind,enabled,health_status,consecutive_failures,circuit_open,last_success_at,last_failure_at,last_error,created_at,updated_at FROM sources WHERE id=?",(cur.lastrowid,)).fetchone()))
|
||||
def update_source(self,sid,payload,db,user):
|
||||
if not db.execute("SELECT id FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone():return self.send_json(404,{"error":"not_found"})
|
||||
if 'enabled' not in payload:return self.send_json(400,{"error":"enabled_required"})
|
||||
value=int(bool(payload['enabled']));db.execute("UPDATE sources SET enabled=?,updated_at=CURRENT_TIMESTAMP WHERE id=?",(value,sid));self.audit(db,user,'source.enabled' if value else 'source.disabled',str(sid));db.commit();return self.send_json(200,row_json(db.execute("SELECT * FROM sources WHERE id=?",(sid,)).fetchone()))
|
||||
def create_query(self,payload,db,user):
|
||||
sid=payload.get('source_id');name=str(payload.get('name','')).strip();query=payload.get('query',{})
|
||||
if not isinstance(sid,int) or not name or not isinstance(query,dict) or contains_secret(query):return self.send_json(400,{"error":"invalid_query"})
|
||||
if not db.execute("SELECT id FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone():return self.send_json(404,{"error":"not_found"})
|
||||
try:cur=db.execute("INSERT INTO discovery_queries(organization_id,source_id,name,query_json) VALUES(?,?,?,?)",(user['organization_id'],sid,name,json.dumps(query,sort_keys=True)))
|
||||
except sqlite3.IntegrityError:return self.send_json(409,{"error":"duplicate_query"})
|
||||
self.audit(db,user,'discovery_query.created',str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT * FROM discovery_queries WHERE id=?",(cur.lastrowid,)).fetchone()))
|
||||
def run_query(self,qid,db,user):
|
||||
if not db.execute("SELECT id FROM discovery_queries WHERE id=? AND organization_id=?",(qid,user['organization_id'])).fetchone():return self.send_json(404,{"error":"not_found"})
|
||||
return self.create_job({"type":"source_discovery","_accepted":True,"payload":{"discovery_query_id":qid},"idempotency_key":f"discovery-query-{qid}-{int(time.time())}"},db,user)
|
||||
def test_source(self,sid,db,user):
|
||||
source=db.execute("SELECT * FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone()
|
||||
if not source:return self.send_json(404,{"error":"not_found"})
|
||||
try: result=adapter_for(source['kind']).validate(json.loads(source['config_json'])); ok=result.valid; error='; '.join(result.errors) if not ok else None
|
||||
except Exception as exc:ok=False;error=str(exc)[:300]
|
||||
if ok:db.execute("UPDATE sources SET health_status='healthy',consecutive_failures=0,circuit_open=0,last_success_at=CURRENT_TIMESTAMP,last_error=NULL WHERE id=?",(sid,));action='source.test.succeeded'
|
||||
else:db.execute("UPDATE sources SET health_status='unhealthy',consecutive_failures=consecutive_failures+1,circuit_open=CASE WHEN consecutive_failures+1>=3 THEN 1 ELSE circuit_open END,last_failure_at=CURRENT_TIMESTAMP,last_error=? WHERE id=?",(error,sid));action='source.test.failed'
|
||||
self.audit(db,user,action,str(sid));db.commit();return self.send_json(200,{"ok":ok,"errors":[] if ok else [error]})
|
||||
def ingest_source(self,sid,payload,db,user):
|
||||
source=db.execute("SELECT * FROM sources WHERE id=? AND organization_id=?",(sid,user['organization_id'])).fetchone()
|
||||
if not source:return self.send_json(404,{"error":"not_found"})
|
||||
if not source['enabled'] or source['circuit_open']:return self.send_json(409,{"error":"source_disabled"})
|
||||
if contains_secret(payload):return self.send_json(400,{"error":"secret_not_permitted"})
|
||||
config={k:v for k,v in payload.items() if k not in ('source_url','query_context','cursor','rate_policy')}
|
||||
if len(json.dumps(config).encode())>5*1024*1024:return self.send_json(400,{"error":"ingest_limits"})
|
||||
if isinstance(config.get('rows'),list) and (len(config['rows'])>1000 or any(not isinstance(r,dict) or len(r)>50 or any(len(str(v))>10000 for v in r.values()) for r in config['rows'])):return self.send_json(400,{"error":"ingest_limits"})
|
||||
if isinstance(config.get('csv'),str) and config['csv'].count('\n')>1001:return self.send_json(400,{"error":"ingest_limits"})
|
||||
try:page=adapter_for(source['kind']).discover(config)
|
||||
except (ValueError,KeyError) as exc:return self.send_json(400,{"error":"invalid_ingest","detail":str(exc)})
|
||||
inserted=0
|
||||
for record in page.records[:1000]:
|
||||
raw=json.dumps(record,sort_keys=True,separators=(',',':'));digest=hashlib.sha256(raw.encode()).hexdigest()
|
||||
try:db.execute("INSERT INTO source_records(organization_id,source_id,content_hash,raw_json,normalized_json,source_url,query_context_json,cursor_json,rate_policy_json) VALUES(?,?,?,?,?,?,?,?,?)",(user['organization_id'],sid,digest,raw,raw,str(payload.get('source_url','')),json.dumps(payload.get('query_context',{}),sort_keys=True),json.dumps(payload.get('cursor',{}),sort_keys=True),json.dumps(payload.get('rate_policy',{}),sort_keys=True)));inserted+=1
|
||||
except sqlite3.IntegrityError:pass
|
||||
db.execute("UPDATE sources SET health_status='healthy',consecutive_failures=0,last_success_at=CURRENT_TIMESTAMP,last_error=NULL WHERE id=?",(sid,));self.audit(db,user,'source.ingested',f'{sid}:{inserted}');db.commit();return self.send_json(201 if inserted else 200,{"inserted":inserted,"records":len(page.records)})
|
||||
def log_message(self,*_):pass
|
||||
|
||||
def _job_worker(server):
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Deterministic, network-free discovery source contracts and adapters."""
|
||||
from __future__ import annotations
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Mapping, Protocol, Sequence
|
||||
import csv, io, re
|
||||
|
||||
SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "private_key", "credential"}
|
||||
|
||||
def contains_secret(value: Any, path: str = "") -> str | None:
|
||||
if isinstance(value, Mapping):
|
||||
for key, child in value.items():
|
||||
key_text = str(key).lower()
|
||||
if key_text in SECRET_KEYS or any(x in key_text for x in ("password", "token", "secret", "api_key", "private_key")):
|
||||
return path + str(key)
|
||||
found = contains_secret(child, path + str(key) + ".")
|
||||
if found: return found
|
||||
elif isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)):
|
||||
for index, child in enumerate(value):
|
||||
found = contains_secret(child, path + str(index) + ".")
|
||||
if found: return found
|
||||
return None
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ValidationResult:
|
||||
valid: bool
|
||||
errors: list[str] = field(default_factory=list)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DiscoveryPage:
|
||||
records: list[dict[str, str]]
|
||||
next_cursor: str | None = None
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SourceHealth:
|
||||
status: str = "unknown"
|
||||
consecutive_failures: int = 0
|
||||
circuit_open: bool = False
|
||||
last_error: str | None = None
|
||||
|
||||
class DiscoverySource(Protocol):
|
||||
kind: str
|
||||
def validate(self, config: Mapping[str, Any]) -> ValidationResult: ...
|
||||
def discover(self, config: Mapping[str, Any], cursor: str | None = None) -> DiscoveryPage: ...
|
||||
|
||||
_FIELDS = ("name", "website", "email", "phone", "description")
|
||||
|
||||
def normalize_record(row: Mapping[str, Any]) -> dict[str, str]:
|
||||
result = {field: str(row.get(field, "")).strip() for field in _FIELDS}
|
||||
# Accept common CSV spellings without retaining arbitrary sensitive fields.
|
||||
aliases = {"company": "name", "url": "website", "domain": "website"}
|
||||
for key, target in aliases.items():
|
||||
if not result[target] and row.get(key) is not None: result[target] = str(row[key]).strip()
|
||||
return result
|
||||
|
||||
class ManualSource:
|
||||
kind = "manual"
|
||||
def validate(self, config: Mapping[str, Any]) -> ValidationResult:
|
||||
if not isinstance(config, Mapping): return ValidationResult(False, ["config must be an object"])
|
||||
found = contains_secret(config)
|
||||
if found: return ValidationResult(False, [f"secret field is not permitted: {found}"])
|
||||
rows = config.get("rows")
|
||||
if not isinstance(rows, list): return ValidationResult(False, ["rows must be a list"])
|
||||
return ValidationResult(True)
|
||||
def discover(self, config: Mapping[str, Any], cursor: str | None = None) -> DiscoveryPage:
|
||||
validation = self.validate(config)
|
||||
if not validation.valid: raise ValueError(validation.errors[0])
|
||||
rows = [normalize_record(row) for row in config["rows"] if isinstance(row, Mapping)]
|
||||
return DiscoveryPage(rows, None, {"adapter": self.kind})
|
||||
|
||||
class CsvSource:
|
||||
kind = "csv"
|
||||
def validate(self, config: Mapping[str, Any]) -> ValidationResult:
|
||||
if not isinstance(config, Mapping): return ValidationResult(False, ["config must be an object"])
|
||||
found = contains_secret(config)
|
||||
if found: return ValidationResult(False, [f"secret field is not permitted: {found}"])
|
||||
if not isinstance(config.get("csv"), str): return ValidationResult(False, ["csv must be text"])
|
||||
try:
|
||||
reader = csv.DictReader(io.StringIO(config["csv"]));
|
||||
if not reader.fieldnames: return ValidationResult(False, ["CSV header is required"])
|
||||
except csv.Error as exc: return ValidationResult(False, [f"invalid CSV: {exc}"])
|
||||
return ValidationResult(True)
|
||||
def discover(self, config: Mapping[str, Any], cursor: str | None = None) -> DiscoveryPage:
|
||||
validation = self.validate(config)
|
||||
if not validation.valid: raise ValueError(validation.errors[0])
|
||||
reader = csv.DictReader(io.StringIO(config["csv"].replace("\r\n", "\n")))
|
||||
records = [normalize_record({str(k).strip().lower(): v for k, v in row.items()}) for row in reader]
|
||||
return DiscoveryPage(records, None, {"adapter": self.kind, "columns": reader.fieldnames or []})
|
||||
|
||||
ADAPTERS = {"manual": ManualSource, "csv": CsvSource}
|
||||
|
||||
def adapter_for(kind: str) -> DiscoverySource:
|
||||
try: return ADAPTERS[kind]()
|
||||
except KeyError: raise ValueError("unsupported source kind")
|
||||
@@ -114,3 +114,34 @@ CREATE TABLE IF NOT EXISTS job_events (
|
||||
UNIQUE(job_id,sequence)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_job_events_job_sequence ON job_events(job_id,sequence);
|
||||
|
||||
-- Phase 5 source framework (additive-safe; credentials contain metadata only).
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id),
|
||||
name TEXT NOT NULL, kind TEXT NOT NULL CHECK(kind IN ('csv','manual')), enabled INTEGER NOT NULL DEFAULT 0,
|
||||
config_json TEXT NOT NULL DEFAULT '{}', health_status TEXT NOT NULL DEFAULT 'unknown',
|
||||
consecutive_failures INTEGER NOT NULL DEFAULT 0, circuit_open INTEGER NOT NULL DEFAULT 0,
|
||||
last_success_at TEXT, last_failure_at TEXT, last_error TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,name)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_sources_org ON sources(organization_id,id);
|
||||
CREATE TABLE IF NOT EXISTS source_credentials (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id), provider TEXT NOT NULL, key_name TEXT NOT NULL,
|
||||
secret_ref TEXT NOT NULL DEFAULT '', metadata_json TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(source_id,provider,key_name)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS discovery_queries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL, query_json TEXT NOT NULL DEFAULT '{}', enabled INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,name)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_discovery_queries_org ON discovery_queries(organization_id,id);
|
||||
CREATE TABLE IF NOT EXISTS source_records (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
|
||||
discovery_query_id INTEGER REFERENCES discovery_queries(id) ON DELETE SET NULL, content_hash TEXT NOT NULL, raw_json TEXT NOT NULL,
|
||||
normalized_json TEXT NOT NULL, source_url TEXT NOT NULL DEFAULT '', query_context_json TEXT NOT NULL DEFAULT '{}', processing_status TEXT NOT NULL DEFAULT 'raw',
|
||||
cursor_json TEXT NOT NULL DEFAULT '{}', rate_policy_json TEXT NOT NULL DEFAULT '{}', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,source_id,content_hash)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_records_org ON source_records(organization_id,id DESC);
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
import json, os, sqlite3, threading, unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
from app.main import create_server
|
||||
from app.sources import CsvSource, ManualSource
|
||||
|
||||
class SourceAdapterTests(unittest.TestCase):
|
||||
def test_csv_adapter_is_deterministic_and_normalizes(self):
|
||||
src = CsvSource()
|
||||
a = src.discover({'csv': 'Name,Website,Email\n Acme ,https://acme.test,a@acme.test\n'})
|
||||
b = src.discover({'csv': 'Name,Website,Email\n Acme ,https://acme.test,a@acme.test\n'})
|
||||
self.assertEqual(a.records, b.records)
|
||||
self.assertEqual(a.records[0]['name'], 'Acme')
|
||||
self.assertEqual(a.records[0]['email'], 'a@acme.test')
|
||||
|
||||
def test_manual_validation_rejects_secret_fields(self):
|
||||
result = ManualSource().validate({'rows': [{'name': 'x', 'api_key': 'secret'}]})
|
||||
self.assertFalse(result.valid)
|
||||
self.assertIn('secret', result.errors[0].lower())
|
||||
|
||||
class SourceApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp=TemporaryDirectory(); os.environ['BOOTSTRAP_ADMIN_EMAIL']='owner@example.test'; os.environ['BOOTSTRAP_ADMIN_PASSWORD']='development-password'
|
||||
self.server=create_server('127.0.0.1',0,self.tmp.name+'/x.db'); self.thread=threading.Thread(target=self.server.serve_forever,daemon=True); self.thread.start(); self.c=HTTPConnection('127.0.0.1',self.server.server_port); self.cookie=None
|
||||
self.req('POST','/api/v1/auth/login',{'email':'owner@example.test','password':'development-password'})
|
||||
def tearDown(self): self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
|
||||
def req(self,m,p,x=None,cookie=True):
|
||||
body=json.dumps(x).encode() if x is not None else None; h={'Content-Type':'application/json'} if body else {};
|
||||
if cookie and self.cookie:h['Cookie']=self.cookie
|
||||
self.c.request(m,p,body,h); r=self.c.getresponse(); sc=r.getheader('Set-Cookie');
|
||||
if sc:self.cookie=sc.split(';',1)[0]
|
||||
raw=r.read(); return r.status,json.loads(raw or b'{}')
|
||||
def test_source_lifecycle_ingestion_idempotency_health_and_disabled(self):
|
||||
s, source=self.req('POST','/api/v1/sources',{'name':'Import','kind':'manual','config':{}}); self.assertEqual(s,201)
|
||||
sid=source['id']; self.assertFalse(source['enabled'])
|
||||
self.assertEqual(self.req('PATCH',f'/api/v1/sources/{sid}',{'enabled':True})[0],200)
|
||||
payload={'rows':[{'name':'Acme','website':'https://acme.test'}], 'source_url':'file://import.csv','query_context':{'q':'test'}}
|
||||
self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/ingest',payload)[0],201)
|
||||
self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/ingest',payload)[0],200)
|
||||
self.assertEqual(len(self.req('GET','/api/v1/source-records')[1]['items']),1)
|
||||
self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/test')[0],200)
|
||||
self.assertEqual(self.req('PATCH',f'/api/v1/sources/{sid}',{'enabled':False})[0],200)
|
||||
self.assertEqual(self.req('POST',f'/api/v1/sources/{sid}/ingest',payload)[0],409)
|
||||
db=sqlite3.connect(self.tmp.name+'/x.db'); self.assertTrue(db.execute("select 1 from audit_log where action='source.disabled'").fetchone()); db.close()
|
||||
def test_queries_enqueue_and_records_are_tenant_scoped(self):
|
||||
_,source=self.req('POST','/api/v1/sources',{'name':'CSV','kind':'csv','enabled':True})
|
||||
_,q=self.req('POST','/api/v1/discovery-queries',{'source_id':source['id'],'name':'q','query':{'csv':'name\nA'}})
|
||||
status,job=self.req('POST',f"/api/v1/discovery-queries/{q['id']}/run",{})
|
||||
self.assertEqual(status,202); self.assertEqual(job['type'],'source_discovery')
|
||||
self.assertEqual(self.req('GET','/api/v1/source-records?page_size=101')[0],400)
|
||||
def test_sources_require_auth(self):
|
||||
self.cookie=None; self.assertEqual(self.req('GET','/api/v1/sources',cookie=False)[0],401)
|
||||
|
||||
if __name__=='__main__': unittest.main()
|
||||
+8
-2
@@ -1,6 +1,6 @@
|
||||
# ProspectOS web — Phase 4 boundary
|
||||
# ProspectOS web — Phase 5 boundary
|
||||
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor; it does not discover prospects, scan DNS/websites, or send outreach.
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach.
|
||||
|
||||
## Configure and run
|
||||
|
||||
@@ -24,6 +24,12 @@ If not set, the UI uses `localStorage.prospect_api_base` when present; otherwise
|
||||
|
||||
The API remains the source of truth for tenant isolation, pagination bounds, filters, pipeline transitions, notes, audit records, and suppression. See `apps/api/README.md` for the route contract.
|
||||
|
||||
## Phase 5 source UI contract
|
||||
|
||||
The web client may display registered source metadata, query mode, approval/terms state, rate-limit status, retention class, health, and circuit state returned by the API. It must label `dry_run` as a plan/validation result and distinguish operator-supplied CSV/manual references from independently verified evidence. It must not offer a live-source control unless the API reports explicit approval and operational enablement; client visibility is never an authorization control.
|
||||
|
||||
CSV and manual reference workflows must show source attribution, adapter/version, observed time, and any retention/redaction status. Raw source payloads should be hidden or minimized in the UI and remain tenant-scoped. A circuit-open or rate-limited source must be presented as unavailable/deferred, not as an empty discovery result. The current static client has no network discovery implementation; these are display and contract requirements for a future approved integration.
|
||||
|
||||
## Phase 4 job/live-log UI contract
|
||||
|
||||
A future job view should show `queued`, `running`, `succeeded`, `failed`, or `cancelled`, the current attempt, timestamps, safe error text, and a clear terminal state. It should display persisted events in sequence order, resume from the last cursor after refresh/reconnect, and tolerate duplicate events. Create/retry requests should send an idempotency key and show the returned job identity rather than starting duplicate work.
|
||||
|
||||
+14
-2
@@ -63,13 +63,25 @@
|
||||
async function jobAction(action) { const job = jobs.find(item => String(item.id) === String(selectedJobId)); if (!job || !canManageJobs()) return; const endpointPath = action === 'cancel' ? `/api/v1/jobs/${encodeURIComponent(job.id)}/cancel` : `/api/v1/jobs/${encodeURIComponent(job.id)}/retry`; const label = action === 'cancel' ? 'cancel' : 'retry'; try { await jobsRequest(endpointPath, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({})}); jobMessage(`Job ${label} requested.`); await loadJobs({silent:true}); await loadJobDetail(job.id); } catch (error) { if (error.message !== 'unauthorized') jobMessage(error.message || `Unable to ${label} job.`, true); } }
|
||||
function updateJobPermissions() { if ($('startDemoJobBtn')) $('startDemoJobBtn').disabled = !canManageJobs(); }
|
||||
|
||||
let sources = [], selectedSourceId = null;
|
||||
const sourceState = source => Boolean(source?.enabled ?? source?.active);
|
||||
const sourceStatus = source => sourceState(source) ? 'enabled' : 'disabled';
|
||||
const sourceItems = payload => Array.isArray(payload) ? payload : (payload?.sources || payload?.items || payload?.records || []);
|
||||
function sourceMessage(text, error = false) { const el = $('sourcesMessage'); if (el) { el.textContent = text || ''; el.className = `sources-message${error ? ' error' : ''}`; } }
|
||||
function renderSourceSelect() { const select = $('discoverySource'); if (!select) return; select.innerHTML = `<option value="">Select a source</option>${sources.map(s => `<option value="${esc(s.id)}">${esc(s.name || s.label || `Source ${s.id}`)} · ${sourceStatus(s)}</option>`).join('')}`; }
|
||||
function renderSources() { renderSourceSelect(); const list = $('sourcesList'); if (!sources.length) { list.innerHTML = '<div class="source-empty">No sources returned by the workspace.</div>'; return; } list.innerHTML = sources.map(source => { const status = sourceStatus(source), health = source.health || source.health_status || 'Not tested', terms = source.terms_reviewed ?? source.terms_status ?? 'Not reviewed', config = source.config || {}; return `<article class="source-row" data-source-id="${esc(source.id)}"><div class="source-row-main"><strong>${esc(source.name || source.label || `Source ${source.id}`)}</strong><small>${esc(source.kind || source.source_type || source.type || 'manual')}${source.url || config.url ? ` · ${esc(source.url || config.url)}` : ''}</small></div><span class="source-status ${status}">${status}</span><dl class="source-facts"><div><dt>Owner</dt><dd>${esc(source.owner || source.owner_name || config.owner || 'Not assigned')}</dd></div><div><dt>Terms</dt><dd>${esc(String(source.terms_reviewed ?? source.terms_status ?? (config.terms_url ? 'Provided' : 'Not reviewed')))}</dd></div><div><dt>Rate limit</dt><dd>${esc(source.rate_limit || source.rate_limit_label || config.rate_limit || 'Not set')}</dd></div><div><dt>Health</dt><dd>${esc(String(health))}</dd></div></dl><div class="source-actions"><button class="button ghost compact" type="button" data-source-action="test" data-source-id="${esc(source.id)}">Test</button><button class="button ${status === 'enabled' ? 'danger' : 'primary'} compact" type="button" data-source-action="toggle" data-source-id="${esc(source.id)}">${status === 'enabled' ? 'Disable' : 'Enable'}</button></div></article>`; }).join(''); }
|
||||
function renderSourceRecords(items) { const list = $('sourceRecordsList'); if (!items.length) { list.innerHTML = '<div class="source-empty">No source records returned by the workspace.</div>'; return; } list.innerHTML = `<div class="source-record-table"><table><thead><tr><th>Record</th><th>Source</th><th>Status</th><th>Observed</th></tr></thead><tbody>${items.slice(0,25).map(record => `<tr><td>${esc(record.name || record.title || record.external_id || record.id || 'Unnamed record')}</td><td>${esc(record.source_name || record.source || 'Unknown source')}</td><td><span class="status">${esc(record.status || 'Pending')}</span></td><td>${esc(record.observed_at || record.created_at || 'Time unavailable')}</td></tr>`).join('')}</tbody></table></div>`; }
|
||||
async function loadSources() { sourceMessage('Loading sources…'); $('sourcesList').innerHTML = '<div class="detail-loading" aria-live="polite">Loading source registry…</div>'; $('sourceRecordsList').innerHTML = '<div class="detail-loading" aria-live="polite">Loading source records…</div>'; try { const [sourcePayload, recordPayload] = await Promise.all([jsonRequest('/api/v1/sources'), jsonRequest('/api/v1/source-records?page_size=25')]); sources = sourceItems(sourcePayload); renderSources(); renderSourceRecords(sourceItems(recordPayload)); $('sourcesUpdatedAt').textContent = `Updated ${new Date().toLocaleTimeString([], {hour:'2-digit', minute:'2-digit'})}`; sourceMessage(sources.some(sourceState) ? '' : 'No live source is enabled.'); } catch (error) { sources = []; renderSources(); renderSourceRecords([]); if (error.message !== 'unauthorized') sourceMessage(error.message || 'Unable to load sources.', true); } }
|
||||
async function saveSource(event) { event.preventDefault(); const form = event.currentTarget, fields = Object.fromEntries(new FormData(form).entries()); if (fields.source_type === 'csv' && !fields.csv_content.trim()) { message('sourceFormMessage', 'CSV content is required for a CSV source.', true); return; } const config = {url:fields.url, terms_url:fields.terms_url, owner:fields.owner, rate_limit:fields.rate_limit}; if (fields.source_type === 'csv') config.csv = fields.csv_content; else config.rows = []; try { await jsonRequest('/api/v1/sources', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({name:fields.name, kind:fields.source_type, config, enabled:false})}); message('sourceFormMessage', 'Source saved. It remains disabled until explicitly enabled.'); form.reset(); $('sourceCsvField').hidden = true; await loadSources(); } catch (error) { if (error.message !== 'unauthorized') message('sourceFormMessage', error.message || 'Unable to save source.', true); } }
|
||||
async function sourceAction(id, action) { const source = sources.find(item => String(item.id) === String(id)); if (!source) return; try { if (action === 'test') { await jsonRequest(`/api/v1/sources/${encodeURIComponent(id)}/test`, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({})}); sourceMessage('Source test completed.'); } else { const enabled = sourceState(source); await jsonRequest(`/api/v1/sources/${encodeURIComponent(id)}`, {method:'PATCH', headers:{'Content-Type':'application/json'}, body:JSON.stringify({enabled:!enabled})}); sourceMessage(`Source ${enabled ? 'disabled' : 'enabled'} by the workspace.`); } await loadSources(); } catch (error) { if (error.message !== 'unauthorized') sourceMessage(error.message || `Unable to ${action} source.`, true); } }
|
||||
async function runDiscovery(dryRun) { const form = $('discoveryForm'), data = Object.fromEntries(new FormData(form).entries()); data.dry_run = Boolean(dryRun); if (!data.source_id || !data.query.trim()) { message('discoveryMessage', 'Select a source and enter a query.', true); return; } const source = sources.find(item => String(item.id) === String(data.source_id)); if (!dryRun && !sourceState(source)) { message('discoveryMessage', 'This source is disabled. Enable it only after review.', true); return; } const button = dryRun ? $('discoveryDryRunBtn') : $('discoveryRunBtn'); button.disabled = true; message('discoveryMessage', dryRun ? 'Validating query…' : 'Starting discovery…'); try { const query = await jsonRequest('/api/v1/discovery-queries', {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({source_id:Number(data.source_id), name:data.query.trim().slice(0,80), query:{text:data.query.trim()}, dry_run:data.dry_run})}); if (!dryRun) await jsonRequest(`/api/v1/discovery-queries/${encodeURIComponent(query.id)}/run`, {method:'POST', headers:{'Content-Type':'application/json'}, body:JSON.stringify({})}); message('discoveryMessage', dryRun ? 'Dry run completed; no discovery job was started.' : 'Discovery request accepted. Check Jobs for progress.'); } catch (error) { if (error.message !== 'unauthorized') message('discoveryMessage', error.message || 'Discovery request failed.', true); } finally { button.disabled = false; } }
|
||||
function parseCsv(text){const lines=text.trim().split(/\r?\n/).filter(Boolean),cells=line=>line.match(/("[^"]*(?:""[^"]*)*"|[^,]+)(?=,|$)/g)?.map(x=>x.replace(/^"|"$/g,'').replaceAll('""','"'))||[];if(!lines.length)return[];const headers=cells(lines[0]);return lines.slice(1,11).map(l=>Object.fromEntries(cells(l).map((v,i)=>[headers[i]||`column_${i+1}`,v])));}
|
||||
function renderCsv(rows){if(!rows.length){$('csvPreview').innerHTML='<span>⊞</span><p>No data rows found</p>';return;}const h=Object.keys(rows[0]);$('csvPreview').className='csv-table';$('csvPreview').innerHTML=`<table><thead><tr>${h.map(x=>`<th>${esc(x)}</th>`).join('')}</tr></thead><tbody>${rows.map(r=>`<tr>${h.map(x=>`<td>${esc(r[x])}</td>`).join('')}</tr>`).join('')}</tbody></table><small class="muted">Showing up to 10 rows · Preview only; nothing added yet.</small>`;}
|
||||
async function login(event){event.preventDefault();const form=event.currentTarget,messageEl=$('loginMessage'),data=Object.fromEntries(new FormData(form).entries());messageEl.textContent='Signing in…';messageEl.className='form-message';try{const res=await fetch(endpoint('/api/v1/auth/login'),{method:'POST',headers:{'Content-Type':'application/json'},credentials:'include',body:JSON.stringify(data)});const body=await res.json().catch(()=>({}));if(!res.ok)throw new Error(body.error||'Invalid email or password.');await bootstrap();}catch(e){if(e.message!=='unauthorized'){messageEl.textContent=e.message;messageEl.className='form-message error';}}}
|
||||
async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}}
|
||||
async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
|
||||
async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
|
||||
document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
$('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
|
||||
$('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
|
||||
bootstrap();
|
||||
})();
|
||||
|
||||
@@ -31,6 +31,7 @@
|
||||
<a class="nav-item" href="#explorer"><span>⌕</span> Prospect explorer</a>
|
||||
<a class="nav-item" href="#add"><span>＋</span> Add prospects</a>
|
||||
<a class="nav-item" href="#jobs" data-nav="jobs"><span>◷</span> Jobs</a>
|
||||
<a class="nav-item" href="#sources" data-nav="sources"><span>⌁</span> Sources</a>
|
||||
</nav>
|
||||
<div class="sidebar-foot"><div class="live-dot"></div><div><strong>Workspace live</strong><small>Data sync is healthy</small></div></div>
|
||||
</aside>
|
||||
@@ -66,6 +67,17 @@
|
||||
<aside class="job-detail panel" id="jobDetailPanel"><div class="empty-detail"><span class="empty-icon">◷</span><h3>Select a job</h3><p>Inspect progress, structured errors, and the event timeline.</p></div></aside>
|
||||
</div>
|
||||
</section>
|
||||
<section class="sources-section" id="sources" aria-labelledby="sourcesTitle">
|
||||
<div class="sources-header panel"><div><p class="eyebrow">GOVERNANCE</p><h2 id="sourcesTitle">Sources</h2><p class="muted">Review source ownership, terms, limits, and health before using discovery.</p></div><button class="button ghost" id="sourcesRefreshBtn" type="button">↻ Refresh</button></div>
|
||||
<div class="source-safety" role="note"><strong>Discovery is disabled by default.</strong> No live source is enabled in this workspace. Enable a source only after its terms, owner, rate limit, and health have been reviewed.</div>
|
||||
<div id="sourcesMessage" class="sources-message" role="status" aria-live="polite"></div>
|
||||
<div class="sources-grid">
|
||||
<article class="panel source-config-panel"><div class="panel-heading"><div><p class="eyebrow">CONFIGURATION</p><h3>Add a source</h3></div><span class="small-label">Manual or CSV</span></div><form id="sourceForm"><div class="form-grid"><label>Source name<input name="name" required placeholder="Public business directory"></label><label>Source type<select name="source_type" id="sourceType"><option value="manual">Manual / API</option><option value="csv">CSV upload</option></select></label><label>Source URL <span class="optional">optional</span><input name="url" type="url" placeholder="https://…"></label><label>Terms URL <span class="optional">required for enablement</span><input name="terms_url" type="url" placeholder="https://…/terms"></label><label>Owner<input name="owner" required placeholder="Team or accountable person"></label><label>Rate limit<input name="rate_limit" required placeholder="e.g. 60 requests/hour"></label></div><label class="source-csv-field" id="sourceCsvField" hidden>CSV content<textarea name="csv_content" id="sourceCsvContent" rows="4" placeholder="Paste CSV content; it is sent only when you save this source."></textarea></label><div class="form-footer"><p id="sourceFormMessage" class="form-message" role="status"></p><button class="button primary" type="submit">Save source</button></div></form></article>
|
||||
<article class="panel discovery-panel"><div class="panel-heading"><div><p class="eyebrow">DISCOVERY</p><h3>Query a source</h3></div><span class="small-label">No automatic runs</span></div><form id="discoveryForm"><label>Source<select name="source_id" id="discoverySource" required><option value="">Select a source</option></select></label><label>Query<input name="query" required placeholder="e.g. renewable energy firms in Cape Town"></label><label class="checkbox-label"><input type="checkbox" name="dry_run" id="discoveryDryRun" checked> Dry run (preview only)</label><div class="form-footer"><p id="discoveryMessage" class="form-message" role="status"></p><div class="discovery-actions"><button class="button ghost" id="discoveryDryRunBtn" type="submit">Validate query</button><button class="button primary" id="discoveryRunBtn" type="button">Run discovery</button></div></div></form></article>
|
||||
</div>
|
||||
<div class="sources-list panel"><div class="panel-heading"><div><p class="eyebrow">REGISTRY</p><h3>Configured sources</h3></div><span id="sourcesUpdatedAt" class="small-label">Not loaded</span></div><div id="sourcesList" class="sources-list-body"><div class="source-empty">Sign in to load sources from the workspace.</div></div></div>
|
||||
<div class="source-records panel"><div class="panel-heading"><div><p class="eyebrow">RECENT OUTPUT</p><h3>Recent source records</h3></div></div><div id="sourceRecordsList" class="source-records-body"><div class="source-empty">No records loaded.</div></div></div>
|
||||
</section>
|
||||
<section class="lower-grid" id="add"><article class="panel add-panel"><div class="panel-heading"><div><p class="eyebrow">INTAKE</p><h2>Add a prospect</h2></div><span class="small-label">Manual entry</span></div><form id="addForm"><div class="form-grid"><label>Company name<input required name="name" placeholder="Acme Inc."></label><label>Website <span class="optional">optional</span><input name="website" type="url" placeholder="https://acme.com"></label><label>Location<input name="location" placeholder="Cape Town, ZA"></label><label>Notes <span class="optional">optional</span><input name="description" placeholder="Why this is a fit…"></label></div><div class="form-footer"><p id="formMessage" class="form-message" role="status"></p><button class="button primary" type="submit">Add to review queue</button></div></form></article>
|
||||
<article class="panel csv-panel"><div class="panel-heading"><div><p class="eyebrow">BULK INTAKE</p><h2>CSV preview</h2></div><label class="button ghost upload-label" for="csvInput">↑ Choose CSV</label><input id="csvInput" type="file" accept=".csv,text/csv" hidden></div><p class="muted">Preview rows before adding them to your review queue.</p><div id="csvPreview" class="csv-empty"><span>⊞</span><p>No file selected</p><small>CSV stays in your browser until you confirm.</small></div></article></section>
|
||||
<footer><span>ProspectOS</span><span>Evidence-led prospecting · <a href="#explorer">Review queue</a></span></footer>
|
||||
|
||||
@@ -24,5 +24,11 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
|
||||
,['Job status counts and controls',()=>['jobCountQueued','jobCountRunning','jobCountSucceeded','jobCountFailed','jobCountCancelled','startDemoJobBtn','jobsRefreshBtn'].every(id=>!!d.querySelector('#'+id))]
|
||||
,['Authenticated jobs API contract',()=>js.includes("/api/v1/jobs")&&js.includes('jobsRequest')&&js.includes('/cancel')&&js.includes('/retry')]
|
||||
,['Job detail timeline and structured states',()=>!!d.querySelector('#jobDetailPanel')&&js.includes('event_timeline')&&js.includes('structured_error')&&js.includes('progress')]
|
||||
,['Sources navigation and safety boundary',()=>!!d.querySelector('[data-nav="sources"]')&&!!d.querySelector('#sources')&&js.includes('No live source is enabled')]
|
||||
,['Source registry status and governance fields',()=>!!d.querySelector('#sourcesList')&&['owner','terms','rate_limit','health'].every(x=>js.includes(x))&&js.includes('/api/v1/sources')]
|
||||
,['Source configuration and controls use authenticated helper',()=>!!d.querySelector('#sourceForm')&&!!d.querySelector('#sourceType')&&js.includes('saveSource')&&js.includes('/test')&&js.includes('method:\'PATCH\'')&&js.includes('jsonRequest')]
|
||||
,['Discovery dry-run and run controls',()=>!!d.querySelector('#discoveryForm')&&!!d.querySelector('#discoveryDryRunBtn')&&!!d.querySelector('#discoveryRunBtn')&&js.includes('/api/v1/sources/discovery')&&js.includes('dry_run')]
|
||||
,['Recent source records and error/loading states',()=>!!d.querySelector('#sourceRecordsList')&&js.includes('source-record-table')&&js.includes('Loading source registry')&&js.includes('Unable to load sources')]
|
||||
,['No live source enabled copy is explicit',()=>d.querySelector('#sources')?.textContent.includes('No live source is enabled')&&!js.includes('demoSources')]
|
||||
];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
|
||||
</script>
|
||||
|
||||
+3
-1
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user