From 25ee7931abc1d28e97b1d2e279addb811a278c0b Mon Sep 17 00:00:00 2001 From: Marco0300 Date: Thu, 3 Sep 2026 08:46:22 +0200 Subject: [PATCH] add reversible prospect deduplication --- README.md | 14 ++++++- apps/api/README.md | 14 ++++++- apps/api/app/domain.py | 64 ++++++++++++++++++++++++++++++- apps/api/app/main.py | 63 ++++++++++++++++++++++++++++-- apps/api/schema.sql | 8 ++++ apps/api/tests/test_phase6.py | 30 +++++++++++++++ apps/api/tests/test_phase6_api.py | 35 +++++++++++++++++ apps/web/README.md | 12 +++++- apps/web/app.js | 35 ++++++++++++++++- apps/web/index.html | 3 ++ apps/web/smoke-test.html | 3 ++ apps/web/styles.css | 1 + docs/OPERATIONS.md | 10 +++++ docs/SECURITY.md | 11 ++++++ 14 files changed, 290 insertions(+), 13 deletions(-) create mode 100644 apps/api/tests/test_phase6.py create mode 100644 apps/api/tests/test_phase6_api.py diff --git a/README.md b/README.md index 99dc7b3..4227039 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,12 @@ # Prospect Intelligence Platform -A safety-first Phase 5 design/implementation boundary for **manual**, evidence-led prospect qualification and controlled source ingestion. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 5 defines source adapters, discovery-query records, raw-source retention, and health controls; it does **not** enable network discovery. **Automated outreach is disabled, and no live source may be enabled without explicit approval.** +A safety-first Phase 6 design/implementation boundary for **manual**, evidence-led prospect qualification and controlled source ingestion. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 6 defines deterministic South African normalization and deduplication review semantics in addition to the Phase 5 source controls; it does **not** enable network discovery. **Automated outreach is disabled, and no live source may be enabled without explicit approval.** ## Included - Dependency-free Python/SQLite API under `apps/api`. - Tenant-scoped business detail APIs with child intelligence/evidence records, provenance fields, notes, pipeline state, and audit history. -- Server-side normalization, conservative website classification, exact deduplication, versioned scoring, and suppression checks. +- Server-side normalization, conservative website classification, exact deduplication, versioned scoring, and suppression checks. Phase 6 documents the SA phone/location canonical forms and the review-only fuzzy-match contract. - Bounded list pagination and server-side filters so a tenant cannot request an unbounded prospect collection. - Responsive static dashboard under `apps/web` with authenticated explorer filters, paginated results, detail review, manual intake, notes/pipeline context, evidence provenance, and browser-only CSV preview. - Docker Compose runtime with non-root containers, read-only filesystems, health checks, and a named SQLite data volume. @@ -86,6 +86,16 @@ A discovery query is a tenant-scoped, bounded, auditable request that can be val SQLite, the in-process worker, and the named local volume are suitable for the pilot only; production migration, durable queue/worker leases, event retention/backup, SSE delivery, and tested backup/restore remain unfinished. Redis and Celery are not implemented. The development password fallback is PBKDF2 rather than production Argon2id. Before production, complete the gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`, including MFA, TLS, CSRF protection, source approval and terms review, rate limiting, circuit monitoring, tenant-scoped job/event authorization, idempotent side-effect handling, durable raw-source/audit retention, SSRF-safe fetching if a future scanner is approved, and tested backups/restores. +## Phase 6 normalization and deduplication boundary + +Normalization is deterministic and versioned. For South African data, phone values are stripped to digits, local 10-digit `0` forms and `00 27` forms are converted to canonical `+27...`, and unknown international numbers retain their explicit country code; presentation punctuation must not create a second identity. Locations derive whitespace/case/diacritic-folded province, city, and suburb fields. A normalized value is not proof that the underlying observation is correct. + +Exact keys (for example, canonical domain, email, or phone) may identify duplicate candidates. Fuzzy matching is deterministic and suggestion-only: the same inputs and normalization version produce the same candidate, score, and reason. A suggested match must never merge automatically. Use the documented thresholds: `>=0.90` is a strong suggestion, `0.75–0.8999` is a review suggestion, and `<0.75` is not surfaced as a suggestion. A human with permission must explicitly confirm each merge. + +Every confirmed merge must create a tenant-scoped, immutable-enough merge snapshot before mutation, recording the surviving and absorbed IDs, normalized comparison inputs, score/reasons, acting user, timestamp, and schema/normalization versions. The operation must be reversible from that snapshot. It must preserve or re-parent every child, evidence item, provenance/source-record link, note, pipeline/audit history, and original source identity; conflicts remain visible for human resolution rather than being silently overwritten. Cross-tenant candidates are never comparable or mergeable, and each suggestion, confirmation, rejection, reversal, and preservation/conflict decision belongs in the audit trail. + +The MVP now exposes deterministic match suggestions at `GET /api/v1/businesses/{id}/matches`, explicit merge confirmation in the web review dialog, tenant-scoped merge history, and `POST /api/v1/merge-history/{id}/reverse`. The implementation remains a pilot boundary: hardening is still needed for a dedicated merge permission, stronger server-side confirmation semantics, full snapshot conflict handling, and production-grade rollback guarantees. Do not describe a normalized or suggested match as verified identity, discovery, enrichment, or outreach authorization. + ## Verification ```bash diff --git a/apps/api/README.md b/apps/api/README.md index 1661adb..487b157 100644 --- a/apps/api/README.md +++ b/apps/api/README.md @@ -1,6 +1,6 @@ -# Prospect Platform API — Phase 5 boundary +# Prospect Platform API — Phase 6 boundary -Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 5 source-ingestion contract. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach. +Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 6 normalization/deduplication plus Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach. ## Run @@ -88,6 +88,16 @@ Pipeline state and verification are review metadata, not outreach authorization. All SQL uses parameters and all responses are JSON. Scores include `score_version` and `score_factors` for traceability. Provenance is supplied by the operator/source record; the MVP does not validate external sources or independently refresh evidence. +### Phase 6 normalization and duplicate-review contract + +The server is the normalization authority. SA phone input is canonicalized using explicit `+27` context for local `0` numbers, with punctuation/spacing removed while retaining a display/original value. Location input retains the raw observation and derives a comparison form plus country/province/municipality/city tokens; missing or ambiguous locality must remain missing/ambiguous, not guessed. The normalization/schema version must be stored with derived values so reprocessing is deterministic. + +Exact duplicate keys are deterministic. `GET /api/v1/businesses/{id}/matches` compares only active businesses in the authenticated organization and produces a sorted, deterministic score, score version, and explainable reasons. Its default candidate cutoff is `0.72`; policy bands are `>=0.90` strong suggestion, `0.75–0.8999` review suggestion, and `<0.75` no suggestion. Fuzzy comparison is **suggestion-only** and there is no automatic merge at any score. `POST /api/v1/businesses/{id}/merge` requires an authenticated mutating-role user and an explicit target; the web client also requires a human confirmation. A production merge permission and server-verifiable confirmation token remain hardening work. + +Before a confirmed merge, the current route persists a tenant-scoped `merge_history` snapshot of the source business and child rows (`business_identifiers`, `contacts`, `domains`, `websites`, `evidence`, `pipeline_entries`, `interactions`, and `notes`), plus child IDs/counts, actor, and timestamp. It re-parents those children to the target without deleting source records; `GET /api/v1/merge-history` reads the ledger and `POST /api/v1/merge-history/{id}/reverse` restores the source/child links. Candidate queries, merges, snapshots, reversal, and history reads apply the organization predicate; a cross-tenant ID behaves as not found. Audit events record merge and reversal actions. + +Remaining limitations: the snapshot currently focuses on the source graph rather than a full two-parent conflict snapshot; the merge route does not yet enforce a dedicated merge permission or cryptographically bound confirmation payload; and preservation/conflict semantics need production-grade transactional and concurrency tests. It must not claim that normalization proves identity or that deduplication performs discovery or outreach. + ## Pagination and filtering rules List and child-record endpoints are deliberately bounded. For business lists, use `page` (starting at 1) and `page_size` within the server-enforced maximum; invalid values are rejected rather than allowing an unbounded query. Supported filters are applied inside the tenant-scoped query before pagination: `q`, `score_min`, `score_max`, `website_class`, and `pipeline_stage`. The UI's page and filter controls are convenience clients, not authorization controls. A filtered page is not a count of the entire unfiltered tenant unless the response explicitly says so. diff --git a/apps/api/app/domain.py b/apps/api/app/domain.py index 9ad28ae..14681ad 100644 --- a/apps/api/app/domain.py +++ b/apps/api/app/domain.py @@ -2,9 +2,12 @@ from __future__ import annotations import re +import unicodedata +from difflib import SequenceMatcher from urllib.parse import urlparse SCORE_VERSION = "mvp-1" +MATCH_SCORE_VERSION = "phase6-1" _SOCIAL = {"facebook.com", "instagram.com", "linkedin.com", "twitter.com", "x.com", "youtube.com", "tiktok.com"} @@ -20,7 +23,38 @@ def normalize_domain(value: str | None) -> str: def normalize_phone(value: str | None) -> str: - return re.sub(r"[^0-9+]", "", (value or "").strip()) + raw = str(value or "").strip() + if not raw: + return "" + # Keep a leading international plus and digits only; never invent a country + # code for an unknown number. South African local and 00 prefixes are safe + # canonicalization cases because their numbering plan is unambiguous. + compact = re.sub(r"[^0-9+]", "", raw) + if compact.startswith("00"): + compact = "+" + compact[2:] + if compact.startswith("+27"): + rest = compact[3:] + if rest.startswith("0"): + rest = rest[1:] + return "+27" + rest + if compact.startswith("0") and len(compact) == 10: + return "+27" + compact[1:] + if compact.startswith("+"): + return "+" + re.sub(r"\D", "", compact[1:]) + return re.sub(r"\D", "", compact) + + +def _location_part(value: object) -> str: + text = " ".join(str(value or "").split()).strip().lower() + return "".join(c for c in unicodedata.normalize("NFKD", text) if not unicodedata.combining(c)) + + +def normalize_location(value: object = None, *, province=None, city=None, suburb=None) -> dict: + if isinstance(value, dict): + province, city, suburb = value.get("province", province), value.get("city", city), value.get("suburb", suburb) + elif value is not None and not any(x is not None for x in (province, city, suburb)): + province = value + return {"province": _location_part(province), "city": _location_part(city), "suburb": _location_part(suburb)} def normalize_business(raw: dict) -> dict: @@ -31,9 +65,37 @@ def normalize_business(raw: dict) -> dict: phone = normalize_phone(raw.get("phone")) result = dict(raw) result.update({"name": name, "email": email, "website": website, "website_domain": domain, "phone": phone}) + result.update(normalize_location(raw.get("location", raw))) return result +def match_businesses(source: dict, candidates: list[dict], threshold: float = 0.72) -> list[dict]: + """Return deterministic, explainable suggestions; this function never merges.""" + left = normalize_business(source) + output = [] + for raw in candidates: + right = normalize_business(raw) + signals = [] + if left["website_domain"] and left["website_domain"] == right["website_domain"]: + signals.append((1.0, "exact_website_domain")) + if left["email"] and left["email"] == right["email"]: + signals.append((1.0, "exact_email")) + if left["phone"] and left["phone"] == right["phone"]: + signals.append((1.0, "exact_phone")) + if left["name"] and right["name"]: + similarity = SequenceMatcher(None, re.sub(r"[^a-z0-9]", "", left["name"].lower()), re.sub(r"[^a-z0-9]", "", right["name"].lower())).ratio() + if similarity >= 0.65: signals.append((similarity, "similar_name")) + for field, reason in (("province", "same_province"), ("city", "same_city"), ("suburb", "same_suburb")): + if left[field] and left[field] == right[field]: signals.append((0.08, reason)) + if not signals: continue + exact = [s for s, r in signals if r.startswith("exact_")] + name = next((s for s, r in signals if r == "similar_name"), 0.0) + confidence = max(exact or [0.0]) if exact else min(0.99, 0.65 * name + sum(s for s, r in signals if r.startswith("same_"))) + if confidence >= threshold: + output.append({"id": raw.get("id"), "confidence": round(confidence, 4), "reasons": [r for _, r in signals], "score_version": MATCH_SCORE_VERSION}) + return sorted(output, key=lambda x: (-x["confidence"], x["id"] if isinstance(x["id"], int) else str(x["id"]))) + + def classify_website(website_or_domain: str | None) -> str: domain = normalize_domain(website_or_domain) if not domain: diff --git a/apps/api/app/main.py b/apps/api/app/main.py index 57ceb9e..32b1b7e 100644 --- a/apps/api/app/main.py +++ b/apps/api/app/main.py @@ -7,10 +7,10 @@ from pathlib import Path from urllib.parse import parse_qs, urlparse if __package__ in (None, ""): sys.path.insert(0, str(Path(__file__).resolve().parents[1])) - from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone + from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses from app.sources import adapter_for, contains_secret else: - from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone + from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses from .sources import adapter_for, contains_secret ORGANIZATION_ID = "demo-tenant" SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql" @@ -44,7 +44,7 @@ def connect(db_path: str) -> sqlite3.Connection: db = sqlite3.connect(db_path); db.row_factory = sqlite3.Row; db.execute("PRAGMA foreign_keys = ON"); db.executescript(SCHEMA.read_text()) # Upgrade databases created by Phase 1/2 without destroying data. cols = {r[1] for r in db.execute("PRAGMA table_info(businesses)")} - for col, definition in (("verified", "INTEGER NOT NULL DEFAULT 0"), ("verified_at", "TEXT"), ("updated_at", "TEXT")): + for col, definition in (("verified", "INTEGER NOT NULL DEFAULT 0"), ("verified_at", "TEXT"), ("updated_at", "TEXT"), ("province", "TEXT NOT NULL DEFAULT ''"), ("city", "TEXT NOT NULL DEFAULT ''"), ("suburb", "TEXT NOT NULL DEFAULT ''"), ("merge_status", "TEXT NOT NULL DEFAULT 'active'"), ("merged_into_id", "INTEGER")): if col not in cols: db.execute(f"ALTER TABLE businesses ADD COLUMN {col} {definition}") db.execute("UPDATE businesses SET updated_at=COALESCE(updated_at,created_at) WHERE updated_at IS NULL") db.execute("INSERT OR IGNORE INTO organizations (id,name) VALUES (?,?)", (ORGANIZATION_ID, "Demo organization")) @@ -112,6 +112,7 @@ class ApiHandler(BaseHTTPRequestHandler): if path=="/api/v1/dashboard/summary": row=db.execute("SELECT COUNT(*) businesses,COALESCE(AVG(score),0) average_score FROM businesses WHERE organization_id=?",(org,)).fetchone(); return self.send_json(200,{"organization_id":org,"businesses":row["businesses"],"average_score":round(row["average_score"],2),"suppressed":db.execute("SELECT COUNT(*) FROM suppressions WHERE organization_id=?",(org,)).fetchone()[0]}) if path=="/api/v1/businesses": return self.list_businesses(db,org,parse_qs(parsed.query)) + if path=="/api/v1/merge-history": return self.list_merge_history(db,org) if path=="/api/v1/sources": return self.list_sources(db,org) if path=="/api/v1/discovery-queries": return self.list_queries(db,org) if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query)) @@ -122,6 +123,7 @@ class ApiHandler(BaseHTTPRequestHandler): if not ident.isdigit(): return self.send_json(404,{"error":"not_found"}) row=self.business(db,int(ident),org) if not row:return self.send_json(404,{"error":"not_found"}) + if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org) payload=row_json(row); payload.update(self.nested(db,int(ident),org)); return self.send_json(200,payload) return self.send_json(404,{"error":"not_found"}) finally: db.close() @@ -219,6 +221,9 @@ class ApiHandler(BaseHTTPRequestHandler): if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user) if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user) if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org) + if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"): + ident=path.split("/")[4] + return self.reverse_merge(int(ident) if ident.isdigit() else -1,db,user) bits=path.split("/") if len(bits)==6 and bits[3] == "sources" and bits[4].isdigit() and bits[5] in {"test","ingest"}: return self.test_source(int(bits[4]),db,user) if bits[5]=="test" else self.ingest_source(int(bits[4]),payload,db,user) @@ -226,6 +231,7 @@ class ApiHandler(BaseHTTPRequestHandler): if len(bits)==7 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES and bits[6]=="": pass if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES:return self.create_child(int(bits[4]) if bits[4].isdigit() else -1,bits[5],payload,db,user) if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="verify":return self.verify_business(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user) + if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="merge":return self.merge_business(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user) return self.send_json(404,{"error":"not_found"}) finally:db.close() def do_PATCH(self): @@ -252,7 +258,7 @@ class ApiHandler(BaseHTTPRequestHandler): if is_suppressed(b,suppressions):return self.send_json(409,{"error":"suppressed"}) fields=[(c,b[c]) for c in ("website_domain","email","phone") if b[c]] if fields and db.execute("SELECT id FROM businesses WHERE organization_id=? AND ("+" OR ".join(f"{c}=?" for c,_ in fields)+")",[org]+[v for _,v in fields]).fetchone():return self.send_json(409,{"error":"duplicate"}) - scored=score_business(b);cur=db.execute("INSERT INTO businesses(organization_id,name,website,website_domain,email,phone,description,score,score_version,score_factors,website_class) VALUES(?,?,?,?,?,?,?,?,?,?,?)",(org,b["name"],b["website"],b["website_domain"],b["email"],b["phone"],str(b.get("description","")),scored["score"],scored["score_version"],json.dumps(scored["factors"]),scored["website_class"])); self.audit(db,user,"business.created",str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT * FROM businesses WHERE id=?",(cur.lastrowid,)).fetchone())) + scored=score_business(b);cur=db.execute("INSERT INTO businesses(organization_id,name,website,website_domain,email,phone,description,province,city,suburb,score,score_version,score_factors,website_class) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)",(org,b["name"],b["website"],b["website_domain"],b["email"],b["phone"],str(b.get("description","")),b["province"],b["city"],b["suburb"],scored["score"],scored["score_version"],json.dumps(scored["factors"]),scored["website_class"])); self.audit(db,user,"business.created",str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT * FROM businesses WHERE id=?",(cur.lastrowid,)).fetchone())) def create_suppression(self,payload,db,user): kind,value=payload.get("kind"),str(payload.get("value","")).strip().lower() if kind not in {"email","domain","phone"} or not value:return self.send_json(400,{"error":"invalid_suppression"}) @@ -364,6 +370,55 @@ class ApiHandler(BaseHTTPRequestHandler): try:db.execute("INSERT INTO source_records(organization_id,source_id,content_hash,raw_json,normalized_json,source_url,query_context_json,cursor_json,rate_policy_json) VALUES(?,?,?,?,?,?,?,?,?)",(user['organization_id'],sid,digest,raw,raw,str(payload.get('source_url','')),json.dumps(payload.get('query_context',{}),sort_keys=True),json.dumps(payload.get('cursor',{}),sort_keys=True),json.dumps(payload.get('rate_policy',{}),sort_keys=True)));inserted+=1 except sqlite3.IntegrityError:pass db.execute("UPDATE sources SET health_status='healthy',consecutive_failures=0,last_success_at=CURRENT_TIMESTAMP,last_error=NULL WHERE id=?",(sid,));self.audit(db,user,'source.ingested',f'{sid}:{inserted}');db.commit();return self.send_json(201 if inserted else 200,{"inserted":inserted,"records":len(page.records)}) + def matches(self,bid,db,org): + source=self.business(db,bid,org) + if not source:return self.send_json(404,{"error":"not_found"}) + try: threshold=float(parse_qs(urlparse(self.path).query).get("threshold",["0.72"])[0]) + except ValueError:return self.send_json(400,{"error":"invalid_threshold"}) + rows=[row_json(r) for r in db.execute("SELECT * FROM businesses WHERE organization_id=? AND id<>? AND merge_status='active' ORDER BY id",(org,bid))] + return self.send_json(200,{"business_id":bid,"threshold":threshold,"items":match_businesses(row_json(source),rows,threshold)}) + + def list_merge_history(self,db,org): + rows=[row_json(r) for r in db.execute("SELECT * FROM merge_history WHERE organization_id=? ORDER BY id DESC",(org,))] + for row in rows: + for key in ("source_snapshot_json","child_reassignment_json"): + try: row[key]=json.loads(row[key]) + except (TypeError,ValueError): pass + return self.send_json(200,{"organization_id":org,"items":rows}) + + def merge_business(self,bid,payload,db,user): + org=user["organization_id"]; target_id=payload.get("target_business_id",payload.get("target_id")) + if not isinstance(target_id,int) or target_id==bid:return self.send_json(400,{"error":"target_required"}) + source=self.business(db,bid,org); target=self.business(db,target_id,org) + if not source or not target:return self.send_json(404,{"error":"not_found"}) + if source["merge_status"] != "active":return self.send_json(409,{"error":"source_already_merged"}) + snapshot={"business":row_json(source),"children":{}} + child_meta={} + for table in ("business_identifiers","contacts","domains","websites","evidence","pipeline_entries","interactions","notes"): + rows=[row_json(r) for r in db.execute(f"SELECT * FROM {table} WHERE business_id=? AND organization_id=? ORDER BY id",(bid,org))] + snapshot["children"][table]=rows; child_meta[table]={"ids":[r["id"] for r in rows],"count":len(rows)} + if rows: db.execute(f"UPDATE {table} SET business_id=? WHERE business_id=? AND organization_id=?",(target_id,bid,org)) + cur=db.execute("INSERT INTO merge_history(organization_id,source_business_id,target_business_id,source_snapshot_json,child_reassignment_json,actor_user_id) VALUES(?,?,?,?,?,?)",(org,bid,target_id,json.dumps(snapshot,sort_keys=True),json.dumps(child_meta,sort_keys=True),user["id"])) + db.execute("UPDATE businesses SET merge_status='merged',merged_into_id=?,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",(target_id,bid,org)) + self.audit(db,user,"business.merged",f"{bid}->{target_id}");db.commit() + return self.send_json(200,{"merge_history_id":cur.lastrowid,"source_business_id":bid,"target_business_id":target_id,"status":"merged","reassigned":child_meta}) + + def reverse_merge(self,hid,db,user): + row=db.execute("SELECT * FROM merge_history WHERE id=? AND organization_id=?",(hid,user["organization_id"])).fetchone() + if not row:return self.send_json(404,{"error":"not_found"}) + if not row["reversible"]:return self.send_json(409,{"error":"merge_not_reversible"}) + source=self.business(db,row["source_business_id"],user["organization_id"]); target=self.business(db,row["target_business_id"],user["organization_id"]) + if not source or not target:return self.send_json(409,{"error":"business_missing"}) + snapshot=json.loads(row["source_snapshot_json"]); ids=json.loads(row["child_reassignment_json"]) + for table, meta in ids.items(): + if not meta.get("ids"):continue + marks=",".join("?" for _ in meta["ids"]) + db.execute(f"UPDATE {table} SET business_id=? WHERE business_id=? AND organization_id=? AND id IN ({marks})",[source["id"],target["id"],user["organization_id"]]+meta["ids"]) + db.execute("UPDATE businesses SET merge_status='active',merged_into_id=NULL,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",(source["id"],user["organization_id"])) + db.execute("UPDATE merge_history SET reversible=0,reversed_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",(hid,user["organization_id"])) + self.audit(db,user,"business.merge_reversed",str(hid));db.commit() + return self.send_json(200,{"id":hid,"status":"reversed","source_business_id":source["id"],"target_business_id":target["id"]}) + def log_message(self,*_):pass def _job_worker(server): diff --git a/apps/api/schema.sql b/apps/api/schema.sql index ffd2154..c4276a2 100644 --- a/apps/api/schema.sql +++ b/apps/api/schema.sql @@ -25,6 +25,14 @@ CREATE TABLE IF NOT EXISTS businesses ( score_version TEXT NOT NULL DEFAULT 'mvp-1', score_factors TEXT NOT NULL DEFAULT '[]', website_class TEXT NOT NULL DEFAULT 'missing', verified INTEGER NOT NULL DEFAULT 0, verified_at TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP ); +-- Phase 6 normalized location and reversible merge ledger. +CREATE TABLE IF NOT EXISTS merge_history ( + id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), + source_business_id INTEGER NOT NULL REFERENCES businesses(id), target_business_id INTEGER NOT NULL REFERENCES businesses(id), + source_snapshot_json TEXT NOT NULL, child_reassignment_json TEXT NOT NULL DEFAULT '{}', actor_user_id INTEGER REFERENCES users(id) ON DELETE SET NULL, + reversible INTEGER NOT NULL DEFAULT 1, reversed_at TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP +); +CREATE INDEX IF NOT EXISTS idx_merge_history_org ON merge_history(organization_id,created_at DESC,id DESC); CREATE INDEX IF NOT EXISTS idx_businesses_org ON businesses(organization_id); CREATE INDEX IF NOT EXISTS idx_businesses_score ON businesses(organization_id,score DESC,id); CREATE INDEX IF NOT EXISTS idx_businesses_class ON businesses(organization_id,website_class); diff --git a/apps/api/tests/test_phase6.py b/apps/api/tests/test_phase6.py new file mode 100644 index 0000000..ae7ee89 --- /dev/null +++ b/apps/api/tests/test_phase6.py @@ -0,0 +1,30 @@ +import unittest +from app.domain import normalize_phone, normalize_location, normalize_business, match_businesses + +class Phase6DomainTests(unittest.TestCase): + def test_south_african_phone_formats_share_canonical_value(self): + self.assertEqual(normalize_phone("082 555 1234"), "+27825551234") + self.assertEqual(normalize_phone("0027 82 555 1234"), "+27825551234") + self.assertEqual(normalize_phone("+27 (82) 555-1234"), "+27825551234") + + def test_unknown_international_phone_is_not_rewritten(self): + self.assertEqual(normalize_phone("+44 (20) 1234 5678"), "+442012345678") + self.assertEqual(normalize_phone("555-1234"), "5551234") + + def test_location_normalization_and_business_fields(self): + self.assertEqual(normalize_location({"province": " Gauteng ", "city": " Johannesburg ", "suburb": " Sandton "}), {"province": "gauteng", "city": "johannesburg", "suburb": "sandton"}) + b = normalize_business({"name":" Acme ", "province":" Gauteng ", "city":" Johannesburg ", "suburb":" Sandton "}) + self.assertEqual((b["province"], b["city"], b["suburb"]), ("gauteng", "johannesburg", "sandton")) + + def test_matching_has_deterministic_confidence_and_reasons(self): + a = {"name":"Acme Consulting", "email":"hello@acme.co.za", "city":"Johannesburg"} + b = {"name":"Acme Consultng", "email":"hello@acme.co.za", "city":"Johannesburg"} + first = match_businesses(a, [dict(b, id=2), {"id":3,"name":"Unrelated Shop"}], threshold=0.5) + second = match_businesses(a, [dict(b, id=2), {"id":3,"name":"Unrelated Shop"}], threshold=0.5) + self.assertEqual(first, second) + self.assertEqual(first[0]["id"], 2) + self.assertGreaterEqual(first[0]["confidence"], 0.5) + self.assertTrue(first[0]["reasons"]) + self.assertEqual([x["id"] for x in match_businesses(a, [{"id":3,"name":"Unrelated Shop"}], threshold=0.8)], []) + +if __name__ == "__main__": unittest.main() diff --git a/apps/api/tests/test_phase6_api.py b/apps/api/tests/test_phase6_api.py new file mode 100644 index 0000000..2cbc2d5 --- /dev/null +++ b/apps/api/tests/test_phase6_api.py @@ -0,0 +1,35 @@ +import json, os, sqlite3, threading, unittest +from http.client import HTTPConnection +from tempfile import TemporaryDirectory +from app.main import create_server + +class Phase6ApiTests(unittest.TestCase): + def setUp(self): + self.tmp=TemporaryDirectory(); os.environ['BOOTSTRAP_ADMIN_EMAIL']='owner@phase6.test'; os.environ['BOOTSTRAP_ADMIN_PASSWORD']='password' + self.server=create_server('127.0.0.1',0,self.tmp.name+'/db.sqlite'); self.thread=threading.Thread(target=self.server.serve_forever,daemon=True); self.thread.start(); self.conn=HTTPConnection('127.0.0.1',self.server.server_port,timeout=3); self.cookie=None + self.request('POST','/api/v1/auth/login',{'email':'owner@phase6.test','password':'password'}) + def tearDown(self): self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup() + def request(self,method,path,payload=None): + body=json.dumps(payload).encode() if payload is not None else None; headers={'Content-Type':'application/json'} if body else {} + if self.cookie: headers['Cookie']=self.cookie + self.conn.request(method,path,body,headers); r=self.conn.getresponse(); c=r.getheader('Set-Cookie'); + if c: self.cookie=c.split(';',1)[0] + return r.status,json.loads(r.read() or b'{}') + def test_merge_preserves_children_and_reverse_restores_ownership(self): + _, source=self.request('POST','/api/v1/businesses',{'name':'Acme Consulting','phone':'082 555 1234'}) + _, target=self.request('POST','/api/v1/businesses',{'name':'Acme Consulting HQ','city':'Johannesburg'}) + self.assertEqual(self.request('POST',f"/api/v1/businesses/{source['id']}/notes",{'body':'evidence'})[0],201) + self.assertEqual(self.request('POST',f"/api/v1/businesses/{source['id']}/evidence",{'kind':'source','claim':'claim'})[0],201) + status, merged=self.request('POST',f"/api/v1/businesses/{source['id']}/merge",{'target_business_id':target['id']}) + self.assertEqual(status,200); self.assertEqual(merged['status'],'merged') + status, detail=self.request('GET',f"/api/v1/businesses/{target['id']}"); self.assertEqual(status,200); self.assertEqual(len(detail['notes']),1); self.assertEqual(len(detail['evidence']),1) + status, source_detail=self.request('GET',f"/api/v1/businesses/{source['id']}"); self.assertEqual(status,200); self.assertEqual(source_detail['merge_status'],'merged') + status, reversed_=self.request('POST',f"/api/v1/merge-history/{merged['merge_history_id']}/reverse",{}); self.assertEqual(status,200); self.assertEqual(reversed_['status'],'reversed') + _, restored=self.request('GET',f"/api/v1/businesses/{source['id']}"); self.assertEqual(len(restored['notes']),1); self.assertEqual(len(restored['evidence']),1) + def test_matches_endpoint_is_thresholded_and_history_is_tenant_scoped(self): + _, a=self.request('POST','/api/v1/businesses',{'name':'Bright Co','email':'hello@bright.test'}) + _, b=self.request('POST','/api/v1/businesses',{'name':'Bright Company','website':'https://bright.test'}) + status, matches=self.request('GET',f"/api/v1/businesses/{a['id']}/matches?threshold=0.4"); self.assertEqual(status,200); self.assertEqual(matches['items'][0]['id'],b['id']); self.assertIn('similar_name',matches['items'][0]['reasons']) + self.assertEqual(self.request('GET','/api/v1/merge-history')[0],200) + +if __name__=='__main__': unittest.main() diff --git a/apps/web/README.md b/apps/web/README.md index 1d7a4c9..15203a9 100644 --- a/apps/web/README.md +++ b/apps/web/README.md @@ -1,6 +1,6 @@ -# ProspectOS web — Phase 5 boundary +# ProspectOS web — Phase 6 boundary -Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach. +Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts and Phase 6 normalization/deduplication concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach. ## Configure and run @@ -24,6 +24,14 @@ If not set, the UI uses `localStorage.prospect_api_base` when present; otherwise The API remains the source of truth for tenant isolation, pagination bounds, filters, pipeline transitions, notes, audit records, and suppression. See `apps/api/README.md` for the route contract. +## Phase 6 normalization and deduplication UI contract + +The UI may display the API's normalized SA phone and location values alongside the original observation, normalization version, and any ambiguity warning. It must not silently replace the source value or imply that a canonical form verifies identity. Duplicate candidates must show deterministic score, threshold band (`strong` `>=0.90`, `review` `0.75–0.8999`, or `none` `<0.75`), and explainable matching reasons. + +Suggestions are review aids only. A merge flow must identify the surviving record, list all parents/children/evidence/notes/source records that will be preserved, show conflicts, and require an explicit human confirmation before calling an authorized API mutation. The UI must offer rejection and, where implemented, reversal using the merge snapshot; it must never auto-merge based on a score. Candidate, merge, snapshot, and audit data are tenant-scoped by the API, not by hidden UI state. + +The current static MVP requests `/matches`, renders a **Human review required** list with confidence/reasons, asks for **Confirm merge**, and displays merge history with **Reverse merge** when the API marks it reversible. The API remains authoritative; these controls are not a substitute for server-side authorization. Existing normalization and match display remain suggestion-only; no merge happens without explicit operator confirmation. + ## Phase 5 source UI contract The web client may display registered source metadata, query mode, approval/terms state, rate-limit status, retention class, health, and circuit state returned by the API. It must label `dry_run` as a plan/validation result and distinguish operator-supplied CSV/manual references from independently verified evidence. It must not offer a live-source control unless the API reports explicit approval and operational enablement; client visibility is never an authorization control. diff --git a/apps/web/app.js b/apps/web/app.js index 3e7cd86..c40132b 100644 --- a/apps/web/app.js +++ b/apps/web/app.js @@ -25,7 +25,38 @@ async function selectProspect(id){selectedId=Number(id);selectedDetail=null;renderRows();$('detailPanel').innerHTML='
Loading prospect detail…
';await loadDetail(selectedId);} async function loadDetail(id){try{const detail=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}`);selectedDetail=detail;const index=prospects.findIndex(p=>Number(p.id)===Number(id));if(index>=0)prospects[index]={...prospects[index],...detail};renderDetail(detail);}catch(error){if(error.message!=='unauthorized')$('detailPanel').innerHTML=``;}} const listItems=(items,empty,label)=>Array.isArray(items)&&items.length?``:`

${empty}

`; - function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`

PROSPECT DETAIL

${esc(p.name)}

${esc(p.website_domain||'no detected website')}

${esc(review)}
Fit score${s}/ 100
${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence

Pipeline stage

Contacts ${contacts.length}

${listItems(contacts,'No contacts added.','email')}

Domains & websites

${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}

Evidence timeline

${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`

✓ ${esc(labelFactor(x))}${esc(p.confidence||'Medium')}

`).join(''):''}

Notes ${notes.length}

${listItems(notes,'No notes added.','body')}

Review status

${esc(review)}

${st!=='suppressed'?'':''}

${blocked?`

${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}

`:''}`;} + function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`

PROSPECT DETAIL

${esc(p.name)}

${esc(p.website_domain||'no detected website')}

${esc(review)}
Fit score${s}/ 100
${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence

Pipeline stage

Contacts ${contacts.length}

${listItems(contacts,'No contacts added.','email')}

Domains & websites

${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}

Evidence timeline

${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`

✓ ${esc(labelFactor(x))}${esc(p.confidence||'Medium')}

`).join(''):''}

Notes ${notes.length}

${listItems(notes,'No notes added.','body')}

Review status

${esc(review)}

${st!=='suppressed'?'':''}

${blocked?`

${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}

`:''}`;renderDedupPanel();} + let mergeSource = null, mergeTarget = null, mergeBusy = false; + const payloadItems = (payload, keys) => { for (const key of keys) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; }; + const suggestionId = item => item.target_id ?? item.business_id ?? item.prospect_id ?? item.id; + const suggestionName = item => item.target_name || item.business_name || item.prospect_name || item.name || `Prospect ${suggestionId(item)}`; + const suggestionConfidence = item => item.confidence ?? item.score ?? item.match_confidence ?? 'Unknown'; + const suggestionReasons = item => item.reasons || item.reason || item.match_reasons || item.explanation || []; + const reasonItems = reasons => Array.isArray(reasons) ? reasons : [reasons]; + function renderDedupPanel() { + const detail = $('detailPanel'); if (!detail || !selectedId) return; + let panel = $('dedupPanel'); + if (!panel) { panel = document.createElement('section'); panel.id = 'dedupPanel'; panel.className = 'dedup-panel'; panel.dataset.smoke = 'deduplication'; detail.appendChild(panel); } + panel.innerHTML = '
Loading match suggestions…
'; + loadMatchSuggestions(selectedId); + } + async function loadMatchSuggestions(id) { + const panel = $('dedupPanel'); if (!panel) return; + try { + const payload = await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/matches`); + const suggestions = payloadItems(payload, ['suggestions','matches','items']); + panel.innerHTML = `

DEDUPLICATION

Possible matches ${suggestions.length}

Human review required
${suggestions.length ? `
${suggestions.map(item => { const confidence=String(suggestionConfidence(item)); const reasons=reasonItems(suggestionReasons(item)); return `
${esc(suggestionName(item))}${esc(confidence)} confidence
    ${reasons.map(reason => `
  • ${esc(typeof reason === 'string' ? reason : reason.text || reason.description || JSON.stringify(reason))}
    • `).join('')}
`; }).join('')}
` : '

No possible matches returned. Nothing was merged automatically.

'}
Loading merge history…
`; + loadMergeHistory(id); + } catch (error) { if (error.message !== 'unauthorized') panel.innerHTML = `
`; } + } + async function loadMergeHistory(id) { + const history = $('mergeHistory'); if (!history) return; + try { const payload = await jsonRequest('/api/v1/merge-history'); const items = payloadItems(payload, ['history','merges','items']).filter(item => Number(item.source_business_id) === Number(id) || Number(item.target_business_id) === Number(id)); history.innerHTML = `

Merge history ${items.length}

${items.length ? `
${items.map(item => { const mergeId=item.merge_id||item.id, source=item.source_name||item.source_business_name||('Prospect '+(item.source_business_id||'source')), target=item.target_name||item.target_business_name||('Prospect '+(item.target_business_id||'target')); return `
${esc(source)} → ${esc(target)}${esc(item.created_at||item.merged_at||'Time unavailable')} · ${esc(item.status||'Merged')}
${item.reversible !== false && item.reversed_at == null ? `` : `${item.reversed_at ? 'Reversed' : 'Not reversible'}`}
`; }).join('')}
` : '

No merges recorded for this prospect.

'}`; } catch (error) { if (error.message !== 'unauthorized') history.innerHTML = ``; } + } + function openMergeDialog(targetId, targetName) { mergeSource={id:selectedId,name:selectedDetail?.name||prospects.find(p=>Number(p.id)===Number(selectedId))?.name||`Prospect ${selectedId}`}; mergeTarget={id:targetId,name:targetName}; $('mergeDialogCopy').innerHTML=`You are about to merge ${esc(mergeSource.name)} (source) into ${esc(mergeTarget.name)} (target). Review both records before confirming.`; $('mergeDialogMessage').textContent=''; $('mergeDialogMessage').className='form-message'; $('mergeDialog').hidden=false; $('confirmMergeBtn').disabled=false; } + function closeMergeDialog() { if (mergeBusy) return; $('mergeDialog').hidden=true; mergeSource=null; mergeTarget=null; } + async function confirmMerge() { if (!mergeSource || !mergeTarget || mergeBusy) return; mergeBusy=true; const button=$('confirmMergeBtn'); button.disabled=true; $('mergeDialogMessage').textContent='Merging records…'; $('mergeDialogMessage').className='form-message'; try { await jsonRequest(`/api/v1/businesses/${encodeURIComponent(mergeSource.id)}/merge`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({target_id:Number(mergeTarget.id),review_required:true})}); $('mergeDialogMessage').textContent='Merge completed and recorded in history.'; await loadData(); closeMergeDialog(); if (selectedId) await loadDetail(selectedId); } catch (error) { if (error.message !== 'unauthorized') { $('mergeDialogMessage').textContent=error.message||'Unable to merge records.'; $('mergeDialogMessage').className='form-message error'; button.disabled=false; } } finally { mergeBusy=false; } } + async function reverseMerge(id) { if (!window.confirm('Reverse this merge? The original records will be restored.')) return; try { await jsonRequest(`/api/v1/merge-history/${encodeURIComponent(id)}/reverse`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({review_required:true})}); if(selectedId) { await loadDetail(selectedId); await loadData(); } } catch(error) { const history=$('mergeHistory'); if(error.message!=='unauthorized'&&history) history.insertAdjacentHTML('afterbegin',``); } } function message(id,text,error=false){const el=$(id);if(el){el.textContent=text;el.className=`form-message${error?' error':''}`;}} async function saveContact(form){const data=Object.fromEntries(new FormData(form).entries());if(!data.email.trim()){message('contactMessage','Email is required.',true);return;}try{await jsonRequest(`/api/v1/businesses/${selectedId}/contacts`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(data)});message('contactMessage','Contact added.');await loadDetail(selectedId);}catch(e){if(e.message!=='unauthorized')message('contactMessage',e.message,true);}} async function saveNote(form){const data=Object.fromEntries(new FormData(form).entries());if(!data.body.trim()){message('noteMessage','Note cannot be empty.',true);return;}try{await jsonRequest(`/api/v1/businesses/${selectedId}/notes`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(data)});message('noteMessage','Note added.');await loadDetail(selectedId);}catch(e){if(e.message!=='unauthorized')message('noteMessage',e.message,true);}} @@ -81,7 +112,7 @@ async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}} async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}} document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);}); - document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);}); + document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);}); $('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView())); bootstrap(); })(); diff --git a/apps/web/index.html b/apps/web/index.html index 1b390e2..e132237 100644 --- a/apps/web/index.html +++ b/apps/web/index.html @@ -84,6 +84,9 @@ + diff --git a/apps/web/smoke-test.html b/apps/web/smoke-test.html index eeb2a4e..623b5e5 100644 --- a/apps/web/smoke-test.html +++ b/apps/web/smoke-test.html @@ -30,5 +30,8 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j ,['Discovery dry-run and run controls',()=>!!d.querySelector('#discoveryForm')&&!!d.querySelector('#discoveryDryRunBtn')&&!!d.querySelector('#discoveryRunBtn')&&js.includes('/api/v1/sources/discovery')&&js.includes('dry_run')] ,['Recent source records and error/loading states',()=>!!d.querySelector('#sourceRecordsList')&&js.includes('source-record-table')&&js.includes('Loading source registry')&&js.includes('Unable to load sources')] ,['No live source enabled copy is explicit',()=>d.querySelector('#sources')?.textContent.includes('No live source is enabled')&&!js.includes('demoSources')] + ,['Deduplication review UI contract',()=>!!d.querySelector('#mergeDialog')&&js.includes('/matches')&&js.includes('review-required')&&js.includes('confidence')&&js.includes('reasons')] + ,['Merge actions are explicit and reversible',()=>js.includes('/merge-history')&&js.includes('/reverse')&&js.includes('This action is reversible')&&js.includes('Confirm merge')&&!js.includes('autoMerge')] + ,['Deduplication loading and errors',()=>js.includes('Loading match suggestions')&&js.includes('Unable to load match suggestions')&&js.includes('merge-history')&&js.includes('Unable to load merge history')] ];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `
  • ${ok?'PASS':'FAIL'} — ${name}
    • `}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;}; diff --git a/apps/web/styles.css b/apps/web/styles.css index bd64686..9b828f1 100644 --- a/apps/web/styles.css +++ b/apps/web/styles.css @@ -1,5 +1,6 @@ :root{--ink:#172033;--muted:#6d7890;--line:#e7eaf1;--surface:#fff;--bg:#f7f8fb;--violet:#6756e8;--violet-soft:#efedff;--green:#16845b;--green-soft:#e5f7ef;--amber:#b87513;--amber-soft:#fff3dd;--red:#b84d55;--red-soft:#fff0f1;--shadow:0 10px 30px rgba(33,36,75,.05)}*{box-sizing:border-box}html{scroll-behavior:smooth}body{margin:0;background:var(--bg);color:var(--ink);font:14px/1.45 Inter,ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}.app-shell{display:flex;min-height:100vh}.sidebar{width:238px;background:#17152e;color:#e5e4f2;display:flex;flex-direction:column;padding:28px 16px;position:fixed;inset:0 auto 0 0}.brand{color:#fff;display:flex;align-items:center;gap:10px;text-decoration:none;font-size:20px;font-weight:750;padding:0 14px 44px;letter-spacing:-.5px}.brand-mark{width:27px;height:27px;border-radius:8px;background:#7263f3;display:grid;place-items:center;font-size:16px}.brand-light{font-weight:400;color:#a7a5c1}.nav-item{display:flex;align-items:center;gap:13px;color:#a8a7bd;text-decoration:none;padding:12px 15px;border-radius:9px;margin:3px 0}.nav-item span{font-size:20px;width:18px;text-align:center}.nav-item.active,.nav-item:hover{color:#fff;background:#2a2749}.sidebar-foot{margin-top:auto;border-top:1px solid #302d4b;padding:20px 14px 4px;display:flex;gap:9px;align-items:flex-start;font-size:12px}.sidebar-foot small{display:block;color:#85839e;margin-top:3px}.live-dot{background:#45d99c;width:7px;height:7px;border-radius:50%;margin-top:5px;box-shadow:0 0 0 4px #23463d}.main{margin-left:238px;flex:1;min-width:0}.topbar{height:76px;background:#fff;border-bottom:1px solid var(--line);display:flex;align-items:center;justify-content:space-between;padding:0 4.5%;color:var(--muted)}.crumb span{padding:0 10px;color:#c2c5ce}.top-actions{display:flex;align-items:center;gap:22px}.api-status{color:#7d8494;font-size:12px}.api-status.live{color:var(--green)}.icon-button{border:0;background:none;color:#7c8497;font-size:21px;cursor:pointer}.avatar{width:33px;height:33px;border-radius:50%;display:grid;place-items:center;background:#e5e2ff;color:#5648c8;font-weight:700;font-size:11px}.content{max-width:1450px;margin:auto;padding:40px 4.5% 28px}.hero{display:flex;justify-content:space-between;align-items:end;margin-bottom:30px}.eyebrow{color:#8d94a4;font-size:10px;letter-spacing:1.6px;font-weight:750;margin:0 0 9px}.hero h1{font-size:30px;letter-spacing:-1px;margin:0 0 7px}.hero h1 span{color:#7666f1}.hero-sub{color:var(--muted);margin:0}.hero-sub strong{color:var(--ink)}.button{border:0;border-radius:8px;padding:10px 15px;font-weight:650;cursor:pointer;white-space:nowrap}.primary{background:var(--violet);color:#fff;box-shadow:0 6px 14px #6756e833}.primary:hover{background:#5848d7}.ghost{background:#fff;border:1px solid var(--line);color:#5d6678}.ghost:hover{border-color:#bcb6ff;color:var(--violet)}.metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:24px}.metric-card,.panel{background:var(--surface);border:1px solid var(--line);border-radius:12px;box-shadow:var(--shadow)}.metric-card{padding:20px;display:flex;gap:14px;min-height:130px}.metric-icon{height:40px;width:40px;border-radius:11px;display:grid;place-items:center;font-size:22px}.violet{background:var(--violet-soft);color:var(--violet)}.amber{background:var(--amber-soft);color:var(--amber)}.green{background:var(--green-soft);color:var(--green)}.blue{background:#e9f2ff;color:#3e80d5}.metric-card p{margin:2px 0 5px;color:var(--muted);font-size:12px}.metric-card h2{margin:0 0 6px;font-size:26px;letter-spacing:-1px}.trend{font-size:11px;font-weight:700}.trend em{font-style:normal;font-weight:400;color:#a2a8b5}.up{color:var(--green)}.neutral{color:#8b93a2}.workspace-grid{display:grid;grid-template-columns:minmax(0,1.6fr) minmax(290px,.75fr);gap:18px;margin-bottom:24px}.panel{padding:24px}.panel-heading{display:flex;justify-content:space-between;align-items:start;margin-bottom:20px}.panel h2{font-size:18px;margin:0;letter-spacing:-.3px}.filters{display:grid;grid-template-columns:minmax(160px,1fr) 150px 150px;gap:9px;margin-bottom:16px}.search-wrap{display:flex;align-items:center;border:1px solid var(--line);border-radius:8px;background:#fff;color:#a2a8b5;padding:0 11px}.search-wrap input{border:0;outline:0;padding:10px 8px;width:100%;font:inherit;color:var(--ink);background:transparent}.filters select{border:1px solid var(--line);border-radius:8px;padding:0 10px;color:#596478;background:#fff;font:inherit}.table-meta{color:#9299a8;font-size:11px;display:flex;justify-content:space-between;margin:0 0 9px}.legend{display:flex;gap:6px;align-items:center}.legend-dot{width:7px;height:7px;border-radius:50%;display:inline-block;margin-left:8px}.high-dot{background:#52bf93}.review-dot{background:#e8a84f}.table-scroll{overflow-x:auto}table{border-collapse:collapse;width:100%;min-width:650px}th{text-align:left;color:#9aa1af;font-size:10px;letter-spacing:.6px;text-transform:uppercase;font-weight:700;padding:11px 8px;border-bottom:1px solid var(--line)}td{padding:15px 8px;border-bottom:1px solid #f0f1f5;vertical-align:middle;color:#485367;font-size:12px}tbody tr{cursor:pointer;transition:background .15s}tbody tr:hover,tbody tr.selected{background:#faf9ff}td:first-child{color:var(--ink);font-weight:700;font-size:13px}.company-sub{display:block;font-size:11px;color:#99a0ae;font-weight:400;margin-top:2px}.score{display:inline-flex;align-items:center;gap:5px;border-radius:15px;padding:4px 8px;font-weight:750;font-size:11px}.score.high{color:var(--green);background:var(--green-soft)}.score.medium{color:var(--amber);background:var(--amber-soft)}.score.low{color:#788193;background:#eef0f4}.evidence{color:#596478}.evidence strong{display:block;color:var(--ink);font-size:12px}.fresh{font-size:11px}.fresh.good{color:var(--green)}.fresh.stale{color:var(--amber)}.status{font-size:10px;border-radius:4px;padding:4px 6px;font-weight:700}.status.review{background:var(--amber-soft);color:var(--amber)}.status.reviewed{background:var(--green-soft);color:var(--green)}.status.suppressed{background:var(--red-soft);color:var(--red)}.row-arrow{font-size:18px;color:#aeb4c0}.detail-panel{min-height:420px}.empty-detail{text-align:center;color:var(--muted);padding:55px 20px}.empty-icon{display:grid;place-items:center;margin:auto auto 17px;background:var(--violet-soft);color:var(--violet);width:48px;height:48px;border-radius:50%;font-size:23px}.empty-detail h3{color:var(--ink);margin:0 0 8px}.empty-detail p{margin:auto;max-width:220px;font-size:12px}.detail-head{display:flex;justify-content:space-between;gap:10px}.detail-head h3{margin:0;font-size:19px}.detail-domain{color:#949baa;font-size:12px;margin:3px 0 20px}.detail-score{display:flex;align-items:center;justify-content:space-between;background:#f9f8ff;padding:14px;border-radius:9px;margin-bottom:18px}.detail-score b{font-size:27px;color:var(--violet)}.detail-score small{display:block;color:var(--muted)}.detail-block{border-top:1px solid var(--line);padding:15px 0}.detail-block h4{font-size:10px;color:#8c94a4;text-transform:uppercase;letter-spacing:1px;margin:0 0 10px}.detail-block p{font-size:12px;margin:5px 0;color:#556176}.evidence-line{display:flex;justify-content:space-between;gap:10px}.confidence{color:var(--violet);font-weight:700}.disabled-action{width:100%;margin-top:5px;color:#a0a6b3;background:#f0f1f4;cursor:not-allowed}.disabled-reason{font-size:11px;color:var(--red);margin:8px 0 0}.lower-grid{display:grid;grid-template-columns:1.3fr 1fr;gap:18px}.small-label,.optional{color:#9ca3b1;font-size:11px;font-weight:400}.form-grid{display:grid;grid-template-columns:1fr 1fr;gap:16px}label{display:block;color:#596478;font-size:12px;font-weight:650}label input{display:block;width:100%;margin-top:7px;border:1px solid var(--line);border-radius:7px;padding:10px 11px;font:inherit;outline:0}label input:focus{border-color:#9489f5;box-shadow:0 0 0 3px #eeeaff}.form-footer{display:flex;align-items:center;justify-content:space-between;margin-top:20px}.form-message{font-size:11px;color:var(--green);margin:0}.form-message.error{color:var(--red)}.muted{color:var(--muted);font-size:12px}.csv-empty{border:1px dashed #d9dce8;border-radius:9px;text-align:center;padding:22px;color:#adb3c0}.csv-empty span{font-size:26px}.csv-empty p{margin:4px 0;font-size:12px;color:#737d90}.csv-empty small{font-size:10px}.csv-table{max-height:150px;overflow:auto;font-size:11px}.csv-table table{min-width:400px}.csv-table th,.csv-table td{padding:7px}.upload-label{display:inline-block}footer{display:flex;justify-content:space-between;color:#a0a6b2;font-size:11px;padding:30px 2px 0}footer a{color:var(--violet);text-decoration:none}.mobile-menu{display:none;border:0;background:transparent;font-size:21px;color:var(--ink)}@media(max-width:1050px){.metrics{grid-template-columns:repeat(2,1fr)}.workspace-grid{grid-template-columns:1fr}.detail-panel{min-height:auto}.lower-grid{grid-template-columns:1fr}}@media(max-width:700px){.sidebar{transform:translateX(-100%);transition:transform .2s;z-index:5;width:230px}.sidebar.open{transform:translateX(0)}.main{margin-left:0}.topbar{padding:0 20px}.mobile-menu{display:block}.crumb{font-size:12px}.top-actions{gap:12px}.api-status{display:none}.content{padding:28px 16px}.hero{align-items:start;gap:18px;flex-direction:column}.hero h1{font-size:25px}.metrics{grid-template-columns:1fr 1fr;gap:10px}.metric-card{padding:15px;min-height:112px;gap:9px}.metric-icon{width:34px;height:34px;font-size:18px}.metric-card h2{font-size:22px}.panel{padding:18px}.filters{grid-template-columns:1fr;gap:8px}.filters select{height:38px}.table-meta{align-items:start;gap:8px;flex-direction:column}.legend{display:none}.form-grid{grid-template-columns:1fr}.form-footer{align-items:start;gap:14px;flex-direction:column}.form-footer .button{width:100%}footer{flex-direction:column;gap:5px}}.login-screen{min-height:100vh;display:grid;place-items:center;padding:24px;background:radial-gradient(circle at 15% 10%,#efedff 0,transparent 34%),var(--bg)}.login-card{width:min(100%,430px);padding:42px 40px;background:var(--surface);border:1px solid var(--line);border-radius:18px;box-shadow:0 24px 70px rgba(33,36,75,.11)}.login-brand{padding:0;margin-bottom:42px}.login-card h1{margin:0;font-size:32px;letter-spacing:-1px}.login-subtitle{color:var(--muted);margin:8px 0 28px}.login-card form{display:grid;gap:17px}.login-card label input{padding:12px}.login-submit{width:100%;display:flex;justify-content:center;gap:9px;margin-top:3px;padding:12px}.login-note{color:#9299a8;text-align:center;font-size:11px;margin:22px 0 0}.user-identity{color:#596478;font-size:12px;font-weight:650}.logout-button{border:1px solid var(--line);border-radius:7px;background:#fff;color:#596478;font:inherit;font-size:11px;font-weight:650;padding:7px 10px;cursor:pointer}.logout-button:hover{border-color:#bcb6ff;color:var(--violet)}[hidden]{display:none!important}@media(max-width:700px){.login-card{padding:32px 22px}.login-brand{margin-bottom:32px}.user-identity{display:none}.logout-button{padding:6px 8px}} .detail-list{margin:.5rem 0 1rem;padding-left:1.2rem}.detail-list li{margin:.25rem 0}.detail-block{border-top:1px solid var(--line);padding:14px 0}.detail-block h4{margin:0 0 9px}.count{color:var(--muted);font-size:12px;font-weight:400}.compact{padding:7px 10px;font-size:12px}.inline-form,.compact-form{display:flex;gap:7px;flex-wrap:wrap;align-items:center}.compact-form input,.compact-form textarea,.inline-form select{border:1px solid var(--line);border-radius:7px;padding:8px;font:inherit;min-width:0;flex:1}.compact-form textarea{flex-basis:100%;resize:vertical}.detail-loading,.detail-error{padding:28px 4px;color:var(--muted)}.detail-error h3{color:var(--ink)}.review-status{margin:0 0 10px}.page-size{display:flex;align-items:center;gap:5px;color:var(--muted);font-size:12px}.page-size select{border:1px solid var(--line);border-radius:6px;padding:5px}.form-message.error{color:var(--red)} .sources-section{margin-top:28px;scroll-margin-top:24px}.sources-header{display:flex;align-items:flex-start;justify-content:space-between;gap:24px}.sources-header h2{margin:.15rem 0 .25rem}.source-safety{margin:14px 0;padding:12px 15px;border:1px solid #f1d7a5;border-radius:9px;background:var(--amber-soft);color:#76500d}.sources-message{min-height:22px;padding:8px 2px;color:var(--green)}.sources-message.error{color:var(--red)}.sources-grid{display:grid;grid-template-columns:1fr 1fr;gap:18px}.source-config-panel,.discovery-panel,.sources-list,.source-records{min-width:0}.source-csv-field{display:block;margin-top:12px}.source-csv-field textarea{width:100%;resize:vertical}.checkbox-label{display:flex;flex-direction:row;align-items:center;gap:8px;margin-top:14px}.checkbox-label input{width:auto}.discovery-actions{display:flex;gap:8px;flex-wrap:wrap}.source-row{display:grid;grid-template-columns:minmax(170px,1fr) auto;gap:12px 18px;padding:16px;border-top:1px solid var(--line);align-items:start}.source-row-main{display:flex;flex-direction:column;gap:3px;min-width:0}.source-row-main strong{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.source-row-main small,.source-facts{color:var(--muted);font-size:12px}.source-status{border-radius:999px;padding:4px 9px;font-size:11px;font-weight:700;text-transform:capitalize;background:var(--red-soft);color:var(--red)}.source-status.enabled{background:var(--green-soft);color:var(--green)}.source-facts{grid-column:1 / -1;display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:0}.source-facts div{border:1px solid var(--line);border-radius:7px;padding:8px}.source-facts dt{font-size:10px;text-transform:uppercase;letter-spacing:.05em}.source-facts dd{margin:3px 0 0;color:var(--ink);overflow-wrap:anywhere}.source-actions{grid-column:1 / -1;display:flex;gap:7px}.button.danger{border-color:#e9b8bd;color:var(--red);background:var(--red-soft)}.source-empty{padding:34px 18px;color:var(--muted);text-align:center}.source-record-table{overflow-x:auto}.source-record-table table{min-width:650px}.source-record-table td,.source-record-table th{padding:11px 14px}@media(max-width:900px){.sources-grid{grid-template-columns:1fr}.source-facts{grid-template-columns:repeat(2,1fr)}}@media(max-width:700px){.sources-header{flex-direction:column}.source-row{grid-template-columns:1fr}.source-status{justify-self:start}.source-facts{grid-column:1;grid-template-columns:1fr 1fr}.source-actions{grid-column:1}.discovery-actions .button{flex:1}} +.dedup-panel{border-top:1px solid var(--line);margin-top:18px;padding-top:16px}.dedup-heading{display:flex;align-items:flex-start;justify-content:space-between;gap:12px}.dedup-heading h4{margin:.1rem 0}.match-list,.history-list{display:grid;gap:9px;margin-top:10px}.match-card,.history-row{border:1px solid var(--line);border-radius:9px;padding:11px;background:#fcfcfe}.match-card-head,.history-row{display:flex;align-items:flex-start;justify-content:space-between;gap:10px}.match-card-head strong{overflow-wrap:anywhere}.match-confidence{font-size:11px;color:var(--green);font-weight:700;white-space:nowrap}.match-reasons{margin:7px 0;padding-left:18px;color:var(--muted);font-size:12px}.match-reasons li{margin:3px 0}.history-row small{display:block;color:var(--muted);font-size:11px;margin-top:3px}.dedup-error{padding:14px 0;color:var(--muted)}.dedup-error h4{color:var(--ink);margin:0 0 4px}.merge-dialog{position:fixed;z-index:20;inset:0;background:rgba(23,32,51,.46);display:grid;place-items:center;padding:20px}.merge-dialog[hidden]{display:none}.merge-dialog-card{width:min(520px,100%);background:var(--surface);border-radius:14px;padding:22px;box-shadow:0 24px 70px rgba(23,32,51,.25)}.merge-dialog-card h2{margin:.1rem 0}.merge-dialog-card p{margin:12px 0}.merge-warning{border:1px solid #f1d7a5;border-radius:8px;padding:11px;background:var(--amber-soft);color:#76500d}.merge-dialog-actions{display:flex;justify-content:flex-end;gap:8px;margin-top:18px} .jobs-section{margin-top:28px;scroll-margin-top:24px}.jobs-header{display:flex;align-items:flex-start;justify-content:space-between;gap:24px}.jobs-header h2{margin:.15rem 0 .25rem}.jobs-header p{margin:.25rem 0 0}.jobs-actions{display:flex;gap:8px;flex-wrap:wrap}.jobs-message{min-height:22px;padding:8px 2px;color:var(--green)}.jobs-message.error{color:var(--red)}.job-counts{display:grid;grid-template-columns:repeat(5,1fr);gap:12px;margin:0 0 14px}.job-count{background:var(--surface);border:1px solid var(--line);border-left:4px solid var(--violet);border-radius:10px;padding:14px 16px;box-shadow:var(--shadow)}.job-count span{display:block;color:var(--muted);font-size:12px}.job-count strong{display:block;font-size:25px;margin-top:4px}.job-count.queued{border-left-color:var(--amber)}.job-count.running{border-left-color:#4d8bd8}.job-count.succeeded{border-left-color:var(--green)}.job-count.failed{border-left-color:var(--red)}.job-count.cancelled{border-left-color:#8d879c}.jobs-grid{display:grid;grid-template-columns:minmax(0,1.05fr) minmax(320px,.95fr);gap:18px}.jobs-list,.job-detail{min-height:320px}.jobs-list-body{border-top:1px solid var(--line)}.job-empty{padding:38px 18px;color:var(--muted);text-align:center}.job-row{display:grid;grid-template-columns:minmax(0,1fr) auto 42px;gap:12px;align-items:center;width:100%;padding:14px 16px;border:0;border-bottom:1px solid var(--line);background:transparent;color:inherit;text-align:left;cursor:pointer;font:inherit}.job-row:hover,.job-row.selected{background:var(--violet-soft)}.job-row-main{display:flex;flex-direction:column;gap:3px;min-width:0}.job-row-main strong{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.job-row-main small,.job-row-progress{color:var(--muted);font-size:12px}.job-row-state{border-radius:999px;padding:4px 8px;font-size:11px;font-weight:700;white-space:nowrap;background:var(--violet-soft);color:var(--violet)}.job-row-state.queued{background:var(--amber-soft);color:var(--amber)}.job-row-state.running{background:#e8f1ff;color:#3269ad}.job-row-state.succeeded{background:var(--green-soft);color:var(--green)}.job-row-state.failed{background:var(--red-soft);color:var(--red)}.job-row-state.cancelled{background:#f0eef4;color:#716a7c}.job-detail{padding:22px}.job-detail-head{display:flex;justify-content:space-between;gap:12px;border-bottom:1px solid var(--line);padding-bottom:16px}.job-detail-head h3{margin:.15rem 0}.job-progress{padding:18px 0}.job-progress-meta{display:flex;justify-content:space-between;gap:12px;font-size:12px;color:var(--muted)}.job-progress-meta strong{color:var(--ink)}.progress-track{height:8px;background:#eef0f5;border-radius:999px;overflow:hidden;margin-top:10px}.progress-track span{display:block;height:100%;background:var(--violet);border-radius:inherit;transition:width .25s}.structured-error{background:var(--red-soft);border:1px solid #f2cdd0;border-radius:8px;padding:12px;margin:4px 0 16px;color:var(--red)}.structured-error p{margin:5px 0}.structured-error pre{white-space:pre-wrap;font-size:11px;margin:8px 0 0}.job-detail-actions{display:flex;gap:8px;min-height:34px}.event-timeline{border-top:1px solid var(--line);margin-top:16px;padding-top:16px}.event-timeline h4{margin:0}.event-timeline ol{list-style:none;padding:0;margin:12px 0 0}.event-timeline li{display:flex;gap:10px;position:relative;padding:0 0 15px}.event-timeline li:not(:last-child):before{content:"";position:absolute;left:4px;top:10px;bottom:0;border-left:1px solid var(--line)}.timeline-dot{z-index:1;width:9px;height:9px;margin-top:4px;border-radius:50%;background:var(--violet);flex:none}.event-timeline li div{display:flex;flex-direction:column;gap:2px}.event-timeline small,.timeline-progress{font-size:11px;color:var(--muted)}@media(max-width:900px){.jobs-grid{grid-template-columns:1fr}.job-counts{grid-template-columns:repeat(3,1fr)}}@media(max-width:700px){.jobs-header{flex-direction:column}.job-counts{grid-template-columns:repeat(2,1fr)}.job-row{grid-template-columns:minmax(0,1fr) auto}.job-row-progress{display:none}} diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 36182ef..061318f 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -41,6 +41,16 @@ Before enabling any adapter, verify the registry entry has a stable ID/version, Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation. +## Phase 6 normalization and deduplication operations + +Normalization and duplicate review are data-integrity operations, not discovery. Record the normalization and algorithm versions with every derived SA phone/location value and suggestion. Verify that local South African phone forms are interpreted only with explicit `+27` context, that original values remain available, and that ambiguous locations are flagged rather than guessed. Re-running the same input/version must produce the same canonical values, score, band, and reasons. + +Operate the review queue with the fixed bands: `>=0.90` strong suggestion, `0.75–0.8999` review suggestion, `<0.75` no suggestion. No band authorizes a merge. A permitted human must choose the survivor and explicitly confirm; operators must be able to reject a suggestion. Before any merge, verify the tenant, permissions, conflict list, and that a complete pre-merge snapshot was durably written. Preserve every child, evidence/provenance item, source-record/raw lineage link, note, pipeline state, and audit entry; never resolve a conflict by deleting the losing source record. + +After a merge, read back the survivor and audit trail, check child/evidence/source-record counts and provenance links, and record the snapshot ID. For reversal, use only the snapshot, verify the same tenant and authorization, restore the prior graph transactionally, and read back the restored records and audit event. If snapshot creation, preservation, or audit append fails, stop the mutation and investigate; do not retry blindly. Cross-tenant candidates must be absent, not merely hidden by the UI. + +The current Compose/MVP runtime exposes the Phase 6 match-suggestion, explicit merge-confirmation, tenant-scoped merge-history, and reversal routes; no separate worker is required. It has basic canonicalization and exact duplicate handling plus pilot-grade fuzzy review. Monitor merge audit events and verify child/source-record counts after each operation. Do not schedule or report deduplication as automated discovery, enrichment, verification, or outreach. + ## Phase 4 jobs and live logging The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects. diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 695a615..74cf39d 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -18,6 +18,17 @@ - Phase 4 job/live-log controls are not enabled in the current runtime. If added, job IDs, idempotency keys, status, cancellation, retries, and event cursors must all be authorized against the authenticated organization; never accept a job or child identifier as authorization by itself. - Persisted job events must be append-only, sequence-ordered per job, replayable from a cursor, and redacted to safe operational data. Never emit credentials, session cookies, API keys, full request bodies, or unnecessary contact/prospect data in polling responses, SSE frames, logs, or error details. +## Phase 6 normalization and deduplication controls + +- Treat normalization as a deterministic, versioned derivation, never as identity proof. SA phone normalization must use an explicit `+27` context for local `0` forms and retain the original/display value; location normalization must retain raw input and flag missing or ambiguous province/municipality/city rather than guessing. +- Exact duplicate keys and fuzzy suggestions must be reproducible from the same tenant data, normalization version, and algorithm version. Expose the score and reasons, and use `>=0.90` for a strong suggestion, `0.75–0.8999` for a review suggestion, and `<0.75` for no suggestion. Thresholds are triage policy, not permission to merge. +- Never auto-merge. Require an authenticated, authorized human to select the survivor and explicitly confirm each merge. Enforce this server-side; a hidden UI control, submitted score, or candidate ID is not authorization. +- Create a tenant-scoped merge snapshot before mutation. It must cover the complete parent/child graph, contacts, domains/websites, evidence and provenance, source-record links/raw lineage, notes, pipeline state, and audit context, plus actor/time, algorithm/normalization versions, score/reasons, and conflict decisions. Protect snapshots from ordinary mutation and make them sufficient for reversal. +- Preserve child, evidence, provenance, and source-record identity by re-parenting or retaining links; never discard a losing observation or silently overwrite conflicting values. A merge or reversal must be transactional or fail closed. +- Audit candidate generation (without leaking cross-tenant data), suggestion/rejection, confirmation, snapshot creation, merge, conflict resolution, and reversal with tenant, actor, target IDs, timestamp, versions, and safe details. Audit reads and snapshots require the same tenant predicate as business reads; cross-tenant candidates are never compared. + +The MVP provides deterministic match suggestions, an explicit human confirmation dialog, a tenant-scoped merge ledger/snapshot, and reversal routes. These are pilot controls, not a compliance-grade merge system: add a dedicated merge permission, server-verifiable confirmation/re-authentication, complete parent/conflict snapshots, transactional concurrency protection, and tamper-evident retention before production. Do not present derived normalization or a suggestion as verified identity, discovery, enrichment, or outreach permission. + ## Phase 5 source security controls Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.