From f1efe39de4f5b7e2a2b9c21d82fa88dd27908376 Mon Sep 17 00:00:00 2001 From: Marco0300 Date: Thu, 3 Sep 2026 10:43:18 +0200 Subject: [PATCH] add conservative domain intelligence --- README.md | 14 ++- apps/api/README.md | 20 ++- apps/api/app/domain_intelligence.py | 174 +++++++++++++++++++++++++++ apps/api/app/main.py | 69 ++++++++++- apps/api/schema.sql | 17 +++ apps/api/tests/test_phase7_domain.py | 92 ++++++++++++++ apps/web/README.md | 6 +- apps/web/app.js | 37 +++++- apps/web/smoke-test.html | 4 + apps/web/styles.css | 2 +- docs/OPERATIONS.md | 12 +- docs/SECURITY.md | 12 ++ 12 files changed, 446 insertions(+), 13 deletions(-) create mode 100644 apps/api/app/domain_intelligence.py create mode 100644 apps/api/tests/test_phase7_domain.py diff --git a/README.md b/README.md index 4227039..7653261 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Prospect Intelligence Platform -A safety-first Phase 6 design/implementation boundary for **manual**, evidence-led prospect qualification and controlled source ingestion. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 6 defines deterministic South African normalization and deduplication review semantics in addition to the Phase 5 source controls; it does **not** enable network discovery. **Automated outreach is disabled, and no live source may be enabled without explicit approval.** +A safety-first Phase 7 design/implementation boundary for **manual**, evidence-led prospect qualification, controlled source ingestion, and domain intelligence review. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 7 defines conservative registrable-domain/PSL and DNS observation semantics in addition to the Phase 6 normalization and deduplication rules; it does **not** enable network discovery. **Automated outreach is disabled, and no live source may be enabled without explicit approval.** ## Included @@ -96,6 +96,18 @@ Every confirmed merge must create a tenant-scoped, immutable-enough merge snapsh The MVP now exposes deterministic match suggestions at `GET /api/v1/businesses/{id}/matches`, explicit merge confirmation in the web review dialog, tenant-scoped merge history, and `POST /api/v1/merge-history/{id}/reverse`. The implementation remains a pilot boundary: hardening is still needed for a dedicated merge permission, stronger server-side confirmation semantics, full snapshot conflict handling, and production-grade rollback guarantees. Do not describe a normalized or suggested match as verified identity, discovery, enrichment, or outreach authorization. +## Phase 7 domain intelligence boundary + +Domain intelligence is an observation and review aid, not proof of business identity, control of a domain, or availability. A domain normalizer may derive a lowercase ASCII/Unicode comparison form and a **registrable domain** using a versioned Public Suffix List (PSL). The PSL is an input with update/version drift: unknown, private, malformed, single-label, localhost, and IP-literal values must remain unresolved rather than guessed. A subdomain is not automatically a separate candidate, and a public suffix itself is never a registrable domain. + +DNS status is explicit: `not_checked`, `pending`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error` describe the check outcome, not a business conclusion. MX, NS, and TXT observations may be absent, partial, truncated, stale, resolver-dependent, or blocked; no record is not proof that mail, delegation, ownership, or a business relationship is absent. Store the resolver/source, observed time, TTL where supplied, and uncertainty/error metadata. Caches must be bounded and keyed by normalized query/type/class plus resolver policy, honor an observed TTL without extending authority, and expose freshness/staleness; cached data must never be presented as a fresh check. + +Association confidence is separate from DNS status and from duplicate score. It must be derived from explainable, tenant-scoped evidence (for example, an operator citation, an exact business-domain observation, or corroborating DNS facts), retain the algorithm/version and uncertainty reasons, and remain suggestion-only. Candidate generation must reject cross-tenant records, public-suffix-only values, malformed or IP-only inputs, and suppressed/merged targets as applicable; it must not auto-attach a domain or infer ownership from a shared, parked, wildcard, sibling-subdomain, homograph, or merely resolvable domain. Every candidate needs human review, provenance, and an auditable accept/reject decision. + +The platform must not claim that a domain is available, unregistered, or safe to acquire without an explicitly authorized availability provider registered with current product/legal/security approval, terms, tenant scope, rate limits, retention, and operational enablement. DNS `nxdomain` or `no_data` is not an availability result. Provider outages, rate limits, stale responses, conflicting results, and unknown status must remain `unknown`/`unavailable`, fail closed, and never trigger purchase, outreach, or automated follow-up. + +Phase 7 remains a documentation/contract boundary in this MVP: there is no live DNS resolver, PSL-backed enrichment worker, cache service, or availability provider in Compose. Production work still includes selecting and versioning the PSL, implementing bounded DNS resolution and TTL-aware cache invalidation, defining MX/NS/TXT parsing and uncertainty retention, adding association review/permission/audit tests, and completing an approved availability-provider integration with SSRF/network egress controls, monitoring, retention, and incident/rollback procedures. + ## Verification ```bash diff --git a/apps/api/README.md b/apps/api/README.md index 487b157..c06a396 100644 --- a/apps/api/README.md +++ b/apps/api/README.md @@ -1,6 +1,6 @@ -# Prospect Platform API — Phase 6 boundary +# Prospect Platform API — Phase 7 boundary -Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 6 normalization/deduplication plus Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach. +Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach. ## Run @@ -18,6 +18,8 @@ Set `PROSPECT_API_PORT` or pass `--port`; set `PROSPECT_API_DB` or pass `--db` t All protected endpoints require the server-side session cookie. Every query is constrained by the authenticated user's `organization_id`; IDs from another tenant behave as not found and must not disclose whether a record exists. +Phase 7 domain routes (all tenant-scoped) are `POST /api/v1/businesses/{id}/domains/check`, `GET /api/v1/businesses/{id}/domains/check?domain=...`, `GET /api/v1/domain-checks`, `GET /api/v1/businesses/{id}/domain-candidates`, and `POST /api/v1/businesses/{id}/domain-candidates/check-availability`. The current implementation is intentionally conservative: a successful address lookup is reported as `ok`, unresolved/empty results as `unknown`, and an availability check returns `unknown`/`not_configured` because no provider is enabled. Treat these as observation states, not ownership or availability claims. + ## Phase 4 jobs and live logging (target contract) The intended job resource has a stable `job_id`, tenant/creator metadata, operation/payload fingerprint, `status`, `attempt`, timestamps, cancellation state, and terminal error/result metadata. Its lifecycle is `queued` → `running` → exactly one terminal state: `succeeded`, `failed`, or `cancelled`. State transitions and worker messages must be persisted transactionally with tenant and job identifiers; terminal jobs are immutable except for controlled retention/redaction. @@ -102,6 +104,18 @@ Remaining limitations: the snapshot currently focuses on the source graph rather List and child-record endpoints are deliberately bounded. For business lists, use `page` (starting at 1) and `page_size` within the server-enforced maximum; invalid values are rejected rather than allowing an unbounded query. Supported filters are applied inside the tenant-scoped query before pagination: `q`, `score_min`, `score_max`, `website_class`, and `pipeline_stage`. The UI's page and filter controls are convenience clients, not authorization controls. A filtered page is not a count of the entire unfiltered tenant unless the response explicitly says so. +## Phase 7 domain-intelligence contract + +Domain observations are tenant-scoped, provenance-bearing inputs. Registrable-domain normalization must use a pinned/versioned PSL rather than a last-two-label heuristic. Preserve the original value and normalized form; return `unknown` when the PSL cannot classify a value (including an unknown/private suffix, public suffix, malformed name, single-label name, localhost, or IP literal). IDN/punycode and case/label handling must be deterministic and must not turn a subdomain into an independent business identity. + +DNS checks must expose an explicit status—`not_checked`, `pending`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`—and never collapse failure or absence into a negative business fact. MX, NS, and TXT are independently uncertain observations: retain record type, normalized response, resolver/source, observed time, TTL if supplied, truncation/partial indicators, and error/uncertainty reason. A missing MX does not prove that email is unavailable; an NS result does not prove control; TXT content does not prove ownership. + +Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class and resolver policy, and TTL-aware. Do not extend authority beyond the received TTL; expose `observed_at`, `expires_at`/freshness, and stale or refresh state. A cache hit is not a fresh lookup, and resolver policy/PSL version changes require invalidation or re-evaluation. There is no DNS resolver or cache service in the current runtime. + +Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification. + +Domain availability is `unknown` unless an explicitly authorized availability provider supplies it. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented. + ## Remaining limitations and production migration work -SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Production migration work must add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies. +SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Phase 7 production work must add a reviewed PSL update/version policy, bounded DNS workers, TTL-aware cache storage/invalidation, uncertainty-aware MX/NS/TXT schema and tests, association candidate safeguards and permissions, and an authorized availability provider with egress/rate/circuit/retention controls. It must also add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies. diff --git a/apps/api/app/domain_intelligence.py b/apps/api/app/domain_intelligence.py new file mode 100644 index 0000000..f0a5c71 --- /dev/null +++ b/apps/api/app/domain_intelligence.py @@ -0,0 +1,174 @@ +"""Conservative, dependency-free domain intelligence helpers. + +This module deliberately reports uncertainty rather than inferring DNS or registrar facts. +""" +from __future__ import annotations + +import ipaddress +import re +import socket +import threading +import time +import unicodedata +from datetime import datetime, timezone +from difflib import SequenceMatcher +from urllib.parse import urlparse + +# Small explicitly maintained PSL subset. Unknown suffixes are never guessed. +PUBLIC_SUFFIXES = frozenset({ + "com", "org", "net", "za", "co.za", "org.za", "net.za", "ac.za", "gov.za", "edu.za", +}) +DEFAULT_TTL_SECONDS = 300 +MAX_DOMAIN_LENGTH = 253 +MAX_CANDIDATES = 20 + + +def _host(value: str | None) -> str: + raw = str(value or "").strip().lower() + if not raw: + return "" + parsed = urlparse(raw if "://" in raw else "//" + raw) + host = (parsed.hostname or "").rstrip(".").lower() + if host.startswith("www."): + host = host[4:] + return host + + +def _valid_hostname(host: str) -> bool: + if not host or len(host) > MAX_DOMAIN_LENGTH or "." not in host: + return False + try: + ipaddress.ip_address(host) + return False + except ValueError: + pass + if any(len(label) > 63 or not re.fullmatch(r"[a-z0-9](?:[a-z0-9-]*[a-z0-9])?", label) for label in host.split(".")): + return False + return True + + +def normalize_registrable_domain(value: str | None) -> str: + """Return the registrable domain, or ``unknown`` for unsupported/unsafe input.""" + host = _host(value) + if not _valid_hostname(host): + return "unknown" + labels = host.split(".") + suffix = next((".".join(labels[-n:]) for n in (3, 2, 1) if ".".join(labels[-n:]) in PUBLIC_SUFFIXES), None) + if not suffix or len(labels) <= suffix.count(".") + 1: + return "unknown" + return ".".join(labels[-(suffix.count(".") + 2):]) + + +def domain_info(value: str | None) -> dict: + host = _host(value) + registered = normalize_registrable_domain(value) + return {"input": str(value or ""), "hostname": host, "registrable_domain": registered, + "status": "ok" if registered != "unknown" else "unknown"} + + +def _metadata(ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict: + now = time.time() + checked = datetime.fromtimestamp(now, timezone.utc).replace(microsecond=0).isoformat() + return {"checked_at": checked, "ttl_seconds": ttl_seconds, + "cache_expires_at": datetime.fromtimestamp(now + ttl_seconds, timezone.utc).replace(microsecond=0).isoformat()} + + +def resolve_domain(domain: str, *, timeout: float = 3.0, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict: + """Resolve only a validated hostname using stdlib socket; never fetches a URL.""" + normalized = normalize_registrable_domain(domain) + host = _host(domain) + result = {"domain": normalized, "hostname": host, "status": "unknown", "addresses": [], **_metadata(ttl_seconds)} + if normalized == "unknown" or not _valid_hostname(host): + return result + resolver = resolver or socket.getaddrinfo + try: + # getaddrinfo has no per-call timeout. Run it in a daemon worker so a + # resolver stall cannot block an HTTP handler indefinitely. + outcome = {} + def lookup(): + try: outcome["records"] = resolver(host, None, 0, socket.SOCK_STREAM) + except BaseException as exc: outcome["exception"] = exc + worker = threading.Thread(target=lookup, daemon=True) + worker.start(); worker.join(max(0.01, float(timeout))) + if worker.is_alive(): + result["status"] = "timeout" + return result + if "exception" in outcome: raise outcome["exception"] + records = outcome.get("records", []) + addresses = sorted({str(item[4][0]) for item in records if item and len(item) > 4 and item[0] in (socket.AF_INET, socket.AF_INET6)}) + result["addresses"] = addresses + result["status"] = "ok" if addresses else "unknown" + except socket.gaierror as exc: + code = getattr(exc, "errno", None) + if code is None and exc.args: code = exc.args[0] + result["status"] = "nxdomain" if code in (socket.EAI_NONAME, socket.EAI_NODATA, -2, -5) else "error" + result["error_code"] = code + except (TimeoutError, socket.timeout): + result["status"] = "timeout" + except OSError as exc: + result["status"] = "error"; result["error_code"] = exc.__class__.__name__ + return result + + +def capability_hook(domain: str, capability: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict: + """MX/NS/TXT hook. No optional resolver means not_configured, not empty.""" + normalized = normalize_registrable_domain(domain) + base = {"domain": normalized, "capability": capability, "status": "unknown", "records": [], **_metadata(ttl_seconds)} + if normalized == "unknown": return base + if capability not in {"mx", "ns", "txt"}: base.update(status="error", error_code="unsupported_capability"); return base + if resolver is None: + base.update(status="not_configured", reason="resolver_not_configured") + return base + try: + records = resolver(normalized, capability) + base.update(status="ok" if records else "unknown", records=list(records or [])) + except TimeoutError: base["status"] = "timeout" + except socket.gaierror: base["status"] = "nxdomain" + except Exception as exc: base.update(status="error", error_code=exc.__class__.__name__) + return base + + +def resolve_mx(domain: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict: + return capability_hook(domain, "mx", resolver=resolver, ttl_seconds=ttl_seconds) + + +def resolve_ns(domain: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict: + return capability_hook(domain, "ns", resolver=resolver, ttl_seconds=ttl_seconds) + + +def resolve_txt(domain: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict: + return capability_hook(domain, "txt", resolver=resolver, ttl_seconds=ttl_seconds) + + +registrable_domain = normalize_registrable_domain + + +def association_confidence(candidate_domain: str, observed_domain: str, *, business_name: str = "") -> dict: + candidate = normalize_registrable_domain(candidate_domain); observed = normalize_registrable_domain(observed_domain) + if candidate == "unknown" or observed == "unknown": return {"level":"unknown", "score":0.0, "reasons":["unsupported_domain"]} + if candidate == observed: return {"level":"high", "score":1.0, "reasons":["same_registrable_domain"]} + token = re.sub(r"[^a-z0-9]", "", unicodedata.normalize("NFKD", str(business_name).lower())) + score = SequenceMatcher(None, token, candidate.split(".")[0]).ratio() if token else 0.0 + return {"level":"medium" if score >= .8 else "low", "score":round(score, 4), "reasons":["name_domain_similarity"] if score >= .8 else ["different_registrable_domain"]} + + +def _slug(value: object) -> str: + text = unicodedata.normalize("NFKD", str(value or "")).encode("ascii", "ignore").decode().lower() + words = re.findall(r"[a-z0-9]+", text) + return "-".join(words)[:40].strip("-") + + +def generate_candidate_domains(business_name: str, service: str = "", location: str = "", *, suffixes=("co.za", "com"), limit: int = MAX_CANDIDATES) -> list[str]: + limit = max(0, min(int(limit), MAX_CANDIDATES)); name, svc, loc = _slug(business_name), _slug(service), _slug(location) + if not name: return [] + parts = [name, f"{name}-{svc}" if svc else "", f"{name}-{loc}" if loc else "", f"{svc}-{loc}" if svc and loc else ""] + if svc: parts += [f"{name}{svc}"] + out=[] + for base in parts: + if not base or len(base) > 63: continue + for suffix in suffixes: + if suffix not in PUBLIC_SUFFIXES: continue + domain = f"{base}.{suffix}" + if normalize_registrable_domain(domain) != "unknown" and domain not in out: out.append(domain) + if len(out) >= limit: return out + return out diff --git a/apps/api/app/main.py b/apps/api/app/main.py index 32b1b7e..da6adc3 100644 --- a/apps/api/app/main.py +++ b/apps/api/app/main.py @@ -9,15 +9,17 @@ if __package__ in (None, ""): sys.path.insert(0, str(Path(__file__).resolve().parents[1])) from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses from app.sources import adapter_for, contains_secret + from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains else: from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses from .sources import adapter_for, contains_secret + from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains ORGANIZATION_ID = "demo-tenant" SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql" SESSION_DAYS = 7 PBKDF2_ITERATIONS = 300_000 MUTATING_ROLES = {"owner", "admin", "researcher"} -JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery"} +JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"} JOB_PAGE_SIZE = 100 SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"} CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)} @@ -117,16 +119,79 @@ class ApiHandler(BaseHTTPRequestHandler): if path=="/api/v1/discovery-queries": return self.list_queries(db,org) if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query)) if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query)) + if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query)) if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query)) if path.startswith("/api/v1/businesses/"): bits=path.split("/"); ident=bits[4] if len(bits)>4 else "" if not ident.isdigit(): return self.send_json(404,{"error":"not_found"}) row=self.business(db,int(ident),org) if not row:return self.send_json(404,{"error":"not_found"}) + if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query)) + if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org) if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org) payload=row_json(row); payload.update(self.nested(db,int(ident),org)); return self.send_json(200,payload) return self.send_json(404,{"error":"not_found"}) finally: db.close() + def _domain_result(self, row, cache_hit=False): + try: result=json.loads(row["result_json"] or "{}") + except (TypeError,ValueError): result={} + result.update({"id":row["id"],"business_id":row["business_id"],"domain":row["domain"],"status":row["status"],"cache_hit":cache_hit,"checked_at":row["checked_at"],"cache_expires_at":row["cache_expires_at"]}) + return result + + def _check_domain(self, bid, domain, db, user): + normalized=normalize_registrable_domain(domain) + if normalized == "unknown": return self.send_json(400,{"error":"unsupported_domain","status":"unknown"}) + now=datetime.now(timezone.utc).replace(microsecond=0).isoformat() + cache_key=normalized+":a_aaaa:v1" + cached=db.execute("SELECT * FROM domain_checks WHERE organization_id=? AND business_id=? AND cache_key=? AND cache_expires_at>? ORDER BY id DESC LIMIT 1",(user["organization_id"],bid,cache_key,now)).fetchone() + if cached:return self.send_json(200,self._domain_result(cached,True)) + result=resolve_domain(normalized) + try: cur=db.execute("INSERT INTO domain_checks(organization_id,business_id,domain,status,result_json,cache_key,checked_at,cache_expires_at) VALUES(?,?,?,?,?,?,?,?)",(user["organization_id"],bid,normalized,result["status"],json.dumps(result,sort_keys=True),cache_key,result["checked_at"],result["cache_expires_at"])) + except sqlite3.IntegrityError: + existing=db.execute("SELECT id FROM domain_checks WHERE organization_id=? AND business_id=? AND cache_key=?",(user["organization_id"],bid,cache_key)).fetchone() + if not existing: return self.send_json(409,{"error":"domain_check_conflict"}) + db.execute("UPDATE domain_checks SET domain=?,status=?,result_json=?,checked_at=?,cache_expires_at=? WHERE id=?",(normalized,result["status"],json.dumps(result,sort_keys=True),result["checked_at"],result["cache_expires_at"],existing["id"])) + self.audit(db,user,"domain.checked",f"{bid}:{normalized}:{result['status']}"); db.commit() + return self.send_json(200,self._domain_result(db.execute("SELECT * FROM domain_checks WHERE id=?",(existing["id"],)).fetchone())) + self.audit(db,user,"domain.checked",f"{bid}:{normalized}:{result['status']}"); db.commit() + return self.send_json(200,self._domain_result(db.execute("SELECT * FROM domain_checks WHERE id=?",(cur.lastrowid,)).fetchone())) + + def post_domain_check(self,bid,payload,db,user): + if not self.business(db,bid,user["organization_id"]): return self.send_json(404,{"error":"not_found"}) + domain=payload.get("domain") or self.business(db,bid,user["organization_id"])["website_domain"] + if not str(domain).strip(): return self.send_json(400,{"error":"domain_required"}) + return self._check_domain(bid,str(domain),db,user) + + def get_domain_check(self,bid,db,user,query): + domain=(query.get("domain") or [""])[0] + if not domain:return self.send_json(400,{"error":"domain_required"}) + return self._check_domain(bid,domain,db,user) + + def list_domain_checks(self,db,org,query): + try: limit=max(1,min(int((query.get("page_size") or [50])[0]),100)); offset=max(0,int((query.get("offset") or [0])[0])) + except (ValueError,TypeError): return self.send_json(400,{"error":"invalid_pagination"}) + rows=db.execute("SELECT * FROM domain_checks WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall() + return self.send_json(200,{"organization_id":org,"items":[self._domain_result(r) for r in rows[:limit]],"limit":limit,"offset":offset,"has_more":len(rows)>limit}) + + def list_domain_candidates(self,bid,db,org): + business=self.business(db,bid,org) + if not business:return self.send_json(404,{"error":"not_found"}) + rows=db.execute("SELECT * FROM domain_candidates WHERE organization_id=? AND business_id=? ORDER BY rank,id",(org,bid)).fetchall() + if not rows: + generated=generate_candidate_domains(business["name"],business["description"],business["city"]) + for rank,domain in enumerate(generated,1): + db.execute("INSERT OR IGNORE INTO domain_candidates(organization_id,business_id,domain,rank) VALUES(?,?,?,?)",(org,bid,domain,rank)) + db.commit(); rows=db.execute("SELECT * FROM domain_candidates WHERE organization_id=? AND business_id=? ORDER BY rank,id",(org,bid)).fetchall() + return self.send_json(200,{"business_id":bid,"items":[row_json(r) for r in rows]}) + + def check_availability(self,bid,payload,db,user): + if not self.business(db,bid,user["organization_id"]):return self.send_json(404,{"error":"not_found"}) + candidates=db.execute("SELECT domain FROM domain_candidates WHERE organization_id=? AND business_id=? ORDER BY rank,id",(user["organization_id"],bid)).fetchall() + domains=[r["domain"] for r in candidates] + if isinstance(payload.get("domains"),list): domains=[normalize_registrable_domain(x) for x in payload["domains"] if normalize_registrable_domain(x)!="unknown"][:20] + self.audit(db,user,"domain.availability.checked",str(bid)); db.commit() + return self.send_json(200,{"business_id":bid,"status":"unknown","reason":"not_configured","provider_configured":False,"items":[{"domain":d,"status":"unknown","reason":"not_configured"} for d in domains]}) + def list_jobs(self, db, org, query): try: limit=max(1,min(int(query.get("page_size",[50])[0]),JOB_PAGE_SIZE)); offset=max(0,int(query.get("offset",[0])[0])) except (ValueError, TypeError): return self.send_json(400,{"error":"invalid_pagination"}) @@ -221,6 +286,8 @@ class ApiHandler(BaseHTTPRequestHandler): if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user) if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user) if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org) + if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user) + if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user) if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"): ident=path.split("/")[4] return self.reverse_merge(int(ident) if ident.isdigit() else -1,db,user) diff --git a/apps/api/schema.sql b/apps/api/schema.sql index c4276a2..11d4763 100644 --- a/apps/api/schema.sql +++ b/apps/api/schema.sql @@ -153,3 +153,20 @@ CREATE TABLE IF NOT EXISTS source_records ( UNIQUE(organization_id,source_id,content_hash) ); CREATE INDEX IF NOT EXISTS idx_source_records_org ON source_records(organization_id,id DESC); + +-- Phase 7 domain intelligence (additive-safe; DNS results are explicitly stateful). +CREATE TABLE IF NOT EXISTS domain_checks ( + id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), + business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE, domain TEXT NOT NULL, + status TEXT NOT NULL, result_json TEXT NOT NULL DEFAULT '{}', cache_key TEXT NOT NULL, + checked_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, cache_expires_at TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(organization_id, business_id, cache_key) +); +CREATE INDEX IF NOT EXISTS idx_domain_checks_org ON domain_checks(organization_id,created_at DESC,id DESC); +CREATE INDEX IF NOT EXISTS idx_domain_checks_business ON domain_checks(organization_id,business_id,created_at DESC); +CREATE TABLE IF NOT EXISTS domain_candidates ( + id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE, + domain TEXT NOT NULL, source TEXT NOT NULL DEFAULT 'generated', rank INTEGER NOT NULL DEFAULT 0, metadata_json TEXT NOT NULL DEFAULT '{}', + created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,business_id,domain) +); +CREATE INDEX IF NOT EXISTS idx_domain_candidates_business ON domain_candidates(organization_id,business_id,rank,id); diff --git a/apps/api/tests/test_phase7_domain.py b/apps/api/tests/test_phase7_domain.py new file mode 100644 index 0000000..5a2dd3f --- /dev/null +++ b/apps/api/tests/test_phase7_domain.py @@ -0,0 +1,92 @@ +import json +import os +import sqlite3 +import threading +import unittest +from http.client import HTTPConnection +from tempfile import TemporaryDirectory +from unittest.mock import patch + +from app.domain_intelligence import ( + normalize_registrable_domain, resolve_domain, generate_candidate_domains, + association_confidence, resolve_mx, +) +from app.main import create_server + + +class DomainIntelligenceTests(unittest.TestCase): + def test_psl_normalization_and_unknown_suffix(self): + self.assertEqual(normalize_registrable_domain('https://WWW.shop.example.co.za/path'), 'example.co.za') + self.assertEqual(normalize_registrable_domain('foo.example.com'), 'example.com') + self.assertEqual(normalize_registrable_domain('foo.example.invalidtld'), 'unknown') + + def test_candidate_generation_is_bounded_and_safe(self): + out = generate_candidate_domains('Acme & Sons (Pty) Ltd', 'Solar Panels', 'Cape Town') + self.assertLessEqual(len(out), 20) + self.assertTrue(out) + for value in out: + self.assertRegex(value, r'^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.(?:co\.za|com)$') + self.assertNotIn('--', value) + + def test_resolution_distinguishes_nxdomain_timeout_and_error(self): + self.assertEqual(resolve_domain('missing.example.com', resolver=lambda *a: (_ for _ in ()).throw(socket_gaierror_name()))['status'], 'nxdomain') + self.assertEqual(resolve_domain('slow.example.com', resolver=lambda *a: (_ for _ in ()).throw(TimeoutError()))['status'], 'timeout') + self.assertEqual(resolve_domain('bad.example.com', resolver=lambda *a: (_ for _ in ()).throw(OSError('boom')))['status'], 'error') + + def test_local_resolution_and_association_confidence(self): + with patch('app.domain_intelligence.socket.getaddrinfo', return_value=[(2, 1, 6, '', ('1.2.3.4', 0))]): + result = resolve_domain('example.com') + self.assertEqual(result['status'], 'ok') + self.assertEqual(result['addresses'], ['1.2.3.4']) + self.assertEqual(association_confidence('acme.co.za', 'acme.co.za')['level'], 'high') + self.assertEqual(association_confidence('acme.co.za', 'other.co.za')['level'], 'low') + + def test_optional_capabilities_are_not_falsely_empty(self): + self.assertEqual(resolve_mx('example.com')['status'], 'not_configured') + + +def socket_gaierror_name(): + import socket + return socket.gaierror(socket.EAI_NONAME) + + +class DomainApiTests(unittest.TestCase): + def setUp(self): + self.tmp = TemporaryDirectory(); self.db_path = self.tmp.name + '/db.sqlite' + os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'domain-owner@example.test'; os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password' + self.server = create_server('127.0.0.1', 0, self.db_path); self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start() + self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None + self.request('POST', '/api/v1/auth/login', {'email':'domain-owner@example.test','password':'password'}) + + def tearDown(self): + self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup() + + def request(self, method, path, payload=None): + body = json.dumps(payload).encode() if payload is not None else None; headers = {'Content-Type':'application/json'} if body else {} + if self.cookie: headers['Cookie'] = self.cookie + self.conn.request(method, path, body, headers); response = self.conn.getresponse(); c = response.getheader('Set-Cookie') + if c: self.cookie = c.split(';', 1)[0] + return response.status, json.loads(response.read() or b'{}') + + def test_check_cache_candidates_and_no_false_availability(self): + status, business = self.request('POST', '/api/v1/businesses', {'name':'Acme Solar','website':'https://example.com','city':'Cape Town','province':'Western Cape'}) + self.assertEqual(status, 201); bid = business['id'] + with patch('app.main.resolve_domain', return_value={'status':'unknown','addresses':[],'checked_at':'2026-01-01T00:00:00+00:00','cache_expires_at':'2099-01-01T00:00:00+00:00'}): + status, first = self.request('POST', f'/api/v1/businesses/{bid}/domains/check', {'domain':'example.com'}) + self.assertEqual(status, 200); self.assertIn(first['status'], ('unknown','ok')) + status, second = self.request('GET', f'/api/v1/businesses/{bid}/domains/check?domain=example.com') + self.assertEqual(status, 200); self.assertTrue(second.get('cache_hit')) + status, candidates = self.request('GET', f'/api/v1/businesses/{bid}/domain-candidates') + self.assertEqual(status, 200); self.assertTrue(candidates['items']) + status, availability = self.request('POST', f'/api/v1/businesses/{bid}/domain-candidates/check-availability', {}) + self.assertEqual(status, 200); self.assertEqual(availability['status'], 'unknown'); self.assertEqual(availability['reason'], 'not_configured') + self.assertEqual(self.request('GET', '/api/v1/domain-checks')[0], 200) + + def test_business_domain_checks_are_tenant_scoped(self): + _, business = self.request('POST', '/api/v1/businesses', {'name':'Private'}) + self.cookie = None + self.request('POST', '/api/v1/auth/logout') + self.assertEqual(self.request('GET', f'/api/v1/businesses/{business["id"]}/domains/check?domain=example.com')[0], 401) + + +if __name__ == '__main__': unittest.main() diff --git a/apps/web/README.md b/apps/web/README.md index 15203a9..c0e75c1 100644 --- a/apps/web/README.md +++ b/apps/web/README.md @@ -1,6 +1,6 @@ -# ProspectOS web — Phase 6 boundary +# ProspectOS web — Phase 7 boundary -Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts and Phase 6 normalization/deduplication concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach. +Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts, Phase 6 normalization/deduplication, and Phase 7 domain-intelligence concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach. ## Configure and run @@ -62,4 +62,4 @@ A zero-dependency static smoke page (`smoke-test.html`) checks the key DOM contr ## Remaining limitations -The static client has no background discovery, DNS/website scanner, enrichment scheduler, outreach integration, or SSE delivery. It cannot make missing provenance authoritative and should display API-provided limitations rather than infer them. CSV preview is capped for display and is not an import workflow. Production deployment still requires durable job execution, tenant-scoped controls, idempotency verification, SSE/polling verification, and the security and operations gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`. +The static client has no background discovery, DNS/website scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. For future domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability; any availability label requires an API result from an authorized provider and explicit human review. It cannot make missing provenance authoritative and should display API-provided limitations rather than infer them. CSV preview is capped for display and is not an import workflow. Production deployment still requires durable job execution, tenant-scoped controls, idempotency verification, PSL/DNS/cache implementation and tests, candidate review permissions/audit, authorized availability-provider controls, SSE/polling verification, and the security and operations gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`. diff --git a/apps/web/app.js b/apps/web/app.js index c40132b..ab5e45b 100644 --- a/apps/web/app.js +++ b/apps/web/app.js @@ -12,7 +12,7 @@ const freshness = (p) => { const raw=p.updated_at||p.last_checked_at||p.created_at; if(!raw)return {label:'Unknown',cls:'stale'}; const days=Math.max(0,Math.floor((Date.now()-new Date(raw).getTime())/86400000)); return {label:days===0?'Today':`${days}d ago`,cls:days<=7?'good':'stale'}; }; const scoreFor = (p) => Number.isFinite(Number(p.score)) ? Number(p.score) : (p.score_factors||p.factors||[]).reduce((n,f)=>n+({named_business:20,business_site:30,email:25,phone:15,description:10}[f]||0),0); const labelFactor = (f) => ({named_business:'Named business',business_site:'Business site',email:'Email found',phone:'Phone found',description:'Description'}[f] || String(f).replaceAll('_',' ')); - async function request(path, options = {}) { const response=await fetch(endpoint(path),{...options,credentials:'include'}); if(response.status===401){showLogin('Your session has expired. Please sign in again.');throw new Error('unauthorized');} return response; } + async function request(path, options = {}) { const response=await fetch(endpoint(path),{...options,credentials:'include'}); if(response.status===401){showLogin('Your session has expired. Please sign in again.');throw new Error('unauthorized');} if(response.status===403)throw new Error('Tenant/workspace access denied.'); return response; } async function jsonRequest(path, options = {}) { const res=await request(path,options); const body=await res.json().catch(()=>({})); if(!res.ok)throw new Error(body.error||body.message||'Request failed'); return body; } function showLogin(message=''){currentUser=null;$('dashboardShell').hidden=true;$('loginScreen').hidden=false;$('loginMessage').textContent=message;$('loginMessage').className=`form-message${message?' error':''}`;} function showDashboard(user){currentUser=user||{};const name=currentUser.name||currentUser.full_name||currentUser.email||'Workspace member';const role=currentUser.role||currentUser.roles?.[0]||'Member';$('userIdentity').textContent=`${name} · ${role}`;$('userAvatar').textContent=name.split(/\s+/).map(x=>x[0]).join('').slice(0,2).toUpperCase();$('loginScreen').hidden=true;$('dashboardShell').hidden=false;updateJobPermissions();} @@ -25,7 +25,38 @@ async function selectProspect(id){selectedId=Number(id);selectedDetail=null;renderRows();$('detailPanel').innerHTML='
Loading prospect detail…
';await loadDetail(selectedId);} async function loadDetail(id){try{const detail=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}`);selectedDetail=detail;const index=prospects.findIndex(p=>Number(p.id)===Number(id));if(index>=0)prospects[index]={...prospects[index],...detail};renderDetail(detail);}catch(error){if(error.message!=='unauthorized')$('detailPanel').innerHTML=``;}} const listItems=(items,empty,label)=>Array.isArray(items)&&items.length?``:`

${empty}

`; - function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`

PROSPECT DETAIL

${esc(p.name)}

${esc(p.website_domain||'no detected website')}

${esc(review)}
Fit score${s}/ 100
${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence

Pipeline stage

Contacts ${contacts.length}

${listItems(contacts,'No contacts added.','email')}

Domains & websites

${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}

Evidence timeline

${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`

✓ ${esc(labelFactor(x))}${esc(p.confidence||'Medium')}

`).join(''):''}

Notes ${notes.length}

${listItems(notes,'No notes added.','body')}

Review status

${esc(review)}

${st!=='suppressed'?'':''}

${blocked?`

${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}

`:''}`;renderDedupPanel();} + const domainValue = item => typeof item === 'string' ? item : item?.domain || item?.hostname || item?.name || item?.value || ''; + const domainForProspect = p => p?.website_domain || (Array.isArray(p?.domains) ? domainValue(p.domains[0]) : '') || ''; + const domainState = value => { const state=String(value ?? '').toLowerCase().replaceAll('_','-'); return state==='nxdomain'?'NXDOMAIN':state==='error'?'Error':state==='not-configured'?'Not configured':state==='unknown'||!state?'Unknown':state.replaceAll('-',' '); }; + const capabilityState = value => { if (value === true || ['present','found','yes','true','ok'].includes(String(value).toLowerCase())) return 'Present'; if (value === false || ['absent','not-found','no','false'].includes(String(value).toLowerCase())) return 'Not found'; return domainState(value); }; + const checkedTime = item => item?.checked_at || item?.checkedAt || item?.last_checked_at || item?.checked_time || 'Time unavailable'; + const cacheState = item => item?.cache_hit === true ? 'Hit' : item?.cache_state || item?.cacheStatus || item?.cache || (item?.cache_hit === false ? 'Miss' : 'Unknown'); + const confidenceText = item => item?.confidence ?? item?.score ?? 'Unknown'; + const reasonsText = item => { const reasons=item?.reasons || item?.reason || item?.explanation || []; return Array.isArray(reasons) ? reasons : [reasons]; }; + function renderDomainPanel(p){ + const panel=$('domainIntelligencePanel'); if(!panel)return; + panel.innerHTML=`

DOMAIN INTELLIGENCE

DNS checks

DNS results are evidence only. Unknown, error, and NXDOMAIN states are never treated as ownership or availability.

Loading DNS results…
Loading candidate domains…
`; + loadDomainIntelligence(p.id); + } + function renderDomainCheck(result){ + const el=$('domainCheckState'); if(!el)return; + if(!result){el.innerHTML='

No DNS check has been returned. Run a check to see evidence.

';return;} + const dns=result.dns || result.dns_state || result.state || result.status || 'unknown', records=result.records || result.capabilities || result; + const capability=(key, aliases=[])=>{for(const k of [key,...aliases])if(records?.[k]!==undefined)return records[k];return undefined;}; + el.innerHTML=`
${esc(domainState(dns))}Checked ${esc(checkedTime(result))}Cache: ${esc(domainState(cacheState(result)))}
${['A','AAAA','MX','NS','TXT'].map(type=>`
${type}
${esc(capabilityState(capability(type,type.toLowerCase())))}
`).join('')}
${result.error||result.error_code||result.message?``:''}`; + } + function renderCandidates(payload){ + const panel=$('candidateDomainPanel'); if(!panel)return; + const candidates=payloadItems(payload,['candidates','candidate_domains','domains','items']); + panel.innerHTML=`

CANDIDATE DOMAINS

Possible matches ${candidates.length}

Human review required
${candidates.length?`
${candidates.map((item,index)=>{const domain=domainValue(item);return `
${esc(domain||'Unknown domain')}${esc(confidenceText(item))} confidence
    ${reasonsText(item).map(reason=>`
  • ${esc(typeof reason==='string'?reason:reason?.text||reason?.description||JSON.stringify(reason))}
    • `).join('')||'
  • Reason unavailable
    • '}

`;}).join('')}
`:'

No candidate domains returned.

'}

Availability is not configured here. This action will never claim a domain is available or owned.

`; + } + async function loadDomainIntelligence(id){ + const check=$('domainCheckState'), candidates=$('candidateDomainPanel'); + try{const [checkPayload,candidatePayload]=await Promise.all([jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/domains/check?domain=${encodeURIComponent(domainForProspect(selectedDetail || prospects.find(item=>Number(item.id)===Number(id)) || {}))}`),jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/domain-candidates`)]);renderDomainCheck(checkPayload);renderCandidates(candidatePayload);}catch(error){if(error.message!=='unauthorized'){if(check)check.innerHTML=``;if(candidates)candidates.innerHTML='

Candidate domains unavailable.

';}} + } + async function runDomainCheck(){const button=$('runDomainCheckBtn');if(!selectedId||!button)return;button.disabled=true;const state=$('domainCheckState');if(state)state.innerHTML='
Checking DNS…
';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domains/check`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain:domainForProspect(selectedDetail||{})})});renderDomainCheck(result);}catch(error){if(error.message!=='unauthorized'&&state)state.innerHTML=``;}finally{button.disabled=false;}} + async function checkDomainAvailability(domain,button){const result=button.closest('.candidate-card')?.querySelector('.availability-result');if(!domain||!result)return;button.disabled=true;result.textContent='Checking availability…';try{const body=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domain-candidates/check-availability`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain})});const state=body?.status||body?.state||body?.availability||'unknown';result.textContent=['not-configured','unknown','error'].includes(String(state).toLowerCase())?`Availability: ${domainState(state)}. No ownership conclusion.`:'Availability: Unknown. No ownership conclusion.';}catch(error){if(error.message!=='unauthorized')result.textContent=`Availability: Unknown. ${error.message}`;}finally{button.disabled=false;}} + function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`

PROSPECT DETAIL

${esc(p.name)}

${esc(p.website_domain||'no detected website')}

${esc(review)}
Fit score${s}/ 100
${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence

Pipeline stage

Contacts ${contacts.length}

${listItems(contacts,'No contacts added.','email')}

Loading domain intelligence…

Domains & websites

${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}

Evidence timeline

${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`

✓ ${esc(labelFactor(x))}${esc(p.confidence||'Medium')}

`).join(''):''}

Notes ${notes.length}

${listItems(notes,'No notes added.','body')}

Review status

${esc(review)}

${st!=='suppressed'?'':''}

${blocked?`

${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}

`:''}`;renderDomainPanel(p);renderDedupPanel();} let mergeSource = null, mergeTarget = null, mergeBusy = false; const payloadItems = (payload, keys) => { for (const key of keys) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; }; const suggestionId = item => item.target_id ?? item.business_id ?? item.prospect_id ?? item.id; @@ -112,7 +143,7 @@ async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}} async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}} document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);}); - document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);}); + document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);}); $('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView())); bootstrap(); })(); diff --git a/apps/web/smoke-test.html b/apps/web/smoke-test.html index 623b5e5..c01474f 100644 --- a/apps/web/smoke-test.html +++ b/apps/web/smoke-test.html @@ -33,5 +33,9 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j ,['Deduplication review UI contract',()=>!!d.querySelector('#mergeDialog')&&js.includes('/matches')&&js.includes('review-required')&&js.includes('confidence')&&js.includes('reasons')] ,['Merge actions are explicit and reversible',()=>js.includes('/merge-history')&&js.includes('/reverse')&&js.includes('This action is reversible')&&js.includes('Confirm merge')&&!js.includes('autoMerge')] ,['Deduplication loading and errors',()=>js.includes('Loading match suggestions')&&js.includes('Unable to load match suggestions')&&js.includes('merge-history')&&js.includes('Unable to load merge history')] + ,['Domain intelligence smoke marker and controls',()=>!!d.querySelector('[data-smoke="domain-intelligence"]')&&js.includes('/domains/check')&&js.includes('/domain-candidates')&&js.includes('runDomainCheckBtn')] + ,['DNS capability and uncertainty labels',()=>['DNS checks','A','AAAA','MX','NS','TXT','NXDOMAIN','Unknown','Error','Cache'].every(x=>js.includes(x))] + ,['Candidate confidence, reasons, and safe availability',()=>js.includes('confidence')&&js.includes('reasons')&&js.includes('check-availability')&&js.includes('Not configured')&&js.includes('No ownership conclusion')&&!js.includes('Domain is available')] + ,['Domain loading, error, tenant, and auth states',()=>js.includes('Loading DNS results')&&js.includes('Unable to load DNS results')&&js.includes('Tenant/workspace access denied')&&js.includes("credentials:'include'")] ];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `
  • ${ok?'PASS':'FAIL'} — ${name}
    • `}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;}; diff --git a/apps/web/styles.css b/apps/web/styles.css index 9b828f1..e44a53e 100644 --- a/apps/web/styles.css +++ b/apps/web/styles.css @@ -1,6 +1,6 @@ :root{--ink:#172033;--muted:#6d7890;--line:#e7eaf1;--surface:#fff;--bg:#f7f8fb;--violet:#6756e8;--violet-soft:#efedff;--green:#16845b;--green-soft:#e5f7ef;--amber:#b87513;--amber-soft:#fff3dd;--red:#b84d55;--red-soft:#fff0f1;--shadow:0 10px 30px rgba(33,36,75,.05)}*{box-sizing:border-box}html{scroll-behavior:smooth}body{margin:0;background:var(--bg);color:var(--ink);font:14px/1.45 Inter,ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif}.app-shell{display:flex;min-height:100vh}.sidebar{width:238px;background:#17152e;color:#e5e4f2;display:flex;flex-direction:column;padding:28px 16px;position:fixed;inset:0 auto 0 0}.brand{color:#fff;display:flex;align-items:center;gap:10px;text-decoration:none;font-size:20px;font-weight:750;padding:0 14px 44px;letter-spacing:-.5px}.brand-mark{width:27px;height:27px;border-radius:8px;background:#7263f3;display:grid;place-items:center;font-size:16px}.brand-light{font-weight:400;color:#a7a5c1}.nav-item{display:flex;align-items:center;gap:13px;color:#a8a7bd;text-decoration:none;padding:12px 15px;border-radius:9px;margin:3px 0}.nav-item span{font-size:20px;width:18px;text-align:center}.nav-item.active,.nav-item:hover{color:#fff;background:#2a2749}.sidebar-foot{margin-top:auto;border-top:1px solid #302d4b;padding:20px 14px 4px;display:flex;gap:9px;align-items:flex-start;font-size:12px}.sidebar-foot small{display:block;color:#85839e;margin-top:3px}.live-dot{background:#45d99c;width:7px;height:7px;border-radius:50%;margin-top:5px;box-shadow:0 0 0 4px #23463d}.main{margin-left:238px;flex:1;min-width:0}.topbar{height:76px;background:#fff;border-bottom:1px solid var(--line);display:flex;align-items:center;justify-content:space-between;padding:0 4.5%;color:var(--muted)}.crumb span{padding:0 10px;color:#c2c5ce}.top-actions{display:flex;align-items:center;gap:22px}.api-status{color:#7d8494;font-size:12px}.api-status.live{color:var(--green)}.icon-button{border:0;background:none;color:#7c8497;font-size:21px;cursor:pointer}.avatar{width:33px;height:33px;border-radius:50%;display:grid;place-items:center;background:#e5e2ff;color:#5648c8;font-weight:700;font-size:11px}.content{max-width:1450px;margin:auto;padding:40px 4.5% 28px}.hero{display:flex;justify-content:space-between;align-items:end;margin-bottom:30px}.eyebrow{color:#8d94a4;font-size:10px;letter-spacing:1.6px;font-weight:750;margin:0 0 9px}.hero h1{font-size:30px;letter-spacing:-1px;margin:0 0 7px}.hero h1 span{color:#7666f1}.hero-sub{color:var(--muted);margin:0}.hero-sub strong{color:var(--ink)}.button{border:0;border-radius:8px;padding:10px 15px;font-weight:650;cursor:pointer;white-space:nowrap}.primary{background:var(--violet);color:#fff;box-shadow:0 6px 14px #6756e833}.primary:hover{background:#5848d7}.ghost{background:#fff;border:1px solid var(--line);color:#5d6678}.ghost:hover{border-color:#bcb6ff;color:var(--violet)}.metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:16px;margin-bottom:24px}.metric-card,.panel{background:var(--surface);border:1px solid var(--line);border-radius:12px;box-shadow:var(--shadow)}.metric-card{padding:20px;display:flex;gap:14px;min-height:130px}.metric-icon{height:40px;width:40px;border-radius:11px;display:grid;place-items:center;font-size:22px}.violet{background:var(--violet-soft);color:var(--violet)}.amber{background:var(--amber-soft);color:var(--amber)}.green{background:var(--green-soft);color:var(--green)}.blue{background:#e9f2ff;color:#3e80d5}.metric-card p{margin:2px 0 5px;color:var(--muted);font-size:12px}.metric-card h2{margin:0 0 6px;font-size:26px;letter-spacing:-1px}.trend{font-size:11px;font-weight:700}.trend em{font-style:normal;font-weight:400;color:#a2a8b5}.up{color:var(--green)}.neutral{color:#8b93a2}.workspace-grid{display:grid;grid-template-columns:minmax(0,1.6fr) minmax(290px,.75fr);gap:18px;margin-bottom:24px}.panel{padding:24px}.panel-heading{display:flex;justify-content:space-between;align-items:start;margin-bottom:20px}.panel h2{font-size:18px;margin:0;letter-spacing:-.3px}.filters{display:grid;grid-template-columns:minmax(160px,1fr) 150px 150px;gap:9px;margin-bottom:16px}.search-wrap{display:flex;align-items:center;border:1px solid var(--line);border-radius:8px;background:#fff;color:#a2a8b5;padding:0 11px}.search-wrap input{border:0;outline:0;padding:10px 8px;width:100%;font:inherit;color:var(--ink);background:transparent}.filters select{border:1px solid var(--line);border-radius:8px;padding:0 10px;color:#596478;background:#fff;font:inherit}.table-meta{color:#9299a8;font-size:11px;display:flex;justify-content:space-between;margin:0 0 9px}.legend{display:flex;gap:6px;align-items:center}.legend-dot{width:7px;height:7px;border-radius:50%;display:inline-block;margin-left:8px}.high-dot{background:#52bf93}.review-dot{background:#e8a84f}.table-scroll{overflow-x:auto}table{border-collapse:collapse;width:100%;min-width:650px}th{text-align:left;color:#9aa1af;font-size:10px;letter-spacing:.6px;text-transform:uppercase;font-weight:700;padding:11px 8px;border-bottom:1px solid var(--line)}td{padding:15px 8px;border-bottom:1px solid #f0f1f5;vertical-align:middle;color:#485367;font-size:12px}tbody tr{cursor:pointer;transition:background .15s}tbody tr:hover,tbody tr.selected{background:#faf9ff}td:first-child{color:var(--ink);font-weight:700;font-size:13px}.company-sub{display:block;font-size:11px;color:#99a0ae;font-weight:400;margin-top:2px}.score{display:inline-flex;align-items:center;gap:5px;border-radius:15px;padding:4px 8px;font-weight:750;font-size:11px}.score.high{color:var(--green);background:var(--green-soft)}.score.medium{color:var(--amber);background:var(--amber-soft)}.score.low{color:#788193;background:#eef0f4}.evidence{color:#596478}.evidence strong{display:block;color:var(--ink);font-size:12px}.fresh{font-size:11px}.fresh.good{color:var(--green)}.fresh.stale{color:var(--amber)}.status{font-size:10px;border-radius:4px;padding:4px 6px;font-weight:700}.status.review{background:var(--amber-soft);color:var(--amber)}.status.reviewed{background:var(--green-soft);color:var(--green)}.status.suppressed{background:var(--red-soft);color:var(--red)}.row-arrow{font-size:18px;color:#aeb4c0}.detail-panel{min-height:420px}.empty-detail{text-align:center;color:var(--muted);padding:55px 20px}.empty-icon{display:grid;place-items:center;margin:auto auto 17px;background:var(--violet-soft);color:var(--violet);width:48px;height:48px;border-radius:50%;font-size:23px}.empty-detail h3{color:var(--ink);margin:0 0 8px}.empty-detail p{margin:auto;max-width:220px;font-size:12px}.detail-head{display:flex;justify-content:space-between;gap:10px}.detail-head h3{margin:0;font-size:19px}.detail-domain{color:#949baa;font-size:12px;margin:3px 0 20px}.detail-score{display:flex;align-items:center;justify-content:space-between;background:#f9f8ff;padding:14px;border-radius:9px;margin-bottom:18px}.detail-score b{font-size:27px;color:var(--violet)}.detail-score small{display:block;color:var(--muted)}.detail-block{border-top:1px solid var(--line);padding:15px 0}.detail-block h4{font-size:10px;color:#8c94a4;text-transform:uppercase;letter-spacing:1px;margin:0 0 10px}.detail-block p{font-size:12px;margin:5px 0;color:#556176}.evidence-line{display:flex;justify-content:space-between;gap:10px}.confidence{color:var(--violet);font-weight:700}.disabled-action{width:100%;margin-top:5px;color:#a0a6b3;background:#f0f1f4;cursor:not-allowed}.disabled-reason{font-size:11px;color:var(--red);margin:8px 0 0}.lower-grid{display:grid;grid-template-columns:1.3fr 1fr;gap:18px}.small-label,.optional{color:#9ca3b1;font-size:11px;font-weight:400}.form-grid{display:grid;grid-template-columns:1fr 1fr;gap:16px}label{display:block;color:#596478;font-size:12px;font-weight:650}label input{display:block;width:100%;margin-top:7px;border:1px solid var(--line);border-radius:7px;padding:10px 11px;font:inherit;outline:0}label input:focus{border-color:#9489f5;box-shadow:0 0 0 3px #eeeaff}.form-footer{display:flex;align-items:center;justify-content:space-between;margin-top:20px}.form-message{font-size:11px;color:var(--green);margin:0}.form-message.error{color:var(--red)}.muted{color:var(--muted);font-size:12px}.csv-empty{border:1px dashed #d9dce8;border-radius:9px;text-align:center;padding:22px;color:#adb3c0}.csv-empty span{font-size:26px}.csv-empty p{margin:4px 0;font-size:12px;color:#737d90}.csv-empty small{font-size:10px}.csv-table{max-height:150px;overflow:auto;font-size:11px}.csv-table table{min-width:400px}.csv-table th,.csv-table td{padding:7px}.upload-label{display:inline-block}footer{display:flex;justify-content:space-between;color:#a0a6b2;font-size:11px;padding:30px 2px 0}footer a{color:var(--violet);text-decoration:none}.mobile-menu{display:none;border:0;background:transparent;font-size:21px;color:var(--ink)}@media(max-width:1050px){.metrics{grid-template-columns:repeat(2,1fr)}.workspace-grid{grid-template-columns:1fr}.detail-panel{min-height:auto}.lower-grid{grid-template-columns:1fr}}@media(max-width:700px){.sidebar{transform:translateX(-100%);transition:transform .2s;z-index:5;width:230px}.sidebar.open{transform:translateX(0)}.main{margin-left:0}.topbar{padding:0 20px}.mobile-menu{display:block}.crumb{font-size:12px}.top-actions{gap:12px}.api-status{display:none}.content{padding:28px 16px}.hero{align-items:start;gap:18px;flex-direction:column}.hero h1{font-size:25px}.metrics{grid-template-columns:1fr 1fr;gap:10px}.metric-card{padding:15px;min-height:112px;gap:9px}.metric-icon{width:34px;height:34px;font-size:18px}.metric-card h2{font-size:22px}.panel{padding:18px}.filters{grid-template-columns:1fr;gap:8px}.filters select{height:38px}.table-meta{align-items:start;gap:8px;flex-direction:column}.legend{display:none}.form-grid{grid-template-columns:1fr}.form-footer{align-items:start;gap:14px;flex-direction:column}.form-footer .button{width:100%}footer{flex-direction:column;gap:5px}}.login-screen{min-height:100vh;display:grid;place-items:center;padding:24px;background:radial-gradient(circle at 15% 10%,#efedff 0,transparent 34%),var(--bg)}.login-card{width:min(100%,430px);padding:42px 40px;background:var(--surface);border:1px solid var(--line);border-radius:18px;box-shadow:0 24px 70px rgba(33,36,75,.11)}.login-brand{padding:0;margin-bottom:42px}.login-card h1{margin:0;font-size:32px;letter-spacing:-1px}.login-subtitle{color:var(--muted);margin:8px 0 28px}.login-card form{display:grid;gap:17px}.login-card label input{padding:12px}.login-submit{width:100%;display:flex;justify-content:center;gap:9px;margin-top:3px;padding:12px}.login-note{color:#9299a8;text-align:center;font-size:11px;margin:22px 0 0}.user-identity{color:#596478;font-size:12px;font-weight:650}.logout-button{border:1px solid var(--line);border-radius:7px;background:#fff;color:#596478;font:inherit;font-size:11px;font-weight:650;padding:7px 10px;cursor:pointer}.logout-button:hover{border-color:#bcb6ff;color:var(--violet)}[hidden]{display:none!important}@media(max-width:700px){.login-card{padding:32px 22px}.login-brand{margin-bottom:32px}.user-identity{display:none}.logout-button{padding:6px 8px}} .detail-list{margin:.5rem 0 1rem;padding-left:1.2rem}.detail-list li{margin:.25rem 0}.detail-block{border-top:1px solid var(--line);padding:14px 0}.detail-block h4{margin:0 0 9px}.count{color:var(--muted);font-size:12px;font-weight:400}.compact{padding:7px 10px;font-size:12px}.inline-form,.compact-form{display:flex;gap:7px;flex-wrap:wrap;align-items:center}.compact-form input,.compact-form textarea,.inline-form select{border:1px solid var(--line);border-radius:7px;padding:8px;font:inherit;min-width:0;flex:1}.compact-form textarea{flex-basis:100%;resize:vertical}.detail-loading,.detail-error{padding:28px 4px;color:var(--muted)}.detail-error h3{color:var(--ink)}.review-status{margin:0 0 10px}.page-size{display:flex;align-items:center;gap:5px;color:var(--muted);font-size:12px}.page-size select{border:1px solid var(--line);border-radius:6px;padding:5px}.form-message.error{color:var(--red)} .sources-section{margin-top:28px;scroll-margin-top:24px}.sources-header{display:flex;align-items:flex-start;justify-content:space-between;gap:24px}.sources-header h2{margin:.15rem 0 .25rem}.source-safety{margin:14px 0;padding:12px 15px;border:1px solid #f1d7a5;border-radius:9px;background:var(--amber-soft);color:#76500d}.sources-message{min-height:22px;padding:8px 2px;color:var(--green)}.sources-message.error{color:var(--red)}.sources-grid{display:grid;grid-template-columns:1fr 1fr;gap:18px}.source-config-panel,.discovery-panel,.sources-list,.source-records{min-width:0}.source-csv-field{display:block;margin-top:12px}.source-csv-field textarea{width:100%;resize:vertical}.checkbox-label{display:flex;flex-direction:row;align-items:center;gap:8px;margin-top:14px}.checkbox-label input{width:auto}.discovery-actions{display:flex;gap:8px;flex-wrap:wrap}.source-row{display:grid;grid-template-columns:minmax(170px,1fr) auto;gap:12px 18px;padding:16px;border-top:1px solid var(--line);align-items:start}.source-row-main{display:flex;flex-direction:column;gap:3px;min-width:0}.source-row-main strong{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.source-row-main small,.source-facts{color:var(--muted);font-size:12px}.source-status{border-radius:999px;padding:4px 9px;font-size:11px;font-weight:700;text-transform:capitalize;background:var(--red-soft);color:var(--red)}.source-status.enabled{background:var(--green-soft);color:var(--green)}.source-facts{grid-column:1 / -1;display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:0}.source-facts div{border:1px solid var(--line);border-radius:7px;padding:8px}.source-facts dt{font-size:10px;text-transform:uppercase;letter-spacing:.05em}.source-facts dd{margin:3px 0 0;color:var(--ink);overflow-wrap:anywhere}.source-actions{grid-column:1 / -1;display:flex;gap:7px}.button.danger{border-color:#e9b8bd;color:var(--red);background:var(--red-soft)}.source-empty{padding:34px 18px;color:var(--muted);text-align:center}.source-record-table{overflow-x:auto}.source-record-table table{min-width:650px}.source-record-table td,.source-record-table th{padding:11px 14px}@media(max-width:900px){.sources-grid{grid-template-columns:1fr}.source-facts{grid-template-columns:repeat(2,1fr)}}@media(max-width:700px){.sources-header{flex-direction:column}.source-row{grid-template-columns:1fr}.source-status{justify-self:start}.source-facts{grid-column:1;grid-template-columns:1fr 1fr}.source-actions{grid-column:1}.discovery-actions .button{flex:1}} -.dedup-panel{border-top:1px solid var(--line);margin-top:18px;padding-top:16px}.dedup-heading{display:flex;align-items:flex-start;justify-content:space-between;gap:12px}.dedup-heading h4{margin:.1rem 0}.match-list,.history-list{display:grid;gap:9px;margin-top:10px}.match-card,.history-row{border:1px solid var(--line);border-radius:9px;padding:11px;background:#fcfcfe}.match-card-head,.history-row{display:flex;align-items:flex-start;justify-content:space-between;gap:10px}.match-card-head strong{overflow-wrap:anywhere}.match-confidence{font-size:11px;color:var(--green);font-weight:700;white-space:nowrap}.match-reasons{margin:7px 0;padding-left:18px;color:var(--muted);font-size:12px}.match-reasons li{margin:3px 0}.history-row small{display:block;color:var(--muted);font-size:11px;margin-top:3px}.dedup-error{padding:14px 0;color:var(--muted)}.dedup-error h4{color:var(--ink);margin:0 0 4px}.merge-dialog{position:fixed;z-index:20;inset:0;background:rgba(23,32,51,.46);display:grid;place-items:center;padding:20px}.merge-dialog[hidden]{display:none}.merge-dialog-card{width:min(520px,100%);background:var(--surface);border-radius:14px;padding:22px;box-shadow:0 24px 70px rgba(23,32,51,.25)}.merge-dialog-card h2{margin:.1rem 0}.merge-dialog-card p{margin:12px 0}.merge-warning{border:1px solid #f1d7a5;border-radius:8px;padding:11px;background:var(--amber-soft);color:#76500d}.merge-dialog-actions{display:flex;justify-content:flex-end;gap:8px;margin-top:18px} +.domain-intelligence{background:#fbfbff;border-radius:10px;padding:15px;margin-top:6px}.domain-panel-heading{display:flex;align-items:flex-start;justify-content:space-between;gap:12px}.domain-panel-heading h4{margin:.1rem 0}.domain-safety{border:1px solid #dcd8ff;border-radius:8px;background:var(--violet-soft);color:#5145a7;padding:9px 11px;font-size:12px;margin:10px 0}.domain-check-meta{display:flex;flex-wrap:wrap;align-items:center;gap:9px;color:var(--muted);font-size:11px;margin:10px 0}.domain-state{border-radius:999px;padding:4px 9px;background:var(--amber-soft);color:var(--amber);font-weight:700;text-transform:capitalize}.domain-state.nxdomain{background:var(--red-soft);color:var(--red);text-transform:none}.domain-state.error{background:var(--red-soft);color:var(--red)}.dns-capabilities{display:grid;grid-template-columns:repeat(5,1fr);gap:7px;margin:0}.dns-capabilities div{border:1px solid var(--line);border-radius:7px;padding:8px;background:#fff}.dns-capabilities dt{font-size:10px;font-weight:800;color:var(--muted);letter-spacing:.06em}.dns-capabilities dd{margin:3px 0 0;font-size:12px;overflow-wrap:anywhere}.domain-error{color:var(--red);font-size:12px}.candidate-domain-panel{border-top:1px solid var(--line);margin-top:15px;padding-top:15px}.candidate-domain-panel h4{margin:.1rem 0}.candidate-list{display:grid;gap:9px;margin-top:10px}.candidate-card{border:1px solid var(--line);border-radius:9px;padding:11px;background:#fff}.candidate-head{display:flex;justify-content:space-between;gap:10px}.candidate-head strong{overflow-wrap:anywhere}.availability-result{color:var(--muted);font-size:12px;margin:7px 0 0}.compact-safety{margin-bottom:0}.dedup-panel{border-top:1px solid var(--line);margin-top:18px;padding-top:16px}.dedup-heading{display:flex;align-items:flex-start;justify-content:space-between;gap:12px}.dedup-heading h4{margin:.1rem 0}.match-list,.history-list{display:grid;gap:9px;margin-top:10px}.match-card,.history-row{border:1px solid var(--line);border-radius:9px;padding:11px;background:#fcfcfe}.match-card-head,.history-row{display:flex;align-items:flex-start;justify-content:space-between;gap:10px}.match-card-head strong{overflow-wrap:anywhere}.match-confidence{font-size:11px;color:var(--green);font-weight:700;white-space:nowrap}.match-reasons{margin:7px 0;padding-left:18px;color:var(--muted);font-size:12px}.match-reasons li{margin:3px 0}.history-row small{display:block;color:var(--muted);font-size:11px;margin-top:3px}.dedup-error{padding:14px 0;color:var(--muted)}.dedup-error h4{color:var(--ink);margin:0 0 4px}.merge-dialog{position:fixed;z-index:20;inset:0;background:rgba(23,32,51,.46);display:grid;place-items:center;padding:20px}.merge-dialog[hidden]{display:none}.merge-dialog-card{width:min(520px,100%);background:var(--surface);border-radius:14px;padding:22px;box-shadow:0 24px 70px rgba(23,32,51,.25)}.merge-dialog-card h2{margin:.1rem 0}.merge-dialog-card p{margin:12px 0}.merge-warning{border:1px solid #f1d7a5;border-radius:8px;padding:11px;background:var(--amber-soft);color:#76500d}.merge-dialog-actions{display:flex;justify-content:flex-end;gap:8px;margin-top:18px} .jobs-section{margin-top:28px;scroll-margin-top:24px}.jobs-header{display:flex;align-items:flex-start;justify-content:space-between;gap:24px}.jobs-header h2{margin:.15rem 0 .25rem}.jobs-header p{margin:.25rem 0 0}.jobs-actions{display:flex;gap:8px;flex-wrap:wrap}.jobs-message{min-height:22px;padding:8px 2px;color:var(--green)}.jobs-message.error{color:var(--red)}.job-counts{display:grid;grid-template-columns:repeat(5,1fr);gap:12px;margin:0 0 14px}.job-count{background:var(--surface);border:1px solid var(--line);border-left:4px solid var(--violet);border-radius:10px;padding:14px 16px;box-shadow:var(--shadow)}.job-count span{display:block;color:var(--muted);font-size:12px}.job-count strong{display:block;font-size:25px;margin-top:4px}.job-count.queued{border-left-color:var(--amber)}.job-count.running{border-left-color:#4d8bd8}.job-count.succeeded{border-left-color:var(--green)}.job-count.failed{border-left-color:var(--red)}.job-count.cancelled{border-left-color:#8d879c}.jobs-grid{display:grid;grid-template-columns:minmax(0,1.05fr) minmax(320px,.95fr);gap:18px}.jobs-list,.job-detail{min-height:320px}.jobs-list-body{border-top:1px solid var(--line)}.job-empty{padding:38px 18px;color:var(--muted);text-align:center}.job-row{display:grid;grid-template-columns:minmax(0,1fr) auto 42px;gap:12px;align-items:center;width:100%;padding:14px 16px;border:0;border-bottom:1px solid var(--line);background:transparent;color:inherit;text-align:left;cursor:pointer;font:inherit}.job-row:hover,.job-row.selected{background:var(--violet-soft)}.job-row-main{display:flex;flex-direction:column;gap:3px;min-width:0}.job-row-main strong{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.job-row-main small,.job-row-progress{color:var(--muted);font-size:12px}.job-row-state{border-radius:999px;padding:4px 8px;font-size:11px;font-weight:700;white-space:nowrap;background:var(--violet-soft);color:var(--violet)}.job-row-state.queued{background:var(--amber-soft);color:var(--amber)}.job-row-state.running{background:#e8f1ff;color:#3269ad}.job-row-state.succeeded{background:var(--green-soft);color:var(--green)}.job-row-state.failed{background:var(--red-soft);color:var(--red)}.job-row-state.cancelled{background:#f0eef4;color:#716a7c}.job-detail{padding:22px}.job-detail-head{display:flex;justify-content:space-between;gap:12px;border-bottom:1px solid var(--line);padding-bottom:16px}.job-detail-head h3{margin:.15rem 0}.job-progress{padding:18px 0}.job-progress-meta{display:flex;justify-content:space-between;gap:12px;font-size:12px;color:var(--muted)}.job-progress-meta strong{color:var(--ink)}.progress-track{height:8px;background:#eef0f5;border-radius:999px;overflow:hidden;margin-top:10px}.progress-track span{display:block;height:100%;background:var(--violet);border-radius:inherit;transition:width .25s}.structured-error{background:var(--red-soft);border:1px solid #f2cdd0;border-radius:8px;padding:12px;margin:4px 0 16px;color:var(--red)}.structured-error p{margin:5px 0}.structured-error pre{white-space:pre-wrap;font-size:11px;margin:8px 0 0}.job-detail-actions{display:flex;gap:8px;min-height:34px}.event-timeline{border-top:1px solid var(--line);margin-top:16px;padding-top:16px}.event-timeline h4{margin:0}.event-timeline ol{list-style:none;padding:0;margin:12px 0 0}.event-timeline li{display:flex;gap:10px;position:relative;padding:0 0 15px}.event-timeline li:not(:last-child):before{content:"";position:absolute;left:4px;top:10px;bottom:0;border-left:1px solid var(--line)}.timeline-dot{z-index:1;width:9px;height:9px;margin-top:4px;border-radius:50%;background:var(--violet);flex:none}.event-timeline li div{display:flex;flex-direction:column;gap:2px}.event-timeline small,.timeline-progress{font-size:11px;color:var(--muted)}@media(max-width:900px){.jobs-grid{grid-template-columns:1fr}.job-counts{grid-template-columns:repeat(3,1fr)}}@media(max-width:700px){.jobs-header{flex-direction:column}.job-counts{grid-template-columns:repeat(2,1fr)}.job-row{grid-template-columns:minmax(0,1fr) auto}.job-row-progress{display:none}} diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 061318f..2448e9e 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -51,6 +51,16 @@ After a merge, read back the survivor and audit trail, check child/evidence/sour The current Compose/MVP runtime exposes the Phase 6 match-suggestion, explicit merge-confirmation, tenant-scoped merge-history, and reversal routes; no separate worker is required. It has basic canonicalization and exact duplicate handling plus pilot-grade fuzzy review. Monitor merge audit events and verify child/source-record counts after each operation. Do not schedule or report deduplication as automated discovery, enrichment, verification, or outreach. +## Phase 7 domain-intelligence operations + +Phase 7 is a review contract only; the current Compose stack has no DNS worker, PSL service, DNS cache, or availability provider. Do not add a resolver or provider by pointing a job at an arbitrary URL. Any future implementation must record a pinned PSL version, preserve raw and registrable-domain forms, and report unresolved reasons for public suffixes, unknown/private suffixes, malformed/single-label values, localhost, and IP literals. IDN/punycode handling and subdomain treatment must be deterministic. + +Operate DNS as bounded, observable evidence collection. Surface `not_checked`, `pending`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error`; never translate an empty/error result into a business conclusion. Keep MX, NS, and TXT records separate and record resolver/source, observed time, TTL, freshness/staleness, truncation/partial state, and uncertainty/error reason. A cached answer is not a fresh check: caches must be bounded, tenant-isolated, keyed by normalized query/type/class and resolver policy, honor TTL, and be invalidated/re-evaluated when the PSL or resolver policy changes. + +Review domain-to-business candidates separately from DNS results. Verify tenant scope before comparison, reject public-suffix-only and malformed/IP candidates, and flag shared, parked, wildcard, homograph, sibling-subdomain, and conflicting-evidence cases. Association confidence must show evidence, reasons, algorithm/version, freshness, and uncertainty; it is suggestion-only. Require an authorized human accept/reject decision and audit it. Never auto-attach a domain or treat DNS as proof of ownership. + +Availability is `unknown` unless the API reports a result from an authorized provider. Before enabling one, verify current product/legal/security approval, terms owner, allowed tenant scope, request/concurrency and timeout limits, retention/deletion class, health/circuit state, and explicit operational enablement. `nxdomain`, `no_data`, timeout, stale cache, or provider error is not “available.” On provider outage, rate-limit, terms/approval expiry, circuit-open, or conflicting result, fail closed and report unknown/deferred; never purchase, reserve, contact, or retry through an unapproved provider. + ## Phase 4 jobs and live logging The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects. @@ -107,7 +117,7 @@ Do not run `docker compose down -v` on a data-bearing environment: it removes th ## Production migration and scaling path -Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker operations, metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and scanners are possible future components—not implicit Compose dependencies and not implemented by this MVP. No automated discovery or outreach may be inferred from the scaling path. +Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events and domain observations, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker operations, bounded DNS/PSL processing, TTL-aware cache invalidation, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS status/error rates, cache freshness, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and scanners are possible future components—not implicit Compose dependencies and not implemented by this MVP. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path. ## Incident checklist diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 74cf39d..fcaa9a9 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -29,6 +29,17 @@ The MVP provides deterministic match suggestions, an explicit human confirmation dialog, a tenant-scoped merge ledger/snapshot, and reversal routes. These are pilot controls, not a compliance-grade merge system: add a dedicated merge permission, server-verifiable confirmation/re-authentication, complete parent/conflict snapshots, transactional concurrency protection, and tamper-evident retention before production. Do not present derived normalization or a suggestion as verified identity, discovery, enrichment, or outreach permission. +## Phase 7 domain-intelligence controls + +- Registrable-domain derivation must use a pinned/versioned PSL. Never use a naive last-two-label fallback. Preserve the raw input and return unresolved for unknown/private suffixes, public suffixes, malformed/single-label names, localhost, and IP literals; handle IDN/punycode deterministically and treat subdomains conservatively. +- DNS status is not a business fact. Expose `not_checked`, `pending`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error` separately. A failed, blocked, stale, or empty response must not be upgraded to “no domain,” “no email,” or “unowned.” +- MX, NS, and TXT observations require independent provenance, resolver/source, observed time, TTL, truncation/partial and uncertainty/error metadata. MX absence does not prove mail inability; NS does not prove control; TXT does not prove ownership. Treat TXT values as untrusted input and redact secrets where applicable. +- DNS caches must be bounded and keyed by normalized name/type/class plus resolver policy and PSL version where relevant. Honor received TTLs without extending authority, expose freshness/staleness, invalidate on policy/PSL changes, and prevent one tenant's cached evidence from being disclosed to another. A cache hit is not a fresh check. +- Association confidence is an explainable, versioned review signal separate from DNS status and duplicate score. Candidate generation must enforce tenant scope before comparison, reject public-suffix-only/malformed/IP candidates, flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases, and require authorized human accept/reject decisions. Never auto-attach or infer ownership. +- Domain availability must remain `unknown` unless an explicitly authorized provider is registered and checked at execution time for current product/legal/security approval, terms, tenant scope, rate/concurrency limits, retention, and health/circuit state. DNS `nxdomain`/`no_data`, stale cache, timeout, or provider failure is never availability. Fail closed; do not purchase, reserve, contact, or report availability from DNS alone. + +No Phase 7 resolver, cache, or availability provider is enabled in the current Compose runtime. Before production, add egress/SSRF controls, provider and PSL update review, retention/deletion handling, monitoring, permission/audit coverage, and failure/rollback tests for all domain observations. + ## Phase 5 source security controls Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization. @@ -57,6 +68,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in 11. **Jobs and idempotency:** require tenant-scoped idempotency keys for side-effecting job creation and bind each key to a request fingerprint; reject conflicting reuse and make retries safe against duplicate side effects. Persist lifecycle transitions transactionally and define lease/timeout/recovery behavior. 12. **Live delivery:** SSE, if introduced, must authenticate before opening the stream, enforce tenant scope on every replay query, bound event/backlog size, support `Last-Event-ID`/cursor replay, send heartbeats, and provide polling fallback. Treat event-stream connections as untrusted clients and avoid cross-tenant timing/detail leaks. 13. **Worker boundary:** the current SQLite/in-process MVP is not durable or horizontally safe. A production worker migration requires reviewed queue semantics, leases, visibility timeouts, dead-letter handling, concurrency limits, cancellation races, metrics, and deployment isolation. Redis/Celery are not implemented today. +14. **Domain intelligence:** implement and security-review PSL pin/update handling, bounded DNS resolution, TTL-aware cache isolation/invalidation, uncertainty-preserving MX/NS/TXT parsing, tenant-scoped association decisions, and an authorized availability provider before exposing any live domain status. No DNS response may authorize acquisition, ownership, outreach, or verification. ## Source and contact policy