add conservative domain intelligence
This commit is contained in:
Marco0300
+17
-3
@@ -1,6 +1,6 @@
|
||||
# Prospect Platform API — Phase 6 boundary
|
||||
# Prospect Platform API — Phase 7 boundary
|
||||
|
||||
Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 6 normalization/deduplication plus Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
|
||||
Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
|
||||
|
||||
## Run
|
||||
|
||||
@@ -18,6 +18,8 @@ Set `PROSPECT_API_PORT` or pass `--port`; set `PROSPECT_API_DB` or pass `--db` t
|
||||
|
||||
All protected endpoints require the server-side session cookie. Every query is constrained by the authenticated user's `organization_id`; IDs from another tenant behave as not found and must not disclose whether a record exists.
|
||||
|
||||
Phase 7 domain routes (all tenant-scoped) are `POST /api/v1/businesses/{id}/domains/check`, `GET /api/v1/businesses/{id}/domains/check?domain=...`, `GET /api/v1/domain-checks`, `GET /api/v1/businesses/{id}/domain-candidates`, and `POST /api/v1/businesses/{id}/domain-candidates/check-availability`. The current implementation is intentionally conservative: a successful address lookup is reported as `ok`, unresolved/empty results as `unknown`, and an availability check returns `unknown`/`not_configured` because no provider is enabled. Treat these as observation states, not ownership or availability claims.
|
||||
|
||||
## Phase 4 jobs and live logging (target contract)
|
||||
|
||||
The intended job resource has a stable `job_id`, tenant/creator metadata, operation/payload fingerprint, `status`, `attempt`, timestamps, cancellation state, and terminal error/result metadata. Its lifecycle is `queued` → `running` → exactly one terminal state: `succeeded`, `failed`, or `cancelled`. State transitions and worker messages must be persisted transactionally with tenant and job identifiers; terminal jobs are immutable except for controlled retention/redaction.
|
||||
@@ -102,6 +104,18 @@ Remaining limitations: the snapshot currently focuses on the source graph rather
|
||||
|
||||
List and child-record endpoints are deliberately bounded. For business lists, use `page` (starting at 1) and `page_size` within the server-enforced maximum; invalid values are rejected rather than allowing an unbounded query. Supported filters are applied inside the tenant-scoped query before pagination: `q`, `score_min`, `score_max`, `website_class`, and `pipeline_stage`. The UI's page and filter controls are convenience clients, not authorization controls. A filtered page is not a count of the entire unfiltered tenant unless the response explicitly says so.
|
||||
|
||||
## Phase 7 domain-intelligence contract
|
||||
|
||||
Domain observations are tenant-scoped, provenance-bearing inputs. Registrable-domain normalization must use a pinned/versioned PSL rather than a last-two-label heuristic. Preserve the original value and normalized form; return `unknown` when the PSL cannot classify a value (including an unknown/private suffix, public suffix, malformed name, single-label name, localhost, or IP literal). IDN/punycode and case/label handling must be deterministic and must not turn a subdomain into an independent business identity.
|
||||
|
||||
DNS checks must expose an explicit status—`not_checked`, `pending`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`—and never collapse failure or absence into a negative business fact. MX, NS, and TXT are independently uncertain observations: retain record type, normalized response, resolver/source, observed time, TTL if supplied, truncation/partial indicators, and error/uncertainty reason. A missing MX does not prove that email is unavailable; an NS result does not prove control; TXT content does not prove ownership.
|
||||
|
||||
Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class and resolver policy, and TTL-aware. Do not extend authority beyond the received TTL; expose `observed_at`, `expires_at`/freshness, and stale or refresh state. A cache hit is not a fresh lookup, and resolver policy/PSL version changes require invalidation or re-evaluation. There is no DNS resolver or cache service in the current runtime.
|
||||
|
||||
Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification.
|
||||
|
||||
Domain availability is `unknown` unless an explicitly authorized availability provider supplies it. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
|
||||
|
||||
## Remaining limitations and production migration work
|
||||
|
||||
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Production migration work must add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.
|
||||
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Phase 7 production work must add a reviewed PSL update/version policy, bounded DNS workers, TTL-aware cache storage/invalidation, uncertainty-aware MX/NS/TXT schema and tests, association candidate safeguards and permissions, and an authorized availability provider with egress/rate/circuit/retention controls. It must also add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Conservative, dependency-free domain intelligence helpers.
|
||||
|
||||
This module deliberately reports uncertainty rather than inferring DNS or registrar facts.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import re
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
import unicodedata
|
||||
from datetime import datetime, timezone
|
||||
from difflib import SequenceMatcher
|
||||
from urllib.parse import urlparse
|
||||
|
||||
# Small explicitly maintained PSL subset. Unknown suffixes are never guessed.
|
||||
PUBLIC_SUFFIXES = frozenset({
|
||||
"com", "org", "net", "za", "co.za", "org.za", "net.za", "ac.za", "gov.za", "edu.za",
|
||||
})
|
||||
DEFAULT_TTL_SECONDS = 300
|
||||
MAX_DOMAIN_LENGTH = 253
|
||||
MAX_CANDIDATES = 20
|
||||
|
||||
|
||||
def _host(value: str | None) -> str:
|
||||
raw = str(value or "").strip().lower()
|
||||
if not raw:
|
||||
return ""
|
||||
parsed = urlparse(raw if "://" in raw else "//" + raw)
|
||||
host = (parsed.hostname or "").rstrip(".").lower()
|
||||
if host.startswith("www."):
|
||||
host = host[4:]
|
||||
return host
|
||||
|
||||
|
||||
def _valid_hostname(host: str) -> bool:
|
||||
if not host or len(host) > MAX_DOMAIN_LENGTH or "." not in host:
|
||||
return False
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
return False
|
||||
except ValueError:
|
||||
pass
|
||||
if any(len(label) > 63 or not re.fullmatch(r"[a-z0-9](?:[a-z0-9-]*[a-z0-9])?", label) for label in host.split(".")):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def normalize_registrable_domain(value: str | None) -> str:
|
||||
"""Return the registrable domain, or ``unknown`` for unsupported/unsafe input."""
|
||||
host = _host(value)
|
||||
if not _valid_hostname(host):
|
||||
return "unknown"
|
||||
labels = host.split(".")
|
||||
suffix = next((".".join(labels[-n:]) for n in (3, 2, 1) if ".".join(labels[-n:]) in PUBLIC_SUFFIXES), None)
|
||||
if not suffix or len(labels) <= suffix.count(".") + 1:
|
||||
return "unknown"
|
||||
return ".".join(labels[-(suffix.count(".") + 2):])
|
||||
|
||||
|
||||
def domain_info(value: str | None) -> dict:
|
||||
host = _host(value)
|
||||
registered = normalize_registrable_domain(value)
|
||||
return {"input": str(value or ""), "hostname": host, "registrable_domain": registered,
|
||||
"status": "ok" if registered != "unknown" else "unknown"}
|
||||
|
||||
|
||||
def _metadata(ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict:
|
||||
now = time.time()
|
||||
checked = datetime.fromtimestamp(now, timezone.utc).replace(microsecond=0).isoformat()
|
||||
return {"checked_at": checked, "ttl_seconds": ttl_seconds,
|
||||
"cache_expires_at": datetime.fromtimestamp(now + ttl_seconds, timezone.utc).replace(microsecond=0).isoformat()}
|
||||
|
||||
|
||||
def resolve_domain(domain: str, *, timeout: float = 3.0, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict:
|
||||
"""Resolve only a validated hostname using stdlib socket; never fetches a URL."""
|
||||
normalized = normalize_registrable_domain(domain)
|
||||
host = _host(domain)
|
||||
result = {"domain": normalized, "hostname": host, "status": "unknown", "addresses": [], **_metadata(ttl_seconds)}
|
||||
if normalized == "unknown" or not _valid_hostname(host):
|
||||
return result
|
||||
resolver = resolver or socket.getaddrinfo
|
||||
try:
|
||||
# getaddrinfo has no per-call timeout. Run it in a daemon worker so a
|
||||
# resolver stall cannot block an HTTP handler indefinitely.
|
||||
outcome = {}
|
||||
def lookup():
|
||||
try: outcome["records"] = resolver(host, None, 0, socket.SOCK_STREAM)
|
||||
except BaseException as exc: outcome["exception"] = exc
|
||||
worker = threading.Thread(target=lookup, daemon=True)
|
||||
worker.start(); worker.join(max(0.01, float(timeout)))
|
||||
if worker.is_alive():
|
||||
result["status"] = "timeout"
|
||||
return result
|
||||
if "exception" in outcome: raise outcome["exception"]
|
||||
records = outcome.get("records", [])
|
||||
addresses = sorted({str(item[4][0]) for item in records if item and len(item) > 4 and item[0] in (socket.AF_INET, socket.AF_INET6)})
|
||||
result["addresses"] = addresses
|
||||
result["status"] = "ok" if addresses else "unknown"
|
||||
except socket.gaierror as exc:
|
||||
code = getattr(exc, "errno", None)
|
||||
if code is None and exc.args: code = exc.args[0]
|
||||
result["status"] = "nxdomain" if code in (socket.EAI_NONAME, socket.EAI_NODATA, -2, -5) else "error"
|
||||
result["error_code"] = code
|
||||
except (TimeoutError, socket.timeout):
|
||||
result["status"] = "timeout"
|
||||
except OSError as exc:
|
||||
result["status"] = "error"; result["error_code"] = exc.__class__.__name__
|
||||
return result
|
||||
|
||||
|
||||
def capability_hook(domain: str, capability: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict:
|
||||
"""MX/NS/TXT hook. No optional resolver means not_configured, not empty."""
|
||||
normalized = normalize_registrable_domain(domain)
|
||||
base = {"domain": normalized, "capability": capability, "status": "unknown", "records": [], **_metadata(ttl_seconds)}
|
||||
if normalized == "unknown": return base
|
||||
if capability not in {"mx", "ns", "txt"}: base.update(status="error", error_code="unsupported_capability"); return base
|
||||
if resolver is None:
|
||||
base.update(status="not_configured", reason="resolver_not_configured")
|
||||
return base
|
||||
try:
|
||||
records = resolver(normalized, capability)
|
||||
base.update(status="ok" if records else "unknown", records=list(records or []))
|
||||
except TimeoutError: base["status"] = "timeout"
|
||||
except socket.gaierror: base["status"] = "nxdomain"
|
||||
except Exception as exc: base.update(status="error", error_code=exc.__class__.__name__)
|
||||
return base
|
||||
|
||||
|
||||
def resolve_mx(domain: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict:
|
||||
return capability_hook(domain, "mx", resolver=resolver, ttl_seconds=ttl_seconds)
|
||||
|
||||
|
||||
def resolve_ns(domain: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict:
|
||||
return capability_hook(domain, "ns", resolver=resolver, ttl_seconds=ttl_seconds)
|
||||
|
||||
|
||||
def resolve_txt(domain: str, *, resolver=None, ttl_seconds: int = DEFAULT_TTL_SECONDS) -> dict:
|
||||
return capability_hook(domain, "txt", resolver=resolver, ttl_seconds=ttl_seconds)
|
||||
|
||||
|
||||
registrable_domain = normalize_registrable_domain
|
||||
|
||||
|
||||
def association_confidence(candidate_domain: str, observed_domain: str, *, business_name: str = "") -> dict:
|
||||
candidate = normalize_registrable_domain(candidate_domain); observed = normalize_registrable_domain(observed_domain)
|
||||
if candidate == "unknown" or observed == "unknown": return {"level":"unknown", "score":0.0, "reasons":["unsupported_domain"]}
|
||||
if candidate == observed: return {"level":"high", "score":1.0, "reasons":["same_registrable_domain"]}
|
||||
token = re.sub(r"[^a-z0-9]", "", unicodedata.normalize("NFKD", str(business_name).lower()))
|
||||
score = SequenceMatcher(None, token, candidate.split(".")[0]).ratio() if token else 0.0
|
||||
return {"level":"medium" if score >= .8 else "low", "score":round(score, 4), "reasons":["name_domain_similarity"] if score >= .8 else ["different_registrable_domain"]}
|
||||
|
||||
|
||||
def _slug(value: object) -> str:
|
||||
text = unicodedata.normalize("NFKD", str(value or "")).encode("ascii", "ignore").decode().lower()
|
||||
words = re.findall(r"[a-z0-9]+", text)
|
||||
return "-".join(words)[:40].strip("-")
|
||||
|
||||
|
||||
def generate_candidate_domains(business_name: str, service: str = "", location: str = "", *, suffixes=("co.za", "com"), limit: int = MAX_CANDIDATES) -> list[str]:
|
||||
limit = max(0, min(int(limit), MAX_CANDIDATES)); name, svc, loc = _slug(business_name), _slug(service), _slug(location)
|
||||
if not name: return []
|
||||
parts = [name, f"{name}-{svc}" if svc else "", f"{name}-{loc}" if loc else "", f"{svc}-{loc}" if svc and loc else ""]
|
||||
if svc: parts += [f"{name}{svc}"]
|
||||
out=[]
|
||||
for base in parts:
|
||||
if not base or len(base) > 63: continue
|
||||
for suffix in suffixes:
|
||||
if suffix not in PUBLIC_SUFFIXES: continue
|
||||
domain = f"{base}.{suffix}"
|
||||
if normalize_registrable_domain(domain) != "unknown" and domain not in out: out.append(domain)
|
||||
if len(out) >= limit: return out
|
||||
return out
|
||||
+68
-1
@@ -9,15 +9,17 @@ if __package__ in (None, ""):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from app.sources import adapter_for, contains_secret
|
||||
from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
else:
|
||||
from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from .sources import adapter_for, contains_secret
|
||||
from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
ORGANIZATION_ID = "demo-tenant"
|
||||
SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
|
||||
SESSION_DAYS = 7
|
||||
PBKDF2_ITERATIONS = 300_000
|
||||
MUTATING_ROLES = {"owner", "admin", "researcher"}
|
||||
JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery"}
|
||||
JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"}
|
||||
JOB_PAGE_SIZE = 100
|
||||
SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
|
||||
CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
|
||||
@@ -117,16 +119,79 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/discovery-queries": return self.list_queries(db,org)
|
||||
if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/businesses/"):
|
||||
bits=path.split("/"); ident=bits[4] if len(bits)>4 else ""
|
||||
if not ident.isdigit(): return self.send_json(404,{"error":"not_found"})
|
||||
row=self.business(db,int(ident),org)
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query))
|
||||
if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org)
|
||||
if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org)
|
||||
payload=row_json(row); payload.update(self.nested(db,int(ident),org)); return self.send_json(200,payload)
|
||||
return self.send_json(404,{"error":"not_found"})
|
||||
finally: db.close()
|
||||
def _domain_result(self, row, cache_hit=False):
|
||||
try: result=json.loads(row["result_json"] or "{}")
|
||||
except (TypeError,ValueError): result={}
|
||||
result.update({"id":row["id"],"business_id":row["business_id"],"domain":row["domain"],"status":row["status"],"cache_hit":cache_hit,"checked_at":row["checked_at"],"cache_expires_at":row["cache_expires_at"]})
|
||||
return result
|
||||
|
||||
def _check_domain(self, bid, domain, db, user):
|
||||
normalized=normalize_registrable_domain(domain)
|
||||
if normalized == "unknown": return self.send_json(400,{"error":"unsupported_domain","status":"unknown"})
|
||||
now=datetime.now(timezone.utc).replace(microsecond=0).isoformat()
|
||||
cache_key=normalized+":a_aaaa:v1"
|
||||
cached=db.execute("SELECT * FROM domain_checks WHERE organization_id=? AND business_id=? AND cache_key=? AND cache_expires_at>? ORDER BY id DESC LIMIT 1",(user["organization_id"],bid,cache_key,now)).fetchone()
|
||||
if cached:return self.send_json(200,self._domain_result(cached,True))
|
||||
result=resolve_domain(normalized)
|
||||
try: cur=db.execute("INSERT INTO domain_checks(organization_id,business_id,domain,status,result_json,cache_key,checked_at,cache_expires_at) VALUES(?,?,?,?,?,?,?,?)",(user["organization_id"],bid,normalized,result["status"],json.dumps(result,sort_keys=True),cache_key,result["checked_at"],result["cache_expires_at"]))
|
||||
except sqlite3.IntegrityError:
|
||||
existing=db.execute("SELECT id FROM domain_checks WHERE organization_id=? AND business_id=? AND cache_key=?",(user["organization_id"],bid,cache_key)).fetchone()
|
||||
if not existing: return self.send_json(409,{"error":"domain_check_conflict"})
|
||||
db.execute("UPDATE domain_checks SET domain=?,status=?,result_json=?,checked_at=?,cache_expires_at=? WHERE id=?",(normalized,result["status"],json.dumps(result,sort_keys=True),result["checked_at"],result["cache_expires_at"],existing["id"]))
|
||||
self.audit(db,user,"domain.checked",f"{bid}:{normalized}:{result['status']}"); db.commit()
|
||||
return self.send_json(200,self._domain_result(db.execute("SELECT * FROM domain_checks WHERE id=?",(existing["id"],)).fetchone()))
|
||||
self.audit(db,user,"domain.checked",f"{bid}:{normalized}:{result['status']}"); db.commit()
|
||||
return self.send_json(200,self._domain_result(db.execute("SELECT * FROM domain_checks WHERE id=?",(cur.lastrowid,)).fetchone()))
|
||||
|
||||
def post_domain_check(self,bid,payload,db,user):
|
||||
if not self.business(db,bid,user["organization_id"]): return self.send_json(404,{"error":"not_found"})
|
||||
domain=payload.get("domain") or self.business(db,bid,user["organization_id"])["website_domain"]
|
||||
if not str(domain).strip(): return self.send_json(400,{"error":"domain_required"})
|
||||
return self._check_domain(bid,str(domain),db,user)
|
||||
|
||||
def get_domain_check(self,bid,db,user,query):
|
||||
domain=(query.get("domain") or [""])[0]
|
||||
if not domain:return self.send_json(400,{"error":"domain_required"})
|
||||
return self._check_domain(bid,domain,db,user)
|
||||
|
||||
def list_domain_checks(self,db,org,query):
|
||||
try: limit=max(1,min(int((query.get("page_size") or [50])[0]),100)); offset=max(0,int((query.get("offset") or [0])[0]))
|
||||
except (ValueError,TypeError): return self.send_json(400,{"error":"invalid_pagination"})
|
||||
rows=db.execute("SELECT * FROM domain_checks WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall()
|
||||
return self.send_json(200,{"organization_id":org,"items":[self._domain_result(r) for r in rows[:limit]],"limit":limit,"offset":offset,"has_more":len(rows)>limit})
|
||||
|
||||
def list_domain_candidates(self,bid,db,org):
|
||||
business=self.business(db,bid,org)
|
||||
if not business:return self.send_json(404,{"error":"not_found"})
|
||||
rows=db.execute("SELECT * FROM domain_candidates WHERE organization_id=? AND business_id=? ORDER BY rank,id",(org,bid)).fetchall()
|
||||
if not rows:
|
||||
generated=generate_candidate_domains(business["name"],business["description"],business["city"])
|
||||
for rank,domain in enumerate(generated,1):
|
||||
db.execute("INSERT OR IGNORE INTO domain_candidates(organization_id,business_id,domain,rank) VALUES(?,?,?,?)",(org,bid,domain,rank))
|
||||
db.commit(); rows=db.execute("SELECT * FROM domain_candidates WHERE organization_id=? AND business_id=? ORDER BY rank,id",(org,bid)).fetchall()
|
||||
return self.send_json(200,{"business_id":bid,"items":[row_json(r) for r in rows]})
|
||||
|
||||
def check_availability(self,bid,payload,db,user):
|
||||
if not self.business(db,bid,user["organization_id"]):return self.send_json(404,{"error":"not_found"})
|
||||
candidates=db.execute("SELECT domain FROM domain_candidates WHERE organization_id=? AND business_id=? ORDER BY rank,id",(user["organization_id"],bid)).fetchall()
|
||||
domains=[r["domain"] for r in candidates]
|
||||
if isinstance(payload.get("domains"),list): domains=[normalize_registrable_domain(x) for x in payload["domains"] if normalize_registrable_domain(x)!="unknown"][:20]
|
||||
self.audit(db,user,"domain.availability.checked",str(bid)); db.commit()
|
||||
return self.send_json(200,{"business_id":bid,"status":"unknown","reason":"not_configured","provider_configured":False,"items":[{"domain":d,"status":"unknown","reason":"not_configured"} for d in domains]})
|
||||
|
||||
def list_jobs(self, db, org, query):
|
||||
try: limit=max(1,min(int(query.get("page_size",[50])[0]),JOB_PAGE_SIZE)); offset=max(0,int(query.get("offset",[0])[0]))
|
||||
except (ValueError, TypeError): return self.send_json(400,{"error":"invalid_pagination"})
|
||||
@@ -221,6 +286,8 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
|
||||
if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
|
||||
if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"):
|
||||
ident=path.split("/")[4]
|
||||
return self.reverse_merge(int(ident) if ident.isdigit() else -1,db,user)
|
||||
|
||||
@@ -153,3 +153,20 @@ CREATE TABLE IF NOT EXISTS source_records (
|
||||
UNIQUE(organization_id,source_id,content_hash)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_source_records_org ON source_records(organization_id,id DESC);
|
||||
|
||||
-- Phase 7 domain intelligence (additive-safe; DNS results are explicitly stateful).
|
||||
CREATE TABLE IF NOT EXISTS domain_checks (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id),
|
||||
business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE, domain TEXT NOT NULL,
|
||||
status TEXT NOT NULL, result_json TEXT NOT NULL DEFAULT '{}', cache_key TEXT NOT NULL,
|
||||
checked_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, cache_expires_at TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id, business_id, cache_key)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_domain_checks_org ON domain_checks(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_domain_checks_business ON domain_checks(organization_id,business_id,created_at DESC);
|
||||
CREATE TABLE IF NOT EXISTS domain_candidates (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id), business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
|
||||
domain TEXT NOT NULL, source TEXT NOT NULL DEFAULT 'generated', rank INTEGER NOT NULL DEFAULT 0, metadata_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,business_id,domain)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_domain_candidates_business ON domain_candidates(organization_id,business_id,rank,id);
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.domain_intelligence import (
|
||||
normalize_registrable_domain, resolve_domain, generate_candidate_domains,
|
||||
association_confidence, resolve_mx,
|
||||
)
|
||||
from app.main import create_server
|
||||
|
||||
|
||||
class DomainIntelligenceTests(unittest.TestCase):
|
||||
def test_psl_normalization_and_unknown_suffix(self):
|
||||
self.assertEqual(normalize_registrable_domain('https://WWW.shop.example.co.za/path'), 'example.co.za')
|
||||
self.assertEqual(normalize_registrable_domain('foo.example.com'), 'example.com')
|
||||
self.assertEqual(normalize_registrable_domain('foo.example.invalidtld'), 'unknown')
|
||||
|
||||
def test_candidate_generation_is_bounded_and_safe(self):
|
||||
out = generate_candidate_domains('Acme & Sons (Pty) Ltd', 'Solar Panels', 'Cape Town')
|
||||
self.assertLessEqual(len(out), 20)
|
||||
self.assertTrue(out)
|
||||
for value in out:
|
||||
self.assertRegex(value, r'^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.(?:co\.za|com)$')
|
||||
self.assertNotIn('--', value)
|
||||
|
||||
def test_resolution_distinguishes_nxdomain_timeout_and_error(self):
|
||||
self.assertEqual(resolve_domain('missing.example.com', resolver=lambda *a: (_ for _ in ()).throw(socket_gaierror_name()))['status'], 'nxdomain')
|
||||
self.assertEqual(resolve_domain('slow.example.com', resolver=lambda *a: (_ for _ in ()).throw(TimeoutError()))['status'], 'timeout')
|
||||
self.assertEqual(resolve_domain('bad.example.com', resolver=lambda *a: (_ for _ in ()).throw(OSError('boom')))['status'], 'error')
|
||||
|
||||
def test_local_resolution_and_association_confidence(self):
|
||||
with patch('app.domain_intelligence.socket.getaddrinfo', return_value=[(2, 1, 6, '', ('1.2.3.4', 0))]):
|
||||
result = resolve_domain('example.com')
|
||||
self.assertEqual(result['status'], 'ok')
|
||||
self.assertEqual(result['addresses'], ['1.2.3.4'])
|
||||
self.assertEqual(association_confidence('acme.co.za', 'acme.co.za')['level'], 'high')
|
||||
self.assertEqual(association_confidence('acme.co.za', 'other.co.za')['level'], 'low')
|
||||
|
||||
def test_optional_capabilities_are_not_falsely_empty(self):
|
||||
self.assertEqual(resolve_mx('example.com')['status'], 'not_configured')
|
||||
|
||||
|
||||
def socket_gaierror_name():
|
||||
import socket
|
||||
return socket.gaierror(socket.EAI_NONAME)
|
||||
|
||||
|
||||
class DomainApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory(); self.db_path = self.tmp.name + '/db.sqlite'
|
||||
os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'domain-owner@example.test'; os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password'
|
||||
self.server = create_server('127.0.0.1', 0, self.db_path); self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
|
||||
self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None
|
||||
self.request('POST', '/api/v1/auth/login', {'email':'domain-owner@example.test','password':'password'})
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
|
||||
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None; headers = {'Content-Type':'application/json'} if body else {}
|
||||
if self.cookie: headers['Cookie'] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse(); c = response.getheader('Set-Cookie')
|
||||
if c: self.cookie = c.split(';', 1)[0]
|
||||
return response.status, json.loads(response.read() or b'{}')
|
||||
|
||||
def test_check_cache_candidates_and_no_false_availability(self):
|
||||
status, business = self.request('POST', '/api/v1/businesses', {'name':'Acme Solar','website':'https://example.com','city':'Cape Town','province':'Western Cape'})
|
||||
self.assertEqual(status, 201); bid = business['id']
|
||||
with patch('app.main.resolve_domain', return_value={'status':'unknown','addresses':[],'checked_at':'2026-01-01T00:00:00+00:00','cache_expires_at':'2099-01-01T00:00:00+00:00'}):
|
||||
status, first = self.request('POST', f'/api/v1/businesses/{bid}/domains/check', {'domain':'example.com'})
|
||||
self.assertEqual(status, 200); self.assertIn(first['status'], ('unknown','ok'))
|
||||
status, second = self.request('GET', f'/api/v1/businesses/{bid}/domains/check?domain=example.com')
|
||||
self.assertEqual(status, 200); self.assertTrue(second.get('cache_hit'))
|
||||
status, candidates = self.request('GET', f'/api/v1/businesses/{bid}/domain-candidates')
|
||||
self.assertEqual(status, 200); self.assertTrue(candidates['items'])
|
||||
status, availability = self.request('POST', f'/api/v1/businesses/{bid}/domain-candidates/check-availability', {})
|
||||
self.assertEqual(status, 200); self.assertEqual(availability['status'], 'unknown'); self.assertEqual(availability['reason'], 'not_configured')
|
||||
self.assertEqual(self.request('GET', '/api/v1/domain-checks')[0], 200)
|
||||
|
||||
def test_business_domain_checks_are_tenant_scoped(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name':'Private'})
|
||||
self.cookie = None
|
||||
self.request('POST', '/api/v1/auth/logout')
|
||||
self.assertEqual(self.request('GET', f'/api/v1/businesses/{business["id"]}/domains/check?domain=example.com')[0], 401)
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
# ProspectOS web — Phase 6 boundary
|
||||
# ProspectOS web — Phase 7 boundary
|
||||
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts and Phase 6 normalization/deduplication concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach.
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts, Phase 6 normalization/deduplication, and Phase 7 domain-intelligence concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach.
|
||||
|
||||
## Configure and run
|
||||
|
||||
@@ -62,4 +62,4 @@ A zero-dependency static smoke page (`smoke-test.html`) checks the key DOM contr
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
The static client has no background discovery, DNS/website scanner, enrichment scheduler, outreach integration, or SSE delivery. It cannot make missing provenance authoritative and should display API-provided limitations rather than infer them. CSV preview is capped for display and is not an import workflow. Production deployment still requires durable job execution, tenant-scoped controls, idempotency verification, SSE/polling verification, and the security and operations gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`.
|
||||
The static client has no background discovery, DNS/website scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. For future domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability; any availability label requires an API result from an authorized provider and explicit human review. It cannot make missing provenance authoritative and should display API-provided limitations rather than infer them. CSV preview is capped for display and is not an import workflow. Production deployment still requires durable job execution, tenant-scoped controls, idempotency verification, PSL/DNS/cache implementation and tests, candidate review permissions/audit, authorized availability-provider controls, SSE/polling verification, and the security and operations gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`.
|
||||
|
||||
+34
-3
@@ -12,7 +12,7 @@
|
||||
const freshness = (p) => { const raw=p.updated_at||p.last_checked_at||p.created_at; if(!raw)return {label:'Unknown',cls:'stale'}; const days=Math.max(0,Math.floor((Date.now()-new Date(raw).getTime())/86400000)); return {label:days===0?'Today':`${days}d ago`,cls:days<=7?'good':'stale'}; };
|
||||
const scoreFor = (p) => Number.isFinite(Number(p.score)) ? Number(p.score) : (p.score_factors||p.factors||[]).reduce((n,f)=>n+({named_business:20,business_site:30,email:25,phone:15,description:10}[f]||0),0);
|
||||
const labelFactor = (f) => ({named_business:'Named business',business_site:'Business site',email:'Email found',phone:'Phone found',description:'Description'}[f] || String(f).replaceAll('_',' '));
|
||||
async function request(path, options = {}) { const response=await fetch(endpoint(path),{...options,credentials:'include'}); if(response.status===401){showLogin('Your session has expired. Please sign in again.');throw new Error('unauthorized');} return response; }
|
||||
async function request(path, options = {}) { const response=await fetch(endpoint(path),{...options,credentials:'include'}); if(response.status===401){showLogin('Your session has expired. Please sign in again.');throw new Error('unauthorized');} if(response.status===403)throw new Error('Tenant/workspace access denied.'); return response; }
|
||||
async function jsonRequest(path, options = {}) { const res=await request(path,options); const body=await res.json().catch(()=>({})); if(!res.ok)throw new Error(body.error||body.message||'Request failed'); return body; }
|
||||
function showLogin(message=''){currentUser=null;$('dashboardShell').hidden=true;$('loginScreen').hidden=false;$('loginMessage').textContent=message;$('loginMessage').className=`form-message${message?' error':''}`;}
|
||||
function showDashboard(user){currentUser=user||{};const name=currentUser.name||currentUser.full_name||currentUser.email||'Workspace member';const role=currentUser.role||currentUser.roles?.[0]||'Member';$('userIdentity').textContent=`${name} · ${role}`;$('userAvatar').textContent=name.split(/\s+/).map(x=>x[0]).join('').slice(0,2).toUpperCase();$('loginScreen').hidden=true;$('dashboardShell').hidden=false;updateJobPermissions();}
|
||||
@@ -25,7 +25,38 @@
|
||||
async function selectProspect(id){selectedId=Number(id);selectedDetail=null;renderRows();$('detailPanel').innerHTML='<div class="detail-loading" aria-live="polite">Loading prospect detail…</div>';await loadDetail(selectedId);}
|
||||
async function loadDetail(id){try{const detail=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}`);selectedDetail=detail;const index=prospects.findIndex(p=>Number(p.id)===Number(id));if(index>=0)prospects[index]={...prospects[index],...detail};renderDetail(detail);}catch(error){if(error.message!=='unauthorized')$('detailPanel').innerHTML=`<div class="detail-error" role="alert"><h3>Unable to load detail</h3><p>${esc(error.message)}</p><button class="button ghost" id="retryDetailBtn" type="button">Try again</button></div>`;}}
|
||||
const listItems=(items,empty,label)=>Array.isArray(items)&&items.length?`<ul class="detail-list">${items.map(item=>`<li>${esc(typeof item==='string'?item:item[label]||item.value||item.name||JSON.stringify(item))}</li>`).join('')}</ul>`:`<p class="muted">${empty}</p>`;
|
||||
function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderDedupPanel();}
|
||||
const domainValue = item => typeof item === 'string' ? item : item?.domain || item?.hostname || item?.name || item?.value || '';
|
||||
const domainForProspect = p => p?.website_domain || (Array.isArray(p?.domains) ? domainValue(p.domains[0]) : '') || '';
|
||||
const domainState = value => { const state=String(value ?? '').toLowerCase().replaceAll('_','-'); return state==='nxdomain'?'NXDOMAIN':state==='error'?'Error':state==='not-configured'?'Not configured':state==='unknown'||!state?'Unknown':state.replaceAll('-',' '); };
|
||||
const capabilityState = value => { if (value === true || ['present','found','yes','true','ok'].includes(String(value).toLowerCase())) return 'Present'; if (value === false || ['absent','not-found','no','false'].includes(String(value).toLowerCase())) return 'Not found'; return domainState(value); };
|
||||
const checkedTime = item => item?.checked_at || item?.checkedAt || item?.last_checked_at || item?.checked_time || 'Time unavailable';
|
||||
const cacheState = item => item?.cache_hit === true ? 'Hit' : item?.cache_state || item?.cacheStatus || item?.cache || (item?.cache_hit === false ? 'Miss' : 'Unknown');
|
||||
const confidenceText = item => item?.confidence ?? item?.score ?? 'Unknown';
|
||||
const reasonsText = item => { const reasons=item?.reasons || item?.reason || item?.explanation || []; return Array.isArray(reasons) ? reasons : [reasons]; };
|
||||
function renderDomainPanel(p){
|
||||
const panel=$('domainIntelligencePanel'); if(!panel)return;
|
||||
panel.innerHTML=`<div class="domain-panel-heading"><div><p class="eyebrow">DOMAIN INTELLIGENCE</p><h4>DNS checks</h4></div><button class="button ghost compact" id="runDomainCheckBtn" type="button">↻ Check DNS</button></div><p class="domain-safety">DNS results are evidence only. Unknown, error, and NXDOMAIN states are never treated as ownership or availability.</p><div id="domainCheckState" class="domain-check-state"><div class="detail-loading" aria-live="polite">Loading DNS results…</div></div><div class="candidate-domain-panel" id="candidateDomainPanel"><div class="detail-loading" aria-live="polite">Loading candidate domains…</div></div>`;
|
||||
loadDomainIntelligence(p.id);
|
||||
}
|
||||
function renderDomainCheck(result){
|
||||
const el=$('domainCheckState'); if(!el)return;
|
||||
if(!result){el.innerHTML='<p class="muted">No DNS check has been returned. Run a check to see evidence.</p>';return;}
|
||||
const dns=result.dns || result.dns_state || result.state || result.status || 'unknown', records=result.records || result.capabilities || result;
|
||||
const capability=(key, aliases=[])=>{for(const k of [key,...aliases])if(records?.[k]!==undefined)return records[k];return undefined;};
|
||||
el.innerHTML=`<div class="domain-check-meta"><span class="domain-state ${String(dns).toLowerCase().replaceAll('_','-')}">${esc(domainState(dns))}</span><span>Checked ${esc(checkedTime(result))}</span><span>Cache: ${esc(domainState(cacheState(result)))}</span></div><dl class="dns-capabilities">${['A','AAAA','MX','NS','TXT'].map(type=>`<div><dt>${type}</dt><dd>${esc(capabilityState(capability(type,type.toLowerCase())))}</dd></div>`).join('')}</dl>${result.error||result.error_code||result.message?`<p class="domain-error" role="alert">${esc(result.error||result.error_code||result.message)}</p>`:''}`;
|
||||
}
|
||||
function renderCandidates(payload){
|
||||
const panel=$('candidateDomainPanel'); if(!panel)return;
|
||||
const candidates=payloadItems(payload,['candidates','candidate_domains','domains','items']);
|
||||
panel.innerHTML=`<div class="domain-panel-heading"><div><p class="eyebrow">CANDIDATE DOMAINS</p><h4>Possible matches <span class="count">${candidates.length}</span></h4></div><span class="small-label">Human review required</span></div>${candidates.length?`<div class="candidate-list">${candidates.map((item,index)=>{const domain=domainValue(item);return `<article class="candidate-card" data-candidate-index="${index}"><div class="candidate-head"><strong>${esc(domain||'Unknown domain')}</strong><span class="match-confidence">${esc(confidenceText(item))} confidence</span></div><ul class="match-reasons">${reasonsText(item).map(reason=>`<li>${esc(typeof reason==='string'?reason:reason?.text||reason?.description||JSON.stringify(reason))}</li>`).join('')||'<li>Reason unavailable</li>'}</ul><button class="button ghost compact" type="button" data-domain-availability="${esc(domain)}">Check availability</button><p class="availability-result" aria-live="polite"></p></article>`;}).join('')}</div>`:'<p class="muted">No candidate domains returned.</p>'}<p class="domain-safety compact-safety">Availability is not configured here. This action will never claim a domain is available or owned.</p>`;
|
||||
}
|
||||
async function loadDomainIntelligence(id){
|
||||
const check=$('domainCheckState'), candidates=$('candidateDomainPanel');
|
||||
try{const [checkPayload,candidatePayload]=await Promise.all([jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/domains/check?domain=${encodeURIComponent(domainForProspect(selectedDetail || prospects.find(item=>Number(item.id)===Number(id)) || {}))}`),jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/domain-candidates`)]);renderDomainCheck(checkPayload);renderCandidates(candidatePayload);}catch(error){if(error.message!=='unauthorized'){if(check)check.innerHTML=`<div class="detail-error" role="alert"><strong>Unable to load DNS results</strong><p>${esc(error.message)}</p><button class="button ghost compact" id="retryDomainBtn" type="button">Try again</button></div>`;if(candidates)candidates.innerHTML='<p class="muted">Candidate domains unavailable.</p>';}}
|
||||
}
|
||||
async function runDomainCheck(){const button=$('runDomainCheckBtn');if(!selectedId||!button)return;button.disabled=true;const state=$('domainCheckState');if(state)state.innerHTML='<div class="detail-loading" aria-live="polite">Checking DNS…</div>';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domains/check`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain:domainForProspect(selectedDetail||{})})});renderDomainCheck(result);}catch(error){if(error.message!=='unauthorized'&&state)state.innerHTML=`<div class="detail-error" role="alert"><strong>DNS check failed</strong><p>${esc(error.message)}</p></div>`;}finally{button.disabled=false;}}
|
||||
async function checkDomainAvailability(domain,button){const result=button.closest('.candidate-card')?.querySelector('.availability-result');if(!domain||!result)return;button.disabled=true;result.textContent='Checking availability…';try{const body=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domain-candidates/check-availability`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain})});const state=body?.status||body?.state||body?.availability||'unknown';result.textContent=['not-configured','unknown','error'].includes(String(state).toLowerCase())?`Availability: ${domainState(state)}. No ownership conclusion.`:'Availability: Unknown. No ownership conclusion.';}catch(error){if(error.message!=='unauthorized')result.textContent=`Availability: Unknown. ${error.message}`;}finally{button.disabled=false;}}
|
||||
function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderDomainPanel(p);renderDedupPanel();}
|
||||
let mergeSource = null, mergeTarget = null, mergeBusy = false;
|
||||
const payloadItems = (payload, keys) => { for (const key of keys) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; };
|
||||
const suggestionId = item => item.target_id ?? item.business_id ?? item.prospect_id ?? item.id;
|
||||
@@ -112,7 +143,7 @@
|
||||
async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}}
|
||||
async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
|
||||
document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
$('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
|
||||
bootstrap();
|
||||
})();
|
||||
|
||||
@@ -33,5 +33,9 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
|
||||
,['Deduplication review UI contract',()=>!!d.querySelector('#mergeDialog')&&js.includes('/matches')&&js.includes('review-required')&&js.includes('confidence')&&js.includes('reasons')]
|
||||
,['Merge actions are explicit and reversible',()=>js.includes('/merge-history')&&js.includes('/reverse')&&js.includes('This action is reversible')&&js.includes('Confirm merge')&&!js.includes('autoMerge')]
|
||||
,['Deduplication loading and errors',()=>js.includes('Loading match suggestions')&&js.includes('Unable to load match suggestions')&&js.includes('merge-history')&&js.includes('Unable to load merge history')]
|
||||
,['Domain intelligence smoke marker and controls',()=>!!d.querySelector('[data-smoke="domain-intelligence"]')&&js.includes('/domains/check')&&js.includes('/domain-candidates')&&js.includes('runDomainCheckBtn')]
|
||||
,['DNS capability and uncertainty labels',()=>['DNS checks','A','AAAA','MX','NS','TXT','NXDOMAIN','Unknown','Error','Cache'].every(x=>js.includes(x))]
|
||||
,['Candidate confidence, reasons, and safe availability',()=>js.includes('confidence')&&js.includes('reasons')&&js.includes('check-availability')&&js.includes('Not configured')&&js.includes('No ownership conclusion')&&!js.includes('Domain is available')]
|
||||
,['Domain loading, error, tenant, and auth states',()=>js.includes('Loading DNS results')&&js.includes('Unable to load DNS results')&&js.includes('Tenant/workspace access denied')&&js.includes("credentials:'include'")]
|
||||
];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
|
||||
</script>
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user