add public contact extraction
This commit is contained in:
Marco0300
@@ -120,6 +120,16 @@ Scan history is tenant-scoped and append-oriented. Results and cache entries are
|
||||
|
||||
The website scanner remains a pilot boundary. Compose does not provide a production egress proxy, durable scan queue, distributed crawl coordinator, hardened DNS resolver, or compliance-grade result store. Production still requires independent SSRF testing (including DNS rebinding and redirect chains), egress/network policy, resource isolation, durable retention/deletion, authenticated scan-history authorization, rate limits and abuse controls, observability, and a reviewed policy for content types, robots/terms, caching, and incident response. Scans must never trigger acquisition, verification, enrichment, or outreach automatically.
|
||||
|
||||
## Phase 9 public official-site contact extraction boundary
|
||||
|
||||
Phase 9 adds a **passive, suggestion-only** contact-observation workflow. When explicitly enabled, extraction may inspect bounded HTML from the business's approved/public official-site origin and its same-site contact/about pages; it is not general web search, crawling, enrichment, identity verification, or outreach. Only public page content and explicitly permitted `mailto:`/visible contact values may be considered. Do not submit forms, authenticate, bypass access controls, probe SMTP, send test messages, or contact a person or organization.
|
||||
|
||||
Every candidate contact must retain provenance: source URL and page location/context, extraction method, observed time, scanner/extractor and policy versions, and the exact uncertainty/reason code. Confidence is an explainable review signal, not deliverability, consent, ownership, or permission to contact. Classify role addresses separately from person addresses and classify free-mail domains separately from business-domain addresses; neither classification is proof of identity. Syntax validation is only a parse result. MX/DNS status is independently uncertain (`not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`), and no MX result may be presented as deliverability.
|
||||
|
||||
False-positive exclusions must reject or quarantine values from asset URLs, image/file names, scripts/styles, example/test/placeholder domains, documentation text, tracking addresses, and malformed or unsupported schemes. Apply tenant-scoped suppressions before a candidate is persisted, returned, exported, or queued for review; suppressed values remain do-not-contact and suppression always wins over confidence, role, syntax, MX, pipeline, or verification state. Extraction is bounded by per-request and aggregate page/URL, byte, time, redirect, candidate, and concurrency limits. Store only the minimum contact value and lineage required for review, apply a documented retention/deletion class, and redact secrets and unnecessary personal data from logs and audit events.
|
||||
|
||||
Phase 9 does not authorize automated outreach. There is no SMTP probing, SMTP banner/VRFY/EXPN check, email validation message, send endpoint, campaign queue, or follow-up action. An extracted address is an observation requiring human review and explicit policy authorization before any separate future contact workflow.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
|
||||
+13
-1
@@ -1,4 +1,4 @@
|
||||
# Prospect Platform API — Phase 8 boundary
|
||||
# Prospect Platform API — Phase 9 boundary
|
||||
|
||||
Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
|
||||
|
||||
@@ -68,6 +68,8 @@ Implementations should expose source/query/job state without leaking raw payload
|
||||
The business detail includes the supported child collections: `contacts`, `domains`, `websites`, `evidence`, `pipeline`, and `notes`. The child collection routes are:
|
||||
|
||||
- `POST /api/v1/businesses/{id}/contacts` — manually add a contact; suppression matching marks a matching contact as do-not-contact.
|
||||
- `GET /api/v1/businesses/{id}/contacts/extract` — read the tenant-scoped Phase 9 official-site extraction projection.
|
||||
- `POST /api/v1/businesses/{id}/contacts/extract` — extract bounded public contacts from the approved official site/same-site pages and persist provenance-bearing observations; the request is passive and must not probe SMTP or send outreach. Suppression matching marks matches `suppressed`/`do_not_contact` and remains authoritative.
|
||||
- `POST /api/v1/businesses/{id}/domains` — manually add a domain observation.
|
||||
- `POST /api/v1/businesses/{id}/websites` — manually add a website observation/classification.
|
||||
- `POST /api/v1/businesses/{id}/evidence` — manually add evidence with its kind, claim, and source URL/reference. This records provenance supplied by the operator; it does not scan or independently verify the URL.
|
||||
@@ -126,6 +128,16 @@ Classifications must be conservative, explainable, and derived only from bounded
|
||||
|
||||
Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
|
||||
|
||||
## Phase 9 official-site contact extraction contract
|
||||
|
||||
The optional Phase 9 extractor is a passive, authenticated, tenant-scoped observation of a business's approved/public official-site origin. It may inspect bounded HTML and same-site contact/about pages only; it must not become a search engine, unrestricted crawler, or arbitrary URL fetcher. The extractor must use the existing SSRF-safe URL, redirect, content-type, and resource-budget controls, and must never submit forms, execute target JavaScript, use credentials/cookies, probe SMTP, issue SMTP `VRFY`/`EXPN`, send validation mail, or perform outreach.
|
||||
|
||||
For each candidate, return/store the normalized address only with provenance (official-site/page URL, page or DOM context, extraction method, observed time, extractor/policy version) and an explainable confidence/reason list. Preserve uncertainty rather than inventing facts. `syntax_valid`/`syntax_invalid` is a parser outcome only. Role classification (`role`/`person`/`unknown`) and free-mail classification (`free_mail`/`business_domain`/`unknown`) are independent review labels; they do not prove identity, consent, ownership, or deliverability. MX/DNS is a separate observation with resolver/source, observed time, TTL/freshness where available, and one of `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`; MX absence or failure remains unknown and must never be treated as invalid or undeliverable.
|
||||
|
||||
Exclude false positives before persistence and response: values in scripts/styles/comments or asset URLs/file names, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party pages. Enforce hard limits for total extraction time, pages/URLs, redirects, response and retained bytes, candidates per page/request, and concurrency. Suppression matching is server-side and tenant-scoped, before storing, returning, exporting, or presenting a candidate; suppressed contacts are marked do-not-contact and cannot be revived by a later classification or review action. Retention must be explicit and bounded for extracted values, page provenance, DNS/MX observations, caches, and audit records; logs must not contain full contact payloads when a redacted identifier is sufficient.
|
||||
|
||||
Extraction results are suggestions only and do not create a send/contact capability. The API exposes no SMTP-probe, validation-message, outreach, or campaign endpoint. If the feature is disabled, unapproved, over limit, blocked, or uncertain, fail closed with an explicit status/reason rather than an empty successful result. The current MVP remains pilot-only until extraction limits, suppression enforcement, retention/deletion jobs, provenance/audit coverage, and tenant-isolation tests are production hardened.
|
||||
|
||||
## Remaining limitations and production migration work
|
||||
|
||||
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Conservative extraction of public business contact signals from approved HTML.
|
||||
|
||||
This module never fetches URLs. Callers must provide HTML obtained from the approved
|
||||
website scanner and a source URL already associated with the business.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import html as html_lib
|
||||
import re
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
MAX_HTML_BYTES = 512 * 1024
|
||||
MAX_RESULTS = 100
|
||||
EMAIL_RE = re.compile(r"(?<![\w.+-])[A-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?(?:\.[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?)+", re.I)
|
||||
PHONE_RE = re.compile(r"(?<!\w)(\+?\d[\d ()().-]{6,}\d)(?!\w)")
|
||||
SOCIAL_HOSTS = {"facebook.com", "instagram.com", "linkedin.com", "twitter.com", "x.com", "youtube.com", "tiktok.com"}
|
||||
FREE_MAIL = {"gmail.com", "googlemail.com", "yahoo.com", "yahoo.co.uk", "hotmail.com", "outlook.com", "live.com", "icloud.com", "aol.com", "proton.me", "protonmail.com", "mail.com"}
|
||||
ROLE_NAMES = {"info", "hello", "contact", "sales", "support", "admin", "office", "enquiries", "inquiries", "accounts", "billing", "careers", "hr", "help", "marketing", "bookings", "reception", "service", "customerservice"}
|
||||
EXAMPLE_DOMAINS = {"example.com", "example.org", "example.net", "example.test", "invalid", "localhost"}
|
||||
|
||||
|
||||
def _clean(value: str) -> str:
|
||||
return re.sub(r"\s+", " ", html_lib.unescape(value or "")).strip()
|
||||
|
||||
|
||||
def _decode_obfuscation(value: str) -> str:
|
||||
value = html_lib.unescape(unquote(value or ""))
|
||||
value = re.sub(r"\s*(?:\[|\(|\{|\s)at(?:\]|\)|\}|\s)\s*", "@", value, flags=re.I)
|
||||
value = re.sub(r"\s*(?:\[|\(|\{|\s)dot(?:\]|\)|\}|\s)*", ".", value, flags=re.I)
|
||||
return value
|
||||
|
||||
|
||||
def valid_email(value: str) -> bool:
|
||||
value = value.strip().lower()
|
||||
if len(value) > 254 or value.count("@") != 1 or value.split("@", 1)[1] in EXAMPLE_DOMAINS:
|
||||
return False
|
||||
return bool(EMAIL_RE.fullmatch(value)) and not any(x in value for x in ("password", "token", "secret", "credential", "apikey"))
|
||||
|
||||
|
||||
def normalize_phone(value: str) -> str:
|
||||
value = value.strip()
|
||||
digits = re.sub(r"\D", "", value)
|
||||
if value.startswith("+") and digits:
|
||||
return "+" + digits
|
||||
return digits
|
||||
|
||||
|
||||
class _ContactParser(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.text: list[str] = []
|
||||
self.links: list[tuple[str, str]] = []
|
||||
self.form_fields: list[str] = []
|
||||
self._ignore = 0
|
||||
self._anchor = ""
|
||||
self._in_form = False
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = {str(k).lower(): str(v or "") for k, v in attrs}
|
||||
tag = tag.lower()
|
||||
if tag in {"script", "style", "noscript", "template", "svg"}:
|
||||
self._ignore += 1
|
||||
if tag == "a": self._anchor = attrs.get("href", "")
|
||||
if tag == "form": self._in_form = True
|
||||
if self._in_form and tag in {"input", "textarea", "select"}:
|
||||
name = attrs.get("name", "") or attrs.get("id", "")
|
||||
typ = attrs.get("type", "")
|
||||
self.form_fields.append(_clean(" ".join((name, typ, attrs.get("placeholder", "")))))
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
tag = tag.lower()
|
||||
if tag in {"script", "style", "noscript", "template", "svg"} and self._ignore: self._ignore -= 1
|
||||
if tag == "a": self._anchor = ""
|
||||
if tag == "form": self._in_form = False
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._ignore: return
|
||||
if data.strip(): self.text.append(data)
|
||||
if self._anchor: self.links.append((self._anchor, data))
|
||||
|
||||
|
||||
def _record(kind, value, label, source_url, confidence, classification="unknown", *, suppressed=False, provenance="visible_text"):
|
||||
return {"kind": kind, "value": value, "label": label[:200], "classification": classification,
|
||||
"confidence": round(max(0.0, min(1.0, confidence)), 2), "source_url": source_url,
|
||||
"public_business": True, "mx_status": "unknown", "suppressed": bool(suppressed),
|
||||
"do_not_contact": bool(suppressed), "provenance": provenance}
|
||||
|
||||
|
||||
def extract_contacts(source_html: str, source_url: str, *, suppressions=None, max_results=MAX_RESULTS) -> list[dict]:
|
||||
if not isinstance(source_html, str) or len(source_html.encode("utf-8")) > MAX_HTML_BYTES:
|
||||
raise ValueError("html_too_large")
|
||||
try: max_results = int(max_results)
|
||||
except (ValueError, TypeError): raise ValueError("invalid_limits")
|
||||
if max_results < 1 or max_results > MAX_RESULTS: raise ValueError("invalid_limits")
|
||||
parser = _ContactParser(); parser.feed(source_html)
|
||||
visible = _decode_obfuscation(_clean(" ".join(parser.text)))
|
||||
suppression = {(str(x.get("kind", "")), str(x.get("value", "")).lower()) for x in (suppressions or [])}
|
||||
out, seen = [], set()
|
||||
def add(kind, value, label, confidence, classification="unknown", provenance="visible_text"):
|
||||
value = value.strip().lower() if kind == "email" else value.strip()
|
||||
if kind == "email":
|
||||
if not valid_email(value): return
|
||||
local, domain = value.rsplit("@", 1)
|
||||
classification = "free_mail" if domain in FREE_MAIL else ("role" if local in ROLE_NAMES else "named")
|
||||
key = (kind, value); blocked = ("email", value) in suppression or ("domain", domain) in suppression
|
||||
else:
|
||||
if kind in {"phone", "whatsapp"}: value = normalize_phone(value)
|
||||
if len(re.sub(r"\D", "", value)) < 7: return
|
||||
key = (kind, value); blocked = (kind, value.lower()) in suppression
|
||||
if key in seen or len(out) >= max_results: return
|
||||
seen.add(key); out.append(_record(kind, value, label or kind.title(), source_url, confidence, classification, suppressed=blocked, provenance=provenance))
|
||||
for href, anchor_text in parser.links:
|
||||
raw = _decode_obfuscation(href)
|
||||
if raw.lower().startswith("mailto:"):
|
||||
address = raw[7:].split("?", 1)[0]
|
||||
add("email", address, _clean(anchor_text), 0.98, provenance="mailto")
|
||||
elif raw.lower().startswith("tel:"):
|
||||
add("phone", raw[4:].split("?", 1)[0], _clean(anchor_text), 0.98, provenance="tel")
|
||||
else:
|
||||
parsed = urlparse(raw if "://" in raw else "https://" + raw)
|
||||
host = (parsed.hostname or "").lower().removeprefix("www.")
|
||||
if host == "wa.me" or "whatsapp" in host:
|
||||
number = re.sub(r"\D", "", parsed.path)
|
||||
if number: add("whatsapp", "+" + number, _clean(anchor_text) or "WhatsApp", 0.98, provenance="whatsapp_link")
|
||||
elif any(host == d or host.endswith("." + d) for d in SOCIAL_HOSTS):
|
||||
add("social", raw, _clean(anchor_text) or host, 0.95, provenance="social_link")
|
||||
for match in EMAIL_RE.finditer(visible):
|
||||
context = visible[max(0, match.start() - 40):match.start()]
|
||||
if re.search(r"(?:password|passwd|token|secret|credential|api[_ -]?key|authorization)\s*[:=]?\s*$", context, re.I):
|
||||
continue
|
||||
add("email", match.group(0), "Email", 0.88)
|
||||
for value in PHONE_RE.findall(visible): add("phone", value, "Phone", 0.82)
|
||||
# A form is provenance, not proof of a destination address.
|
||||
if parser.form_fields:
|
||||
for field in parser.form_fields[:5]:
|
||||
if re.search(r"email|contact|phone|whatsapp", field, re.I):
|
||||
add("form", field, "Contact form", 0.7, provenance="form_field")
|
||||
return out
|
||||
@@ -11,11 +11,13 @@ if __package__ in (None, ""):
|
||||
from app.sources import adapter_for, contains_secret
|
||||
from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
from app.website_scanner import scan_website, validate_url
|
||||
from app.contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS
|
||||
else:
|
||||
from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from .sources import adapter_for, contains_secret
|
||||
from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
from .website_scanner import scan_website, validate_url
|
||||
from .contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS
|
||||
ORGANIZATION_ID = "demo-tenant"
|
||||
SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
|
||||
SESSION_DAYS = 7
|
||||
@@ -25,6 +27,7 @@ JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"}
|
||||
JOB_PAGE_SIZE = 100
|
||||
WEBSITE_SCAN_PAGE_SIZE = 100
|
||||
WEBSITE_SCAN_CACHE_SECONDS = 3600
|
||||
CONTACT_EXTRACTION_PAGE_SIZE = 100
|
||||
SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
|
||||
CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
|
||||
|
||||
@@ -125,6 +128,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/website-scans": return self.list_website_scans(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/contact-extractions": return self.list_contact_extractions(db,org,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/businesses/"):
|
||||
bits=path.split("/"); ident=bits[4] if len(bits)>4 else ""
|
||||
@@ -133,6 +137,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
if len(bits)==7 and bits[5:]==["websites","scan"]: return self.get_latest_website_scan(int(ident),db,user)
|
||||
if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query))
|
||||
if len(bits)==7 and bits[5:]==["contacts","extract"]: return self.send_json(405,{"error":"method_not_allowed"})
|
||||
if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org)
|
||||
if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org)
|
||||
payload=row_json(row); payload.update(self.nested(db,int(ident),org)); return self.send_json(200,payload)
|
||||
@@ -194,6 +199,64 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
rows = db.execute("SELECT * FROM website_scans WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
|
||||
return self.send_json(200, {"organization_id": org, "items": [self._website_scan_result(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def _contact_json(self, row):
|
||||
result = row_json(row)
|
||||
for key in ("public_business", "suppressed", "do_not_contact"):
|
||||
if key in result: result[key] = bool(result[key])
|
||||
return result
|
||||
|
||||
def list_contact_extractions(self, db, org, query):
|
||||
try:
|
||||
limit = int((query.get("page_size") or [50])[0]); offset = max(0, int((query.get("offset") or [0])[0]))
|
||||
if limit < 1 or limit > CONTACT_EXTRACTION_PAGE_SIZE: raise ValueError
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
|
||||
params = [org]; where = ["organization_id=?"]
|
||||
business_id = (query.get("business_id") or [""])[0]
|
||||
if business_id.isdigit(): where.append("business_id=?"); params.append(int(business_id))
|
||||
rows = db.execute("SELECT * FROM contact_extractions WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
|
||||
return self.send_json(200, {"organization_id": org, "items": [self._contact_json(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def extract_business_contacts(self, bid, payload, db, user):
|
||||
org = user["organization_id"]; business = self.business(db, bid, org)
|
||||
if not business: return self.send_json(404, {"error": "not_found"})
|
||||
scan_id = payload.get("website_scan_id", payload.get("scan_id"))
|
||||
scan = None
|
||||
if scan_id is not None:
|
||||
if not isinstance(scan_id, int): return self.send_json(400, {"error": "invalid_scan"})
|
||||
scan = db.execute("SELECT * FROM website_scans WHERE id=? AND business_id=? AND organization_id=?", (scan_id, bid, org)).fetchone()
|
||||
if not scan: return self.send_json(404, {"error": "scan_not_found"})
|
||||
try: stored = json.loads(scan["result_json"] or "{}")
|
||||
except (TypeError, ValueError): stored = {}
|
||||
source_url = str(payload.get("source_url") or scan["input_url"]).strip()
|
||||
source_html = payload.get("html") if isinstance(payload.get("html"), str) else stored.get("html")
|
||||
if source_html is None: return self.send_json(409, {"error": "scan_html_unavailable"})
|
||||
if source_url != scan["input_url"] and source_url != (stored.get("final_url") or ""): return self.send_json(400, {"error": "source_not_approved"})
|
||||
else:
|
||||
source_url = str(payload.get("source_url") or "").strip()
|
||||
source_html = payload.get("html")
|
||||
if not source_url or not isinstance(source_html, str): return self.send_json(400, {"error": "approved_scan_required"})
|
||||
parsed = urlparse(source_url)
|
||||
official_hosts = {str(business["website_domain"]).lower().strip(".")}
|
||||
if business["website"]:
|
||||
official_hosts.add((urlparse(business["website"]).hostname or "").lower().strip("."))
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.hostname.lower().strip(".") not in official_hosts:
|
||||
return self.send_json(400, {"error": "source_not_approved"})
|
||||
if len(source_html.encode("utf-8")) > MAX_HTML_BYTES: return self.send_json(413, {"error": "html_too_large"})
|
||||
try: requested_limit = int(payload.get("limit", MAX_RESULTS))
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_limits"})
|
||||
if requested_limit < 1 or requested_limit > MAX_RESULTS: return self.send_json(400, {"error": "invalid_limits"})
|
||||
key = str(payload.get("idempotency_key") or hashlib.sha256((str(scan_id or "") + source_url + source_html).encode()).hexdigest())[:200]
|
||||
existing = db.execute("SELECT * FROM contact_extractions WHERE organization_id=? AND business_id=? AND extraction_key=? ORDER BY id", (org, bid, key)).fetchall()
|
||||
if existing: return self.send_json(200, {"business_id": bid, "extraction_key": key, "items": [self._contact_json(r) for r in existing], "idempotent": True})
|
||||
suppressions = [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=?", (org,))]
|
||||
try: found = extract_contacts(source_html, source_url, suppressions=suppressions, max_results=requested_limit)
|
||||
except ValueError as exc: return self.send_json(400, {"error": str(exc)})
|
||||
for item in found:
|
||||
db.execute("INSERT INTO contact_extractions(organization_id,business_id,website_scan_id,extraction_key,kind,value,label,classification,confidence,source_url,public_business,mx_status,suppressed,do_not_contact,provenance) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", (org,bid,scan["id"] if scan else None,key,item["kind"],item["value"],item["label"],item["classification"],item["confidence"],item["source_url"],int(item["public_business"]),item["mx_status"],int(item["suppressed"]),int(item["do_not_contact"]),item["provenance"]))
|
||||
self.audit(db, user, "contacts.extracted", f"{bid}:{len(found)}:{key}"); db.commit()
|
||||
rows = db.execute("SELECT * FROM contact_extractions WHERE organization_id=? AND business_id=? AND extraction_key=? ORDER BY id", (org,bid,key)).fetchall()
|
||||
return self.send_json(201, {"business_id": bid, "extraction_key": key, "items": [self._contact_json(r) for r in rows], "idempotent": False})
|
||||
|
||||
def get_latest_website_scan(self, bid, db, user):
|
||||
row = db.execute("SELECT * FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "scan_not_found"})
|
||||
@@ -333,8 +396,10 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/sources":return self.create_source(payload,db,user)
|
||||
if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
|
||||
if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
|
||||
if path=="/api/v1/contact-extractions": return self.send_json(405,{"error":"method_not_allowed"})
|
||||
if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"websites"] and path.split("/")[6]=="scan": return self.scan_business_website(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"contacts"] and path.split("/")[6]=="extract": return self.extract_business_contacts(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"):
|
||||
|
||||
@@ -169,6 +169,8 @@ def scan_website(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int =
|
||||
if not 0 < int(max_redirects) <= MAX_REDIRECTS or not 0 < int(max_pages) <= MAX_PAGES: raise ValueError("invalid_limits")
|
||||
fetched = _fetch(url, timeout=max(0.1, min(float(timeout), 10.0)), max_bytes=max(1, min(int(max_bytes), MAX_BYTES)), max_redirects=int(max_redirects))
|
||||
result.update({k: fetched[k] for k in ("status", "final_url", "redirect_chain", "elapsed_ms", "tls", "certificate_status")}); result["size_bytes"] = len(fetched["body"])
|
||||
if fetched["content_type"] in {"text/html", "application/xhtml+xml"}:
|
||||
result["html"] = fetched["body"].decode("utf-8", "replace")
|
||||
if fetched["content_type"] not in {"text/html", "application/xhtml+xml"}:
|
||||
result["classification"] = classify_website(fetched["status"], fetched["final_url"], ""); return result
|
||||
parser = _PageParser(); parser.feed(fetched["body"].decode("utf-8", "replace"))
|
||||
|
||||
@@ -188,3 +188,27 @@ CREATE TABLE IF NOT EXISTS website_scans (
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_org ON website_scans(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_business ON website_scans(organization_id,business_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_cache ON website_scans(organization_id,cache_key,cache_expires_at);
|
||||
|
||||
-- Phase 9 contact extraction provenance. MX is intentionally a state value, not a probe.
|
||||
CREATE TABLE IF NOT EXISTS contact_extractions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id),
|
||||
business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
|
||||
website_scan_id INTEGER REFERENCES website_scans(id) ON DELETE SET NULL,
|
||||
extraction_key TEXT NOT NULL,
|
||||
kind TEXT NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
label TEXT NOT NULL DEFAULT '',
|
||||
classification TEXT NOT NULL DEFAULT 'unknown',
|
||||
confidence REAL NOT NULL DEFAULT 0,
|
||||
source_url TEXT NOT NULL,
|
||||
public_business INTEGER NOT NULL DEFAULT 1,
|
||||
mx_status TEXT NOT NULL DEFAULT 'unknown',
|
||||
suppressed INTEGER NOT NULL DEFAULT 0,
|
||||
do_not_contact INTEGER NOT NULL DEFAULT 0,
|
||||
provenance TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,business_id,extraction_key,kind,value)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_contact_extractions_org ON contact_extractions(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_contact_extractions_business ON contact_extractions(organization_id,business_id,id DESC);
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server
|
||||
|
||||
|
||||
class ContactExtractionApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory()
|
||||
os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'extract-owner@example.test'
|
||||
os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password'
|
||||
self.server = create_server('127.0.0.1', 0, self.tmp.name + '/db.sqlite')
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
|
||||
self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None
|
||||
self.request('POST', '/api/v1/auth/login', {'email': 'extract-owner@example.test', 'password': 'password'})
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
|
||||
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None
|
||||
headers = {'Content-Type': 'application/json'} if body else {}
|
||||
if self.cookie: headers['Cookie'] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse()
|
||||
cookie = response.getheader('Set-Cookie')
|
||||
if cookie: self.cookie = cookie.split(';', 1)[0]
|
||||
return response.status, json.loads(response.read() or b'{}')
|
||||
|
||||
def test_official_html_extracts_with_provenance_suppression_and_idempotency(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Acme', 'website': 'https://acme.test'})
|
||||
self.request('POST', '/api/v1/suppressions', {'kind': 'email', 'value': 'sales@acme.test'})
|
||||
payload = {'source_url': 'https://acme.test/contact', 'html': '<a href="mailto:sales@acme.test">Sales</a><p>info [at] acme [dot] test</p>', 'idempotency_key': 'extract-1'}
|
||||
status, result = self.request('POST', f"/api/v1/businesses/{business['id']}/contacts/extract", payload)
|
||||
self.assertEqual(status, 201); self.assertEqual(len(result['items']), 2)
|
||||
sales = next(x for x in result['items'] if x['value'] == 'sales@acme.test')
|
||||
self.assertTrue(sales['suppressed']); self.assertTrue(sales['do_not_contact']); self.assertEqual(sales['provenance'], 'mailto'); self.assertEqual(sales['mx_status'], 'unknown')
|
||||
self.assertEqual(self.request('POST', f"/api/v1/businesses/{business['id']}/contacts/extract", payload)[1]['idempotent'], True)
|
||||
self.assertEqual(self.request('GET', '/api/v1/contact-extractions?page_size=1')[1]['limit'], 1)
|
||||
|
||||
def test_arbitrary_and_oversized_sources_are_rejected(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Acme', 'website': 'https://acme.test'})
|
||||
path = f"/api/v1/businesses/{business['id']}/contacts/extract"
|
||||
self.assertEqual(self.request('POST', path, {'source_url': 'https://evil.test', 'html': '<p>x@y.test</p>'})[1]['error'], 'source_not_approved')
|
||||
self.assertEqual(self.request('POST', path, {'source_url': 'https://acme.test', 'html': 'x' * (512 * 1024 + 1)})[0], 413)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,42 @@
|
||||
import unittest
|
||||
from app.contact_extractor import extract_contacts
|
||||
|
||||
|
||||
class ContactExtractorTests(unittest.TestCase):
|
||||
def test_extracts_public_mailto_obfuscated_phone_and_ignores_false_positives(self):
|
||||
html = '''<html><body>
|
||||
<a href="mailto:sales@acme.test">Email Sales</a>
|
||||
<span>info [at] acme.test</span><span>+27 (12) 345-6789</span>
|
||||
<a href="https://wa.me/27123456789">WhatsApp</a>
|
||||
<script>const token = 'abc@example.com';</script>
|
||||
<img src="https://cdn.thirdparty.test/x?email=bad@thirdparty.test">
|
||||
<span>john@example.com</span>
|
||||
</body></html>'''
|
||||
result = extract_contacts(html, 'https://acme.test/contact')
|
||||
values = {(x['kind'], x['value']) for x in result}
|
||||
self.assertIn(('email', 'sales@acme.test'), values)
|
||||
self.assertIn(('email', 'info@acme.test'), values)
|
||||
self.assertIn(('phone', '+27123456789'), values)
|
||||
self.assertIn(('whatsapp', '+27123456789'), values)
|
||||
self.assertNotIn(('email', 'abc@example.com'), values)
|
||||
self.assertNotIn(('email', 'bad@thirdparty.test'), values)
|
||||
self.assertNotIn(('email', 'john@example.com'), values)
|
||||
|
||||
def test_excludes_credentials_adjacent_to_email_like_values(self):
|
||||
result = extract_contacts('<p>API key: foo@acme.test password: bar@acme.test</p>', 'https://acme.test/')
|
||||
self.assertEqual(result, [])
|
||||
|
||||
def test_classifies_role_named_free_mail_and_suppression(self):
|
||||
html = '<p>support@acme.test alice@acme.test bob@gmail.com</p>'
|
||||
result = extract_contacts(html, 'https://acme.test/', suppressions=[{'kind':'email','value':'support@acme.test'}])
|
||||
by = {x['value']: x for x in result}
|
||||
self.assertEqual(by['support@acme.test']['classification'], 'role')
|
||||
self.assertTrue(by['support@acme.test']['do_not_contact'])
|
||||
self.assertTrue(by['support@acme.test']['suppressed'])
|
||||
self.assertEqual(by['alice@acme.test']['classification'], 'named')
|
||||
self.assertEqual(by['bob@gmail.com']['classification'], 'free_mail')
|
||||
self.assertEqual(by['alice@acme.test']['mx_status'], 'unknown')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+10
-2
@@ -1,4 +1,4 @@
|
||||
# ProspectOS web — Phase 8 boundary
|
||||
# ProspectOS web — Phase 9 boundary
|
||||
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow, Phase 4 MVP job monitor, and Phase 8 scan-result/history presentation when supplied by the API. The browser does not fetch targets, submit forms, execute scan JavaScript, or send outreach; SSRF controls and budgets are server-side.
|
||||
|
||||
@@ -66,6 +66,14 @@ The current client renders job status/counts, detail, structured errors, progres
|
||||
|
||||
A zero-dependency static smoke page (`smoke-test.html`) checks the key DOM contract in an iframe and reports pass/fail. Browser smoke coverage should treat pagination, detail child records, provenance rendering, pipeline/notes actions, and the disabled outreach state as separate checks.
|
||||
|
||||
## Phase 9 contact-observation UI contract
|
||||
|
||||
When the API supplies Phase 9 results, the UI may render contacts extracted from the approved/public official site and same-site contact/about pages. Display source URL/page context, extraction method, observed time, extractor/policy version, confidence and reasons, syntax status, role classification, free-mail classification, and independent MX/DNS status/freshness. Use explicit labels such as **Observation**, **Human review required**, and **Unknown**; never label a candidate verified, deliverable, owned, consented, or ready for outreach. The UI must show suppressed contacts as **Do not contact**, preserve the suppression reason, and never hide or override suppression through filters, refreshes, exports, or cached results.
|
||||
|
||||
Extraction is not a browser crawler. The browser must not fetch target pages directly, submit contact forms, execute target JavaScript, send credentials/cookies, probe SMTP, send validation messages, or expose an outreach/send control. Render bounded/partial/blocked/timeout/error results distinctly from an empty successful result, including candidate/page/byte/time limit reasons. Do not display false-positive candidates from assets, scripts/styles, example/test/placeholder values, tracking addresses, malformed schemes, or unrelated third-party pages.
|
||||
|
||||
The API remains authoritative for official-site scope, tenant isolation, suppression enforcement, limits, retention, provenance, and permissions. A cached extraction must show its observed time and freshness, never “live.” Candidate confidence, role/free-mail labels, syntax, and MX/DNS uncertainty are review metadata only and cannot enable a contact action.
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
The static client has no client-side crawler, scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. It can display server-provided Phase 8 observations, but production still requires server-side SSRF/DNS-rebinding/redirect controls, hard size/time/crawl budgets, durable scan history/cache isolation and retention, abuse/rate controls, and authenticated permission/audit coverage. For domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability. CSV preview is capped for display and is not an import workflow.
|
||||
The static client has no client-side crawler, scanner, contact extractor, enrichment scheduler, outreach integration, availability provider, or SSE delivery. It can display server-provided Phase 9 observations, but production still requires server-side official-site scoping, SSRF/DNS-rebinding/redirect controls, hard extraction/page/byte/time/candidate budgets, durable history/cache isolation and retention/deletion, abuse/rate controls, suppression regression tests, and authenticated provenance/audit coverage. For domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability. CSV preview is capped for display and is not an import workflow.
|
||||
|
||||
+15
-2
@@ -74,7 +74,20 @@
|
||||
return `<div class="website-scan-meta"><span><b>Scan status</b><span class="website-status ${String(status).toLowerCase().replaceAll(' ','-')}">${esc(status)}</span></span><span>Checked ${esc(websiteScanValue(scan,['checked_at','last_checked_at','checked_time']))}</span><span>Cache: ${esc(websiteStateLabel(websiteScanValue(scan,['cache_state','cache_status','cache'], 'Unknown')))}</span></div>${error?`<div class="website-scan-error" role="alert"><strong>${esc(websiteStateLabel(scan.error_code||'Error'))}</strong><p>${esc(error)}</p></div>`:''}<div class="website-scan-grid"><div><dt>HTTP status</dt><dd>${esc(http)}</dd></div><div><dt>Final URL</dt><dd class="breakable">${esc(finalUrl)}</dd></div><div><dt>Redirect chain</dt><dd class="breakable">${esc(value(chain)||'None')}</dd></div><div><dt>Classification</dt><dd>${esc(classification)}</dd></div><div><dt>Title</dt><dd>${esc(websiteScanValue(scan,['title']))}</dd></div><div><dt>Meta description</dt><dd>${esc(websiteScanValue(scan,['meta_description','description']))}</dd></div><div><dt>Viewport</dt><dd>${esc(scan?.responsive_signal===true?'Responsive signal present':scan?.viewport||'Unknown')}</dd></div><div><dt>HTTPS / TLS</dt><dd>${esc(tls)}${scan?.certificate_status?` · ${esc(websiteStateLabel(scan.certificate_status))}`:''}</dd></div><div><dt>Load time</dt><dd>${esc(websiteScanValue(scan,['elapsed_ms','load_time_ms','duration_ms'],'Unknown'))}${scan?.elapsed_ms!=null?' ms':''}</dd></div><div><dt>Response size</dt><dd>${esc(websiteScanValue(scan,['response_size','size_bytes','body_size'],'Unknown'))}${scan?.size_bytes!=null?' bytes':''}</dd></div></div><div class="website-contact-signals"><h5>Contact signals</h5><div class="signal-list">${contact.map(([label,val])=>`<span class="signal ${val===true?'present':val===false?'absent':'unknown'}"><b>${esc(label)}</b>${esc(signalLabel(val))}</span>`).join('')}</div></div>`;
|
||||
}
|
||||
async function loadWebsiteScan(id, {scan=false}={}){const state=$('websiteScanState'), scanButton=$('scanWebsiteBtn'), refreshButton=$('refreshWebsiteScanBtn');if(!state||!id)return;[scanButton,refreshButton].forEach(button=>{if(button)button.disabled=true;});state.innerHTML='<div class="detail-loading" aria-live="polite">Loading website scan…</div>';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/websites/scan`,scan?{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({url:websiteUrlFor(selectedDetail||prospects.find(item=>Number(item.id)===Number(id))||{})})}:{});if(selectedDetail&&Number(selectedDetail.id)===Number(id))selectedDetail={...selectedDetail,website_scan:result};state.innerHTML=renderWebsiteScan(result);}catch(error){if(error.message!=='unauthorized')state.innerHTML=`<div class="detail-error" role="alert"><strong>${scan?'Website scan failed':'Unable to load website scan'}</strong><p>${esc(error.message)}</p><button class="button ghost compact" id="retryWebsiteScanBtn" type="button">Try again</button></div>`;}finally{[scanButton,refreshButton].forEach(button=>{if(button)button.disabled=false;});}}
|
||||
function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block website-scan-panel" id="websiteScanPanel" data-smoke="website-scan"><div class="detail-loading">Loading website scan…</div></section><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderWebsiteScanPanel(p);renderDomainPanel(p);renderDedupPanel();}
|
||||
const extractedContactItems = payload => { for (const key of ['contacts','extracted_contacts','items','results']) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; };
|
||||
const extractedContactValue = (item, keys, fallback='Unknown') => { for (const key of keys) if (item?.[key] !== undefined && item[key] !== null && item[key] !== '') return item[key]; return fallback; };
|
||||
const extractedContactStatus = item => { const status=String(extractedContactValue(item,['suppression_status','suppression','status'],'Unknown')).toLowerCase().replaceAll('_','-'); return status==='suppressed'||item?.suppressed===true ? 'Suppressed' : status==='not-suppressed'||status==='allowed'||item?.suppressed===false ? 'Not suppressed' : status.replaceAll('-',' ').replace(/\\b\\w/g,c=>c.toUpperCase()); };
|
||||
function renderExtractedContacts(payload){
|
||||
const state=$('contactExtractionState'); if(!state)return;
|
||||
if(!payload){state.innerHTML='<p class="muted">No extracted contacts returned. Extract from approved public business pages to review contact evidence.</p>';return;}
|
||||
const contacts=extractedContactItems(payload);
|
||||
if(!contacts.length){state.innerHTML='<div class="contact-extraction-empty"><strong>No public contacts found</strong><p class="muted">No contact details were found on the approved public business pages checked.</p></div>';return;}
|
||||
state.innerHTML=`<div class="extracted-contact-list">${contacts.map(item=>{const type=extractedContactValue(item,['type','contact_type','kind'],'Other'),email=extractedContactValue(item,['email','value','address']),emailClass=extractedContactValue(item,['email_class','classification','class'],'Unknown'),validation=extractedContactValue(item,['validation','validation_status','verified_status'],'Unknown'),confidence=extractedContactValue(item,['confidence','confidence_score','score']),source=extractedContactValue(item,['source_url','url','source'],'Unknown source URL'),suppression=extractedContactStatus(item);return `<article class="extracted-contact" data-contact-type="${esc(type)}"><div class="extracted-contact-head"><strong>${esc(email)}</strong><span class="contact-confidence">${esc(confidence)} confidence</span></div><dl class="extracted-contact-facts"><div><dt>Type</dt><dd>${esc(type)}</dd></div><div><dt>Email class</dt><dd>${esc(emailClass)}</dd></div><div><dt>Validation</dt><dd>${esc(validation)}</dd></div><div><dt>Suppression</dt><dd class="suppression-status">${esc(suppression)}</dd></div><div class="contact-source"><dt>Source URL</dt><dd class="breakable"><a href="${esc(source)}" target="_blank" rel="noreferrer">${esc(source)}</a></dd></div></dl></article>`;}).join('')}</div>`;
|
||||
}
|
||||
function renderContactExtractionPanel(p){const panel=$('contactExtractionPanel');if(!panel)return;panel.innerHTML=`<div class="contact-extraction-heading"><div><p class="eyebrow">PUBLIC CONTACT EXTRACTION</p><h4>Extracted contacts <span class="count">${extractedContactItems(p.extracted_contacts||p.contact_extraction).length}</span></h4></div><div class="contact-extraction-actions"><button class="button ghost compact" id="refreshContactExtractionBtn" type="button">↻ Refresh</button><button class="button primary compact" id="extractContactsBtn" type="button">Extract public contacts</button></div></div><p class="contact-extraction-safety">Extraction uses approved public business pages only. It does not probe SMTP, verify mailbox access, or send outreach.</p><div id="contactExtractionState" aria-live="polite">${p.extracted_contacts||p.contact_extraction?renderExtractedContactsMarkup(p.extracted_contacts||p.contact_extraction):'<p class="muted">No public contacts extracted yet. Start an extraction to review evidence.</p>'}</div>`;}
|
||||
function renderExtractedContactsMarkup(payload){const contacts=extractedContactItems(payload);if(!contacts.length)return '<p class="muted">No public contacts extracted yet. Start an extraction to review evidence.</p>';return '<p class="muted">Previously extracted contacts are available. Refresh to check approved public business pages again.</p>';}
|
||||
async function loadContactExtraction(id,{extract=false}={}){const state=$('contactExtractionState'),extractButton=$('extractContactsBtn'),refreshButton=$('refreshContactExtractionBtn');if(!state||!id)return;[extractButton,refreshButton].forEach(button=>{if(button)button.disabled=true;});state.innerHTML='<div class="detail-loading" aria-live="polite">Loading public contacts…</div>';try{const scan=selectedDetail?.website_scan||selectedDetail?.websiteScan||{};const payload={approved_public_pages_only:true,smtp_probing:false,outreach:false,website_scan_id:scan.id,source_url:scan.final_url||scan.input_url,html:scan.html};const result=await jsonRequest(extract?`/api/v1/businesses/${encodeURIComponent(id)}/contacts/extract`:`/api/v1/contact-extractions?business_id=${encodeURIComponent(id)}`,extract?{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(payload)}:{});if(selectedDetail&&Number(selectedDetail.id)===Number(id))selectedDetail={...selectedDetail,extracted_contacts:result.contacts||result.extracted_contacts||result.items||result};renderExtractedContacts(result);}catch(error){if(error.message!=='unauthorized')state.innerHTML=`<div class="detail-error" role="alert"><strong>${extract?'Contact extraction failed':'Unable to load extracted contacts'}</strong><p>${esc(error.message)}</p><button class="button ghost compact" id="retryContactExtractionBtn" type="button">Try again</button></div>`;}finally{[extractButton,refreshButton].forEach(button=>{if(button)button.disabled=false;});}}
|
||||
function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block contact-extraction-panel" id="contactExtractionPanel" data-smoke="contact-extraction"><div class="detail-loading">Loading public contacts…</div></section><section class="detail-block website-scan-panel" id="websiteScanPanel" data-smoke="website-scan"><div class="detail-loading">Loading website scan…</div></section><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderContactExtractionPanel(p);loadContactExtraction(p.id);renderWebsiteScanPanel(p);renderDomainPanel(p);renderDedupPanel();}
|
||||
let mergeSource = null, mergeTarget = null, mergeBusy = false;
|
||||
const payloadItems = (payload, keys) => { for (const key of keys) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; };
|
||||
const suggestionId = item => item.target_id ?? item.business_id ?? item.prospect_id ?? item.id;
|
||||
@@ -161,7 +174,7 @@
|
||||
async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}}
|
||||
async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
|
||||
document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='scanWebsiteBtn'&&selectedId)loadWebsiteScan(selectedId,{scan:true});if(e.target.id==='refreshWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='retryWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='extractContactsBtn'&&selectedId)loadContactExtraction(selectedId,{extract:true});if(e.target.id==='refreshContactExtractionBtn'&&selectedId)loadContactExtraction(selectedId,{extract:true});if(e.target.id==='retryContactExtractionBtn'&&selectedId)loadContactExtraction(selectedId);if(e.target.id==='scanWebsiteBtn'&&selectedId)loadWebsiteScan(selectedId,{scan:true});if(e.target.id==='refreshWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='retryWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
$('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
|
||||
bootstrap();
|
||||
})();
|
||||
|
||||
@@ -42,5 +42,9 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
|
||||
,['Website scan loading and error states',()=>js.includes('Loading website scan')&&js.includes('Unable to load website scan')&&js.includes('Website scan failed')&&js.includes('role="alert"')]
|
||||
,['Website scan avoids browser execution claims',()=>!js.includes('screenshot')&&!js.includes('browser execution')&&!js.includes('page screenshot')]
|
||||
,['Website scan responsive styles',()=>js.includes('website-scan-panel')&&js.includes('website-scan-grid')&&js.includes('@media')]
|
||||
,['Public contact extraction smoke marker and controls',()=>!!d.querySelector('[data-smoke="contact-extraction"]')&&!!d.querySelector('#extractContactsBtn')&&!!d.querySelector('#refreshContactExtractionBtn')&&js.includes('/contacts/extract')&&js.includes('jsonRequest')]
|
||||
,['Extracted contact evidence fields',()=>['Extracted contacts','Type','Email class','Validation','confidence','Source URL','Suppression'].every(x=>js.includes(x))]
|
||||
,['Contact extraction safety and states',()=>js.includes('approved public business pages only')&&js.includes('SMTP')&&js.includes('outreach')&&js.includes('Loading public contacts')&&js.includes('No public contacts found')&&js.includes('Contact extraction failed')]
|
||||
,['Contact extraction responsive styles',()=>js.includes('contact-extraction-panel')&&js.includes('extracted-contact-facts')&&js.includes('@media')]
|
||||
];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
|
||||
</script>
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
+12
-1
@@ -69,6 +69,16 @@ Monitor per-tenant and global scan counts, active concurrency, queue age, total/
|
||||
|
||||
Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.
|
||||
|
||||
## Phase 9 official-site contact extraction operations
|
||||
|
||||
Operate contact extraction as passive observation of an approved/public official-site origin, limited to bounded same-site contact/about pages. Before enabling it, verify tenant scope, official-site approval, extractor/policy version, retention class, suppression source, and configured hard limits for pages/URLs, redirects, bytes, time, candidates, and concurrency. Do not add arbitrary URLs or search results to the scope.
|
||||
|
||||
Review each result with its source/page URL and context, extraction method, observed time, confidence/reasons, syntax status, role/free-mail labels, and independent MX/DNS status/freshness. Treat `not_checked`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error` as uncertainty—not invalidity or non-deliverability. Syntax, role, free-mail, and MX labels never authorize contact. Values found in scripts/styles/assets/file names, examples/placeholders, tracking addresses, malformed schemes, or third-party content are false positives and must be excluded or quarantined.
|
||||
|
||||
Verify suppression matching before persistence, response, cache, export, or review-queue insertion. Suppressed contacts remain **do not contact**, regardless of later confidence, classification, syntax, MX, pipeline, or verification changes. Monitor extraction attempts, pages/bytes/candidates, limit hits, blocked destinations, parse errors, false-positive exclusions, suppression matches, cache freshness, retention/deletion jobs, and provenance/audit failures. On a suppression or provenance failure, stop the affected write path and investigate; do not retry blindly.
|
||||
|
||||
There is no SMTP probing, SMTP `VRFY`/`EXPN`, validation email, outreach worker, campaign queue, or follow-up action. Never contact a discovered address. If extraction is disabled, unapproved, out of budget, or uncertain, report deferred/blocked/unknown with the reason. Retain only the minimum value and lineage for the approved retention period; redact addresses and page content from routine logs.
|
||||
|
||||
## Phase 4 jobs and live logging
|
||||
|
||||
The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.
|
||||
@@ -90,7 +100,8 @@ Before deployment:
|
||||
3. Restrict host/network exposure at the ingress/firewall.
|
||||
4. Verify both unauthenticated health checks and review logs for unexpected errors, cross-tenant errors, or sensitive data.
|
||||
5. Exercise tenant-scoped list/detail/child routes with bounded pagination and filters, and verify that notes/pipeline changes appear in the intended tenant's audit trail only.
|
||||
6. Record the image digest and configuration revision for rollback.
|
||||
6. If Phase 9 is enabled, run official-site fixtures covering provenance/confidence, role and free-mail labels, syntax failures, every MX/DNS uncertainty state, false-positive exclusions, limit exhaustion, retention/deletion, and suppression-before-persistence/response/export. Confirm no SMTP or outreach network activity.
|
||||
7. Record the image digest and configuration revision for rollback.
|
||||
|
||||
## Data, backups, and retention
|
||||
|
||||
|
||||
@@ -51,6 +51,18 @@ No Phase 7 resolver, cache, or availability provider is enabled in the current C
|
||||
|
||||
No production-grade scanner egress proxy, isolated worker, or durable scan store is supplied by the current Compose runtime. Before enabling scanning in production, add SSRF/DNS-rebinding/redirect-chain tests, egress deny-by-default policy, abuse/rate controls, authenticated history authorization, retention/deletion, monitoring, and incident procedures. Scans must never cause form submission, acquisition, verification, enrichment, or outreach.
|
||||
|
||||
## Phase 9 public official-site contact extraction controls
|
||||
|
||||
- Extraction is passive and suggestion-only. Scope it to an explicitly approved/public official-site origin and bounded same-site contact/about pages; never use it as general search, unrestricted crawling, identity verification, or enrichment.
|
||||
- Apply the Phase 8 SSRF, redirect, content-type, timeout, byte, page, URL, candidate, and concurrency limits. Fail closed on disabled/unapproved scope, limit exhaustion, partial content, blocked/unsafe destinations, or resolver errors, with an explicit status and uncertainty reason rather than an empty success.
|
||||
- Parse only permitted public HTML/visible contact values and `mailto:` links. Never submit forms, execute JavaScript, send credentials/cookies, probe SMTP or SMTP `VRFY`/`EXPN`, send validation email, or make any outbound contact. Treat HTML, attributes, scripts, and extracted text as untrusted input.
|
||||
- Retain provenance for every candidate: source/page URL and context, extraction method, observed time, extractor/policy version, confidence algorithm/version, and uncertainty reasons. Confidence is a triage signal—not ownership, consent, deliverability, or contact permission.
|
||||
- Keep `syntax_valid`/`syntax_invalid` separate from role classification and free-mail classification. Role/person/unknown and free-mail/business-domain/unknown are labels only. MX/DNS must remain independently uncertain with resolver/source, observed time, TTL/freshness, and explicit `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error` states; no MX outcome proves deliverability.
|
||||
- Exclude asset/file-name addresses, script/style/comment text, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party content. Suppression matching must happen before persistence, response, export, cache, or review queue insertion; a match is immutable do-not-contact until an authorized suppression change, and suppression always wins.
|
||||
- Minimize and protect contact values and lineage. Bound retention for raw/extracted values, provenance, MX/DNS observations, caches, and audit records; redact full addresses and page content from logs where a safe hash/identifier suffices. Tenant predicates and deletion handling apply to every result, cache, export, and audit read.
|
||||
|
||||
No SMTP probing or outreach capability is permitted by this phase. A public address remains an unverified observation requiring human review and separate future product/legal/security approval before any contact workflow could exist.
|
||||
|
||||
## Phase 5 source security controls
|
||||
|
||||
Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
|
||||
@@ -80,6 +92,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in
|
||||
12. **Live delivery:** SSE, if introduced, must authenticate before opening the stream, enforce tenant scope on every replay query, bound event/backlog size, support `Last-Event-ID`/cursor replay, send heartbeats, and provide polling fallback. Treat event-stream connections as untrusted clients and avoid cross-tenant timing/detail leaks.
|
||||
13. **Worker boundary:** the current SQLite/in-process MVP is not durable or horizontally safe. A production worker migration requires reviewed queue semantics, leases, visibility timeouts, dead-letter handling, concurrency limits, cancellation races, metrics, and deployment isolation. Redis/Celery are not implemented today.
|
||||
14. **Domain intelligence:** implement and security-review PSL pin/update handling, bounded DNS resolution, TTL-aware cache isolation/invalidation, uncertainty-preserving MX/NS/TXT parsing, tenant-scoped association decisions, and an authorized availability provider before exposing any live domain status. No DNS response may authorize acquisition, ownership, outreach, or verification.
|
||||
15. **Official-site contacts:** production extraction requires approved official-site scoping, false-positive fixtures, syntax/role/free-mail classification tests, explicit MX uncertainty handling, pre-persistence suppression tests, hard limit/retention/deletion controls, provenance/audit coverage, and a permanent prohibition on SMTP probing, validation mail, and outreach.
|
||||
|
||||
## Source and contact policy
|
||||
|
||||
|
||||
Reference in New Issue
Block a user