add ssrf-safe website analysis
This commit is contained in:
Marco0300
+14
-4
@@ -1,6 +1,6 @@
|
||||
# Prospect Platform API — Phase 7 boundary
|
||||
# Prospect Platform API — Phase 8 boundary
|
||||
|
||||
Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
|
||||
Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
|
||||
|
||||
## Run
|
||||
|
||||
@@ -114,8 +114,18 @@ Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class
|
||||
|
||||
Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification.
|
||||
|
||||
Domain availability is `unknown` unless an explicitly authorized availability provider supplies it. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
|
||||
Domain availability is `unknown` unless the API reports a result from an authorized provider. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
|
||||
|
||||
## Phase 8 website-scanning contract
|
||||
|
||||
Website scans are tenant-scoped, authenticated, bounded observations. Accept only `http` and `https`; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Resolve and validate the destination immediately before connecting, block loopback/private/link-local/multicast/reserved/cloud-metadata ranges, and repeat the protocol/host/DNS/IP checks for every redirect. Redirects must have a small fixed maximum and cannot escape the allowed protocol policy. DNS rebinding protections must validate the address actually used for the connection.
|
||||
|
||||
Apply hard per-scan budgets for wall-clock time, connect/read timeouts, response/body bytes (including decompression), redirects, crawl depth, discovered links, and concurrency/retries. Crawl only explicitly allowed same-policy links; do not submit forms, send cookies or credentials, execute JavaScript, run plugins, or emulate a browser. Unsupported content, a budget exhaustion, timeout, DNS failure, redirect rejection, or partial response is an explicit `unknown`/`blocked`/`partial`/`error` result, never a successful empty page.
|
||||
|
||||
Classifications must be conservative, explainable, and derived only from bounded fetched content. They are observations, not proof of ownership, identity, consent, deliverability, security, or contact permission. Persist the normalized URL, redirect chain, response metadata, observed time, scanner/policy/version, applied budgets, cache status, and uncertainty/error reasons; redact response bodies and secrets unless an approved minimal excerpt is required.
|
||||
|
||||
Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
|
||||
|
||||
## Remaining limitations and production migration work
|
||||
|
||||
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Phase 7 production work must add a reviewed PSL update/version policy, bounded DNS workers, TTL-aware cache storage/invalidation, uncertainty-aware MX/NS/TXT schema and tests, association candidate safeguards and permissions, and an authorized availability provider with egress/rate/circuit/retention controls. It must also add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.
|
||||
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.
|
||||
|
||||
@@ -10,10 +10,12 @@ if __package__ in (None, ""):
|
||||
from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from app.sources import adapter_for, contains_secret
|
||||
from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
from app.website_scanner import scan_website, validate_url
|
||||
else:
|
||||
from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from .sources import adapter_for, contains_secret
|
||||
from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
from .website_scanner import scan_website, validate_url
|
||||
ORGANIZATION_ID = "demo-tenant"
|
||||
SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
|
||||
SESSION_DAYS = 7
|
||||
@@ -21,6 +23,8 @@ PBKDF2_ITERATIONS = 300_000
|
||||
MUTATING_ROLES = {"owner", "admin", "researcher"}
|
||||
JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"}
|
||||
JOB_PAGE_SIZE = 100
|
||||
WEBSITE_SCAN_PAGE_SIZE = 100
|
||||
WEBSITE_SCAN_CACHE_SECONDS = 3600
|
||||
SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
|
||||
CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
|
||||
|
||||
@@ -120,12 +124,14 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/website-scans": return self.list_website_scans(db,org,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/businesses/"):
|
||||
bits=path.split("/"); ident=bits[4] if len(bits)>4 else ""
|
||||
if not ident.isdigit(): return self.send_json(404,{"error":"not_found"})
|
||||
row=self.business(db,int(ident),org)
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
if len(bits)==7 and bits[5:]==["websites","scan"]: return self.get_latest_website_scan(int(ident),db,user)
|
||||
if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query))
|
||||
if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org)
|
||||
if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org)
|
||||
@@ -173,6 +179,48 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
rows=db.execute("SELECT * FROM domain_checks WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall()
|
||||
return self.send_json(200,{"organization_id":org,"items":[self._domain_result(r) for r in rows[:limit]],"limit":limit,"offset":offset,"has_more":len(rows)>limit})
|
||||
|
||||
def _website_scan_result(self, row, cache_hit=False):
|
||||
result = json.loads(row["result_json"] or "{}")
|
||||
result.update({"id": row["id"], "business_id": row["business_id"], "website_id": row["website_id"], "input_url": row["input_url"], "classification": row["classification"], "scanned_at": row["scanned_at"], "cache_expires_at": row["cache_expires_at"], "cache_hit": cache_hit})
|
||||
return result
|
||||
|
||||
def list_website_scans(self, db, org, query):
|
||||
try:
|
||||
limit = int((query.get("page_size") or [50])[0]); offset = max(0, int((query.get("offset") or [0])[0]))
|
||||
if limit < 1 or limit > WEBSITE_SCAN_PAGE_SIZE: raise ValueError
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
|
||||
params = [org]; where = ["organization_id=?"]
|
||||
if (query.get("business_id") or [""])[0].isdigit(): where.append("business_id=?"); params.append(int(query["business_id"][0]))
|
||||
rows = db.execute("SELECT * FROM website_scans WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
|
||||
return self.send_json(200, {"organization_id": org, "items": [self._website_scan_result(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def get_latest_website_scan(self, bid, db, user):
|
||||
row = db.execute("SELECT * FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "scan_not_found"})
|
||||
return self.send_json(200, self._website_scan_result(row, False))
|
||||
|
||||
def scan_business_website(self, bid, payload, db, user):
|
||||
org = user["organization_id"]; business = self.business(db, bid, org)
|
||||
if not business: return self.send_json(404, {"error": "not_found"})
|
||||
requested = str(payload.get("url", "")).strip() if isinstance(payload, dict) else ""
|
||||
if not requested:
|
||||
child = db.execute("SELECT * FROM websites WHERE business_id=? AND organization_id=? ORDER BY id LIMIT 1", (bid, org)).fetchone()
|
||||
requested = (child["url"] if child else business["website"]) or ""
|
||||
try: safe_url = validate_url(requested)
|
||||
except ValueError as exc:
|
||||
self.audit(db, user, "website.scan.rejected", f"{bid}:{str(exc)}"); db.commit()
|
||||
return self.send_json(400, {"error": "unsafe_url", "reason": str(exc)})
|
||||
website = db.execute("SELECT id FROM websites WHERE business_id=? AND organization_id=? AND url=? ORDER BY id LIMIT 1", (bid, org, safe_url)).fetchone()
|
||||
cache_key = hashlib.sha256(safe_url.encode()).hexdigest(); now = datetime.now(timezone.utc).replace(microsecond=0); expires = now + timedelta(seconds=WEBSITE_SCAN_CACHE_SECONDS)
|
||||
cached = db.execute("SELECT * FROM website_scans WHERE organization_id=? AND business_id=? AND cache_key=? AND cache_expires_at>? ORDER BY id DESC LIMIT 1", (org, bid, cache_key, now.isoformat())).fetchone()
|
||||
if cached:
|
||||
self.audit(db, user, "website.scan.cache_hit", f"{bid}:{safe_url}"); db.commit()
|
||||
return self.send_json(200, self._website_scan_result(cached, True))
|
||||
result = scan_website(safe_url); result["business_id"] = bid
|
||||
cur = db.execute("INSERT INTO website_scans(organization_id,business_id,website_id,input_url,classification,result_json,cache_key,scanned_at,cache_expires_at) VALUES(?,?,?,?,?,?,?,?,?)", (org, bid, website["id"] if website else None, safe_url, result["classification"], json.dumps(result, sort_keys=True), cache_key, now.isoformat(), expires.isoformat()))
|
||||
self.audit(db, user, "website.scanned", f"{bid}:{result['classification']}"); db.commit()
|
||||
return self.send_json(201, self._website_scan_result(db.execute("SELECT * FROM website_scans WHERE id=?", (cur.lastrowid,)).fetchone()))
|
||||
|
||||
def list_domain_candidates(self,bid,db,org):
|
||||
business=self.business(db,bid,org)
|
||||
if not business:return self.send_json(404,{"error":"not_found"})
|
||||
@@ -286,6 +334,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
|
||||
if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
|
||||
if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"websites"] and path.split("/")[6]=="scan": return self.scan_business_website(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"):
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
"""Bounded, passive and SSRF-safe website analysis using only the stdlib."""
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import ipaddress
|
||||
import re
|
||||
import socket
|
||||
import ssl
|
||||
import time
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import urljoin, urlparse
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import HTTPRedirectHandler, Request, build_opener
|
||||
|
||||
MAX_BYTES = 512 * 1024
|
||||
MAX_REDIRECTS = 5
|
||||
MAX_PAGES = 1
|
||||
DEFAULT_TIMEOUT = 5.0
|
||||
_METADATA_IPS = {ipaddress.ip_address("169.254.169.254"), ipaddress.ip_address("100.100.100.200")}
|
||||
|
||||
|
||||
def _resolved_addresses(host: str, timeout: float) -> list[str]:
|
||||
try:
|
||||
socket.setdefaulttimeout(timeout)
|
||||
records = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM)
|
||||
except (OSError, socket.gaierror) as exc:
|
||||
raise ValueError("dns_failure") from exc
|
||||
addresses = sorted({str(r[4][0]) for r in records if len(r) > 4})
|
||||
if not addresses:
|
||||
raise ValueError("dns_failure")
|
||||
for value in addresses:
|
||||
try:
|
||||
ip = ipaddress.ip_address(value)
|
||||
except ValueError as exc:
|
||||
raise ValueError("unsafe_address") from exc
|
||||
if ip in _METADATA_IPS or not ip.is_global or ip.is_loopback or ip.is_private or ip.is_link_local or ip.is_multicast or ip.is_reserved or ip.is_unspecified:
|
||||
raise ValueError("unsafe_address")
|
||||
return addresses
|
||||
|
||||
|
||||
def validate_url(value: str, *, timeout: float = DEFAULT_TIMEOUT) -> str:
|
||||
raw = str(value or "").strip()
|
||||
parsed = urlparse(raw)
|
||||
if parsed.scheme.lower() not in {"http", "https"} or not parsed.hostname or parsed.username or parsed.password:
|
||||
raise ValueError("invalid_url")
|
||||
if parsed.fragment:
|
||||
raw = raw.split("#", 1)[0]
|
||||
parsed = urlparse(raw)
|
||||
host = (parsed.hostname or "").rstrip(".").lower()
|
||||
if len(raw) > 2048 or len(host) > 253:
|
||||
raise ValueError("invalid_url")
|
||||
try:
|
||||
ipaddress.ip_address(host)
|
||||
_resolved_addresses(host, timeout)
|
||||
except ValueError:
|
||||
_resolved_addresses(host, timeout)
|
||||
return parsed.geturl()
|
||||
|
||||
|
||||
class _Redirects(HTTPRedirectHandler):
|
||||
def __init__(self, timeout: float, max_redirects: int):
|
||||
self.timeout = timeout
|
||||
self.max_redirects = max_redirects
|
||||
self.chain: list[str] = []
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
if len(self.chain) >= self.max_redirects:
|
||||
raise ValueError("redirect_limit")
|
||||
target = validate_url(urljoin(req.full_url, newurl), timeout=self.timeout)
|
||||
self.chain.append(target)
|
||||
return Request(target, headers={"User-Agent": "ProspectPlatformWebsiteScanner/1.0", "Accept": "text/html,application/xhtml+xml"}, method="GET")
|
||||
|
||||
|
||||
def _fetch(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int = MAX_BYTES, max_redirects: int = MAX_REDIRECTS) -> dict:
|
||||
safe_url = validate_url(url, timeout=timeout)
|
||||
redirects = _Redirects(timeout, max_redirects)
|
||||
opener = build_opener(redirects)
|
||||
request = Request(safe_url, headers={"User-Agent": "ProspectPlatformWebsiteScanner/1.0", "Accept": "text/html,application/xhtml+xml"}, method="GET")
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with opener.open(request, timeout=timeout) as response:
|
||||
chunks, total = [], 0
|
||||
while True:
|
||||
chunk = response.read(min(65536, max_bytes - total + 1))
|
||||
if not chunk:
|
||||
break
|
||||
total += len(chunk)
|
||||
if total > max_bytes:
|
||||
raise ValueError("response_too_large")
|
||||
chunks.append(chunk)
|
||||
final_url = validate_url(response.geturl(), timeout=timeout)
|
||||
return {"status": int(response.status), "final_url": final_url, "redirect_chain": redirects.chain, "body": b"".join(chunks), "content_type": response.headers.get_content_type(), "elapsed_ms": round((time.monotonic() - started) * 1000, 2), "tls": final_url.startswith("https://"), "certificate_status": "valid" if final_url.startswith("https://") else "not_applicable"}
|
||||
except HTTPError as exc:
|
||||
# HTTP errors are still useful website observations; read only the bounded body.
|
||||
body = exc.read(max_bytes + 1)
|
||||
if len(body) > max_bytes: raise ValueError("response_too_large") from exc
|
||||
return {"status": int(exc.code), "final_url": validate_url(exc.geturl(), timeout=timeout), "redirect_chain": redirects.chain, "body": body, "content_type": exc.headers.get_content_type() if exc.headers else "text/html", "elapsed_ms": round((time.monotonic() - started) * 1000, 2), "tls": str(exc.geturl()).startswith("https://"), "certificate_status": "valid" if str(exc.geturl()).startswith("https://") else "not_applicable"}
|
||||
except ssl.SSLCertVerificationError as exc:
|
||||
raise ValueError("certificate_invalid") from exc
|
||||
except ValueError:
|
||||
raise
|
||||
except TimeoutError as exc:
|
||||
raise ValueError("timeout") from exc
|
||||
except OSError as exc:
|
||||
raise ValueError("connection_failed") from exc
|
||||
|
||||
|
||||
class _PageParser(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.title = ""; self.meta_description = ""; self.language = ""; self.headings: list[str] = []
|
||||
self.viewport = False; self.cms_hints: set[str] = set(); self.contact_page = False; self.form = False
|
||||
self.mail = False; self.phone = False; self.whatsapp = False; self.social = False; self._tag = ""; self._buf: list[str] = []
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = {str(k).lower(): str(v or "") for k, v in attrs}; tag = tag.lower()
|
||||
self._tag = tag
|
||||
if tag == "html": self.language = attrs.get("lang", "")[:20]
|
||||
if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}: self._buf = []
|
||||
if tag == "title": self._buf = []
|
||||
if tag == "meta":
|
||||
name = attrs.get("name", "").lower()
|
||||
if name == "description": self.meta_description = attrs.get("content", "")[:1000]
|
||||
if name == "viewport": self.viewport = True
|
||||
generator = attrs.get("content", "").lower()
|
||||
if name == "generator": self._cms(generator)
|
||||
if tag == "form": self.form = True
|
||||
if tag == "a":
|
||||
href = attrs.get("href", "").lower()
|
||||
self.contact_page |= any(x in href for x in ("contact", "get-in-touch", "reach-us"))
|
||||
self.mail |= href.startswith("mailto:"); self.whatsapp |= "wa.me" in href or "whatsapp" in href
|
||||
self.social |= any(x in href for x in ("facebook.com", "instagram.com", "linkedin.com", "twitter.com", "x.com", "youtube.com", "tiktok.com"))
|
||||
if tag in {"script", "link"}:
|
||||
text = " ".join(attrs.values()).lower(); self._cms(text)
|
||||
|
||||
def _cms(self, text):
|
||||
for key, terms in {"wordpress": ("wordpress", "wp-content"), "drupal": ("drupal",), "joomla": ("joomla",), "shopify": ("shopify",), "wix": ("wix.com",)}.items():
|
||||
if any(term in text for term in terms): self.cms_hints.add(key)
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._tag in {"title", "h1", "h2", "h3", "h4", "h5", "h6"}: self._buf.append(data)
|
||||
if re.search(r"(?:tel:|\+?\d[\d ()-]{6,})", data): self.phone = True
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
tag = tag.lower()
|
||||
if tag == "title" and self._buf: self.title = " ".join("".join(self._buf).split())[:500]
|
||||
if tag in {"h1", "h2", "h3", "h4", "h5", "h6"} and self._buf:
|
||||
self.headings.append(" ".join("".join(self._buf).split())[:300])
|
||||
self._tag = ""
|
||||
|
||||
|
||||
def classify_website(status, final_url, body, *, error=None) -> str:
|
||||
if error:
|
||||
return "blocked" if error in {"timeout", "connection_failed", "dns_failure", "unsafe_address", "certificate_invalid", "redirect_limit", "response_too_large"} else "unknown"
|
||||
if status is None: return "unknown"
|
||||
if 400 <= status or status < 200: return "broken"
|
||||
text = re.sub(r"<[^>]+>", " ", body if isinstance(body, str) else body.decode("utf-8", "replace")).lower()
|
||||
if status in {301, 302, 303, 307, 308} and not text.strip(): return "redirect_only"
|
||||
if re.search(r"domain (is )?for sale|buy this domain|parking page|parked free", text): return "parked"
|
||||
if re.search(r"under construction|coming soon|website coming", text): return "under_construction"
|
||||
if re.search(r"placeholder|lorem ipsum|sample page|default web page", text): return "placeholder"
|
||||
if status < 300 and len(re.sub(r"\s+", "", text)) >= 8: return "healthy"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def scan_website(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int = MAX_BYTES, max_redirects: int = MAX_REDIRECTS, max_pages: int = MAX_PAGES) -> dict:
|
||||
result = {"input_url": str(url), "status": None, "final_url": None, "redirect_chain": [], "title": "", "meta_description": "", "language": "", "headings": [], "responsive_signal": None, "cms_hints": [], "contact_page_signal": None, "form_signal": None, "mail_signal": None, "phone_signal": None, "whatsapp_signal": None, "social_signal": None, "elapsed_ms": None, "size_bytes": 0, "tls": None, "certificate_status": "unknown", "error_code": None}
|
||||
try:
|
||||
if not 0 < int(max_redirects) <= MAX_REDIRECTS or not 0 < int(max_pages) <= MAX_PAGES: raise ValueError("invalid_limits")
|
||||
fetched = _fetch(url, timeout=max(0.1, min(float(timeout), 10.0)), max_bytes=max(1, min(int(max_bytes), MAX_BYTES)), max_redirects=int(max_redirects))
|
||||
result.update({k: fetched[k] for k in ("status", "final_url", "redirect_chain", "elapsed_ms", "tls", "certificate_status")}); result["size_bytes"] = len(fetched["body"])
|
||||
if fetched["content_type"] not in {"text/html", "application/xhtml+xml"}:
|
||||
result["classification"] = classify_website(fetched["status"], fetched["final_url"], ""); return result
|
||||
parser = _PageParser(); parser.feed(fetched["body"].decode("utf-8", "replace"))
|
||||
for key in ("title", "meta_description", "language", "headings", "viewport", "cms_hints", "contact_page", "form", "mail", "phone", "whatsapp", "social"):
|
||||
result[{"viewport":"responsive_signal","cms_hints":"cms_hints","contact_page":"contact_page_signal","form":"form_signal","mail":"mail_signal","phone":"phone_signal","whatsapp":"whatsapp_signal","social":"social_signal"}.get(key,key)] = sorted(parser.cms_hints) if key == "cms_hints" else getattr(parser, key)
|
||||
result["classification"] = classify_website(result["status"], result["final_url"], fetched["body"])
|
||||
except ValueError as exc:
|
||||
result["error_code"] = str(exc); result["classification"] = classify_website(None, url, "", error=str(exc))
|
||||
return result
|
||||
@@ -170,3 +170,21 @@ CREATE TABLE IF NOT EXISTS domain_candidates (
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,business_id,domain)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_domain_candidates_business ON domain_candidates(organization_id,business_id,rank,id);
|
||||
|
||||
-- Phase 8 passive website analysis history; each scan is retained.
|
||||
CREATE TABLE IF NOT EXISTS website_scans (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id),
|
||||
business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
|
||||
website_id INTEGER REFERENCES websites(id) ON DELETE SET NULL,
|
||||
input_url TEXT NOT NULL,
|
||||
classification TEXT NOT NULL,
|
||||
result_json TEXT NOT NULL DEFAULT '{}',
|
||||
cache_key TEXT NOT NULL,
|
||||
scanned_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
cache_expires_at TEXT,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_org ON website_scans(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_business ON website_scans(organization_id,business_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_cache ON website_scans(organization_id,cache_key,cache_expires_at);
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.main import create_server
|
||||
|
||||
|
||||
class WebsiteScanApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory()
|
||||
os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'scan-owner@example.test'
|
||||
os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password'
|
||||
self.server = create_server('127.0.0.1', 0, self.tmp.name + '/db.sqlite')
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
|
||||
self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None
|
||||
self.request('POST', '/api/v1/auth/login', {'email': 'scan-owner@example.test', 'password': 'password'})
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
|
||||
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None; headers = {'Content-Type': 'application/json'} if body else {}
|
||||
if self.cookie: headers['Cookie'] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse(); cookie = response.getheader('Set-Cookie')
|
||||
if cookie: self.cookie = cookie.split(';', 1)[0]
|
||||
return response.status, json.loads(response.read() or b'{}')
|
||||
|
||||
def test_scan_is_cached_history_is_listed_and_audited(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Scan Co', 'website': 'https://scan.example'})
|
||||
result = {'classification': 'healthy', 'status': 200, 'final_url': 'https://scan.example/', 'redirect_chain': []}
|
||||
with patch('app.main.validate_url', return_value='https://scan.example/'), patch('app.main.scan_website', return_value=result) as scanner:
|
||||
first_status, first = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
|
||||
second_status, second = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
|
||||
self.assertEqual(first_status, 201); self.assertEqual(second_status, 200); self.assertTrue(second['cache_hit']); scanner.assert_called_once()
|
||||
status, history = self.request('GET', '/api/v1/website-scans?page_size=1')
|
||||
self.assertEqual(status, 200); self.assertEqual(len(history['items']), 1); self.assertFalse(history['has_more'])
|
||||
self.assertEqual(history['items'][0]['classification'], 'healthy')
|
||||
self.assertEqual(self.request('GET', '/api/v1/website-scans?business_id=999999')[1]['items'], [])
|
||||
|
||||
def test_get_latest_scan_returns_scan_payload(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Read Scan Co', 'website': 'https://read.example'})
|
||||
result = {'classification': 'unknown', 'status': 200, 'final_url': 'https://read.example/', 'redirect_chain': []}
|
||||
with patch('app.main.validate_url', return_value='https://read.example/'), patch('app.main.scan_website', return_value=result):
|
||||
self.assertEqual(self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})[0], 201)
|
||||
status, payload = self.request('GET', f"/api/v1/businesses/{business['id']}/websites/scan")
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(payload['classification'], 'unknown')
|
||||
self.assertEqual(payload['business_id'], business['id'])
|
||||
|
||||
def test_unsafe_scan_is_rejected_without_fetching(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Private Scan', 'website': 'http://127.0.0.1/'})
|
||||
with patch('app.main.scan_website') as scanner:
|
||||
status, payload = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
|
||||
self.assertEqual(status, 400); self.assertEqual(payload['error'], 'unsafe_url'); scanner.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,59 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.website_scanner import classify_website, validate_url, scan_website
|
||||
|
||||
|
||||
class WebsiteScannerTests(unittest.TestCase):
|
||||
def test_rejects_unsafe_schemes_and_addresses(self):
|
||||
for url in ('file:///etc/passwd', 'ftp://example.com', 'http://127.0.0.1/', 'http://169.254.169.254/latest/meta-data'):
|
||||
with self.subTest(url=url):
|
||||
with self.assertRaises(ValueError):
|
||||
validate_url(url)
|
||||
|
||||
def test_classification_is_conservative(self):
|
||||
self.assertEqual(classify_website(200, 'https://example.com', '<html><title>Acme</title><h1>Welcome</h1></html>'), 'healthy')
|
||||
self.assertEqual(classify_website(404, 'https://example.com', ''), 'broken')
|
||||
self.assertEqual(classify_website(200, 'https://example.com', '<html>under construction - coming soon</html>'), 'under_construction')
|
||||
self.assertEqual(classify_website(200, 'https://example.com', '<html>domain for sale parking</html>'), 'parked')
|
||||
self.assertEqual(classify_website(200, 'https://example.com', '<html>placeholder page</html>'), 'placeholder')
|
||||
self.assertEqual(classify_website(301, 'https://example.com', ''), 'redirect_only')
|
||||
self.assertEqual(classify_website(None, 'https://example.com', '', error='timeout'), 'blocked')
|
||||
self.assertEqual(classify_website(None, 'https://example.com', ''), 'unknown')
|
||||
|
||||
def test_metadata_and_signals_are_extracted_from_fixture(self):
|
||||
html = '''<!doctype html><html lang="en"><head><title>Acme</title>
|
||||
<meta name="description" content="Solar experts"><meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="generator" content="WordPress 6"><link rel="stylesheet" href="/style.css"></head>
|
||||
<body><h1>Acme Solar</h1><h2>Contact us</h2><a href="/contact">Contact</a><a href="mailto:hi@acme.test">Email</a>
|
||||
<a href="https://wa.me/27123456789">WhatsApp</a><a href="https://facebook.com/acme">Facebook</a><form action="/contact"></form></body></html>'''
|
||||
with patch('app.website_scanner._fetch', return_value={
|
||||
'status': 200, 'final_url': 'https://acme.test/', 'redirect_chain': [],
|
||||
'body': html.encode(), 'content_type': 'text/html', 'elapsed_ms': 12, 'tls': True, 'certificate_status': 'valid'
|
||||
}):
|
||||
result = scan_website('https://acme.test/')
|
||||
self.assertEqual(result['classification'], 'healthy')
|
||||
self.assertEqual(result['title'], 'Acme')
|
||||
self.assertEqual(result['meta_description'], 'Solar experts')
|
||||
self.assertEqual(result['language'], 'en')
|
||||
self.assertEqual(result['headings'], ['Acme Solar', 'Contact us'])
|
||||
self.assertTrue(result['responsive_signal'])
|
||||
self.assertIn('wordpress', result['cms_hints'])
|
||||
self.assertTrue(result['contact_page_signal'])
|
||||
self.assertTrue(result['form_signal'])
|
||||
self.assertTrue(result['mail_signal'])
|
||||
self.assertTrue(result['whatsapp_signal'])
|
||||
self.assertTrue(result['social_signal'])
|
||||
self.assertEqual(result['certificate_status'], 'valid')
|
||||
|
||||
def test_limits_are_reported_not_as_missing_contact_data(self):
|
||||
with patch('app.website_scanner._fetch', side_effect=ValueError('response_too_large')):
|
||||
result = scan_website('https://example.com')
|
||||
self.assertEqual(result['classification'], 'blocked')
|
||||
self.assertEqual(result['error_code'], 'response_too_large')
|
||||
self.assertIsNone(result['contact_page_signal'])
|
||||
self.assertIsNone(result['mail_signal'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
+9
-3
@@ -1,6 +1,6 @@
|
||||
# ProspectOS web — Phase 7 boundary
|
||||
# ProspectOS web — Phase 8 boundary
|
||||
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts, Phase 6 normalization/deduplication, and Phase 7 domain-intelligence concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach.
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow, Phase 4 MVP job monitor, and Phase 8 scan-result/history presentation when supplied by the API. The browser does not fetch targets, submit forms, execute scan JavaScript, or send outreach; SSRF controls and budgets are server-side.
|
||||
|
||||
## Configure and run
|
||||
|
||||
@@ -32,6 +32,12 @@ Suggestions are review aids only. A merge flow must identify the surviving recor
|
||||
|
||||
The current static MVP requests `/matches`, renders a **Human review required** list with confidence/reasons, asks for **Confirm merge**, and displays merge history with **Reverse merge** when the API marks it reversible. The API remains authoritative; these controls are not a substitute for server-side authorization. Existing normalization and match display remain suggestion-only; no merge happens without explicit operator confirmation.
|
||||
|
||||
## Phase 8 website-scanning UI contract
|
||||
|
||||
The UI may request a tenant-scoped scan through an authenticated API route when enabled and render the returned classification, status, redirect chain, observed time, scanner/policy version, cache freshness, applied budgets, and uncertainty/error reasons. The server permits only `http` and `https` targets. It must label cached data as cached/stale rather than “live,” keep `unknown`, `blocked`, `partial`, `timeout`, and `error` distinct from positive observations, and never turn a conservative classification into identity, ownership, consent, deliverability, or outreach permission.
|
||||
|
||||
The browser must not directly fetch arbitrary target URLs, follow redirects for scanning, submit forms, send cookies/credentials, execute target JavaScript, or expose response bodies unnecessarily. Scan history and cache controls are tenant-scoped API capabilities, not hidden client state. A result that is budget-limited or incomplete must remain visibly incomplete; no UI timer may imply that a scan completed.
|
||||
|
||||
## Phase 5 source UI contract
|
||||
|
||||
The web client may display registered source metadata, query mode, approval/terms state, rate-limit status, retention class, health, and circuit state returned by the API. It must label `dry_run` as a plan/validation result and distinguish operator-supplied CSV/manual references from independently verified evidence. It must not offer a live-source control unless the API reports explicit approval and operational enablement; client visibility is never an authorization control.
|
||||
@@ -62,4 +68,4 @@ A zero-dependency static smoke page (`smoke-test.html`) checks the key DOM contr
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
The static client has no background discovery, DNS/website scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. For future domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability; any availability label requires an API result from an authorized provider and explicit human review. It cannot make missing provenance authoritative and should display API-provided limitations rather than infer them. CSV preview is capped for display and is not an import workflow. Production deployment still requires durable job execution, tenant-scoped controls, idempotency verification, PSL/DNS/cache implementation and tests, candidate review permissions/audit, authorized availability-provider controls, SSE/polling verification, and the security and operations gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`.
|
||||
The static client has no client-side crawler, scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. It can display server-provided Phase 8 observations, but production still requires server-side SSRF/DNS-rebinding/redirect controls, hard size/time/crawl budgets, durable scan history/cache isolation and retention, abuse/rate controls, and authenticated permission/audit coverage. For domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability. CSV preview is capped for display and is not an import workflow.
|
||||
|
||||
+20
-2
@@ -56,7 +56,25 @@
|
||||
}
|
||||
async function runDomainCheck(){const button=$('runDomainCheckBtn');if(!selectedId||!button)return;button.disabled=true;const state=$('domainCheckState');if(state)state.innerHTML='<div class="detail-loading" aria-live="polite">Checking DNS…</div>';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domains/check`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain:domainForProspect(selectedDetail||{})})});renderDomainCheck(result);}catch(error){if(error.message!=='unauthorized'&&state)state.innerHTML=`<div class="detail-error" role="alert"><strong>DNS check failed</strong><p>${esc(error.message)}</p></div>`;}finally{button.disabled=false;}}
|
||||
async function checkDomainAvailability(domain,button){const result=button.closest('.candidate-card')?.querySelector('.availability-result');if(!domain||!result)return;button.disabled=true;result.textContent='Checking availability…';try{const body=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domain-candidates/check-availability`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain})});const state=body?.status||body?.state||body?.availability||'unknown';result.textContent=['not-configured','unknown','error'].includes(String(state).toLowerCase())?`Availability: ${domainState(state)}. No ownership conclusion.`:'Availability: Unknown. No ownership conclusion.';}catch(error){if(error.message!=='unauthorized')result.textContent=`Availability: Unknown. ${error.message}`;}finally{button.disabled=false;}}
|
||||
function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderDomainPanel(p);renderDedupPanel();}
|
||||
const websiteScanValue = (scan, keys, fallback='Unknown') => { for (const key of keys) if (scan?.[key] !== undefined && scan[key] !== null && scan[key] !== '') return scan[key]; return fallback; };
|
||||
const websiteStateLabel = value => { const state=String(value ?? 'unknown').toLowerCase().replaceAll('_','-'); return ({queued:'Queued',running:'Running',succeeded:'Complete',completed:'Complete',healthy:'Healthy',broken:'Broken','under-construction':'Under construction',parked:'Parked',placeholder:'Placeholder','redirect-only':'Redirect only',blocked:'Blocked',unknown:'Unknown',error:'Error'})[state] || state.replaceAll('-',' ').replace(/\b\w/g,c=>c.toUpperCase()); };
|
||||
const signalLabel = (value, positive='Present') => value === true ? positive : value === false ? 'Not found' : websiteStateLabel(value);
|
||||
const websiteScanFor = p => p?.website_scan || p?.websiteScan || p?.scan || null;
|
||||
const websiteUrlFor = p => p?.website || p?.website_url || p?.url || (domainForProspect(p) ? `https://${domainForProspect(p)}` : '');
|
||||
function renderWebsiteScanPanel(p){
|
||||
const panel=$('websiteScanPanel'); if(!panel)return;
|
||||
panel.innerHTML=`<div class="website-scan-heading"><div><p class="eyebrow">WEBSITE ANALYSIS</p><h4>Website scan</h4></div><div class="website-scan-actions"><button class="button ghost compact" id="refreshWebsiteScanBtn" type="button">↻ Refresh</button><button class="button primary compact" id="scanWebsiteBtn" type="button">Scan website</button></div></div><p class="website-scan-safety">Website analysis is authenticated workspace evidence from the HTTP scanner. It does not enable outreach.</p><div id="websiteScanState" aria-live="polite">${websiteScanFor(p)?renderWebsiteScan(websiteScanFor(p)):'<p class="muted">No website scan has been returned. Start a scan to see HTTP and contact signals.</p>'}</div>`;
|
||||
loadWebsiteScan(p.id);
|
||||
}
|
||||
function renderWebsiteScan(scan){
|
||||
const status=websiteStateLabel(websiteScanValue(scan,['scan_status','status','state'])), classification=websiteStateLabel(websiteScanValue(scan,['classification','class'],'unknown')), http=websiteScanValue(scan,['http_status','status_code','status'], 'Unknown'), finalUrl=websiteScanValue(scan,['final_url','url'],'Unknown'), chain=websiteScanValue(scan,['redirect_chain','redirects'],[]), tls=scan?.tls===true||scan?.https===true||scan?.certificate_status==='valid'?'Secure':scan?.tls===false||scan?.https===false?'Not secure':websiteStateLabel(scan?.certificate_status||'Unknown');
|
||||
const contact=[['Contact page',scan?.contact_page_signal],['Email',scan?.mail_signal],['Phone',scan?.phone_signal],['WhatsApp',scan?.whatsapp_signal],['Social',scan?.social_signal],['Form',scan?.form_signal]];
|
||||
const value=v=>Array.isArray(v)?(v.length?v.join(' → '):'None'):v;
|
||||
const error=scan?.error||scan?.error_code||scan?.message;
|
||||
return `<div class="website-scan-meta"><span><b>Scan status</b><span class="website-status ${String(status).toLowerCase().replaceAll(' ','-')}">${esc(status)}</span></span><span>Checked ${esc(websiteScanValue(scan,['checked_at','last_checked_at','checked_time']))}</span><span>Cache: ${esc(websiteStateLabel(websiteScanValue(scan,['cache_state','cache_status','cache'], 'Unknown')))}</span></div>${error?`<div class="website-scan-error" role="alert"><strong>${esc(websiteStateLabel(scan.error_code||'Error'))}</strong><p>${esc(error)}</p></div>`:''}<div class="website-scan-grid"><div><dt>HTTP status</dt><dd>${esc(http)}</dd></div><div><dt>Final URL</dt><dd class="breakable">${esc(finalUrl)}</dd></div><div><dt>Redirect chain</dt><dd class="breakable">${esc(value(chain)||'None')}</dd></div><div><dt>Classification</dt><dd>${esc(classification)}</dd></div><div><dt>Title</dt><dd>${esc(websiteScanValue(scan,['title']))}</dd></div><div><dt>Meta description</dt><dd>${esc(websiteScanValue(scan,['meta_description','description']))}</dd></div><div><dt>Viewport</dt><dd>${esc(scan?.responsive_signal===true?'Responsive signal present':scan?.viewport||'Unknown')}</dd></div><div><dt>HTTPS / TLS</dt><dd>${esc(tls)}${scan?.certificate_status?` · ${esc(websiteStateLabel(scan.certificate_status))}`:''}</dd></div><div><dt>Load time</dt><dd>${esc(websiteScanValue(scan,['elapsed_ms','load_time_ms','duration_ms'],'Unknown'))}${scan?.elapsed_ms!=null?' ms':''}</dd></div><div><dt>Response size</dt><dd>${esc(websiteScanValue(scan,['response_size','size_bytes','body_size'],'Unknown'))}${scan?.size_bytes!=null?' bytes':''}</dd></div></div><div class="website-contact-signals"><h5>Contact signals</h5><div class="signal-list">${contact.map(([label,val])=>`<span class="signal ${val===true?'present':val===false?'absent':'unknown'}"><b>${esc(label)}</b>${esc(signalLabel(val))}</span>`).join('')}</div></div>`;
|
||||
}
|
||||
async function loadWebsiteScan(id, {scan=false}={}){const state=$('websiteScanState'), scanButton=$('scanWebsiteBtn'), refreshButton=$('refreshWebsiteScanBtn');if(!state||!id)return;[scanButton,refreshButton].forEach(button=>{if(button)button.disabled=true;});state.innerHTML='<div class="detail-loading" aria-live="polite">Loading website scan…</div>';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/websites/scan`,scan?{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({url:websiteUrlFor(selectedDetail||prospects.find(item=>Number(item.id)===Number(id))||{})})}:{});if(selectedDetail&&Number(selectedDetail.id)===Number(id))selectedDetail={...selectedDetail,website_scan:result};state.innerHTML=renderWebsiteScan(result);}catch(error){if(error.message!=='unauthorized')state.innerHTML=`<div class="detail-error" role="alert"><strong>${scan?'Website scan failed':'Unable to load website scan'}</strong><p>${esc(error.message)}</p><button class="button ghost compact" id="retryWebsiteScanBtn" type="button">Try again</button></div>`;}finally{[scanButton,refreshButton].forEach(button=>{if(button)button.disabled=false;});}}
|
||||
function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block website-scan-panel" id="websiteScanPanel" data-smoke="website-scan"><div class="detail-loading">Loading website scan…</div></section><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderWebsiteScanPanel(p);renderDomainPanel(p);renderDedupPanel();}
|
||||
let mergeSource = null, mergeTarget = null, mergeBusy = false;
|
||||
const payloadItems = (payload, keys) => { for (const key of keys) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; };
|
||||
const suggestionId = item => item.target_id ?? item.business_id ?? item.prospect_id ?? item.id;
|
||||
@@ -143,7 +161,7 @@
|
||||
async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}}
|
||||
async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
|
||||
document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='scanWebsiteBtn'&&selectedId)loadWebsiteScan(selectedId,{scan:true});if(e.target.id==='refreshWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='retryWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
|
||||
$('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
|
||||
bootstrap();
|
||||
})();
|
||||
|
||||
@@ -37,5 +37,10 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
|
||||
,['DNS capability and uncertainty labels',()=>['DNS checks','A','AAAA','MX','NS','TXT','NXDOMAIN','Unknown','Error','Cache'].every(x=>js.includes(x))]
|
||||
,['Candidate confidence, reasons, and safe availability',()=>js.includes('confidence')&&js.includes('reasons')&&js.includes('check-availability')&&js.includes('Not configured')&&js.includes('No ownership conclusion')&&!js.includes('Domain is available')]
|
||||
,['Domain loading, error, tenant, and auth states',()=>js.includes('Loading DNS results')&&js.includes('Unable to load DNS results')&&js.includes('Tenant/workspace access denied')&&js.includes("credentials:'include'")]
|
||||
,['Website scan smoke marker and authenticated controls',()=>!!d.querySelector('[data-smoke="website-scan"]')&&!!d.querySelector('#scanWebsiteBtn')&&!!d.querySelector('#refreshWebsiteScanBtn')&&js.includes('/websites/scan')&&js.includes("method:'POST'")&&js.includes("credentials:'include'")]
|
||||
,['Website scan fields and conservative states',()=>['Scan status','HTTP status','Final URL','Redirect chain','Classification','Title','Meta description','Viewport','HTTPS / TLS','Load time','Response size','Contact signals','Checked','Cache','Blocked','Unknown','Error'].every(x=>js.includes(x))]
|
||||
,['Website scan loading and error states',()=>js.includes('Loading website scan')&&js.includes('Unable to load website scan')&&js.includes('Website scan failed')&&js.includes('role="alert"')]
|
||||
,['Website scan avoids browser execution claims',()=>!js.includes('screenshot')&&!js.includes('browser execution')&&!js.includes('page screenshot')]
|
||||
,['Website scan responsive styles',()=>js.includes('website-scan-panel')&&js.includes('website-scan-grid')&&js.includes('@media')]
|
||||
];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
|
||||
</script>
|
||||
|
||||
+2
-1
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user