add ssrf-safe website analysis

README.md

@@ -1,6 +1,6 @@
 # Prospect Intelligence Platform
 
-A safety-first Phase 7 design/implementation boundary for **manual**, evidence-led prospect qualification, controlled source ingestion, and domain intelligence review. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 7 defines conservative registrable-domain/PSL and DNS observation semantics in addition to the Phase 6 normalization and deduplication rules; it does **not** enable network discovery. **Automated outreach is disabled, and no live source may be enabled without explicit approval.**
+A safety-first Phase 8 design/implementation boundary for **manual**, evidence-led prospect qualification, controlled source ingestion, bounded website scanning, and domain intelligence review. Phase 8 website scanning is a conservative observation workflow: it never submits forms, executes JavaScript, follows unsafe protocols, or authorizes outreach. **Automated outreach is disabled, and no live source may be enabled without explicit approval.**
 
 ## Included
 
@@ -80,7 +80,7 @@ Authenticated browser requests use a server-side session cookie; login creates a
 
 ## Phase 5 source boundary and remaining limitations
 
-Phase 5 defines a source adapter contract and registry; it does not implement network discovery, DNS resolution, website/HTTP scanning, enrichment scheduling, or a live external-source adapter. A source adapter must declare its identity, terms owner, permitted purpose, rate limits, retention class, query/result schema, dry-run behavior, and health/circuit controls. CSV and manual reference adapters may be used for operator-supplied data; they must preserve source attribution and raw source records, and must not silently turn preview data into outreach or verified facts.
+Phase 5 defines a source adapter contract and registry; Phase 8 adds a bounded website-observation adapter, but it does not implement general network discovery, enrichment scheduling, or a live external-source adapter. A source adapter must declare its identity, terms owner, permitted purpose, rate limits, retention class, query/result schema, dry-run behavior, and health/circuit controls. CSV and manual reference adapters may be used for operator-supplied data; they must preserve source attribution and raw source records, and must not silently turn preview data into outreach or verified facts.
 
 A discovery query is a tenant-scoped, bounded, auditable request that can be validated and dry-run without contacting a source. Any live source requires explicit product/legal/security approval, a registered adapter, and an operational enablement decision; absent all three, execution must fail closed. Circuit-open, rate-limit, terms, or approval failures must produce a safe non-live result. Raw source records are retained only under the approved retention class and must exclude secrets and unnecessary personal data.
 
@@ -108,6 +108,18 @@ The platform must not claim that a domain is available, unregistered, or safe to
 
 Phase 7 remains a documentation/contract boundary in this MVP: there is no live DNS resolver, PSL-backed enrichment worker, cache service, or availability provider in Compose. Production work still includes selecting and versioning the PSL, implementing bounded DNS resolution and TTL-aware cache invalidation, defining MX/NS/TXT parsing and uncertainty retention, adding association review/permission/audit tests, and completing an approved availability-provider integration with SSRF/network egress controls, monitoring, retention, and incident/rollback procedures.
 
+## Phase 8 website scanning boundary
+
+Website scanning is a bounded, tenant-scoped observation—not a crawler, browser, verifier, or outreach mechanism. A scan may fetch only `http` and `https` URLs after strict parsing and normalization. It must reject credentials, non-web schemes (`file:`, `ftp:`, `gopher:`, `data:`, `javascript:`, and similar), malformed hosts, localhost, IP literals where policy disallows them, and targets in loopback, private, link-local, multicast, reserved, or cloud-metadata ranges. DNS is resolved immediately before connection and the destination is revalidated at connection time; every redirect is limited, normalized, and revalidated for protocol, hostname, DNS, and IP range before it is followed. DNS answers must not be trusted from the initial validation alone (including rebinding changes).
+
+Each scan enforces hard budgets: total wall-clock/request time, response bytes, body bytes retained, redirect count, and page/link crawl count and depth. Budgets apply across redirects and discovered links, with bounded concurrency, retries, and response decompression; a limit, timeout, DNS error, unsupported content type, or partial fetch produces an explicit incomplete/unknown outcome rather than an empty result. The scanner fetches HTML and other explicitly allowed small resources only; it does not submit forms, send credentials, execute JavaScript, load browser plugins, or perform arbitrary subresource requests.
+
+Classifications are conservative and explainable. `unknown`, `blocked`, `timeout`, `partial`, and `error` remain distinct from a positive observation. A page can be classified only from bounded fetched content and must retain URL, redirect chain, response metadata, observed time, scanner/policy version, limits, and uncertainty reasons. A detected contact form, script, tracking tag, or business phrase is an observation—not proof of ownership, consent, deliverability, safety, or permission to contact.
+
+Scan history is tenant-scoped and append-oriented. Results and cache entries are keyed by normalized URL plus scanner/policy/version inputs, bounded by size and retention, and expose `observed_at`, freshness/expiry, and whether a result came from cache. A cache hit is never represented as a fresh scan; policy, DNS, or scanner-version changes require revalidation/invalidation. History must not leak response bodies, secrets, cookies, authorization headers, or unnecessary personal data across tenants.
+
+The website scanner remains a pilot boundary. Compose does not provide a production egress proxy, durable scan queue, distributed crawl coordinator, hardened DNS resolver, or compliance-grade result store. Production still requires independent SSRF testing (including DNS rebinding and redirect chains), egress/network policy, resource isolation, durable retention/deletion, authenticated scan-history authorization, rate limits and abuse controls, observability, and a reviewed policy for content types, robots/terms, caching, and incident response. Scans must never trigger acquisition, verification, enrichment, or outreach automatically.
+
 ## Verification
 
 ```bash

apps/api/README.md

@@ -1,6 +1,6 @@
-# Prospect Platform API — Phase 7 boundary
+# Prospect Platform API — Phase 8 boundary
 
-Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
+Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
 
 ## Run
 
@@ -114,8 +114,18 @@ Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class
 
 Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification.
 
-Domain availability is `unknown` unless an explicitly authorized availability provider supplies it. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
+Domain availability is `unknown` unless the API reports a result from an authorized provider. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
+
+## Phase 8 website-scanning contract
+
+Website scans are tenant-scoped, authenticated, bounded observations. Accept only `http` and `https`; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Resolve and validate the destination immediately before connecting, block loopback/private/link-local/multicast/reserved/cloud-metadata ranges, and repeat the protocol/host/DNS/IP checks for every redirect. Redirects must have a small fixed maximum and cannot escape the allowed protocol policy. DNS rebinding protections must validate the address actually used for the connection.
+
+Apply hard per-scan budgets for wall-clock time, connect/read timeouts, response/body bytes (including decompression), redirects, crawl depth, discovered links, and concurrency/retries. Crawl only explicitly allowed same-policy links; do not submit forms, send cookies or credentials, execute JavaScript, run plugins, or emulate a browser. Unsupported content, a budget exhaustion, timeout, DNS failure, redirect rejection, or partial response is an explicit `unknown`/`blocked`/`partial`/`error` result, never a successful empty page.
+
+Classifications must be conservative, explainable, and derived only from bounded fetched content. They are observations, not proof of ownership, identity, consent, deliverability, security, or contact permission. Persist the normalized URL, redirect chain, response metadata, observed time, scanner/policy/version, applied budgets, cache status, and uncertainty/error reasons; redact response bodies and secrets unless an approved minimal excerpt is required.
+
+Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
 
 ## Remaining limitations and production migration work
 
-SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Phase 7 production work must add a reviewed PSL update/version policy, bounded DNS workers, TTL-aware cache storage/invalidation, uncertainty-aware MX/NS/TXT schema and tests, association candidate safeguards and permissions, and an authorized availability provider with egress/rate/circuit/retention controls. It must also add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.
+SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.

apps/api/app/main.py

@@ -10,10 +10,12 @@ if __package__ in (None, ""):
     from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
     from app.sources import adapter_for, contains_secret
     from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
+    from app.website_scanner import scan_website, validate_url
 else:
     from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
     from .sources import adapter_for, contains_secret
     from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
+    from .website_scanner import scan_website, validate_url
 ORGANIZATION_ID = "demo-tenant"
 SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
 SESSION_DAYS = 7
@@ -21,6 +23,8 @@ PBKDF2_ITERATIONS = 300_000
 MUTATING_ROLES = {"owner", "admin", "researcher"}
 JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"}
 JOB_PAGE_SIZE = 100
+WEBSITE_SCAN_PAGE_SIZE = 100
+WEBSITE_SCAN_CACHE_SECONDS = 3600
 SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
 CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
 
@@ -120,12 +124,14 @@ class ApiHandler(BaseHTTPRequestHandler):
             if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query))
             if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
             if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query))
+            if path=="/api/v1/website-scans": return self.list_website_scans(db,org,parse_qs(parsed.query))
             if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
             if path.startswith("/api/v1/businesses/"):
                 bits=path.split("/"); ident=bits[4] if len(bits)>4 else ""
                 if not ident.isdigit(): return self.send_json(404,{"error":"not_found"})
                 row=self.business(db,int(ident),org)
                 if not row:return self.send_json(404,{"error":"not_found"})
+                if len(bits)==7 and bits[5:]==["websites","scan"]: return self.get_latest_website_scan(int(ident),db,user)
                 if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query))
                 if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org)
                 if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org)
@@ -173,6 +179,48 @@ class ApiHandler(BaseHTTPRequestHandler):
         rows=db.execute("SELECT * FROM domain_checks WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall()
         return self.send_json(200,{"organization_id":org,"items":[self._domain_result(r) for r in rows[:limit]],"limit":limit,"offset":offset,"has_more":len(rows)>limit})
 
+    def _website_scan_result(self, row, cache_hit=False):
+        result = json.loads(row["result_json"] or "{}")
+        result.update({"id": row["id"], "business_id": row["business_id"], "website_id": row["website_id"], "input_url": row["input_url"], "classification": row["classification"], "scanned_at": row["scanned_at"], "cache_expires_at": row["cache_expires_at"], "cache_hit": cache_hit})
+        return result
+
+    def list_website_scans(self, db, org, query):
+        try:
+            limit = int((query.get("page_size") or [50])[0]); offset = max(0, int((query.get("offset") or [0])[0]))
+            if limit < 1 or limit > WEBSITE_SCAN_PAGE_SIZE: raise ValueError
+        except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
+        params = [org]; where = ["organization_id=?"]
+        if (query.get("business_id") or [""])[0].isdigit(): where.append("business_id=?"); params.append(int(query["business_id"][0]))
+        rows = db.execute("SELECT * FROM website_scans WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
+        return self.send_json(200, {"organization_id": org, "items": [self._website_scan_result(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
+
+    def get_latest_website_scan(self, bid, db, user):
+        row = db.execute("SELECT * FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, user["organization_id"])).fetchone()
+        if not row: return self.send_json(404, {"error": "scan_not_found"})
+        return self.send_json(200, self._website_scan_result(row, False))
+
+    def scan_business_website(self, bid, payload, db, user):
+        org = user["organization_id"]; business = self.business(db, bid, org)
+        if not business: return self.send_json(404, {"error": "not_found"})
+        requested = str(payload.get("url", "")).strip() if isinstance(payload, dict) else ""
+        if not requested:
+            child = db.execute("SELECT * FROM websites WHERE business_id=? AND organization_id=? ORDER BY id LIMIT 1", (bid, org)).fetchone()
+            requested = (child["url"] if child else business["website"]) or ""
+        try: safe_url = validate_url(requested)
+        except ValueError as exc:
+            self.audit(db, user, "website.scan.rejected", f"{bid}:{str(exc)}"); db.commit()
+            return self.send_json(400, {"error": "unsafe_url", "reason": str(exc)})
+        website = db.execute("SELECT id FROM websites WHERE business_id=? AND organization_id=? AND url=? ORDER BY id LIMIT 1", (bid, org, safe_url)).fetchone()
+        cache_key = hashlib.sha256(safe_url.encode()).hexdigest(); now = datetime.now(timezone.utc).replace(microsecond=0); expires = now + timedelta(seconds=WEBSITE_SCAN_CACHE_SECONDS)
+        cached = db.execute("SELECT * FROM website_scans WHERE organization_id=? AND business_id=? AND cache_key=? AND cache_expires_at>? ORDER BY id DESC LIMIT 1", (org, bid, cache_key, now.isoformat())).fetchone()
+        if cached:
+            self.audit(db, user, "website.scan.cache_hit", f"{bid}:{safe_url}"); db.commit()
+            return self.send_json(200, self._website_scan_result(cached, True))
+        result = scan_website(safe_url); result["business_id"] = bid
+        cur = db.execute("INSERT INTO website_scans(organization_id,business_id,website_id,input_url,classification,result_json,cache_key,scanned_at,cache_expires_at) VALUES(?,?,?,?,?,?,?,?,?)", (org, bid, website["id"] if website else None, safe_url, result["classification"], json.dumps(result, sort_keys=True), cache_key, now.isoformat(), expires.isoformat()))
+        self.audit(db, user, "website.scanned", f"{bid}:{result['classification']}"); db.commit()
+        return self.send_json(201, self._website_scan_result(db.execute("SELECT * FROM website_scans WHERE id=?", (cur.lastrowid,)).fetchone()))
+
     def list_domain_candidates(self,bid,db,org):
         business=self.business(db,bid,org)
         if not business:return self.send_json(404,{"error":"not_found"})
@@ -286,6 +334,7 @@ class ApiHandler(BaseHTTPRequestHandler):
             if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
             if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
             if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
+            if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"websites"] and path.split("/")[6]=="scan": return self.scan_business_website(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
             if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
             if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
             if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"):

apps/api/app/website_scanner.py

@@ -0,0 +1,180 @@
+"""Bounded, passive and SSRF-safe website analysis using only the stdlib."""
+from __future__ import annotations
+
+import html
+import ipaddress
+import re
+import socket
+import ssl
+import time
+from html.parser import HTMLParser
+from urllib.parse import urljoin, urlparse
+from urllib.error import HTTPError
+from urllib.request import HTTPRedirectHandler, Request, build_opener
+
+MAX_BYTES = 512 * 1024
+MAX_REDIRECTS = 5
+MAX_PAGES = 1
+DEFAULT_TIMEOUT = 5.0
+_METADATA_IPS = {ipaddress.ip_address("169.254.169.254"), ipaddress.ip_address("100.100.100.200")}
+
+
+def _resolved_addresses(host: str, timeout: float) -> list[str]:
+    try:
+        socket.setdefaulttimeout(timeout)
+        records = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM)
+    except (OSError, socket.gaierror) as exc:
+        raise ValueError("dns_failure") from exc
+    addresses = sorted({str(r[4][0]) for r in records if len(r) > 4})
+    if not addresses:
+        raise ValueError("dns_failure")
+    for value in addresses:
+        try:
+            ip = ipaddress.ip_address(value)
+        except ValueError as exc:
+            raise ValueError("unsafe_address") from exc
+        if ip in _METADATA_IPS or not ip.is_global or ip.is_loopback or ip.is_private or ip.is_link_local or ip.is_multicast or ip.is_reserved or ip.is_unspecified:
+            raise ValueError("unsafe_address")
+    return addresses
+
+
+def validate_url(value: str, *, timeout: float = DEFAULT_TIMEOUT) -> str:
+    raw = str(value or "").strip()
+    parsed = urlparse(raw)
+    if parsed.scheme.lower() not in {"http", "https"} or not parsed.hostname or parsed.username or parsed.password:
+        raise ValueError("invalid_url")
+    if parsed.fragment:
+        raw = raw.split("#", 1)[0]
+        parsed = urlparse(raw)
+    host = (parsed.hostname or "").rstrip(".").lower()
+    if len(raw) > 2048 or len(host) > 253:
+        raise ValueError("invalid_url")
+    try:
+        ipaddress.ip_address(host)
+        _resolved_addresses(host, timeout)
+    except ValueError:
+        _resolved_addresses(host, timeout)
+    return parsed.geturl()
+
+
+class _Redirects(HTTPRedirectHandler):
+    def __init__(self, timeout: float, max_redirects: int):
+        self.timeout = timeout
+        self.max_redirects = max_redirects
+        self.chain: list[str] = []
+
+    def redirect_request(self, req, fp, code, msg, headers, newurl):
+        if len(self.chain) >= self.max_redirects:
+            raise ValueError("redirect_limit")
+        target = validate_url(urljoin(req.full_url, newurl), timeout=self.timeout)
+        self.chain.append(target)
+        return Request(target, headers={"User-Agent": "ProspectPlatformWebsiteScanner/1.0", "Accept": "text/html,application/xhtml+xml"}, method="GET")
+
+
+def _fetch(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int = MAX_BYTES, max_redirects: int = MAX_REDIRECTS) -> dict:
+    safe_url = validate_url(url, timeout=timeout)
+    redirects = _Redirects(timeout, max_redirects)
+    opener = build_opener(redirects)
+    request = Request(safe_url, headers={"User-Agent": "ProspectPlatformWebsiteScanner/1.0", "Accept": "text/html,application/xhtml+xml"}, method="GET")
+    started = time.monotonic()
+    try:
+        with opener.open(request, timeout=timeout) as response:
+            chunks, total = [], 0
+            while True:
+                chunk = response.read(min(65536, max_bytes - total + 1))
+                if not chunk:
+                    break
+                total += len(chunk)
+                if total > max_bytes:
+                    raise ValueError("response_too_large")
+                chunks.append(chunk)
+            final_url = validate_url(response.geturl(), timeout=timeout)
+            return {"status": int(response.status), "final_url": final_url, "redirect_chain": redirects.chain, "body": b"".join(chunks), "content_type": response.headers.get_content_type(), "elapsed_ms": round((time.monotonic() - started) * 1000, 2), "tls": final_url.startswith("https://"), "certificate_status": "valid" if final_url.startswith("https://") else "not_applicable"}
+    except HTTPError as exc:
+        # HTTP errors are still useful website observations; read only the bounded body.
+        body = exc.read(max_bytes + 1)
+        if len(body) > max_bytes: raise ValueError("response_too_large") from exc
+        return {"status": int(exc.code), "final_url": validate_url(exc.geturl(), timeout=timeout), "redirect_chain": redirects.chain, "body": body, "content_type": exc.headers.get_content_type() if exc.headers else "text/html", "elapsed_ms": round((time.monotonic() - started) * 1000, 2), "tls": str(exc.geturl()).startswith("https://"), "certificate_status": "valid" if str(exc.geturl()).startswith("https://") else "not_applicable"}
+    except ssl.SSLCertVerificationError as exc:
+        raise ValueError("certificate_invalid") from exc
+    except ValueError:
+        raise
+    except TimeoutError as exc:
+        raise ValueError("timeout") from exc
+    except OSError as exc:
+        raise ValueError("connection_failed") from exc
+
+
+class _PageParser(HTMLParser):
+    def __init__(self):
+        super().__init__(convert_charrefs=True)
+        self.title = ""; self.meta_description = ""; self.language = ""; self.headings: list[str] = []
+        self.viewport = False; self.cms_hints: set[str] = set(); self.contact_page = False; self.form = False
+        self.mail = False; self.phone = False; self.whatsapp = False; self.social = False; self._tag = ""; self._buf: list[str] = []
+
+    def handle_starttag(self, tag, attrs):
+        attrs = {str(k).lower(): str(v or "") for k, v in attrs}; tag = tag.lower()
+        self._tag = tag
+        if tag == "html": self.language = attrs.get("lang", "")[:20]
+        if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}: self._buf = []
+        if tag == "title": self._buf = []
+        if tag == "meta":
+            name = attrs.get("name", "").lower()
+            if name == "description": self.meta_description = attrs.get("content", "")[:1000]
+            if name == "viewport": self.viewport = True
+            generator = attrs.get("content", "").lower()
+            if name == "generator": self._cms(generator)
+        if tag == "form": self.form = True
+        if tag == "a":
+            href = attrs.get("href", "").lower()
+            self.contact_page |= any(x in href for x in ("contact", "get-in-touch", "reach-us"))
+            self.mail |= href.startswith("mailto:"); self.whatsapp |= "wa.me" in href or "whatsapp" in href
+            self.social |= any(x in href for x in ("facebook.com", "instagram.com", "linkedin.com", "twitter.com", "x.com", "youtube.com", "tiktok.com"))
+        if tag in {"script", "link"}:
+            text = " ".join(attrs.values()).lower(); self._cms(text)
+
+    def _cms(self, text):
+        for key, terms in {"wordpress": ("wordpress", "wp-content"), "drupal": ("drupal",), "joomla": ("joomla",), "shopify": ("shopify",), "wix": ("wix.com",)}.items():
+            if any(term in text for term in terms): self.cms_hints.add(key)
+
+    def handle_data(self, data):
+        if self._tag in {"title", "h1", "h2", "h3", "h4", "h5", "h6"}: self._buf.append(data)
+        if re.search(r"(?:tel:|\+?\d[\d ()-]{6,})", data): self.phone = True
+
+    def handle_endtag(self, tag):
+        tag = tag.lower()
+        if tag == "title" and self._buf: self.title = " ".join("".join(self._buf).split())[:500]
+        if tag in {"h1", "h2", "h3", "h4", "h5", "h6"} and self._buf:
+            self.headings.append(" ".join("".join(self._buf).split())[:300])
+        self._tag = ""
+
+
+def classify_website(status, final_url, body, *, error=None) -> str:
+    if error:
+        return "blocked" if error in {"timeout", "connection_failed", "dns_failure", "unsafe_address", "certificate_invalid", "redirect_limit", "response_too_large"} else "unknown"
+    if status is None: return "unknown"
+    if 400 <= status or status < 200: return "broken"
+    text = re.sub(r"<[^>]+>", " ", body if isinstance(body, str) else body.decode("utf-8", "replace")).lower()
+    if status in {301, 302, 303, 307, 308} and not text.strip(): return "redirect_only"
+    if re.search(r"domain (is )?for sale|buy this domain|parking page|parked free", text): return "parked"
+    if re.search(r"under construction|coming soon|website coming", text): return "under_construction"
+    if re.search(r"placeholder|lorem ipsum|sample page|default web page", text): return "placeholder"
+    if status < 300 and len(re.sub(r"\s+", "", text)) >= 8: return "healthy"
+    return "unknown"
+
+
+def scan_website(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int = MAX_BYTES, max_redirects: int = MAX_REDIRECTS, max_pages: int = MAX_PAGES) -> dict:
+    result = {"input_url": str(url), "status": None, "final_url": None, "redirect_chain": [], "title": "", "meta_description": "", "language": "", "headings": [], "responsive_signal": None, "cms_hints": [], "contact_page_signal": None, "form_signal": None, "mail_signal": None, "phone_signal": None, "whatsapp_signal": None, "social_signal": None, "elapsed_ms": None, "size_bytes": 0, "tls": None, "certificate_status": "unknown", "error_code": None}
+    try:
+        if not 0 < int(max_redirects) <= MAX_REDIRECTS or not 0 < int(max_pages) <= MAX_PAGES: raise ValueError("invalid_limits")
+        fetched = _fetch(url, timeout=max(0.1, min(float(timeout), 10.0)), max_bytes=max(1, min(int(max_bytes), MAX_BYTES)), max_redirects=int(max_redirects))
+        result.update({k: fetched[k] for k in ("status", "final_url", "redirect_chain", "elapsed_ms", "tls", "certificate_status")}); result["size_bytes"] = len(fetched["body"])
+        if fetched["content_type"] not in {"text/html", "application/xhtml+xml"}:
+            result["classification"] = classify_website(fetched["status"], fetched["final_url"], ""); return result
+        parser = _PageParser(); parser.feed(fetched["body"].decode("utf-8", "replace"))
+        for key in ("title", "meta_description", "language", "headings", "viewport", "cms_hints", "contact_page", "form", "mail", "phone", "whatsapp", "social"):
+            result[{"viewport":"responsive_signal","cms_hints":"cms_hints","contact_page":"contact_page_signal","form":"form_signal","mail":"mail_signal","phone":"phone_signal","whatsapp":"whatsapp_signal","social":"social_signal"}.get(key,key)] = sorted(parser.cms_hints) if key == "cms_hints" else getattr(parser, key)
+        result["classification"] = classify_website(result["status"], result["final_url"], fetched["body"])
+    except ValueError as exc:
+        result["error_code"] = str(exc); result["classification"] = classify_website(None, url, "", error=str(exc))
+    return result

apps/api/schema.sql

@@ -170,3 +170,21 @@ CREATE TABLE IF NOT EXISTS domain_candidates (
  created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,business_id,domain)
 );
 CREATE INDEX IF NOT EXISTS idx_domain_candidates_business ON domain_candidates(organization_id,business_id,rank,id);
+
+-- Phase 8 passive website analysis history; each scan is retained.
+CREATE TABLE IF NOT EXISTS website_scans (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ organization_id TEXT NOT NULL REFERENCES organizations(id),
+ business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
+ website_id INTEGER REFERENCES websites(id) ON DELETE SET NULL,
+ input_url TEXT NOT NULL,
+ classification TEXT NOT NULL,
+ result_json TEXT NOT NULL DEFAULT '{}',
+ cache_key TEXT NOT NULL,
+ scanned_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ cache_expires_at TEXT,
+ created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+CREATE INDEX IF NOT EXISTS idx_website_scans_org ON website_scans(organization_id,created_at DESC,id DESC);
+CREATE INDEX IF NOT EXISTS idx_website_scans_business ON website_scans(organization_id,business_id,created_at DESC,id DESC);
+CREATE INDEX IF NOT EXISTS idx_website_scans_cache ON website_scans(organization_id,cache_key,cache_expires_at);

apps/api/tests/test_phase8_api.py

@@ -0,0 +1,62 @@
+import json
+import os
+import threading
+import unittest
+from http.client import HTTPConnection
+from tempfile import TemporaryDirectory
+from unittest.mock import patch
+
+from app.main import create_server
+
+
+class WebsiteScanApiTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = TemporaryDirectory()
+        os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'scan-owner@example.test'
+        os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password'
+        self.server = create_server('127.0.0.1', 0, self.tmp.name + '/db.sqlite')
+        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
+        self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None
+        self.request('POST', '/api/v1/auth/login', {'email': 'scan-owner@example.test', 'password': 'password'})
+
+    def tearDown(self):
+        self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
+
+    def request(self, method, path, payload=None):
+        body = json.dumps(payload).encode() if payload is not None else None; headers = {'Content-Type': 'application/json'} if body else {}
+        if self.cookie: headers['Cookie'] = self.cookie
+        self.conn.request(method, path, body, headers); response = self.conn.getresponse(); cookie = response.getheader('Set-Cookie')
+        if cookie: self.cookie = cookie.split(';', 1)[0]
+        return response.status, json.loads(response.read() or b'{}')
+
+    def test_scan_is_cached_history_is_listed_and_audited(self):
+        _, business = self.request('POST', '/api/v1/businesses', {'name': 'Scan Co', 'website': 'https://scan.example'})
+        result = {'classification': 'healthy', 'status': 200, 'final_url': 'https://scan.example/', 'redirect_chain': []}
+        with patch('app.main.validate_url', return_value='https://scan.example/'), patch('app.main.scan_website', return_value=result) as scanner:
+            first_status, first = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
+            second_status, second = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
+        self.assertEqual(first_status, 201); self.assertEqual(second_status, 200); self.assertTrue(second['cache_hit']); scanner.assert_called_once()
+        status, history = self.request('GET', '/api/v1/website-scans?page_size=1')
+        self.assertEqual(status, 200); self.assertEqual(len(history['items']), 1); self.assertFalse(history['has_more'])
+        self.assertEqual(history['items'][0]['classification'], 'healthy')
+        self.assertEqual(self.request('GET', '/api/v1/website-scans?business_id=999999')[1]['items'], [])
+
+    def test_get_latest_scan_returns_scan_payload(self):
+        _, business = self.request('POST', '/api/v1/businesses', {'name': 'Read Scan Co', 'website': 'https://read.example'})
+        result = {'classification': 'unknown', 'status': 200, 'final_url': 'https://read.example/', 'redirect_chain': []}
+        with patch('app.main.validate_url', return_value='https://read.example/'), patch('app.main.scan_website', return_value=result):
+            self.assertEqual(self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})[0], 201)
+        status, payload = self.request('GET', f"/api/v1/businesses/{business['id']}/websites/scan")
+        self.assertEqual(status, 200)
+        self.assertEqual(payload['classification'], 'unknown')
+        self.assertEqual(payload['business_id'], business['id'])
+
+    def test_unsafe_scan_is_rejected_without_fetching(self):
+        _, business = self.request('POST', '/api/v1/businesses', {'name': 'Private Scan', 'website': 'http://127.0.0.1/'})
+        with patch('app.main.scan_website') as scanner:
+            status, payload = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
+        self.assertEqual(status, 400); self.assertEqual(payload['error'], 'unsafe_url'); scanner.assert_not_called()
+
+
+if __name__ == '__main__':
+    unittest.main()

apps/api/tests/test_phase8_website.py

@@ -0,0 +1,59 @@
+import unittest
+from unittest.mock import patch
+
+from app.website_scanner import classify_website, validate_url, scan_website
+
+
+class WebsiteScannerTests(unittest.TestCase):
+    def test_rejects_unsafe_schemes_and_addresses(self):
+        for url in ('file:///etc/passwd', 'ftp://example.com', 'http://127.0.0.1/', 'http://169.254.169.254/latest/meta-data'):
+            with self.subTest(url=url):
+                with self.assertRaises(ValueError):
+                    validate_url(url)
+
+    def test_classification_is_conservative(self):
+        self.assertEqual(classify_website(200, 'https://example.com', '<html><title>Acme</title><h1>Welcome</h1></html>'), 'healthy')
+        self.assertEqual(classify_website(404, 'https://example.com', ''), 'broken')
+        self.assertEqual(classify_website(200, 'https://example.com', '<html>under construction - coming soon</html>'), 'under_construction')
+        self.assertEqual(classify_website(200, 'https://example.com', '<html>domain for sale parking</html>'), 'parked')
+        self.assertEqual(classify_website(200, 'https://example.com', '<html>placeholder page</html>'), 'placeholder')
+        self.assertEqual(classify_website(301, 'https://example.com', ''), 'redirect_only')
+        self.assertEqual(classify_website(None, 'https://example.com', '', error='timeout'), 'blocked')
+        self.assertEqual(classify_website(None, 'https://example.com', ''), 'unknown')
+
+    def test_metadata_and_signals_are_extracted_from_fixture(self):
+        html = '''<!doctype html><html lang="en"><head><title>Acme</title>
+        <meta name="description" content="Solar experts"><meta name="viewport" content="width=device-width, initial-scale=1">
+        <meta name="generator" content="WordPress 6"><link rel="stylesheet" href="/style.css"></head>
+        <body><h1>Acme Solar</h1><h2>Contact us</h2><a href="/contact">Contact</a><a href="mailto:hi@acme.test">Email</a>
+        <a href="https://wa.me/27123456789">WhatsApp</a><a href="https://facebook.com/acme">Facebook</a><form action="/contact"></form></body></html>'''
+        with patch('app.website_scanner._fetch', return_value={
+            'status': 200, 'final_url': 'https://acme.test/', 'redirect_chain': [],
+            'body': html.encode(), 'content_type': 'text/html', 'elapsed_ms': 12, 'tls': True, 'certificate_status': 'valid'
+        }):
+            result = scan_website('https://acme.test/')
+        self.assertEqual(result['classification'], 'healthy')
+        self.assertEqual(result['title'], 'Acme')
+        self.assertEqual(result['meta_description'], 'Solar experts')
+        self.assertEqual(result['language'], 'en')
+        self.assertEqual(result['headings'], ['Acme Solar', 'Contact us'])
+        self.assertTrue(result['responsive_signal'])
+        self.assertIn('wordpress', result['cms_hints'])
+        self.assertTrue(result['contact_page_signal'])
+        self.assertTrue(result['form_signal'])
+        self.assertTrue(result['mail_signal'])
+        self.assertTrue(result['whatsapp_signal'])
+        self.assertTrue(result['social_signal'])
+        self.assertEqual(result['certificate_status'], 'valid')
+
+    def test_limits_are_reported_not_as_missing_contact_data(self):
+        with patch('app.website_scanner._fetch', side_effect=ValueError('response_too_large')):
+            result = scan_website('https://example.com')
+        self.assertEqual(result['classification'], 'blocked')
+        self.assertEqual(result['error_code'], 'response_too_large')
+        self.assertIsNone(result['contact_page_signal'])
+        self.assertIsNone(result['mail_signal'])
+
+
+if __name__ == '__main__':
+    unittest.main()

apps/web/README.md

@@ -1,6 +1,6 @@
-# ProspectOS web — Phase 7 boundary
+# ProspectOS web — Phase 8 boundary
 
-Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow and a Phase 4 MVP job monitor. Phase 5 source concepts, Phase 6 normalization/deduplication, and Phase 7 domain-intelligence concepts are display/contract boundaries only; the UI does not perform network discovery, scan DNS/websites, or send outreach.
+Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow, Phase 4 MVP job monitor, and Phase 8 scan-result/history presentation when supplied by the API. The browser does not fetch targets, submit forms, execute scan JavaScript, or send outreach; SSRF controls and budgets are server-side.
 
 ## Configure and run
 
@@ -32,6 +32,12 @@ Suggestions are review aids only. A merge flow must identify the surviving recor
 
 The current static MVP requests `/matches`, renders a **Human review required** list with confidence/reasons, asks for **Confirm merge**, and displays merge history with **Reverse merge** when the API marks it reversible. The API remains authoritative; these controls are not a substitute for server-side authorization. Existing normalization and match display remain suggestion-only; no merge happens without explicit operator confirmation.
 
+## Phase 8 website-scanning UI contract
+
+The UI may request a tenant-scoped scan through an authenticated API route when enabled and render the returned classification, status, redirect chain, observed time, scanner/policy version, cache freshness, applied budgets, and uncertainty/error reasons. The server permits only `http` and `https` targets. It must label cached data as cached/stale rather than “live,” keep `unknown`, `blocked`, `partial`, `timeout`, and `error` distinct from positive observations, and never turn a conservative classification into identity, ownership, consent, deliverability, or outreach permission.
+
+The browser must not directly fetch arbitrary target URLs, follow redirects for scanning, submit forms, send cookies/credentials, execute target JavaScript, or expose response bodies unnecessarily. Scan history and cache controls are tenant-scoped API capabilities, not hidden client state. A result that is budget-limited or incomplete must remain visibly incomplete; no UI timer may imply that a scan completed.
+
 ## Phase 5 source UI contract
 
 The web client may display registered source metadata, query mode, approval/terms state, rate-limit status, retention class, health, and circuit state returned by the API. It must label `dry_run` as a plan/validation result and distinguish operator-supplied CSV/manual references from independently verified evidence. It must not offer a live-source control unless the API reports explicit approval and operational enablement; client visibility is never an authorization control.
@@ -62,4 +68,4 @@ A zero-dependency static smoke page (`smoke-test.html`) checks the key DOM contr
 
 ## Remaining limitations
 
-The static client has no background discovery, DNS/website scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. For future domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability; any availability label requires an API result from an authorized provider and explicit human review. It cannot make missing provenance authoritative and should display API-provided limitations rather than infer them. CSV preview is capped for display and is not an import workflow. Production deployment still requires durable job execution, tenant-scoped controls, idempotency verification, PSL/DNS/cache implementation and tests, candidate review permissions/audit, authorized availability-provider controls, SSE/polling verification, and the security and operations gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`.
+The static client has no client-side crawler, scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. It can display server-provided Phase 8 observations, but production still requires server-side SSRF/DNS-rebinding/redirect controls, hard size/time/crawl budgets, durable scan history/cache isolation and retention, abuse/rate controls, and authenticated permission/audit coverage. For domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability. CSV preview is capped for display and is not an import workflow.

apps/web/app.js

@@ -56,7 +56,25 @@
   }
   async function runDomainCheck(){const button=$('runDomainCheckBtn');if(!selectedId||!button)return;button.disabled=true;const state=$('domainCheckState');if(state)state.innerHTML='<div class="detail-loading" aria-live="polite">Checking DNS…</div>';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domains/check`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain:domainForProspect(selectedDetail||{})})});renderDomainCheck(result);}catch(error){if(error.message!=='unauthorized'&&state)state.innerHTML=`<div class="detail-error" role="alert"><strong>DNS check failed</strong><p>${esc(error.message)}</p></div>`;}finally{button.disabled=false;}}
   async function checkDomainAvailability(domain,button){const result=button.closest('.candidate-card')?.querySelector('.availability-result');if(!domain||!result)return;button.disabled=true;result.textContent='Checking availability…';try{const body=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(selectedId)}/domain-candidates/check-availability`,{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({domain})});const state=body?.status||body?.state||body?.availability||'unknown';result.textContent=['not-configured','unknown','error'].includes(String(state).toLowerCase())?`Availability: ${domainState(state)}. No ownership conclusion.`:'Availability: Unknown. No ownership conclusion.';}catch(error){if(error.message!=='unauthorized')result.textContent=`Availability: Unknown. ${error.message}`;}finally{button.disabled=false;}}
-  function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderDomainPanel(p);renderDedupPanel();}
+  const websiteScanValue = (scan, keys, fallback='Unknown') => { for (const key of keys) if (scan?.[key] !== undefined && scan[key] !== null && scan[key] !== '') return scan[key]; return fallback; };
+  const websiteStateLabel = value => { const state=String(value ?? 'unknown').toLowerCase().replaceAll('_','-'); return ({queued:'Queued',running:'Running',succeeded:'Complete',completed:'Complete',healthy:'Healthy',broken:'Broken','under-construction':'Under construction',parked:'Parked',placeholder:'Placeholder','redirect-only':'Redirect only',blocked:'Blocked',unknown:'Unknown',error:'Error'})[state] || state.replaceAll('-',' ').replace(/\b\w/g,c=>c.toUpperCase()); };
+  const signalLabel = (value, positive='Present') => value === true ? positive : value === false ? 'Not found' : websiteStateLabel(value);
+  const websiteScanFor = p => p?.website_scan || p?.websiteScan || p?.scan || null;
+  const websiteUrlFor = p => p?.website || p?.website_url || p?.url || (domainForProspect(p) ? `https://${domainForProspect(p)}` : '');
+  function renderWebsiteScanPanel(p){
+    const panel=$('websiteScanPanel'); if(!panel)return;
+    panel.innerHTML=`<div class="website-scan-heading"><div><p class="eyebrow">WEBSITE ANALYSIS</p><h4>Website scan</h4></div><div class="website-scan-actions"><button class="button ghost compact" id="refreshWebsiteScanBtn" type="button">↻ Refresh</button><button class="button primary compact" id="scanWebsiteBtn" type="button">Scan website</button></div></div><p class="website-scan-safety">Website analysis is authenticated workspace evidence from the HTTP scanner. It does not enable outreach.</p><div id="websiteScanState" aria-live="polite">${websiteScanFor(p)?renderWebsiteScan(websiteScanFor(p)):'<p class="muted">No website scan has been returned. Start a scan to see HTTP and contact signals.</p>'}</div>`;
+    loadWebsiteScan(p.id);
+  }
+  function renderWebsiteScan(scan){
+    const status=websiteStateLabel(websiteScanValue(scan,['scan_status','status','state'])), classification=websiteStateLabel(websiteScanValue(scan,['classification','class'],'unknown')), http=websiteScanValue(scan,['http_status','status_code','status'], 'Unknown'), finalUrl=websiteScanValue(scan,['final_url','url'],'Unknown'), chain=websiteScanValue(scan,['redirect_chain','redirects'],[]), tls=scan?.tls===true||scan?.https===true||scan?.certificate_status==='valid'?'Secure':scan?.tls===false||scan?.https===false?'Not secure':websiteStateLabel(scan?.certificate_status||'Unknown');
+    const contact=[['Contact page',scan?.contact_page_signal],['Email',scan?.mail_signal],['Phone',scan?.phone_signal],['WhatsApp',scan?.whatsapp_signal],['Social',scan?.social_signal],['Form',scan?.form_signal]];
+    const value=v=>Array.isArray(v)?(v.length?v.join(' → '):'None'):v;
+    const error=scan?.error||scan?.error_code||scan?.message;
+    return `<div class="website-scan-meta"><span><b>Scan status</b><span class="website-status ${String(status).toLowerCase().replaceAll(' ','-')}">${esc(status)}</span></span><span>Checked ${esc(websiteScanValue(scan,['checked_at','last_checked_at','checked_time']))}</span><span>Cache: ${esc(websiteStateLabel(websiteScanValue(scan,['cache_state','cache_status','cache'], 'Unknown')))}</span></div>${error?`<div class="website-scan-error" role="alert"><strong>${esc(websiteStateLabel(scan.error_code||'Error'))}</strong><p>${esc(error)}</p></div>`:''}<div class="website-scan-grid"><div><dt>HTTP status</dt><dd>${esc(http)}</dd></div><div><dt>Final URL</dt><dd class="breakable">${esc(finalUrl)}</dd></div><div><dt>Redirect chain</dt><dd class="breakable">${esc(value(chain)||'None')}</dd></div><div><dt>Classification</dt><dd>${esc(classification)}</dd></div><div><dt>Title</dt><dd>${esc(websiteScanValue(scan,['title']))}</dd></div><div><dt>Meta description</dt><dd>${esc(websiteScanValue(scan,['meta_description','description']))}</dd></div><div><dt>Viewport</dt><dd>${esc(scan?.responsive_signal===true?'Responsive signal present':scan?.viewport||'Unknown')}</dd></div><div><dt>HTTPS / TLS</dt><dd>${esc(tls)}${scan?.certificate_status?` · ${esc(websiteStateLabel(scan.certificate_status))}`:''}</dd></div><div><dt>Load time</dt><dd>${esc(websiteScanValue(scan,['elapsed_ms','load_time_ms','duration_ms'],'Unknown'))}${scan?.elapsed_ms!=null?' ms':''}</dd></div><div><dt>Response size</dt><dd>${esc(websiteScanValue(scan,['response_size','size_bytes','body_size'],'Unknown'))}${scan?.size_bytes!=null?' bytes':''}</dd></div></div><div class="website-contact-signals"><h5>Contact signals</h5><div class="signal-list">${contact.map(([label,val])=>`<span class="signal ${val===true?'present':val===false?'absent':'unknown'}"><b>${esc(label)}</b>${esc(signalLabel(val))}</span>`).join('')}</div></div>`;
+  }
+  async function loadWebsiteScan(id, {scan=false}={}){const state=$('websiteScanState'), scanButton=$('scanWebsiteBtn'), refreshButton=$('refreshWebsiteScanBtn');if(!state||!id)return;[scanButton,refreshButton].forEach(button=>{if(button)button.disabled=true;});state.innerHTML='<div class="detail-loading" aria-live="polite">Loading website scan…</div>';try{const result=await jsonRequest(`/api/v1/businesses/${encodeURIComponent(id)}/websites/scan`,scan?{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({url:websiteUrlFor(selectedDetail||prospects.find(item=>Number(item.id)===Number(id))||{})})}:{});if(selectedDetail&&Number(selectedDetail.id)===Number(id))selectedDetail={...selectedDetail,website_scan:result};state.innerHTML=renderWebsiteScan(result);}catch(error){if(error.message!=='unauthorized')state.innerHTML=`<div class="detail-error" role="alert"><strong>${scan?'Website scan failed':'Unable to load website scan'}</strong><p>${esc(error.message)}</p><button class="button ghost compact" id="retryWebsiteScanBtn" type="button">Try again</button></div>`;}finally{[scanButton,refreshButton].forEach(button=>{if(button)button.disabled=false;});}}
+  function renderDetail(p){const s=scoreFor(p),st=statusOf(p),f=freshness(p),factors=p.score_factors||p.factors||[],blocked=st==='review'||st==='suppressed',stage=p.pipeline_stage||p.pipeline?.stage||(Array.isArray(p.pipeline)?p.pipeline.at(-1)?.stage:'')||'new',contacts=p.contacts||[],domains=p.domains||[],websites=p.websites||[],evidence=p.evidence||p.evidence_timeline||[],notes=p.notes||[],review=p.review_status||p.review|| (st==='review'?'Needs review':st[0].toUpperCase()+st.slice(1));$('detailPanel').innerHTML=`<div class="detail-head"><div><p class="eyebrow">PROSPECT DETAIL</p><h3>${esc(p.name)}</h3><p class="detail-domain">${esc(p.website_domain||'no detected website')}</p></div><span class="status ${st}">${esc(review)}</span></div><div class="detail-score"><div><small>Fit score</small><b>${s}<small>/ 100</small></b></div><span class="score ${scoreClass(s)}">${esc(p.confidence||(s>=80?'High':s>=60?'Medium':'Low'))} confidence</span></div><div class="detail-block"><h4>Pipeline stage</h4><form id="pipelineForm" class="inline-form"><select name="stage" aria-label="Pipeline stage"><option value="new" ${stage==='new'?'selected':''}>New</option><option value="qualified" ${stage==='qualified'?'selected':''}>Qualified</option><option value="review" ${stage==='review'?'selected':''}>Review</option><option value="suppressed" ${stage==='suppressed'?'selected':''}>Suppressed</option></select><button class="button ghost compact" type="submit">Save stage</button></form><p id="pipelineMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Contacts <span class="count">${contacts.length}</span></h4>${listItems(contacts,'No contacts added.','email')}<form id="contactForm" class="compact-form"><input name="name" placeholder="Contact name" aria-label="Contact name"><input name="email" type="email" placeholder="Email" aria-label="Contact email" required><button class="button ghost compact" type="submit">Add contact</button></form><p id="contactMessage" class="form-message" role="status"></p></div><section class="detail-block website-scan-panel" id="websiteScanPanel" data-smoke="website-scan"><div class="detail-loading">Loading website scan…</div></section><section class="detail-block domain-intelligence" id="domainIntelligencePanel" data-smoke="domain-intelligence"><div class="detail-loading">Loading domain intelligence…</div></section><div class="detail-block"><h4>Domains & websites</h4>${listItems(domains,'No domains recorded.','domain')}${listItems(websites,'No websites recorded.','url')}</div><div class="detail-block"><h4>Evidence timeline</h4>${listItems(evidence,'No evidence events recorded.','description')}${factors.length?factors.map(x=>`<p class="evidence-line"><span>✓ ${esc(labelFactor(x))}</span><span class="confidence">${esc(p.confidence||'Medium')}</span></p>`).join(''):''}</div><div class="detail-block"><h4>Notes <span class="count">${notes.length}</span></h4>${listItems(notes,'No notes added.','body')}<form id="noteForm" class="compact-form"><textarea name="body" rows="2" placeholder="Add a review note…" required></textarea><button class="button ghost compact" type="submit">Add note</button></form><p id="noteMessage" class="form-message" role="status"></p></div><div class="detail-block"><h4>Review status</h4><p class="review-status">${esc(review)}</p>${st!=='suppressed'?'<button class="button primary compact" id="verifyBtn" type="button">Mark verified</button>':''}<p id="verifyMessage" class="form-message" role="status"></p></div>${blocked?`<button class="button disabled-action" disabled aria-disabled="true">Outreach unavailable</button><p class="disabled-reason">${st==='suppressed'?'Suppressed records cannot be contacted.':'Review this prospect before outreach is available.'}</p>`:''}`;renderWebsiteScanPanel(p);renderDomainPanel(p);renderDedupPanel();}
   let mergeSource = null, mergeTarget = null, mergeBusy = false;
   const payloadItems = (payload, keys) => { for (const key of keys) if (Array.isArray(payload?.[key])) return payload[key]; return Array.isArray(payload) ? payload : []; };
   const suggestionId = item => item.target_id ?? item.business_id ?? item.prospect_id ?? item.id;
@@ -143,7 +161,7 @@
   async function logout(){try{await fetch(endpoint('/api/v1/auth/logout'),{method:'POST',credentials:'include'});}finally{showLogin('You have been signed out.');$('loginForm').reset();}}
   async function bootstrap(){try{const res=await fetch(endpoint('/api/v1/auth/me'),{credentials:'include'});if(res.status===401){showLogin();return;}if(!res.ok)throw new Error('Could not verify session.');const user=await res.json();showDashboard(user.user||user);await loadData();await loadJobs();await loadSources();}catch(e){if(e.message!=='unauthorized')showLogin('Unable to connect to the workspace. Try again.');}}
   document.addEventListener('submit',e=>{if(e.target.id==='contactForm')saveContact(e.target);if(e.target.id==='noteForm')saveNote(e.target);if(e.target.id==='pipelineForm')saveStage(e.target);});
-  document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
+  document.addEventListener('click',e=>{if(e.target.id==='verifyBtn')verify();if(e.target.id==='scanWebsiteBtn'&&selectedId)loadWebsiteScan(selectedId,{scan:true});if(e.target.id==='refreshWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='retryWebsiteScanBtn'&&selectedId)loadWebsiteScan(selectedId);if(e.target.id==='runDomainCheckBtn')runDomainCheck();if(e.target.id==='retryDomainBtn'&&selectedId)loadDomainIntelligence(selectedId);const availability=e.target.closest?.('[data-domain-availability]');if(availability)checkDomainAvailability(availability.dataset.domain,availability);if(e.target.id==='retryDetailBtn'&&selectedId)loadDetail(selectedId);if(e.target.id==='retryDedupBtn'&&selectedId)loadMatchSuggestions(selectedId);if(e.target.id==='retryHistoryBtn'&&selectedId)loadMergeHistory(selectedId);if(e.target.id==='cancelMergeBtn'||e.target.id==='cancelMergeBtnSecondary')closeMergeDialog();if(e.target.id==='confirmMergeBtn')confirmMerge();const mergeButton=e.target.closest?.('[data-merge-target]');if(mergeButton)openMergeDialog(mergeButton.dataset.mergeTarget,mergeButton.dataset.mergeTargetName);const reverseButton=e.target.closest?.('[data-reverse-merge]');if(reverseButton)reverseMerge(reverseButton.dataset.reverseMerge);if(e.target.id==='retryJobDetailBtn'&&selectedJobId)loadJobDetail(selectedJobId);if(e.target.id==='cancelJobBtn')jobAction('cancel');if(e.target.id==='retryJobBtn')jobAction('retry');const row=e.target.closest?.('[data-job-id]');if(row)loadJobDetail(row.dataset.jobId);});
   $('loginForm').addEventListener('submit',login);$('logoutBtn').addEventListener('click',logout);$('searchInput').addEventListener('input',()=>{page=1;renderRows();});['scoreFilter','statusFilter','websiteClassFilter','pipelineFilter'].forEach(id=>$(id).addEventListener('change',()=>{page=1;loadData();}));$('pageSize').addEventListener('change',e=>{pageSize=Number(e.target.value);page=1;loadData();});$('nextPageBtn').addEventListener('click',()=>{if(hasNextPage){page+=1;loadData();}});$('refreshBtn').addEventListener('click',loadData);$('jobsRefreshBtn').addEventListener('click',()=>loadJobs());$('startDemoJobBtn').addEventListener('click',startDemoJob);$('sourcesRefreshBtn').addEventListener('click',loadSources);$('sourceForm').addEventListener('submit',saveSource);$('sourceType').addEventListener('change',e=>{$('sourceCsvField').hidden=e.target.value!=='csv';});$('discoveryForm').addEventListener('submit',e=>{e.preventDefault();runDiscovery(true);});$('discoveryRunBtn').addEventListener('click',()=>runDiscovery(false));$('sourcesList').addEventListener('click',e=>{const button=e.target.closest?.('[data-source-action]');if(button)sourceAction(button.dataset.sourceId,button.dataset.sourceAction);});$('addForm').addEventListener('submit',addProspect);$('csvInput').addEventListener('change',e=>{const file=e.target.files[0];if(file){const reader=new FileReader();reader.onload=()=>renderCsv(parseCsv(reader.result));reader.readAsText(file);}});$('menuBtn').addEventListener('click',()=>document.querySelector('.sidebar').classList.toggle('open'));document.querySelectorAll('[data-scroll]').forEach(b=>b.addEventListener('click',()=>document.querySelector(b.dataset.scroll)?.scrollIntoView()));
   bootstrap();
 })();

apps/web/smoke-test.html

@@ -37,5 +37,10 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
  ,['DNS capability and uncertainty labels',()=>['DNS checks','A','AAAA','MX','NS','TXT','NXDOMAIN','Unknown','Error','Cache'].every(x=>js.includes(x))]
  ,['Candidate confidence, reasons, and safe availability',()=>js.includes('confidence')&&js.includes('reasons')&&js.includes('check-availability')&&js.includes('Not configured')&&js.includes('No ownership conclusion')&&!js.includes('Domain is available')]
  ,['Domain loading, error, tenant, and auth states',()=>js.includes('Loading DNS results')&&js.includes('Unable to load DNS results')&&js.includes('Tenant/workspace access denied')&&js.includes("credentials:'include'")]
+ ,['Website scan smoke marker and authenticated controls',()=>!!d.querySelector('[data-smoke="website-scan"]')&&!!d.querySelector('#scanWebsiteBtn')&&!!d.querySelector('#refreshWebsiteScanBtn')&&js.includes('/websites/scan')&&js.includes("method:'POST'")&&js.includes("credentials:'include'")]
+ ,['Website scan fields and conservative states',()=>['Scan status','HTTP status','Final URL','Redirect chain','Classification','Title','Meta description','Viewport','HTTPS / TLS','Load time','Response size','Contact signals','Checked','Cache','Blocked','Unknown','Error'].every(x=>js.includes(x))]
+ ,['Website scan loading and error states',()=>js.includes('Loading website scan')&&js.includes('Unable to load website scan')&&js.includes('Website scan failed')&&js.includes('role="alert"')]
+ ,['Website scan avoids browser execution claims',()=>!js.includes('screenshot')&&!js.includes('browser execution')&&!js.includes('page screenshot')]
+ ,['Website scan responsive styles',()=>js.includes('website-scan-panel')&&js.includes('website-scan-grid')&&js.includes('@media')]
 ];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
 </script>

docs/OPERATIONS.md

@@ -61,13 +61,21 @@ Review domain-to-business candidates separately from DNS results. Verify tenant
 
 Availability is `unknown` unless the API reports a result from an authorized provider. Before enabling one, verify current product/legal/security approval, terms owner, allowed tenant scope, request/concurrency and timeout limits, retention/deletion class, health/circuit state, and explicit operational enablement. `nxdomain`, `no_data`, timeout, stale cache, or provider error is not “available.” On provider outage, rate-limit, terms/approval expiry, circuit-open, or conflicting result, fail closed and report unknown/deferred; never purchase, reserve, contact, or retry through an unapproved provider.
 
+## Phase 8 website-scanning operations
+
+Website scans are bounded evidence collection, not browser sessions. The API must accept only `http`/`https`, resolve and validate DNS/IP immediately before connection, block loopback/private/link-local/multicast/reserved/cloud-metadata destinations, and repeat those checks for every redirect. Review redirect chains for protocol or host-policy violations; DNS rebinding or an unsafe resolved address is a blocked scan, not a retry opportunity.
+
+Monitor per-tenant and global scan counts, active concurrency, queue age, total/connect/read timeouts, response/decompressed bytes, redirect and crawl depth/link counts, cache hit/freshness, blocked destinations, DNS failures, partial results, and classification/error rates. Enforce hard budgets across redirects and discovered links. Classifications are conservative observations only: never submit forms, send cookies/credentials, execute JavaScript, or treat an HTTP 200/empty page as proof of a business fact. A scan that hits a limit is explicitly incomplete/unknown.
+
+Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.
+
 ## Phase 4 jobs and live logging
 
 The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.
 
 Creation must use a tenant-scoped idempotency key and request fingerprint. A repeated identical request returns the existing job/attempt; a conflicting payload is rejected. Cancellation is cooperative and race-safe, while retry is an explicit authorized new attempt linked to the original job and must not repeat completed side effects. Do not treat HTTP acceptance as completion, and do not reconstruct history from ephemeral container logs.
 
-The MVP has no SSE handler, durable queue, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.
+There is no SSE handler, durable queue, scan worker/isolation boundary, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.
 
 ## Configuration and deployment
 
@@ -117,7 +125,7 @@ Do not run `docker compose down -v` on a data-bearing environment: it removes th
 
 ## Production migration and scaling path
 
-Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events and domain observations, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker operations, bounded DNS/PSL processing, TTL-aware cache invalidation, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS status/error rates, cache freshness, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and scanners are possible future components—not implicit Compose dependencies and not implemented by this MVP. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.
+Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, domain observations, and scan history/cache, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker and scanner-isolation operations, bounded DNS/PSL/website processing, TTL/freshness-aware cache invalidation, SSRF/DNS-rebinding/redirect-chain tests, hard size/time/crawl budgets, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS/scanner status/error rates, cache freshness, blocked destinations, crawl-budget exhaustion, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and production scanners are possible future components—not implicit Compose dependencies. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.
 
 ## Incident checklist
 

docs/SECURITY.md

@@ -40,6 +40,17 @@ The MVP provides deterministic match suggestions, an explicit human confirmation
 
 No Phase 7 resolver, cache, or availability provider is enabled in the current Compose runtime. Before production, add egress/SSRF controls, provider and PSL update review, retention/deletion handling, monitoring, permission/audit coverage, and failure/rollback tests for all domain observations.
 
+## Phase 8 website-scanning controls
+
+- Scanning is an authenticated, tenant-scoped observation. Allow only `http` and `https`; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Never allow `file:`, `ftp:`, `gopher:`, `data:`, `javascript:`, or equivalent protocol smuggling.
+- Resolve immediately before connection and validate the actual destination address. Block loopback, private, link-local, multicast, reserved, and cloud-metadata ranges for IPv4 and IPv6. Re-run protocol, hostname, DNS, and IP checks on every redirect and protect against DNS rebinding; do not rely on an initial DNS check or an HTTP `Host` header.
+- Enforce hard budgets for connect/read/total time, response and decompressed bytes, retained body size, redirects, crawl depth/links, retries, and concurrency. Abort on budget exhaustion. Do not allow compression, redirects, or retries to bypass limits.
+- Fetch only explicitly allowed content types and links. Never submit forms, send user cookies/credentials/authorization headers, execute JavaScript, run plugins, or make arbitrary subresource requests. Treat fetched content and all TXT/HTML/script text as untrusted input and escape it on display.
+- Classify conservatively: `unknown`, `blocked`, `partial`, `timeout`, and `error` are not empty success and are not negative business facts. A classification is evidence of bounded content only—not ownership, identity, consent, deliverability, security, or permission to contact.
+- Persist scan history/cache with tenant isolation, normalized URL, policy/scanner version, redirect policy, observed time, freshness/expiry, applied budgets, and uncertainty/error metadata. Bound size/retention, redact secrets and response bodies, and invalidate/re-evaluate after policy, DNS, or scanner-version changes. A cache hit must be visibly non-fresh.
+
+No production-grade scanner egress proxy, isolated worker, or durable scan store is supplied by the current Compose runtime. Before enabling scanning in production, add SSRF/DNS-rebinding/redirect-chain tests, egress deny-by-default policy, abuse/rate controls, authenticated history authorization, retention/deletion, monitoring, and incident procedures. Scans must never cause form submission, acquisition, verification, enrichment, or outreach.
+
 ## Phase 5 source security controls
 
 Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
@@ -59,7 +70,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in
 2. **MFA:** require phishing-resistant or TOTP MFA for administrator accounts in production, including the bootstrap admin before granting ongoing administrative access. Define recovery, enrollment, reset, and revocation procedures; do not treat a password-only bootstrap as production-ready.
 3. **Authentication and authorization:** enforce authorization server-side on every protected route, including every child-record, note, pipeline, and audit route. Rotate/regenerate sessions at login and privilege changes, expire idle/absolute sessions, revoke on logout/password reset, and test tenant isolation.
 4. **Cookies and CSRF:** use `HttpOnly`, `Secure` (production HTTPS), and an appropriate `SameSite` policy. Browser state-changing endpoints require CSRF tokens (or a rigorously reviewed equivalent); do not rely on CORS or cookie flags alone.
-5. **SSRF and future scanners:** no scanner is enabled in this release. If a future approved feature fetches a URL, allow only `http`/`https`, validate DNS/IP targets, block loopback/private/link-local/cloud-metadata ranges after resolution, limit redirects, enforce size/time limits, and re-check each redirect.
+5. **SSRF and scanner production hardening:** the Phase 8 scanner is bounded and conservative, but production still requires an egress proxy/isolation boundary, DNS-rebinding and redirect-chain regression tests, deny-by-default network policy, abuse controls, and durable scan-history/cache retention. Allow only `http`/`https`; never submit forms or execute JavaScript.
 6. **Input/output safety:** validate schema and content types, bound request and note/evidence sizes, parameterize database queries, escape output, and reject unsafe provenance URLs or markup. Treat operator-entered notes and sources as untrusted data.
 7. **Audit and retention:** the current audit/activity behavior is an MVP trail, not an immutable compliance log. Define append-only guarantees, retention, redaction/deletion rules, access controls, alerting, and export procedures before production.
 8. **Secrets:** inject production secrets from a secret manager or orchestrator secret store. Do not place them in images, Compose files, source, CI logs, or committed `.env` files. Remove bootstrap variables after first-run provisioning.