add ssrf-safe website analysis

apps/api/README.md

@@ -1,6 +1,6 @@
-# Prospect Platform API — Phase 7 boundary
+# Prospect Platform API — Phase 8 boundary
 
-Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
+Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
 
 ## Run
 
@@ -114,8 +114,18 @@ Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class
 
 Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification.
 
-Domain availability is `unknown` unless an explicitly authorized availability provider supplies it. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
+Domain availability is `unknown` unless the API reports a result from an authorized provider. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
+
+## Phase 8 website-scanning contract
+
+Website scans are tenant-scoped, authenticated, bounded observations. Accept only `http` and `https`; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Resolve and validate the destination immediately before connecting, block loopback/private/link-local/multicast/reserved/cloud-metadata ranges, and repeat the protocol/host/DNS/IP checks for every redirect. Redirects must have a small fixed maximum and cannot escape the allowed protocol policy. DNS rebinding protections must validate the address actually used for the connection.
+
+Apply hard per-scan budgets for wall-clock time, connect/read timeouts, response/body bytes (including decompression), redirects, crawl depth, discovered links, and concurrency/retries. Crawl only explicitly allowed same-policy links; do not submit forms, send cookies or credentials, execute JavaScript, run plugins, or emulate a browser. Unsupported content, a budget exhaustion, timeout, DNS failure, redirect rejection, or partial response is an explicit `unknown`/`blocked`/`partial`/`error` result, never a successful empty page.
+
+Classifications must be conservative, explainable, and derived only from bounded fetched content. They are observations, not proof of ownership, identity, consent, deliverability, security, or contact permission. Persist the normalized URL, redirect chain, response metadata, observed time, scanner/policy/version, applied budgets, cache status, and uncertainty/error reasons; redact response bodies and secrets unless an approved minimal excerpt is required.
+
+Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
 
 ## Remaining limitations and production migration work
 
-SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Phase 7 production work must add a reviewed PSL update/version policy, bounded DNS workers, TTL-aware cache storage/invalidation, uncertainty-aware MX/NS/TXT schema and tests, association candidate safeguards and permissions, and an authorized availability provider with egress/rate/circuit/retention controls. It must also add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.
+SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.

apps/api/app/main.py

@@ -10,10 +10,12 @@ if __package__ in (None, ""):
     from app.domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
     from app.sources import adapter_for, contains_secret
     from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
+    from app.website_scanner import scan_website, validate_url
 else:
     from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
     from .sources import adapter_for, contains_secret
     from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
+    from .website_scanner import scan_website, validate_url
 ORGANIZATION_ID = "demo-tenant"
 SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
 SESSION_DAYS = 7
@@ -21,6 +23,8 @@ PBKDF2_ITERATIONS = 300_000
 MUTATING_ROLES = {"owner", "admin", "researcher"}
 JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"}
 JOB_PAGE_SIZE = 100
+WEBSITE_SCAN_PAGE_SIZE = 100
+WEBSITE_SCAN_CACHE_SECONDS = 3600
 SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
 CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
 
@@ -120,12 +124,14 @@ class ApiHandler(BaseHTTPRequestHandler):
             if path=="/api/v1/source-records": return self.list_source_records(db,org,parse_qs(parsed.query))
             if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
             if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query))
+            if path=="/api/v1/website-scans": return self.list_website_scans(db,org,parse_qs(parsed.query))
             if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
             if path.startswith("/api/v1/businesses/"):
                 bits=path.split("/"); ident=bits[4] if len(bits)>4 else ""
                 if not ident.isdigit(): return self.send_json(404,{"error":"not_found"})
                 row=self.business(db,int(ident),org)
                 if not row:return self.send_json(404,{"error":"not_found"})
+                if len(bits)==7 and bits[5:]==["websites","scan"]: return self.get_latest_website_scan(int(ident),db,user)
                 if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query))
                 if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org)
                 if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org)
@@ -173,6 +179,48 @@ class ApiHandler(BaseHTTPRequestHandler):
         rows=db.execute("SELECT * FROM domain_checks WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?",(org,limit+1,offset)).fetchall()
         return self.send_json(200,{"organization_id":org,"items":[self._domain_result(r) for r in rows[:limit]],"limit":limit,"offset":offset,"has_more":len(rows)>limit})
 
+    def _website_scan_result(self, row, cache_hit=False):
+        result = json.loads(row["result_json"] or "{}")
+        result.update({"id": row["id"], "business_id": row["business_id"], "website_id": row["website_id"], "input_url": row["input_url"], "classification": row["classification"], "scanned_at": row["scanned_at"], "cache_expires_at": row["cache_expires_at"], "cache_hit": cache_hit})
+        return result
+
+    def list_website_scans(self, db, org, query):
+        try:
+            limit = int((query.get("page_size") or [50])[0]); offset = max(0, int((query.get("offset") or [0])[0]))
+            if limit < 1 or limit > WEBSITE_SCAN_PAGE_SIZE: raise ValueError
+        except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
+        params = [org]; where = ["organization_id=?"]
+        if (query.get("business_id") or [""])[0].isdigit(): where.append("business_id=?"); params.append(int(query["business_id"][0]))
+        rows = db.execute("SELECT * FROM website_scans WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
+        return self.send_json(200, {"organization_id": org, "items": [self._website_scan_result(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
+
+    def get_latest_website_scan(self, bid, db, user):
+        row = db.execute("SELECT * FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, user["organization_id"])).fetchone()
+        if not row: return self.send_json(404, {"error": "scan_not_found"})
+        return self.send_json(200, self._website_scan_result(row, False))
+
+    def scan_business_website(self, bid, payload, db, user):
+        org = user["organization_id"]; business = self.business(db, bid, org)
+        if not business: return self.send_json(404, {"error": "not_found"})
+        requested = str(payload.get("url", "")).strip() if isinstance(payload, dict) else ""
+        if not requested:
+            child = db.execute("SELECT * FROM websites WHERE business_id=? AND organization_id=? ORDER BY id LIMIT 1", (bid, org)).fetchone()
+            requested = (child["url"] if child else business["website"]) or ""
+        try: safe_url = validate_url(requested)
+        except ValueError as exc:
+            self.audit(db, user, "website.scan.rejected", f"{bid}:{str(exc)}"); db.commit()
+            return self.send_json(400, {"error": "unsafe_url", "reason": str(exc)})
+        website = db.execute("SELECT id FROM websites WHERE business_id=? AND organization_id=? AND url=? ORDER BY id LIMIT 1", (bid, org, safe_url)).fetchone()
+        cache_key = hashlib.sha256(safe_url.encode()).hexdigest(); now = datetime.now(timezone.utc).replace(microsecond=0); expires = now + timedelta(seconds=WEBSITE_SCAN_CACHE_SECONDS)
+        cached = db.execute("SELECT * FROM website_scans WHERE organization_id=? AND business_id=? AND cache_key=? AND cache_expires_at>? ORDER BY id DESC LIMIT 1", (org, bid, cache_key, now.isoformat())).fetchone()
+        if cached:
+            self.audit(db, user, "website.scan.cache_hit", f"{bid}:{safe_url}"); db.commit()
+            return self.send_json(200, self._website_scan_result(cached, True))
+        result = scan_website(safe_url); result["business_id"] = bid
+        cur = db.execute("INSERT INTO website_scans(organization_id,business_id,website_id,input_url,classification,result_json,cache_key,scanned_at,cache_expires_at) VALUES(?,?,?,?,?,?,?,?,?)", (org, bid, website["id"] if website else None, safe_url, result["classification"], json.dumps(result, sort_keys=True), cache_key, now.isoformat(), expires.isoformat()))
+        self.audit(db, user, "website.scanned", f"{bid}:{result['classification']}"); db.commit()
+        return self.send_json(201, self._website_scan_result(db.execute("SELECT * FROM website_scans WHERE id=?", (cur.lastrowid,)).fetchone()))
+
     def list_domain_candidates(self,bid,db,org):
         business=self.business(db,bid,org)
         if not business:return self.send_json(404,{"error":"not_found"})
@@ -286,6 +334,7 @@ class ApiHandler(BaseHTTPRequestHandler):
             if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
             if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
             if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
+            if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"websites"] and path.split("/")[6]=="scan": return self.scan_business_website(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
             if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
             if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
             if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"):

apps/api/app/website_scanner.py

@@ -0,0 +1,180 @@
+"""Bounded, passive and SSRF-safe website analysis using only the stdlib."""
+from __future__ import annotations
+
+import html
+import ipaddress
+import re
+import socket
+import ssl
+import time
+from html.parser import HTMLParser
+from urllib.parse import urljoin, urlparse
+from urllib.error import HTTPError
+from urllib.request import HTTPRedirectHandler, Request, build_opener
+
+MAX_BYTES = 512 * 1024
+MAX_REDIRECTS = 5
+MAX_PAGES = 1
+DEFAULT_TIMEOUT = 5.0
+_METADATA_IPS = {ipaddress.ip_address("169.254.169.254"), ipaddress.ip_address("100.100.100.200")}
+
+
+def _resolved_addresses(host: str, timeout: float) -> list[str]:
+    try:
+        socket.setdefaulttimeout(timeout)
+        records = socket.getaddrinfo(host, None, type=socket.SOCK_STREAM)
+    except (OSError, socket.gaierror) as exc:
+        raise ValueError("dns_failure") from exc
+    addresses = sorted({str(r[4][0]) for r in records if len(r) > 4})
+    if not addresses:
+        raise ValueError("dns_failure")
+    for value in addresses:
+        try:
+            ip = ipaddress.ip_address(value)
+        except ValueError as exc:
+            raise ValueError("unsafe_address") from exc
+        if ip in _METADATA_IPS or not ip.is_global or ip.is_loopback or ip.is_private or ip.is_link_local or ip.is_multicast or ip.is_reserved or ip.is_unspecified:
+            raise ValueError("unsafe_address")
+    return addresses
+
+
+def validate_url(value: str, *, timeout: float = DEFAULT_TIMEOUT) -> str:
+    raw = str(value or "").strip()
+    parsed = urlparse(raw)
+    if parsed.scheme.lower() not in {"http", "https"} or not parsed.hostname or parsed.username or parsed.password:
+        raise ValueError("invalid_url")
+    if parsed.fragment:
+        raw = raw.split("#", 1)[0]
+        parsed = urlparse(raw)
+    host = (parsed.hostname or "").rstrip(".").lower()
+    if len(raw) > 2048 or len(host) > 253:
+        raise ValueError("invalid_url")
+    try:
+        ipaddress.ip_address(host)
+        _resolved_addresses(host, timeout)
+    except ValueError:
+        _resolved_addresses(host, timeout)
+    return parsed.geturl()
+
+
+class _Redirects(HTTPRedirectHandler):
+    def __init__(self, timeout: float, max_redirects: int):
+        self.timeout = timeout
+        self.max_redirects = max_redirects
+        self.chain: list[str] = []
+
+    def redirect_request(self, req, fp, code, msg, headers, newurl):
+        if len(self.chain) >= self.max_redirects:
+            raise ValueError("redirect_limit")
+        target = validate_url(urljoin(req.full_url, newurl), timeout=self.timeout)
+        self.chain.append(target)
+        return Request(target, headers={"User-Agent": "ProspectPlatformWebsiteScanner/1.0", "Accept": "text/html,application/xhtml+xml"}, method="GET")
+
+
+def _fetch(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int = MAX_BYTES, max_redirects: int = MAX_REDIRECTS) -> dict:
+    safe_url = validate_url(url, timeout=timeout)
+    redirects = _Redirects(timeout, max_redirects)
+    opener = build_opener(redirects)
+    request = Request(safe_url, headers={"User-Agent": "ProspectPlatformWebsiteScanner/1.0", "Accept": "text/html,application/xhtml+xml"}, method="GET")
+    started = time.monotonic()
+    try:
+        with opener.open(request, timeout=timeout) as response:
+            chunks, total = [], 0
+            while True:
+                chunk = response.read(min(65536, max_bytes - total + 1))
+                if not chunk:
+                    break
+                total += len(chunk)
+                if total > max_bytes:
+                    raise ValueError("response_too_large")
+                chunks.append(chunk)
+            final_url = validate_url(response.geturl(), timeout=timeout)
+            return {"status": int(response.status), "final_url": final_url, "redirect_chain": redirects.chain, "body": b"".join(chunks), "content_type": response.headers.get_content_type(), "elapsed_ms": round((time.monotonic() - started) * 1000, 2), "tls": final_url.startswith("https://"), "certificate_status": "valid" if final_url.startswith("https://") else "not_applicable"}
+    except HTTPError as exc:
+        # HTTP errors are still useful website observations; read only the bounded body.
+        body = exc.read(max_bytes + 1)
+        if len(body) > max_bytes: raise ValueError("response_too_large") from exc
+        return {"status": int(exc.code), "final_url": validate_url(exc.geturl(), timeout=timeout), "redirect_chain": redirects.chain, "body": body, "content_type": exc.headers.get_content_type() if exc.headers else "text/html", "elapsed_ms": round((time.monotonic() - started) * 1000, 2), "tls": str(exc.geturl()).startswith("https://"), "certificate_status": "valid" if str(exc.geturl()).startswith("https://") else "not_applicable"}
+    except ssl.SSLCertVerificationError as exc:
+        raise ValueError("certificate_invalid") from exc
+    except ValueError:
+        raise
+    except TimeoutError as exc:
+        raise ValueError("timeout") from exc
+    except OSError as exc:
+        raise ValueError("connection_failed") from exc
+
+
+class _PageParser(HTMLParser):
+    def __init__(self):
+        super().__init__(convert_charrefs=True)
+        self.title = ""; self.meta_description = ""; self.language = ""; self.headings: list[str] = []
+        self.viewport = False; self.cms_hints: set[str] = set(); self.contact_page = False; self.form = False
+        self.mail = False; self.phone = False; self.whatsapp = False; self.social = False; self._tag = ""; self._buf: list[str] = []
+
+    def handle_starttag(self, tag, attrs):
+        attrs = {str(k).lower(): str(v or "") for k, v in attrs}; tag = tag.lower()
+        self._tag = tag
+        if tag == "html": self.language = attrs.get("lang", "")[:20]
+        if tag in {"h1", "h2", "h3", "h4", "h5", "h6"}: self._buf = []
+        if tag == "title": self._buf = []
+        if tag == "meta":
+            name = attrs.get("name", "").lower()
+            if name == "description": self.meta_description = attrs.get("content", "")[:1000]
+            if name == "viewport": self.viewport = True
+            generator = attrs.get("content", "").lower()
+            if name == "generator": self._cms(generator)
+        if tag == "form": self.form = True
+        if tag == "a":
+            href = attrs.get("href", "").lower()
+            self.contact_page |= any(x in href for x in ("contact", "get-in-touch", "reach-us"))
+            self.mail |= href.startswith("mailto:"); self.whatsapp |= "wa.me" in href or "whatsapp" in href
+            self.social |= any(x in href for x in ("facebook.com", "instagram.com", "linkedin.com", "twitter.com", "x.com", "youtube.com", "tiktok.com"))
+        if tag in {"script", "link"}:
+            text = " ".join(attrs.values()).lower(); self._cms(text)
+
+    def _cms(self, text):
+        for key, terms in {"wordpress": ("wordpress", "wp-content"), "drupal": ("drupal",), "joomla": ("joomla",), "shopify": ("shopify",), "wix": ("wix.com",)}.items():
+            if any(term in text for term in terms): self.cms_hints.add(key)
+
+    def handle_data(self, data):
+        if self._tag in {"title", "h1", "h2", "h3", "h4", "h5", "h6"}: self._buf.append(data)
+        if re.search(r"(?:tel:|\+?\d[\d ()-]{6,})", data): self.phone = True
+
+    def handle_endtag(self, tag):
+        tag = tag.lower()
+        if tag == "title" and self._buf: self.title = " ".join("".join(self._buf).split())[:500]
+        if tag in {"h1", "h2", "h3", "h4", "h5", "h6"} and self._buf:
+            self.headings.append(" ".join("".join(self._buf).split())[:300])
+        self._tag = ""
+
+
+def classify_website(status, final_url, body, *, error=None) -> str:
+    if error:
+        return "blocked" if error in {"timeout", "connection_failed", "dns_failure", "unsafe_address", "certificate_invalid", "redirect_limit", "response_too_large"} else "unknown"
+    if status is None: return "unknown"
+    if 400 <= status or status < 200: return "broken"
+    text = re.sub(r"<[^>]+>", " ", body if isinstance(body, str) else body.decode("utf-8", "replace")).lower()
+    if status in {301, 302, 303, 307, 308} and not text.strip(): return "redirect_only"
+    if re.search(r"domain (is )?for sale|buy this domain|parking page|parked free", text): return "parked"
+    if re.search(r"under construction|coming soon|website coming", text): return "under_construction"
+    if re.search(r"placeholder|lorem ipsum|sample page|default web page", text): return "placeholder"
+    if status < 300 and len(re.sub(r"\s+", "", text)) >= 8: return "healthy"
+    return "unknown"
+
+
+def scan_website(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int = MAX_BYTES, max_redirects: int = MAX_REDIRECTS, max_pages: int = MAX_PAGES) -> dict:
+    result = {"input_url": str(url), "status": None, "final_url": None, "redirect_chain": [], "title": "", "meta_description": "", "language": "", "headings": [], "responsive_signal": None, "cms_hints": [], "contact_page_signal": None, "form_signal": None, "mail_signal": None, "phone_signal": None, "whatsapp_signal": None, "social_signal": None, "elapsed_ms": None, "size_bytes": 0, "tls": None, "certificate_status": "unknown", "error_code": None}
+    try:
+        if not 0 < int(max_redirects) <= MAX_REDIRECTS or not 0 < int(max_pages) <= MAX_PAGES: raise ValueError("invalid_limits")
+        fetched = _fetch(url, timeout=max(0.1, min(float(timeout), 10.0)), max_bytes=max(1, min(int(max_bytes), MAX_BYTES)), max_redirects=int(max_redirects))
+        result.update({k: fetched[k] for k in ("status", "final_url", "redirect_chain", "elapsed_ms", "tls", "certificate_status")}); result["size_bytes"] = len(fetched["body"])
+        if fetched["content_type"] not in {"text/html", "application/xhtml+xml"}:
+            result["classification"] = classify_website(fetched["status"], fetched["final_url"], ""); return result
+        parser = _PageParser(); parser.feed(fetched["body"].decode("utf-8", "replace"))
+        for key in ("title", "meta_description", "language", "headings", "viewport", "cms_hints", "contact_page", "form", "mail", "phone", "whatsapp", "social"):
+            result[{"viewport":"responsive_signal","cms_hints":"cms_hints","contact_page":"contact_page_signal","form":"form_signal","mail":"mail_signal","phone":"phone_signal","whatsapp":"whatsapp_signal","social":"social_signal"}.get(key,key)] = sorted(parser.cms_hints) if key == "cms_hints" else getattr(parser, key)
+        result["classification"] = classify_website(result["status"], result["final_url"], fetched["body"])
+    except ValueError as exc:
+        result["error_code"] = str(exc); result["classification"] = classify_website(None, url, "", error=str(exc))
+    return result

apps/api/schema.sql

@@ -170,3 +170,21 @@ CREATE TABLE IF NOT EXISTS domain_candidates (
  created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(organization_id,business_id,domain)
 );
 CREATE INDEX IF NOT EXISTS idx_domain_candidates_business ON domain_candidates(organization_id,business_id,rank,id);
+
+-- Phase 8 passive website analysis history; each scan is retained.
+CREATE TABLE IF NOT EXISTS website_scans (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ organization_id TEXT NOT NULL REFERENCES organizations(id),
+ business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
+ website_id INTEGER REFERENCES websites(id) ON DELETE SET NULL,
+ input_url TEXT NOT NULL,
+ classification TEXT NOT NULL,
+ result_json TEXT NOT NULL DEFAULT '{}',
+ cache_key TEXT NOT NULL,
+ scanned_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
+ cache_expires_at TEXT,
+ created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+CREATE INDEX IF NOT EXISTS idx_website_scans_org ON website_scans(organization_id,created_at DESC,id DESC);
+CREATE INDEX IF NOT EXISTS idx_website_scans_business ON website_scans(organization_id,business_id,created_at DESC,id DESC);
+CREATE INDEX IF NOT EXISTS idx_website_scans_cache ON website_scans(organization_id,cache_key,cache_expires_at);

apps/api/tests/test_phase8_api.py

@@ -0,0 +1,62 @@
+import json
+import os
+import threading
+import unittest
+from http.client import HTTPConnection
+from tempfile import TemporaryDirectory
+from unittest.mock import patch
+
+from app.main import create_server
+
+
+class WebsiteScanApiTests(unittest.TestCase):
+    def setUp(self):
+        self.tmp = TemporaryDirectory()
+        os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'scan-owner@example.test'
+        os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password'
+        self.server = create_server('127.0.0.1', 0, self.tmp.name + '/db.sqlite')
+        self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
+        self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None
+        self.request('POST', '/api/v1/auth/login', {'email': 'scan-owner@example.test', 'password': 'password'})
+
+    def tearDown(self):
+        self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
+
+    def request(self, method, path, payload=None):
+        body = json.dumps(payload).encode() if payload is not None else None; headers = {'Content-Type': 'application/json'} if body else {}
+        if self.cookie: headers['Cookie'] = self.cookie
+        self.conn.request(method, path, body, headers); response = self.conn.getresponse(); cookie = response.getheader('Set-Cookie')
+        if cookie: self.cookie = cookie.split(';', 1)[0]
+        return response.status, json.loads(response.read() or b'{}')
+
+    def test_scan_is_cached_history_is_listed_and_audited(self):
+        _, business = self.request('POST', '/api/v1/businesses', {'name': 'Scan Co', 'website': 'https://scan.example'})
+        result = {'classification': 'healthy', 'status': 200, 'final_url': 'https://scan.example/', 'redirect_chain': []}
+        with patch('app.main.validate_url', return_value='https://scan.example/'), patch('app.main.scan_website', return_value=result) as scanner:
+            first_status, first = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
+            second_status, second = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
+        self.assertEqual(first_status, 201); self.assertEqual(second_status, 200); self.assertTrue(second['cache_hit']); scanner.assert_called_once()
+        status, history = self.request('GET', '/api/v1/website-scans?page_size=1')
+        self.assertEqual(status, 200); self.assertEqual(len(history['items']), 1); self.assertFalse(history['has_more'])
+        self.assertEqual(history['items'][0]['classification'], 'healthy')
+        self.assertEqual(self.request('GET', '/api/v1/website-scans?business_id=999999')[1]['items'], [])
+
+    def test_get_latest_scan_returns_scan_payload(self):
+        _, business = self.request('POST', '/api/v1/businesses', {'name': 'Read Scan Co', 'website': 'https://read.example'})
+        result = {'classification': 'unknown', 'status': 200, 'final_url': 'https://read.example/', 'redirect_chain': []}
+        with patch('app.main.validate_url', return_value='https://read.example/'), patch('app.main.scan_website', return_value=result):
+            self.assertEqual(self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})[0], 201)
+        status, payload = self.request('GET', f"/api/v1/businesses/{business['id']}/websites/scan")
+        self.assertEqual(status, 200)
+        self.assertEqual(payload['classification'], 'unknown')
+        self.assertEqual(payload['business_id'], business['id'])
+
+    def test_unsafe_scan_is_rejected_without_fetching(self):
+        _, business = self.request('POST', '/api/v1/businesses', {'name': 'Private Scan', 'website': 'http://127.0.0.1/'})
+        with patch('app.main.scan_website') as scanner:
+            status, payload = self.request('POST', f"/api/v1/businesses/{business['id']}/websites/scan", {})
+        self.assertEqual(status, 400); self.assertEqual(payload['error'], 'unsafe_url'); scanner.assert_not_called()
+
+
+if __name__ == '__main__':
+    unittest.main()

apps/api/tests/test_phase8_website.py

@@ -0,0 +1,59 @@
+import unittest
+from unittest.mock import patch
+
+from app.website_scanner import classify_website, validate_url, scan_website
+
+
+class WebsiteScannerTests(unittest.TestCase):
+    def test_rejects_unsafe_schemes_and_addresses(self):
+        for url in ('file:///etc/passwd', 'ftp://example.com', 'http://127.0.0.1/', 'http://169.254.169.254/latest/meta-data'):
+            with self.subTest(url=url):
+                with self.assertRaises(ValueError):
+                    validate_url(url)
+
+    def test_classification_is_conservative(self):
+        self.assertEqual(classify_website(200, 'https://example.com', '<html><title>Acme</title><h1>Welcome</h1></html>'), 'healthy')
+        self.assertEqual(classify_website(404, 'https://example.com', ''), 'broken')
+        self.assertEqual(classify_website(200, 'https://example.com', '<html>under construction - coming soon</html>'), 'under_construction')
+        self.assertEqual(classify_website(200, 'https://example.com', '<html>domain for sale parking</html>'), 'parked')
+        self.assertEqual(classify_website(200, 'https://example.com', '<html>placeholder page</html>'), 'placeholder')
+        self.assertEqual(classify_website(301, 'https://example.com', ''), 'redirect_only')
+        self.assertEqual(classify_website(None, 'https://example.com', '', error='timeout'), 'blocked')
+        self.assertEqual(classify_website(None, 'https://example.com', ''), 'unknown')
+
+    def test_metadata_and_signals_are_extracted_from_fixture(self):
+        html = '''<!doctype html><html lang="en"><head><title>Acme</title>
+        <meta name="description" content="Solar experts"><meta name="viewport" content="width=device-width, initial-scale=1">
+        <meta name="generator" content="WordPress 6"><link rel="stylesheet" href="/style.css"></head>
+        <body><h1>Acme Solar</h1><h2>Contact us</h2><a href="/contact">Contact</a><a href="mailto:hi@acme.test">Email</a>
+        <a href="https://wa.me/27123456789">WhatsApp</a><a href="https://facebook.com/acme">Facebook</a><form action="/contact"></form></body></html>'''
+        with patch('app.website_scanner._fetch', return_value={
+            'status': 200, 'final_url': 'https://acme.test/', 'redirect_chain': [],
+            'body': html.encode(), 'content_type': 'text/html', 'elapsed_ms': 12, 'tls': True, 'certificate_status': 'valid'
+        }):
+            result = scan_website('https://acme.test/')
+        self.assertEqual(result['classification'], 'healthy')
+        self.assertEqual(result['title'], 'Acme')
+        self.assertEqual(result['meta_description'], 'Solar experts')
+        self.assertEqual(result['language'], 'en')
+        self.assertEqual(result['headings'], ['Acme Solar', 'Contact us'])
+        self.assertTrue(result['responsive_signal'])
+        self.assertIn('wordpress', result['cms_hints'])
+        self.assertTrue(result['contact_page_signal'])
+        self.assertTrue(result['form_signal'])
+        self.assertTrue(result['mail_signal'])
+        self.assertTrue(result['whatsapp_signal'])
+        self.assertTrue(result['social_signal'])
+        self.assertEqual(result['certificate_status'], 'valid')
+
+    def test_limits_are_reported_not_as_missing_contact_data(self):
+        with patch('app.website_scanner._fetch', side_effect=ValueError('response_too_large')):
+            result = scan_website('https://example.com')
+        self.assertEqual(result['classification'], 'blocked')
+        self.assertEqual(result['error_code'], 'response_too_large')
+        self.assertIsNone(result['contact_page_signal'])
+        self.assertIsNone(result['mail_signal'])
+
+
+if __name__ == '__main__':
+    unittest.main()