add ssrf-safe website analysis

docs/OPERATIONS.md

@@ -61,13 +61,21 @@ Review domain-to-business candidates separately from DNS results. Verify tenant
 
 Availability is `unknown` unless the API reports a result from an authorized provider. Before enabling one, verify current product/legal/security approval, terms owner, allowed tenant scope, request/concurrency and timeout limits, retention/deletion class, health/circuit state, and explicit operational enablement. `nxdomain`, `no_data`, timeout, stale cache, or provider error is not “available.” On provider outage, rate-limit, terms/approval expiry, circuit-open, or conflicting result, fail closed and report unknown/deferred; never purchase, reserve, contact, or retry through an unapproved provider.
 
+## Phase 8 website-scanning operations
+
+Website scans are bounded evidence collection, not browser sessions. The API must accept only `http`/`https`, resolve and validate DNS/IP immediately before connection, block loopback/private/link-local/multicast/reserved/cloud-metadata destinations, and repeat those checks for every redirect. Review redirect chains for protocol or host-policy violations; DNS rebinding or an unsafe resolved address is a blocked scan, not a retry opportunity.
+
+Monitor per-tenant and global scan counts, active concurrency, queue age, total/connect/read timeouts, response/decompressed bytes, redirect and crawl depth/link counts, cache hit/freshness, blocked destinations, DNS failures, partial results, and classification/error rates. Enforce hard budgets across redirects and discovered links. Classifications are conservative observations only: never submit forms, send cookies/credentials, execute JavaScript, or treat an HTTP 200/empty page as proof of a business fact. A scan that hits a limit is explicitly incomplete/unknown.
+
+Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.
+
 ## Phase 4 jobs and live logging
 
 The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.
 
 Creation must use a tenant-scoped idempotency key and request fingerprint. A repeated identical request returns the existing job/attempt; a conflicting payload is rejected. Cancellation is cooperative and race-safe, while retry is an explicit authorized new attempt linked to the original job and must not repeat completed side effects. Do not treat HTTP acceptance as completion, and do not reconstruct history from ephemeral container logs.
 
-The MVP has no SSE handler, durable queue, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.
+There is no SSE handler, durable queue, scan worker/isolation boundary, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.
 
 ## Configuration and deployment
 
@@ -117,7 +125,7 @@ Do not run `docker compose down -v` on a data-bearing environment: it removes th
 
 ## Production migration and scaling path
 
-Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events and domain observations, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker operations, bounded DNS/PSL processing, TTL-aware cache invalidation, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS status/error rates, cache freshness, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and scanners are possible future components—not implicit Compose dependencies and not implemented by this MVP. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.
+Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, domain observations, and scan history/cache, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker and scanner-isolation operations, bounded DNS/PSL/website processing, TTL/freshness-aware cache invalidation, SSRF/DNS-rebinding/redirect-chain tests, hard size/time/crawl budgets, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS/scanner status/error rates, cache freshness, blocked destinations, crawl-budget exhaustion, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and production scanners are possible future components—not implicit Compose dependencies. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.
 
 ## Incident checklist
 

docs/SECURITY.md

@@ -40,6 +40,17 @@ The MVP provides deterministic match suggestions, an explicit human confirmation
 
 No Phase 7 resolver, cache, or availability provider is enabled in the current Compose runtime. Before production, add egress/SSRF controls, provider and PSL update review, retention/deletion handling, monitoring, permission/audit coverage, and failure/rollback tests for all domain observations.
 
+## Phase 8 website-scanning controls
+
+- Scanning is an authenticated, tenant-scoped observation. Allow only `http` and `https`; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Never allow `file:`, `ftp:`, `gopher:`, `data:`, `javascript:`, or equivalent protocol smuggling.
+- Resolve immediately before connection and validate the actual destination address. Block loopback, private, link-local, multicast, reserved, and cloud-metadata ranges for IPv4 and IPv6. Re-run protocol, hostname, DNS, and IP checks on every redirect and protect against DNS rebinding; do not rely on an initial DNS check or an HTTP `Host` header.
+- Enforce hard budgets for connect/read/total time, response and decompressed bytes, retained body size, redirects, crawl depth/links, retries, and concurrency. Abort on budget exhaustion. Do not allow compression, redirects, or retries to bypass limits.
+- Fetch only explicitly allowed content types and links. Never submit forms, send user cookies/credentials/authorization headers, execute JavaScript, run plugins, or make arbitrary subresource requests. Treat fetched content and all TXT/HTML/script text as untrusted input and escape it on display.
+- Classify conservatively: `unknown`, `blocked`, `partial`, `timeout`, and `error` are not empty success and are not negative business facts. A classification is evidence of bounded content only—not ownership, identity, consent, deliverability, security, or permission to contact.
+- Persist scan history/cache with tenant isolation, normalized URL, policy/scanner version, redirect policy, observed time, freshness/expiry, applied budgets, and uncertainty/error metadata. Bound size/retention, redact secrets and response bodies, and invalidate/re-evaluate after policy, DNS, or scanner-version changes. A cache hit must be visibly non-fresh.
+
+No production-grade scanner egress proxy, isolated worker, or durable scan store is supplied by the current Compose runtime. Before enabling scanning in production, add SSRF/DNS-rebinding/redirect-chain tests, egress deny-by-default policy, abuse/rate controls, authenticated history authorization, retention/deletion, monitoring, and incident procedures. Scans must never cause form submission, acquisition, verification, enrichment, or outreach.
+
 ## Phase 5 source security controls
 
 Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
@@ -59,7 +70,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in
 2. **MFA:** require phishing-resistant or TOTP MFA for administrator accounts in production, including the bootstrap admin before granting ongoing administrative access. Define recovery, enrollment, reset, and revocation procedures; do not treat a password-only bootstrap as production-ready.
 3. **Authentication and authorization:** enforce authorization server-side on every protected route, including every child-record, note, pipeline, and audit route. Rotate/regenerate sessions at login and privilege changes, expire idle/absolute sessions, revoke on logout/password reset, and test tenant isolation.
 4. **Cookies and CSRF:** use `HttpOnly`, `Secure` (production HTTPS), and an appropriate `SameSite` policy. Browser state-changing endpoints require CSRF tokens (or a rigorously reviewed equivalent); do not rely on CORS or cookie flags alone.
-5. **SSRF and future scanners:** no scanner is enabled in this release. If a future approved feature fetches a URL, allow only `http`/`https`, validate DNS/IP targets, block loopback/private/link-local/cloud-metadata ranges after resolution, limit redirects, enforce size/time limits, and re-check each redirect.
+5. **SSRF and scanner production hardening:** the Phase 8 scanner is bounded and conservative, but production still requires an egress proxy/isolation boundary, DNS-rebinding and redirect-chain regression tests, deny-by-default network policy, abuse controls, and durable scan-history/cache retention. Allow only `http`/`https`; never submit forms or execute JavaScript.
 6. **Input/output safety:** validate schema and content types, bound request and note/evidence sizes, parameterize database queries, escape output, and reject unsafe provenance URLs or markup. Treat operator-entered notes and sources as untrusted data.
 7. **Audit and retention:** the current audit/activity behavior is an MVP trail, not an immutable compliance log. Define append-only guarantees, retention, redaction/deletion rules, access controls, alerting, and export procedures before production.
 8. **Secrets:** inject production secrets from a secret manager or orchestrator secret store. Do not place them in images, Compose files, source, CI logs, or committed `.env` files. Remove bootstrap variables after first-run provisioning.