add public contact extraction

This commit is contained in:
Marco0300
2026-09-03 11:15:54 +02:00
parent fb89a28f2c
commit 89eb7e07e6
14 changed files with 402 additions and 7 deletions
+13
View File
@@ -51,6 +51,18 @@ No Phase 7 resolver, cache, or availability provider is enabled in the current C
No production-grade scanner egress proxy, isolated worker, or durable scan store is supplied by the current Compose runtime. Before enabling scanning in production, add SSRF/DNS-rebinding/redirect-chain tests, egress deny-by-default policy, abuse/rate controls, authenticated history authorization, retention/deletion, monitoring, and incident procedures. Scans must never cause form submission, acquisition, verification, enrichment, or outreach.
## Phase 9 public official-site contact extraction controls
- Extraction is passive and suggestion-only. Scope it to an explicitly approved/public official-site origin and bounded same-site contact/about pages; never use it as general search, unrestricted crawling, identity verification, or enrichment.
- Apply the Phase 8 SSRF, redirect, content-type, timeout, byte, page, URL, candidate, and concurrency limits. Fail closed on disabled/unapproved scope, limit exhaustion, partial content, blocked/unsafe destinations, or resolver errors, with an explicit status and uncertainty reason rather than an empty success.
- Parse only permitted public HTML/visible contact values and `mailto:` links. Never submit forms, execute JavaScript, send credentials/cookies, probe SMTP or SMTP `VRFY`/`EXPN`, send validation email, or make any outbound contact. Treat HTML, attributes, scripts, and extracted text as untrusted input.
- Retain provenance for every candidate: source/page URL and context, extraction method, observed time, extractor/policy version, confidence algorithm/version, and uncertainty reasons. Confidence is a triage signal—not ownership, consent, deliverability, or contact permission.
- Keep `syntax_valid`/`syntax_invalid` separate from role classification and free-mail classification. Role/person/unknown and free-mail/business-domain/unknown are labels only. MX/DNS must remain independently uncertain with resolver/source, observed time, TTL/freshness, and explicit `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error` states; no MX outcome proves deliverability.
- Exclude asset/file-name addresses, script/style/comment text, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party content. Suppression matching must happen before persistence, response, export, cache, or review queue insertion; a match is immutable do-not-contact until an authorized suppression change, and suppression always wins.
- Minimize and protect contact values and lineage. Bound retention for raw/extracted values, provenance, MX/DNS observations, caches, and audit records; redact full addresses and page content from logs where a safe hash/identifier suffices. Tenant predicates and deletion handling apply to every result, cache, export, and audit read.
No SMTP probing or outreach capability is permitted by this phase. A public address remains an unverified observation requiring human review and separate future product/legal/security approval before any contact workflow could exist.
## Phase 5 source security controls
Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
@@ -80,6 +92,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in
12. **Live delivery:** SSE, if introduced, must authenticate before opening the stream, enforce tenant scope on every replay query, bound event/backlog size, support `Last-Event-ID`/cursor replay, send heartbeats, and provide polling fallback. Treat event-stream connections as untrusted clients and avoid cross-tenant timing/detail leaks.
13. **Worker boundary:** the current SQLite/in-process MVP is not durable or horizontally safe. A production worker migration requires reviewed queue semantics, leases, visibility timeouts, dead-letter handling, concurrency limits, cancellation races, metrics, and deployment isolation. Redis/Celery are not implemented today.
14. **Domain intelligence:** implement and security-review PSL pin/update handling, bounded DNS resolution, TTL-aware cache isolation/invalidation, uncertainty-preserving MX/NS/TXT parsing, tenant-scoped association decisions, and an authorized availability provider before exposing any live domain status. No DNS response may authorize acquisition, ownership, outreach, or verification.
15. **Official-site contacts:** production extraction requires approved official-site scoping, false-positive fixtures, syntax/role/free-mail classification tests, explicit MX uncertainty handling, pre-persistence suppression tests, hard limit/retention/deletion controls, provenance/audit coverage, and a permanent prohibition on SMTP probing, validation mail, and outreach.
## Source and contact policy