add public contact extraction
This commit is contained in:
@@ -51,6 +51,18 @@ No Phase 7 resolver, cache, or availability provider is enabled in the current C
|
||||
|
||||
No production-grade scanner egress proxy, isolated worker, or durable scan store is supplied by the current Compose runtime. Before enabling scanning in production, add SSRF/DNS-rebinding/redirect-chain tests, egress deny-by-default policy, abuse/rate controls, authenticated history authorization, retention/deletion, monitoring, and incident procedures. Scans must never cause form submission, acquisition, verification, enrichment, or outreach.
|
||||
|
||||
## Phase 9 public official-site contact extraction controls
|
||||
|
||||
- Extraction is passive and suggestion-only. Scope it to an explicitly approved/public official-site origin and bounded same-site contact/about pages; never use it as general search, unrestricted crawling, identity verification, or enrichment.
|
||||
- Apply the Phase 8 SSRF, redirect, content-type, timeout, byte, page, URL, candidate, and concurrency limits. Fail closed on disabled/unapproved scope, limit exhaustion, partial content, blocked/unsafe destinations, or resolver errors, with an explicit status and uncertainty reason rather than an empty success.
|
||||
- Parse only permitted public HTML/visible contact values and `mailto:` links. Never submit forms, execute JavaScript, send credentials/cookies, probe SMTP or SMTP `VRFY`/`EXPN`, send validation email, or make any outbound contact. Treat HTML, attributes, scripts, and extracted text as untrusted input.
|
||||
- Retain provenance for every candidate: source/page URL and context, extraction method, observed time, extractor/policy version, confidence algorithm/version, and uncertainty reasons. Confidence is a triage signal—not ownership, consent, deliverability, or contact permission.
|
||||
- Keep `syntax_valid`/`syntax_invalid` separate from role classification and free-mail classification. Role/person/unknown and free-mail/business-domain/unknown are labels only. MX/DNS must remain independently uncertain with resolver/source, observed time, TTL/freshness, and explicit `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error` states; no MX outcome proves deliverability.
|
||||
- Exclude asset/file-name addresses, script/style/comment text, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party content. Suppression matching must happen before persistence, response, export, cache, or review queue insertion; a match is immutable do-not-contact until an authorized suppression change, and suppression always wins.
|
||||
- Minimize and protect contact values and lineage. Bound retention for raw/extracted values, provenance, MX/DNS observations, caches, and audit records; redact full addresses and page content from logs where a safe hash/identifier suffices. Tenant predicates and deletion handling apply to every result, cache, export, and audit read.
|
||||
|
||||
No SMTP probing or outreach capability is permitted by this phase. A public address remains an unverified observation requiring human review and separate future product/legal/security approval before any contact workflow could exist.
|
||||
|
||||
## Phase 5 source security controls
|
||||
|
||||
Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
|
||||
@@ -80,6 +92,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in
|
||||
12. **Live delivery:** SSE, if introduced, must authenticate before opening the stream, enforce tenant scope on every replay query, bound event/backlog size, support `Last-Event-ID`/cursor replay, send heartbeats, and provide polling fallback. Treat event-stream connections as untrusted clients and avoid cross-tenant timing/detail leaks.
|
||||
13. **Worker boundary:** the current SQLite/in-process MVP is not durable or horizontally safe. A production worker migration requires reviewed queue semantics, leases, visibility timeouts, dead-letter handling, concurrency limits, cancellation races, metrics, and deployment isolation. Redis/Celery are not implemented today.
|
||||
14. **Domain intelligence:** implement and security-review PSL pin/update handling, bounded DNS resolution, TTL-aware cache isolation/invalidation, uncertainty-preserving MX/NS/TXT parsing, tenant-scoped association decisions, and an authorized availability provider before exposing any live domain status. No DNS response may authorize acquisition, ownership, outreach, or verification.
|
||||
15. **Official-site contacts:** production extraction requires approved official-site scoping, false-positive fixtures, syntax/role/free-mail classification tests, explicit MX uncertainty handling, pre-persistence suppression tests, hard limit/retention/deletion controls, provenance/audit coverage, and a permanent prohibition on SMTP probing, validation mail, and outreach.
|
||||
|
||||
## Source and contact policy
|
||||
|
||||
|
||||
Reference in New Issue
Block a user