root/MarketingTool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add public contact extraction

Browse Source
This commit is contained in:
Marco0300
2026-09-03 11:15:54 +02:00
parent fb89a28f2c
commit 89eb7e07e6
14 changed files with 402 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/OPERATIONS.md
+12 -1
View File
@@ -69,6 +69,16 @@ Monitor per-tenant and global scan counts, active concurrency, queue age, total/
Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.
## Phase 9 official-site contact extraction operations
Operate contact extraction as passive observation of an approved/public official-site origin, limited to bounded same-site contact/about pages. Before enabling it, verify tenant scope, official-site approval, extractor/policy version, retention class, suppression source, and configured hard limits for pages/URLs, redirects, bytes, time, candidates, and concurrency. Do not add arbitrary URLs or search results to the scope.
Review each result with its source/page URL and context, extraction method, observed time, confidence/reasons, syntax status, role/free-mail labels, and independent MX/DNS status/freshness. Treat `not_checked`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error` as uncertainty—not invalidity or non-deliverability. Syntax, role, free-mail, and MX labels never authorize contact. Values found in scripts/styles/assets/file names, examples/placeholders, tracking addresses, malformed schemes, or third-party content are false positives and must be excluded or quarantined.
Verify suppression matching before persistence, response, cache, export, or review-queue insertion. Suppressed contacts remain **do not contact**, regardless of later confidence, classification, syntax, MX, pipeline, or verification changes. Monitor extraction attempts, pages/bytes/candidates, limit hits, blocked destinations, parse errors, false-positive exclusions, suppression matches, cache freshness, retention/deletion jobs, and provenance/audit failures. On a suppression or provenance failure, stop the affected write path and investigate; do not retry blindly.
There is no SMTP probing, SMTP `VRFY`/`EXPN`, validation email, outreach worker, campaign queue, or follow-up action. Never contact a discovered address. If extraction is disabled, unapproved, out of budget, or uncertain, report deferred/blocked/unknown with the reason. Retain only the minimum value and lineage for the approved retention period; redact addresses and page content from routine logs.
## Phase 4 jobs and live logging
The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.
@@ -90,7 +100,8 @@ Before deployment:
3. Restrict host/network exposure at the ingress/firewall.
4. Verify both unauthenticated health checks and review logs for unexpected errors, cross-tenant errors, or sensitive data.
5. Exercise tenant-scoped list/detail/child routes with bounded pagination and filters, and verify that notes/pipeline changes appear in the intended tenant's audit trail only.
6. Record the image digest and configuration revision for rollback.
6. If Phase 9 is enabled, run official-site fixtures covering provenance/confidence, role and free-mail labels, syntax failures, every MX/DNS uncertainty state, false-positive exclusions, limit exhaustion, retention/deletion, and suppression-before-persistence/response/export. Confirm no SMTP or outreach network activity.
7. Record the image digest and configuration revision for rollback.
## Data, backups, and retention