add public contact extraction

This commit is contained in:
Marco0300
2026-09-03 11:15:54 +02:00
parent fb89a28f2c
commit 89eb7e07e6
14 changed files with 402 additions and 7 deletions
+13 -1
View File
@@ -1,4 +1,4 @@
# Prospect Platform API — Phase 8 boundary
# Prospect Platform API — Phase 9 boundary
Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
@@ -68,6 +68,8 @@ Implementations should expose source/query/job state without leaking raw payload
The business detail includes the supported child collections: `contacts`, `domains`, `websites`, `evidence`, `pipeline`, and `notes`. The child collection routes are:
- `POST /api/v1/businesses/{id}/contacts` — manually add a contact; suppression matching marks a matching contact as do-not-contact.
- `GET /api/v1/businesses/{id}/contacts/extract` — read the tenant-scoped Phase 9 official-site extraction projection.
- `POST /api/v1/businesses/{id}/contacts/extract` — extract bounded public contacts from the approved official site/same-site pages and persist provenance-bearing observations; the request is passive and must not probe SMTP or send outreach. Suppression matching marks matches `suppressed`/`do_not_contact` and remains authoritative.
- `POST /api/v1/businesses/{id}/domains` — manually add a domain observation.
- `POST /api/v1/businesses/{id}/websites` — manually add a website observation/classification.
- `POST /api/v1/businesses/{id}/evidence` — manually add evidence with its kind, claim, and source URL/reference. This records provenance supplied by the operator; it does not scan or independently verify the URL.
@@ -126,6 +128,16 @@ Classifications must be conservative, explainable, and derived only from bounded
Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
## Phase 9 official-site contact extraction contract
The optional Phase 9 extractor is a passive, authenticated, tenant-scoped observation of a business's approved/public official-site origin. It may inspect bounded HTML and same-site contact/about pages only; it must not become a search engine, unrestricted crawler, or arbitrary URL fetcher. The extractor must use the existing SSRF-safe URL, redirect, content-type, and resource-budget controls, and must never submit forms, execute target JavaScript, use credentials/cookies, probe SMTP, issue SMTP `VRFY`/`EXPN`, send validation mail, or perform outreach.
For each candidate, return/store the normalized address only with provenance (official-site/page URL, page or DOM context, extraction method, observed time, extractor/policy version) and an explainable confidence/reason list. Preserve uncertainty rather than inventing facts. `syntax_valid`/`syntax_invalid` is a parser outcome only. Role classification (`role`/`person`/`unknown`) and free-mail classification (`free_mail`/`business_domain`/`unknown`) are independent review labels; they do not prove identity, consent, ownership, or deliverability. MX/DNS is a separate observation with resolver/source, observed time, TTL/freshness where available, and one of `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`; MX absence or failure remains unknown and must never be treated as invalid or undeliverable.
Exclude false positives before persistence and response: values in scripts/styles/comments or asset URLs/file names, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party pages. Enforce hard limits for total extraction time, pages/URLs, redirects, response and retained bytes, candidates per page/request, and concurrency. Suppression matching is server-side and tenant-scoped, before storing, returning, exporting, or presenting a candidate; suppressed contacts are marked do-not-contact and cannot be revived by a later classification or review action. Retention must be explicit and bounded for extracted values, page provenance, DNS/MX observations, caches, and audit records; logs must not contain full contact payloads when a redacted identifier is sufficient.
Extraction results are suggestions only and do not create a send/contact capability. The API exposes no SMTP-probe, validation-message, outreach, or campaign endpoint. If the feature is disabled, unapproved, over limit, blocked, or uncertain, fail closed with an explicit status/reason rather than an empty successful result. The current MVP remains pilot-only until extraction limits, suppression enforcement, retention/deletion jobs, provenance/audit coverage, and tenant-isolation tests are production hardened.
## Remaining limitations and production migration work
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.