root/MarketingTool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add ssrf-safe website analysis

Browse Source
This commit is contained in:
Marco0300
2026-09-03 11:07:34 +02:00
parent f1efe39de4
commit fb89a28f2c
13 changed files with 454 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/api/README.md
+14 -4
View File
@@ -1,6 +1,6 @@
# Prospect Platform API — Phase 7 boundary
# Prospect Platform API — Phase 8 boundary
Dependency-light JSON API for tenant-scoped, **manual** prospect workflows and the Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. Source queries and adapter results must remain auditable and fail closed; the current runtime does not perform network discovery, DNS/website scanning, or outreach.
Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
## Run
@@ -114,8 +114,18 @@ Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class
Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification.
Domain availability is `unknown` unless an explicitly authorized availability provider supplies it. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
Domain availability is `unknown` unless the API reports a result from an authorized provider. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. `nxdomain`, `no_data`, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.
## Phase 8 website-scanning contract
Website scans are tenant-scoped, authenticated, bounded observations. Accept only `http` and `https`; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Resolve and validate the destination immediately before connecting, block loopback/private/link-local/multicast/reserved/cloud-metadata ranges, and repeat the protocol/host/DNS/IP checks for every redirect. Redirects must have a small fixed maximum and cannot escape the allowed protocol policy. DNS rebinding protections must validate the address actually used for the connection.
Apply hard per-scan budgets for wall-clock time, connect/read timeouts, response/body bytes (including decompression), redirects, crawl depth, discovered links, and concurrency/retries. Crawl only explicitly allowed same-policy links; do not submit forms, send cookies or credentials, execute JavaScript, run plugins, or emulate a browser. Unsupported content, a budget exhaustion, timeout, DNS failure, redirect rejection, or partial response is an explicit `unknown`/`blocked`/`partial`/`error` result, never a successful empty page.
Classifications must be conservative, explainable, and derived only from bounded fetched content. They are observations, not proof of ownership, identity, consent, deliverability, security, or contact permission. Persist the normalized URL, redirect chain, response metadata, observed time, scanner/policy/version, applied budgets, cache status, and uncertainty/error reasons; redact response bodies and secrets unless an approved minimal excerpt is required.
Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
## Remaining limitations and production migration work
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Phase 7 production work must add a reviewed PSL update/version policy, bounded DNS workers, TTL-aware cache storage/invalidation, uncertainty-aware MX/NS/TXT schema and tests, association candidate safeguards and permissions, and an authorized availability provider with egress/rate/circuit/retention controls. It must also add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.