Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.

Run

From this directory:

python3 app/main.py
# listens on http://127.0.0.1:8000
python3 -m unittest discover

Set PROSPECT_API_PORT or pass --port; set PROSPECT_API_DB or pass --db to override the default prospects.db.

Endpoint contract

All protected endpoints require the server-side session cookie. Every query is constrained by the authenticated user's organization_id; IDs from another tenant behave as not found and must not disclose whether a record exists.

Phase 7 domain routes (all tenant-scoped) are POST /api/v1/businesses/{id}/domains/check, GET /api/v1/businesses/{id}/domains/check?domain=..., GET /api/v1/domain-checks, GET /api/v1/businesses/{id}/domain-candidates, and POST /api/v1/businesses/{id}/domain-candidates/check-availability. The current implementation is intentionally conservative: a successful address lookup is reported as ok, unresolved/empty results as unknown, and an availability check returns unknown/not_configured because no provider is enabled. Treat these as observation states, not ownership or availability claims.

Phase 4 jobs and live logging (target contract)

The intended job resource has a stable job_id, tenant/creator metadata, operation/payload fingerprint, status, attempt, timestamps, cancellation state, and terminal error/result metadata. Its lifecycle is queued → running → exactly one terminal state: succeeded, failed, or cancelled. State transitions and worker messages must be persisted transactionally with tenant and job identifiers; terminal jobs are immutable except for controlled retention/redaction.

Idempotency and events

Mutating job creation should require an idempotency key (for example, an Idempotency-Key header). The key must be scoped to the authenticated tenant and operation, stored with a request fingerprint, and return the original job/result for an exact replay. Reuse with a different payload must be rejected rather than creating a second job. Idempotency must cover side effects, not merely the HTTP response.

Live events should be append-only and ordered per job with a durable integer sequence/cursor. A consumer can resume from the last acknowledged sequence, tolerate duplicate delivery, and detect gaps. Events must contain only safe operational detail; do not persist passwords, session cookies, API keys, or unnecessary prospect/contact data.

Control and delivery semantics

The MVP job routes are POST /api/v1/jobs, GET /api/v1/jobs, GET /api/v1/jobs/{id}, GET /api/v1/jobs/{id}/events, POST /api/v1/jobs/{id}/cancel, and POST /api/v1/jobs/{id}/retry. They are tenant-scoped and backed by SQLite persistence. Cancellation is best effort and must be race-safe with workers; retry creates a new attempt/lineage and must not repeat completed side effects. Polling supports a bounded cursor and backoff in the web monitor. An SSE endpoint may stream the same persisted sequence events with Last-Event-ID, heartbeats, disconnect/reconnect support, and polling fallback, but SSE is not implemented in this MVP.

The current runtime has no durable job queue and no Redis/Celery integration. Its in-process/SQLite worker is single-instance, non-durable across process loss, and unsuitable for horizontal scaling or guaranteed execution; it is pilot-only.

Phase 5 source adapter contract

Phase 5 treats a source as a registered, reviewable capability rather than an arbitrary URL or scraper. Every adapter contract must identify:

a stable adapter/source ID and version, owner, permitted purpose, and terms/robots contact;
accepted discovery-query fields, result schema, provenance fields, and validation/error behavior;
request and concurrency rate limits, retry/backoff rules, timeout/size bounds, and a retention class for raw source records;
health signals and circuit-breaker states (closed, open, half-open), including fail-closed behavior when unhealthy; and
an explicit dry_run mode that validates and plans work without contacting a source or writing prospect facts.

The initial safe adapters are csv and manual_reference. CSV input may be parsed and previewed; a manual reference records operator-supplied source identity, citation/reference, observed value, and timestamp. Neither adapter independently verifies a source or authorizes outreach. Raw source records should be retained immutably enough to reproduce the normalized result, with tenant/source/query identifiers, capture time, adapter version, and redaction/retention metadata; never retain secrets or unnecessary personal data.

A source registry entry must include its terms owner, approval status/expiry, allowed tenants or scopes, rate-limit policy, retention class, and health/circuit policy. A discovery query is bounded and tenant-scoped, and its execution mode must be explicit (dry_run by default). A live network source is not implemented and must be rejected unless product, legal, and security approval is recorded and operations explicitly enables the registered adapter. No query, job acceptance, or successful parse may be described as network discovery.

Phase 5 source controls (contract, not current live routes)

Implementations should expose source/query/job state without leaking raw payloads across tenants, including source approval, terms, rate-limit, retention, health, and circuit-open reason. On rate-limit, terms, approval, or circuit failure, return a safe non-live outcome and preserve an audit event; do not silently retry against another source. Dry-run must be side-effect-free with respect to external sources and prospect facts. These are Phase 5 design requirements; the current MVP has no network adapter or discovery endpoint.

Health and workspace

GET /api/v1/health/live — unauthenticated liveness check.
GET /api/v1/auth/me — current authenticated user and tenant.
GET /api/v1/dashboard/summary — tenant-scoped counts and score summary.

Prospect and child intelligence records

GET /api/v1/businesses — paginated tenant list. Supports bounded page/page_size, text q, score_min/score_max, website_class, and pipeline_stage filters, with stable ordering. Responses contain items, page, page_size, and has_next; callers must not assume all records are returned.
POST /api/v1/businesses — manual business creation. name is required; website, email, phone, description, and other explicitly supported intake fields are optional. Normalization, scoring, deduplication, and suppression are enforced server-side.
GET /api/v1/businesses/{id} — tenant-scoped business detail, including the permitted child intelligence/evidence projection and current pipeline/review context. The business detail includes the supported child collections: contacts, domains, websites, evidence, pipeline, and notes. The child collection routes are:
POST /api/v1/businesses/{id}/contacts — manually add a contact; suppression matching marks a matching contact as do-not-contact.
GET /api/v1/businesses/{id}/contacts/extract — read the tenant-scoped Phase 9 official-site extraction projection.
POST /api/v1/businesses/{id}/contacts/extract — extract bounded public contacts from the approved official site/same-site pages and persist provenance-bearing observations; the request is passive and must not probe SMTP or send outreach. Suppression matching marks matches suppressed/do_not_contact and remains authoritative.
POST /api/v1/businesses/{id}/domains — manually add a domain observation.
POST /api/v1/businesses/{id}/websites — manually add a website observation/classification.
POST /api/v1/businesses/{id}/evidence — manually add evidence with its kind, claim, and source URL/reference. This records provenance supplied by the operator; it does not scan or independently verify the URL.
POST /api/v1/businesses/{id}/notes — add a manual note.

Child records are subordinate to their parent business. A child ID is never sufficient authorization: the API verifies both the child ID and the parent business's organization. Do not use a missing source or a score as proof that a website or DNS check occurred.

Pipeline, verification, and audit behavior

POST /api/v1/businesses/{id}/pipeline — record an allowed human workflow-stage transition, with server-side validation and an audit event.
POST /api/v1/businesses/{id}/verify — record the permitted human verification action and its audit event; it does not perform an external check.
Each business detail response returns the tenant-scoped child collections and current verification/pipeline context. Audit events are retained in the workspace audit log; the detail projection includes the relevant mutation context where supported.

Pipeline state and verification are review metadata, not outreach authorization. Suppression always wins, and the API exposes no send/contact endpoint. State changes, child records, and notes are human-entered; they do not trigger discovery, scanning, or outbound messaging.

Existing safety and intake routes

POST /api/v1/suppressions — add {kind: email|domain|phone, value: ...} for the current tenant. Future matching business creation is blocked.
POST /api/v1/imports/preview — preview {rows: [...]} without writing; reports accepted, duplicates, suppressed, and normalized rows. It is not an import/persistence endpoint.

All SQL uses parameters and all responses are JSON. Scores include score_version and score_factors for traceability. Provenance is supplied by the operator/source record; the MVP does not validate external sources or independently refresh evidence.

Phase 6 normalization and duplicate-review contract

The server is the normalization authority. SA phone input is canonicalized using explicit +27 context for local 0 numbers, with punctuation/spacing removed while retaining a display/original value. Location input retains the raw observation and derives a comparison form plus country/province/municipality/city tokens; missing or ambiguous locality must remain missing/ambiguous, not guessed. The normalization/schema version must be stored with derived values so reprocessing is deterministic.

Exact duplicate keys are deterministic. GET /api/v1/businesses/{id}/matches compares only active businesses in the authenticated organization and produces a sorted, deterministic score, score version, and explainable reasons. Its default candidate cutoff is 0.72; policy bands are >=0.90 strong suggestion, 0.75–0.8999 review suggestion, and <0.75 no suggestion. Fuzzy comparison is suggestion-only and there is no automatic merge at any score. POST /api/v1/businesses/{id}/merge requires an authenticated mutating-role user and an explicit target; the web client also requires a human confirmation. A production merge permission and server-verifiable confirmation token remain hardening work.

Before a confirmed merge, the current route persists a tenant-scoped merge_history snapshot of the source business and child rows (business_identifiers, contacts, domains, websites, evidence, pipeline_entries, interactions, and notes), plus child IDs/counts, actor, and timestamp. It re-parents those children to the target without deleting source records; GET /api/v1/merge-history reads the ledger and POST /api/v1/merge-history/{id}/reverse restores the source/child links. Candidate queries, merges, snapshots, reversal, and history reads apply the organization predicate; a cross-tenant ID behaves as not found. Audit events record merge and reversal actions.

Remaining limitations: the snapshot currently focuses on the source graph rather than a full two-parent conflict snapshot; the merge route does not yet enforce a dedicated merge permission or cryptographically bound confirmation payload; and preservation/conflict semantics need production-grade transactional and concurrency tests. It must not claim that normalization proves identity or that deduplication performs discovery or outreach.

Pagination and filtering rules

List and child-record endpoints are deliberately bounded. For business lists, use page (starting at 1) and page_size within the server-enforced maximum; invalid values are rejected rather than allowing an unbounded query. Supported filters are applied inside the tenant-scoped query before pagination: q, score_min, score_max, website_class, and pipeline_stage. The UI's page and filter controls are convenience clients, not authorization controls. A filtered page is not a count of the entire unfiltered tenant unless the response explicitly says so.

Phase 7 domain-intelligence contract

Domain observations are tenant-scoped, provenance-bearing inputs. Registrable-domain normalization must use a pinned/versioned PSL rather than a last-two-label heuristic. Preserve the original value and normalized form; return unknown when the PSL cannot classify a value (including an unknown/private suffix, public suffix, malformed name, single-label name, localhost, or IP literal). IDN/punycode and case/label handling must be deterministic and must not turn a subdomain into an independent business identity.

DNS checks must expose an explicit status—not_checked, pending, resolved, nxdomain, no_data, timeout, servfail, blocked, or error—and never collapse failure or absence into a negative business fact. MX, NS, and TXT are independently uncertain observations: retain record type, normalized response, resolver/source, observed time, TTL if supplied, truncation/partial indicators, and error/uncertainty reason. A missing MX does not prove that email is unavailable; an NS result does not prove control; TXT content does not prove ownership.

Any DNS cache must be bounded, tenant-safe, keyed by normalized name/type/class and resolver policy, and TTL-aware. Do not extend authority beyond the received TTL; expose observed_at, expires_at/freshness, and stale or refresh state. A cache hit is not a fresh lookup, and resolver policy/PSL version changes require invalidation or re-evaluation. There is no DNS resolver or cache service in the current runtime.

Association confidence is a separate, explainable, versioned review signal—not DNS status, duplicate score, or identity proof. Candidate-domain generation must apply the organization predicate before comparison, reject public-suffix-only/malformed/IP candidates, avoid automatic attachment, and flag shared/parked/wildcard/homograph/sibling-subdomain and conflicting-evidence cases. Human accept/reject decisions, reasons, provenance, and confidence version must be auditable; no candidate may authorize outreach or verification.

Domain availability is unknown unless the API reports a result from an authorized provider. The provider must be registered with current product/legal/security approval, terms owner, tenant scope, rate/concurrency limits, retention, health/circuit policy, and operational enablement checked at execution time. nxdomain, no_data, timeout, stale cache, or provider failure is not an availability result. Never purchase, reserve, contact, or report a domain as available from DNS alone. These safeguards are contract requirements only; no live availability provider is implemented.

Phase 8 website-scanning contract

Website scans are tenant-scoped, authenticated, bounded observations. Accept only http and https; reject credentials, unsupported schemes, malformed/localhost/single-label hosts, and disallowed IP literals. Resolve and validate the destination immediately before connecting, block loopback/private/link-local/multicast/reserved/cloud-metadata ranges, and repeat the protocol/host/DNS/IP checks for every redirect. Redirects must have a small fixed maximum and cannot escape the allowed protocol policy. DNS rebinding protections must validate the address actually used for the connection.

Apply hard per-scan budgets for wall-clock time, connect/read timeouts, response/body bytes (including decompression), redirects, crawl depth, discovered links, and concurrency/retries. Crawl only explicitly allowed same-policy links; do not submit forms, send cookies or credentials, execute JavaScript, run plugins, or emulate a browser. Unsupported content, a budget exhaustion, timeout, DNS failure, redirect rejection, or partial response is an explicit unknown/blocked/partial/error result, never a successful empty page.

Classifications must be conservative, explainable, and derived only from bounded fetched content. They are observations, not proof of ownership, identity, consent, deliverability, security, or contact permission. Persist the normalized URL, redirect chain, response metadata, observed time, scanner/policy/version, applied budgets, cache status, and uncertainty/error reasons; redact response bodies and secrets unless an approved minimal excerpt is required.

Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose observed_at and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.

Phase 9 official-site contact extraction contract

The optional Phase 9 extractor is a passive, authenticated, tenant-scoped observation of a business's approved/public official-site origin. It may inspect bounded HTML and same-site contact/about pages only; it must not become a search engine, unrestricted crawler, or arbitrary URL fetcher. The extractor must use the existing SSRF-safe URL, redirect, content-type, and resource-budget controls, and must never submit forms, execute target JavaScript, use credentials/cookies, probe SMTP, issue SMTP VRFY/EXPN, send validation mail, or perform outreach.

For each candidate, return/store the normalized address only with provenance (official-site/page URL, page or DOM context, extraction method, observed time, extractor/policy version) and an explainable confidence/reason list. Preserve uncertainty rather than inventing facts. syntax_valid/syntax_invalid is a parser outcome only. Role classification (role/person/unknown) and free-mail classification (free_mail/business_domain/unknown) are independent review labels; they do not prove identity, consent, ownership, or deliverability. MX/DNS is a separate observation with resolver/source, observed time, TTL/freshness where available, and one of not_checked, resolved, nxdomain, no_data, timeout, servfail, blocked, or error; MX absence or failure remains unknown and must never be treated as invalid or undeliverable.

Exclude false positives before persistence and response: values in scripts/styles/comments or asset URLs/file names, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party pages. Enforce hard limits for total extraction time, pages/URLs, redirects, response and retained bytes, candidates per page/request, and concurrency. Suppression matching is server-side and tenant-scoped, before storing, returning, exporting, or presenting a candidate; suppressed contacts are marked do-not-contact and cannot be revived by a later classification or review action. Retention must be explicit and bounded for extracted values, page provenance, DNS/MX observations, caches, and audit records; logs must not contain full contact payloads when a redacted identifier is sufficient.

Extraction results are suggestions only and do not create a send/contact capability. The API exposes no SMTP-probe, validation-message, outreach, or campaign endpoint. If the feature is disabled, unapproved, over limit, blocked, or uncertain, fail closed with an explicit status/reason rather than an empty successful result. The current MVP remains pilot-only until extraction limits, suppression enforcement, retention/deletion jobs, provenance/audit coverage, and tenant-isolation tests are production hardened.

Remaining limitations and production migration work

SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.

README.md Unescape Escape

Prospect Platform API — Phase 9 boundary

Run