Prospect Intelligence Platform

A safety-first Phase 6 design/implementation boundary for manual, evidence-led prospect qualification and controlled source ingestion. The current runtime remains a manual vertical slice: it stores tenant-owned businesses and child intelligence records, keeps provenance with each evidence item, supports a review pipeline, and records operationally relevant changes. Phase 6 defines deterministic South African normalization and deduplication review semantics in addition to the Phase 5 source controls; it does not enable network discovery. Automated outreach is disabled, and no live source may be enabled without explicit approval.

Included

Dependency-free Python/SQLite API under apps/api.
Tenant-scoped business detail APIs with child intelligence/evidence records, provenance fields, notes, pipeline state, and audit history.
Server-side normalization, conservative website classification, exact deduplication, versioned scoring, and suppression checks. Phase 6 documents the SA phone/location canonical forms and the review-only fuzzy-match contract.
Bounded list pagination and server-side filters so a tenant cannot request an unbounded prospect collection.
Responsive static dashboard under apps/web with authenticated explorer filters, paginated results, detail review, manual intake, notes/pipeline context, evidence provenance, and browser-only CSV preview.
Docker Compose runtime with non-root containers, read-only filesystems, health checks, and a named SQLite data volume.
Browser authentication with server-side sessions and an optional first-run admin bootstrap.
Phase 4 MVP job monitor and SQLite-backed job/event schema/API surface, with the production limitations documented below.
Phase 5 source-ingestion contract: an approved source registry owns adapter terms, rate limits, retention, and health/circuit policy; CSV and manual reference adapters are the safe initial adapters.
Discovery queries are recorded as bounded, auditable intent and dry-run plans. Recording a query does not perform network discovery or imply that results exist.

Current workflow and Phase 4/5 boundary

A permitted workspace member manually creates or reviews a prospect.
The business detail response is the aggregate record for that tenant; related intelligence/evidence rows are returned only through the tenant-scoped detail surface.
Each manually entered intelligence item should retain its source/provenance (for example, source label or URL, observed value, and captured/verified time). Missing provenance is a data-quality limitation, not permission to infer facts.
Members use the pipeline state and notes to coordinate human review. A state change or note is an application event and is included in the record's audit/activity history where exposed by the API.
Suppression remains a hard safety boundary. Suppressed or unreviewed records must not be treated as eligible for contact.

The API applies the organization/tenant boundary server-side to list, detail, child-record, notes, pipeline, and audit reads and writes. Clients must use the returned pagination metadata and follow next/previous links or tokens rather than assuming that one response contains the whole tenant dataset. See apps/api/README.md for the route contract and limits.

Phase 4 jobs/live logging contract

The planned asynchronous contract is: create one tenant-scoped job, return a stable job identifier, and move it through queued → running → a terminal state (succeeded, failed, cancelled). Each accepted request should carry an idempotency key whose scope and request fingerprint prevent duplicate jobs while allowing a safe replay of the original result. A job should persist append-only events with a monotonically increasing per-job sequence number, timestamp, level/type, safe message, and job/tenant identifiers.

Clients should poll a tenant-scoped job status/events endpoint using after_sequence (or an equivalent cursor), with bounded backoff and terminal-state handling. SSE is a planned low-latency delivery option, not a current implementation; polling remains the compatibility fallback. Cancellation and retry must be explicit, authorized controls: cancellation is cooperative and may finish as cancelled or report that the job is already terminal; retry creates a new attempt while retaining the original job/idempotency lineage and must not duplicate side effects.

The current MVP has SQLite job/event persistence, job status/list/detail and event APIs, cancellation/retry controls, and a browser job monitor that polls while work is active. SSE is not implemented; it remains a future delivery optimization over the persisted cursor. There is no Redis/Celery worker: the current in-process worker is suitable only for development/pilot use and must not be treated as durable, horizontally scalable execution.

Run locally

cd apps/api
python3 -m unittest discover -v
python3 app/main.py --host 127.0.0.1 --port 8000 --db /tmp/prospects.db

Serve the UI separately:

cd apps/web
python3 -m http.server 8080

Open http://127.0.0.1:8080. Set window.API_BASE in the browser console to http://127.0.0.1:8000 when testing the authenticated API locally, then sign in with the configured workspace credentials.

API smoke calls

curl http://127.0.0.1:8000/api/v1/health/live
curl 'http://127.0.0.1:8000/api/v1/businesses?page=1&page_size=25&pipeline_stage=new'
curl http://127.0.0.1:8000/api/v1/businesses/1
curl -X POST http://127.0.0.1:8000/api/v1/businesses \
  -H 'content-type: application/json' \
  -d '{"name":"Example Plumbing","website":"https://example.invalid","email":"info@example.invalid","phone":"+27 21 555 0100"}'

The protected calls require the authenticated session cookie. Exact child-record, notes, pipeline, and audit routes are documented in apps/api/README.md and are never cross-tenant addressable by changing an ID.

Compose

cp .env.example .env
docker compose config --quiet
docker compose up --build -d
curl -fsS http://localhost:8000/api/v1/health/live
curl -fsS http://localhost:8080/healthz
docker compose down

Compose passes the optional BOOTSTRAP_ADMIN_EMAIL and BOOTSTRAP_ADMIN_PASSWORD values to the API. Set both in an untracked .env only when provisioning a fresh instance, then remove them and rotate the password after the bootstrap admin is created. No credentials belong in this repository.

Authenticated browser requests use a server-side session cookie; login creates a session and logout invalidates it. The liveness endpoints (GET /api/v1/health/live and GET /healthz) intentionally remain unauthenticated so Docker, ingress, and monitoring health checks can use them. Authentication is not a substitute for tenant/authorization checks: protected routes must enforce the session and organization boundary server-side.

Phase 5 source boundary and remaining limitations

Phase 5 defines a source adapter contract and registry; it does not implement network discovery, DNS resolution, website/HTTP scanning, enrichment scheduling, or a live external-source adapter. A source adapter must declare its identity, terms owner, permitted purpose, rate limits, retention class, query/result schema, dry-run behavior, and health/circuit controls. CSV and manual reference adapters may be used for operator-supplied data; they must preserve source attribution and raw source records, and must not silently turn preview data into outreach or verified facts.

A discovery query is a tenant-scoped, bounded, auditable request that can be validated and dry-run without contacting a source. Any live source requires explicit product/legal/security approval, a registered adapter, and an operational enablement decision; absent all three, execution must fail closed. Circuit-open, rate-limit, terms, or approval failures must produce a safe non-live result. Raw source records are retained only under the approved retention class and must exclude secrets and unnecessary personal data.

SQLite, the in-process worker, and the named local volume are suitable for the pilot only; production migration, durable queue/worker leases, event retention/backup, SSE delivery, and tested backup/restore remain unfinished. Redis and Celery are not implemented. The development password fallback is PBKDF2 rather than production Argon2id. Before production, complete the gates in docs/SECURITY.md and docs/OPERATIONS.md, including MFA, TLS, CSRF protection, source approval and terms review, rate limiting, circuit monitoring, tenant-scoped job/event authorization, idempotent side-effect handling, durable raw-source/audit retention, SSRF-safe fetching if a future scanner is approved, and tested backups/restores.

Phase 6 normalization and deduplication boundary

Normalization is deterministic and versioned. For South African data, phone values are stripped to digits, local 10-digit 0 forms and 00 27 forms are converted to canonical +27..., and unknown international numbers retain their explicit country code; presentation punctuation must not create a second identity. Locations derive whitespace/case/diacritic-folded province, city, and suburb fields. A normalized value is not proof that the underlying observation is correct.

Exact keys (for example, canonical domain, email, or phone) may identify duplicate candidates. Fuzzy matching is deterministic and suggestion-only: the same inputs and normalization version produce the same candidate, score, and reason. A suggested match must never merge automatically. Use the documented thresholds: >=0.90 is a strong suggestion, 0.75–0.8999 is a review suggestion, and <0.75 is not surfaced as a suggestion. A human with permission must explicitly confirm each merge.

Every confirmed merge must create a tenant-scoped, immutable-enough merge snapshot before mutation, recording the surviving and absorbed IDs, normalized comparison inputs, score/reasons, acting user, timestamp, and schema/normalization versions. The operation must be reversible from that snapshot. It must preserve or re-parent every child, evidence item, provenance/source-record link, note, pipeline/audit history, and original source identity; conflicts remain visible for human resolution rather than being silently overwritten. Cross-tenant candidates are never comparable or mergeable, and each suggestion, confirmation, rejection, reversal, and preservation/conflict decision belongs in the audit trail.

The MVP now exposes deterministic match suggestions at GET /api/v1/businesses/{id}/matches, explicit merge confirmation in the web review dialog, tenant-scoped merge history, and POST /api/v1/merge-history/{id}/reverse. The implementation remains a pilot boundary: hardening is still needed for a dedicated merge permission, stronger server-side confirmation semantics, full snapshot conflict handling, and production-grade rollback guarantees. Do not describe a normalized or suggested match as verified identity, discovery, enrichment, or outreach authorization.

Verification

python3 -m unittest discover -v -s apps/api/tests -t apps/api
python3 -m compileall -q apps/api apps/web
git diff --check
docker compose config --quiet

See apps/api/README.md, apps/web/README.md, docs/SECURITY.md, and docs/OPERATIONS.md for details.

11 KiB Raw Blame History Unescape Escape