Compare commits
10
Commits
fb89a28f2c
...
af9862a794
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
af9862a794 | ||
|
|
561efe8855 | ||
|
|
9622f76977 | ||
|
|
6b41d5b9ee | ||
|
|
537a2a6f0a | ||
|
|
c0909132f2 | ||
|
|
c93dbd1ab4 | ||
|
|
de97a2337d | ||
|
|
655780ff88 | ||
|
|
89eb7e07e6 |
+11
-6
@@ -1,12 +1,17 @@
|
||||
# Local, non-secret runtime configuration. Copy to .env; never commit real credentials.
|
||||
APP_ENV=development
|
||||
# Portable deployment example. Keep this file secret-free; supply SESSION_SECRET via a secret manager.
|
||||
APP_ENV=production
|
||||
LOG_LEVEL=INFO
|
||||
API_PORT=8000
|
||||
WEB_PORT=8080
|
||||
CORS_ORIGINS=http://localhost:8080
|
||||
# Optional first-run admin bootstrap. Set both only for a fresh deployment;
|
||||
# remove them after the admin account has been created. Never commit credentials.
|
||||
CORS_ORIGINS=https://your-approved-web-origin.example
|
||||
DATA_DIR=/data
|
||||
# Required in production; generate at least 32 random characters outside this file.
|
||||
SESSION_SECRET=
|
||||
# Optional first-run admin bootstrap. Remove both immediately after provisioning.
|
||||
BOOTSTRAP_ADMIN_EMAIL=
|
||||
BOOTSTRAP_ADMIN_PASSWORD=
|
||||
# Deliberately fixed to false in compose for this MVP.
|
||||
# Hard safety default; this release has no delivery capability.
|
||||
AUTOMATED_OUTREACH_ENABLED=false
|
||||
# Backup operations (host-side, never mounted into the web container).
|
||||
BACKUP_DIR=/var/backups/prospect-platform
|
||||
BACKUP_RETENTION=30
|
||||
|
||||
@@ -86,6 +86,18 @@ A discovery query is a tenant-scoped, bounded, auditable request that can be val
|
||||
|
||||
SQLite, the in-process worker, and the named local volume are suitable for the pilot only; production migration, durable queue/worker leases, event retention/backup, SSE delivery, and tested backup/restore remain unfinished. Redis and Celery are not implemented. The development password fallback is PBKDF2 rather than production Argon2id. Before production, complete the gates in `docs/SECURITY.md` and `docs/OPERATIONS.md`, including MFA, TLS, CSRF protection, source approval and terms review, rate limiting, circuit monitoring, tenant-scoped job/event authorization, idempotent side-effect handling, durable raw-source/audit retention, SSRF-safe fetching if a future scanner is approved, and tested backups/restores.
|
||||
|
||||
## Phase 16 pilot benchmark boundary
|
||||
|
||||
Phase 16 defines a local, reproducible pilot methodology for evaluating classification and scoring against labeled **synthetic fixtures**. A fixture has a stable ID, schema/label-policy version, input, expected class and safety state, and rationale; it contains no live personal data, credentials, uncontrolled network response, or production record. The fixture manifest is kept separate from implementation outputs and is versioned/hash-recorded with every run. Positive, negative, unknown, blocked, partial, timeout, error, stale, conflicting, suppressed, malformed, boundary, cache, and cross-tenant cases must be represented where applicable.
|
||||
|
||||
Reports include a confusion matrix and support plus precision (`TP/(TP+FP)`) and recall (`TP/(TP+FN)`) for each declared class, with `N/A` for zero denominators. Unknown/blocked/partial/error and safety outcomes are reported separately. Synthetic precision/recall is not evidence of identity, ownership, consent, deliverability, fairness, generalization, operational reliability, or production usefulness; labels may be cleaner or less representative than real data. See `docs/BENCHMARKS.md` for the complete protocol, limitations, reproducibility fields, human-review sampling, and acceptance thresholds.
|
||||
|
||||
Classification/scoring reproducibility requires identical deterministic output fields across repeated fresh-process runs, using recorded commit, fixture hash, algorithm/rule-set versions, runtime/dependency/image versions, non-secret configuration, seed, locale/timezone, rounding, and tie-breaking. Latency must report cold and warm cache separately with cache state, freshness, hit/miss counts, sample count, median, p95, p99, maximum, errors, and timeout handling; averages alone are insufficient. A cache hit is never a fresh observation.
|
||||
|
||||
The pilot gates are synthetic-only: deterministic fields must reproduce at 100%; where positive support is at least 20, precision must be `>=0.90` and recall `>=0.85`; suppressed/blocked/malformed/unknown safety cases must remain non-eligible/non-positive as applicable; tenant-isolation probes must pass 100%; and every latency sample must retain a duration and cache-state label. Human review is blinded and reproducible, stratified across predictions, uncertainty/suppression, and disagreements, with at least 20 fixtures or 10% of the set (whichever is larger, capped by the declared budget). Passing these gates makes no production claim.
|
||||
|
||||
Remote push and deployment prerequisites remain unresolved. Local docs, tests, Compose validation, or a local benchmark do not constitute a remote push or production deployment. Remote publication still requires repository authentication and branch permission; deployment still requires the Phase 15 reviewed revision, host/Docker/Compose access, protected directory, secrets, DNS/TLS/Virtualmin, encrypted off-host backups, monitoring, rollback ownership, and operational approval. No deployment is performed by this change.
|
||||
|
||||
## Phase 6 normalization and deduplication boundary
|
||||
|
||||
Normalization is deterministic and versioned. For South African data, phone values are stripped to digits, local 10-digit `0` forms and `00 27` forms are converted to canonical `+27...`, and unknown international numbers retain their explicit country code; presentation punctuation must not create a second identity. Locations derive whitespace/case/diacritic-folded province, city, and suburb fields. A normalized value is not proof that the underlying observation is correct.
|
||||
@@ -120,6 +132,68 @@ Scan history is tenant-scoped and append-oriented. Results and cache entries are
|
||||
|
||||
The website scanner remains a pilot boundary. Compose does not provide a production egress proxy, durable scan queue, distributed crawl coordinator, hardened DNS resolver, or compliance-grade result store. Production still requires independent SSRF testing (including DNS rebinding and redirect chains), egress/network policy, resource isolation, durable retention/deletion, authenticated scan-history authorization, rate limits and abuse controls, observability, and a reviewed policy for content types, robots/terms, caching, and incident response. Scans must never trigger acquisition, verification, enrichment, or outreach automatically.
|
||||
|
||||
## Phase 9 public official-site contact extraction boundary
|
||||
|
||||
Phase 9 adds a **passive, suggestion-only** contact-observation workflow. When explicitly enabled, extraction may inspect bounded HTML from the business's approved/public official-site origin and its same-site contact/about pages; it is not general web search, crawling, enrichment, identity verification, or outreach. Only public page content and explicitly permitted `mailto:`/visible contact values may be considered. Do not submit forms, authenticate, bypass access controls, probe SMTP, send test messages, or contact a person or organization.
|
||||
|
||||
Every candidate contact must retain provenance: source URL and page location/context, extraction method, observed time, scanner/extractor and policy versions, and the exact uncertainty/reason code. Confidence is an explainable review signal, not deliverability, consent, ownership, or permission to contact. Classify role addresses separately from person addresses and classify free-mail domains separately from business-domain addresses; neither classification is proof of identity. Syntax validation is only a parse result. MX/DNS status is independently uncertain (`not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`), and no MX result may be presented as deliverability.
|
||||
|
||||
False-positive exclusions must reject or quarantine values from asset URLs, image/file names, scripts/styles, example/test/placeholder domains, documentation text, tracking addresses, and malformed or unsupported schemes. Apply tenant-scoped suppressions before a candidate is persisted, returned, exported, or queued for review; suppressed values remain do-not-contact and suppression always wins over confidence, role, syntax, MX, pipeline, or verification state. Extraction is bounded by per-request and aggregate page/URL, byte, time, redirect, candidate, and concurrency limits. Store only the minimum contact value and lineage required for review, apply a documented retention/deletion class, and redact secrets and unnecessary personal data from logs and audit events.
|
||||
|
||||
Phase 9 does not authorize automated outreach. There is no SMTP probing, SMTP banner/VRFY/EXPN check, email validation message, send endpoint, campaign queue, or follow-up action. An extracted address is an observation requiring human review and explicit policy authorization before any separate future contact workflow.
|
||||
|
||||
## Phase 10 configurable scoring boundary
|
||||
|
||||
Phase 10 separates **fit score**, **priority band**, and **contact eligibility**. A score is a deterministic ranking signal; it never authorizes contact. Rules are represented by a named, versioned rule set with explicit weights, thresholds, band definitions, eligibility gates, freshness policy, and suppression behavior. The active rule-set identifier/version is stored with each result so a historical score can be explained without silently applying today's policy.
|
||||
|
||||
A reproducible calculation uses the tenant-scoped business snapshot, normalized values, eligible evidence observations, rule-set/version, algorithm/version, and calculation time/freshness inputs. Explanations must retain the contributing factors, normalized inputs or evidence references, weights/points, exclusions, uncertainty reasons, and the final band. Do not accept a client-submitted score, band, eligibility flag, or rule version as authoritative.
|
||||
|
||||
Priority bands are policy labels (for example, high/medium/low or an explicitly configured equivalent) and must be derived from the versioned thresholds. Eligibility is evaluated separately and fail-closed: suppression/do-not-contact, stale or expired required evidence, unresolved/uncertain required signals, missing policy prerequisites, and authorization/tenant failures can make a prospect ineligible regardless of score. `suppressed` always wins and must remain visible; stale and uncertain observations must not be silently treated as absent or positive.
|
||||
|
||||
Recalculation is an explicit, tenant-scoped operation. It must snapshot the input/rule versions, record before/after score, band, eligibility, explanation, actor/job, timestamp, and reason in the audit trail, and be idempotent or safely repeatable. A policy/rule change must not rewrite history without an auditable recalculation; partial or failed recalculation must report its incomplete state rather than presenting mixed results as current.
|
||||
|
||||
Phase 10 remains a pilot boundary unless the runtime exposes all of the above controls end to end. Production work includes administrative rule-set lifecycle/approval, immutable calculation inputs, deterministic rounding/tie-breaking, scheduled recalculation with leases, retention and export semantics for explanations/audit, and regression tests proving suppression, stale, uncertain, and cross-tenant isolation behavior. See the API, security, and operations contracts for the authoritative safeguards.
|
||||
|
||||
## Phase 11 dashboard and review workflow boundary
|
||||
|
||||
Phase 11 defines the operator-facing review workflow around **saved filters**, a tenant-scoped **review queue**, and deliberately bounded bulk actions. A saved filter is a named, server-persisted filter definition (including search, score/status/pipeline/eligibility criteria, sort, and page-size preferences) owned by one tenant. Save, load, rename, and delete operations must authorize against the authenticated tenant; filter definitions are configuration, not authorization, and must not contain secrets or unbounded query instructions.
|
||||
|
||||
The review queue is a derived view of the current tenant's matching businesses. Queue counts and rows must use the same filter predicate, exclude records the caller cannot read, and make suppression and merge state explicit. Suppressed/do-not-contact records are never eligible for contact or bulk contact action; merged/non-active records are not merge-eligible. Match suggestions remain review-only: a score cannot auto-merge, and every merge requires an explicit permitted human confirmation and a reversible, tenant-scoped snapshot.
|
||||
|
||||
Bulk actions must be bounded by a server-enforced maximum, an explicit set of tenant-scoped record IDs or an immutable filter snapshot, and a preview/confirmation step. The API must re-check authorization, suppression, merge eligibility, and current record state at execution time, use an idempotency key or equivalent safe retry control, and return per-record outcomes rather than claiming that the whole selection succeeded. A filtered count is not permission to act on an unbounded result set, and bulk actions must never imply outreach or bypass review.
|
||||
|
||||
Dashboard counts are clickable only when their scope and query semantics are clear. A click must carry the exact tenant-scoped predicate that produced the count; it must not silently broaden to all records. Labels must distinguish the current page from the full matching set, and loading, stale, unavailable, and error states must not be rendered as zero. Every saved-filter mutation, queue decision, bulk preview/confirmation/result, suppression decision, and merge/reversal must emit an audit event with tenant, actor, action, timestamp, filter/selection snapshot, bounded counts, per-item outcomes, and relevant policy/version identifiers. Audit details must be redacted and tenant-scoped.
|
||||
|
||||
The Phase 11 slice now provides tenant-scoped saved-filter storage/listing, a bounded review-queue read, clickable dashboard filter metadata, and a bounded bulk review route (`POST /api/v1/businesses/bulk-review`) for verify/reject/assign. The current implementation is still pilot-grade: saved-filter update/delete routes are not wired, the queue does not yet expose full matching-set counts or filter snapshots, dashboard click metadata is not a complete predicate contract, bulk actions lack preview/idempotency/per-record outcomes, and audit coverage is aggregate for bulk operations. Production work requires those hardening items plus regression tests for suppression precedence, merge eligibility, cross-tenant IDs, stale counts, and partial bulk failure.
|
||||
|
||||
## Phase 12 CRM pipeline, interactions, outcomes, reporting, and suppression center
|
||||
|
||||
Phase 12 adds the CRM coordination contract around a tenant-scoped pipeline, an append-oriented interaction timeline, normalized interaction outcomes, bounded reporting, and a suppression center. These features are review and record-keeping tools; they do **not** turn the platform into an outreach system. Every read, write, export, report, and background operation must carry the authenticated `organization_id` scope, and a child ID, report ID, filter, or aggregate count is never authorization.
|
||||
|
||||
### Pipeline state and transition rules
|
||||
|
||||
The canonical lifecycle is `new` → `contacted` → `qualified` → `proposal` → `negotiation` → `won` or `lost`. The API exposes these configured stages; any future paused/disqualified state must be explicitly added to the tenant's stage configuration and may be reopened only by an authorized human with a reason. A transition must name the target state, actor, timestamp, and reason/source; the API validates transitions server-side and records the before/after state in the audit trail. Repeating the current state is idempotent, not a new transition. Direct jumps, client-supplied history, edits to historical transitions, and transitions for merged/inactive records are rejected. Reopening `lost`, `paused`, or `disqualified` creates a new transition and does not rewrite history.
|
||||
|
||||
An interaction may suggest a state change, but it never changes pipeline state implicitly. A state change and its related interaction/outcome are separate auditable events, and a failed or partial write must not leave a fabricated outcome. `won`/`lost` require an explicit outcome and reason; `won` is not proof of payment or fulfillment. Suppression/do-not-contact overrides every pipeline state and makes contact eligibility false.
|
||||
|
||||
### Interactions and outcome taxonomy
|
||||
|
||||
Interactions are append-only, tenant-scoped records of an operator-observed event. The record should retain the business/contact reference when known, channel (`note`, `phone`, `email`, `meeting`, or `other`), occurred time, actor, bounded redacted summary, provenance, and correlation/idempotency key. The current product does not send or validate communication: an interaction records what an operator says happened, not what the platform performed.
|
||||
|
||||
Outcomes are normalized and mutually explicit: `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, and `other`. `other` is an explicit catch-all, not proof of success or failure; missing/unknown data must not be silently converted to a negative outcome. `disqualified` and `lost` require an explicit reason under the configured workflow. `do_not_contact` is a separate safety state, immediately creates or confirms tenant suppression, and cannot be downgraded by a later positive outcome. Free-text notes supplement but do not replace the taxonomy; corrections append a new record rather than mutating history.
|
||||
|
||||
### Reporting, audit, tenant scope, and retention
|
||||
|
||||
Reports use an explicit `as_of` time, tenant scope, timezone, date interval, and inclusion policy. Pipeline reports count the latest effective state per active business; interaction reports count events by `occurred_at` (not ingestion time); outcome reports count normalized outcomes and may include a separate `unknown` bucket. Suppressed, merged, deleted, and inactive records must be labeled and excluded from contact-eligible totals; they must not silently disappear from safety/audit counts. Page counts, matching-set counts, and distinct-business counts are different metrics and must be named. Late-arriving or corrected interactions preserve original and corrected timestamps and are never double-counted without an explicit correction policy.
|
||||
|
||||
Audit events cover pipeline transitions, interaction/outcome creation or correction, suppression changes, report/export requests, and report results. Store tenant, actor, target, action, timestamps, safe reason, bounded filter/as-of snapshot or hash, policy/version, correlation/idempotency ID, and per-item outcomes where applicable. Redact full contact values and free text unless required for the approved purpose. Retain CRM records, suppression decisions, interaction provenance, report snapshots, and audit events according to the approved tenant/data-retention schedule; deletion or legal-hold behavior must be explicit and auditable. A report cache is tenant-keyed, bounded, and labeled with its `as_of`/freshness; it is never a live authorization decision.
|
||||
|
||||
### Suppression center and remaining outreach limitations
|
||||
|
||||
Suppression is a tenant-scoped deny list for email, domain, phone, and other approved identifiers. Matching is normalized server-side and must run before persistence, display, export, report inclusion as eligible, queueing, or any future action. Suppression wins over pipeline state, outcome, score, verification, cached data, and operator intent. The center must show the source, reason, actor, created/updated time, scope, and effective status; removal or expiry requires explicit authorization, reason, audit, and re-evaluation. Existing records remain visible as **Do not contact** and are not silently deleted. No Phase 12 route may send email/SMS, probe SMTP, validate an address by message, create a campaign, schedule follow-up delivery, or imply consent/deliverability. Any future outreach requires separate product, legal, security, and operational approval and must remain disabled by default.
|
||||
|
||||
Phase 12 remains pilot-grade until transition validation, immutable interaction/outcome history, suppression precedence, report definitions/timezones, retention/deletion jobs, export controls, idempotent writes, and cross-tenant regression tests are exercised end to end. The current Compose stack still has no durable CRM worker, scheduler, delivery provider, or outreach capability.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
@@ -129,4 +203,48 @@ git diff --check
|
||||
docker compose config --quiet
|
||||
```
|
||||
|
||||
See `apps/api/README.md`, `apps/web/README.md`, `docs/SECURITY.md`, and `docs/OPERATIONS.md` for details.
|
||||
## Phase 13 optional AI assistance boundary
|
||||
|
||||
Phase 13 adds optional, human-reviewed AI assistance for summarization, classification, and evidence-oriented suggestions. AI is a drafting aid, not a source of truth, verifier, identity resolver, score/eligibility authority, or CRM actor. It is disabled unless an explicitly configured provider and tenant-scoped policy permit the requested operation. A provider failure, timeout, quota/rate limit, approval expiry, missing evidence, or ambiguous result fails closed to `unknown`/`unavailable`; it must never be represented as a successful empty result.
|
||||
|
||||
Provider configuration is optional and deny-by-default. A deployment may configure a primary provider and an optional fallback, but each provider must have an approved purpose/capability, tenant scope, data-processing/retention terms, region/egress policy, model/version, timeout/token budget, rate limit, cost ceiling, and operational enablement. Fallback is permitted only to another pre-approved provider for the same purpose and data class; it must not broaden tenant scope, retention, prompt data, or authority. No provider credentials belong in source, Compose files, logs, or committed `.env` files. The current Compose stack does not provision an AI provider; production enablement remains gated configuration work.
|
||||
|
||||
AI requests must minimize data before transmission: send only the fields and evidence excerpts needed for the approved task, redact secrets and unnecessary personal/contact data, avoid raw page bodies and credentials, and record a redacted request/policy fingerprint rather than a prompt containing sensitive data. Every generated suggestion must cite the tenant-scoped evidence IDs/source references and preserve evidence hash, citation, observed time, provider/model/version, policy version, and uncertainty. Hashes identify the exact evidence snapshot for reproducibility; they do not prove that the source is true. Missing, conflicting, stale, suppressed, or low-quality evidence must remain visible and must not be filled with invented facts.
|
||||
|
||||
AI output is an untrusted draft. It requires an authorized human approval/rejection (and an explicit reason for material changes) before it can become a stored claim, score input, pipeline update, contact decision, export, or any other consequential record. Approval must re-check tenant scope, suppression, evidence freshness, policy/version, and the unchanged evidence hash; stale or changed inputs require re-review. AI cannot create or edit CRM interactions/outcomes as if communication occurred, send messages, create campaigns, schedule follow-ups, contact prospects, merge records, acquire domains, or perform autonomous outreach. Suppressed/do-not-contact records remain visible for safety review and are never made eligible by an AI result.
|
||||
|
||||
Phase 13 is pilot-only. Before production, implement provider allowlisting and secret management, data-processing agreements, prompt/output redaction tests, evidence hash/citation verification, approval and rollback semantics, immutable audit coverage, tenant-isolation tests, retention/deletion/legal-hold jobs, cost/rate monitoring, incident disablement, and evaluation for hallucination, prompt injection, bias, and stale/conflicting evidence. See the API, web, security, and operations contracts for the authoritative limitations.
|
||||
|
||||
## Phase 14 draft-only outreach boundary
|
||||
|
||||
Phase 14 prepares outreach records without sending them. The platform may assemble a bounded, tenant-scoped draft from approved evidence, but it has no send endpoint, delivery worker, campaign scheduler, SMTP probe, validation message, or autonomous follow-up. `AUTOMATED_OUTREACH_ENABLED=false` remains the enforced Compose default and a draft is never a queued or delivered message.
|
||||
|
||||
A future outreach provider must be explicitly registered and allowlisted with an owner, capability/purpose, tenant and data-class scope, processing region/retention terms, model or API version where applicable, timeout and payload limits, per-tenant/global rate caps, daily cost/message caps, health/circuit policy, approval expiry, and operational enablement. Provider credentials belong only in a deployment secret manager; never place them in source, Compose, browser code, logs, audit payloads, or committed `.env` files. Provider failure, timeout, quota, circuit-open state, expired approval, or policy mismatch fails closed and must not silently switch to an unapproved fallback.
|
||||
|
||||
A draft is eligible only after server-side gates pass: the intended recipient and organization are tenant-scoped; the contact is not suppressed or do-not-contact; required evidence is present, cited, sufficiently fresh, and permitted for the stated purpose; consent/legal basis and jurisdiction/channel policy are recorded; the current score/eligibility state is not stale or uncertain; and a permitted human explicitly approves the exact draft. Public availability, a high score, a verified pipeline state, an extracted address, or an AI citation is not consent, legal basis, deliverability, or permission to contact. Suppression always wins and must be re-checked immediately before approval and any future send attempt.
|
||||
|
||||
Drafts must retain evidence citation IDs/source references, exact evidence snapshot hash, observed times, policy/provider versions, recipient/channel, redacted content fingerprint, actor, approval decision/reason, expiry, and correlation/idempotency key. The UI and API must display citations and uncertainty, preserve unsupported/conflicting evidence, and label the result **Draft only — human approval required**. Human approval is explicit, tenant-scoped, single-purpose, and bound to the unchanged draft/evidence hash; edits, stale evidence, policy changes, or approval expiry require a new review. Approval does not itself send or authorize a later send.
|
||||
|
||||
Any future side-effecting operation must require a tenant-scoped idempotency key bound to a request fingerprint and draft/provider/policy version. Exact replay returns the original bounded result; conflicting reuse is rejected. Enforce caps before provider calls and across retries, fallbacks, workers, and tenants, with bounded backoff and no cap bypass by rotation. Append audit events for draft creation, gate outcomes, citations, approval/rejection/expiry, provider attempts, cap denials, suppression decisions, retries, and any future send result; redact secrets, full prompts, credentials, and unnecessary contact content.
|
||||
|
||||
Phase 14 is preparation only and remains pilot-grade. The current Compose stack has no outreach provider, draft/send API, consent ledger, legal-policy engine, durable approval workflow, production secret manager, durable queue/worker, delivery telemetry, bounce/complaint handling, or compliance-grade audit/retention service. Before production, obtain jurisdiction-specific legal review and documented consent/legal-basis policy, implement provider contracts/DPA and secret isolation, durable transactional idempotency and audit, suppression synchronization, approval expiry/rollback, rate/cost controls, delivery feedback, incident kill switch, retention/deletion/legal-hold workflows, and end-to-end tenant-isolation and no-send tests.
|
||||
|
||||
See `docs/DEPLOYMENT.md` for Phase 15 portable backup/restore, readiness, rollback, monitoring, and systemd/Virtualmin operations. No deployment is performed by this repository change.
|
||||
|
||||
## Phase 15 production-readiness boundary
|
||||
|
||||
Phase 15 documents a portable deployment pattern for a Virtualmin-managed VPS; it does not claim that this repository provisions Virtualmin, TLS, DNS, backups, or a production database. The supported baseline is Docker Engine plus the Compose v2 plugin on a Linux host, with Virtualmin (or another reverse proxy) terminating HTTPS and forwarding only to the published web/API ports. The operator owns firewalling, DNS, certificates, Docker access, host patching, resource capacity, and an off-host backup destination.
|
||||
|
||||
Before deployment, verify the prerequisites in `docs/RELEASE_CHECKLIST.md`: a reviewed commit, Docker/Compose, DNS and TLS, a protected deployment directory, secret injection, backup destination, monitoring, and a tested rollback owner. Copy `.env.example` to an untracked `.env` only for non-secret defaults. Production secrets and bootstrap credentials must come from a secret manager or protected Virtualmin deployment environment; remove bootstrap values after first-run provisioning and never commit or print them.
|
||||
|
||||
The API liveness endpoint (`/api/v1/health/live`) and web `/healthz` are public process checks; API `/api/v1/health/ready` additionally verifies SQLite readiness. They are suitable for Docker and monitoring but do not prove backups, tenant authorization, or external dependencies. Production ingress must not route traffic until both Compose services report `healthy` and the deployment smoke tests pass. The current image initializes SQLite from `schema.sql` and has no standalone migration runner. Treat schema changes as a versioned, backup-first migration: validate on a restored copy, record the schema/data checks, and keep the previous image/config available for rollback.
|
||||
|
||||
The named Docker volume is not a backup. Stop or quiesce writes, create an encrypted off-host backup, verify it, and perform a restore drill before calling a deployment protected. Define retention for the database, audit/source lineage, logs, and backups; apply legal holds and deletion rules deliberately. Do not run `docker compose down -v` on a data-bearing host. Outbound traffic is deny-by-default for product behavior: `AUTOMATED_OUTREACH_ENABLED=false` is fixed in Compose and this release has no send/provider/worker path. Unexpected egress is an incident. SQLite, the in-process worker, HTTP-only local Compose, lack of durable migrations/PITR, and the limited SQLite-only readiness check are explicit limitations, not hidden guarantees.
|
||||
|
||||
## Phase 17 final acceptance
|
||||
|
||||
Phase 17 records the final local acceptance decision in `docs/FINAL_ACCEPTANCE.md`: **local acceptance passed; production deployment was not attempted and remains blocked**. The verified local head is `9622f769776637a40fdae797adccba91445cd351`. The acceptance run passed 100 API tests, the deterministic Phase 16 benchmark, Python and shell syntax checks, JSON validation, Compose configuration validation, and safety invariants.
|
||||
|
||||
The bounded capacity smoke is deterministic and in-memory only: a synthetic 1,000-item collection returns a maximum page of 100, and a 5,000-item synthetic batch retains at most 100 items. This demonstrates bounded behavior, not production throughput, concurrency, durability, availability, or an SLO. Reproducibility requires the recorded commit, fixture hash, rule/algorithm versions, runtime/dependency/image versions, non-secret configuration, seed, locale/timezone, rounding, and tie-breaking; see `docs/FINAL_ACCEPTANCE.md` and `docs/BENCHMARKS.md`.
|
||||
|
||||
Remote push is still blocked by repository authentication and branch-permission prerequisites. Production additionally requires host/Docker/Compose access, protected deployment paths, secret injection, DNS/TLS/Virtualmin, encrypted off-host backups and restore evidence, monitoring, rollback ownership, operational approval, and all applicable provider, terms/DPA, consent/legal-basis, and retention prerequisites. No local check, benchmark, or commit is a remote push or production deployment. `AUTOMATED_OUTREACH_ENABLED=false` remains the required default.
|
||||
|
||||
+88
-1
@@ -1,4 +1,4 @@
|
||||
# Prospect Platform API — Phase 8 boundary
|
||||
# Prospect Platform API — Phase 9 boundary
|
||||
|
||||
Dependency-light JSON API for tenant-scoped prospect workflows and the Phase 8 bounded website-scanning, Phase 7 domain-intelligence, Phase 6 normalization/deduplication, and Phase 5 source-ingestion contracts. Core domain rules use Python's standard library and persistence is SQLite. Scan requests/results, where enabled, must remain auditable and fail closed; scanning never submits forms, executes JavaScript, or authorizes outreach.
|
||||
|
||||
@@ -57,6 +57,7 @@ Implementations should expose source/query/job state without leaking raw payload
|
||||
### Health and workspace
|
||||
|
||||
- `GET /api/v1/health/live` — unauthenticated liveness check.
|
||||
- `GET /api/v1/health/ready` — unauthenticated readiness check; verifies SQLite connectivity and returns 503 when unavailable.
|
||||
- `GET /api/v1/auth/me` — current authenticated user and tenant.
|
||||
- `GET /api/v1/dashboard/summary` — tenant-scoped counts and score summary.
|
||||
|
||||
@@ -68,6 +69,8 @@ Implementations should expose source/query/job state without leaking raw payload
|
||||
The business detail includes the supported child collections: `contacts`, `domains`, `websites`, `evidence`, `pipeline`, and `notes`. The child collection routes are:
|
||||
|
||||
- `POST /api/v1/businesses/{id}/contacts` — manually add a contact; suppression matching marks a matching contact as do-not-contact.
|
||||
- `GET /api/v1/businesses/{id}/contacts/extract` — read the tenant-scoped Phase 9 official-site extraction projection.
|
||||
- `POST /api/v1/businesses/{id}/contacts/extract` — extract bounded public contacts from the approved official site/same-site pages and persist provenance-bearing observations; the request is passive and must not probe SMTP or send outreach. Suppression matching marks matches `suppressed`/`do_not_contact` and remains authoritative.
|
||||
- `POST /api/v1/businesses/{id}/domains` — manually add a domain observation.
|
||||
- `POST /api/v1/businesses/{id}/websites` — manually add a website observation/classification.
|
||||
- `POST /api/v1/businesses/{id}/evidence` — manually add evidence with its kind, claim, and source URL/reference. This records provenance supplied by the operator; it does not scan or independently verify the URL.
|
||||
@@ -126,6 +129,90 @@ Classifications must be conservative, explainable, and derived only from bounded
|
||||
|
||||
Scan history and cache reads/writes require the same tenant predicate as business routes. Keys include normalized URL, scanner/policy version, and relevant request/redirect policy; entries are size- and retention-bounded, expose `observed_at` and freshness/expiry, and never make a cache hit look like a fresh scan. Invalidate or re-evaluate entries after policy, DNS, or scanner-version changes. No scan result may trigger enrichment, acquisition, verification, or outreach.
|
||||
|
||||
## Phase 9 official-site contact extraction contract
|
||||
|
||||
The optional Phase 9 extractor is a passive, authenticated, tenant-scoped observation of a business's approved/public official-site origin. It may inspect bounded HTML and same-site contact/about pages only; it must not become a search engine, unrestricted crawler, or arbitrary URL fetcher. The extractor must use the existing SSRF-safe URL, redirect, content-type, and resource-budget controls, and must never submit forms, execute target JavaScript, use credentials/cookies, probe SMTP, issue SMTP `VRFY`/`EXPN`, send validation mail, or perform outreach.
|
||||
|
||||
For each candidate, return/store the normalized address only with provenance (official-site/page URL, page or DOM context, extraction method, observed time, extractor/policy version) and an explainable confidence/reason list. Preserve uncertainty rather than inventing facts. `syntax_valid`/`syntax_invalid` is a parser outcome only. Role classification (`role`/`person`/`unknown`) and free-mail classification (`free_mail`/`business_domain`/`unknown`) are independent review labels; they do not prove identity, consent, ownership, or deliverability. MX/DNS is a separate observation with resolver/source, observed time, TTL/freshness where available, and one of `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error`; MX absence or failure remains unknown and must never be treated as invalid or undeliverable.
|
||||
|
||||
Exclude false positives before persistence and response: values in scripts/styles/comments or asset URLs/file names, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party pages. Enforce hard limits for total extraction time, pages/URLs, redirects, response and retained bytes, candidates per page/request, and concurrency. Suppression matching is server-side and tenant-scoped, before storing, returning, exporting, or presenting a candidate; suppressed contacts are marked do-not-contact and cannot be revived by a later classification or review action. Retention must be explicit and bounded for extracted values, page provenance, DNS/MX observations, caches, and audit records; logs must not contain full contact payloads when a redacted identifier is sufficient.
|
||||
|
||||
Extraction results are suggestions only and do not create a send/contact capability. The API exposes no SMTP-probe, validation-message, outreach, or campaign endpoint. If the feature is disabled, unapproved, over limit, blocked, or uncertain, fail closed with an explicit status/reason rather than an empty successful result. The current MVP remains pilot-only until extraction limits, suppression enforcement, retention/deletion jobs, provenance/audit coverage, and tenant-isolation tests are production hardened.
|
||||
|
||||
## Phase 10 scoring contract
|
||||
|
||||
Scoring is a tenant-scoped, deterministic derivation with three separate outputs: `score` (fit/ranking), `priority_band` (versioned threshold policy), and `eligibility` (whether a later, separately authorized workflow may act). A high score or priority band never authorizes contact. The API must calculate these values server-side from a named rule set and immutable rule-set version; weights, thresholds, required evidence, freshness windows, suppression precedence, and uncertainty handling are configuration, not undocumented code defaults.
|
||||
|
||||
Each result should expose the rule-set ID/version and algorithm/version, calculation timestamp, input/evidence snapshot or stable references, contributing factors, points/weights, exclusions, band decision, eligibility decision/reasons, and stale/uncertain state. Explanations are reviewable lineage, not proof of identity, consent, deliverability, or permission. Normalize and round deterministically, define tie-breaking, and reject client-supplied score/band/eligibility/version fields.
|
||||
|
||||
Eligibility is evaluated independently of score. Tenant-scoped suppression/do-not-contact is an unconditional ineligible result. Required evidence that is stale, expired, missing, blocked, partial, or uncertain must produce an explicit reason and fail closed according to the active rule set; it must not be converted into a zero, a positive signal, or an empty successful result. A score may remain visible for triage while eligibility is `ineligible` or `unknown`, and suppression must remain visible after recalculation.
|
||||
|
||||
Recalculation must be an explicit authenticated operation, preferably represented by the existing tenant-scoped job contract for larger sets. It must capture the requested rule-set/version, input snapshot, actor/job/idempotency lineage, started/completed time, counts and failures, and before/after score, band, eligibility, and explanation changes. Every rule change and recalculation is auditable; retries cannot duplicate or erase history, and a partial run must be marked incomplete. Audit and explanation reads use the same organization predicate as business reads, and cross-tenant IDs/jobs/rule sets behave as not found.
|
||||
|
||||
The current MVP's scoring surface is limited compared with the Phase 10 contract: production still needs an authorized rule-set management API, approval/activation and rollback semantics, immutable evidence snapshots, scheduled/durable recalculation, concurrency protection, deterministic migration of old scores, and comprehensive tests for suppression, stale/uncertain evidence, audit completeness, and tenant isolation.
|
||||
|
||||
## Phase 11 dashboard and review workflow contract
|
||||
|
||||
Phase 11 adds the API contract for saved filters and review operations without weakening the tenant boundary. A saved filter is a named, tenant-owned record containing a validated, bounded predicate (search, score/status/pipeline/eligibility filters, sort, and page-size preference). Save/load/update/delete/list routes must scope by `organization_id`, reject unknown or unbounded fields, and never treat a client-provided filter ID as authorization. Sharing, if added, must be explicit and remain within the tenant; filter definitions must not store secrets.
|
||||
|
||||
A review queue is a derived, tenant-scoped projection of businesses matching the saved/current filter. Its response must identify the predicate/snapshot, ordering, page or cursor, bounded `items`, and whether counts are page counts or full matching-set counts. Suppressed/do-not-contact records must remain visible as safety state when policy requires review, but are never contact-eligible. Merged/non-active businesses are excluded from merge candidates and must not be acted on as active records. Queue counts are not authorization and must be recomputed under the caller's tenant and permission scope.
|
||||
|
||||
Bulk operations must accept only a bounded selection of IDs or a server-created immutable filter snapshot, enforce a maximum batch size before execution, and require preview followed by explicit confirmation. At execution time the server must re-check tenant ownership, permissions, suppression, active/merge eligibility, and current versions. Require an idempotency key or equivalent safe retry behavior, prevent duplicate side effects, and return a per-record result (`succeeded`, `skipped`, or `failed` with a safe reason) plus bounded totals. A request accepted or previewed is not completion. Bulk review actions do not create an outreach capability and must not auto-merge records.
|
||||
|
||||
Clickable dashboard counts must link to the exact tenant-scoped predicate that produced them. The API must distinguish `page_count` from `matching_count`/`has_more`; clients must not turn a page count into a global total or silently drop eligibility/suppression criteria on navigation. Loading, stale, error, and unavailable counts are distinct from zero. Saved-filter changes, queue decisions, bulk preview/confirmation/execution, suppression/eligibility decisions, and merge/reversal operations require audit records containing tenant, actor, action, timestamp, filter/selection snapshot or hash, bounded counts, per-item outcomes, policy/version context, and a correlation/idempotency identifier. Audit reads use the same organization predicate and redact secrets and unnecessary personal/contact data.
|
||||
|
||||
The current Phase 11 slice exposes `GET /api/v1/saved-filters`, `POST /api/v1/saved-filters`, `GET /api/v1/review-queue`, and `POST /api/v1/businesses/bulk-review`. Saved filters are durable and bounded, the queue is capped at 100 rows per request, and bulk verify/reject/assign accepts at most 100 explicit IDs. The slice remains pilot-grade: update/delete saved-filter handlers are not routed, review-queue results do not yet expose a full matching-set count or immutable filter snapshot, dashboard clickable-filter metadata is not a complete predicate, bulk execution has no preview/idempotency/per-record outcome contract, and bulk audit is one aggregate event. Do not infer stronger guarantees from the existing list filters.
|
||||
|
||||
## Phase 12 CRM API contract
|
||||
|
||||
Phase 12 introduces tenant-scoped CRM records for pipeline state, append-only interactions, normalized outcomes, bounded reports, and a suppression center. All routes must use the authenticated session's `organization_id`; a business, interaction, outcome, report, export, suppression, cursor, or filter ID from another tenant behaves as not found. The server—not the web client—enforces role permissions, state transitions, suppression, batch/report limits, and redaction.
|
||||
|
||||
Pipeline transitions use `new` → `contacted` → `qualified` → `proposal` → `negotiation` → `won`/`lost`. Any paused/disqualified state must be explicitly configured before use and requires a reasoned, authorized reopen. The API accepts only policy-approved transitions, rejects direct jumps and changes to merged/inactive records, and appends actor, before/after state, reason, timestamp, and correlation/idempotency metadata. Same-state retries are idempotent. Reopening a terminal-for-now state creates a new event; it never edits history. Interactions do not implicitly advance the pipeline. `won` and `lost` require an explicit outcome/reason.
|
||||
|
||||
Interaction records contain a bounded safe summary, channel, occurred/recorded timestamps, actor, business/contact reference, provenance, and idempotency lineage. Outcomes are normalized to `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, or `other`. `other` is an explicit catch-all, not proof of success or failure; missing data must not be silently assigned a negative outcome. Corrections append a superseding event and preserve the original. `do_not_contact` is safety-critical and cannot be overridden by score, stage, a later outcome, or a client payload.
|
||||
|
||||
Reports must require bounded date ranges and page/row limits and expose their tenant, timezone, `as_of`, freshness, filter snapshot, and semantics. State reports use the latest effective state per active business; interaction/outcome reports use `occurred_at`; counts distinguish events from distinct businesses and page counts from matching-set counts. Suppressed, merged, inactive, and unknown records are labeled and never counted as contact-eligible. Report/export requests and results are audited, report caches are tenant-keyed, and report data must not become an authorization shortcut.
|
||||
|
||||
Suppression endpoints accept only approved normalized identifier kinds and record source, reason, actor, scope, and timestamps. Matching occurs before writes, responses, exports, reports, caches, and queues. Suppressed contacts remain visible as `suppressed`/`do_not_contact` for safety review; deletion or unsuppression requires an authorized, reasoned, audited operation and does not retroactively rewrite interaction history. No endpoint sends messages, probes SMTP, performs validation mail, creates campaigns, or schedules delivery. Outreach remains disabled and requires a separate approved product/security/legal design.
|
||||
|
||||
Every CRM mutation and report/export operation emits an audit record with tenant, actor, action, target, before/after or bounded result, policy/version, timestamps, correlation/idempotency ID, and safe reason. Audit and retention reads use the same organization predicate. CRM records, contact references, suppression decisions, report snapshots, and audit details require explicit retention classes, deletion/legal-hold semantics, and redacted logs. These are the Phase 12 contract; production readiness additionally requires durable migrations, worker/retry behavior, report reproducibility, export authorization, retention jobs, and transition/outcome/suppression/cross-tenant tests.
|
||||
|
||||
## Phase 14 draft-only outreach API contract
|
||||
|
||||
Phase 14 is a preparation contract, not a delivery feature. A future authenticated route may create a tenant-scoped outreach **draft** from a bounded approved evidence set, but the current API exposes no send, delivery, campaign, SMTP, validation-message, or autonomous follow-up endpoint. The server must reject any request that attempts to send or that treats draft creation/approval as delivery. `AUTOMATED_OUTREACH_ENABLED=false` is the no-send default in Compose and must be enforced server-side, not only by the UI.
|
||||
|
||||
Draft creation must evaluate and persist gate results for tenant/recipient scope, normalized suppression/do-not-contact, consent or other documented legal basis, jurisdiction/channel policy, evidence permission/freshness, current eligibility, provider approval, and content policy. Suppression is an unconditional deny. Public contact data, pipeline state, score, verification metadata, or AI evidence citations do not establish consent, lawful basis, deliverability, or permission to contact. A blocked, missing, stale, conflicting, or uncertain gate returns an explicit non-send reason rather than an empty success.
|
||||
|
||||
Provider configuration is server-side and deny-by-default. A registered provider must include an allowlisted ID/version, purpose/capability, permitted tenant/data class, processing region and retention terms, timeout/payload limits, per-tenant and global rate caps, daily message/cost ceilings, health/circuit state, approval owner/expiry, and explicit operations enablement. Credentials are secret-manager references only and must never be accepted from clients or returned in responses/logs. Fallbacks, if ever enabled, must be pre-approved for the same purpose, data class, policy, caps, citations, and authority; provider failure, timeout, quota, circuit-open, or expired approval fails closed.
|
||||
|
||||
Every draft must retain recipient/channel, bounded redacted content or a safe content hash, evidence IDs/source citations, exact evidence snapshot hash and observed times, policy/provider versions, gate outcomes, actor, approval status/expiry, and correlation ID. Drafts are immutable or versioned: an edit creates a new version and invalidates approval. Approval/rejection is an explicit authorized human operation bound to the unchanged draft/evidence/policy hash; it must re-check tenant scope, suppression, legal/consent state, freshness, and provider approval, and record actor, time, reason, before/after status, and audit event. An approved draft still requires a separately authorized future send operation.
|
||||
|
||||
Any future send or other side effect must require an `Idempotency-Key` scoped to tenant, operation, draft version, recipient, provider, and policy fingerprint. Exact retries return the original result; a different request under the same key is rejected. Enforce rate/message/cost caps before attempts and across retries, fallbacks, and workers; use bounded retry/backoff and circuit breaking. Audit draft/gate/approval/provider/cap/suppression events with safe per-item outcomes and redacted payloads. No route may infer completion from request acceptance.
|
||||
|
||||
The Phase 13 implementation remains documentation-only/pilot preparation: there is no draft persistence/API, consent ledger, legal-policy evaluator, configured provider, durable approval queue, delivery adapter, bounce/complaint feedback, secret manager, or production-grade audit/retention workflow in the current runtime. Production requires those components plus DPA/provider and jurisdictional legal review, suppression synchronization, kill switch, rollback/revocation, deletion/legal-hold verification, and integration tests proving no-send default, citation/hash binding, stale/uncertain gate failure, cap enforcement, idempotent replay/conflict rejection, approval expiry, and cross-tenant isolation.
|
||||
|
||||
## Phase 15 deployment contract
|
||||
|
||||
The API image is portable but intentionally small: it runs as a non-root user, writes only `/data`, and initializes the SQLite schema from the image-bundled `schema.sql`. It does not provision Virtualmin, DNS, TLS, a reverse proxy, a secret manager, a durable queue, or a production database. A Virtualmin deployment must provide Docker Engine with Compose v2, a protected checkout, host firewalling, HTTPS termination, a private deployment network, sufficient CPU/RAM/disk, and an encrypted off-host backup destination. Publish the API only where the reverse proxy requires it; preferably expose the web entry point publicly and keep the API port private to the host/network.
|
||||
|
||||
Configuration is environment-only. `APP_ENV`, `LOG_LEVEL`, `CORS_ORIGINS`, and port values may be non-secret deployment settings. `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` are one-time provisioning inputs, must be supplied together through a protected environment/secret store, and must be removed and rotated immediately after bootstrap. Never accept provider credentials from clients or store secrets in source, images, Compose, committed `.env`, logs, traces, metrics, or audit payloads. `AUTOMATED_OUTREACH_ENABLED=false` is the enforced Compose default; no outbound provider or send route exists in this release.
|
||||
|
||||
`GET /api/v1/health/live` is an unauthenticated liveness check and `GET /api/v1/health/ready` is an unauthenticated SQLite readiness check. Operators must gate traffic on Compose `healthy` status plus an authenticated smoke test and migration validation; do not interpret either response as proof that backups, tenant authorization, workers, or external dependencies are ready.
|
||||
|
||||
The current runtime has no standalone migration, backup, restore, retention, or rollback command. Before changing an image or schema, quiesce writes, take and verify an encrypted backup, restore it into an isolated copy, run the API test suite and schema/data invariants, then deploy. Keep the previous image digest and configuration revision available; rollback must restore the application image/config first and only restore data when compatibility and operator approval are established. SQLite remains pilot-scale: no HA, PITR, durable worker lease/recovery, or online migration is implied. See `docs/OPERATIONS.md` and `docs/RELEASE_CHECKLIST.md` for the operator procedure.
|
||||
|
||||
## Remaining limitations and production migration work
|
||||
|
||||
SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Website scanning remains subject to pilot limits: no production egress proxy/isolation, distributed crawl coordinator, hardened resolver, or compliance-grade scan-history retention. Production work must add SSRF/DNS-rebinding/redirect-chain tests, egress policy, authenticated history/cache isolation, budget/abuse enforcement, durable result retention/deletion, observability, reviewed content/robots/terms policy, and the Phase 7 PSL/DNS/availability controls. Redis/Celery/Postgres remain future options, not implemented dependencies.
|
||||
|
||||
## Phase 13 optional AI assistance API contract
|
||||
|
||||
AI assistance is an optional, authenticated, tenant-scoped drafting capability. It may summarize or classify already-held evidence and propose review text; it must not discover facts, verify identity/ownership/deliverability, calculate authoritative score/eligibility, mutate CRM state, merge records, acquire domains, send messages, create campaigns, schedule follow-ups, or perform any autonomous CRM/outreach action. The current Compose runtime does not configure an AI provider or expose a production AI worker; any future route must be explicitly enabled and documented rather than inferred from a provider setting.
|
||||
|
||||
Provider selection is server-side and deny-by-default. A primary provider and optional fallback may be configured only from an allowlist of approved provider IDs. Each provider registration must include capability/purpose, model/version, tenant/data-class scope, processing region and retention terms, timeout/token/request budgets, rate and cost limits, approval owner/expiry, health/circuit state, and operational enablement. Fallback may run only for the same approved purpose and input data class; it must preserve the same tenant scope, redaction policy, evidence set, citation contract, and authority level. Provider credentials are secrets and must never appear in request payloads, prompts, responses, logs, Compose, or committed environment files. Provider outage, timeout, quota, policy rejection, or expired approval returns an explicit `unavailable`/`unknown` result and does not silently invent facts or present invented facts as supported claims.
|
||||
|
||||
Before a request leaves the tenant boundary, minimize and redact data: include only the bounded evidence fields required for the task; remove credentials, tokens, session data, secrets, unnecessary contact values, raw page bodies, and unrelated personal data; and enforce input/output size limits. Store a redacted request/policy fingerprint, not a sensitive prompt. Every suggestion must carry tenant-scoped evidence IDs/citations, a hash of the exact evidence snapshot used, observed/captured times, uncertainty/conflict reasons, provider/model/version, and policy version. Hashes and citations provide reproducible lineage, not truth or independent verification. If evidence is absent, suppressed, stale, conflicting, blocked, or uncertain, preserve that state and return no unsupported claim.
|
||||
|
||||
AI output is untrusted until an authorized human approves it. Approval/rejection must be an explicit tenant-scoped operation with actor, time, reason, output/version, evidence hash/citation set, and before/after value in the audit trail. At approval time re-check authorization, suppression, evidence freshness, provider/policy approval, and hash equality; changed evidence requires a new review. A rejected or expired suggestion must not be applied by retry, fallback, cache, or background work. Approval never converts a citation into proof, consent, deliverability, or outreach permission.
|
||||
|
||||
AI records, prompts/fingerprints, outputs, citations, evidence snapshots, approvals, and audit events require explicit retention classes, deletion/legal-hold behavior, tenant-keyed access, and redacted operational logs. Preserve enough hashed lineage to explain an approved result without retaining unnecessary source content. Cross-tenant business, evidence, suggestion, approval, provider, job, cache, and audit IDs behave as not found. Production requires provider contracts/DPA review, secret isolation, immutable/tamper-evident audit, deletion verification, cost/rate monitoring, prompt-injection and hallucination tests, human-review SLAs, kill-switch procedures, and durable worker/retry semantics; none are implied by the current MVP.
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Evidence-bounded AI assistance primitives for Phase 13.
|
||||
|
||||
This module deliberately has no network or model dependency. The local provider is
|
||||
an auditable formatter over stored records; other providers are reported as
|
||||
not_configured rather than guessed at.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
MAX_INPUT_ITEMS = 100
|
||||
MAX_FIELD_CHARS = 500
|
||||
MAX_OUTPUT_CHARS = 12_000
|
||||
SUPPORTED_KINDS = ("summary", "qualification_explanation", "missing_data_questions", "research_note")
|
||||
_SECRET_RE = re.compile(r"(?i)(password|passwd|secret|token|api[_-]?key|authorization|private[_-]?key|credential)\s*[:=]\s*[^\s,;]+")
|
||||
|
||||
|
||||
def _text(value: Any, limit: int = MAX_FIELD_CHARS) -> str:
|
||||
value = "" if value is None else str(value)
|
||||
value = _SECRET_RE.sub(r"\1: [REDACTED]", value)
|
||||
return value[:limit]
|
||||
|
||||
|
||||
def redact(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
return {str(k)[:80]: ("[REDACTED]" if re.search(r"(?i)(password|passwd|secret|token|api[_-]?key|authorization|private[_-]?key|credential)", str(k)) else redact(v)) for k, v in list(value.items())[:100]}
|
||||
if isinstance(value, list):
|
||||
return [redact(v) for v in value[:MAX_INPUT_ITEMS]]
|
||||
if isinstance(value, str):
|
||||
return _text(value)
|
||||
return value
|
||||
|
||||
|
||||
def evidence_hashes(evidence: list[dict[str, Any]]) -> list[str]:
|
||||
return [hashlib.sha256(json.dumps(redact(item), sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest() for item in evidence]
|
||||
|
||||
|
||||
def _citation(item: dict[str, Any]) -> dict[str, Any]:
|
||||
return {"evidence_id": int(item["id"]), "kind": _text(item.get("kind", "evidence"), 80), "url": _text(item.get("url", ""), 500)}
|
||||
|
||||
|
||||
def _claim(item: dict[str, Any]) -> str:
|
||||
return _text(item.get("claim", ""), MAX_FIELD_CHARS).strip()
|
||||
|
||||
|
||||
def build_local_suggestions(business: dict[str, Any], scans: list[dict[str, Any]], contacts: list[dict[str, Any]], evidence: list[dict[str, Any]], score_history: list[dict[str, Any]] | None = None) -> dict[str, Any]:
|
||||
"""Create deterministic suggestions using only supplied stored data.
|
||||
|
||||
Every claim-bearing item cites one or more rows from ``evidence``. No
|
||||
contact details are emitted, and contacts are used only as aggregate counts.
|
||||
"""
|
||||
evidence = [redact(x) for x in evidence[:MAX_INPUT_ITEMS] if _claim(x)]
|
||||
citations = [_citation(x) for x in evidence]
|
||||
claims = [_claim(x) for x in evidence]
|
||||
suggestions: list[dict[str, Any]] = []
|
||||
name = _text(business.get("name", "this business"), 200)
|
||||
if claims:
|
||||
joined = " ".join(f"{claim} [evidence:{item['id']}]" for claim, item in zip(claims[:5], evidence[:5]))
|
||||
suggestions.append({"type": "summary", "text": f"Stored evidence for {name}: {joined}", "citations": citations[:5]})
|
||||
score = business.get("score")
|
||||
if score is not None:
|
||||
suggestions.append({"type": "qualification_explanation", "text": f"The stored qualification score is {_text(score, 30)}; review the cited evidence before relying on it. [evidence:{evidence[0]['id']}]", "citations": citations[:1]})
|
||||
missing = []
|
||||
if not _text(business.get("website", "")).strip(): missing.append("official website")
|
||||
if not contacts: missing.append("public contact evidence")
|
||||
if missing:
|
||||
suggestions.append({"type": "missing_data_questions", "text": "Confirm whether the following data is available: " + ", ".join(missing) + f". [evidence:{evidence[0]['id']}]", "citations": citations[:1]})
|
||||
suggestions.append({"type": "research_note", "text": f"Draft note: independently verify the stored claims for {name}; do not infer facts beyond the cited records. [evidence:{evidence[0]['id']}]", "citations": citations[:1]})
|
||||
else:
|
||||
# No claim is fabricated. A question is safe but has no citation, so
|
||||
# return no suggestions and let the caller expose the missing-data state.
|
||||
suggestions = []
|
||||
output = {"provider": "local", "version": "deterministic-v1", "suggestions": suggestions, "grounded": True, "claim_policy": "stored_evidence_only"}
|
||||
encoded = json.dumps(output, sort_keys=True, ensure_ascii=False)
|
||||
return json.loads(encoded[:MAX_OUTPUT_CHARS]) if len(encoded) <= MAX_OUTPUT_CHARS else {"provider": "local", "version": "deterministic-v1", "suggestions": suggestions[:1], "grounded": True, "claim_policy": "stored_evidence_only"}
|
||||
|
||||
|
||||
def provider_name() -> str | None:
|
||||
value = os.environ.get("AI_PROVIDER", "").strip().lower()
|
||||
return value or None
|
||||
|
||||
|
||||
def generate(business: dict[str, Any], scans: list[dict[str, Any]], contacts: list[dict[str, Any]], evidence: list[dict[str, Any]], score_history: list[dict[str, Any]] | None = None) -> tuple[str, str, str, dict[str, Any]]:
|
||||
provider = provider_name()
|
||||
hashes = evidence_hashes(evidence)
|
||||
metadata = {"input_counts": {"business": 1, "scans": min(len(scans), MAX_INPUT_ITEMS), "contacts": min(len(contacts), MAX_INPUT_ITEMS), "evidence": min(len(evidence), MAX_INPUT_ITEMS)}, "redacted": True, "max_input_items": MAX_INPUT_ITEMS, "max_field_chars": MAX_FIELD_CHARS, "evidence_hashes": hashes}
|
||||
if provider not in {"local", "deterministic"}:
|
||||
return "not_configured", provider or "", "", metadata
|
||||
return "succeeded", "local", "deterministic-v1", {**metadata, "output": build_local_suggestions(business, scans, contacts, evidence, score_history)}
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Environment configuration validation for portable deployments."""
|
||||
from dataclasses import dataclass
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ConfigError(ValueError):
|
||||
"""Raised when deployment configuration is unsafe or malformed."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Config:
|
||||
app_env: str
|
||||
data_dir: Path
|
||||
session_secret: str
|
||||
outreach_enabled: bool
|
||||
log_level: str
|
||||
|
||||
|
||||
def _env(values, key, default=""):
|
||||
return str(values.get(key, default) or "").strip()
|
||||
|
||||
|
||||
def load_config(values=None):
|
||||
values = os.environ if values is None else values
|
||||
app_env = _env(values, "APP_ENV", "development").lower()
|
||||
if app_env not in {"development", "test", "staging", "production"}:
|
||||
raise ConfigError("APP_ENV must be development, test, staging, or production")
|
||||
data_dir = Path(_env(values, "DATA_DIR", ".") or ".").expanduser()
|
||||
if not data_dir.is_absolute():
|
||||
data_dir = (Path.cwd() / data_dir).resolve()
|
||||
if data_dir.exists() and not data_dir.is_dir():
|
||||
raise ConfigError("DATA_DIR must be a directory")
|
||||
secret = _env(values, "SESSION_SECRET")
|
||||
if app_env == "production" and (len(secret) < 32 or secret.lower() in {"change-me", "development", "dev"}):
|
||||
raise ConfigError("SESSION_SECRET must be at least 32 characters in production")
|
||||
outreach = _env(values, "AUTOMATED_OUTREACH_ENABLED", "false").lower()
|
||||
if outreach not in {"", "0", "false", "no", "off"}:
|
||||
raise ConfigError("automated outreach is disabled in this release")
|
||||
log_level = _env(values, "LOG_LEVEL", "INFO").upper()
|
||||
if log_level not in {"QUIET", "ERROR", "WARNING", "INFO", "DEBUG"}:
|
||||
raise ConfigError("LOG_LEVEL is invalid")
|
||||
return Config(app_env, data_dir, secret, False, log_level)
|
||||
@@ -0,0 +1,139 @@
|
||||
"""Conservative extraction of public business contact signals from approved HTML.
|
||||
|
||||
This module never fetches URLs. Callers must provide HTML obtained from the approved
|
||||
website scanner and a source URL already associated with the business.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import html as html_lib
|
||||
import re
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
MAX_HTML_BYTES = 512 * 1024
|
||||
MAX_RESULTS = 100
|
||||
EMAIL_RE = re.compile(r"(?<![\w.+-])[A-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?(?:\.[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?)+", re.I)
|
||||
PHONE_RE = re.compile(r"(?<!\w)(\+?\d[\d ()().-]{6,}\d)(?!\w)")
|
||||
SOCIAL_HOSTS = {"facebook.com", "instagram.com", "linkedin.com", "twitter.com", "x.com", "youtube.com", "tiktok.com"}
|
||||
FREE_MAIL = {"gmail.com", "googlemail.com", "yahoo.com", "yahoo.co.uk", "hotmail.com", "outlook.com", "live.com", "icloud.com", "aol.com", "proton.me", "protonmail.com", "mail.com"}
|
||||
ROLE_NAMES = {"info", "hello", "contact", "sales", "support", "admin", "office", "enquiries", "inquiries", "accounts", "billing", "careers", "hr", "help", "marketing", "bookings", "reception", "service", "customerservice"}
|
||||
EXAMPLE_DOMAINS = {"example.com", "example.org", "example.net", "example.test", "invalid", "localhost"}
|
||||
|
||||
|
||||
def _clean(value: str) -> str:
|
||||
return re.sub(r"\s+", " ", html_lib.unescape(value or "")).strip()
|
||||
|
||||
|
||||
def _decode_obfuscation(value: str) -> str:
|
||||
value = html_lib.unescape(unquote(value or ""))
|
||||
value = re.sub(r"\s*(?:\[|\(|\{|\s)at(?:\]|\)|\}|\s)\s*", "@", value, flags=re.I)
|
||||
value = re.sub(r"\s*(?:\[|\(|\{|\s)dot(?:\]|\)|\}|\s)*", ".", value, flags=re.I)
|
||||
return value
|
||||
|
||||
|
||||
def valid_email(value: str) -> bool:
|
||||
value = value.strip().lower()
|
||||
if len(value) > 254 or value.count("@") != 1 or value.split("@", 1)[1] in EXAMPLE_DOMAINS:
|
||||
return False
|
||||
return bool(EMAIL_RE.fullmatch(value)) and not any(x in value for x in ("password", "token", "secret", "credential", "apikey"))
|
||||
|
||||
|
||||
def normalize_phone(value: str) -> str:
|
||||
value = value.strip()
|
||||
digits = re.sub(r"\D", "", value)
|
||||
if value.startswith("+") and digits:
|
||||
return "+" + digits
|
||||
return digits
|
||||
|
||||
|
||||
class _ContactParser(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.text: list[str] = []
|
||||
self.links: list[tuple[str, str]] = []
|
||||
self.form_fields: list[str] = []
|
||||
self._ignore = 0
|
||||
self._anchor = ""
|
||||
self._in_form = False
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
attrs = {str(k).lower(): str(v or "") for k, v in attrs}
|
||||
tag = tag.lower()
|
||||
if tag in {"script", "style", "noscript", "template", "svg"}:
|
||||
self._ignore += 1
|
||||
if tag == "a": self._anchor = attrs.get("href", "")
|
||||
if tag == "form": self._in_form = True
|
||||
if self._in_form and tag in {"input", "textarea", "select"}:
|
||||
name = attrs.get("name", "") or attrs.get("id", "")
|
||||
typ = attrs.get("type", "")
|
||||
self.form_fields.append(_clean(" ".join((name, typ, attrs.get("placeholder", "")))))
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
tag = tag.lower()
|
||||
if tag in {"script", "style", "noscript", "template", "svg"} and self._ignore: self._ignore -= 1
|
||||
if tag == "a": self._anchor = ""
|
||||
if tag == "form": self._in_form = False
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._ignore: return
|
||||
if data.strip(): self.text.append(data)
|
||||
if self._anchor: self.links.append((self._anchor, data))
|
||||
|
||||
|
||||
def _record(kind, value, label, source_url, confidence, classification="unknown", *, suppressed=False, provenance="visible_text"):
|
||||
return {"kind": kind, "value": value, "label": label[:200], "classification": classification,
|
||||
"confidence": round(max(0.0, min(1.0, confidence)), 2), "source_url": source_url,
|
||||
"public_business": True, "mx_status": "unknown", "suppressed": bool(suppressed),
|
||||
"do_not_contact": bool(suppressed), "provenance": provenance}
|
||||
|
||||
|
||||
def extract_contacts(source_html: str, source_url: str, *, suppressions=None, max_results=MAX_RESULTS) -> list[dict]:
|
||||
if not isinstance(source_html, str) or len(source_html.encode("utf-8")) > MAX_HTML_BYTES:
|
||||
raise ValueError("html_too_large")
|
||||
try: max_results = int(max_results)
|
||||
except (ValueError, TypeError): raise ValueError("invalid_limits")
|
||||
if max_results < 1 or max_results > MAX_RESULTS: raise ValueError("invalid_limits")
|
||||
parser = _ContactParser(); parser.feed(source_html)
|
||||
visible = _decode_obfuscation(_clean(" ".join(parser.text)))
|
||||
suppression = {(str(x.get("kind", "")), str(x.get("value", "")).lower()) for x in (suppressions or [])}
|
||||
out, seen = [], set()
|
||||
def add(kind, value, label, confidence, classification="unknown", provenance="visible_text"):
|
||||
value = value.strip().lower() if kind == "email" else value.strip()
|
||||
if kind == "email":
|
||||
if not valid_email(value): return
|
||||
local, domain = value.rsplit("@", 1)
|
||||
classification = "free_mail" if domain in FREE_MAIL else ("role" if local in ROLE_NAMES else "named")
|
||||
key = (kind, value); blocked = ("email", value) in suppression or ("domain", domain) in suppression
|
||||
else:
|
||||
if kind in {"phone", "whatsapp"}: value = normalize_phone(value)
|
||||
if len(re.sub(r"\D", "", value)) < 7: return
|
||||
key = (kind, value); blocked = (kind, value.lower()) in suppression
|
||||
if key in seen or len(out) >= max_results: return
|
||||
seen.add(key); out.append(_record(kind, value, label or kind.title(), source_url, confidence, classification, suppressed=blocked, provenance=provenance))
|
||||
for href, anchor_text in parser.links:
|
||||
raw = _decode_obfuscation(href)
|
||||
if raw.lower().startswith("mailto:"):
|
||||
address = raw[7:].split("?", 1)[0]
|
||||
add("email", address, _clean(anchor_text), 0.98, provenance="mailto")
|
||||
elif raw.lower().startswith("tel:"):
|
||||
add("phone", raw[4:].split("?", 1)[0], _clean(anchor_text), 0.98, provenance="tel")
|
||||
else:
|
||||
parsed = urlparse(raw if "://" in raw else "https://" + raw)
|
||||
host = (parsed.hostname or "").lower().removeprefix("www.")
|
||||
if host == "wa.me" or "whatsapp" in host:
|
||||
number = re.sub(r"\D", "", parsed.path)
|
||||
if number: add("whatsapp", "+" + number, _clean(anchor_text) or "WhatsApp", 0.98, provenance="whatsapp_link")
|
||||
elif any(host == d or host.endswith("." + d) for d in SOCIAL_HOSTS):
|
||||
add("social", raw, _clean(anchor_text) or host, 0.95, provenance="social_link")
|
||||
for match in EMAIL_RE.finditer(visible):
|
||||
context = visible[max(0, match.start() - 40):match.start()]
|
||||
if re.search(r"(?:password|passwd|token|secret|credential|api[_ -]?key|authorization)\s*[:=]?\s*$", context, re.I):
|
||||
continue
|
||||
add("email", match.group(0), "Email", 0.88)
|
||||
for value in PHONE_RE.findall(visible): add("phone", value, "Phone", 0.82)
|
||||
# A form is provenance, not proof of a destination address.
|
||||
if parser.form_fields:
|
||||
for field in parser.form_fields[:5]:
|
||||
if re.search(r"email|contact|phone|whatsapp", field, re.I):
|
||||
add("form", field, "Contact form", 0.7, provenance="form_field")
|
||||
return out
|
||||
+739
-13
@@ -11,11 +11,19 @@ if __package__ in (None, ""):
|
||||
from app.sources import adapter_for, contains_secret
|
||||
from app.domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
from app.website_scanner import scan_website, validate_url
|
||||
from app.contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS
|
||||
from app.scoring import DEFAULT_RULES, signals_for_business, evaluate_score, SCORE_VERSION
|
||||
from app.ai_assistance import generate as generate_ai, MAX_INPUT_ITEMS, MAX_OUTPUT_CHARS
|
||||
from app.config import load_config
|
||||
else:
|
||||
from .domain import deduplication_key, deduplicate_businesses, is_suppressed, normalize_business, score_business, normalize_domain, normalize_phone, match_businesses
|
||||
from .sources import adapter_for, contains_secret
|
||||
from .domain_intelligence import normalize_registrable_domain, resolve_domain, generate_candidate_domains
|
||||
from .website_scanner import scan_website, validate_url
|
||||
from .contact_extractor import extract_contacts, MAX_HTML_BYTES, MAX_RESULTS
|
||||
from .scoring import DEFAULT_RULES, signals_for_business, evaluate_score, SCORE_VERSION
|
||||
from .ai_assistance import generate as generate_ai, MAX_INPUT_ITEMS, MAX_OUTPUT_CHARS
|
||||
from .config import load_config
|
||||
ORGANIZATION_ID = "demo-tenant"
|
||||
SCHEMA = Path(__file__).resolve().parents[1] / "schema.sql"
|
||||
SESSION_DAYS = 7
|
||||
@@ -25,6 +33,7 @@ JOB_TYPES = {"noop", "prospect_recalculate", "source_discovery", "domain_check"}
|
||||
JOB_PAGE_SIZE = 100
|
||||
WEBSITE_SCAN_PAGE_SIZE = 100
|
||||
WEBSITE_SCAN_CACHE_SECONDS = 3600
|
||||
CONTACT_EXTRACTION_PAGE_SIZE = 100
|
||||
SECRET_KEYS = {"password", "passwd", "secret", "token", "api_key", "apikey", "authorization", "credential", "private_key"}
|
||||
CHILD_TABLES = {"contacts": ("name", "email", "phone", "title", "do_not_contact"), "domains": ("domain", "kind"), "websites": ("url", "website_class"), "evidence": ("kind", "url", "claim"), "notes": ("body",)}
|
||||
|
||||
@@ -50,10 +59,24 @@ def connect(db_path: str) -> sqlite3.Connection:
|
||||
db = sqlite3.connect(db_path); db.row_factory = sqlite3.Row; db.execute("PRAGMA foreign_keys = ON"); db.executescript(SCHEMA.read_text())
|
||||
# Upgrade databases created by Phase 1/2 without destroying data.
|
||||
cols = {r[1] for r in db.execute("PRAGMA table_info(businesses)")}
|
||||
for col, definition in (("verified", "INTEGER NOT NULL DEFAULT 0"), ("verified_at", "TEXT"), ("updated_at", "TEXT"), ("province", "TEXT NOT NULL DEFAULT ''"), ("city", "TEXT NOT NULL DEFAULT ''"), ("suburb", "TEXT NOT NULL DEFAULT ''"), ("merge_status", "TEXT NOT NULL DEFAULT 'active'"), ("merged_into_id", "INTEGER")):
|
||||
for col, definition in (("verified", "INTEGER NOT NULL DEFAULT 0"), ("verified_at", "TEXT"), ("updated_at", "TEXT"), ("province", "TEXT NOT NULL DEFAULT ''"), ("city", "TEXT NOT NULL DEFAULT ''"), ("suburb", "TEXT NOT NULL DEFAULT ''"), ("merge_status", "TEXT NOT NULL DEFAULT 'active'"), ("merged_into_id", "INTEGER"), ("review_status", "TEXT NOT NULL DEFAULT 'pending'"), ("assigned_to", "TEXT NOT NULL DEFAULT ''"), ("review_metadata_json", "TEXT NOT NULL DEFAULT '{}'")):
|
||||
if col not in cols: db.execute(f"ALTER TABLE businesses ADD COLUMN {col} {definition}")
|
||||
db.execute("UPDATE businesses SET updated_at=COALESCE(updated_at,created_at) WHERE updated_at IS NULL")
|
||||
# Phase 12 is additive-safe for databases created before CRM metadata existed.
|
||||
for table, additions in {
|
||||
"pipeline_entries": (("notes", "TEXT NOT NULL DEFAULT ''"), ("next_action", "TEXT NOT NULL DEFAULT ''"), ("follow_up_at", "TEXT"), ("actor_user_id", "INTEGER"), ("idempotency_key", "TEXT"), ("version", "INTEGER NOT NULL DEFAULT 1")),
|
||||
"interactions": (("outcome", "TEXT NOT NULL DEFAULT 'other'"), ("notes", "TEXT NOT NULL DEFAULT ''"), ("next_action", "TEXT NOT NULL DEFAULT ''"), ("follow_up_at", "TEXT"), ("actor_user_id", "INTEGER"), ("idempotency_key", "TEXT")),
|
||||
"suppressions": (("active", "INTEGER NOT NULL DEFAULT 1"), ("updated_at", "TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP"), ("actor_user_id", "INTEGER")),
|
||||
}.items():
|
||||
existing = {r[1] for r in db.execute(f"PRAGMA table_info({table})")}
|
||||
for col, definition in additions:
|
||||
if col not in existing: db.execute(f"ALTER TABLE {table} ADD COLUMN {col} {definition}")
|
||||
db.execute("CREATE UNIQUE INDEX IF NOT EXISTS uq_pipeline_idempotency ON pipeline_entries(organization_id,idempotency_key) WHERE idempotency_key IS NOT NULL AND idempotency_key <> ''")
|
||||
db.execute("CREATE UNIQUE INDEX IF NOT EXISTS uq_interaction_idempotency ON interactions(organization_id,idempotency_key) WHERE idempotency_key IS NOT NULL AND idempotency_key <> ''")
|
||||
db.execute("INSERT OR IGNORE INTO organizations (id,name) VALUES (?,?)", (ORGANIZATION_ID, "Demo organization"))
|
||||
for organization in db.execute("SELECT id FROM organizations").fetchall():
|
||||
for rule in DEFAULT_RULES:
|
||||
db.execute("INSERT OR IGNORE INTO score_rules(organization_id,code,name,description,condition_json,points,max_applications,enabled,version) VALUES(?,?,?,?,?,?,?,?,?)", (organization["id"], rule["code"], rule["name"], rule["description"], json.dumps(rule["condition_json"], sort_keys=True), rule["points"], rule["max_applications"], rule["enabled"], rule["version"]))
|
||||
email, password = os.environ.get("BOOTSTRAP_ADMIN_EMAIL"), os.environ.get("BOOTSTRAP_ADMIN_PASSWORD")
|
||||
if email and password and not db.execute("SELECT id FROM users WHERE email=?", (email.strip().lower(),)).fetchone():
|
||||
ph, salt = hash_password(password); db.execute("INSERT INTO users (organization_id,email,password_hash,password_salt,role) VALUES (?,?,?,?,?)", (ORGANIZATION_ID,email.strip().lower(),ph,salt,"owner"))
|
||||
@@ -98,14 +121,415 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
db.execute("INSERT INTO audit_log (organization_id,user_id,action,details) VALUES (?,?,?,?)",(user["organization_id"],user["id"],action,details))
|
||||
def business(self, db, ident, org): return db.execute("SELECT * FROM businesses WHERE id=? AND organization_id=?",(ident,org)).fetchone()
|
||||
def nested(self, db, bid, org):
|
||||
result={"contacts":[],"domains":[],"websites":[],"evidence":[],"pipeline":[],"notes":[]}
|
||||
tables={"contacts":"contacts","domains":"domains","websites":"websites","evidence":"evidence","pipeline":"pipeline_entries","notes":"notes"}
|
||||
result={"contacts":[],"domains":[],"websites":[],"evidence":[],"pipeline":[],"interactions":[],"notes":[]}
|
||||
tables={"contacts":"contacts","domains":"domains","websites":"websites","evidence":"evidence","pipeline":"pipeline_entries","interactions":"interactions","notes":"notes"}
|
||||
for key, table in tables.items():
|
||||
result[key]=[row_json(r) for r in db.execute(f"SELECT * FROM {table} WHERE business_id=? AND organization_id=? ORDER BY id",(bid,org))]
|
||||
return result
|
||||
def _bounded_filters(self, value):
|
||||
def walk(item, depth=0):
|
||||
if depth > 4: raise ValueError("filters_too_deep")
|
||||
if isinstance(item, dict):
|
||||
if len(item) > 30: raise ValueError("filters_too_large")
|
||||
return {str(k)[:80]: walk(v, depth + 1) for k, v in item.items()}
|
||||
if isinstance(item, list):
|
||||
if len(item) > 50: raise ValueError("filters_too_large")
|
||||
return [walk(v, depth + 1) for v in item]
|
||||
if isinstance(item, str):
|
||||
if len(item) > 500: raise ValueError("filter_value_too_large")
|
||||
return item
|
||||
if item is None or isinstance(item, (bool, int, float)):
|
||||
return item
|
||||
raise ValueError("invalid_filters")
|
||||
if not isinstance(value, dict): raise ValueError("invalid_filters")
|
||||
result = walk(value)
|
||||
if len(json.dumps(result, separators=(",", ":"), ensure_ascii=False).encode()) > 8192: raise ValueError("filters_too_large")
|
||||
return result
|
||||
|
||||
def _saved_filter_json(self, row):
|
||||
item = row_json(row)
|
||||
try: item["filters"] = json.loads(item.pop("filters_json") or "{}")
|
||||
except (TypeError, ValueError): item["filters"] = {}
|
||||
return item
|
||||
|
||||
def list_saved_filters(self, db, user):
|
||||
rows = db.execute("SELECT * FROM saved_filters WHERE organization_id=? AND user_id=? ORDER BY updated_at DESC,id DESC", (user["organization_id"], user["id"])).fetchall()
|
||||
return self.send_json(200, {"organization_id": user["organization_id"], "items": [self._saved_filter_json(r) for r in rows]})
|
||||
|
||||
def save_filter(self, payload, db, user):
|
||||
name = str(payload.get("name", "")).strip()
|
||||
raw = payload.get("filters", payload.get("filter", payload.get("filters_json", {})))
|
||||
if not name or len(name) > 120: return self.send_json(400, {"error": "invalid_saved_filter"})
|
||||
try: filters = self._bounded_filters(raw)
|
||||
except ValueError as exc: return self.send_json(400, {"error": str(exc)})
|
||||
try:
|
||||
cur = db.execute("INSERT INTO saved_filters(organization_id,user_id,name,filters_json) VALUES(?,?,?,?)", (user["organization_id"], user["id"], name, json.dumps(filters, sort_keys=True, separators=(",", ":"))))
|
||||
except sqlite3.IntegrityError: return self.send_json(409, {"error": "duplicate_saved_filter"})
|
||||
self.audit(db, user, "saved_filter.created", str(cur.lastrowid)); db.commit()
|
||||
return self.send_json(201, self._saved_filter_json(db.execute("SELECT * FROM saved_filters WHERE id=?", (cur.lastrowid,)).fetchone()))
|
||||
|
||||
def update_saved_filter(self, fid, payload, db, user):
|
||||
row = db.execute("SELECT * FROM saved_filters WHERE id=? AND organization_id=? AND user_id=?", (fid, user["organization_id"], user["id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
fields = []; values = []
|
||||
if "name" in payload:
|
||||
name = str(payload["name"]).strip()
|
||||
if not name or len(name) > 120: return self.send_json(400, {"error": "invalid_saved_filter"})
|
||||
fields.append("name=?"); values.append(name)
|
||||
if any(k in payload for k in ("filters", "filter", "filters_json")):
|
||||
try: filters = self._bounded_filters(payload.get("filters", payload.get("filter", payload.get("filters_json"))))
|
||||
except ValueError as exc: return self.send_json(400, {"error": str(exc)})
|
||||
fields.append("filters_json=?"); values.append(json.dumps(filters, sort_keys=True, separators=(",", ":")))
|
||||
if not fields: return self.send_json(400, {"error": "no_changes"})
|
||||
values += [fid, user["organization_id"], user["id"]]
|
||||
try: db.execute("UPDATE saved_filters SET " + ",".join(fields) + ",updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=? AND user_id=?", values)
|
||||
except sqlite3.IntegrityError: return self.send_json(409, {"error": "duplicate_saved_filter"})
|
||||
self.audit(db, user, "saved_filter.updated", str(fid)); db.commit()
|
||||
return self.send_json(200, self._saved_filter_json(db.execute("SELECT * FROM saved_filters WHERE id=?", (fid,)).fetchone()))
|
||||
|
||||
def delete_saved_filter(self, fid, db, user):
|
||||
row = db.execute("SELECT id FROM saved_filters WHERE id=? AND organization_id=? AND user_id=?", (fid, user["organization_id"], user["id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
db.execute("DELETE FROM saved_filters WHERE id=? AND organization_id=? AND user_id=?", (fid, user["organization_id"], user["id"]))
|
||||
self.audit(db, user, "saved_filter.deleted", str(fid)); db.commit()
|
||||
return self.send_json(200, {"ok": True, "id": fid})
|
||||
|
||||
def _review_item(self, row, db, org):
|
||||
item = row_json(row)
|
||||
suppressed = is_suppressed(dict(row), [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1", (org,))])
|
||||
merged = row["merge_status"] == "merged"
|
||||
item.update({"suppressed": bool(suppressed), "merged": bool(merged), "outreach_eligible": not suppressed and not merged and row["review_status"] not in ("rejected",), "review_flags": [x for x, yes in (("suppressed", suppressed), ("merged", merged), ("rejected", row["review_status"] == "rejected")) if yes]})
|
||||
return item
|
||||
|
||||
def review_queue(self, db, org, query):
|
||||
try:
|
||||
limit = int((query.get("page_size") or [50])[0]); offset = int((query.get("offset") or [0])[0])
|
||||
if limit < 1 or limit > 100 or offset < 0: raise ValueError
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
|
||||
where, params = ["b.organization_id=?"], [org]
|
||||
status = (query.get("review_status") or query.get("status") or [""])[0].strip()
|
||||
if status:
|
||||
if status not in {"pending", "verified", "rejected", "assigned"}: return self.send_json(400, {"error": "invalid_review_status"})
|
||||
where.append("b.review_status=?"); params.append(status)
|
||||
for key, op in (("score_min", ">="), ("score_max", "<=")):
|
||||
raw = (query.get(key) or [""])[0]
|
||||
if raw:
|
||||
try: value = int(raw)
|
||||
except ValueError: return self.send_json(400, {"error": "invalid_score"})
|
||||
if value < 0 or value > 100: return self.send_json(400, {"error": "invalid_score"})
|
||||
where.append("b.score" + op + "?"); params.append(value)
|
||||
priority = (query.get("priority") or query.get("priority_band") or [""])[0].strip()
|
||||
if priority:
|
||||
bounds = {"high": (70, 100), "medium": (40, 69), "low": (0, 39)}
|
||||
if priority not in bounds: return self.send_json(400, {"error": "invalid_priority"})
|
||||
where += ["b.score BETWEEN ? AND ?"]; params += list(bounds[priority])
|
||||
website = (query.get("website_state") or query.get("website_class") or [""])[0].strip()
|
||||
if website: where.append("b.website_class=?"); params.append(website)
|
||||
freshness = (query.get("freshness_days") or [""])[0]
|
||||
if freshness:
|
||||
try: days = int(freshness)
|
||||
except ValueError: return self.send_json(400, {"error": "invalid_freshness"})
|
||||
if days < 0 or days > 3650: return self.send_json(400, {"error": "invalid_freshness"})
|
||||
where.append("b.updated_at >= datetime('now', ?)"); params.append(f"-{days} days")
|
||||
source_health = (query.get("source_health") or [""])[0].strip()
|
||||
if source_health:
|
||||
where.append("EXISTS (SELECT 1 FROM sources s WHERE s.organization_id=b.organization_id AND s.health_status=?)"); params.append(source_health)
|
||||
rows = db.execute("SELECT b.* FROM businesses b WHERE " + " AND ".join(where) + " ORDER BY b.score DESC,b.updated_at DESC,b.id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
|
||||
return self.send_json(200, {"organization_id": org, "items": [self._review_item(r, db, org) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
OUTCOMES = {"connected", "no_answer", "left_message", "meeting_booked", "meeting_held", "qualified", "disqualified", "won", "lost", "other"}
|
||||
STAGES = {"new", "contacted", "qualified", "proposal", "negotiation", "won", "lost"}
|
||||
|
||||
def _crm_suppressed(self, db, org, business):
|
||||
return is_suppressed(dict(business), [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1", (org,))])
|
||||
|
||||
def _date_where(self, query, column, params):
|
||||
start = (query.get("from") or query.get("start") or [""])[0]
|
||||
end = (query.get("to") or query.get("end") or [""])[0]
|
||||
for value in (start, end):
|
||||
if value and (len(value) > 30 or not re.match(r"^\\d{4}-\\d{2}-\\d{2}(?:T[^ ]*)?$", value)):
|
||||
raise ValueError("invalid_date")
|
||||
if start and end:
|
||||
try:
|
||||
left=datetime.fromisoformat(start.replace("Z","+00:00")).date(); right=datetime.fromisoformat(end.replace("Z","+00:00")).date()
|
||||
if (right-left).days > 366: raise ValueError("date_range_too_large")
|
||||
except ValueError as exc:
|
||||
if str(exc) == "date_range_too_large": raise
|
||||
raise ValueError("invalid_date_range")
|
||||
if start: params.append(start); clause = f"{column}>=?"
|
||||
else: clause = ""
|
||||
if end: params.append(end); clause += (" AND " if clause else "") + f"{column}<=?"
|
||||
return clause
|
||||
|
||||
def list_pipeline(self, db, org, query):
|
||||
params=[org]; where=["p.organization_id=?"]
|
||||
bid=(query.get("business_id") or [""])[0]
|
||||
if bid.isdigit(): where.append("p.business_id=?"); params.append(int(bid))
|
||||
stage=(query.get("stage") or [""])[0]
|
||||
if stage: where.append("p.stage=?"); params.append(stage)
|
||||
rows=db.execute("SELECT p.* FROM pipeline_entries p WHERE " + " AND ".join(where) + " ORDER BY p.updated_at DESC,p.id DESC",params).fetchall()
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in rows]})
|
||||
|
||||
def list_stages(self, db, org):
|
||||
rows=db.execute("SELECT * FROM pipeline_stages WHERE organization_id=? AND active=1 ORDER BY position,id",(org,)).fetchall()
|
||||
if not rows: return self.send_json(200,{"organization_id":org,"items":[{"name":x,"position":i,"active":True} for i,x in enumerate(("new","contacted","qualified","proposal","negotiation","won","lost"))]})
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in rows]})
|
||||
|
||||
def list_interactions(self, db, org, query):
|
||||
params=[org]; where=["i.organization_id=?"]
|
||||
bid=(query.get("business_id") or [""])[0]
|
||||
if bid.isdigit(): where.append("i.business_id=?"); params.append(int(bid))
|
||||
outcome=(query.get("outcome") or [""])[0]
|
||||
if outcome: where.append("i.outcome=?"); params.append(outcome)
|
||||
rows=db.execute("SELECT i.* FROM interactions i WHERE " + " AND ".join(where) + " ORDER BY i.created_at DESC,i.id DESC",params).fetchall()
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in rows]})
|
||||
|
||||
def list_suppressions(self, db, org, query):
|
||||
rows=db.execute("SELECT * FROM suppressions WHERE organization_id=? ORDER BY id DESC",(org,)).fetchall()
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in rows]})
|
||||
|
||||
def report(self, db, org, kind, query):
|
||||
try:
|
||||
if kind == "pipeline":
|
||||
params=[org]; clause=self._date_where(query,"p.created_at",params); sql="SELECT p.stage, p.status, COUNT(*) count FROM pipeline_entries p WHERE p.organization_id=?" + ((" AND "+clause) if clause else "") + " GROUP BY p.stage,p.status ORDER BY p.stage,p.status"
|
||||
elif kind == "outcomes":
|
||||
params=[org]; clause=self._date_where(query,"i.created_at",params); sql="SELECT i.outcome, COUNT(*) count FROM interactions i WHERE i.organization_id=?" + ((" AND "+clause) if clause else "") + " GROUP BY i.outcome ORDER BY i.outcome"
|
||||
else:
|
||||
params=[org]; clause=self._date_where(query,"activity_at",params); sql="SELECT activity_type, COUNT(*) count FROM (SELECT 'pipeline' activity_type, created_at activity_at FROM pipeline_entries WHERE organization_id=? UNION ALL SELECT 'interaction',created_at FROM interactions WHERE organization_id=?) WHERE 1=1" + ((" AND "+clause) if clause else "") + " GROUP BY activity_type"
|
||||
params=[org,org] + params[1:]
|
||||
rows=db.execute(sql,params).fetchall(); return self.send_json(200,{"organization_id":org,"items":[dict(r) for r in rows]})
|
||||
except ValueError as exc: return self.send_json(400,{"error":str(exc)})
|
||||
|
||||
def _ai_run_json(self, row, suggestions=None):
|
||||
item = row_json(row)
|
||||
for key in ("input_evidence_hashes_json", "prompt_metadata_json", "data_minimization_json", "output_json"):
|
||||
source = item.pop(key, "{}" if key != "input_evidence_hashes_json" else "[]")
|
||||
try: item[key[:-5] if key.endswith("_json") else key] = json.loads(source or ("{}" if key != "input_evidence_hashes_json" else "[]"))
|
||||
except (TypeError, ValueError): item[key[:-5] if key.endswith("_json") else key] = {} if key != "input_evidence_hashes_json" else []
|
||||
if suggestions is not None: item["suggestions"] = suggestions
|
||||
return item
|
||||
|
||||
# Phase 14: outreach is a reviewable preparation workflow only. There is
|
||||
# deliberately no provider client in this service and send never performs
|
||||
# network I/O.
|
||||
OUTREACH_PROVIDERS = {"smtp", "sendgrid", "twilio", "whatsapp"}
|
||||
OUTREACH_KINDS = {"email", "phone", "whatsapp"}
|
||||
OUTREACH_DAILY_CAP = 100
|
||||
OUTREACH_BATCH_CAP = 25
|
||||
|
||||
def _provider_json(self, row, org):
|
||||
if not row:
|
||||
return {"organization_id": org, "provider": "", "enabled": False,
|
||||
"policy": {"consent_required": True}, "daily_cap": self.OUTREACH_DAILY_CAP,
|
||||
"batch_cap": self.OUTREACH_BATCH_CAP}
|
||||
item = {"id": row["id"], "organization_id": org, "provider": row["provider"],
|
||||
"enabled": bool(row["enabled"]), "daily_cap": row["daily_cap"],
|
||||
"batch_cap": row["batch_cap"]}
|
||||
try: item["policy"] = json.loads(row["policy_json"] or "{}")
|
||||
except (TypeError, ValueError): item["policy"] = {"consent_required": True}
|
||||
return item
|
||||
|
||||
def provider_config(self, db, user, payload=None):
|
||||
org = user["organization_id"]
|
||||
row = db.execute("SELECT * FROM outreach_provider_configs WHERE organization_id=?", (org,)).fetchone()
|
||||
if payload is None:
|
||||
return self.send_json(200, self._provider_json(row, org))
|
||||
provider = str(payload.get("provider", row["provider"] if row else "")).strip().lower()
|
||||
if provider and provider not in self.OUTREACH_PROVIDERS: return self.send_json(400, {"error": "invalid_provider"})
|
||||
enabled = bool(payload.get("enabled", bool(row["enabled"]) if row else False))
|
||||
policy = payload.get("policy", payload.get("legal_policy", {} if not row else None))
|
||||
if policy is None:
|
||||
try: policy = json.loads(row["policy_json"] or "{}")
|
||||
except (TypeError, ValueError): policy = {"consent_required": True}
|
||||
if not isinstance(policy, dict) or len(policy) > 20: return self.send_json(400, {"error": "invalid_policy"})
|
||||
policy = {str(k)[:80]: v for k, v in policy.items()}
|
||||
policy.setdefault("consent_required", True)
|
||||
try:
|
||||
daily = int(payload.get("daily_cap", row["daily_cap"] if row else self.OUTREACH_DAILY_CAP)); batch = int(payload.get("batch_cap", row["batch_cap"] if row else self.OUTREACH_BATCH_CAP))
|
||||
except (TypeError, ValueError): return self.send_json(400, {"error": "invalid_limits"})
|
||||
if daily < 1 or daily > self.OUTREACH_DAILY_CAP or batch < 1 or batch > self.OUTREACH_BATCH_CAP: return self.send_json(400, {"error": "invalid_limits"})
|
||||
secret = payload.get("secret", None)
|
||||
fingerprint = row["secret_fingerprint"] if row else ""
|
||||
if secret is not None:
|
||||
if not isinstance(secret, str) or not secret or len(secret) > 4096: return self.send_json(400, {"error": "invalid_secret"})
|
||||
fingerprint = hashlib.sha256(secret.encode()).hexdigest()
|
||||
if enabled and (not provider or not fingerprint): return self.send_json(400, {"error": "provider_credentials_required"})
|
||||
if row:
|
||||
db.execute("UPDATE outreach_provider_configs SET provider=?,enabled=?,secret_fingerprint=?,policy_json=?,daily_cap=?,batch_cap=?,updated_at=CURRENT_TIMESTAMP WHERE organization_id=?", (provider, int(enabled), fingerprint, json.dumps(policy, sort_keys=True), daily, batch, org))
|
||||
else:
|
||||
db.execute("INSERT INTO outreach_provider_configs(organization_id,provider,enabled,secret_fingerprint,policy_json,daily_cap,batch_cap) VALUES(?,?,?,?,?,?,?)", (org, provider, int(enabled), fingerprint, json.dumps(policy, sort_keys=True), daily, batch))
|
||||
self.audit(db, user, "outreach.provider_config.updated", provider or "disabled"); db.commit()
|
||||
return self.send_json(200, self._provider_json(db.execute("SELECT * FROM outreach_provider_configs WHERE organization_id=?", (org,)).fetchone(), org))
|
||||
|
||||
def _draft_json(self, row):
|
||||
item = row_json(row)
|
||||
for field, default in (("template_json", {}), ("citations_json", []), ("provenance_json", {})):
|
||||
key = field[:-5]
|
||||
try: item[key] = json.loads(item.pop(field) or json.dumps(default))
|
||||
except (TypeError, ValueError): item[key] = default
|
||||
item["target_verified"] = bool(item.get("target_verified")); item["consent_confirmed"] = bool(item.get("consent_confirmed"))
|
||||
item["target"] = {"kind": item.pop("target_kind"), "value": item.pop("target_value"), "verified": item["target_verified"]}
|
||||
return item
|
||||
|
||||
def _template(self, text, business, evidence):
|
||||
pattern = re.compile(r"{{\s*([^}]+?)\s*}}")
|
||||
used = []
|
||||
def replace(match):
|
||||
token = match.group(1).strip()
|
||||
if token == "business.name": return str(business["name"])
|
||||
m = re.fullmatch(r"evidence\.(\d+)\.claim", token)
|
||||
if m:
|
||||
index = int(m.group(1));
|
||||
if index < 1 or index > len(evidence): raise ValueError("unsupported_template_variable")
|
||||
used.append(evidence[index - 1]["id"]); return str(evidence[index - 1]["claim"])
|
||||
raise ValueError("unsupported_template_variable")
|
||||
return pattern.sub(replace, text), sorted(set(used))
|
||||
|
||||
def _target_exists(self, db, bid, org, kind, value):
|
||||
if kind == "email":
|
||||
return bool(db.execute("SELECT id FROM businesses WHERE id=? AND organization_id=? AND email=?", (bid, org, value)).fetchone() or db.execute("SELECT id FROM contacts WHERE business_id=? AND organization_id=? AND email=?", (bid, org, value)).fetchone() or db.execute("SELECT id FROM contact_extractions WHERE business_id=? AND organization_id=? AND kind='email' AND value=?", (bid, org, value)).fetchone())
|
||||
return bool(db.execute("SELECT id FROM businesses WHERE id=? AND organization_id=? AND phone=?", (bid, org, value)).fetchone() or db.execute("SELECT id FROM contacts WHERE business_id=? AND organization_id=? AND phone=?", (bid, org, value)).fetchone() or db.execute("SELECT id FROM contact_extractions WHERE business_id=? AND organization_id=? AND kind IN ('phone','whatsapp') AND value=?", (bid, org, value)).fetchone())
|
||||
|
||||
def create_outreach_draft(self, bid, payload, db, user):
|
||||
org = user["organization_id"]; business = self.business(db, bid, org)
|
||||
if not business: return self.send_json(404, {"error": "not_found"})
|
||||
target = payload.get("target", {}); kind = str(target.get("kind", "email")).lower(); value = str(target.get("value", "")).strip().lower()
|
||||
if kind not in self.OUTREACH_KINDS or not value or len(value) > 320 or not isinstance(target.get("verified", False), bool): return self.send_json(400, {"error": "invalid_target"})
|
||||
subject, body = str(payload.get("subject", "")).strip(), str(payload.get("body", "")).strip()
|
||||
if not subject or not body or len(subject) > 500 or len(body) > 10000: return self.send_json(400, {"error": "invalid_draft"})
|
||||
if contains_secret(payload): return self.send_json(400, {"error": "secret_not_permitted"})
|
||||
try:
|
||||
evidence = [dict(r) for r in db.execute("SELECT id,kind,url,claim,created_at FROM evidence WHERE business_id=? AND organization_id=? ORDER BY id", (bid, org))]
|
||||
subject, sids = self._template(subject, business, evidence); body, bids = self._template(body, business, evidence); ids = sorted(set(sids + bids))
|
||||
except ValueError as exc: return self.send_json(400, {"error": str(exc)})
|
||||
key = str(payload.get("idempotency_key", "")).strip() or hashlib.sha256(json.dumps({"business_id": bid, "target": target, "subject": subject, "body": body}, sort_keys=True).encode()).hexdigest()
|
||||
if len(key) > 200: return self.send_json(400, {"error": "invalid_idempotency_key"})
|
||||
existing = db.execute("SELECT * FROM outreach_drafts WHERE organization_id=? AND idempotency_key=?", (org, key)).fetchone()
|
||||
if existing: return self.send_json(200, self._draft_json(existing))
|
||||
if db.execute("SELECT COUNT(*) FROM outreach_drafts WHERE organization_id=? AND created_at>=datetime('now','-1 day')", (org,)).fetchone()[0] >= self.OUTREACH_DAILY_CAP: return self.send_json(429, {"error": "outreach_daily_cap"})
|
||||
cur = db.execute("INSERT INTO outreach_drafts(organization_id,business_id,target_kind,target_value,target_verified,subject,body,template_json,citations_json,provenance_json,legal_basis,consent_confirmed,actor_user_id,idempotency_key) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)", (org, bid, kind, value, int(target["verified"]), subject, body, json.dumps({"variables": ids}, sort_keys=True), json.dumps(ids), json.dumps({str(x): {"type": "evidence", "id": x} for x in ids}, sort_keys=True), str(payload.get("legal_basis", ""))[:100], int(bool(payload.get("consent_confirmed", False))), user["id"], key))
|
||||
self.audit(db, user, "outreach_draft.created", str(cur.lastrowid)); db.commit()
|
||||
return self.send_json(201, self._draft_json(db.execute("SELECT * FROM outreach_drafts WHERE id=?", (cur.lastrowid,)).fetchone()))
|
||||
|
||||
def list_outreach_drafts(self, db, user, query):
|
||||
try: limit = int((query.get("page_size") or [self.OUTREACH_BATCH_CAP])[0]); offset = int((query.get("offset") or [0])[0])
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
|
||||
if limit < 1 or limit > self.OUTREACH_BATCH_CAP or offset < 0: return self.send_json(400, {"error": "invalid_pagination"})
|
||||
rows = db.execute("SELECT * FROM outreach_drafts WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?", (user["organization_id"], limit + 1, offset)).fetchall()
|
||||
return self.send_json(200, {"organization_id": user["organization_id"], "items": [self._draft_json(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def update_outreach_draft(self, did, payload, db, user):
|
||||
row = db.execute("SELECT * FROM outreach_drafts WHERE id=? AND organization_id=?", (did, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
if row["status"] in ("approved", "sent"): return self.send_json(409, {"error": "draft_locked"})
|
||||
fields, values = [], []
|
||||
subject, body = row["subject"], row["body"]
|
||||
if "subject" in payload: subject = str(payload["subject"]).strip()
|
||||
if "body" in payload: body = str(payload["body"]).strip()
|
||||
if ("subject" in payload and (not subject or len(subject) > 500)) or ("body" in payload and (not body or len(body) > 10000)): return self.send_json(400, {"error": "invalid_draft"})
|
||||
if "subject" in payload or "body" in payload:
|
||||
business = self.business(db, row["business_id"], user["organization_id"])
|
||||
evidence = [dict(r) for r in db.execute("SELECT id,kind,url,claim,created_at FROM evidence WHERE business_id=? AND organization_id=? ORDER BY id", (row["business_id"], user["organization_id"]))]
|
||||
try:
|
||||
subject, sids = self._template(subject, business, evidence); body, bids = self._template(body, business, evidence)
|
||||
except ValueError as exc: return self.send_json(400, {"error": str(exc)})
|
||||
ids = sorted(set(sids + bids))
|
||||
fields += ["subject=?", "body=?", "template_json=?", "citations_json=?", "provenance_json=?"]
|
||||
values += [subject, body, json.dumps({"variables": ids}, sort_keys=True), json.dumps(ids), json.dumps({str(x): {"type": "evidence", "id": x} for x in ids}, sort_keys=True)]
|
||||
if not fields: return self.send_json(400, {"error": "no_changes"})
|
||||
fields.append("updated_at=CURRENT_TIMESTAMP"); values += [did, user["organization_id"]]
|
||||
db.execute("UPDATE outreach_drafts SET " + ",".join(fields) + " WHERE id=? AND organization_id=?", values); self.audit(db, user, "outreach_draft.updated", str(did)); db.commit()
|
||||
return self.send_json(200, self._draft_json(db.execute("SELECT * FROM outreach_drafts WHERE id=?", (did,)).fetchone()))
|
||||
|
||||
def approve_outreach_draft(self, did, db, user):
|
||||
row = db.execute("SELECT * FROM outreach_drafts WHERE id=? AND organization_id=?", (did, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
if row["status"] != "pending_review": return self.send_json(409, {"error": "draft_not_reviewable"})
|
||||
db.execute("UPDATE outreach_drafts SET status='approved',approved_by=?,approved_at=CURRENT_TIMESTAMP,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?", (user["id"], did, user["organization_id"])); self.audit(db, user, "outreach_draft.approved", str(did)); db.commit()
|
||||
return self.send_json(200, self._draft_json(db.execute("SELECT * FROM outreach_drafts WHERE id=?", (did,)).fetchone()))
|
||||
|
||||
def send_outreach_draft(self, did, db, user):
|
||||
org = user["organization_id"]; row = db.execute("SELECT * FROM outreach_drafts WHERE id=? AND organization_id=?", (did, org)).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
cfg = db.execute("SELECT * FROM outreach_provider_configs WHERE organization_id=?", (org,)).fetchone(); reasons = []
|
||||
if not cfg or not cfg["enabled"] or not cfg["provider"] or not cfg["secret_fingerprint"]: reasons.append("provider")
|
||||
business = self.business(db, row["business_id"], org)
|
||||
if not row["target_verified"] or not self._target_exists(db, row["business_id"], org, row["target_kind"], row["target_value"]): reasons.append("verified_target")
|
||||
suppressed = is_suppressed({"email": row["target_value"] if row["target_kind"] == "email" else "", "phone": row["target_value"] if row["target_kind"] != "email" else "", "website_domain": business["website_domain"] if business else ""}, [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1", (org,))])
|
||||
if suppressed: reasons.append("suppressed")
|
||||
policy = {}
|
||||
if cfg:
|
||||
try: policy = json.loads(cfg["policy_json"] or "{}")
|
||||
except (TypeError, ValueError): policy = {"consent_required": True}
|
||||
if policy.get("consent_required", True) and (not row["consent_confirmed"] or not row["legal_basis"]): reasons.append("consent_or_legal_policy")
|
||||
if row["status"] != "approved": reasons.append("approved_draft")
|
||||
if cfg and db.execute("SELECT COUNT(*) FROM outreach_drafts WHERE organization_id=? AND status='sent' AND sent_at>=datetime('now','-1 day')", (org,)).fetchone()[0] >= cfg["daily_cap"]: reasons.append("daily_cap")
|
||||
if reasons:
|
||||
status = "not_configured" if "provider" in reasons and (not cfg or not cfg["enabled"] or not cfg["provider"] or not cfg["secret_fingerprint"]) else "blocked"; self.audit(db, user, "outreach_draft.send_blocked", f"{did}:{','.join(reasons)}"); db.commit()
|
||||
return self.send_json(409, {"status": status, "blocked_reasons": reasons, "network_send": False, "id": did})
|
||||
reasons.append("network_send_disabled"); self.audit(db, user, "outreach_draft.send_blocked", f"{did}:network_send_disabled"); db.commit()
|
||||
return self.send_json(409, {"status": "blocked", "blocked_reasons": reasons, "network_send": False, "id": did})
|
||||
|
||||
def suggest_ai(self, bid, payload, db, user):
|
||||
org = user["organization_id"]
|
||||
business = self.business(db, bid, org)
|
||||
if not business: return self.send_json(404, {"error": "not_found"})
|
||||
business_dict = row_json(business)
|
||||
suppressions = [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1", (org,))]
|
||||
contacts = [dict(r) for r in db.execute("SELECT * FROM contacts WHERE business_id=? AND organization_id=?", (bid, org))]
|
||||
extracted = [dict(r) for r in db.execute("SELECT * FROM contact_extractions WHERE business_id=? AND organization_id=?", (bid, org))]
|
||||
if is_suppressed(business_dict, suppressions) or any(bool(c.get("do_not_contact") or c.get("suppressed")) for c in contacts + extracted):
|
||||
return self.send_json(409, {"error": "ai_blocked_suppressed_business", "business_id": bid})
|
||||
try:
|
||||
requested = int(payload.get("max_items", MAX_INPUT_ITEMS))
|
||||
if requested < 1 or requested > MAX_INPUT_ITEMS: raise ValueError
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_limits"})
|
||||
scans = [dict(r) for r in db.execute("SELECT id,input_url,classification,result_json,scanned_at FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT ?", (bid, org, requested)).fetchall()]
|
||||
evidence = [dict(r) for r in db.execute("SELECT id,kind,url,claim,created_at FROM evidence WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT ?", (bid, org, requested)).fetchall()]
|
||||
history = [dict(r) for r in db.execute("SELECT score,eligible,priority_band,score_version,explanations_json,created_at FROM score_history WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT ?", (bid, org, requested)).fetchall()]
|
||||
status, provider, version, metadata = generate_ai(business_dict, scans[:requested], contacts[:requested], evidence[:requested], history[:requested])
|
||||
output = metadata.pop("output", {}) if status == "succeeded" else {"suggestions": [], "grounded": True, "claim_policy": "stored_evidence_only"}
|
||||
hashes = metadata.get("evidence_hashes", [])
|
||||
cur = db.execute("INSERT INTO ai_runs(organization_id,business_id,input_evidence_hashes_json,model,provider,version,prompt_metadata_json,data_minimization_json,status,approval_state,output_json,actor_user_id) VALUES(?,?,?,?,?,?,?,?,?,?,?,?)", (org, bid, json.dumps(hashes, sort_keys=True), "local-deterministic" if provider else "", provider, version, json.dumps({"request": {"max_items": requested}, "output_limit": MAX_OUTPUT_CHARS}, sort_keys=True), json.dumps(metadata, sort_keys=True), status, "pending", json.dumps(output, sort_keys=True), user["id"]))
|
||||
run_id = cur.lastrowid
|
||||
for suggestion in output.get("suggestions", []):
|
||||
db.execute("INSERT INTO ai_suggestions(ai_run_id,organization_id,business_id,suggestion_type,citations_json,output_json) VALUES(?,?,?,?,?,?)", (run_id, org, bid, str(suggestion.get("type", ""))[:80], json.dumps(suggestion.get("citations", []), sort_keys=True), json.dumps(suggestion, sort_keys=True)))
|
||||
self.audit(db, user, "ai.suggested", str(run_id)); db.commit()
|
||||
row = db.execute("SELECT * FROM ai_runs WHERE id=? AND organization_id=?", (run_id, org)).fetchone()
|
||||
return self.send_json(201, self._ai_run_json(row, output.get("suggestions", [])))
|
||||
|
||||
def list_ai_runs(self, db, user, query):
|
||||
try:
|
||||
limit = int((query.get("page_size") or [50])[0]); offset = max(0, int((query.get("offset") or [0])[0]))
|
||||
if limit < 1 or limit > 100: raise ValueError
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
|
||||
rows = db.execute("SELECT * FROM ai_runs WHERE organization_id=? ORDER BY id DESC LIMIT ? OFFSET ?", (user["organization_id"], limit + 1, offset)).fetchall()
|
||||
items = []
|
||||
for row in rows[:limit]:
|
||||
suggestions = [json.loads(x[0]) for x in db.execute("SELECT output_json FROM ai_suggestions WHERE ai_run_id=? AND organization_id=? ORDER BY id", (row["id"], user["organization_id"]))]
|
||||
items.append(self._ai_run_json(row, suggestions))
|
||||
return self.send_json(200, {"organization_id": user["organization_id"], "items": items, "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def decide_ai(self, run_id, decision, db, user):
|
||||
row = db.execute("SELECT * FROM ai_runs WHERE id=? AND organization_id=?", (run_id, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
if decision == "approve" and row["status"] != "succeeded": return self.send_json(409, {"error": "run_not_approvable"})
|
||||
if row["approval_state"] != "pending": return self.send_json(409, {"error": "already_decided"})
|
||||
now = datetime.now(timezone.utc).replace(microsecond=0).isoformat()
|
||||
if decision == "approve": db.execute("UPDATE ai_runs SET approval_state='approved',approved_at=? WHERE id=? AND organization_id=?", (now, run_id, user["organization_id"]))
|
||||
else: db.execute("UPDATE ai_runs SET approval_state='rejected',rejected_at=? WHERE id=? AND organization_id=?", (now, run_id, user["organization_id"]))
|
||||
self.audit(db, user, "ai." + decision, str(run_id)); db.commit()
|
||||
return self.send_json(200, self._ai_run_json(db.execute("SELECT * FROM ai_runs WHERE id=?", (run_id,)).fetchone()))
|
||||
|
||||
def do_GET(self):
|
||||
parsed=urlparse(self.path); path=parsed.path.rstrip("/")
|
||||
if path=="/api/v1/health/live": return self.send_json(200,{"status":"ok","organization_id":ORGANIZATION_ID})
|
||||
if path=="/api/v1/health/live": return self.send_json(200,{"status":"ok","organization_id":ORGANIZATION_ID,"outreach_enabled":False})
|
||||
if path=="/api/v1/health/ready":
|
||||
try:
|
||||
check=connect(self.server.db_path); check.execute("SELECT 1"); check.close()
|
||||
return self.send_json(200,{"status":"ok","ready":True,"database":"ok","outreach_enabled":False})
|
||||
except (sqlite3.Error, OSError):
|
||||
return self.send_json(503,{"status":"not_ready","ready":False,"database":"error","outreach_enabled":False})
|
||||
db=self.db()
|
||||
try:
|
||||
user=self.require_auth(db)
|
||||
@@ -116,7 +540,14 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if user["role"] not in {"owner","admin"}: return self.send_json(403,{"error":"forbidden"})
|
||||
return self.send_json(200,{"items":[dict(r) for r in db.execute("SELECT id,email,role,organization_id,created_at FROM users WHERE organization_id=? ORDER BY id",(org,))]})
|
||||
if path=="/api/v1/dashboard/summary":
|
||||
row=db.execute("SELECT COUNT(*) businesses,COALESCE(AVG(score),0) average_score FROM businesses WHERE organization_id=?",(org,)).fetchone(); return self.send_json(200,{"organization_id":org,"businesses":row["businesses"],"average_score":round(row["average_score"],2),"suppressed":db.execute("SELECT COUNT(*) FROM suppressions WHERE organization_id=?",(org,)).fetchone()[0]})
|
||||
row=db.execute("SELECT COUNT(*) businesses,COALESCE(AVG(score),0) average_score FROM businesses WHERE organization_id=?",(org,)).fetchone()
|
||||
counts={"new":db.execute("SELECT COUNT(*) FROM businesses WHERE organization_id=? AND created_at>=datetime('now','-7 days')",(org,)).fetchone()[0],"hot":db.execute("SELECT COUNT(*) FROM businesses WHERE organization_id=? AND score>=70 AND merge_status='active'",(org,)).fetchone()[0],"review":db.execute("SELECT COUNT(*) FROM businesses WHERE organization_id=? AND review_status='pending'",(org,)).fetchone()[0],"source_health":db.execute("SELECT COUNT(*) FROM sources WHERE organization_id=? AND health_status IN ('healthy','unhealthy')",(org,)).fetchone()[0],"active_jobs":db.execute("SELECT COUNT(*) FROM jobs WHERE organization_id=? AND status IN ('queued','running')",(org,)).fetchone()[0]}
|
||||
clickable={key:{"count":value,"filter":{"dashboard_filter":key}} for key,value in counts.items()}
|
||||
return self.send_json(200,{"organization_id":org,"businesses":row["businesses"],"average_score":round(row["average_score"],2),"suppressed":db.execute("SELECT COUNT(*) FROM suppressions WHERE organization_id=?",(org,)).fetchone()[0],"counts":counts,"clickable_filters":clickable,"quick_filters":[{"key":key,"count":value,"filter":{"dashboard_filter":key}} for key,value in counts.items()]})
|
||||
if path=="/api/v1/saved-filters": return self.list_saved_filters(db,user)
|
||||
if path=="/api/v1/review-queue": return self.review_queue(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/score-rules": return self.list_score_rules(db,org)
|
||||
if path=="/api/v1/scoring/summary": return self.scoring_summary(db,org)
|
||||
if path=="/api/v1/businesses": return self.list_businesses(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/merge-history": return self.list_merge_history(db,org)
|
||||
if path=="/api/v1/sources": return self.list_sources(db,org)
|
||||
@@ -125,6 +556,16 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if path=="/api/v1/jobs": return self.list_jobs(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/domain-checks": return self.list_domain_checks(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/website-scans": return self.list_website_scans(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/contact-extractions": return self.list_contact_extractions(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/pipeline-entries": return self.list_pipeline(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/pipeline-stages": return self.list_stages(db,org)
|
||||
if path=="/api/v1/outcomes": return self.send_json(200,{"items":sorted(self.OUTCOMES)})
|
||||
if path=="/api/v1/interactions": return self.list_interactions(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/suppressions": return self.list_suppressions(db,org,parse_qs(parsed.query))
|
||||
if path=="/api/v1/ai-runs": return self.list_ai_runs(db,user,parse_qs(parsed.query))
|
||||
if path=="/api/v1/outreach/drafts": return self.list_outreach_drafts(db,user,parse_qs(parsed.query))
|
||||
if path=="/api/v1/outreach/provider-config": return self.provider_config(db,user)
|
||||
if path in ("/api/v1/reports/pipeline","/api/v1/reports/outcomes","/api/v1/reports/activity"): return self.report(db,org,path.rsplit('/',1)[1],parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/jobs/"): return self.get_job_route(db,org,path,parse_qs(parsed.query))
|
||||
if path.startswith("/api/v1/businesses/"):
|
||||
bits=path.split("/"); ident=bits[4] if len(bits)>4 else ""
|
||||
@@ -133,6 +574,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
if len(bits)==7 and bits[5:]==["websites","scan"]: return self.get_latest_website_scan(int(ident),db,user)
|
||||
if len(bits)==7 and bits[5:]==["domains","check"]: return self.get_domain_check(int(ident),db,user,parse_qs(parsed.query))
|
||||
if len(bits)==7 and bits[5:]==["contacts","extract"]: return self.send_json(405,{"error":"method_not_allowed"})
|
||||
if len(bits)==6 and bits[5]=="domain-candidates": return self.list_domain_candidates(int(ident),db,org)
|
||||
if len(bits)==6 and bits[5]=="matches": return self.matches(int(ident),db,org)
|
||||
payload=row_json(row); payload.update(self.nested(db,int(ident),org)); return self.send_json(200,payload)
|
||||
@@ -194,6 +636,64 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
rows = db.execute("SELECT * FROM website_scans WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
|
||||
return self.send_json(200, {"organization_id": org, "items": [self._website_scan_result(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def _contact_json(self, row):
|
||||
result = row_json(row)
|
||||
for key in ("public_business", "suppressed", "do_not_contact"):
|
||||
if key in result: result[key] = bool(result[key])
|
||||
return result
|
||||
|
||||
def list_contact_extractions(self, db, org, query):
|
||||
try:
|
||||
limit = int((query.get("page_size") or [50])[0]); offset = max(0, int((query.get("offset") or [0])[0]))
|
||||
if limit < 1 or limit > CONTACT_EXTRACTION_PAGE_SIZE: raise ValueError
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_pagination"})
|
||||
params = [org]; where = ["organization_id=?"]
|
||||
business_id = (query.get("business_id") or [""])[0]
|
||||
if business_id.isdigit(): where.append("business_id=?"); params.append(int(business_id))
|
||||
rows = db.execute("SELECT * FROM contact_extractions WHERE " + " AND ".join(where) + " ORDER BY id DESC LIMIT ? OFFSET ?", params + [limit + 1, offset]).fetchall()
|
||||
return self.send_json(200, {"organization_id": org, "items": [self._contact_json(r) for r in rows[:limit]], "limit": limit, "offset": offset, "has_more": len(rows) > limit})
|
||||
|
||||
def extract_business_contacts(self, bid, payload, db, user):
|
||||
org = user["organization_id"]; business = self.business(db, bid, org)
|
||||
if not business: return self.send_json(404, {"error": "not_found"})
|
||||
scan_id = payload.get("website_scan_id", payload.get("scan_id"))
|
||||
scan = None
|
||||
if scan_id is not None:
|
||||
if not isinstance(scan_id, int): return self.send_json(400, {"error": "invalid_scan"})
|
||||
scan = db.execute("SELECT * FROM website_scans WHERE id=? AND business_id=? AND organization_id=?", (scan_id, bid, org)).fetchone()
|
||||
if not scan: return self.send_json(404, {"error": "scan_not_found"})
|
||||
try: stored = json.loads(scan["result_json"] or "{}")
|
||||
except (TypeError, ValueError): stored = {}
|
||||
source_url = str(payload.get("source_url") or scan["input_url"]).strip()
|
||||
source_html = payload.get("html") if isinstance(payload.get("html"), str) else stored.get("html")
|
||||
if source_html is None: return self.send_json(409, {"error": "scan_html_unavailable"})
|
||||
if source_url != scan["input_url"] and source_url != (stored.get("final_url") or ""): return self.send_json(400, {"error": "source_not_approved"})
|
||||
else:
|
||||
source_url = str(payload.get("source_url") or "").strip()
|
||||
source_html = payload.get("html")
|
||||
if not source_url or not isinstance(source_html, str): return self.send_json(400, {"error": "approved_scan_required"})
|
||||
parsed = urlparse(source_url)
|
||||
official_hosts = {str(business["website_domain"]).lower().strip(".")}
|
||||
if business["website"]:
|
||||
official_hosts.add((urlparse(business["website"]).hostname or "").lower().strip("."))
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.hostname.lower().strip(".") not in official_hosts:
|
||||
return self.send_json(400, {"error": "source_not_approved"})
|
||||
if len(source_html.encode("utf-8")) > MAX_HTML_BYTES: return self.send_json(413, {"error": "html_too_large"})
|
||||
try: requested_limit = int(payload.get("limit", MAX_RESULTS))
|
||||
except (ValueError, TypeError): return self.send_json(400, {"error": "invalid_limits"})
|
||||
if requested_limit < 1 or requested_limit > MAX_RESULTS: return self.send_json(400, {"error": "invalid_limits"})
|
||||
key = str(payload.get("idempotency_key") or hashlib.sha256((str(scan_id or "") + source_url + source_html).encode()).hexdigest())[:200]
|
||||
existing = db.execute("SELECT * FROM contact_extractions WHERE organization_id=? AND business_id=? AND extraction_key=? ORDER BY id", (org, bid, key)).fetchall()
|
||||
if existing: return self.send_json(200, {"business_id": bid, "extraction_key": key, "items": [self._contact_json(r) for r in existing], "idempotent": True})
|
||||
suppressions = [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1", (org,))]
|
||||
try: found = extract_contacts(source_html, source_url, suppressions=suppressions, max_results=requested_limit)
|
||||
except ValueError as exc: return self.send_json(400, {"error": str(exc)})
|
||||
for item in found:
|
||||
db.execute("INSERT INTO contact_extractions(organization_id,business_id,website_scan_id,extraction_key,kind,value,label,classification,confidence,source_url,public_business,mx_status,suppressed,do_not_contact,provenance) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", (org,bid,scan["id"] if scan else None,key,item["kind"],item["value"],item["label"],item["classification"],item["confidence"],item["source_url"],int(item["public_business"]),item["mx_status"],int(item["suppressed"]),int(item["do_not_contact"]),item["provenance"]))
|
||||
self.audit(db, user, "contacts.extracted", f"{bid}:{len(found)}:{key}"); db.commit()
|
||||
rows = db.execute("SELECT * FROM contact_extractions WHERE organization_id=? AND business_id=? AND extraction_key=? ORDER BY id", (org,bid,key)).fetchall()
|
||||
return self.send_json(201, {"business_id": bid, "extraction_key": key, "items": [self._contact_json(r) for r in rows], "idempotent": False})
|
||||
|
||||
def get_latest_website_scan(self, bid, db, user):
|
||||
row = db.execute("SELECT * FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "scan_not_found"})
|
||||
@@ -293,6 +793,81 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
db.execute("UPDATE jobs SET status='queued',error_code=NULL,completed_at=NULL,updated_at=CURRENT_TIMESTAMP WHERE id=?",(jid,));self.add_job_event(db,jid,user["organization_id"],"retry","Job retry queued",job["progress"]);self.audit(db,user,"job.retried",str(jid));db.commit();getattr(self.server,"job_wakeup",threading.Event()).set();return self.send_json(200,job_json(db.execute("SELECT * FROM jobs WHERE id=?",(jid,)).fetchone()))
|
||||
return self.send_json(404,{"error":"not_found"})
|
||||
|
||||
def ensure_score_rules(self, db, org):
|
||||
for rule in DEFAULT_RULES:
|
||||
db.execute("INSERT OR IGNORE INTO score_rules(organization_id,code,name,description,condition_json,points,max_applications,enabled,version) VALUES(?,?,?,?,?,?,?,?,?)", (org, rule["code"], rule["name"], rule["description"], json.dumps(rule["condition_json"], sort_keys=True), rule["points"], rule["max_applications"], rule["enabled"], rule["version"]))
|
||||
|
||||
def list_score_rules(self, db, org):
|
||||
self.ensure_score_rules(db, org); db.commit()
|
||||
rows = db.execute("SELECT * FROM score_rules WHERE organization_id=? ORDER BY code,id", (org,)).fetchall()
|
||||
items = []
|
||||
for row in rows:
|
||||
item = row_json(row)
|
||||
try: item["condition_json"] = json.loads(item["condition_json"])
|
||||
except (TypeError, ValueError): item["condition_json"] = {}
|
||||
item["enabled"] = bool(item["enabled"]); items.append(item)
|
||||
return self.send_json(200, {"organization_id": org, "items": items})
|
||||
|
||||
def create_score_rule(self, payload, db, user):
|
||||
code = str(payload.get("code", "")).strip(); name = str(payload.get("name", "")).strip(); condition = payload.get("condition_json", payload.get("condition", {}))
|
||||
try: points = int(payload.get("points", 0)); maximum = int(payload.get("max_applications", 1)); version = int(payload.get("version", 1))
|
||||
except (TypeError, ValueError): return self.send_json(400, {"error": "invalid_rule"})
|
||||
if not code or not name or not isinstance(condition, dict) or maximum < 1 or version < 1 or points < -100 or points > 100: return self.send_json(400, {"error": "invalid_rule"})
|
||||
try:
|
||||
cur = db.execute("INSERT INTO score_rules(organization_id,code,name,description,condition_json,points,max_applications,enabled,version) VALUES(?,?,?,?,?,?,?,?,?)", (user["organization_id"], code, name, str(payload.get("description", "")), json.dumps(condition, sort_keys=True), points, maximum, int(bool(payload.get("enabled", True))), version))
|
||||
except sqlite3.IntegrityError: return self.send_json(409, {"error": "duplicate_rule"})
|
||||
self.audit(db, user, "score_rule.created", code); db.commit()
|
||||
row = db.execute("SELECT * FROM score_rules WHERE id=?", (cur.lastrowid,)).fetchone(); item = row_json(row); item["condition_json"] = condition; item["enabled"] = bool(item["enabled"])
|
||||
return self.send_json(201, item)
|
||||
|
||||
def update_score_rule(self, rid, payload, db, user):
|
||||
row = db.execute("SELECT * FROM score_rules WHERE id=? AND organization_id=?", (rid, user["organization_id"])).fetchone()
|
||||
if not row: return self.send_json(404, {"error": "not_found"})
|
||||
allowed = {"name", "description", "condition_json", "points", "max_applications", "enabled", "version"}; values = {k: payload[k] for k in allowed if k in payload}
|
||||
if not values: return self.send_json(400, {"error": "no_changes"})
|
||||
if "condition_json" in values and not isinstance(values["condition_json"], dict): return self.send_json(400, {"error": "invalid_rule"})
|
||||
if "points" in values:
|
||||
try: values["points"] = int(values["points"])
|
||||
except (TypeError, ValueError): return self.send_json(400, {"error": "invalid_rule"})
|
||||
columns=[]; params=[]
|
||||
for key, value in values.items(): columns.append(key + "=?"); params.append(json.dumps(value, sort_keys=True) if key == "condition_json" else (int(bool(value)) if key == "enabled" else value))
|
||||
params += [rid, user["organization_id"]]; db.execute("UPDATE score_rules SET " + ",".join(columns) + ",updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?", params); self.audit(db, user, "score_rule.updated", str(rid)); db.commit()
|
||||
item = row_json(db.execute("SELECT * FROM score_rules WHERE id=?", (rid,)).fetchone())
|
||||
try: item["condition_json"] = json.loads(item["condition_json"])
|
||||
except (TypeError, ValueError): item["condition_json"] = {}
|
||||
item["enabled"] = bool(item["enabled"]); return self.send_json(200, item)
|
||||
|
||||
def recalculate_score(self, bid, payload, db, user):
|
||||
org = user["organization_id"]; business = self.business(db, bid, org)
|
||||
if not business: return self.send_json(404, {"error": "not_found"})
|
||||
scans = db.execute("SELECT result_json,classification,scanned_at FROM website_scans WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, org)).fetchone(); website = {"classification": business["website_class"]}
|
||||
if scans:
|
||||
try: website.update(json.loads(scans["result_json"] or "{}"))
|
||||
except (TypeError, ValueError): pass
|
||||
website["classification"] = scans["classification"]
|
||||
contacts = [dict(r) for r in db.execute("SELECT public_business,suppressed,do_not_contact FROM contact_extractions WHERE business_id=? AND organization_id=?", (bid, org))]
|
||||
drow = db.execute("SELECT status,result_json,checked_at FROM domain_checks WHERE business_id=? AND organization_id=? ORDER BY id DESC LIMIT 1", (bid, org)).fetchone(); domain = dict(drow) if drow else {}
|
||||
if drow:
|
||||
try: domain.update(json.loads(drow["result_json"] or "{}"))
|
||||
except (TypeError, ValueError): pass
|
||||
suppressed = is_suppressed(dict(business), [dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1", (org,))])
|
||||
signals = signals_for_business(dict(business), website, contacts, domain, suppressed); self.ensure_score_rules(db, org); rules = [dict(r) for r in db.execute("SELECT * FROM score_rules WHERE organization_id=?", (org,))]; result = evaluate_score(signals, rules)
|
||||
override_score = payload.get("override_score"); override_eligible = payload.get("override_eligible")
|
||||
if override_score is not None or override_eligible is not None:
|
||||
reason = str(payload.get("override_reason", "")).strip()
|
||||
if not reason: return self.send_json(400, {"error": "override_reason_required"})
|
||||
if override_score is not None: result["score"] = max(0, min(100, int(override_score)))
|
||||
if override_eligible is not None and not suppressed: result["eligible"] = bool(override_eligible)
|
||||
result["priority_band"] = "ineligible" if not result["eligible"] else ("high" if result["score"] >= 70 else "medium" if result["score"] >= 40 else "low")
|
||||
db.execute("UPDATE businesses SET score=?,score_version=?,score_factors=?,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?", (result["score"], SCORE_VERSION, json.dumps(result["explanations"], sort_keys=True), bid, org))
|
||||
cur=db.execute("INSERT INTO score_history(organization_id,business_id,score,eligible,priority_band,score_version,explanations_json,signals_json,override_score,override_eligible,override_reason,actor_user_id) VALUES(?,?,?,?,?,?,?,?,?,?,?,?)", (org,bid,result["score"],int(result["eligible"]),result["priority_band"],SCORE_VERSION,json.dumps(result["explanations"],sort_keys=True),json.dumps(signals,sort_keys=True),override_score,override_eligible,payload.get("override_reason"),user["id"]))
|
||||
self.audit(db,user,"business.score_recalculated",f"{bid}:{result['score']}"); db.commit(); result.update({"business_id": bid, "history_id": cur.lastrowid}); return self.send_json(200, result)
|
||||
|
||||
def scoring_summary(self, db, org):
|
||||
row=db.execute("SELECT COUNT(*) businesses,COALESCE(AVG(score),0) average_score,SUM(CASE WHEN score>=70 THEN 1 ELSE 0 END) high_priority FROM businesses WHERE organization_id=? AND merge_status='active'",(org,)).fetchone()
|
||||
bands={r["priority_band"]:r["count"] for r in db.execute("SELECT priority_band,COUNT(*) count FROM score_history WHERE organization_id=? GROUP BY priority_band",(org,))}
|
||||
return self.send_json(200,{"organization_id":org,"businesses":row["businesses"],"average_score":round(row["average_score"],2),"high_priority":row["high_priority"] or 0,"bands":bands,"history_count":db.execute("SELECT COUNT(*) FROM score_history WHERE organization_id=?",(org,)).fetchone()[0]})
|
||||
|
||||
def list_businesses(self,db,org,query):
|
||||
def number(name, default=None):
|
||||
raw=query.get(name,[None])[0]
|
||||
@@ -311,6 +886,27 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
offset=(number("cursor",0) or 0)+(page-1)*size
|
||||
rows=db.execute("SELECT b.* FROM businesses b WHERE "+" AND ".join(where)+" ORDER BY b.score DESC,b.id LIMIT ? OFFSET ?",params+[size+1,offset]).fetchall(); more=len(rows)>size; rows=rows[:size]
|
||||
return self.send_json(200,{"organization_id":org,"items":[row_json(r) for r in rows],"page":page,"page_size":size,"next_cursor":str(offset+size) if more else None})
|
||||
def bulk_review(self, payload, db, user):
|
||||
ids = payload.get("ids", payload.get("business_ids")); action = str(payload.get("action", "")).strip().lower()
|
||||
if not isinstance(ids, list) or not ids or len(ids) > 100 or any(not isinstance(i, int) or i < 1 for i in ids) or len(set(ids)) != len(ids): return self.send_json(400, {"error": "invalid_bulk_ids"})
|
||||
if action not in {"verify", "reject", "assign"}: return self.send_json(400, {"error": "invalid_bulk_action"})
|
||||
if action == "assign":
|
||||
assignee = str(payload.get("assigned_to", payload.get("assignee", ""))).strip()
|
||||
if not assignee or len(assignee) > 120: return self.send_json(400, {"error": "assignee_required"})
|
||||
else: assignee = ""
|
||||
org = user["organization_id"]; marks = ",".join("?" for _ in ids)
|
||||
rows = db.execute("SELECT id FROM businesses WHERE organization_id=? AND id IN (" + marks + ")", [org] + ids).fetchall()
|
||||
if len(rows) != len(ids): return self.send_json(404, {"error": "not_found"})
|
||||
try:
|
||||
if action == "verify": db.execute("UPDATE businesses SET verified=1,verified_at=CURRENT_TIMESTAMP,review_status='verified',updated_at=CURRENT_TIMESTAMP WHERE organization_id=? AND id IN (" + marks + ")", [org] + ids)
|
||||
elif action == "reject": db.execute("UPDATE businesses SET verified=0,review_status='rejected',updated_at=CURRENT_TIMESTAMP WHERE organization_id=? AND id IN (" + marks + ")", [org] + ids)
|
||||
else: db.execute("UPDATE businesses SET assigned_to=?,review_status='assigned',updated_at=CURRENT_TIMESTAMP WHERE organization_id=? AND id IN (" + marks + ")", [assignee, org] + ids)
|
||||
self.audit(db, user, "businesses.bulk_review", json.dumps({"action": action, "ids": ids, "assigned_to": assignee}, sort_keys=True))
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback(); raise
|
||||
return self.send_json(200, {"action": action, "updated": ids, "count": len(ids)})
|
||||
|
||||
def do_POST(self):
|
||||
path=urlparse(self.path).path.rstrip("/")
|
||||
if path=="/api/v1/auth/login":return self.login(self.read_json())
|
||||
@@ -327,14 +923,30 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
return self.create_job(self.read_json(),db,user)
|
||||
if user["role"] not in MUTATING_ROLES:return self.send_json(403,{"error":"forbidden"})
|
||||
payload=self.read_json(); org=user["organization_id"]
|
||||
if path=="/api/v1/saved-filters": return self.save_filter(payload,db,user)
|
||||
if path=="/api/v1/outreach/provider-config": return self.provider_config(db,user,payload)
|
||||
if path=="/api/v1/businesses/bulk-review": return self.bulk_review(payload,db,user)
|
||||
bits_ai=path.split("/")
|
||||
if len(bits_ai)==7 and bits_ai[:4]==["","api","v1","businesses"] and bits_ai[5]=="ai" and bits_ai[6]=="suggest": return self.suggest_ai(int(bits_ai[4]) if bits_ai[4].isdigit() else -1,payload,db,user)
|
||||
if len(bits_ai)==6 and bits_ai[:4]==["","api","v1","ai-runs"] and bits_ai[4].isdigit() and bits_ai[5] in {"approve","reject"}: return self.decide_ai(int(bits_ai[4]),bits_ai[5],db,user)
|
||||
if path=="/api/v1/score-rules": return self.create_score_rule(payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:]==["businesses",path.split("/")[4],"score","recalculate"]: return self.recalculate_score(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if path.startswith("/api/v1/jobs/"):
|
||||
return self.job_action(db,user,path)
|
||||
if path=="/api/v1/businesses":return self.create_business(payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"outreach"] and path.split("/")[6]=="drafts": return self.create_outreach_draft(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
bits_outreach=path.split("/")
|
||||
if len(bits_outreach)==7 and bits_outreach[:4]==["","api","v1","outreach"] and bits_outreach[4]=="drafts" and bits_outreach[5].isdigit() and bits_outreach[6] in {"approve","send"}: return self.approve_outreach_draft(int(bits_outreach[5]),db,user) if bits_outreach[6]=="approve" else self.send_outreach_draft(int(bits_outreach[5]),db,user)
|
||||
if path=="/api/v1/sources":return self.create_source(payload,db,user)
|
||||
if path=="/api/v1/discovery-queries":return self.create_query(payload,db,user)
|
||||
if path=="/api/v1/suppressions":return self.create_suppression(payload,db,user)
|
||||
if path=="/api/v1/suppressions/import":return self.import_suppressions(payload,db,user)
|
||||
if path=="/api/v1/pipeline-stages":return self.create_stage(payload,db,user)
|
||||
if path=="/api/v1/interactions": return self.send_json(400,{"error":"business_id_required"})
|
||||
if path=="/api/v1/contact-extractions": return self.send_json(405,{"error":"method_not_allowed"})
|
||||
if path=="/api/v1/imports/preview":return self.preview_import(payload,db,org)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"websites"] and path.split("/")[6]=="scan": return self.scan_business_website(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"contacts"] and path.split("/")[6]=="extract": return self.extract_business_contacts(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domains"] and path.split("/")[6]=="check": return self.post_domain_check(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if len(path.split("/"))==7 and path.split("/")[3:6]==["businesses",path.split("/")[4],"domain-candidates"] and path.split("/")[6]=="check-availability": return self.check_availability(int(path.split("/")[4]) if path.split("/")[4].isdigit() else -1,payload,db,user)
|
||||
if path.startswith("/api/v1/merge-history/") and path.endswith("/reverse"):
|
||||
@@ -345,6 +957,8 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
return self.test_source(int(bits[4]),db,user) if bits[5]=="test" else self.ingest_source(int(bits[4]),payload,db,user)
|
||||
if len(bits)==6 and bits[3] == "discovery-queries" and bits[4].isdigit() and bits[5]=="run": return self.run_query(int(bits[4]),db,user)
|
||||
if len(bits)==7 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES and bits[6]=="": pass
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5] == "pipeline": return self.update_pipeline(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5] == "interactions": return self.create_interaction(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5] in CHILD_TABLES:return self.create_child(int(bits[4]) if bits[4].isdigit() else -1,bits[5],payload,db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="verify":return self.verify_business(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="merge":return self.merge_business(int(bits[4]) if bits[4].isdigit() else -1,payload,db,user)
|
||||
@@ -357,10 +971,36 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
if not user:return
|
||||
if user["role"] not in MUTATING_ROLES:return self.send_json(403,{"error":"forbidden"})
|
||||
bits=path.split("/")
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","pipeline-stages"] and bits[4].isdigit(): return self.update_stage(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","suppressions"] and bits[4].isdigit(): return self.update_suppression(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","pipeline-entries"] and bits[4].isdigit(): return self.update_pipeline_entry(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","interactions"] and bits[4].isdigit(): return self.update_interaction(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","saved-filters"] and bits[4].isdigit(): return self.update_saved_filter(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","score-rules"] and bits[4].isdigit(): return self.update_score_rule(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","outreach"] and bits[4]=="provider-config": return self.provider_config(db,user,self.read_json())
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","sources"] and bits[4].isdigit(): return self.update_source(int(bits[4]),self.read_json(),db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","outreach"] and bits[4]=="drafts" and bits[5].isdigit(): return self.update_outreach_draft(int(bits[5]),self.read_json(),db,user)
|
||||
if len(bits)==6 and bits[:4]==["","api","v1","businesses"] and bits[5]=="pipeline":return self.update_pipeline(int(bits[4]) if bits[4].isdigit() else -1,self.read_json(),db,user)
|
||||
return self.send_json(404,{"error":"not_found"})
|
||||
finally:db.close()
|
||||
def do_DELETE(self):
|
||||
path=urlparse(self.path).path.rstrip("/"); db=self.db()
|
||||
try:
|
||||
user=self.require_auth(db)
|
||||
if not user:return
|
||||
if user["role"] not in MUTATING_ROLES:return self.send_json(403,{"error":"forbidden"})
|
||||
bits=path.split("/")
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","pipeline-stages"] and bits[4].isdigit(): return self.delete_crm_item("pipeline_stages",int(bits[4]),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","pipeline-entries"] and bits[4].isdigit(): return self.delete_crm_item("pipeline_entries",int(bits[4]),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","interactions"] and bits[4].isdigit(): return self.delete_crm_item("interactions",int(bits[4]),db,user)
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","suppressions"] and bits[4].isdigit():
|
||||
row=db.execute("SELECT id FROM suppressions WHERE id=? AND organization_id=?",(int(bits[4]),user["organization_id"])).fetchone()
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
db.execute("DELETE FROM suppressions WHERE id=? AND organization_id=?",(int(bits[4]),user["organization_id"]));self.audit(db,user,"suppression.deleted",bits[4]);db.commit();return self.send_json(200,{"ok":True,"id":int(bits[4])})
|
||||
if len(bits)==5 and bits[:4]==["","api","v1","saved-filters"] and bits[4].isdigit(): return self.delete_saved_filter(int(bits[4]),db,user)
|
||||
return self.send_json(404,{"error":"not_found"})
|
||||
finally: db.close()
|
||||
|
||||
def login(self,payload):
|
||||
db=self.db(); email=str(payload.get("email"," ")).strip().lower(); password=str(payload.get("password","")); user=db.execute("SELECT * FROM users WHERE email=?",(email,)).fetchone()
|
||||
try:
|
||||
@@ -370,7 +1010,7 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
def create_business(self,payload,db,user):
|
||||
org=user["organization_id"]
|
||||
if not str(payload.get("name","")).strip():return self.send_json(400,{"error":"name_required"})
|
||||
b=normalize_business(payload); suppressions=[dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=?",(org,))]
|
||||
b=normalize_business(payload); suppressions=[dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1",(org,))]
|
||||
if is_suppressed(b,suppressions):return self.send_json(409,{"error":"suppressed"})
|
||||
fields=[(c,b[c]) for c in ("website_domain","email","phone") if b[c]]
|
||||
if fields and db.execute("SELECT id FROM businesses WHERE organization_id=? AND ("+" OR ".join(f"{c}=?" for c,_ in fields)+")",[org]+[v for _,v in fields]).fetchone():return self.send_json(409,{"error":"duplicate"})
|
||||
@@ -378,16 +1018,16 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
def create_suppression(self,payload,db,user):
|
||||
kind,value=payload.get("kind"),str(payload.get("value","")).strip().lower()
|
||||
if kind not in {"email","domain","phone"} or not value:return self.send_json(400,{"error":"invalid_suppression"})
|
||||
try:db.execute("INSERT INTO suppressions(organization_id,kind,value) VALUES(?,?,?)",(user["organization_id"],kind,value))
|
||||
try:db.execute("INSERT INTO suppressions(organization_id,kind,value,actor_user_id,active) VALUES(?,?,?,?,1)",(user["organization_id"],kind,value,user["id"]))
|
||||
except sqlite3.IntegrityError:pass
|
||||
self.audit(db,user,"suppression.created",kind);db.commit();return self.send_json(201,dict(db.execute("SELECT * FROM suppressions WHERE organization_id=? AND kind=? AND value=?",(user["organization_id"],kind,value)).fetchone()))
|
||||
self.audit(db,user,"suppression.created",kind);db.commit();return self.send_json(201,row_json(db.execute("SELECT * FROM suppressions WHERE organization_id=? AND kind=? AND value=?",(user["organization_id"],kind,value)).fetchone()))
|
||||
def child_business(self,db,bid,user):return self.business(db,bid,user["organization_id"])
|
||||
def create_child(self,bid,table,payload,db,user):
|
||||
if not self.child_business(db,bid,user):return self.send_json(404,{"error":"not_found"})
|
||||
if table=="contacts":
|
||||
email=str(payload.get("email","")).strip().lower(); phone=normalize_phone(payload.get("phone"));
|
||||
if email and not re.match(r"^[^@\s]+@[^@\s]+\.[^@\s]+$",email):return self.send_json(400,{"error":"invalid_contact"})
|
||||
suppressed=is_suppressed({"email":email,"phone":phone},[dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=?",(user["organization_id"],))]); values=(str(payload.get("name","")).strip(),email,phone,str(payload.get("title","")).strip(),int(bool(payload.get("do_not_contact"))) or int(suppressed))
|
||||
suppressed=is_suppressed({"email":email,"phone":phone},[dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1",(user["organization_id"],))]); values=(str(payload.get("name","")).strip(),email,phone,str(payload.get("title","")).strip(),int(bool(payload.get("do_not_contact"))) or int(suppressed))
|
||||
elif table=="domains":
|
||||
value=normalize_domain(payload.get("domain"));
|
||||
if not value:return self.send_json(400,{"error":"invalid_domain"})
|
||||
@@ -404,15 +1044,100 @@ class ApiHandler(BaseHTTPRequestHandler):
|
||||
values=(str(payload["body"]).strip(),)
|
||||
columns=CHILD_TABLES[table]; db.execute(f"INSERT INTO {table}(business_id,organization_id,{','.join(columns)}) VALUES(?, ?, {','.join('?' for _ in columns)})",(bid,user["organization_id"])+values); rid=db.execute("SELECT last_insert_rowid()").fetchone()[0];self.audit(db,user,f"{table}.created",str(rid));db.commit();return self.send_json(201,row_json(db.execute(f"SELECT * FROM {table} WHERE id=?",(rid,)).fetchone()))
|
||||
def update_pipeline(self,bid,payload,db,user):
|
||||
if not self.child_business(db,bid,user) or not str(payload.get("stage","")).strip():return self.send_json(404 if not self.child_business(db,bid,user) else 400,{"error":"not_found" if not self.child_business(db,bid,user) else "stage_required"})
|
||||
stage=str(payload["stage"]).strip();status=str(payload.get("status","active")).strip() or "active";db.execute("INSERT INTO pipeline_entries(business_id,organization_id,stage,status) VALUES(?,?,?,?)",(bid,user["organization_id"],stage,status));rid=db.execute("SELECT last_insert_rowid()").fetchone()[0];self.audit(db,user,"pipeline.updated",stage);db.commit();return self.send_json(200,row_json(db.execute("SELECT * FROM pipeline_entries WHERE id=?",(rid,)).fetchone()))
|
||||
if not self.child_business(db,bid,user): return self.send_json(404,{"error":"not_found"})
|
||||
stage=str(payload.get("stage","")).strip(); status=str(payload.get("status","active")).strip() or "active"
|
||||
if not stage or stage not in self.STAGES or status not in {"active","won","lost","paused"}: return self.send_json(400,{"error":"invalid_pipeline"})
|
||||
key=str(payload.get("idempotency_key","")).strip(); org=user["organization_id"]
|
||||
if key:
|
||||
prior=db.execute("SELECT * FROM pipeline_entries WHERE organization_id=? AND idempotency_key=?",(org,key)).fetchone()
|
||||
if prior:return self.send_json(200,row_json(prior))
|
||||
suppressed=self._crm_suppressed(db,org,self.business(db,bid,org))
|
||||
if suppressed:return self.send_json(409,{"error":"do_not_contact","outreach_disabled":True})
|
||||
cur=db.execute("INSERT INTO pipeline_entries(business_id,organization_id,stage,status,notes,next_action,follow_up_at,actor_user_id,idempotency_key) VALUES(?,?,?,?,?,?,?,?,?)",(bid,org,stage,status,str(payload.get("notes",payload.get("body","")))[:5000],str(payload.get("next_action",""))[:500],payload.get("follow_up_at"),user["id"],key or None)); rid=cur.lastrowid
|
||||
self.audit(db,user,"pipeline.created",str(rid)); db.commit(); return self.send_json(200 if getattr(self,"command","")=="PATCH" else 201,row_json(db.execute("SELECT * FROM pipeline_entries WHERE id=?",(rid,)).fetchone()))
|
||||
|
||||
def update_pipeline_entry(self,eid,payload,db,user):
|
||||
org=user["organization_id"]; row=db.execute("SELECT * FROM pipeline_entries WHERE id=? AND organization_id=?",(eid,org)).fetchone()
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
if payload.get("expected_updated_at") and payload["expected_updated_at"] != row["updated_at"]: return self.send_json(409,{"error":"conflict"})
|
||||
values={k:payload[k] for k in ("stage","status","notes","next_action","follow_up_at") if k in payload}
|
||||
if "stage" in values and values["stage"] not in self.STAGES:return self.send_json(400,{"error":"invalid_stage"})
|
||||
if "status" in values and values["status"] not in {"active","won","lost","paused"}:return self.send_json(400,{"error":"invalid_status"})
|
||||
if not values:return self.send_json(400,{"error":"no_changes"})
|
||||
cols=[]; args=[]
|
||||
for k,v in values.items():cols.append(k+"=?");args.append(v)
|
||||
args += [eid,org]; db.execute("UPDATE pipeline_entries SET "+",".join(cols)+",version=version+1,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",args); self.audit(db,user,"pipeline.updated",str(eid));db.commit()
|
||||
return self.send_json(200,row_json(db.execute("SELECT * FROM pipeline_entries WHERE id=?",(eid,)).fetchone()))
|
||||
|
||||
def create_interaction(self,bid,payload,db,user):
|
||||
org=user["organization_id"]; business=self.business(db,bid,org)
|
||||
if not business:return self.send_json(404,{"error":"not_found"})
|
||||
outcome=str(payload.get("outcome","other")).strip().lower(); kind=str(payload.get("kind","")).strip().lower()
|
||||
if not kind or outcome not in self.OUTCOMES:return self.send_json(400,{"error":"invalid_interaction"})
|
||||
if self._crm_suppressed(db,org,business):return self.send_json(409,{"error":"do_not_contact","outreach_disabled":True})
|
||||
key=str(payload.get("idempotency_key","")).strip()
|
||||
if key:
|
||||
prior=db.execute("SELECT * FROM interactions WHERE organization_id=? AND idempotency_key=?",(org,key)).fetchone()
|
||||
if prior:return self.send_json(200,row_json(prior))
|
||||
cur=db.execute("INSERT INTO interactions(business_id,organization_id,kind,body,outcome,notes,next_action,follow_up_at,actor_user_id,idempotency_key) VALUES(?,?,?,?,?,?,?,?,?,?)",(bid,org,kind,str(payload.get("body",payload.get("notes","")))[:5000],outcome,str(payload.get("notes",""))[:5000],str(payload.get("next_action",""))[:500],payload.get("follow_up_at"),user["id"],key or None)); self.audit(db,user,"interaction.created",str(cur.lastrowid));db.commit()
|
||||
return self.send_json(201,row_json(db.execute("SELECT * FROM interactions WHERE id=?",(cur.lastrowid,)).fetchone()))
|
||||
|
||||
def update_interaction(self,iid,payload,db,user):
|
||||
org=user["organization_id"]; row=db.execute("SELECT * FROM interactions WHERE id=? AND organization_id=?",(iid,org)).fetchone()
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
values={k:payload[k] for k in ("kind","body","outcome","notes","next_action","follow_up_at") if k in payload}
|
||||
if "outcome" in values and values["outcome"] not in self.OUTCOMES:return self.send_json(400,{"error":"invalid_outcome"})
|
||||
if not values:return self.send_json(400,{"error":"no_changes"})
|
||||
cols=[];args=[]
|
||||
for k,v in values.items():cols.append(k+"=?");args.append(v)
|
||||
args += [iid,org];db.execute("UPDATE interactions SET "+",".join(cols)+",updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",args);self.audit(db,user,"interaction.updated",str(iid));db.commit();return self.send_json(200,row_json(db.execute("SELECT * FROM interactions WHERE id=?",(iid,)).fetchone()))
|
||||
|
||||
def update_suppression(self,sid,payload,db,user):
|
||||
row=db.execute("SELECT * FROM suppressions WHERE id=? AND organization_id=?",(sid,user["organization_id"])).fetchone()
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
if "active" not in payload:return self.send_json(400,{"error":"active_required"})
|
||||
db.execute("UPDATE suppressions SET active=?,actor_user_id=?,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",(int(bool(payload["active"])),user["id"],sid,user["organization_id"]));self.audit(db,user,"suppression.updated",str(sid));db.commit();return self.send_json(200,row_json(db.execute("SELECT * FROM suppressions WHERE id=?",(sid,)).fetchone()))
|
||||
|
||||
def import_suppressions(self,payload,db,user):
|
||||
items=payload.get("items",payload.get("suppressions"));
|
||||
if not isinstance(items,list) or len(items)>1000:return self.send_json(400,{"error":"invalid_suppression_import"})
|
||||
imported=0
|
||||
for item in items:
|
||||
if not isinstance(item,dict) or item.get("kind") not in {"email","domain","phone"} or not str(item.get("value","")).strip():return self.send_json(400,{"error":"invalid_suppression"})
|
||||
kind=item["kind"];value=str(item["value"]).strip().lower(); cur=db.execute("INSERT OR IGNORE INTO suppressions(organization_id,kind,value,actor_user_id) VALUES(?,?,?,?)",(user["organization_id"],kind,value,user["id"]));imported += cur.rowcount
|
||||
self.audit(db,user,"suppressions.imported",str(imported));db.commit();return self.send_json(201,{"imported":imported,"received":len(items)})
|
||||
|
||||
def create_stage(self,payload,db,user):
|
||||
name=str(payload.get("name","")).strip().lower()
|
||||
if not name or len(name)>80 or not re.match(r"^[a-z0-9_-]+$",name):return self.send_json(400,{"error":"invalid_stage"})
|
||||
try: position=int(payload.get("position",0))
|
||||
except (TypeError,ValueError):return self.send_json(400,{"error":"invalid_stage"})
|
||||
try:cur=db.execute("INSERT INTO pipeline_stages(organization_id,name,position) VALUES(?,?,?)",(user["organization_id"],name,position))
|
||||
except sqlite3.IntegrityError:return self.send_json(409,{"error":"duplicate_stage"})
|
||||
self.audit(db,user,"pipeline_stage.created",str(cur.lastrowid));db.commit();return self.send_json(201,row_json(db.execute("SELECT * FROM pipeline_stages WHERE id=?",(cur.lastrowid,)).fetchone()))
|
||||
|
||||
def update_stage(self,sid,payload,db,user):
|
||||
row=db.execute("SELECT * FROM pipeline_stages WHERE id=? AND organization_id=?",(sid,user["organization_id"])).fetchone()
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
values={k:payload[k] for k in ("name","position","active") if k in payload}
|
||||
if not values:return self.send_json(400,{"error":"no_changes"})
|
||||
if "name" in values and (not isinstance(values["name"],str) or not re.match(r"^[a-z0-9_-]+$",values["name"])):return self.send_json(400,{"error":"invalid_stage"})
|
||||
cols=[];args=[]
|
||||
for k,v in values.items():cols.append(k+"=?");args.append(int(bool(v)) if k=="active" else v)
|
||||
args += [sid,user["organization_id"]];db.execute("UPDATE pipeline_stages SET "+",".join(cols)+",updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",args);self.audit(db,user,"pipeline_stage.updated",str(sid));db.commit();return self.send_json(200,row_json(db.execute("SELECT * FROM pipeline_stages WHERE id=?",(sid,)).fetchone()))
|
||||
|
||||
def delete_crm_item(self,table,ident,db,user):
|
||||
row=db.execute(f"SELECT id FROM {table} WHERE id=? AND organization_id=?",(ident,user["organization_id"])).fetchone()
|
||||
if not row:return self.send_json(404,{"error":"not_found"})
|
||||
db.execute(f"DELETE FROM {table} WHERE id=? AND organization_id=?",(ident,user["organization_id"]));self.audit(db,user,table+".deleted",str(ident));db.commit();return self.send_json(200,{"ok":True,"id":ident})
|
||||
|
||||
def verify_business(self,bid,payload,db,user):
|
||||
if not self.child_business(db,bid,user):return self.send_json(404,{"error":"not_found"})
|
||||
verified=bool(payload.get("verified",True)); now=datetime.now(timezone.utc).replace(microsecond=0).isoformat();db.execute("UPDATE businesses SET verified=?,verified_at=?,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",(int(verified),now if verified else None,bid,user["organization_id"]));self.audit(db,user,"business.verified",str(verified));db.commit();row=self.business(db,bid,user["organization_id"]);return self.send_json(200,row_json(row))
|
||||
verified=bool(payload.get("verified",True)); now=datetime.now(timezone.utc).replace(microsecond=0).isoformat();db.execute("UPDATE businesses SET verified=?,verified_at=?,review_status=?,updated_at=CURRENT_TIMESTAMP WHERE id=? AND organization_id=?",(int(verified),now if verified else None,"verified" if verified else "pending",bid,user["organization_id"]));self.audit(db,user,"business.verified",str(verified));db.commit();row=self.business(db,bid,user["organization_id"]);return self.send_json(200,row_json(row))
|
||||
def preview_import(self,payload,db,org):
|
||||
rows=payload.get("rows",[])
|
||||
if not isinstance(rows,list):return self.send_json(400,{"error":"rows_required"})
|
||||
normalized=deduplicate_businesses([r for r in rows if isinstance(r,dict) and str(r.get("name","")).strip()]); suppressions=[dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=?",(org,))];existing=[row_json(r) for r in db.execute("SELECT * FROM businesses WHERE organization_id=?",(org,))];seen=set();accepted=[];suppressed=0;existing_keys={deduplication_key(x) for x in existing}
|
||||
normalized=deduplicate_businesses([r for r in rows if isinstance(r,dict) and str(r.get("name","")).strip()]); suppressions=[dict(r) for r in db.execute("SELECT kind,value FROM suppressions WHERE organization_id=? AND active=1",(org,))];existing=[row_json(r) for r in db.execute("SELECT * FROM businesses WHERE organization_id=?",(org,))];seen=set();accepted=[];suppressed=0;existing_keys={deduplication_key(x) for x in existing}
|
||||
for b in normalized:
|
||||
key=deduplication_key(b)
|
||||
if is_suppressed(b,suppressions):suppressed+=1
|
||||
@@ -567,6 +1292,7 @@ def _job_worker(server):
|
||||
finally: db.close()
|
||||
|
||||
def create_server(host="127.0.0.1",port=8000,db_path="prospects.db"):
|
||||
load_config()
|
||||
server=ThreadingHTTPServer((host,port),ApiHandler);server.db_path=db_path;connect(db_path).close();server.job_stop=threading.Event();server.job_wakeup=threading.Event();server.job_thread=threading.Thread(target=_job_worker,args=(server,),daemon=True);server.job_thread.start()
|
||||
original_close=server.server_close
|
||||
def close():
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
"""Deterministic, explainable qualification scoring."""
|
||||
from __future__ import annotations
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
SCORE_VERSION = "phase10-1"
|
||||
DEFAULT_RULES = [
|
||||
{"code": "business_name", "name": "Named business", "description": "Business has a usable name", "condition_json": {"signal": "business.name", "operator": "present"}, "points": 15, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "business_site", "name": "Business website", "description": "Business has a non-social website", "condition_json": {"signal": "business.website_class", "operator": "equals", "value": "business_site"}, "points": 20, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "website_healthy", "name": "Healthy website", "description": "Latest website scan is healthy", "condition_json": {"signal": "website.classification", "operator": "equals", "value": "healthy"}, "points": 15, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "contact_email", "name": "Email contact", "description": "A direct business email is available", "condition_json": {"signal": "business.email", "operator": "present"}, "points": 15, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "contact_phone", "name": "Phone contact", "description": "A business phone is available", "condition_json": {"signal": "business.phone", "operator": "present"}, "points": 10, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "extracted_contact", "name": "Extracted contact", "description": "A public, non-suppressed contact was extracted", "condition_json": {"signal": "contacts.public_count", "operator": "gte", "value": 1}, "points": 10, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "domain_verified", "name": "Domain check", "description": "Domain check resolved successfully", "condition_json": {"signal": "domain.status", "operator": "in", "value": ["resolved", "ok", "healthy"]}, "points": 5, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
{"code": "verified_business", "name": "Verified business", "description": "Business has been verified", "condition_json": {"signal": "state.verified", "operator": "truthy"}, "points": 10, "max_applications": 1, "enabled": 1, "version": 1},
|
||||
]
|
||||
|
||||
def _get(data, path):
|
||||
value = data
|
||||
for part in str(path).split("."):
|
||||
if not isinstance(value, dict): return None
|
||||
value = value.get(part)
|
||||
return value
|
||||
|
||||
def _match(condition, signals):
|
||||
if not isinstance(condition, dict): return False
|
||||
if "all" in condition: return all(_match(c, signals) for c in condition["all"])
|
||||
if "any" in condition: return any(_match(c, signals) for c in condition["any"])
|
||||
if "not" in condition: return not _match(condition["not"], signals)
|
||||
value = _get(signals, condition.get("signal", "")); op = condition.get("operator", "truthy"); expected = condition.get("value")
|
||||
section = signals.get(str(condition.get("signal", "")).split(".")[0], {})
|
||||
if isinstance(section, dict) and (section.get("stale") or section.get("uncertain")): return False
|
||||
if op in ("truthy", "present"): return bool(value) if op == "truthy" else value not in (None, "", [], {})
|
||||
if op == "equals": return value == expected
|
||||
if op == "in": return value in (expected if isinstance(expected, list) else [expected])
|
||||
if op in ("gte", "lte", "gt", "lt"):
|
||||
try: return {"gte": value >= expected, "lte": value <= expected, "gt": value > expected, "lt": value < expected}[op]
|
||||
except (TypeError, ValueError): return False
|
||||
return False
|
||||
|
||||
def evaluate_score(signals, rules):
|
||||
total = 0; explanations = []
|
||||
ordered = sorted((dict(r) for r in rules), key=lambda r: (str(r.get("code", "")), int(r.get("id", 0) or 0)))
|
||||
for rule in ordered:
|
||||
enabled = bool(rule.get("enabled", 1)); applied = enabled and _match(_condition(rule), signals)
|
||||
points = int(rule.get("points", 0) or 0) if applied else 0
|
||||
total += points
|
||||
explanations.append({"code": rule.get("code", ""), "name": rule.get("name", rule.get("code", "")), "version": int(rule.get("version", 1) or 1), "enabled": enabled, "applied": applied, "points": points, "reason": (rule.get("description") or rule.get("name") or rule.get("code") or "Rule") + (" (matched)" if applied else " (not matched)")})
|
||||
total = max(0, min(100, total)); state = signals.get("state", {}) if isinstance(signals, dict) else {}
|
||||
eligible = not bool(state.get("suppressed")) and str(state.get("merge_status", "active")) == "active"
|
||||
if not eligible: band = "ineligible"
|
||||
elif total >= 70: band = "high"
|
||||
elif total >= 40: band = "medium"
|
||||
else: band = "low"
|
||||
return {"score": total, "score_version": SCORE_VERSION, "eligible": eligible, "priority_band": band, "explanations": explanations}
|
||||
|
||||
def _condition(rule):
|
||||
raw = rule.get("condition_json", {})
|
||||
if isinstance(raw, str):
|
||||
try: return json.loads(raw)
|
||||
except (TypeError, ValueError): return {}
|
||||
return raw
|
||||
|
||||
def signals_for_business(business, website=None, contacts=None, domain=None, suppressed=False):
|
||||
b = dict(business); website = website or {}; contacts = contacts or []; domain = domain or {}
|
||||
public = [c for c in contacts if c.get("public_business") and not c.get("suppressed") and not c.get("do_not_contact")]
|
||||
return {"business": {"name": b.get("name", ""), "email": b.get("email", ""), "phone": b.get("phone", ""), "description": b.get("description", ""), "website_domain": b.get("website_domain", ""), "website_class": b.get("website_class", "")}, "website": website, "contacts": {"count": len(contacts), "public_count": len(public)}, "domain": domain, "state": {"verified": bool(b.get("verified")), "suppressed": bool(suppressed), "merge_status": b.get("merge_status", "active"), "merged": b.get("merge_status") == "merged"}}
|
||||
@@ -169,6 +169,8 @@ def scan_website(url: str, *, timeout: float = DEFAULT_TIMEOUT, max_bytes: int =
|
||||
if not 0 < int(max_redirects) <= MAX_REDIRECTS or not 0 < int(max_pages) <= MAX_PAGES: raise ValueError("invalid_limits")
|
||||
fetched = _fetch(url, timeout=max(0.1, min(float(timeout), 10.0)), max_bytes=max(1, min(int(max_bytes), MAX_BYTES)), max_redirects=int(max_redirects))
|
||||
result.update({k: fetched[k] for k in ("status", "final_url", "redirect_chain", "elapsed_ms", "tls", "certificate_status")}); result["size_bytes"] = len(fetched["body"])
|
||||
if fetched["content_type"] in {"text/html", "application/xhtml+xml"}:
|
||||
result["html"] = fetched["body"].decode("utf-8", "replace")
|
||||
if fetched["content_type"] not in {"text/html", "application/xhtml+xml"}:
|
||||
result["classification"] = classify_website(fetched["status"], fetched["final_url"], ""); return result
|
||||
parser = _PageParser(); parser.feed(fetched["body"].decode("utf-8", "replace"))
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
{
|
||||
"seed": 1601,
|
||||
"limitations": [
|
||||
"Synthetic, small, English-heavy cases are not representative of production traffic.",
|
||||
"Contact labels cover parser false positives but do not establish consent, identity, or deliverability.",
|
||||
"Website fixtures classify supplied HTML only; they do not measure network, DNS, redirects, or adversarial pages.",
|
||||
"Latency and cache measurements are local pilot signals and vary by host load."
|
||||
],
|
||||
"normalization": [
|
||||
{"name": "SA contact fields", "input": {"name": " Acme Solar ", "website": "HTTPS://WWW.ACME.TEST/path", "phone": "082 555 1234", "email": " SALES@ACME.TEST "}, "expected": {"name": "Acme Solar", "website_domain": "acme.test", "email": "sales@acme.test"}},
|
||||
{"name": "accented location", "input": {"name": "Cafe", "province": " Western Cape ", "city": "CAPE TOWN", "suburb": "Sea Point"}, "expected": {"province": "western cape", "city": "cape town", "suburb": "sea point"}}
|
||||
],
|
||||
"matching": [
|
||||
{
|
||||
"name": "shared exact domain is a positive",
|
||||
"source": {"name": "Acme Solar", "website": "https://www.acme.test", "city": "Johannesburg"},
|
||||
"candidates": [
|
||||
{"id": 1, "name": "Acme Solar (Pty) Ltd", "website": "http://acme.test/", "city": "Johannesburg"},
|
||||
{"id": 2, "name": "Acme Supplies", "website": "https://supplies.test", "city": "Johannesburg"}
|
||||
],
|
||||
"threshold": 0.72,
|
||||
"expected_ids": [1],
|
||||
"forbidden_ids": [2]
|
||||
},
|
||||
{
|
||||
"name": "similar name without corroboration is not a positive",
|
||||
"source": {"name": "Brightline Consulting", "city": "Cape Town"},
|
||||
"candidates": [
|
||||
{"id": 3, "name": "Brightline Consulting", "city": "Durban"},
|
||||
{"id": 4, "name": "Brightline Consulting", "city": "Cape Town"}
|
||||
],
|
||||
"threshold": 0.72,
|
||||
"expected_ids": [4],
|
||||
"forbidden_ids": [3]
|
||||
},
|
||||
{
|
||||
"name": "unrelated businesses are negatives",
|
||||
"source": {"name": "Green Oak Architects", "website": "green-oak.test"},
|
||||
"candidates": [
|
||||
{"id": 5, "name": "Green Oak Accounting", "website": "accounting.test"},
|
||||
{"id": 6, "name": "Red River Bakery", "website": "redriver.test"}
|
||||
],
|
||||
"threshold": 0.72,
|
||||
"expected_ids": [],
|
||||
"forbidden_ids": [5, 6]
|
||||
}
|
||||
],
|
||||
"contacts": [
|
||||
{
|
||||
"name": "public contacts and ignored markup",
|
||||
"source_url": "https://acme.test/contact",
|
||||
"html": "<p>Sales: sales@acme.test</p><p>Call +27 (12) 345-6789</p><script>secret@acme.test</script><img src=\"https://tracker.test/pixel?email=tracker@tracker.test\"><p>API key: token@acme.test</p>",
|
||||
"expected": [["email", "sales@acme.test"], ["phone", "+27123456789"]],
|
||||
"forbidden": [["email", "secret@acme.test"], ["email", "tracker@tracker.test"], ["email", "token@acme.test"]]
|
||||
},
|
||||
{
|
||||
"name": "forms are provenance not email destinations",
|
||||
"source_url": "https://bright.test/contact",
|
||||
"html": "<form><input name=\"email\" placeholder=\"Your email\"></form><p>hello [at] bright.test</p>",
|
||||
"expected": [["email", "hello@bright.test"]],
|
||||
"forbidden": [["email", "email@bright.test"]]
|
||||
}
|
||||
],
|
||||
"websites": [
|
||||
{"name": "healthy", "status": 200, "url": "https://acme.test", "body": "<html><h1>Acme Solar</h1><p>Welcome to our business.</p></html>", "expected": "healthy"},
|
||||
{"name": "parked", "status": 200, "url": "https://parked.test", "body": "Domain for sale - buy this domain", "expected": "parked"},
|
||||
{"name": "construction", "status": 200, "url": "https://new.test", "body": "Website coming soon", "expected": "under_construction"},
|
||||
{"name": "broken", "status": 404, "url": "https://gone.test", "body": "", "expected": "broken"},
|
||||
{"name": "blocked", "status": null, "url": "https://private.test", "body": "", "error": "unsafe_address", "expected": "blocked"},
|
||||
{"name": "unknown is not healthy", "status": null, "url": "https://unknown.test", "body": "", "expected": "unknown"}
|
||||
],
|
||||
"scoring": [
|
||||
{"name": "complete active business", "signals": {"business": {"name": "Acme", "email": "a@acme.test", "phone": "+27123456789"}, "website": {"classification": "healthy"}, "contacts": {"public_count": 1}, "domain": {"status": "resolved"}, "state": {"verified": true, "suppressed": false, "merge_status": "active"}}},
|
||||
{"name": "suppressed remains ineligible", "signals": {"business": {"name": "Suppressed"}, "state": {"suppressed": true, "merge_status": "active"}}, "must_be_ineligible": true},
|
||||
{"name": "stale evidence does not create positive", "signals": {"business": {"name": "Stale"}, "website": {"classification": "healthy", "stale": true}, "state": {"suppressed": false, "merge_status": "active"}}}
|
||||
],
|
||||
"tenant_isolation": {
|
||||
"tenant_a": {"id": 101, "business_ids": [1, 2]},
|
||||
"tenant_b": {"id": 202, "business_ids": [3, 4]},
|
||||
"cross_tenant_ids": [3, 4]
|
||||
}
|
||||
}
|
||||
@@ -89,6 +89,16 @@ CREATE INDEX IF NOT EXISTS idx_pipeline_stage ON pipeline_entries(organization_i
|
||||
CREATE INDEX IF NOT EXISTS idx_notes_business ON notes(business_id,created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_interactions_business ON interactions(business_id,created_at);
|
||||
|
||||
-- Phase 12 CRM workflow metadata and safe, auditable state.
|
||||
CREATE TABLE IF NOT EXISTS pipeline_stages (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL, position INTEGER NOT NULL DEFAULT 0, active INTEGER NOT NULL DEFAULT 1,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,name)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_pipeline_stages_org ON pipeline_stages(organization_id,position,id);
|
||||
CREATE INDEX IF NOT EXISTS idx_interactions_org_created ON interactions(organization_id,created_at DESC,id DESC);
|
||||
|
||||
-- Phase 4 durable background jobs (additive-safe for existing databases).
|
||||
CREATE TABLE IF NOT EXISTS jobs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
@@ -188,3 +198,118 @@ CREATE TABLE IF NOT EXISTS website_scans (
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_org ON website_scans(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_business ON website_scans(organization_id,business_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_website_scans_cache ON website_scans(organization_id,cache_key,cache_expires_at);
|
||||
|
||||
-- Phase 9 contact extraction provenance. MX is intentionally a state value, not a probe.
|
||||
CREATE TABLE IF NOT EXISTS contact_extractions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id),
|
||||
business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
|
||||
website_scan_id INTEGER REFERENCES website_scans(id) ON DELETE SET NULL,
|
||||
extraction_key TEXT NOT NULL,
|
||||
kind TEXT NOT NULL,
|
||||
value TEXT NOT NULL,
|
||||
label TEXT NOT NULL DEFAULT '',
|
||||
classification TEXT NOT NULL DEFAULT 'unknown',
|
||||
confidence REAL NOT NULL DEFAULT 0,
|
||||
source_url TEXT NOT NULL,
|
||||
public_business INTEGER NOT NULL DEFAULT 1,
|
||||
mx_status TEXT NOT NULL DEFAULT 'unknown',
|
||||
suppressed INTEGER NOT NULL DEFAULT 0,
|
||||
do_not_contact INTEGER NOT NULL DEFAULT 0,
|
||||
provenance TEXT NOT NULL DEFAULT '',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,business_id,extraction_key,kind,value)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_contact_extractions_org ON contact_extractions(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_contact_extractions_business ON contact_extractions(organization_id,business_id,id DESC);
|
||||
|
||||
-- Phase 10 deterministic qualification rules and immutable score audit.
|
||||
CREATE TABLE IF NOT EXISTS score_rules (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
code TEXT NOT NULL, name TEXT NOT NULL, description TEXT NOT NULL DEFAULT '', condition_json TEXT NOT NULL DEFAULT '{}',
|
||||
points INTEGER NOT NULL DEFAULT 0, max_applications INTEGER NOT NULL DEFAULT 1, enabled INTEGER NOT NULL DEFAULT 1,
|
||||
version INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,code)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_score_rules_org ON score_rules(organization_id,enabled,code);
|
||||
CREATE TABLE IF NOT EXISTS score_history (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT, organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE, score INTEGER NOT NULL, eligible INTEGER NOT NULL,
|
||||
priority_band TEXT NOT NULL, score_version TEXT NOT NULL, explanations_json TEXT NOT NULL DEFAULT '[]', signals_json TEXT NOT NULL DEFAULT '{}',
|
||||
override_score INTEGER, override_eligible INTEGER, override_reason TEXT, actor_user_id INTEGER REFERENCES users(id) ON DELETE SET NULL,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_score_history_business ON score_history(organization_id,business_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_score_history_org ON score_history(organization_id,created_at DESC,id DESC);
|
||||
|
||||
-- Phase 11 operator workflow: saved views and review state.
|
||||
CREATE TABLE IF NOT EXISTS saved_filters (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
filters_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,user_id,name)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_saved_filters_org_user ON saved_filters(organization_id,user_id,updated_at DESC,id DESC);
|
||||
|
||||
-- Phase 13 optional, evidence-bounded AI assistance. Outputs are drafts only.
|
||||
CREATE TABLE IF NOT EXISTS ai_runs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
business_id INTEGER REFERENCES businesses(id) ON DELETE SET NULL,
|
||||
input_evidence_hashes_json TEXT NOT NULL DEFAULT '[]',
|
||||
model TEXT NOT NULL DEFAULT '', provider TEXT NOT NULL DEFAULT '', version TEXT NOT NULL DEFAULT '',
|
||||
prompt_metadata_json TEXT NOT NULL DEFAULT '{}', data_minimization_json TEXT NOT NULL DEFAULT '{}',
|
||||
status TEXT NOT NULL CHECK(status IN ('not_configured','succeeded','failed','rejected')),
|
||||
approval_state TEXT NOT NULL DEFAULT 'pending' CHECK(approval_state IN ('pending','approved','rejected')),
|
||||
output_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, approved_at TEXT, rejected_at TEXT,
|
||||
actor_user_id INTEGER REFERENCES users(id) ON DELETE SET NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_runs_org_created ON ai_runs(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_runs_business ON ai_runs(organization_id,business_id,created_at DESC,id DESC);
|
||||
CREATE TABLE IF NOT EXISTS ai_suggestions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ai_run_id INTEGER NOT NULL REFERENCES ai_runs(id) ON DELETE CASCADE,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
business_id INTEGER REFERENCES businesses(id) ON DELETE SET NULL,
|
||||
suggestion_type TEXT NOT NULL, citations_json TEXT NOT NULL DEFAULT '[]', output_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_ai_suggestions_run ON ai_suggestions(organization_id,ai_run_id,id);
|
||||
|
||||
-- Phase 14 outreach preparation. Provider configuration is metadata plus a
|
||||
-- one-way secret fingerprint; outbound transport is intentionally disabled.
|
||||
CREATE TABLE IF NOT EXISTS outreach_provider_configs (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
provider TEXT NOT NULL DEFAULT '', enabled INTEGER NOT NULL DEFAULT 0,
|
||||
secret_fingerprint TEXT NOT NULL DEFAULT '',
|
||||
policy_json TEXT NOT NULL DEFAULT '{"consent_required":true}',
|
||||
daily_cap INTEGER NOT NULL DEFAULT 100, batch_cap INTEGER NOT NULL DEFAULT 25,
|
||||
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id)
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS outreach_drafts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
organization_id TEXT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
business_id INTEGER NOT NULL REFERENCES businesses(id) ON DELETE CASCADE,
|
||||
target_kind TEXT NOT NULL CHECK(target_kind IN ('email','phone','whatsapp')),
|
||||
target_value TEXT NOT NULL,
|
||||
target_verified INTEGER NOT NULL DEFAULT 0,
|
||||
subject TEXT NOT NULL DEFAULT '', body TEXT NOT NULL DEFAULT '',
|
||||
template_json TEXT NOT NULL DEFAULT '{}', citations_json TEXT NOT NULL DEFAULT '[]',
|
||||
provenance_json TEXT NOT NULL DEFAULT '{}', status TEXT NOT NULL DEFAULT 'pending_review'
|
||||
CHECK(status IN ('pending_review','approved','blocked','sent')),
|
||||
legal_basis TEXT NOT NULL DEFAULT '', consent_confirmed INTEGER NOT NULL DEFAULT 0,
|
||||
actor_user_id INTEGER REFERENCES users(id) ON DELETE SET NULL,
|
||||
approved_by INTEGER REFERENCES users(id) ON DELETE SET NULL, approved_at TEXT,
|
||||
sent_at TEXT, idempotency_key TEXT, created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(organization_id,idempotency_key)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_outreach_drafts_org ON outreach_drafts(organization_id,created_at DESC,id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_outreach_drafts_business ON outreach_drafts(organization_id,business_id,id DESC);
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server
|
||||
from app.scoring import DEFAULT_RULES, evaluate_score
|
||||
|
||||
|
||||
class ScoringEngineTests(unittest.TestCase):
|
||||
def test_defaults_are_deterministic_and_emit_explanations_and_band(self):
|
||||
signals = {"business": {"name": "Acme", "email": "a@acme.test", "website_domain": "acme.test"}, "website": {"classification": "healthy"}, "state": {"suppressed": False}}
|
||||
first = evaluate_score(signals, DEFAULT_RULES)
|
||||
self.assertEqual(first, evaluate_score(signals, DEFAULT_RULES))
|
||||
self.assertEqual(0 <= first["score"] <= 100, True)
|
||||
self.assertEqual(first["priority_band"], "medium")
|
||||
self.assertTrue(all("code" in item and "reason" in item for item in first["explanations"]))
|
||||
|
||||
def test_disabled_and_versioned_rules_change_score_without_nondeterminism(self):
|
||||
signals = {"business": {"name": "Acme"}, "state": {"suppressed": False}}
|
||||
enabled = evaluate_score(signals, [{"code": "x", "name": "X", "condition_json": {"signal": "business.name", "operator": "present"}, "points": 30, "enabled": 1, "version": 1}])
|
||||
disabled = evaluate_score(signals, [{"code": "x", "name": "X", "condition_json": {"signal": "business.name", "operator": "present"}, "points": 30, "enabled": 0, "version": 2}])
|
||||
self.assertEqual(enabled["score"], 30)
|
||||
self.assertEqual(disabled["score"], 0)
|
||||
|
||||
def test_suppression_is_ineligible_and_stale_uncertain_signals_do_not_penalize(self):
|
||||
signals = {"business": {"name": "Acme"}, "website": {"classification": "unknown", "stale": True}, "domain": {"status": "error"}, "state": {"suppressed": True}}
|
||||
result = evaluate_score(signals, DEFAULT_RULES)
|
||||
self.assertFalse(result["eligible"])
|
||||
self.assertEqual(result["priority_band"], "ineligible")
|
||||
self.assertNotIn("negative", json.dumps(result["explanations"]).lower())
|
||||
|
||||
|
||||
class ScoringApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory(); self.db_path = self.tmp.name + "/db.sqlite"
|
||||
os.environ["BOOTSTRAP_ADMIN_EMAIL"] = "score-owner@example.test"; os.environ["BOOTSTRAP_ADMIN_PASSWORD"] = "password"
|
||||
self.server = create_server("127.0.0.1", 0, self.db_path); threading.Thread(target=self.server.serve_forever, daemon=True).start()
|
||||
self.conn = HTTPConnection("127.0.0.1", self.server.server_port, timeout=4); self.cookie = None
|
||||
self.request("POST", "/api/v1/auth/login", {"email": "score-owner@example.test", "password": "password"})
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.tmp.cleanup()
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None; headers = {"Content-Type": "application/json"} if body else {}
|
||||
if self.cookie: headers["Cookie"] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse(); cookie = response.getheader("Set-Cookie")
|
||||
if cookie: self.cookie = cookie.split(";", 1)[0]
|
||||
return response.status, json.loads(response.read() or b"{}")
|
||||
def test_rule_crud_tenant_scope_and_recalculation_audit(self):
|
||||
status, rules = self.request("GET", "/api/v1/score-rules"); self.assertEqual(status, 200); self.assertEqual(len(rules["items"]), len(DEFAULT_RULES))
|
||||
status, rule = self.request("POST", "/api/v1/score-rules", {"code": "custom", "name": "Custom", "condition_json": {"signal": "business.name", "operator": "present"}, "points": 7}); self.assertEqual(status, 201)
|
||||
self.assertEqual(self.request("PATCH", f"/api/v1/score-rules/{rule['id']}", {"enabled": False})[0], 200)
|
||||
_, business = self.request("POST", "/api/v1/businesses", {"name": "Acme", "website": "https://acme.test", "email": "a@acme.test"})
|
||||
status, result = self.request("POST", f"/api/v1/businesses/{business['id']}/score/recalculate", {"reason": "qa"}); self.assertEqual(status, 200)
|
||||
self.assertIn("explanations", result); self.assertIn("eligible", result); self.assertEqual(result["business_id"], business["id"])
|
||||
status, summary = self.request("GET", "/api/v1/scoring/summary"); self.assertEqual(status, 200); self.assertEqual(summary["businesses"], 1); self.assertGreaterEqual(summary["history_count"], 1)
|
||||
db = sqlite3.connect(self.db_path); self.assertEqual(db.execute("SELECT COUNT(*) FROM score_history").fetchone()[0], 1); db.close()
|
||||
def test_suppressed_business_recalculation_is_force_ineligible(self):
|
||||
_, business = self.request("POST", "/api/v1/businesses", {"name": "Acme", "website": "https://acme.test"})
|
||||
self.request("POST", "/api/v1/suppressions", {"kind": "domain", "value": "acme.test"})
|
||||
status, result = self.request("POST", f"/api/v1/businesses/{business['id']}/score/recalculate", {}); self.assertEqual(status, 200); self.assertFalse(result["eligible"])
|
||||
|
||||
|
||||
if __name__ == "__main__": unittest.main()
|
||||
@@ -0,0 +1,52 @@
|
||||
import sqlite3
|
||||
try:
|
||||
from test_api import ApiSmokeTests
|
||||
except ModuleNotFoundError:
|
||||
from tests.test_api import ApiSmokeTests
|
||||
|
||||
|
||||
class OperatorWorkflowTests(ApiSmokeTests):
|
||||
def test_saved_filter_round_trip_update_delete_and_bounds(self):
|
||||
status, saved = self.request("POST", "/api/v1/saved-filters", {"name": "Hot", "filters": {"score_min": 70, "website_state": "business_site"}})
|
||||
self.assertEqual(status, 201)
|
||||
self.assertEqual(saved["filters"]["score_min"], 70)
|
||||
fid = saved["id"]
|
||||
status, listing = self.request("GET", "/api/v1/saved-filters")
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(listing["items"][0]["name"], "Hot")
|
||||
self.assertEqual(self.request("PATCH", f"/api/v1/saved-filters/{fid}", {"filters": {"priority": "high"}})[0], 200)
|
||||
self.assertEqual(self.request("DELETE", f"/api/v1/saved-filters/{fid}")[0], 200)
|
||||
self.assertEqual(self.request("POST", "/api/v1/saved-filters", {"name": "Bad", "filters": {"x": ["a"] * 51}})[0], 400)
|
||||
|
||||
def test_review_queue_filters_flags_suppression_and_bulk_is_atomic(self):
|
||||
_, first = self.request("POST", "/api/v1/businesses", {"name": "Queue One", "website": "https://queue-one.test", "email": "one@queue.test"})
|
||||
_, second = self.request("POST", "/api/v1/businesses", {"name": "Queue Two", "website": "https://queue-two.test"})
|
||||
self.assertEqual(self.request("POST", "/api/v1/suppressions", {"kind": "domain", "value": "queue-one.test"})[0], 201)
|
||||
status, queue = self.request("GET", "/api/v1/review-queue?review_status=pending&website_state=business_site&page_size=10")
|
||||
self.assertEqual(status, 200)
|
||||
flagged = next(x for x in queue["items"] if x["id"] == first["id"])
|
||||
self.assertTrue(flagged["suppressed"])
|
||||
self.assertFalse(flagged["outreach_eligible"])
|
||||
self.assertEqual(self.request("POST", "/api/v1/businesses/bulk-review", {"ids": [first["id"], second["id"]], "action": "verify"})[0], 200)
|
||||
status, queue = self.request("GET", "/api/v1/review-queue?review_status=verified&page_size=10")
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual({x["id"] for x in queue["items"]}, {first["id"], second["id"]})
|
||||
self.assertFalse(next(x for x in queue["items"] if x["id"] == first["id"])["outreach_eligible"])
|
||||
self.assertEqual(self.request("POST", "/api/v1/businesses/bulk-review", {"ids": [first["id"]] * 101, "action": "reject"})[0], 400)
|
||||
db = sqlite3.connect(self.db_path)
|
||||
self.assertEqual(db.execute("SELECT COUNT(*) FROM audit_log WHERE action='businesses.bulk_review'").fetchone()[0], 1)
|
||||
db.close()
|
||||
|
||||
def test_dashboard_operator_counts_and_clickable_metadata(self):
|
||||
self.request("POST", "/api/v1/businesses", {"name": "Dashboard New", "website": "https://dash.test"})
|
||||
status, summary = self.request("GET", "/api/v1/dashboard/summary")
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(summary["counts"]["new"], 1)
|
||||
for key in ("new", "hot", "review", "source_health", "active_jobs"):
|
||||
self.assertIn(key, summary["clickable_filters"])
|
||||
self.assertEqual(summary["clickable_filters"][key]["count"], summary["counts"][key])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import unittest
|
||||
unittest.main()
|
||||
@@ -0,0 +1,42 @@
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server
|
||||
|
||||
|
||||
class Phase12Tests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory(); os.environ['BOOTSTRAP_ADMIN_EMAIL']='p12@example.test'; os.environ['BOOTSTRAP_ADMIN_PASSWORD']='password-p12'
|
||||
self.server=create_server('127.0.0.1',0,self.tmp.name+'/db.sqlite'); self.thread=threading.Thread(target=self.server.serve_forever,daemon=True); self.thread.start(); self.conn=HTTPConnection('127.0.0.1',self.server.server_port); self.cookie=None
|
||||
self.request('POST','/api/v1/auth/login',{'email':'p12@example.test','password':'password-p12'})
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(timeout=2); os.environ.pop('BOOTSTRAP_ADMIN_EMAIL',None); os.environ.pop('BOOTSTRAP_ADMIN_PASSWORD',None); self.tmp.cleanup()
|
||||
def request(self,m,p,b=None):
|
||||
h={'Content-Type':'application/json'} if b is not None else {}; h['Cookie']=self.cookie if self.cookie else ''
|
||||
self.conn.request(m,p,json.dumps(b).encode() if b is not None else None,h); r=self.conn.getresponse(); sc=r.getheader('Set-Cookie');
|
||||
if sc: self.cookie=sc.split(';',1)[0]
|
||||
return r.status,json.loads(r.read() or b'{}')
|
||||
def test_pipeline_interaction_transition_and_no_contact(self):
|
||||
s,b=self.request('POST','/api/v1/businesses',{'name':'CRM Co','email':'crm@example.test'}); self.assertEqual(s,201)
|
||||
s,p=self.request('POST',f"/api/v1/businesses/{b['id']}/pipeline",{'stage':'new','status':'active','notes':'first','next_action':'qualify','follow_up_at':'2026-09-10T10:00:00Z','idempotency_key':'p1'}); self.assertEqual(s,201); self.assertEqual(p['actor_user_id'],1)
|
||||
s,p2=self.request('PATCH',f"/api/v1/pipeline-entries/{p['id']}",{'stage':'qualified','expected_updated_at':p['updated_at']}); self.assertEqual(s,200); self.assertEqual(p2['stage'],'qualified')
|
||||
s,i=self.request('POST',f"/api/v1/businesses/{b['id']}/interactions",{'kind':'call','outcome':'connected','notes':'good fit','next_action':'demo'}); self.assertEqual(s,201); self.assertEqual(i['outcome'],'connected')
|
||||
self.assertEqual(self.request('GET','/api/v1/reports/pipeline')[0],200); self.assertEqual(self.request('GET','/api/v1/reports/outcomes')[0],200); self.assertEqual(self.request('GET','/api/v1/reports/activity')[0],200)
|
||||
def test_suppression_center_bulk_and_enforces_pipeline_interaction(self):
|
||||
s,b=self.request('POST','/api/v1/businesses',{'name':'Blocked CRM','email':'blocked@example.test'}); self.assertEqual(s,201)
|
||||
s,x=self.request('POST','/api/v1/suppressions/import',{'items':[{'kind':'email','value':'blocked@example.test'},{'kind':'domain','value':'bad.example'}]}); self.assertEqual(s,201); self.assertEqual(x['imported'],2)
|
||||
s,l=self.request('GET','/api/v1/suppressions'); self.assertEqual(s,200); self.assertEqual(len(l['items']),2)
|
||||
self.assertEqual(self.request('POST',f"/api/v1/businesses/{b['id']}/pipeline",{'stage':'new'})[0],409)
|
||||
self.assertEqual(self.request('POST',f"/api/v1/businesses/{b['id']}/interactions",{'kind':'call','outcome':'connected'})[0],409)
|
||||
sid=l['items'][0]['id']; self.assertEqual(self.request('PATCH',f'/api/v1/suppressions/{sid}',{'active':False})[0],200)
|
||||
self.assertEqual(self.request('DELETE',f'/api/v1/suppressions/{sid}')[0],200)
|
||||
def test_validation_and_date_bounds(self):
|
||||
s,b=self.request('POST','/api/v1/businesses',{'name':'Validation CRM'}); self.assertEqual(s,201)
|
||||
self.assertEqual(self.request('POST',f"/api/v1/businesses/{b['id']}/interactions",{'kind':'call','outcome':'not-a-real-outcome'})[0],400)
|
||||
self.assertEqual(self.request('GET','/api/v1/reports/activity?from=2020-01-01&to=2035-01-01')[0],400)
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,82 @@
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.ai_assistance import build_local_suggestions, generate
|
||||
from app.main import create_server
|
||||
from app.main import hash_password
|
||||
|
||||
|
||||
class Phase13ApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory()
|
||||
self.old = {key: os.environ.get(key) for key in ("AI_PROVIDER", "BOOTSTRAP_ADMIN_EMAIL", "BOOTSTRAP_ADMIN_PASSWORD")}
|
||||
os.environ["BOOTSTRAP_ADMIN_EMAIL"] = "ai-owner@example.test"
|
||||
os.environ["BOOTSTRAP_ADMIN_PASSWORD"] = "ai-password"
|
||||
os.environ["AI_PROVIDER"] = "local"
|
||||
self.server = create_server("127.0.0.1", 0, self.tmp.name + "/ai.db")
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
|
||||
self.conn = HTTPConnection("127.0.0.1", self.server.server_port, timeout=3); self.cookie = None
|
||||
self.request("POST", "/api/v1/auth/login", {"email": "ai-owner@example.test", "password": "ai-password"})
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(timeout=2); self.tmp.cleanup()
|
||||
for key, value in self.old.items():
|
||||
if value is None: os.environ.pop(key, None)
|
||||
else: os.environ[key] = value
|
||||
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None; headers = {"Content-Type": "application/json"}
|
||||
if self.cookie: headers["Cookie"] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse(); cookie = response.getheader("Set-Cookie")
|
||||
if cookie: self.cookie = cookie.split(";", 1)[0]
|
||||
return response.status, json.loads(response.read() or b"{}")
|
||||
|
||||
def test_local_fallback_is_deterministic_and_evidence_bounded(self):
|
||||
args = ({"id": 4, "name": "Acme", "score": 10}, [], [], [{"id": 2, "kind": "source", "url": "https://source.test", "claim": "Makes widgets"}])
|
||||
first = build_local_suggestions(*args); second = build_local_suggestions(*args)
|
||||
self.assertEqual(first, second)
|
||||
self.assertIn("[evidence:2]", first["suggestions"][0]["text"])
|
||||
self.assertNotIn("customers", json.dumps(first).lower())
|
||||
|
||||
def test_no_provider_returns_not_configured_without_output(self):
|
||||
os.environ.pop("AI_PROVIDER", None)
|
||||
status, provider, version, metadata = generate({"name": "Acme"}, [], [], [])
|
||||
self.assertEqual(status, "not_configured"); self.assertEqual(provider, ""); self.assertEqual(version, "")
|
||||
self.assertNotIn("output", metadata)
|
||||
|
||||
def test_endpoint_persists_citations_and_approval_without_crm_write(self):
|
||||
status, business = self.request("POST", "/api/v1/businesses", {"name": "Evidence Co", "website": "https://evidence.test"}); self.assertEqual(status, 201)
|
||||
bid = business["id"]
|
||||
self.request("POST", f"/api/v1/businesses/{bid}/evidence", {"kind": "source", "url": "https://source.test", "claim": "Serves Cape Town"})
|
||||
status, run = self.request("POST", f"/api/v1/businesses/{bid}/ai/suggest", {}); self.assertEqual(status, 201)
|
||||
self.assertTrue(all(s["citations"] for s in run["suggestions"]))
|
||||
self.assertEqual(self.request("POST", f"/api/v1/ai-runs/{run['id']}/approve", {})[1]["approval_state"], "approved")
|
||||
self.assertEqual(self.request("POST", f"/api/v1/ai-runs/{run['id']}/reject", {})[0], 409)
|
||||
db = sqlite3.connect(self.tmp.name + "/ai.db")
|
||||
self.assertEqual(db.execute("SELECT COUNT(*) FROM pipeline_entries").fetchone()[0], 0); self.assertEqual(db.execute("SELECT COUNT(*) FROM interactions").fetchone()[0], 0); db.close()
|
||||
|
||||
def test_limits_and_suppressed_business_are_safe(self):
|
||||
status, business = self.request("POST", "/api/v1/businesses", {"name": "Safe Co"}); self.assertEqual(status, 201)
|
||||
bid = business["id"]
|
||||
self.assertEqual(self.request("POST", f"/api/v1/businesses/{bid}/ai/suggest", {"max_items": 101})[0], 400)
|
||||
self.request("POST", "/api/v1/suppressions", {"kind": "domain", "value": "safe.test"})
|
||||
# Directly mark the business with the suppressed domain to exercise the AI guard.
|
||||
db = sqlite3.connect(self.tmp.name + "/ai.db"); db.execute("UPDATE businesses SET website_domain='safe.test' WHERE id=?", (bid,)); db.commit(); db.close()
|
||||
self.assertEqual(self.request("POST", f"/api/v1/businesses/{bid}/ai/suggest", {})[0], 409)
|
||||
def test_tenant_isolation_applies_to_ai_runs_and_business_suggestions(self):
|
||||
status, business = self.request("POST", "/api/v1/businesses", {"name": "Tenant A"}); self.assertEqual(status, 201)
|
||||
ph, salt = hash_password("other-password")
|
||||
db = sqlite3.connect(self.tmp.name + "/ai.db")
|
||||
db.execute("INSERT INTO organizations (id,name) VALUES (?,?)", ("other-tenant", "Other"))
|
||||
db.execute("INSERT INTO users (organization_id,email,password_hash,password_salt,role) VALUES (?,?,?,?,?)", ("other-tenant", "other-ai@example.test", ph, salt, "owner")); db.commit(); db.close()
|
||||
self.cookie = None; self.assertEqual(self.request("POST", "/api/v1/auth/login", {"email": "other-ai@example.test", "password": "other-password"})[0], 200)
|
||||
self.assertEqual(self.request("POST", f"/api/v1/businesses/{business['id']}/ai/suggest", {})[0], 404)
|
||||
self.assertEqual(self.request("GET", "/api/v1/ai-runs")[1]["items"], [])
|
||||
|
||||
|
||||
if __name__ == "__main__": unittest.main()
|
||||
@@ -0,0 +1,82 @@
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server, hash_password
|
||||
|
||||
|
||||
class Phase14ApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory()
|
||||
self.old = {k: os.environ.get(k) for k in ("BOOTSTRAP_ADMIN_EMAIL", "BOOTSTRAP_ADMIN_PASSWORD")}
|
||||
os.environ["BOOTSTRAP_ADMIN_EMAIL"] = "outreach-owner@example.test"
|
||||
os.environ["BOOTSTRAP_ADMIN_PASSWORD"] = "outreach-password"
|
||||
self.db_path = self.tmp.name + "/outreach.db"
|
||||
self.server = create_server("127.0.0.1", 0, self.db_path)
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
|
||||
self.conn = HTTPConnection("127.0.0.1", self.server.server_port, timeout=3); self.cookie = None
|
||||
self.request("POST", "/api/v1/auth/login", {"email": "outreach-owner@example.test", "password": "outreach-password"})
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(timeout=2); self.tmp.cleanup()
|
||||
for k, v in self.old.items():
|
||||
if v is None: os.environ.pop(k, None)
|
||||
else: os.environ[k] = v
|
||||
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None
|
||||
headers = {"Content-Type": "application/json"} if body else {}
|
||||
if self.cookie: headers["Cookie"] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse()
|
||||
if response.getheader("Set-Cookie"): self.cookie = response.getheader("Set-Cookie").split(";", 1)[0]
|
||||
return response.status, json.loads(response.read() or b"{}")
|
||||
|
||||
def business(self, email="target@example.test"):
|
||||
status, item = self.request("POST", "/api/v1/businesses", {"name": "Target Co", "website": "https://target.test", "email": email})
|
||||
self.assertEqual(status, 201); return item
|
||||
|
||||
def test_draft_lifecycle_is_evidence_grounded_and_audited(self):
|
||||
b = self.business(); bid = b["id"]
|
||||
self.request("POST", f"/api/v1/businesses/{bid}/evidence", {"kind": "source", "url": "https://target.test/about", "claim": "Makes solar panels"})
|
||||
status, draft = self.request("POST", f"/api/v1/businesses/{bid}/outreach/drafts", {"target": {"kind": "email", "value": "target@example.test", "verified": True}, "subject": "Hello {{business.name}}", "body": "We saw: {{evidence.1.claim}}", "idempotency_key": "draft-1"})
|
||||
self.assertEqual(status, 201); self.assertEqual(draft["status"], "pending_review"); self.assertIn("Target Co", draft["subject"]); self.assertIn("Makes solar panels", draft["body"])
|
||||
self.assertEqual(len(draft["citations"]), 1); self.assertEqual(self.request("POST", f"/api/v1/businesses/{bid}/outreach/drafts", {"target": {"kind": "email", "value": "target@example.test", "verified": True}, "subject": "ignored", "body": "ignored", "idempotency_key": "draft-1"})[0], 200)
|
||||
self.assertEqual(self.request("PATCH", f"/api/v1/outreach/drafts/{draft['id']}", {"body": "Updated {{evidence.1.claim}}"})[0], 200)
|
||||
status, approved = self.request("POST", f"/api/v1/outreach/drafts/{draft['id']}/approve", {}); self.assertEqual(status, 200); self.assertEqual(approved["status"], "approved")
|
||||
db = sqlite3.connect(self.db_path); actions = [r[0] for r in db.execute("SELECT action FROM audit_log")]; db.close()
|
||||
self.assertTrue({"outreach_draft.created", "outreach_draft.updated", "outreach_draft.approved"}.issubset(actions))
|
||||
|
||||
def test_send_is_not_configured_and_all_gates_are_reported(self):
|
||||
b = self.business(); bid = b["id"]
|
||||
_, d = self.request("POST", f"/api/v1/businesses/{bid}/outreach/drafts", {"target": {"kind": "email", "value": "target@example.test", "verified": False}, "subject": "Hi", "body": "Body"})
|
||||
status, result = self.request("POST", f"/api/v1/outreach/drafts/{d['id']}/send", {}); self.assertEqual(status, 409); self.assertEqual(result["status"], "not_configured"); self.assertIn("provider", result["blocked_reasons"])
|
||||
|
||||
def test_suppression_and_unapproved_target_block_send_and_no_network(self):
|
||||
b = self.business(); bid = b["id"]
|
||||
self.request("PATCH", "/api/v1/outreach/provider-config", {"provider": "smtp", "enabled": True, "secret": "test-secret", "legal_policy": {"consent_required": False}})
|
||||
self.request("POST", "/api/v1/suppressions", {"kind": "email", "value": "target@example.test"})
|
||||
_, d = self.request("POST", f"/api/v1/businesses/{bid}/outreach/drafts", {"target": {"kind": "email", "value": "target@example.test", "verified": True}, "subject": "Hi", "body": "Body"})
|
||||
self.request("POST", f"/api/v1/outreach/drafts/{d['id']}/approve", {})
|
||||
status, result = self.request("POST", f"/api/v1/outreach/drafts/{d['id']}/send", {}); self.assertEqual(status, 409); self.assertEqual(result["status"], "blocked"); self.assertIn("suppressed", result["blocked_reasons"]); self.assertFalse(result["network_send"])
|
||||
|
||||
def test_provider_config_never_returns_secret_and_tenant_isolation(self):
|
||||
status, config = self.request("PATCH", "/api/v1/outreach/provider-config", {"provider": "smtp", "enabled": True, "secret": "super-secret", "legal_policy": {"consent_required": True}})
|
||||
self.assertEqual(status, 200); self.assertNotIn("secret", json.dumps(config).lower()); self.assertTrue(config["enabled"])
|
||||
self.assertNotIn("super-secret", json.dumps(config))
|
||||
ph, salt = hash_password("other-password"); db = sqlite3.connect(self.db_path); db.execute("INSERT INTO organizations(id,name) VALUES('other-tenant','Other')"); db.execute("INSERT INTO users(organization_id,email,password_hash,password_salt,role) VALUES(?,?,?,?,?)", ("other-tenant", "other@example.test", ph, salt, "owner")); db.commit(); db.close()
|
||||
self.cookie = None; self.assertEqual(self.request("POST", "/api/v1/auth/login", {"email": "other@example.test", "password": "other-password"})[0], 200)
|
||||
self.assertEqual(self.request("GET", "/api/v1/outreach/provider-config")[1]["enabled"], False)
|
||||
self.assertEqual(self.request("GET", "/api/v1/outreach/drafts")[1]["items"], [])
|
||||
|
||||
def test_limits_and_unknown_template_variable(self):
|
||||
b = self.business(); bid = b["id"]
|
||||
status, result = self.request("POST", f"/api/v1/businesses/{bid}/outreach/drafts", {"target": {"kind": "email", "value": "target@example.test", "verified": True}, "subject": "Hi {{not.evidence}}", "body": "Body"})
|
||||
self.assertEqual(status, 400); self.assertEqual(result["error"], "unsupported_template_variable")
|
||||
self.assertEqual(self.request("GET", "/api/v1/outreach/drafts?page_size=101")[0], 400)
|
||||
|
||||
|
||||
if __name__ == "__main__": unittest.main()
|
||||
@@ -0,0 +1,73 @@
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from pathlib import Path
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server
|
||||
from app.config import ConfigError, load_config
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
SCRIPTS = ROOT / "scripts"
|
||||
|
||||
|
||||
class Phase15OpsTests(unittest.TestCase):
|
||||
def test_config_defaults_are_safe_and_production_requires_explicit_secret(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
cfg = load_config({"DATA_DIR": tmp})
|
||||
self.assertFalse(cfg.outreach_enabled)
|
||||
self.assertEqual(cfg.app_env, "development")
|
||||
with self.assertRaises(ConfigError):
|
||||
load_config({"APP_ENV": "production", "DATA_DIR": tmp})
|
||||
cfg = load_config({"APP_ENV": "production", "DATA_DIR": tmp, "SESSION_SECRET": "a" * 32})
|
||||
self.assertEqual(cfg.session_secret, "a" * 32)
|
||||
|
||||
def test_health_and_readiness_report_database_state(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
db_path = str(Path(tmp) / "state.db")
|
||||
server = create_server("127.0.0.1", 0, db_path)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
conn = HTTPConnection("127.0.0.1", server.server_port, timeout=3)
|
||||
for path in ("/api/v1/health/live", "/api/v1/health/ready"):
|
||||
conn.request("GET", path)
|
||||
response = conn.getresponse()
|
||||
payload = json.loads(response.read())
|
||||
self.assertEqual(response.status, 200)
|
||||
self.assertEqual(payload["status"], "ok")
|
||||
self.assertFalse(payload["outreach_enabled"])
|
||||
finally:
|
||||
server.shutdown(); server.server_close(); thread.join(timeout=2)
|
||||
|
||||
def test_backup_restore_integrity_and_checksum(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp); db = root / "prospects.db"; backups = root / "backups"
|
||||
conn = sqlite3.connect(db); conn.execute("CREATE TABLE facts (value TEXT)"); conn.execute("INSERT INTO facts VALUES ('before')"); conn.commit(); conn.close()
|
||||
result = subprocess.run([str(SCRIPTS / "backup_sqlite.sh"), str(db), str(backups), "2"], text=True, capture_output=True, check=True)
|
||||
backup = Path(result.stdout.strip().splitlines()[-1])
|
||||
self.assertTrue(backup.exists()); self.assertTrue(Path(str(backup) + ".sha256").exists())
|
||||
self.assertEqual(hashlib.sha256(backup.read_bytes()).hexdigest(), Path(str(backup) + ".sha256").read_text().split()[0])
|
||||
conn = sqlite3.connect(db); conn.execute("UPDATE facts SET value='after'"); conn.commit(); conn.close()
|
||||
subprocess.run([str(SCRIPTS / "restore_sqlite.sh"), str(backup), str(db), "--confirm-restore"], text=True, capture_output=True, check=True)
|
||||
conn = sqlite3.connect(db); self.assertEqual(conn.execute("SELECT value FROM facts").fetchone()[0], "before"); conn.close()
|
||||
self.assertTrue(any(p.name.startswith("pre-restore-") for p in backups.glob("*.db")))
|
||||
|
||||
def test_backup_output_does_not_contain_secret_values(self):
|
||||
with TemporaryDirectory() as tmp:
|
||||
root = Path(tmp); db = root / "prospects.db"; backups = root / "backups"
|
||||
conn = sqlite3.connect(db); conn.execute("CREATE TABLE logs (message TEXT)"); conn.execute("INSERT INTO logs VALUES ('password=should-not-be-logged')"); conn.commit(); conn.close()
|
||||
result = subprocess.run([str(SCRIPTS / "backup_sqlite.sh"), str(db), str(backups), "1"], text=True, capture_output=True, check=True)
|
||||
combined = result.stdout + result.stderr
|
||||
self.assertNotIn("should-not-be-logged", combined)
|
||||
self.assertNotIn("password=", combined)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,61 @@
|
||||
import json
|
||||
import sys
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from app.domain import match_businesses
|
||||
from app.contact_extractor import extract_contacts
|
||||
from app.website_scanner import classify_website
|
||||
from app.scoring import DEFAULT_RULES, evaluate_score
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
FIXTURE_PATH = ROOT / "fixtures" / "phase16.json"
|
||||
sys.path.insert(0, str(ROOT.parent.parent / "scripts"))
|
||||
from benchmark_phase16 import run_benchmark
|
||||
|
||||
|
||||
class Phase16BenchmarkRegressionTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.fixtures = json.loads(FIXTURE_PATH.read_text())
|
||||
|
||||
def test_fixture_is_stable_and_benchmark_cases_are_deterministic(self):
|
||||
self.assertEqual(self.fixtures["seed"], 1601)
|
||||
first = json.dumps(self.fixtures, sort_keys=True, separators=(",", ":"))
|
||||
second = json.dumps(json.loads(FIXTURE_PATH.read_text()), sort_keys=True, separators=(",", ":"))
|
||||
self.assertEqual(first, second)
|
||||
case = self.fixtures["matching"][0]
|
||||
self.assertEqual(
|
||||
match_businesses(case["source"], case["candidates"], threshold=case["threshold"]),
|
||||
match_businesses(case["source"], case["candidates"], threshold=case["threshold"]),
|
||||
)
|
||||
|
||||
def test_benchmark_semantic_report_is_deterministic_and_passes_thresholds(self):
|
||||
first = run_benchmark(measure_latency=False)
|
||||
second = run_benchmark(measure_latency=False)
|
||||
self.assertEqual(first, second)
|
||||
self.assertTrue(first["passed"])
|
||||
self.assertEqual(first["results"]["matching"]["false_positive"], 0)
|
||||
self.assertEqual(first["results"]["contacts"]["false_positive"], 0)
|
||||
|
||||
def test_labeled_false_positives_stay_suppressed(self):
|
||||
for case in self.fixtures["matching"]:
|
||||
predicted = {item["id"] for item in match_businesses(case["source"], case["candidates"], threshold=case["threshold"])}
|
||||
self.assertTrue(predicted.isdisjoint(set(case["forbidden_ids"])), case["name"])
|
||||
|
||||
for case in self.fixtures["contacts"]:
|
||||
values = {(item["kind"], item["value"]) for item in extract_contacts(case["html"], case["source_url"])}
|
||||
self.assertTrue(values.isdisjoint({tuple(item) for item in case["forbidden"]}), case["name"])
|
||||
|
||||
def test_website_labels_and_scores_have_no_unsafe_positive(self):
|
||||
for case in self.fixtures["websites"]:
|
||||
self.assertEqual(classify_website(case["status"], case["url"], case["body"], error=case.get("error")), case["expected"], case["name"])
|
||||
for case in self.fixtures["scoring"]:
|
||||
result = evaluate_score(case["signals"], DEFAULT_RULES)
|
||||
self.assertEqual(result, evaluate_score(case["signals"], DEFAULT_RULES))
|
||||
if case.get("must_be_ineligible"):
|
||||
self.assertFalse(result["eligible"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,54 @@
|
||||
import json
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
import sys
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
|
||||
from final_acceptance import build_capacity_smoke, validate_report
|
||||
|
||||
|
||||
class Phase17AcceptanceTests(unittest.TestCase):
|
||||
def test_capacity_smoke_is_bounded_and_deterministic(self):
|
||||
first = build_capacity_smoke()
|
||||
second = build_capacity_smoke()
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(first["pagination"]["requested_page_size"], 100)
|
||||
self.assertEqual(first["pagination"]["returned_items"], 100)
|
||||
self.assertEqual(first["pagination"]["synthetic_total"], 1000)
|
||||
self.assertEqual(first["large_batch"]["input_items"], 5000)
|
||||
self.assertLessEqual(first["large_batch"]["retained_items"], 100)
|
||||
self.assertTrue(first["bounded"])
|
||||
|
||||
def test_report_schema_rejects_missing_or_wrong_fields(self):
|
||||
report = {
|
||||
"report": "phase17-final-acceptance",
|
||||
"version": 1,
|
||||
"scope": "local-repository-acceptance",
|
||||
"passed": True,
|
||||
"checks": {},
|
||||
"capacity_smoke": build_capacity_smoke(),
|
||||
"blockers": [],
|
||||
"limitations": [],
|
||||
}
|
||||
self.assertEqual(validate_report(report), [])
|
||||
missing = dict(report)
|
||||
del missing["checks"]
|
||||
self.assertTrue(validate_report(missing))
|
||||
wrong = dict(report)
|
||||
wrong["passed"] = "yes"
|
||||
self.assertTrue(validate_report(wrong))
|
||||
|
||||
def test_report_serialization_is_stable(self):
|
||||
from final_acceptance import deterministic_json
|
||||
|
||||
report = {
|
||||
"z": 1,
|
||||
"a": {"items": [{"b": 2, "a": 1}]},
|
||||
}
|
||||
self.assertEqual(deterministic_json(report), deterministic_json(json.loads(deterministic_json(report))))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,52 @@
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import unittest
|
||||
from http.client import HTTPConnection
|
||||
from tempfile import TemporaryDirectory
|
||||
|
||||
from app.main import create_server
|
||||
|
||||
|
||||
class ContactExtractionApiTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = TemporaryDirectory()
|
||||
os.environ['BOOTSTRAP_ADMIN_EMAIL'] = 'extract-owner@example.test'
|
||||
os.environ['BOOTSTRAP_ADMIN_PASSWORD'] = 'password'
|
||||
self.server = create_server('127.0.0.1', 0, self.tmp.name + '/db.sqlite')
|
||||
self.thread = threading.Thread(target=self.server.serve_forever, daemon=True); self.thread.start()
|
||||
self.conn = HTTPConnection('127.0.0.1', self.server.server_port, timeout=4); self.cookie = None
|
||||
self.request('POST', '/api/v1/auth/login', {'email': 'extract-owner@example.test', 'password': 'password'})
|
||||
|
||||
def tearDown(self):
|
||||
self.server.shutdown(); self.server.server_close(); self.thread.join(2); self.tmp.cleanup()
|
||||
|
||||
def request(self, method, path, payload=None):
|
||||
body = json.dumps(payload).encode() if payload is not None else None
|
||||
headers = {'Content-Type': 'application/json'} if body else {}
|
||||
if self.cookie: headers['Cookie'] = self.cookie
|
||||
self.conn.request(method, path, body, headers); response = self.conn.getresponse()
|
||||
cookie = response.getheader('Set-Cookie')
|
||||
if cookie: self.cookie = cookie.split(';', 1)[0]
|
||||
return response.status, json.loads(response.read() or b'{}')
|
||||
|
||||
def test_official_html_extracts_with_provenance_suppression_and_idempotency(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Acme', 'website': 'https://acme.test'})
|
||||
self.request('POST', '/api/v1/suppressions', {'kind': 'email', 'value': 'sales@acme.test'})
|
||||
payload = {'source_url': 'https://acme.test/contact', 'html': '<a href="mailto:sales@acme.test">Sales</a><p>info [at] acme [dot] test</p>', 'idempotency_key': 'extract-1'}
|
||||
status, result = self.request('POST', f"/api/v1/businesses/{business['id']}/contacts/extract", payload)
|
||||
self.assertEqual(status, 201); self.assertEqual(len(result['items']), 2)
|
||||
sales = next(x for x in result['items'] if x['value'] == 'sales@acme.test')
|
||||
self.assertTrue(sales['suppressed']); self.assertTrue(sales['do_not_contact']); self.assertEqual(sales['provenance'], 'mailto'); self.assertEqual(sales['mx_status'], 'unknown')
|
||||
self.assertEqual(self.request('POST', f"/api/v1/businesses/{business['id']}/contacts/extract", payload)[1]['idempotent'], True)
|
||||
self.assertEqual(self.request('GET', '/api/v1/contact-extractions?page_size=1')[1]['limit'], 1)
|
||||
|
||||
def test_arbitrary_and_oversized_sources_are_rejected(self):
|
||||
_, business = self.request('POST', '/api/v1/businesses', {'name': 'Acme', 'website': 'https://acme.test'})
|
||||
path = f"/api/v1/businesses/{business['id']}/contacts/extract"
|
||||
self.assertEqual(self.request('POST', path, {'source_url': 'https://evil.test', 'html': '<p>x@y.test</p>'})[1]['error'], 'source_not_approved')
|
||||
self.assertEqual(self.request('POST', path, {'source_url': 'https://acme.test', 'html': 'x' * (512 * 1024 + 1)})[0], 413)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,42 @@
|
||||
import unittest
|
||||
from app.contact_extractor import extract_contacts
|
||||
|
||||
|
||||
class ContactExtractorTests(unittest.TestCase):
|
||||
def test_extracts_public_mailto_obfuscated_phone_and_ignores_false_positives(self):
|
||||
html = '''<html><body>
|
||||
<a href="mailto:sales@acme.test">Email Sales</a>
|
||||
<span>info [at] acme.test</span><span>+27 (12) 345-6789</span>
|
||||
<a href="https://wa.me/27123456789">WhatsApp</a>
|
||||
<script>const token = 'abc@example.com';</script>
|
||||
<img src="https://cdn.thirdparty.test/x?email=bad@thirdparty.test">
|
||||
<span>john@example.com</span>
|
||||
</body></html>'''
|
||||
result = extract_contacts(html, 'https://acme.test/contact')
|
||||
values = {(x['kind'], x['value']) for x in result}
|
||||
self.assertIn(('email', 'sales@acme.test'), values)
|
||||
self.assertIn(('email', 'info@acme.test'), values)
|
||||
self.assertIn(('phone', '+27123456789'), values)
|
||||
self.assertIn(('whatsapp', '+27123456789'), values)
|
||||
self.assertNotIn(('email', 'abc@example.com'), values)
|
||||
self.assertNotIn(('email', 'bad@thirdparty.test'), values)
|
||||
self.assertNotIn(('email', 'john@example.com'), values)
|
||||
|
||||
def test_excludes_credentials_adjacent_to_email_like_values(self):
|
||||
result = extract_contacts('<p>API key: foo@acme.test password: bar@acme.test</p>', 'https://acme.test/')
|
||||
self.assertEqual(result, [])
|
||||
|
||||
def test_classifies_role_named_free_mail_and_suppression(self):
|
||||
html = '<p>support@acme.test alice@acme.test bob@gmail.com</p>'
|
||||
result = extract_contacts(html, 'https://acme.test/', suppressions=[{'kind':'email','value':'support@acme.test'}])
|
||||
by = {x['value']: x for x in result}
|
||||
self.assertEqual(by['support@acme.test']['classification'], 'role')
|
||||
self.assertTrue(by['support@acme.test']['do_not_contact'])
|
||||
self.assertTrue(by['support@acme.test']['suppressed'])
|
||||
self.assertEqual(by['alice@acme.test']['classification'], 'named')
|
||||
self.assertEqual(by['bob@gmail.com']['classification'], 'free_mail')
|
||||
self.assertEqual(by['alice@acme.test']['mx_status'], 'unknown')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -3,8 +3,12 @@ RUN addgroup -S app && adduser -S -G app app
|
||||
WORKDIR /srv
|
||||
COPY index.html /srv/index.html
|
||||
COPY styles.css /srv/styles.css
|
||||
COPY config.js /srv/config.js
|
||||
COPY asset-manifest.json /srv/asset-manifest.json
|
||||
COPY app.js /srv/app.js
|
||||
COPY healthz /srv/healthz
|
||||
COPY health.html /srv/health.html
|
||||
COPY error.html /srv/error.html
|
||||
RUN chown -R app:app /srv
|
||||
USER app
|
||||
EXPOSE 8080
|
||||
|
||||
+90
-7
@@ -1,17 +1,28 @@
|
||||
# ProspectOS web — Phase 8 boundary
|
||||
# ProspectOS web — Phase 9 boundary
|
||||
|
||||
Self-contained static frontend for the Prospect Platform API. There is no bundler or runtime dependency: serve this directory with any static HTTP server. The UI supports a manual, tenant-scoped review workflow, Phase 4 MVP job monitor, and Phase 8 scan-result/history presentation when supplied by the API. The browser does not fetch targets, submit forms, execute scan JavaScript, or send outreach; SSRF controls and budgets are server-side.
|
||||
|
||||
## Configure and run
|
||||
|
||||
The API base is configurable before `app.js` runs:
|
||||
The public runtime configuration is loaded from `config.js` before `app.js`. It contains no credentials and may safely be replaced during deployment:
|
||||
|
||||
```html
|
||||
<script>window.API_BASE = 'http://127.0.0.1:8000';</script>
|
||||
<script src="app.js"></script>
|
||||
```js
|
||||
window.__PROSPECT_CONFIG__ = Object.freeze({ apiBase: 'https://api.example.invalid', assetVersion: 'phase-15' });
|
||||
```
|
||||
|
||||
If not set, the UI uses `localStorage.prospect_api_base` when present; otherwise it targets the same origin. The dashboard requires an authenticated API session and shows the login screen until `/api/v1/auth/me` succeeds.
|
||||
If `apiBase` is empty, the UI uses `window.API_BASE`, then `localStorage.prospect_api_base` when present, and otherwise targets the same origin. The dashboard requires an authenticated API session and shows the login screen until `/api/v1/auth/me` succeeds. Do not put tokens, passwords, or private keys in `config.js`.
|
||||
|
||||
`asset-manifest.json` records the public entrypoints, cache-busting version, and SHA-256 digests for release verification. The HTML references the static assets with the `phase-15` version query string; update those references and regenerate the manifest when changing the release version.
|
||||
|
||||
## Deployment readiness checks
|
||||
|
||||
Serve this directory from the intended static-server root, then run:
|
||||
|
||||
```sh
|
||||
node scripts/smoke-deployment.mjs http://127.0.0.1:8080
|
||||
```
|
||||
|
||||
The smoke script checks HTTP delivery for the manifest-listed assets, health/error pages, `healthz`, expected markers, and common hardcoded-secret patterns. It validates static readiness only; it does not deploy the app or prove API/production availability.
|
||||
|
||||
## Phase 3 UI contract
|
||||
|
||||
@@ -66,6 +77,78 @@ The current client renders job status/counts, detail, structured errors, progres
|
||||
|
||||
A zero-dependency static smoke page (`smoke-test.html`) checks the key DOM contract in an iframe and reports pass/fail. Browser smoke coverage should treat pagination, detail child records, provenance rendering, pipeline/notes actions, and the disabled outreach state as separate checks.
|
||||
|
||||
## Phase 9 contact-observation UI contract
|
||||
|
||||
When the API supplies Phase 9 results, the UI may render contacts extracted from the approved/public official site and same-site contact/about pages. Display source URL/page context, extraction method, observed time, extractor/policy version, confidence and reasons, syntax status, role classification, free-mail classification, and independent MX/DNS status/freshness. Use explicit labels such as **Observation**, **Human review required**, and **Unknown**; never label a candidate verified, deliverable, owned, consented, or ready for outreach. The UI must show suppressed contacts as **Do not contact**, preserve the suppression reason, and never hide or override suppression through filters, refreshes, exports, or cached results.
|
||||
|
||||
Extraction is not a browser crawler. The browser must not fetch target pages directly, submit contact forms, execute target JavaScript, send credentials/cookies, probe SMTP, send validation messages, or expose an outreach/send control. Render bounded/partial/blocked/timeout/error results distinctly from an empty successful result, including candidate/page/byte/time limit reasons. Do not display false-positive candidates from assets, scripts/styles, example/test/placeholder values, tracking addresses, malformed schemes, or unrelated third-party pages.
|
||||
|
||||
The API remains authoritative for official-site scope, tenant isolation, suppression enforcement, limits, retention, provenance, and permissions. A cached extraction must show its observed time and freshness, never “live.” Candidate confidence, role/free-mail labels, syntax, and MX/DNS uncertainty are review metadata only and cannot enable a contact action.
|
||||
|
||||
## Phase 10 scoring UI contract
|
||||
|
||||
The UI may display server-provided `score`, `priority_band`, and `eligibility` as separate fields. It should show the active rule-set ID/version, calculated time/freshness, explanation factors with their points/weights, exclusions, and stale/uncertain reasons. A band is a triage label, not permission to contact; never derive or override these values solely in browser code.
|
||||
|
||||
Render eligibility independently and prominently: **Eligible**, **Ineligible**, or **Unknown/review required** must not be collapsed into a score band. Show suppression/do-not-contact as a hard, persistent state that wins over score, verification, pipeline, cached data, or refresh. Keep stale, expired, blocked, partial, and uncertain evidence visibly distinct from missing evidence and never present them as a positive or negative fact. There is no outreach control in this phase.
|
||||
|
||||
If the API exposes recalculation, the UI must show the requested rule-set/version, job/progress/partial state, actor/time, and before/after explanation or band changes; acceptance of a request is not completion. Recalculation history and audit details remain tenant-scoped server capabilities. Rule-set administration, activation, rollback, and eligibility policy are not client-side authorization controls.
|
||||
|
||||
Phase 10 remains a pilot display contract until the API supplies stable versioned rule metadata, reproducible input lineage, complete explanation payloads, explicit eligibility reasons, and audited recalculation results. Browser smoke coverage should include score/band disagreement with eligibility, suppression precedence, stale/uncertain rendering, version changes, partial recalculation, and cross-tenant non-disclosure.
|
||||
|
||||
## Phase 11 dashboard and review workflow UI contract
|
||||
|
||||
Phase 11 is the dashboard/review presentation contract for **saved filters**, a **review queue**, and bounded bulk actions. Saved filters must show a named, human-readable summary of the exact search/status/score/pipeline/eligibility predicate, sort, and page-size settings. Save/load/update/delete controls must reflect API authorization and tenant scope; the browser must not treat a filter ID, hidden field, or local-only copy as permission. If filters are shared, the UI must show that they are tenant-scoped and read-only or editable as returned by the API.
|
||||
|
||||
The review queue must make its scope visible: current filter name or predicate, matching-set versus current-page count, ordering, pagination/cursor state, and `has_more`. A row can be selected only from the current tenant-scoped result. Suppressed/do-not-contact state must remain prominent and disable contact-related actions; merged/non-active records are not merge-eligible. Merge suggestions remain **Human review required** and need an explicit confirmation dialog; no score or checkbox may auto-merge.
|
||||
|
||||
Bulk actions must show the bounded selection size and server maximum, provide a preview before confirmation, and report per-record success/skipped/failed outcomes. The UI must refresh or reconcile stale rows after execution, preserve suppression and eligibility reasons, and never describe a partial result as complete. A clickable count must navigate using the exact predicate that produced the count; distinguish full matching-set counts from page counts, and show loading/error/unavailable rather than zero. Count cards are navigation affordances, not authorization controls.
|
||||
|
||||
The UI must expose audit context for saved-filter changes, queue decisions, bulk preview/confirmation/execution, suppression/eligibility decisions, and merge/reversal: actor, time, bounded selection/filter snapshot, result totals, and safe reason/version metadata. It must not render secrets or unnecessary contact data. The API remains authoritative for tenant isolation, permissions, re-checks, idempotency, suppression precedence, and audit persistence.
|
||||
|
||||
The current Phase 11 client now renders saved-view controls, a review queue capped at 100 visible records, selectable rows, and explicit verify/reject bulk review actions. It also renders clickable dashboard metric cards. Current limitations are material: saved views can be created/loaded/deleted in the client but update is not exposed; queue selection is visible-row-only and the UI does not show a server maximum/preview/per-record outcomes; dashboard links use client filter shortcuts rather than a complete server predicate; and suppression/merge eligibility and audit results still depend on the API response. No bulk action sends outreach or auto-merges.
|
||||
|
||||
## Phase 12 CRM UI contract
|
||||
|
||||
The Phase 12 UI presents a tenant-scoped pipeline, append-only interaction timeline, normalized outcomes, bounded reporting, and a suppression center. It must show the exact tenant/filter/as-of/timezone scope of every view and distinguish page counts, matching-set counts, event counts, and distinct-business counts. Loading, stale, unavailable, and error states are not zero. The API is authoritative; a hidden field, report ID, saved filter, or visible row cannot grant access.
|
||||
|
||||
Pipeline controls display the configured stages (`new`, `contacted`, `qualified`, `proposal`, `negotiation`, `won`, `lost`) and require an explicit reason for `won`, `lost`, and any configured reopen action. The UI must not offer direct jumps, edit historical transitions, or advance a stage merely because an interaction was added. Interactions show channel, actor, occurred time, provenance, safe summary, and outcome. Corrections are visibly appended/superseding, not destructive edits. `other` is distinct from a success or failure claim.
|
||||
|
||||
The outcome vocabulary is `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, and `other`. `other` is explicit uncertainty/catch-all metadata, not proof of success or failure; the UI must not invent an outcome for missing data. `do_not_contact` is a separate persistent suppression state, not a deliverability or engagement outcome, and must disable contact-related controls.
|
||||
|
||||
The suppression center shows normalized identifier, source, reason, scope, actor, effective time, and audit context. It must apply to records before display/export/report eligibility and must never silently delete a suppressed record. Unsuppression/removal is an explicit authorized action with confirmation and reason. Report and export screens must show freshness, as-of, timezone, filter snapshot, retention class where applicable, and safe partial/per-record results; they must not imply deliverability or outreach permission.
|
||||
|
||||
There is no send button, message composer, SMTP probe, validation email, campaign, delivery scheduler, or automated follow-up in Phase 12. The browser never contacts a prospect. Suppression, pipeline, outcome, report, and audit controls are presentation layers over server enforcement. The UI remains pilot-grade until browser/API smoke coverage verifies transition rejection, append-only corrections, outcome taxonomy, suppression precedence, report semantics, retention states, and cross-tenant non-disclosure.
|
||||
|
||||
## Phase 14 draft-only outreach UI contract
|
||||
|
||||
Phase 14 adds preparation language only. The browser may display a server-provided outreach draft and its gate status, but it must never call a provider, send a message, schedule delivery, probe SMTP, send validation mail, create a campaign, or imply that draft creation or approval is delivery. Render a persistent **Draft only — human approval required** state and keep `AUTOMATED_OUTREACH_ENABLED=false` visible as the no-send default.
|
||||
|
||||
A draft view must show tenant scope, recipient/channel, provider ID/version when configured, consent/legal-basis status and jurisdiction/policy version, suppression/do-not-contact status, evidence citations/source references, exact evidence snapshot hash, observed/freshness times, uncertainty/conflict reasons, rate/cap status, approval actor/time/reason/expiry, and a safe content fingerprint or bounded redacted preview. Never display provider secrets, raw prompts, credentials, unnecessary personal data, or unsupported claims. Public availability, score, pipeline state, verification, and AI confidence are not consent, lawful basis, deliverability, or permission to contact.
|
||||
|
||||
Approval controls must be absent or disabled unless the API reports all gates passed and the authenticated user is authorized. Approval must be an explicit confirmation of the exact draft version and evidence hash, with a reason where required; edits, changed evidence/policy, stale data, suppression, expired approval, provider failure, or uncertain legal status must invalidate it and require re-review. Rejection and expiry must remain visible. Approval never creates a send control.
|
||||
|
||||
If a future side-effecting API is exposed, the browser must send a tenant-scoped idempotency key and show the original bounded result on exact replay, while presenting conflicting-key, cap, suppression, provider, and gate failures distinctly from success. Display audit context for draft creation, gate decisions, citations, approval/rejection/expiry, retries, cap denials, and any delivery result, with redaction. Never turn a count or visible row into authorization.
|
||||
|
||||
Phase 14 is not implemented as a live outreach workflow. The current static client has no draft composer, consent ledger, provider integration, approval API, send button, delivery status, bounce/complaint handling, or legal-policy engine. Production work requires API-backed draft/version persistence, jurisdiction-specific legal review, provider/DPA and secret-management controls, server-side gates, durable approvals/audit/idempotency, suppression re-checks, rate/cost caps, kill switch, retention/deletion/legal-hold behavior, and browser/API tests proving no outbound network activity.
|
||||
|
||||
## Remaining limitations
|
||||
|
||||
The static client has no client-side crawler, scanner, enrichment scheduler, outreach integration, availability provider, or SSE delivery. It can display server-provided Phase 8 observations, but production still requires server-side SSRF/DNS-rebinding/redirect controls, hard size/time/crawl budgets, durable scan history/cache isolation and retention, abuse/rate controls, and authenticated permission/audit coverage. For domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability. CSV preview is capped for display and is not an import workflow.
|
||||
The static client has no client-side crawler, scanner, contact extractor, enrichment scheduler, outreach integration, availability provider, or SSE delivery. It can display server-provided Phase 9 observations, but production still requires server-side official-site scoping, SSRF/DNS-rebinding/redirect controls, hard extraction/page/byte/time/candidate budgets, durable history/cache isolation and retention/deletion, abuse/rate controls, suppression regression tests, and authenticated provenance/audit coverage. For domain intelligence, display registrable-domain/PSL version and unresolved reasons, DNS status and freshness/TTL (not “available”), independent MX/NS/TXT uncertainty, and association confidence with explainable evidence. Never auto-attach candidate domains or treat `nxdomain`/`no_data` as availability. CSV preview is capped for display and is not an import workflow.
|
||||
|
||||
## Phase 13 optional AI assistance UI contract
|
||||
|
||||
The UI may offer AI drafting only when the authenticated API reports an approved, enabled capability for the current tenant and task. Provider choice, fallback, prompt construction, redaction, budgets, tenant authorization, and suppression checks are server-side; the browser must never receive provider secrets or call an AI vendor directly. Show provider/model/version and `unknown`/`unavailable`, timeout, partial, stale, or policy-blocked states distinctly from an empty or successful result.
|
||||
|
||||
Every suggestion must display its evidence citations, tenant-scoped evidence IDs, exact evidence hash/snapshot identifier, observed time, uncertainty/conflict reasons, and policy/provider/model versions. A citation points to the evidence used; it is not proof that the source is correct, and an AI explanation is not an independently verified fact. Do not render unsupported or invented facts about names, roles, contact details, dates, outcomes, consent, deliverability, ownership, or other claims as facts. Preserve missing and conflicting evidence instead of filling gaps. Suppressed/do-not-contact records remain visible with the safety state and never become actionable because an AI suggestion is confident.
|
||||
|
||||
AI output must be visibly labeled **AI suggestion — human review required** and remain read-only until an authorized human explicitly approves it. Approval must show the proposed change, citations/hash, freshness, tenant scope, and safe reason; rejection and expiry must be available. The UI must require re-review when the evidence hash or policy version changes and must display partial/failed approval rather than implying persistence. Approval does not authorize contact or verification.
|
||||
|
||||
No Phase 13 control may send email/SMS, probe SMTP, create a campaign, schedule follow-up, alter pipeline/interactions/outcomes as if communication occurred, merge records, acquire a domain, or perform autonomous CRM/outreach actions. The browser must not hide or export suppressed data as eligible, and exports/reports must retain safe AI provenance and redaction labels where applicable. Production remains limited until browser/API tests cover citations and hash mismatch, redaction, fallback boundaries, approval/rejection, stale/conflicting evidence, suppression precedence, tenant non-disclosure, and no-autonomy controls; the current Compose stack has no configured AI provider.
|
||||
|
||||
## Phase 15 deployment and readiness
|
||||
|
||||
The web image is a portable static server: it runs as a non-root user, serves only the files copied into `/srv`, and exposes `/healthz`. Virtualmin is responsible for DNS, HTTPS certificates, reverse-proxy routing, firewall rules, and any access control around the site. Set the API base deliberately for the deployed origin; do not put credentials or provider secrets in HTML, JavaScript, local storage, image layers, or `.env` files. `CORS_ORIGINS` must exactly match the approved HTTPS origin rather than a broad wildcard.
|
||||
|
||||
`/healthz` is an unauthenticated process/liveness check. API `/api/v1/health/ready` checks SQLite readiness, but neither endpoint proves tenant authorization, backup validity, or external dependencies. Route traffic only after the web and API containers report `healthy`, the HTTPS proxy reaches the intended containers, and an authenticated browser/API smoke test succeeds. The browser must never be used to test or initiate outbound prospect/provider traffic; `AUTOMATED_OUTREACH_ENABLED=false` remains visible as the no-send default.
|
||||
|
||||
For releases, validate the exact static image and API image together, capture image digests and configuration revision, and retain the prior pair for rollback. If a schema/data migration is involved, the API owner must complete backup/restore and migration validation before the web image is promoted. The current client has no service-worker cache or migration logic; stale browser tabs must be refreshed after a release, and Virtualmin/CDN caching must not serve an old API contract indefinitely. SQLite, HTTP-only local Compose, lack of a readiness endpoint, and lack of a production asset/CDN pipeline are explicit limitations.
|
||||
|
||||
+121
-11
File diff suppressed because one or more lines are too long
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"version": "phase-15",
|
||||
"entrypoints": ["config.js", "app.js", "styles.css"],
|
||||
"publicAssets": ["index.html", "health.html", "error.html", "healthz"],
|
||||
"integrity": {
|
||||
"config.js": "sha256-20f3020432436dcccdbfc86fd56a6a6a49b71fc512a1e434187a5ddc134fda1c",
|
||||
"app.js": "sha256-fc12f49012bb1329ffdd7bdcf655e9ab9097eaf1e0cc73cd0442b19fd57a0374",
|
||||
"styles.css": "sha256-7340dccc648fa917fb497ceeba7286d9c4712b6552960de54cef759cdbda6de4",
|
||||
"index.html": "sha256-37a9d5e2a81941c3c9f9bd3f42edf33a40565bf197cf675389d250dd60eb69db",
|
||||
"health.html": "sha256-c352a6f37aa24628cfc8d5709a70ff2d94181d192ed5fe00916cca9378a61d81",
|
||||
"error.html": "sha256-f3cc28d2dfc9e8af112d0257c6ee47b9e9146e5da8590a754dd589bf43702aaf",
|
||||
"healthz": "sha256-dc51b8c96c2d745df3bd5590d990230a482fd247123599548e0632fdbf97fc22"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
/* Public, non-secret runtime configuration. Replace this file at deploy time if needed. */
|
||||
window.__PROSPECT_CONFIG__ = Object.freeze({
|
||||
apiBase: '',
|
||||
assetVersion: 'phase-15'
|
||||
});
|
||||
@@ -0,0 +1,2 @@
|
||||
<!doctype html>
|
||||
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>ProspectOS · Temporarily unavailable</title><style>body{font:16px system-ui,sans-serif;margin:4rem auto;max-width:42rem;padding:0 1.5rem;color:#172033;background:#f7f8fb}main{background:#fff;border:1px solid #e7eaf1;border-radius:12px;padding:2rem}h1{color:#b84d55}a{color:#6756e8}</style></head><body><main><p>ProspectOS frontend</p><h1>Something went wrong</h1><p>This page could not be loaded. Please try again or return to the <a href="/">workspace</a>.</p></main></body></html>
|
||||
@@ -0,0 +1,2 @@
|
||||
<!doctype html>
|
||||
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>ProspectOS · Healthy</title><style>body{font:16px system-ui,sans-serif;margin:4rem auto;max-width:42rem;padding:0 1.5rem;color:#172033;background:#f7f8fb}main{background:#fff;border:1px solid #e7eaf1;border-radius:12px;padding:2rem}h1{color:#16845b}code{background:#e5f7ef;padding:.15rem .35rem;border-radius:4px}</style></head><body><main><p>ProspectOS frontend</p><h1>Ready</h1><p>Static asset delivery is available. API availability is checked separately by the application.</p><p><code>health.html</code></p></main></body></html>
|
||||
+38
-7
@@ -5,7 +5,7 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>ProspectOS · Pipeline intelligence</title>
|
||||
<meta name="description" content="Prospect discovery and review dashboard">
|
||||
<link rel="stylesheet" href="styles.css">
|
||||
<link rel="stylesheet" href="styles.css?v=phase-15">
|
||||
</head>
|
||||
<body>
|
||||
<section class="login-screen" id="loginScreen" aria-labelledby="loginTitle">
|
||||
@@ -32,6 +32,11 @@
|
||||
<a class="nav-item" href="#add"><span>+</span> Add prospects</a>
|
||||
<a class="nav-item" href="#jobs" data-nav="jobs"><span>◷</span> Jobs</a>
|
||||
<a class="nav-item" href="#sources" data-nav="sources"><span>⌁</span> Sources</a>
|
||||
<a class="nav-item" href="#crmPipeline" data-nav="crm"><span>◫</span> CRM pipeline</a>
|
||||
<a class="nav-item" href="#crmReports" data-nav="reports"><span>▤</span> Reports</a>
|
||||
<a class="nav-item" href="#suppressionCenter" data-nav="suppression"><span>⊘</span> Suppressions</a>
|
||||
<a class="nav-item" href="#outreachSettings" data-nav="outreach-settings"><span>⚙</span> Outreach policy</a>
|
||||
<a class="nav-item" href="#scoreRules" data-nav="score-rules"><span>◈</span> Score rules</a>
|
||||
</nav>
|
||||
<div class="sidebar-foot"><div class="live-dot"></div><div><strong>Workspace live</strong><small>Data sync is healthy</small></div></div>
|
||||
</aside>
|
||||
@@ -40,16 +45,21 @@
|
||||
<div class="content">
|
||||
<section class="hero" id="dashboard"><div><p class="eyebrow">EVIDENCE-LED PROSPECTING</p><h1>Good morning, Alex <span>✦</span></h1><p class="hero-sub">Your pipeline has <strong id="heroCount">0 prospects</strong> ready for review.</p></div><button class="button primary" data-scroll="#add">+ Add prospect</button></section>
|
||||
<section class="metrics" aria-label="Dashboard metrics">
|
||||
<article class="metric-card"><div class="metric-icon violet">◎</div><div><p>Total prospects</p><h2 id="metricTotal">0</h2><span class="trend neutral">● Current workspace</span></div></article>
|
||||
<article class="metric-card"><div class="metric-icon amber">◌</div><div><p>Needs review</p><h2 id="metricReview">0</h2><span class="trend neutral">● Human verification</span></div></article>
|
||||
<article class="metric-card"><div class="metric-icon green">◉</div><div><p>High-fit prospects</p><h2 id="metricHigh">0</h2><span class="trend neutral">● Score 80+</span></div></article>
|
||||
<article class="metric-card"><div class="metric-icon blue">◷</div><div><p>Freshness under 7d</p><h2 id="metricFresh">0%</h2><span class="trend neutral">● Evidence coverage</span></div></article>
|
||||
<a class="metric-card metric-link" data-dashboard-filter="all" href="#explorer"><div class="metric-icon violet">◎</div><div><p>Total prospects</p><h2 id="metricTotal">0</h2><span class="trend neutral">● Current workspace</span></div></a>
|
||||
<a class="metric-card metric-link" data-dashboard-filter="review" href="#explorer"><div class="metric-icon amber">◌</div><div><p>Needs review</p><h2 id="metricReview">0</h2><span class="trend neutral">● Human verification</span></div></a>
|
||||
<a class="metric-card metric-link" data-dashboard-filter="high" href="#explorer"><div class="metric-icon green">◉</div><div><p>High-fit prospects</p><h2 id="metricHigh">0</h2><span class="trend neutral">● Score 80+</span></div></a>
|
||||
<a class="metric-card metric-link" data-dashboard-filter="fresh" href="#explorer"><div class="metric-icon blue">◷</div><div><p>Freshness under 7d</p><h2 id="metricFresh">0%</h2><span class="trend neutral">● Evidence coverage</span></div></a>
|
||||
<a class="metric-card metric-link" data-dashboard-filter="suppressed" href="#explorer"><div class="metric-icon violet">⊘</div><div><p>Suppressed</p><h2 id="metricSuppressed">0</h2><span class="trend neutral">● Do not contact</span></div></a>
|
||||
</section>
|
||||
<section class="score-overview" aria-label="Score overview"><article class="panel score-distribution-panel" id="scoreDistributionPanel" data-smoke="score-distribution"><div class="detail-loading" aria-live="polite">Sign in to load score distribution…</div></article></section>
|
||||
<section class="score-rules-section" id="scoreRules" aria-labelledby="scoreRulesTitle"><article class="panel" id="scoreRulesPanel" data-smoke="score-rules"><div class="detail-loading" aria-live="polite">Sign in to load score rules…</div></article></section>
|
||||
<section class="workspace-grid" id="explorer">
|
||||
<div class="explorer-panel panel"><div class="panel-heading"><div><p class="eyebrow">PIPELINE</p><h2>Prospect explorer</h2></div><button class="button ghost" id="refreshBtn">↻ Refresh</button></div>
|
||||
<div class="filters"><label class="search-wrap"><span>⌕</span><input id="searchInput" type="search" placeholder="Search companies, domains, locations…" autocomplete="off"></label><select id="scoreFilter" aria-label="Filter by score"><option value="all">All scores</option><option value="high">High fit · 80+</option><option value="medium">Medium · 60–79</option><option value="low">Low · under 60</option></select><select id="statusFilter" aria-label="Filter by status"><option value="all">All statuses</option><option value="review">Needs review</option><option value="reviewed">Reviewed</option><option value="suppressed">Suppressed</option></select><select id="websiteClassFilter" aria-label="Filter by website class"><option value="all">All website classes</option><option value="business_site">Business site</option><option value="social_profile">Social profile</option><option value="missing">Missing website</option></select><select id="pipelineFilter" aria-label="Filter by pipeline stage"><option value="all">All pipeline stages</option><option value="new">New</option><option value="qualified">Qualified</option><option value="review">Review</option><option value="suppressed">Suppressed</option></select></div>
|
||||
<div class="saved-view-controls" data-smoke="saved-views"><form id="savedFilterForm" class="inline-form"><input id="savedFilterName" name="name" placeholder="Save current filters as…" maxlength="80" required><button class="button ghost compact" type="submit">Save view</button></form><select id="savedFilterSelect" aria-label="Load saved view"><option value="">Saved views</option></select><button class="button ghost compact" id="deleteSavedFilterBtn" type="button" disabled>Delete view</button><p id="savedFilterMessage" class="form-message" role="status" aria-live="polite"></p></div>
|
||||
<section class="review-queue" data-smoke="review-queue"><div class="queue-heading"><div><p class="eyebrow">OPERATOR QUEUE</p><h3>Review queue <span class="count" id="reviewQueueCount">0</span></h3></div><span class="small-label">Human decision required</span></div><div id="reviewQueueState" class="queue-state" aria-live="polite">Loading review queue…</div><div class="bulk-actions"><label class="checkbox-label"><input id="selectAllReview" type="checkbox"> Select visible</label><span id="selectedReviewCount" class="small-label">0 selected</span><button class="button primary compact" id="bulkVerifyBtn" type="button" disabled>Verify selected</button><button class="button danger compact" id="bulkRejectBtn" type="button" disabled>Reject selected</button></div></section>
|
||||
<div class="table-meta"><span id="resultCount">Showing 0 prospects</span><span class="legend"><span class="legend-dot high-dot"></span> High fit <span class="legend-dot review-dot"></span> Needs review</span><label class="page-size">Rows <select id="pageSize" aria-label="Page size"><option>10</option><option>25</option><option>50</option></select></label><button class="button ghost compact" id="nextPageBtn" type="button">Next page →</button></div>
|
||||
<div class="table-scroll"><table><thead><tr><th>Company</th><th>Fit score</th><th>Evidence</th><th>Freshness</th><th>Status</th><th></th></tr></thead><tbody id="prospectRows"></tbody></table></div>
|
||||
<div id="explorerState" class="explorer-state" aria-live="polite"></div><div class="table-scroll"><table><thead><tr><th>Company</th><th>Fit score</th><th>Evidence</th><th>Freshness</th><th>Status</th><th></th></tr></thead><tbody id="prospectRows"></tbody></table></div>
|
||||
</div>
|
||||
<aside class="detail-panel panel" id="detailPanel"><div class="empty-detail"><span class="empty-icon">◒</span><h3>Select a prospect</h3><p>Review evidence, confidence, and eligibility before taking action.</p></div></aside>
|
||||
</section>
|
||||
@@ -80,6 +90,26 @@
|
||||
</section>
|
||||
<section class="lower-grid" id="add"><article class="panel add-panel"><div class="panel-heading"><div><p class="eyebrow">INTAKE</p><h2>Add a prospect</h2></div><span class="small-label">Manual entry</span></div><form id="addForm"><div class="form-grid"><label>Company name<input required name="name" placeholder="Acme Inc."></label><label>Website <span class="optional">optional</span><input name="website" type="url" placeholder="https://acme.com"></label><label>Location<input name="location" placeholder="Cape Town, ZA"></label><label>Notes <span class="optional">optional</span><input name="description" placeholder="Why this is a fit…"></label></div><div class="form-footer"><p id="formMessage" class="form-message" role="status"></p><button class="button primary" type="submit">Add to review queue</button></div></form></article>
|
||||
<article class="panel csv-panel"><div class="panel-heading"><div><p class="eyebrow">BULK INTAKE</p><h2>CSV preview</h2></div><label class="button ghost upload-label" for="csvInput">↑ Choose CSV</label><input id="csvInput" type="file" accept=".csv,text/csv" hidden></div><p class="muted">Preview rows before adding them to your review queue.</p><div id="csvPreview" class="csv-empty"><span>⊞</span><p>No file selected</p><small>CSV stays in your browser until you confirm.</small></div></article></section>
|
||||
<section class="crm-section" id="crmPipeline" aria-labelledby="crmPipelineTitle" data-smoke="crm-pipeline">
|
||||
<div class="crm-header panel"><div><p class="eyebrow">CRM WORKFLOW</p><h2 id="crmPipelineTitle">Pipeline board</h2><p class="muted">Move prospects through explicit human stages. Stage changes never send outreach.</p></div><div class="crm-actions"><button class="button ghost" id="crmRefreshBtn" type="button">↻ Refresh CRM</button><button class="button ghost" id="pipelineViewToggle" type="button" aria-pressed="false">☷ List view</button></div></div>
|
||||
<div class="crm-safety" role="note"><strong>No outreach from this workspace.</strong> Pipeline actions, interactions, and follow-ups are internal CRM records only. Suppressed or unreviewed prospects remain unavailable for contact.</div>
|
||||
<p id="crmMessage" class="crm-message" role="status" aria-live="polite"></p>
|
||||
<div id="pipelineBoard" class="pipeline-board" aria-live="polite"><div class="detail-loading">Sign in to load the pipeline.</div></div>
|
||||
</section>
|
||||
<section class="crm-section" id="crmActivity" aria-labelledby="crmActivityTitle" data-smoke="crm-interactions">
|
||||
<div class="crm-header panel"><div><p class="eyebrow">RELATIONSHIP HISTORY</p><h2 id="crmActivityTitle">Interactions & follow-ups</h2><p class="muted">Capture outcomes and next steps without contacting anyone.</p></div></div>
|
||||
<div class="crm-two-col"><article class="panel interaction-panel"><div id="interactionState" class="detail-loading">Select a prospect to load interactions.</div></article><article class="panel"><div class="panel-heading"><div><p class="eyebrow">ADD RECORD</p><h3>Log an interaction</h3></div><span class="small-label">Internal only</span></div><form id="interactionForm" class="crm-form"><label>Type<select name="type"><option value="note">Note</option><option value="call">Call</option><option value="meeting">Meeting</option><option value="email">Email (record only)</option></select></label><label>Outcome<select name="outcome"><option value="">Choose outcome</option><option value="no_response">No response</option><option value="interested">Interested</option><option value="not_a_fit">Not a fit</option><option value="follow_up">Follow-up requested</option></select></label><label>Follow-up date <span class="optional">optional</span><input name="follow_up_at" type="date"></label><label>Summary<textarea name="summary" rows="4" required placeholder="What happened? Keep this an internal record."></textarea></label><p id="interactionMessage" class="form-message" role="status"></p><button class="button primary" type="submit">Save interaction</button></form></article></div>
|
||||
</section>
|
||||
<section class="crm-section" id="crmReports" aria-labelledby="crmReportsTitle" data-smoke="crm-reports">
|
||||
<div class="crm-header panel"><div><p class="eyebrow">REPORTING</p><h2 id="crmReportsTitle">CRM reports</h2><p class="muted">Tenant-scoped pipeline, outcomes, and activity summaries from the API.</p></div><button class="button ghost" id="reportsRefreshBtn" type="button">↻ Refresh reports</button></div>
|
||||
<div class="reports-grid"><article class="panel report-panel" id="pipelineReport"><div class="detail-loading">Loading pipeline report…</div></article><article class="panel report-panel" id="outcomesReport"><div class="detail-loading">Loading outcomes report…</div></article><article class="panel report-panel" id="activityReport"><div class="detail-loading">Loading activity report…</div></article></div>
|
||||
</section>
|
||||
<section class="crm-section" id="suppressionCenter" aria-labelledby="suppressionTitle" data-smoke="suppression-center">
|
||||
<div class="crm-header panel"><div><p class="eyebrow">SAFETY CENTER</p><h2 id="suppressionTitle">Suppression center</h2><p class="muted">Review and maintain tenant-scoped do-not-contact rules.</p></div><button class="button ghost" id="suppressionRefreshBtn" type="button">↻ Refresh suppressions</button></div>
|
||||
<div class="suppression-warning" role="alert"><strong>Suppression always wins.</strong> Suppressed contacts and domains cannot be contacted, regardless of score, stage, or interaction outcome.</div>
|
||||
<div class="crm-two-col"><article class="panel"><div class="panel-heading"><div><p class="eyebrow">ADD RULE</p><h3>Add suppression</h3></div><span class="small-label">Explicit confirmation required</span></div><form id="suppressionForm" class="crm-form"><label>Kind<select name="kind"><option value="email">Email</option><option value="domain">Domain</option><option value="phone">Phone</option></select></label><label>Value<input name="value" required placeholder="person@example.com"></label><label>Reason <span class="optional">optional</span><input name="reason" placeholder="Customer request / policy"></label><p id="suppressionMessage" class="form-message" role="status"></p><button class="button danger" type="submit">Add suppression</button></form></article><article class="panel"><div class="panel-heading"><div><p class="eyebrow">REGISTER</p><h3>Current suppressions</h3></div><div class="suppression-bulk-actions"><label class="checkbox-label"><input id="selectAllSuppressions" type="checkbox"> Select all</label><button class="button ghost compact" id="bulkReviewSuppressionsBtn" type="button" disabled>Review selected</button></div></div><div id="suppressionState" class="detail-loading">Sign in to load suppressions.</div></article></div>
|
||||
</section>
|
||||
<section class="crm-section outreach-settings-section" id="outreachSettings" aria-labelledby="outreachSettingsTitle" data-smoke="outreach-provider-policy"><div class="crm-header panel"><div><p class="eyebrow">GOVERNANCE</p><h2 id="outreachSettingsTitle">Outreach provider policy</h2><p class="muted">View approved provider status without exposing credentials or secrets.</p></div><button class="button ghost" id="outreachPolicyRefreshBtn" type="button">↻ Refresh policy</button></div><div class="outreach-settings-safety" role="note"><strong>Sending is disabled by default.</strong> This panel is status-only. Provider configuration never creates a send trigger, and no credentials are displayed.</div><div id="providerPolicyPanel" class="provider-policy-panel" aria-live="polite"><div class="detail-loading">Sign in to load provider policy.</div></div></section>
|
||||
<footer><span>ProspectOS</span><span>Evidence-led prospecting · <a href="#explorer">Review queue</a></span></footer>
|
||||
</div>
|
||||
</main>
|
||||
@@ -87,6 +117,7 @@
|
||||
<div class="merge-dialog" id="mergeDialog" hidden role="dialog" aria-modal="true" aria-labelledby="mergeDialogTitle">
|
||||
<div class="merge-dialog-card"><div class="panel-heading"><div><p class="eyebrow">REVIEW REQUIRED</p><h2 id="mergeDialogTitle">Confirm merge</h2></div><button class="icon-button" id="cancelMergeBtn" type="button" aria-label="Close merge confirmation">×</button></div><p id="mergeDialogCopy"></p><div class="merge-warning"><strong>This action is reversible.</strong> The merge will be recorded in history and can be reversed later.</div><p id="mergeDialogMessage" class="form-message" role="alert" aria-live="polite"></p><div class="merge-dialog-actions"><button class="button ghost" id="cancelMergeBtnSecondary" type="button">Cancel</button><button class="button primary" id="confirmMergeBtn" type="button">Confirm merge</button></div></div>
|
||||
</div>
|
||||
<script src="app.js"></script>
|
||||
<script src="config.js?v=phase-15"></script>
|
||||
<script src="app.js?v=phase-15"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Deterministic, dependency-free frontend final acceptance harness.
|
||||
*
|
||||
* Source checks run by default. HTTP checks are opt-in:
|
||||
* node scripts/final-acceptance.mjs --base-url http://127.0.0.1:8080
|
||||
* node scripts/final-acceptance.mjs --start-server [--port 0]
|
||||
*
|
||||
* stdout is one machine-readable JSON document. Exit codes:
|
||||
* 0 = all requested checks passed
|
||||
* 1 = one or more acceptance checks failed
|
||||
* 2 = invalid CLI/configuration or unreadable source
|
||||
*/
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFile, stat } from 'node:fs/promises';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const webRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
const args = process.argv.slice(2);
|
||||
const valueFor = flag => { const i = args.indexOf(flag); return i >= 0 ? args[i + 1] : undefined; };
|
||||
const startServer = args.includes('--start-server');
|
||||
const baseArg = valueFor('--base-url') || valueFor('--url');
|
||||
const portArg = valueFor('--port') || '8080';
|
||||
if (args.includes('--help') || args.includes('-h')) {
|
||||
console.log(JSON.stringify({ usage: 'node scripts/final-acceptance.mjs [--base-url URL | --start-server] [--port N]', exit_codes: { pass: 0, failed_checks: 1, usage_or_io_error: 2 } }, null, 2));
|
||||
process.exit(0);
|
||||
}
|
||||
if (baseArg && startServer) {
|
||||
console.log(JSON.stringify({ schema: 1, harness: 'prospectos-frontend-final-acceptance', pass: false, error: '--base-url and --start-server are mutually exclusive' }, null, 2));
|
||||
process.exit(2);
|
||||
}
|
||||
const baseUrl = baseArg?.replace(/\/$/, '') || null;
|
||||
const sourceNames = ['index.html', 'app.js', 'styles.css', 'config.js', 'health.html', 'error.html', 'healthz', 'smoke-test.html', 'asset-manifest.json', 'README.md'];
|
||||
const results = [];
|
||||
const check = (id, description, pass, details = '') => results.push({ id, description, pass: Boolean(pass), ...(details ? { details } : {}) });
|
||||
const all = (items, predicate) => items.every(predicate);
|
||||
const listMissing = (items, predicate) => items.filter(item => !predicate(item));
|
||||
const sha256 = bytes => `sha256-${createHash('sha256').update(bytes).digest('hex')}`;
|
||||
let files = {};
|
||||
let manifest;
|
||||
try {
|
||||
files = Object.fromEntries(await Promise.all(sourceNames.map(async name => [name, await readFile(resolve(webRoot, name))])));
|
||||
manifest = JSON.parse(files['asset-manifest.json'].toString('utf8'));
|
||||
check('files.readable', 'all frontend acceptance inputs are readable', true);
|
||||
} catch (error) {
|
||||
check('files.readable', 'all frontend acceptance inputs are readable', false, error.message);
|
||||
}
|
||||
const text = name => files[name]?.toString('utf8') || '';
|
||||
const html = text('index.html');
|
||||
const js = text('app.js');
|
||||
const css = text('styles.css');
|
||||
const config = text('config.js');
|
||||
|
||||
const expectedIds = [
|
||||
'loginScreen', 'loginForm', 'loginEmail', 'loginPassword', 'dashboardShell', 'logoutBtn', 'apiStatus',
|
||||
'explorer', 'detailPanel', 'reviewQueueCount', 'reviewQueueState', 'savedFilterForm', 'savedFilterSelect',
|
||||
'deleteSavedFilterBtn', 'bulkVerifyBtn', 'bulkRejectBtn', 'nextPageBtn', 'jobs', 'jobsList', 'jobDetailPanel',
|
||||
'sources', 'sourcesList', 'sourceRecordsList', 'discoveryForm', 'crmPipeline', 'pipelineBoard', 'crmActivity',
|
||||
'interactionState', 'interactionForm', 'crmReports', 'pipelineReport', 'outcomesReport', 'activityReport',
|
||||
'suppressionCenter', 'suppressionForm', 'suppressionState', 'outreachSettings', 'providerPolicyPanel',
|
||||
'scoreRules', 'scoreRulesPanel', 'scoreDistributionPanel', 'mergeDialog', 'confirmMergeBtn'
|
||||
];
|
||||
const htmlIds = new Set([...html.matchAll(/\bid=["']([^"']+)["']/g)].map(m => m[1]));
|
||||
check('dom.critical-ids', 'critical operator DOM IDs are present', all(expectedIds, id => htmlIds.has(id)), listMissing(expectedIds, id => htmlIds.has(id)).join(', '));
|
||||
const expectedMarkers = ['saved-views', 'review-queue', 'crm-pipeline', 'crm-interactions', 'crm-reports', 'suppression-center', 'outreach-provider-policy', 'score-rules', 'score-distribution'];
|
||||
const smokeMarkers = new Set([...html.matchAll(/data-smoke=["']([^"']+)["']/g)].map(m => m[1]));
|
||||
check('dom.smoke-markers', 'critical sections expose stable smoke markers', all(expectedMarkers, marker => smokeMarkers.has(marker)), listMissing(expectedMarkers, marker => smokeMarkers.has(marker)).join(', '));
|
||||
const dynamicMarkers = ['score-breakdown', 'deduplication'];
|
||||
const dynamicMarkerPresent = marker => js.includes(`dataset.smoke = '${marker}'`) || js.includes(`dataset.smoke='${marker}'`) || js.includes(`data-smoke="${marker}"`) || js.includes(`data-smoke='${marker}'`);
|
||||
check('dom.dynamic-markers', 'detail safety panels define dynamic smoke markers', all(dynamicMarkers, dynamicMarkerPresent), listMissing(dynamicMarkers, dynamicMarkerPresent).join(', '));
|
||||
const routeContracts = [
|
||||
'/api/v1/auth/me', '/api/v1/auth/login', '/api/v1/auth/logout', '/api/v1/dashboard/summary', '/api/v1/businesses',
|
||||
'/api/v1/review-queue', '/api/v1/saved-filters', '/api/v1/businesses/bulk-review', '/api/v1/jobs', '/api/v1/sources',
|
||||
'/api/v1/source-records', '/api/v1/discovery-queries', '/api/v1/merge-history', '/api/v1/scoring/summary', '/api/v1/score-rules',
|
||||
'/api/v1/pipeline-entries', '/api/v1/interactions', '/api/v1/reports/pipeline', '/api/v1/reports/outcomes', '/api/v1/reports/activity',
|
||||
'/api/v1/suppressions', '/api/v1/ai-runs', '/api/v1/outreach/drafts', '/api/v1/outreach/provider-config',
|
||||
'/matches', '/domains/check', '/domain-candidates', '/websites/scan', '/contacts/extract', '/score/recalculate', '/pipeline', '/verify'
|
||||
];
|
||||
check('routes.contracts', 'all critical API route contracts are referenced by the client', all(routeContracts, route => js.includes(route)), listMissing(routeContracts, route => js.includes(route)).join(', '));
|
||||
check('routes.authenticated', 'protected API requests include cookie credentials', js.includes("credentials:'include'") && js.includes('jsonRequest'));
|
||||
check('routes.no-arbitrary-target-fetch', 'browser does not fetch arbitrary target URLs directly', !/fetch\s*\(\s*(?:url|target|website|href)\b/i.test(js));
|
||||
|
||||
const manifestAssets = manifest && [...new Set([...(manifest.entrypoints || []), ...(manifest.publicAssets || [])])];
|
||||
check('manifest.schema', 'asset manifest has a supported schema and asset lists', Boolean(manifest && manifest.schema === 1 && Array.isArray(manifest.entrypoints) && Array.isArray(manifest.publicAssets) && manifest.integrity && manifestAssets.length), manifest ? '' : 'manifest unavailable');
|
||||
if (manifestAssets) {
|
||||
check('manifest.assets-exist', 'every manifest asset exists and is non-empty', all(manifestAssets, asset => files[asset]?.length > 0), listMissing(manifestAssets, asset => files[asset]?.length > 0).join(', '));
|
||||
check('manifest.integrity-complete', 'manifest integrity keys exactly cover manifest assets', manifestAssets.length === Object.keys(manifest.integrity).length && all(manifestAssets, asset => Object.hasOwn(manifest.integrity, asset)), `assets=${manifestAssets.length}, integrity=${Object.keys(manifest.integrity).length}`);
|
||||
const integrityFailures = manifestAssets.filter(asset => sha256(files[asset] || Buffer.alloc(0)) !== manifest.integrity[asset]);
|
||||
check('manifest.integrity', 'manifest SHA-256 digests match local release assets', integrityFailures.length === 0, integrityFailures.join(', '));
|
||||
check('manifest.hash-format', 'manifest integrity values use sha256 hex format', all(manifestAssets, asset => /^sha256-[0-9a-f]{64}$/.test(manifest.integrity[asset] || '')), listMissing(manifestAssets, asset => /^sha256-[0-9a-f]{64}$/.test(manifest.integrity[asset] || '')).join(', '));
|
||||
}
|
||||
|
||||
const linkedAssets = [...new Set([...html.matchAll(/(?:src|href)=["']([^"'#?]+)(?:\?[^"']*)?["']/gi)].map(m => m[1]).filter(asset => !/^(?:https?:|data:|#)/i.test(asset)).map(asset => asset.replace(/^\.\//, '')) )];
|
||||
check('assets.linked-local', 'all local HTML assets exist and are non-empty', all(linkedAssets, asset => files[asset]?.length > 0), listMissing(linkedAssets, asset => files[asset]?.length > 0).join(', '));
|
||||
check('css.responsive', 'responsive CSS covers mobile layouts and critical grids', /@media\s*\(\s*max-width\s*:\s*700px\s*\)/.test(css) && all(['.saved-view-controls', '.website-scan-grid', '.crm-two-col', '.reports-grid', '.outreach-panel', '.pipeline-board'], selector => css.includes(selector)));
|
||||
check('css.layout-contracts', 'critical desktop layout selectors are defined', all(['.app-shell', '.sidebar', '.workspace-grid', '.table-scroll', '.jobs-grid', '.sources-grid', '.score-config-row', '.provider-policy-row'], selector => css.includes(selector)));
|
||||
|
||||
const combined = `${html}\n${js}\n${css}\n${config}\n${text('README.md')}`;
|
||||
const safetyCopy = ['No outreach from this workspace.', 'Suppression always wins.', 'Sending is disabled by default.', 'Approval does not send a message.', 'no outreach will be sent', 'AUTOMATED_OUTREACH_ENABLED=false'];
|
||||
check('safety.no-send-copy', 'persistent no-send and approval safety copy is present', all(safetyCopy, phrase => combined.includes(phrase)), listMissing(safetyCopy, phrase => combined.includes(phrase)).join(' | '));
|
||||
const buttonLabels = [...html.matchAll(/<button\b[^>]*>([\s\S]*?)<\/button>/gi)].map(m => m[1].replace(/<[^>]+>/g, ' '));
|
||||
check('safety.no-send-controls', 'static HTML exposes no send, delivery, or campaign control', !buttonLabels.some(label => /\b(send|deliver|campaign|schedule delivery)\b/i.test(label)));
|
||||
const approvalTokens = ['window.confirm', 'human_approval:true', 'send:false', 'autonomous_action:false'];
|
||||
check('safety.approval-gated', 'approval actions require explicit human confirmation and remain non-delivering', all(approvalTokens, token => js.includes(token)), listMissing(approvalTokens, token => js.includes(token)).join(', '));
|
||||
const suppressionTokens = ['statusOf', "st==='suppressed'", "renderAiState('suppressed')", "renderOutreachState('suppressed')", 'suppression', 'Do not contact'];
|
||||
check('safety.suppression-precedence', 'suppression disables AI/outreach and preserves do-not-contact state', all(suppressionTokens, token => combined.includes(token)), listMissing(suppressionTokens, token => combined.includes(token)).join(', '));
|
||||
check('safety.source-disabled', 'discovery remains disabled until an approved source is enabled', all(['Discovery is disabled by default.', 'enabled:false', 'This source is disabled. Enable it only after review.'], token => combined.includes(token)));
|
||||
check('safety.no-external-network', 'client has no direct provider or target network origins', !/(?:fetch|XMLHttpRequest|WebSocket)\s*\(\s*[`'\"]https?:\/\//i.test(js));
|
||||
const secretPatterns = [
|
||||
/-----BEGIN(?: RSA| EC| OPENSSH)? PRIVATE KEY-----/i,
|
||||
/(?:api[_-]?key|client[_-]?secret|access[_-]?token|refresh[_-]?token|private[_-]?key|authorization)\s*[:=]\s*["'][^"']{8,}/i,
|
||||
/\b(?:AKIA|ASIA)[A-Z0-9]{16}\b/,
|
||||
/\beyJ[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}\.[a-zA-Z0-9_-]{10,}\b/
|
||||
];
|
||||
const secretHits = secretPatterns.flatMap(pattern => [...combined.matchAll(new RegExp(pattern.source, `${pattern.flags.replace('g', '')}g`))].map(m => m[0].slice(0, 80)));
|
||||
check('safety.no-hardcoded-secrets', 'frontend source has no obvious hardcoded secrets', secretHits.length === 0, secretHits.join(' | '));
|
||||
|
||||
async function httpChecks(url) {
|
||||
for (const asset of [...new Set([...(manifestAssets || []), ...linkedAssets])].sort()) {
|
||||
try {
|
||||
const response = await fetch(`${url}/${asset}`);
|
||||
const body = Buffer.from(await response.arrayBuffer());
|
||||
const expected = asset.endsWith('.html') ? 'text/html' : asset.endsWith('.css') ? 'text/css' : asset.endsWith('.js') ? 'javascript' : null;
|
||||
const contentType = response.headers.get('content-type') || '';
|
||||
const integrityPass = !manifest?.integrity?.[asset] || sha256(body) === manifest.integrity[asset];
|
||||
check(`http.${asset}`, `HTTP delivery and integrity: ${asset}`, response.ok && body.length > 0 && (!expected || contentType.includes(expected)) && integrityPass, `${response.status} ${contentType || 'missing content-type'}${integrityPass ? '' : '; integrity mismatch'}`);
|
||||
} catch (error) {
|
||||
check(`http.${asset}`, `HTTP delivery and integrity: ${asset}`, false, error.message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let child;
|
||||
try {
|
||||
let httpUrl = baseUrl;
|
||||
if (startServer) {
|
||||
const port = Number(portArg);
|
||||
if (!Number.isInteger(port) || port < 0 || port > 65535) throw new Error(`invalid --port: ${portArg}`);
|
||||
child = spawn('python3', ['-m', 'http.server', String(port), '--bind', '127.0.0.1', '--directory', webRoot], { stdio: ['ignore', 'ignore', 'ignore'] });
|
||||
const actualPort = await new Promise((resolvePort, reject) => {
|
||||
const timer = setTimeout(() => reject(new Error('timed out waiting for static server')), 5000);
|
||||
const probe = async () => { const candidate = port; try { await fetch(`http://127.0.0.1:${candidate}/healthz`); clearTimeout(timer); resolvePort(candidate); } catch { setTimeout(probe, 50); } };
|
||||
child.once('error', reject); probe();
|
||||
});
|
||||
httpUrl = `http://127.0.0.1:${actualPort}`;
|
||||
}
|
||||
if (httpUrl) await httpChecks(httpUrl.replace(/\/$/, ''));
|
||||
} catch (error) {
|
||||
check('http.server', 'optional static server is reachable', false, error.message);
|
||||
} finally {
|
||||
if (child) child.kill();
|
||||
}
|
||||
|
||||
const failed = results.filter(result => !result.pass);
|
||||
const report = {
|
||||
schema: 1,
|
||||
harness: 'prospectos-frontend-final-acceptance',
|
||||
root: webRoot,
|
||||
http_checks: Boolean(baseUrl || startServer),
|
||||
base_url: baseUrl || (startServer ? 'managed static server' : null),
|
||||
pass: failed.length === 0,
|
||||
totals: { checks: results.length, passed: results.length - failed.length, failed: failed.length },
|
||||
checks: results
|
||||
};
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
process.exitCode = failed.length ? 1 : 0;
|
||||
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env node
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { resolve } from 'node:path';
|
||||
|
||||
const base = (process.argv[2] || 'http://127.0.0.1:8080').replace(/\/$/, '');
|
||||
const manifestPath = resolve(process.cwd(), 'asset-manifest.json');
|
||||
const manifest = JSON.parse(await readFile(manifestPath, 'utf8'));
|
||||
const assets = [...new Set([...(manifest.entrypoints || []), ...(manifest.publicAssets || [])])];
|
||||
const failures = [];
|
||||
const secretPattern = /(api[_-]?key|secret|token|password|private[_-]?key|authorization)\s*[:=]\s*["'][^"']+/i;
|
||||
|
||||
async function check(path, expected = {}) {
|
||||
const response = await fetch(`${base}/${path}`);
|
||||
const body = await response.text();
|
||||
if (!response.ok) failures.push(`${path}: HTTP ${response.status}`);
|
||||
if (expected.contentType && !response.headers.get('content-type')?.includes(expected.contentType)) failures.push(`${path}: unexpected content type`);
|
||||
if (expected.marker && !body.includes(expected.marker)) failures.push(`${path}: missing marker ${expected.marker}`);
|
||||
if (expected.integrity) {
|
||||
const digest = `sha256-${createHash('sha256').update(body).digest('hex')}`;
|
||||
if (digest !== expected.integrity) failures.push(`${path}: integrity mismatch`);
|
||||
}
|
||||
if (secretPattern.test(body)) failures.push(`${path}: possible hardcoded secret`);
|
||||
console.log(`${response.ok ? 'PASS' : 'FAIL'} ${path} (${response.status})`);
|
||||
}
|
||||
|
||||
const expectations = {
|
||||
'index.html': { contentType: 'text/html', marker: 'ProspectOS' },
|
||||
'health.html': { contentType: 'text/html', marker: 'Ready' },
|
||||
'error.html': { contentType: 'text/html', marker: 'Something went wrong' },
|
||||
healthz: { marker: 'ok' }
|
||||
};
|
||||
for (const asset of assets) await check(asset, { ...expectations[asset], integrity: manifest.integrity?.[asset] });
|
||||
if (failures.length) { console.error(`\n${failures.length} deployment smoke check(s) failed`); for (const failure of failures) console.error(`- ${failure}`); process.exitCode = 1; } else console.log(`\nDeployment smoke checks passed for ${base}`);
|
||||
@@ -0,0 +1,136 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Deterministic, Chromium-free frontend pilot smoke harness.
|
||||
*
|
||||
* Usage:
|
||||
* node scripts/smoke-frontend.mjs http://127.0.0.1:8080
|
||||
*
|
||||
* The final stdout value is JSON so CI can consume it directly. Exit code is
|
||||
* non-zero when any check fails. The checks intentionally inspect source HTML,
|
||||
* JavaScript, and CSS instead of executing the application or making API calls.
|
||||
*/
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const webRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
const baseUrl = (process.argv[2] || 'http://127.0.0.1:8080').replace(/\/$/, '');
|
||||
const files = Object.fromEntries(await Promise.all(
|
||||
['index.html', 'app.js', 'styles.css', 'config.js', 'health.html', 'error.html', 'healthz', 'smoke-test.html']
|
||||
.map(async name => [name, await readFile(resolve(webRoot, name), 'utf8')])
|
||||
));
|
||||
|
||||
const results = [];
|
||||
function check(id, description, pass, details = '') {
|
||||
results.push({ id, description, pass: Boolean(pass), ...(details ? { details } : {}) });
|
||||
}
|
||||
function has(text, needles) {
|
||||
return needles.every(needle => text.includes(needle));
|
||||
}
|
||||
function ids(html) {
|
||||
return [...html.matchAll(/\bid=["']([^"']+)["']/g)].map(match => match[1]);
|
||||
}
|
||||
function smokeMarkers(html) {
|
||||
return [...html.matchAll(/data-smoke=["']([^"']+)["']/g)].map(match => match[1]);
|
||||
}
|
||||
|
||||
const html = files['index.html'];
|
||||
const js = files['app.js'];
|
||||
const css = files['styles.css'];
|
||||
const htmlIds = new Set(ids(html));
|
||||
const markers = new Set(smokeMarkers(html));
|
||||
|
||||
check('dom.operator-markers', 'operator-critical DOM markers are present', has(html, [
|
||||
'id="loginScreen"', 'id="dashboardShell"', 'id="explorer"', 'id="detailPanel"',
|
||||
'id="reviewQueueCount"', 'id="jobsList"', 'id="sourcesList"', 'id="pipelineBoard"',
|
||||
'id="interactionState"', 'id="pipelineReport"', 'id="suppressionState"',
|
||||
'id="providerPolicyPanel"', 'id="scoreRulesPanel"', 'id="scoreDistributionPanel"'
|
||||
]));
|
||||
check('dom.safety-markers', 'safety and approval sections have stable smoke markers', [
|
||||
'saved-views', 'review-queue', 'crm-pipeline', 'crm-interactions', 'crm-reports',
|
||||
'suppression-center', 'outreach-provider-policy', 'score-rules', 'score-distribution'
|
||||
].every(marker => markers.has(marker)));
|
||||
check('dom.required-controls', 'operator controls have stable IDs', [
|
||||
'loginEmail', 'loginPassword', 'logoutBtn', 'nextPageBtn', 'bulkVerifyBtn', 'bulkRejectBtn',
|
||||
'savedFilterForm', 'pipelineViewToggle', 'interactionForm', 'suppressionForm', 'outreachPolicyRefreshBtn'
|
||||
].every(id => htmlIds.has(id)) && has(js, [
|
||||
'scanWebsiteBtn', 'extractContactsBtn', 'recalculateScoreBtn', 'generateAiSuggestionBtn', 'createOutreachDraftBtn'
|
||||
]));
|
||||
|
||||
check('css.responsive', 'responsive CSS is present for mobile operator layouts',
|
||||
/@media\s*\(max-width\s*:\s*700px\)/.test(css) &&
|
||||
has(css, ['.saved-view-controls', '.website-scan-grid', '.crm-two-col', '.reports-grid', '.outreach-panel']));
|
||||
check('css.layout-contracts', 'critical layout selectors are defined', has(css, [
|
||||
'.app-shell', '.sidebar', '.workspace-grid', '.table-scroll', '.pipeline-board',
|
||||
'.jobs-grid', '.sources-grid', '.score-config-row', '.provider-policy-row'
|
||||
]));
|
||||
|
||||
check('safety.no-send-copy', 'persistent no-send and approval safety copy is present', has(`${html}\n${js}`, [
|
||||
'No outreach from this workspace.', 'Suppression always wins.', 'Sending is disabled by default.',
|
||||
'Approval does not send a message.', 'no outreach will be sent', 'send:false',
|
||||
'autonomous_action:false'
|
||||
]));
|
||||
check('safety.no-send-controls', 'static HTML exposes no send or delivery button',
|
||||
![...html.matchAll(/<button\b[^>]*>([\s\S]*?)<\/button>/gi)]
|
||||
.some(match => /\b(send|deliver|campaign)\b/i.test(match[1])));
|
||||
check('safety.no-secrets', 'frontend source has no obvious hardcoded secrets',
|
||||
!/(api[_-]?key|secret|token|password|private[_-]?key|authorization)\s*[:=]\s*["'][^"']+/i.test(
|
||||
`${html}\n${js}\n${files['config.js']}`
|
||||
));
|
||||
check('safety.approval-gated', 'approval is explicitly human-confirmed and non-delivering',
|
||||
has(js, ['window.confirm', 'human_approval:true', 'send:false', 'Approval does not send a message.']));
|
||||
|
||||
const routes = [
|
||||
'/api/v1/auth/me', '/api/v1/auth/login', '/api/v1/auth/logout', '/api/v1/businesses',
|
||||
'/api/v1/review-queue', '/api/v1/saved-filters', '/api/v1/businesses/bulk-review',
|
||||
'/api/v1/jobs', '/api/v1/sources', '/api/v1/source-records', '/api/v1/discovery-queries',
|
||||
'/api/v1/merge-history', '/api/v1/scoring/summary', '/api/v1/score-rules',
|
||||
'/api/v1/pipeline-entries', '/api/v1/interactions', '/api/v1/reports/pipeline',
|
||||
'/api/v1/reports/outcomes', '/api/v1/reports/activity', '/api/v1/suppressions',
|
||||
'/api/v1/ai-runs', '/api/v1/outreach/drafts', '/api/v1/outreach/provider-config'
|
||||
];
|
||||
check('routes.contracts', 'operator API route contracts are referenced by the client',
|
||||
routes.every(route => js.includes(route)), routes.filter(route => !js.includes(route)).join(', '));
|
||||
check('routes.authenticated', 'API requests use cookie credentials',
|
||||
js.includes("credentials:'include'") && js.includes('jsonRequest'));
|
||||
check('routes.no-target-fetch', 'client does not directly fetch arbitrary target URLs',
|
||||
!/fetch\s*\(\s*(?:url|target|website|href)\b/i.test(js));
|
||||
|
||||
const linkedAssets = new Set(['index.html', 'config.js', 'app.js', 'styles.css', 'health.html', 'error.html', 'healthz', 'smoke-test.html']);
|
||||
for (const match of html.matchAll(/(?:src|href)=["']([^"'#?]+)(?:\?[^"']*)?["']/gi)) {
|
||||
const asset = match[1];
|
||||
if (!asset.startsWith('http') && !asset.startsWith('data:')) linkedAssets.add(asset.replace(/^\.\//, ''));
|
||||
}
|
||||
check('assets.local', 'all linked static assets exist and are non-empty', [...linkedAssets].every(asset => files[asset]?.length > 0),
|
||||
[...linkedAssets].filter(asset => !files[asset]?.length).join(', '));
|
||||
|
||||
async function verifyHttp() {
|
||||
for (const asset of [...linkedAssets].sort()) {
|
||||
const url = `${baseUrl}/${asset}`;
|
||||
try {
|
||||
const response = await fetch(url);
|
||||
const body = await response.text();
|
||||
const contentType = response.headers.get('content-type') || '';
|
||||
const expectedType = asset.endsWith('.html') ? 'text/html' : asset.endsWith('.css') ? 'text/css' : asset.endsWith('.js') ? 'javascript' : null;
|
||||
const pass = response.ok && body.length > 0 && (!expectedType || contentType.includes(expectedType));
|
||||
check(`http.${asset}`, `HTTP delivery: ${asset}`, pass,
|
||||
pass ? `${response.status} ${contentType}` : `${response.status} ${contentType || 'missing content-type'}`);
|
||||
} catch (error) {
|
||||
check(`http.${asset}`, `HTTP delivery: ${asset}`, false, error.message);
|
||||
}
|
||||
}
|
||||
}
|
||||
await verifyHttp();
|
||||
|
||||
const failed = results.filter(result => !result.pass);
|
||||
const report = {
|
||||
schema: 1,
|
||||
harness: 'prospectos-frontend-pilot-smoke',
|
||||
base_url: baseUrl,
|
||||
chromium_required: false,
|
||||
pass: failed.length === 0,
|
||||
totals: { checks: results.length, passed: results.length - failed.length, failed: failed.length },
|
||||
checks: results
|
||||
};
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
if (failed.length) process.exitCode = 1;
|
||||
@@ -42,5 +42,28 @@ frame.onload=async()=>{const d=frame.contentDocument;const js=await fetch('app.j
|
||||
,['Website scan loading and error states',()=>js.includes('Loading website scan')&&js.includes('Unable to load website scan')&&js.includes('Website scan failed')&&js.includes('role="alert"')]
|
||||
,['Website scan avoids browser execution claims',()=>!js.includes('screenshot')&&!js.includes('browser execution')&&!js.includes('page screenshot')]
|
||||
,['Website scan responsive styles',()=>js.includes('website-scan-panel')&&js.includes('website-scan-grid')&&js.includes('@media')]
|
||||
,['Public contact extraction smoke marker and controls',()=>!!d.querySelector('[data-smoke="contact-extraction"]')&&!!d.querySelector('#extractContactsBtn')&&!!d.querySelector('#refreshContactExtractionBtn')&&js.includes('/contacts/extract')&&js.includes('jsonRequest')]
|
||||
,['Extracted contact evidence fields',()=>['Extracted contacts','Type','Email class','Validation','confidence','Source URL','Suppression'].every(x=>js.includes(x))]
|
||||
,['Contact extraction safety and states',()=>js.includes('approved public business pages only')&&js.includes('SMTP')&&js.includes('outreach')&&js.includes('Loading public contacts')&&js.includes('No public contacts found')&&js.includes('Contact extraction failed')]
|
||||
,['Score breakdown and recalculation contract',()=>!!d.querySelector('[data-smoke="score-breakdown"]')&&js.includes('Score breakdown')&&js.includes('Total score')&&js.includes('Priority')&&js.includes('Eligibility')&&js.includes('Score version')&&js.includes('/score/recalculate')&&js.includes('recalculateScoreBtn')]
|
||||
,['Score rules configuration and permission affordance',()=>!!d.querySelector('[data-smoke="score-rules"]')&&!!d.querySelector('#scoreRules')&&js.includes('/api/v1/score-rules')&&js.includes('enabled')&&js.includes('points')&&js.includes('Read-only for this role')&&js.includes('canEditScores')]
|
||||
,['Score distribution summary and safe states',()=>!!d.querySelector('[data-smoke="score-distribution"]')&&js.includes('/scoring/summary')&&js.includes('Loading score distribution')&&js.includes('Unable to load score distribution')&&js.includes('tenant-scoped')&&js.includes('do not override suppression')]
|
||||
,['Score responsive styles',()=>js.includes('score-breakdown-summary')&&js.includes('score-config-row')&&js.includes('distribution-track')&&js.includes('@media')]
|
||||
,['Phase 11 saved views and review queue controls',()=>!!d.querySelector('[data-smoke="saved-views"]')&&!!d.querySelector('[data-smoke="review-queue"]')&&!!d.querySelector('#savedFilterForm')&&!!d.querySelector('#savedFilterSelect')&&!!d.querySelector('#bulkVerifyBtn')&&!!d.querySelector('#bulkRejectBtn')]
|
||||
,['Phase 11 authenticated saved-view and bulk-review contracts',()=>js.includes('/api/v1/saved-filters')&&js.includes('/api/v1/review-queue')&&js.includes('/api/v1/businesses/bulk-review')&&js.includes('jsonRequest')&&js.includes('window.confirm')]
|
||||
,['Phase 11 bounded selection and safe labels',()=>js.includes('slice(0,100)')&&js.includes('selected visible')&&js.includes('no outreach will be sent')&&d.querySelector('#reviewQueueCount')]
|
||||
,['Phase 11 dashboard cards link to filters and responsive styles',()=>d.querySelectorAll('[data-dashboard-filter]').length>=5&&js.includes('applyFilters')&&js.includes('metricSuppressed')&&js.includes('saved-view-controls')&&js.includes('@media')]
|
||||
,['Phase 12 CRM pipeline board/list and explicit transitions',()=>!!d.querySelector('[data-smoke="crm-pipeline"]')&&!!d.querySelector('#pipelineBoard')&&!!d.querySelector('#pipelineViewToggle')&&js.includes('/api/v1/pipeline-entries')&&js.includes('Stage transition saved')&&js.includes('outreach:false')&&js.includes('window.confirm')]
|
||||
,['Phase 12 interaction timeline and metadata form',()=>!!d.querySelector('[data-smoke="crm-interactions"]')&&!!d.querySelector('#interactionForm')&&!!d.querySelector('#interactionState')&&js.includes('/api/v1/interactions?business_id=')&&js.includes('follow_up_at')&&js.includes('outcome')&&js.includes('No message was sent')]
|
||||
,['Phase 12 reports panels and API states',()=>!!d.querySelector('[data-smoke="crm-reports"]')&&['pipelineReport','outcomesReport','activityReport'].every(id=>!!d.querySelector('#'+id))&&['/api/v1/reports/pipeline','/api/v1/reports/outcomes','/api/v1/reports/activity'].every(path=>js.includes(path))&&js.includes('No')&&js.includes('Unable to load')]
|
||||
,['Phase 12 suppression center controls and safety',()=>!!d.querySelector('[data-smoke="suppression-center"]')&&!!d.querySelector('#suppressionForm')&&!!d.querySelector('#bulkReviewSuppressionsBtn')&&js.includes('/api/v1/suppressions')&&js.includes('data-remove-suppression')&&js.includes('Suppression always wins')&&js.includes('do-not-contact')]
|
||||
,['Phase 12 CRM responsive styles and authenticated requests',()=>js.includes('credentials:\'include\'')&&js.includes('crm-two-col')&&js.includes('reports-grid')&&js.includes('pipeline-board')&&js.includes('@media')]
|
||||
,['Phase 13 AI assistance panel and safety contract',()=>!!d.querySelector('[data-smoke="ai-assistance"]')&&!!d.querySelector('#generateAiSuggestionBtn')&&js.includes('/api/v1/ai-runs')&&js.includes('/api/v1/businesses/${encodeURIComponent(selectedId)}/ai/suggest')&&js.includes('Evidence-grounded suggestion')&&js.includes('No autonomous action')&&js.includes('citations')]
|
||||
,['Phase 13 AI states, provider, and human decisions',()=>['not-configured','Unknown','Error','Suppressed','Provider:','Pending approval','Approve','Reject','human_approval','no outreach will be sent'].every(x=>js.includes(x))&&js.includes('Loading AI assistance')]
|
||||
,['Phase 13 AI authenticated requests and responsive styles',()=>js.includes('/api/v1/ai-runs')&&js.includes('/ai/suggest')&&js.includes('autonomous_action:false')&&js.includes('ai-assistance-panel')&&js.includes('@media')]
|
||||
,['Phase 14 outreach preparation panel and draft-only contract',()=>!!d.querySelector('[data-smoke="outreach-provider-policy"]')&&js.includes('outreach-preparation')&&js.includes('/api/v1/outreach/drafts')&&js.includes('recipient_review_required:true')]
|
||||
,['Phase 14 citations, approval, audit, and no-send safety',()=>['Evidence citations','Recipient review','Evidence hash','Policy:','Approve draft','Approval does not send a message','Send unavailable','send:false','autonomous_action:false'].every(x=>js.includes(x))&&!js.includes('automaticSend')]
|
||||
,['Phase 14 provider policy states and secret-free settings',()=>!!d.querySelector('#providerPolicyPanel')&&js.includes('/api/v1/outreach/provider-config')&&['Configured','Enabled','Not displayed','Loading provider policy','auth-required','suppressed','error'].every(x=>js.includes(x))]
|
||||
,['Phase 14 responsive outreach styles',()=>js.includes('outreach-panel')&&js.includes('provider-policy-row')&&js.includes('@media')]
|
||||
];let passed=0;document.querySelector('#checks').innerHTML=checks.map(([name,test])=>{const ok=test();if(ok)passed++;return `<li class="${ok?'pass':'fail'}">${ok?'PASS':'FAIL'} — ${name}</li>`}).join('');document.querySelector('#summary').textContent=`${passed}/${checks.length} checks passed`;};
|
||||
</script>
|
||||
|
||||
+9
-1
File diff suppressed because one or more lines are too long
+6
-1
@@ -10,6 +10,7 @@ services:
|
||||
API_PORT: 8000
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
CORS_ORIGINS: ${CORS_ORIGINS:-http://localhost:8080}
|
||||
SESSION_SECRET: ${SESSION_SECRET:-}
|
||||
DATA_DIR: /data
|
||||
# Optional first-run admin bootstrap; leave unset after provisioning.
|
||||
BOOTSTRAP_ADMIN_EMAIL: ${BOOTSTRAP_ADMIN_EMAIL:-}
|
||||
@@ -20,6 +21,8 @@ services:
|
||||
volumes:
|
||||
- prospect_api_data:/data
|
||||
read_only: true
|
||||
init: true
|
||||
pids_limit: 128
|
||||
tmpfs:
|
||||
- /tmp
|
||||
security_opt:
|
||||
@@ -27,7 +30,7 @@ services:
|
||||
cap_drop:
|
||||
- ALL
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/v1/health/live', timeout=2)"]
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/api/v1/health/ready', timeout=2)"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
@@ -44,6 +47,8 @@ services:
|
||||
ports:
|
||||
- "${WEB_PORT:-8080}:8080"
|
||||
read_only: true
|
||||
init: true
|
||||
pids_limit: 128
|
||||
tmpfs:
|
||||
- /tmp
|
||||
security_opt:
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
# Phase 16 pilot benchmark methodology
|
||||
|
||||
Phase 16 defines a **pilot evaluation protocol**, not a production performance or quality claim. It evaluates the existing deterministic classification/scoring behavior and any explicitly named pilot path against labeled synthetic fixtures. It does not authorize live discovery, enrichment, acquisition, verification, or outreach, and it does not replace security, privacy, legal, or operational review.
|
||||
|
||||
## Scope and evaluation questions
|
||||
|
||||
The benchmark must answer only these bounded questions:
|
||||
|
||||
1. Given a fixed synthetic input and versioned configuration, does the classifier or scorer produce the expected class, score, band, eligibility, explanation, and uncertainty state?
|
||||
2. How often do pilot outputs agree with the fixture labels under the declared policy?
|
||||
3. What are the cold-cache and warm-cache latency distributions under a documented local workload?
|
||||
4. Are results reproducible across repeated runs, processes, and supported Compose rebuilds?
|
||||
|
||||
A benchmark result is valid only for the fixture release, code revision, dependency/image digests, configuration fingerprint, host profile, and measurement procedure recorded with it. Do not extrapolate it to production traffic, real businesses, real people, live websites, external providers, or a different infrastructure profile.
|
||||
|
||||
## Labeled synthetic fixtures
|
||||
|
||||
Fixtures are synthetic and must contain no live personal data, credentials, provider responses, or uncontrolled network dependencies. Each fixture has a stable ID, schema version, input payload, expected label(s), expected uncertainty/suppression state where applicable, and a short rationale. Keep fixture inputs separate from expected outputs so the evaluator cannot silently derive labels from the implementation.
|
||||
|
||||
The fixture set should include balanced positive and negative examples plus deliberately difficult cases:
|
||||
|
||||
- positive, negative, unknown, blocked, partial, timeout, and error observations;
|
||||
- boundary values around each score/band threshold;
|
||||
- stale, conflicting, missing, suppressed, malformed, and duplicate inputs;
|
||||
- adversarial-looking but harmless text, placeholder/example values, and irrelevant fields;
|
||||
- repeated records and cacheable/non-cacheable cases;
|
||||
- tenant-separation cases proving that identifiers, evidence, and cached results do not cross organizations.
|
||||
|
||||
Publish the fixture manifest and label policy with each benchmark run. Labels describe the expected outcome under the stated policy; they are not assertions that a real-world fact is true.
|
||||
|
||||
## Classification metrics and limitations
|
||||
|
||||
For a declared positive class, report a confusion matrix (`TP`, `FP`, `TN`, `FN`) and:
|
||||
|
||||
- **precision** = `TP / (TP + FP)`;
|
||||
- **recall** = `TP / (TP + FN)`;
|
||||
- support (number of labeled positive and negative fixtures);
|
||||
- unknown/blocked/partial/error rates separately from positive/negative rates;
|
||||
- per-class metrics when there is more than one class.
|
||||
|
||||
Use `N/A`, not zero, when a denominator is zero. Report micro/macro averaging choices for multiclass results and list the exact threshold, tie-breaking, abstention, and suppression rules used. A high precision or recall on synthetic fixtures does not demonstrate identity, ownership, consent, deliverability, safety, generalization, fairness, or production usefulness. Synthetic labels may be simpler, cleaner, or less representative than real inputs; class balance, fixture leakage, threshold tuning, and evaluator assumptions can materially change the result. Precision/recall also do not measure latency, calibration, operational failure modes, or tenant isolation.
|
||||
|
||||
## Reproducible classification and scoring
|
||||
|
||||
Every run records:
|
||||
|
||||
- repository commit and dirty/clean state;
|
||||
- fixture manifest ID/hash and label-policy version;
|
||||
- classifier/scorer algorithm and rule-set versions;
|
||||
- Python/runtime and dependency versions plus Compose/image digests when Compose is used;
|
||||
- explicit environment/configuration values, excluding secrets;
|
||||
- random seed, locale, timezone, rounding mode, numeric precision, and tie-breaking rules;
|
||||
- run command, host/CPU/RAM profile, start/end timestamps, and raw machine-readable results.
|
||||
|
||||
Run each fixture at least twice in fresh processes and compare serialized outputs, not only aggregate metrics. A reproducibility check passes only when class, score, band, eligibility, uncertainty reasons, explanation factors, and version metadata are identical. If a timestamp is intentionally dynamic, compare it through an explicitly documented normalization rule; never hide nondeterminism by rounding away output differences. Preserve failed and incomplete runs.
|
||||
|
||||
## Latency and cache measurements
|
||||
|
||||
Measure a fixed, documented workload with warm-up excluded from the reported sample. Record request/fixture ID, outcome, cache state (`cold`, `warm`, `bypass`, or `not_applicable`), start/end monotonic timestamps, duration, and failure/timeout reason. Report count, minimum, median, p95, p99, maximum, and timeout/error rate for cold and warm paths separately; do not report only an average. Use the same host, process model, concurrency, payload sizes, timeout budgets, and fixture order for comparable runs. State whether serialization, database access, container startup, and network calls are included. The default pilot benchmark is local and network-free; any network-backed experiment must be separately approved and named.
|
||||
|
||||
A cache hit must be labeled as a cache hit and must not be treated as a fresh observation. Record cache key policy, policy/scanner/rule versions, freshness/expiry, invalidation behavior, and hit/miss counts. Compare cold and warm measurements only when the cache contents and invalidation procedure are recorded. Do not infer production capacity, tail latency, cost, or availability from this pilot sample.
|
||||
|
||||
## Pilot acceptance thresholds
|
||||
|
||||
These are release gates for the **synthetic pilot evaluation only**; they are not service-level objectives or production claims:
|
||||
|
||||
| Gate | Pilot threshold | Required evidence |
|
||||
| --- | --- | --- |
|
||||
| Reproducibility | 100% byte-for-byte equality for deterministic output fields across repeated fresh-process runs | Per-fixture comparison report and manifest/config/version metadata |
|
||||
| Positive classification precision | `>= 0.90` when positive support is at least 20; otherwise report `N/A` and do not pass the quality gate | Confusion matrix, support, threshold, and fixture manifest |
|
||||
| Positive classification recall | `>= 0.85` when positive support is at least 20; otherwise report `N/A` and do not pass the quality gate | Confusion matrix, support, threshold, and fixture manifest |
|
||||
| Safety abstention | 100% of labeled suppressed, blocked, malformed, or explicitly unknown cases remain non-eligible/non-positive as applicable | Per-fixture safety assertions and failure list |
|
||||
| Tenant isolation | 100% of cross-tenant access/cache probes are denied or absent without revealing existence | Readback/probe evidence with tenant IDs redacted as needed |
|
||||
| Latency reporting | 100% of samples have monotonic duration and cache-state labels; no missing/error samples are discarded | Raw sample file and summary statistics |
|
||||
|
||||
A failed or `N/A` gate is a pilot finding, not permission to tune labels or silently remove fixtures. Threshold changes require a new methodology/version and a fresh run. No benchmark may be signed off if fixtures, expected labels, failed samples, or configuration metadata are missing.
|
||||
|
||||
## Human review sampling
|
||||
|
||||
Automated metrics are supplemented by blinded human review. Select a reproducible random sample using the recorded seed, stratified at minimum by predicted class, uncertainty state, suppression state, and disagreement (`FP`/`FN`) status. Reviewers must see the bounded synthetic input and output explanation, but not the implementation label or aggregate result while labeling. Use at least 20 fixtures or 10% of the fixture set, whichever is larger, capped at the declared review budget; if fewer fixtures exist, review all and mark statistical power as limited.
|
||||
|
||||
Use two independent reviewers for the disagreement stratum and a random subset of the agreement stratum. Record reviewer IDs as pseudonymous codes, fixture ID, decision, reason code, time, and adjudication outcome. Do not turn reviewer agreement into a production accuracy claim. Disagreements, ambiguous labels, and unsafe outputs are findings requiring documented adjudication or fixture-policy revision before acceptance.
|
||||
|
||||
## Reporting and sign-off
|
||||
|
||||
The report includes methodology version, scope, fixture manifest/hash, run metadata, metrics, per-class support, raw latency/cache summaries, reproducibility result, human-review sample and adjudication, failed gates, limitations, and sign-off owner/date. Store reports with the pilot artifacts, not in application data or logs containing sensitive values.
|
||||
|
||||
Phase 16 is complete only when the report is reproducible locally, the acceptance gates are evaluated honestly, and all limitations are recorded. It does **not** mean the system is production-ready, scientifically validated, or safe for live prospect decisions.
|
||||
|
||||
## Deployment and remote-push prerequisites remain unresolved
|
||||
|
||||
Phase 16 documentation does not deploy the application or push Git history. Remote publication remains blocked until repository authentication and the intended remote/branch permissions are available. Deployment remains blocked until an operator supplies and verifies the Phase 15 prerequisites: reviewed remote revision, Docker/Compose host access, protected deployment directory, secrets, DNS/TLS/Virtualmin configuration, off-host encrypted backup destination, monitoring, rollback owner, and explicit operational approval. A local benchmark pass, local Compose validation, or local commit must not be reported as a remote push or production deployment.
|
||||
|
||||
## Suggested local evidence commands
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -v -s apps/api/tests -t apps/api
|
||||
python3 -m compileall -q apps/api apps/web
|
||||
docker compose config --quiet
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Record command output and exit status with the benchmark report. These commands validate the local artifact; they do not provide production evidence or resolve the remote push/deployment prerequisites.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Portable deployment and recovery runbook (Phase 15)
|
||||
|
||||
## Configuration
|
||||
|
||||
Copy `.env.example` to an untracked deployment environment file. Production requires a secret-manager supplied `SESSION_SECRET` of at least 32 characters and refuses `AUTOMATED_OUTREACH_ENABLED=true`. Keep bootstrap credentials one-time only; remove and rotate them after provisioning. Never place secrets in images, Compose YAML, logs, backups, or public web roots.
|
||||
|
||||
Validate before startup:
|
||||
|
||||
```sh
|
||||
python3 -c 'from app.config import load_config; load_config()'
|
||||
docker compose config --quiet
|
||||
```
|
||||
|
||||
## Health and readiness
|
||||
|
||||
- `GET /api/v1/health/live` is process liveness and unauthenticated.
|
||||
- `GET /api/v1/health/ready` checks SQLite connectivity and returns HTTP 503 until ready.
|
||||
- `scripts/healthcheck.sh` checks readiness and the outreach safety flag.
|
||||
|
||||
Use readiness for load balancers and container health checks; liveness is only for process supervision.
|
||||
|
||||
## SQLite backup and restore
|
||||
|
||||
Backups are host-side artifacts and never include `.env` or secret files. `backup_sqlite.sh` uses SQLite's online backup API for a consistent snapshot, writes with mode 0600 to a temporary file, atomically renames it, writes a SHA-256 sidecar, and retains only the newest configured count.
|
||||
|
||||
```sh
|
||||
scripts/backup_sqlite.sh /var/lib/prospect-platform/prospects.db /var/backups/prospect-platform 30
|
||||
```
|
||||
|
||||
Before restoring, stop application writes, verify the checksum sidecar, and use the explicit confirmation flag. The script first makes a pre-restore backup, then atomically replaces the target only after an integrity check:
|
||||
|
||||
```sh
|
||||
scripts/restore_sqlite.sh /var/backups/prospect-platform/prospects-<timestamp>.db /var/lib/prospect-platform/prospects.db --confirm-restore
|
||||
```
|
||||
|
||||
Validate backup directory permissions and keep copies encrypted/off-host according to the retention policy. Test restores in an isolated directory quarterly. Never use `docker compose down -v` on a data-bearing installation.
|
||||
|
||||
## Rollback
|
||||
|
||||
`scripts/rollback.sh` is deliberately non-destructive: it prints the approved image/tag or digest rollback procedure and executes no stop, delete, restore, or deployment action. Record old/new image digests, configuration revision, backup/checksum, and health/readiness evidence.
|
||||
|
||||
## systemd / Virtualmin
|
||||
|
||||
`systemd/prospect-api.service.example` is a least-privilege service example. Copy it to a reviewed systemd unit, create `/etc/prospect-platform/prospect.env` with mode 0600, use a dedicated user/data directory, and place TLS/reverse proxying in the Virtualmin-managed web tier. Do not put environment files under `public_html`.
|
||||
|
||||
## Monitoring and incident response
|
||||
|
||||
Monitor readiness failures, restart count, HTTP 5xx rate, SQLite backup age/checksum failures, disk usage, and unexpected outbound traffic. Alert when the latest backup is older than the agreed RPO or when a restore drill fails. Routine logs must not contain passwords, tokens, cookies, API keys, full contact values, or free-text notes.
|
||||
|
||||
On incident: record image/config revision and health state; preserve redacted logs and audit evidence; isolate the service for data loss, unauthorized access, or unexpected outbound traffic; rotate secrets through the secret manager; validate restore/readiness and tenant-scoped reads; then document root cause and retention impact. Outreach remains disabled throughout.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Phase 17 Final Acceptance
|
||||
|
||||
## Decision
|
||||
|
||||
**Local acceptance: PASS. Production deployment: BLOCKED / not attempted.**
|
||||
|
||||
This decision applies to the repository at commit `9622f769776637a40fdae797adccba91445cd351` (branch `main`). The acceptance run validates the local checkout and bounded, offline pilot behavior. It is not evidence that the application was pushed to the remote repository or deployed to a production host.
|
||||
|
||||
## Acceptance matrix
|
||||
|
||||
| Area | Local evidence | Result | Production interpretation |
|
||||
| --- | --- | --- | --- |
|
||||
| Phases 1–4: baseline workflow, auth, jobs/events | API regression suite, tenant/auth/job checks | PASS locally | SQLite and the in-process worker remain pilot-only |
|
||||
| Phase 5: source-adapter safety boundary | Source lifecycle, dry-run, CSV/manual-reference tests | PASS locally | No live source is enabled; terms, approval, rate, retention, and circuit controls remain prerequisites |
|
||||
| Phase 6: normalization and reversible deduplication | Canonicalization, threshold, merge/reversal and tenant tests | PASS locally | Merge permission, full conflict snapshots, concurrency, and production rollback hardening remain open |
|
||||
| Phase 7: domain intelligence | PSL/DNS-state/candidate and no-false-availability tests | PASS locally | No production resolver, cache, or authorized availability provider is enabled |
|
||||
| Phase 8: SSRF-safe website observation | Scanner policy, limits, unsafe-target and history tests | PASS locally | Production egress isolation, DNS-rebinding/redirect testing, durable retention, and monitoring remain open |
|
||||
| Phase 9: official-site contact observation | Extraction, provenance, false-positive and suppression tests | PASS locally | No SMTP probing or outreach; legal, retention, and production isolation gates remain |
|
||||
| Phase 10: scoring | Versioned rules, explanations, recalculation and suppression tests | PASS locally | Production approval lifecycle, durable scheduling, snapshots, and rollback remain open |
|
||||
| Phase 11: operator review workflow | Saved views, review queue, dashboard and bounded bulk-action tests | PASS locally | Production audit completeness, idempotency, per-item outcomes, and queue/count hardening remain open |
|
||||
| Phase 12: CRM and suppression center | Pipeline, interactions, outcomes, reports, and suppression tests | PASS locally | Record-keeping only; no delivery provider or outbound worker exists |
|
||||
| Phase 13: evidence-grounded AI assistance | Provider-disabled, evidence/citation, approval, limits, and isolation tests | PASS locally | Provider/DPA, legal, secret-management, evaluation, retention, and operational approval remain required |
|
||||
| Phase 14: draft-only outreach boundary | Draft/gate/idempotency/no-send tests | PASS locally | Outreach remains disabled; consent/legal basis, provider, delivery feedback, and approval controls are not production-complete |
|
||||
| Phase 15: deployment/readiness/recovery assets | Backup/restore, readiness, config, shell, and safety checks | PASS locally | Host, TLS/DNS/Virtualmin, off-host backups, monitoring, and operator access are unavailable |
|
||||
| Phase 16: deterministic pilot benchmark | `scripts/benchmark_phase16.py --no-latency`; synthetic fixture report | PASS locally | Synthetic metrics do not establish production quality, capacity, availability, or SLOs |
|
||||
| Phase 17: final acceptance | This matrix, bounded capacity smoke, reproducibility and blocker review | PASS locally | Production gate remains blocked until external prerequisites are verified |
|
||||
|
||||
## Bounded local capacity smoke
|
||||
|
||||
The Phase 17 smoke is deterministic, in-memory, and deliberately bounded. It is a guard against accidental unbounded retention, not a load test or capacity claim.
|
||||
|
||||
- Synthetic collection: 1,000 items.
|
||||
- Requested page size: 100; returned items: 100; `has_more=true`.
|
||||
- Synthetic large batch: 5,000 input items; retained items: 100; input was truncated.
|
||||
- Enforced smoke limits: maximum page size 100 and maximum retained batch items 100.
|
||||
- Smoke result: `bounded=true`.
|
||||
|
||||
These figures do **not** measure production throughput, concurrency, queue durability, memory pressure, latency SLOs, availability, or safe operating limits. Production capacity requires a reviewed host profile, representative workload, durable worker/database design, observability, and an approved load-test plan.
|
||||
|
||||
## Reproducibility and verification record
|
||||
|
||||
The final acceptance collector passed with:
|
||||
|
||||
- API suite: **100 tests, exit 0** (`python3 -m unittest discover -v -s apps/api/tests -t apps/api`).
|
||||
- Phase 16 semantic benchmark without variable timing: **passed**.
|
||||
- Python compilation of scripts and API modules: **passed**.
|
||||
- Shell syntax checks for backup, healthcheck, restore, and rollback scripts: **passed**.
|
||||
- JSON validation for the benchmark and acceptance schemas/reports: **passed**.
|
||||
- Compose config validation: **passed** (`docker compose -f docker-compose.yml config --quiet`).
|
||||
- Safety invariants: outreach disabled, no send network path, and tenant routes present: **passed**.
|
||||
- Git state: acceptance head resolved to `9622f769776637a40fdae797adccba91445cd351`; working-tree status was recorded before documentation edits.
|
||||
|
||||
The deterministic benchmark reports 100% normalization accuracy, 100% matching precision/recall, 100% contact precision/recall, 100% website-fixture accuracy, reproducible scoring, and zero tenant leakage. The fixture set is synthetic and small; see `docs/BENCHMARKS.md` for methodology and limitations. Timing observations are local pilot signals only.
|
||||
|
||||
Repeatable local checks from the repository root:
|
||||
|
||||
```sh
|
||||
python3 scripts/final_acceptance.py --output /tmp/prospect-final-acceptance.json
|
||||
python3 scripts/benchmark_phase16.py --no-latency --output /tmp/prospect-phase16.json
|
||||
python3 -m unittest discover -v -s apps/api/tests -t apps/api
|
||||
python3 -m compileall -q apps/api apps/web
|
||||
bash -n scripts/*.sh
|
||||
docker compose config --quiet
|
||||
git diff --check
|
||||
```
|
||||
|
||||
## Blockers and prerequisites
|
||||
|
||||
### Remote publication
|
||||
|
||||
Remote push is blocked by missing repository authentication and unverified permission to push the intended branch. The configured remote is `https://repo.mmcloud.co.za/root/MarketingTool.git`; no push was attempted by this acceptance work. A local commit, passing tests, or a clean diff is not a remote publication. Resolve authentication and branch permission, then verify the remote revision independently.
|
||||
|
||||
### Production deployment
|
||||
|
||||
Deployment is blocked because no production host, Docker/Compose access, protected deployment directory, or deployment operator is available in this run. Before promotion, verify all of the following:
|
||||
|
||||
- reviewed remote revision and release owner;
|
||||
- patched Linux host with Docker Engine/Compose v2, adequate CPU/RAM/disk, firewalling, and restricted Docker access;
|
||||
- Virtualmin or equivalent perimeter, DNS control, HTTPS certificates and renewal monitoring, and private API exposure where appropriate;
|
||||
- protected environment/secret manager for `SESSION_SECRET` and any one-time bootstrap values; remove bootstrap values after provisioning and rotate the password;
|
||||
- encrypted off-host backup destination, checksum/restore drill, retention and legal-hold ownership;
|
||||
- monitoring for health, restarts, resources, auth failures, backups, TLS, migrations, and unexpected egress;
|
||||
- named rollback owner and approval to restore data if compatibility is established;
|
||||
- authenticated tenant-isolation smoke tests and migration validation on the target host.
|
||||
|
||||
### Legal and provider enablement
|
||||
|
||||
Real source, AI, DNS/availability, scanner, or outreach providers are not enabled by this local acceptance. Any future provider requires an allowlisted identity, owner, approved purpose/data class, tenant scope, region/retention terms, rate and cost caps, timeout/retry/circuit policy, secret-manager injection, current product/security/legal approval, terms/DPA review where applicable, and explicit operational enablement. Outreach additionally requires jurisdiction-specific legal review, documented consent or lawful basis, suppression synchronization, human approval, delivery feedback, retention/deletion/legal-hold controls, and a tested kill switch. Until those gates are complete, `AUTOMATED_OUTREACH_ENABLED=false` remains mandatory.
|
||||
|
||||
## Rollback decision
|
||||
|
||||
**Decision: retain the current local revision; do not promote or roll back.** There is no production deployment to undo. If a future promotion fails health, integrity, tenant-isolation, migration, or smoke validation, stop traffic and writes as appropriate, record the image/config/backup revisions, restore the previously verified compatible image/config first, and restore data only after schema compatibility and incident-owner approval. Never run an older binary against an incompatible newer schema, and never use `docker compose down -v` on a data-bearing environment. Re-run liveness/readiness, authenticated isolation, and integrity checks before re-enabling traffic.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
Phase 17 completes the **local acceptance documentation and verification boundary**. It does not claim remote push, host provisioning, credential availability, legal approval, provider enablement, or production deployment. Those are separate external gates and must be recorded as evidence in the release checklist before any production go decision.
|
||||
+149
-1
@@ -69,6 +69,28 @@ Monitor per-tenant and global scan counts, active concurrency, queue age, total/
|
||||
|
||||
Scan history and cache entries must retain normalized URL, redirect chain, observed time, scanner/policy version, applied budgets, freshness/expiry, cache status, and uncertainty reasons, with tenant authorization on every read. Keep retention and size bounded; redact response bodies, secrets, cookies, authorization data, and unnecessary personal data. Invalidate or re-evaluate entries when scanner/DNS policy changes. On SSRF indicators, unexpected egress, repeated budget abuse, or unsafe redirect chains, stop/disable scanning, preserve safe metadata, and follow the incident checklist.
|
||||
|
||||
## Phase 9 official-site contact extraction operations
|
||||
|
||||
Operate contact extraction as passive observation of an approved/public official-site origin, limited to bounded same-site contact/about pages. Before enabling it, verify tenant scope, official-site approval, extractor/policy version, retention class, suppression source, and configured hard limits for pages/URLs, redirects, bytes, time, candidates, and concurrency. Do not add arbitrary URLs or search results to the scope.
|
||||
|
||||
Review each result with its source/page URL and context, extraction method, observed time, confidence/reasons, syntax status, role/free-mail labels, and independent MX/DNS status/freshness. Treat `not_checked`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, and `error` as uncertainty—not invalidity or non-deliverability. Syntax, role, free-mail, and MX labels never authorize contact. Values found in scripts/styles/assets/file names, examples/placeholders, tracking addresses, malformed schemes, or third-party content are false positives and must be excluded or quarantined.
|
||||
|
||||
Verify suppression matching before persistence, response, cache, export, or review-queue insertion. Suppressed contacts remain **do not contact**, regardless of later confidence, classification, syntax, MX, pipeline, or verification changes. Monitor extraction attempts, pages/bytes/candidates, limit hits, blocked destinations, parse errors, false-positive exclusions, suppression matches, cache freshness, retention/deletion jobs, and provenance/audit failures. On a suppression or provenance failure, stop the affected write path and investigate; do not retry blindly.
|
||||
|
||||
There is no SMTP probing, SMTP `VRFY`/`EXPN`, validation email, outreach worker, campaign queue, or follow-up action. Never contact a discovered address. If extraction is disabled, unapproved, out of budget, or uncertain, report deferred/blocked/unknown with the reason. Retain only the minimum value and lineage for the approved retention period; redact addresses and page content from routine logs.
|
||||
|
||||
## Phase 10 scoring operations
|
||||
|
||||
Operate scoring as a versioned policy, not as a mutable numeric field. Before activating a rule set, verify its ID/version, owner/approval, weights, thresholds and priority bands, required evidence, freshness windows, suppression precedence, uncertainty behavior, rounding/tie-breaking, tenant scope, and rollback plan. Record the activation/configuration revision; never edit a rule set already used in production history.
|
||||
|
||||
Review score, priority band, and eligibility separately. A high-priority prospect can still be ineligible or unknown. Suppressed/do-not-contact records are hard blocked. Stale, expired, blocked, partial, missing, or uncertain required evidence must retain its state and reason and must not be silently treated as absent, negative, or current. Monitor counts by band and eligibility state, suppression matches, stale/uncertain outcomes, explanation failures, and unexpected score distribution changes.
|
||||
|
||||
Run recalculation only through an authenticated, tenant-scoped operation with an idempotency key or equivalent safe retry control. For each run record rule-set/algorithm versions, input snapshot or cutoff, actor/job, reason, start/end, processed/succeeded/failed counts, and partial status. Verify before/after score, band, eligibility, and explanation changes for representative records; read back the audit events. Do not report a request as complete merely because a job was accepted, and stop on tenant-scope, suppression, snapshot, or audit failures rather than retrying blindly.
|
||||
|
||||
For a rule or evidence-policy change, use a canary or bounded tenant batch, compare old/new explanations and eligibility, preserve the old version for reproducibility, and document rollback/recalculation scope. Ensure cached/list/detail projections do not mix rule versions. Retain and delete calculation inputs, explanations, and audit records under the approved data policy; do not put full contact values or sensitive evidence in routine logs.
|
||||
|
||||
The current Compose/MVP runtime remains pilot-only until durable rule-set storage/approval, scheduled recalculation with worker leases, complete audit/readback, and tenant-isolation and stale/uncertain regression checks are operationally verified.
|
||||
|
||||
## Phase 4 jobs and live logging
|
||||
|
||||
The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.
|
||||
@@ -77,6 +99,28 @@ Creation must use a tenant-scoped idempotency key and request fingerprint. A rep
|
||||
|
||||
There is no SSE handler, durable queue, scan worker/isolation boundary, or worker process in Compose; its in-process worker and SQLite job/event tables are pilot-only. Process loss can lose work, there is no durable lease/recovery or horizontal coordination, and it must not be presented as production execution. Redis and Celery are not implemented.
|
||||
|
||||
## Phase 11 dashboard and review workflow operations
|
||||
|
||||
Operate saved filters as tenant-scoped configuration. Before exposing save/load/update/delete, verify the filter schema, server-side bounds, owner/share permissions, stable ordering, page-size maximum, retention/deletion behavior, and audit event format. A filter is not a permission grant; never troubleshoot a missing record by removing tenant or eligibility predicates.
|
||||
|
||||
The review queue must display the exact filter/predicate snapshot, whether each count is for the current page or the full matching set, and pagination/cursor state. Re-check tenant scope and current suppression/merge state when a row is opened or acted on. Suppressed/do-not-contact records remain blocked; merged/non-active records are not merge-eligible. A clickable count is safe only when it reuses the same bounded, tenant-scoped predicate; do not interpret stale, loading, or failed counts as zero.
|
||||
|
||||
For each bulk operation, verify the server-reported selection size and maximum, inspect the preview, obtain explicit operator confirmation, and ensure an idempotency key/correlation ID is recorded. The server must re-check permissions, tenant ownership, suppression, and merge eligibility at execution. Read back the per-record outcomes and audit event; investigate skipped/failed records and partial completion rather than retrying blindly. Never run an unbounded “all results” action from a count, and never describe bulk review as outreach or automatic deduplication.
|
||||
|
||||
Monitor saved-filter errors, queue count freshness, pagination/cursor failures, cross-tenant denials, suppression/eligibility skips, batch-limit violations, idempotency conflicts, partial bulk failures, audit append/readback failures, and merge snapshot/reversal outcomes. Preserve filter/selection snapshots or safe hashes and bounded totals in operational records, but redact secrets and unnecessary contact data. The current Compose/MVP runtime includes saved-filter creation/listing, a bounded review queue, clickable dashboard metadata, and explicit verify/reject/assign bulk review actions. It remains pilot-only: the remaining Phase 11 limitations are that update/delete saved-filter routes are not wired, queue counts lack complete matching-set/predicate snapshots, bulk operations have no preview/idempotency/per-record result contract, and the audit trail records a batch aggregate. Treat these as release blockers until the API/UI slices are hardened and verified.
|
||||
|
||||
## Phase 12 CRM operations
|
||||
|
||||
Operate CRM as human review and record-keeping, not outbound engagement. Before enabling the slice, verify the tenant/role matrix, canonical transition table (`new` → `contacted` → `qualified` → `proposal` → `negotiation` → `won`/`lost`, plus any explicitly configured paused/disqualified and reopen rules), append-only interaction policy, normalized outcome vocabulary, batch/report limits, suppression source, and retention class. Keep `AUTOMATED_OUTREACH_ENABLED=false` and verify there is no delivery provider, campaign queue, SMTP probe, or follow-up worker.
|
||||
|
||||
For pipeline changes, inspect the server response and audit event, including actor, before/after state, reason, timestamp, and correlation/idempotency ID. Reject direct jumps, edits to history, and actions on merged/inactive records. A same-state retry may be treated as idempotent; a reopen must be a new reasoned event. Adding an interaction does not advance a stage. For outcomes, use only `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, or `other`; preserve `other` as an explicit catch-all and treat `do_not_contact` as a separate immediate hard block.
|
||||
|
||||
Run reports with an explicit bounded date interval, `as_of`, timezone, filter snapshot, and requested metric semantics. Confirm whether totals are latest-state, event-time, distinct-business, page, or matching-set counts. Check freshness and partial/error status before distributing a report; never interpret a page count as a tenant total or a report as authorization. Report and export jobs must be tenant-scoped, idempotent where they have side effects, audited, and redacted.
|
||||
|
||||
Operate the suppression center as the final deny gate. Verify normalized email/domain/phone matching before CRM writes, responses, caches, exports, reports, and any queue. Investigate any record that is not visibly marked **Do not contact** after a match; stop the affected write/report path rather than retrying blindly. Unsuppression/removal requires an authorized reason and audit readback. Retain suppression provenance and history even when the underlying contact is deleted, subject to the approved legal/retention policy.
|
||||
|
||||
Monitor transition rejection and conflict rates, interaction/outcome write and correction failures, unknown outcomes, suppression matches and attempted bypasses, report freshness/partial failures, export denials, idempotency conflicts, cross-tenant denials, audit append/readback failures, and retention/deletion job results. Routine logs must contain no secrets, full contact values, or unnecessary free text. The current Compose/MVP remains pilot-only until durable CRM migrations, retention jobs, reproducible reports, integration tests, and recovery procedures are verified.
|
||||
|
||||
## Configuration and deployment
|
||||
|
||||
Copy `.env.example` for local development. Production values must be supplied by the deployment environment, never committed. `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` are optional API environment variables for first-run admin provisioning only; set them together through a secret store or protected deployment environment, remove them immediately after successful bootstrap, and rotate the password. Do not put real values in Compose files, CI variables visible to logs, images, or committed `.env` files.
|
||||
@@ -90,7 +134,8 @@ Before deployment:
|
||||
3. Restrict host/network exposure at the ingress/firewall.
|
||||
4. Verify both unauthenticated health checks and review logs for unexpected errors, cross-tenant errors, or sensitive data.
|
||||
5. Exercise tenant-scoped list/detail/child routes with bounded pagination and filters, and verify that notes/pipeline changes appear in the intended tenant's audit trail only.
|
||||
6. Record the image digest and configuration revision for rollback.
|
||||
6. If Phase 9 is enabled, run official-site fixtures covering provenance/confidence, role and free-mail labels, syntax failures, every MX/DNS uncertainty state, false-positive exclusions, limit exhaustion, retention/deletion, and suppression-before-persistence/response/export. Confirm no SMTP or outreach network activity.
|
||||
7. Record the image digest and configuration revision for rollback.
|
||||
|
||||
## Data, backups, and retention
|
||||
|
||||
@@ -127,6 +172,30 @@ Do not run `docker compose down -v` on a data-bearing environment: it removes th
|
||||
|
||||
Before production, complete a migration from SQLite to a reviewed production database, add schema/indexes for jobs/idempotency/events, domain observations, and scan history/cache, implement transactional sequence assignment and tenant authorization, and prove cancellation/retry/lease recovery under concurrency. Add durable queue/worker and scanner-isolation operations, bounded DNS/PSL/website processing, TTL/freshness-aware cache invalidation, SSRF/DNS-rebinding/redirect-chain tests, hard size/time/crawl budgets, uncertainty and association-review workflows, and a separately approved availability provider. Add metrics and alerts for queue age, failures, retries, cancellation latency, event lag/gaps, DNS/scanner status/error rates, cache freshness, blocked destinations, crawl-budget exhaustion, provider rate limits/circuit state, and SSE connections; define backup/restore and event-retention drills. Redis, Celery, Postgres, schedulers, discovery adapters, and production scanners are possible future components—not implicit Compose dependencies. No automated discovery, domain acquisition, ownership assertion, or outreach may be inferred from the scaling path.
|
||||
|
||||
## Phase 13 optional AI assistance operations
|
||||
|
||||
Keep AI disabled unless the provider registry, tenant scope, purpose, data class, redaction policy, retention class, rate/token/cost budgets, approval expiry, and operational enablement have been reviewed and recorded. The current Compose stack has no configured provider; do not enable one by adding an arbitrary URL or secret. Store credentials only in the deployment secret manager. A fallback must be pre-approved for the same purpose and input class and must inherit the primary provider's tenant, citation, redaction, retention, and authority constraints.
|
||||
|
||||
Before an AI request, verify the authenticated tenant and permission, active provider/policy version, suppression state, and bounded evidence selection. Minimize and redact inputs; exclude secrets, credentials, session data, raw page bodies, unrelated personal data, and unnecessary full contact values. Monitor request/response size, latency, provider health, fallback rate, quota/rate/cost usage, redaction failures, policy/approval denials, and `unknown`/`unavailable`/partial outcomes. Provider failure, stale or conflicting evidence, prompt-injection indicators, or hash/citation mismatch is a safe non-result—not a retry reason and never permission to present invented facts.
|
||||
|
||||
Review every suggestion as **AI draft — human review required**. Confirm citations resolve within the same tenant, the evidence hash still matches the cited snapshot, observed times/freshness are acceptable, suppression remains clear, and the provider/policy approval is current. An approval must be explicit, reasoned, audited, and read back; changed evidence or policy invalidates the proposal. Rejection, expiry, failed approval, and fallback events must remain auditable. Never report generation or approval request acceptance as persistence or completion.
|
||||
|
||||
The operational path must not alter pipeline/interactions/outcomes as if communication occurred, send outreach, probe SMTP, create campaigns, schedule follow-ups, merge records, acquire domains, or otherwise act autonomously in CRM. Suppressed/do-not-contact records remain visible for safety review and blocked from contact-related actions. Retain only the approved minimum AI lineage (redacted fingerprint, output, citations/hash, versions, approval and audit metadata); apply deletion/legal-hold rules to prompts, outputs, evidence snapshots, caches, and logs and verify deletion without removing required suppression/audit history.
|
||||
|
||||
On suspected provider misuse, data leakage, hallucinated/invented facts, prompt injection, cross-tenant exposure, unexpected outbound traffic, cost runaway, or suppression bypass: disable the AI capability/kill switch, stop affected jobs, preserve redacted evidence and audit metadata, revoke/rotate provider credentials, determine affected tenants and retention obligations, and require security/product/legal review before re-enabling. Production remains blocked until provider contracts/DPA, tenant-isolation, redaction, citation/hash, approval/rollback, evaluation, retention/deletion, monitoring, and recovery tests pass.
|
||||
|
||||
## Phase 14 draft-only outreach operations
|
||||
|
||||
Treat Phase 14 as preparation, never as delivery. Verify `AUTOMATED_OUTREACH_ENABLED=false` in the rendered Compose configuration and confirm there is no provider credential, send route, delivery worker, campaign scheduler, SMTP probe, validation-mail path, or follow-up job. A draft or approval request is not a send and must not be reported as completion.
|
||||
|
||||
Before any future enablement, record the provider allowlist entry, purpose/capability, tenant/data class, processing region/retention terms, timeout and payload bounds, per-tenant/global request/message/cost caps, health/circuit thresholds, approval owner/expiry, legal/consent policy version, and separate operational enablement. Load credentials only through the deployment secret manager; remove bootstrap values after provisioning and never print secrets in logs. A fallback is permitted only when separately approved for the same purpose and constraints.
|
||||
|
||||
For each draft, inspect tenant and recipient scope, normalized suppression/do-not-contact, consent or documented legal basis, jurisdiction/channel policy, evidence IDs/citations and exact snapshot hash, observed/freshness times, uncertainty/conflicts, provider/policy versions, and approval status/expiry. Suppression is the final deny gate. Require an authorized human to approve the unchanged draft version and evidence hash; edits, stale/changed evidence, expired approval, provider/policy changes, or uncertain gates require re-review. Read back the approval audit event. Never use a high score, pipeline state, public address, or AI citation as consent or permission to contact.
|
||||
|
||||
For any future side effect, require a tenant-scoped idempotency key and request fingerprint bound to draft/provider/policy versions. Verify exact replay returns the original result and conflicting reuse is rejected. Enforce caps before provider calls and across retries, fallbacks, workers, and tenants; use bounded backoff and circuit breaking. Monitor gate denials, suppression matches/bypass attempts, approval expiry, citation/hash mismatches, provider latency/errors/quota, cap consumption, retries, duplicate/conflicting keys, and audit append/readback failures. Record bounded per-item outcomes and never infer delivery from HTTP acceptance.
|
||||
|
||||
On suspected suppression bypass, invalid consent/legal basis, cross-tenant exposure, secret leakage, unexpected outbound traffic, provider misuse, cost runaway, citation/hash mismatch, or audit gap: disable the capability/kill switch, stop affected jobs, preserve redacted evidence and audit metadata, revoke/rotate credentials, identify affected tenants and legal-retention obligations, and require product/legal/security review before re-enabling. Production remains blocked until provider/DPA and legal review, durable approval/idempotency/audit, suppression synchronization, delivery feedback, retention/deletion/legal-hold verification, rollback, and no-send integration tests pass. The current Compose/MVP has none of these live outreach components.
|
||||
|
||||
## Incident checklist
|
||||
|
||||
1. Record time, affected service, image/config revision, and observed health state.
|
||||
@@ -135,3 +204,82 @@ Before production, complete a migration from SQLite to a reviewed production dat
|
||||
4. Rotate exposed credentials through the secret manager.
|
||||
5. Validate recovery with health checks and a targeted tenant-isolation/detail smoke test.
|
||||
6. Document root cause, corrective action, and any retention/suppression or audit impact.
|
||||
|
||||
## Phase 15 portable operations
|
||||
|
||||
See `docs/DEPLOYMENT.md` for the production-safe environment contract, `/api/v1/health/ready` readiness semantics, atomic SQLite backup/restore procedures, checksum/retention policy, non-destructive rollback guidance, monitoring references, and the systemd/Virtualmin-compatible service example. Run `scripts/healthcheck.sh` for an operator-safe readiness probe. Backups are host-side and must remain encrypted/off-host; never include `.env` or secret-manager material.
|
||||
|
||||
## Phase 15 production deployment runbook
|
||||
|
||||
### Prerequisites and Virtualmin layout
|
||||
|
||||
Use a dedicated, patched Linux VPS with Docker Engine and Compose v2, adequate disk/RAM/CPU, host firewalling, DNS control, HTTPS certificates with renewal monitoring, and an encrypted off-host backup destination. Virtualmin may host the domain and terminate TLS/reverse-proxy to Compose, but it does not replace Docker health checks, application authorization, backups, or monitoring. Keep the checkout and `.env` outside public web roots with restrictive permissions; expose only the reverse proxy publicly and keep the API binding private where the topology permits.
|
||||
|
||||
### Configuration and first bootstrap
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
chmod 600 .env
|
||||
docker compose config --quiet
|
||||
docker compose up --build -d
|
||||
docker compose ps
|
||||
curl -fsS https://example.invalid/healthz
|
||||
curl -fsS http://127.0.0.1:8000/api/v1/health/live
|
||||
```
|
||||
|
||||
Replace the example hostname with the real HTTPS origin. Supply secrets through the protected deployment environment/secret manager, not shell history or committed files. Set `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWORD` only for a fresh instance, confirm authenticated login, then remove both values and rotate the password. Verify the rendered config still contains `AUTOMATED_OUTREACH_ENABLED=false`; never override it as a routine deployment action.
|
||||
|
||||
### Readiness, monitoring, and release evidence
|
||||
|
||||
The API `/api/v1/health/live` is a liveness check and `/api/v1/health/ready` checks SQLite readiness; web `/healthz` is a liveness check. These endpoints do not prove backups, workers, migrations, or external dependencies, so readiness remains an operator gate: both services must report Compose `healthy`, HTTPS must reach the expected containers, authenticated tenant-scoped smoke tests must pass, and migration validation must be recorded. Monitor container health/restarts, CPU/RAM/disk and `/data` pressure, API latency/error rates, auth failures, backup age/failures, TLS expiry, and unexpected egress. Never log secrets, cookies, full contact values, or request bodies. Record the commit, image digests, rendered non-secret configuration fingerprint, schema/migration result, backup ID, and approver.
|
||||
|
||||
### Backup, restore, retention, and migration validation
|
||||
|
||||
The named volume `prospect-platform-api-data` is live state, not a backup. Before a release or schema change, quiesce writes, take an encrypted backup to an off-host/isolated destination, verify its checksum/manifest, and restore it into a disposable isolated volume. Run `PRAGMA integrity_check`, foreign-key checks, representative tenant-scoped API reads, row-count checks, and the API test suite against the restored copy. Record the result and retain the previous image/config. Apply documented retention to SQLite data, audit/source lineage, operational logs, and backup generations; honor legal holds and verify deletion jobs where present. Do not use `docker compose down -v` on a data-bearing environment.
|
||||
|
||||
There is currently no standalone migration or backup CLI. `schema.sql` is applied by the API startup and additive compatibility behavior is in application code; therefore every schema change requires a reviewed backup-first procedure and isolated restore test. Do not assume startup success means migration success. Stop and roll back the release if integrity, tenant isolation, health, or smoke validation fails.
|
||||
|
||||
### Rollback
|
||||
|
||||
1. Stop promotion and record symptoms, health, commit/image/config revisions, and backup ID.
|
||||
2. Disable the affected Virtualmin route or put the site in maintenance mode; stop writes if data integrity is in doubt.
|
||||
3. Re-deploy the previously verified image pair and exact configuration. Do not run an older binary against a schema it cannot read.
|
||||
4. Re-run health, authenticated tenant-isolation smoke tests, and read-only integrity checks.
|
||||
5. Restore the database only when the backup/schema compatibility is verified and an incident owner approves it; otherwise preserve the newer data and perform forward repair.
|
||||
6. Re-enable traffic only after monitoring is green, then document root cause, retention/legal impact, and follow-up migration work.
|
||||
|
||||
### Explicit limitations
|
||||
|
||||
This repository does not provision Virtualmin/TLS/DNS, provide a dependency-aware readiness service beyond the API's SQLite check, durable migration runner, PITR, HA database, durable queue/worker leases, production egress proxy, or compliance-grade retention service. SQLite and the in-process worker are pilot-only. The release has no outbound provider/send path and must remain outbound-disabled by default. See `docs/RELEASE_CHECKLIST.md` for the short go/no-go gate.
|
||||
|
||||
## Phase 16 pilot benchmark operations
|
||||
|
||||
Run the Phase 16 benchmark only against the versioned, labeled synthetic fixture manifest described in `docs/BENCHMARKS.md`. Confirm that fixtures contain no live personal data, secrets, uncontrolled network responses, or production records. Keep expected labels separate from generated outputs, and record the manifest/hash, label-policy version, commit, clean/dirty state, algorithm and rule-set versions, runtime/dependency/image digests, non-secret configuration fingerprint, seed, locale/timezone, rounding/tie-breaking, host profile, command, and timestamps.
|
||||
|
||||
For each deterministic classifier/scorer, run every fixture at least twice in fresh processes. Compare class, score, band, eligibility, uncertainty reasons, explanation factors, and version metadata; any unexplained difference is a reproducibility failure. Report confusion matrices, support, precision and recall with zero-denominator `N/A`, plus unknown/blocked/partial/error and suppression outcomes separately. Do not tune thresholds, remove hard fixtures, or convert `N/A` into a pass after seeing results without publishing a new methodology version.
|
||||
|
||||
Measure latency after a documented warm-up, using monotonic timestamps and a fixed local, network-free workload unless separate approval exists. Separate cold, warm, bypass, and not-applicable cache states. Record every sample, including errors/timeouts, and report count, minimum, median, p95, p99, maximum, timeout/error rate, cache hits/misses, freshness/expiry, and invalidation behavior. State whether startup, serialization, database, and network time are included. Never present a cache hit as a fresh observation or infer production capacity/SLOs from pilot timings.
|
||||
|
||||
Before sign-off, evaluate the synthetic-only gates: 100% deterministic output equality; precision `>=0.90` and recall `>=0.85` only when positive support is at least 20; 100% safety preservation for suppressed, blocked, malformed, and explicitly unknown fixtures; 100% cross-tenant denial/non-disclosure probes; and complete latency sample/cache labels. Conduct blinded, seeded human review stratified by predicted class, uncertainty/suppression state, and FP/FN disagreement, reviewing at least 20 fixtures or 10% (whichever is larger, subject to the declared cap). Use pseudonymous reviewer IDs, reason codes, independent review for disagreements, and documented adjudication. A passed pilot gate is not production evidence.
|
||||
|
||||
Store the report and raw machine-readable samples with the pilot artifacts, including failed/incomplete runs and limitations. Do not place sensitive fixture values in routine logs. Phase 16 does not enable a worker, live source, or outreach path.
|
||||
|
||||
### Unresolved publication and deployment prerequisites
|
||||
|
||||
Do not report local benchmark or Compose success as deployment. Remote push remains blocked until repository authentication and intended remote/branch permission are supplied. Production deployment remains blocked until the Phase 15 operator prerequisites are verified: reviewed remote revision, Docker/Compose host access, protected deployment directory, secret injection, DNS/TLS/Virtualmin, encrypted off-host backup and restore evidence, monitoring, rollback owner, and operational approval. Record the blocker in the release record and keep the current outbound-disabled configuration.
|
||||
|
||||
## Phase 17 final acceptance and capacity smoke
|
||||
|
||||
The Phase 17 decision is **local acceptance PASS; production deployment BLOCKED and not attempted**. Evidence is collected at local commit `9622f769776637a40fdae797adccba91445cd351`; see `docs/FINAL_ACCEPTANCE.md` for the matrix and release record. The final collector passed 100 API tests, the no-latency Phase 16 benchmark, Python compilation, shell syntax, JSON validation, Compose config, Git state, and safety invariants.
|
||||
|
||||
The capacity smoke is intentionally deterministic and in-memory: 1,000 synthetic items with page size 100 returned 100 and `has_more=true`; a 5,000-item synthetic batch retained 100 and marked truncation. It verifies bounds only. Do not use it to size production hosts, infer throughput/concurrency, set SLOs, or claim durability/availability. A production capacity exercise requires an approved representative workload, host profile, observability, and durable database/worker design.
|
||||
|
||||
For reproducibility, retain the commit, fixture/manifest hash, algorithm and rule-set versions, runtime/dependency/image versions, non-secret configuration fingerprint, seed, locale/timezone, rounding/tie-breaking, command, host profile, timestamps, and raw results. Repeat deterministic fixtures in fresh processes and compare serialized output fields exactly. Keep latency/cache results separate from deterministic acceptance and label cache hits as non-fresh observations.
|
||||
|
||||
## Phase 17 release blockers and rollback decision
|
||||
|
||||
Remote publication remains blocked until repository authentication and intended remote/branch permission are available; no push was attempted. Production remains blocked until a reviewed remote revision, host and Docker/Compose access, protected deployment directory, secret manager, DNS/TLS/Virtualmin perimeter, encrypted off-host backup and restore drill, monitoring, rollback owner, and operational approval are verified. Real source, AI, DNS/availability, scanner, or outreach providers additionally require allowlisting, terms/DPA and legal review, data/retention policy, consent or lawful-basis approval where applicable, rate/cost/circuit controls, and explicit operational enablement. Keep outreach disabled.
|
||||
|
||||
**Rollback decision:** retain the local revision; there is no production deployment to undo. If a future promotion fails health, integrity, migration, tenant-isolation, or smoke checks, stop promotion/traffic and writes as needed, restore the previously verified compatible image/config first, and restore data only after compatibility and incident-owner approval. Re-run health, integrity, and authenticated isolation checks before reopening traffic. Never use `docker compose down -v` on a data-bearing environment.
|
||||
|
||||
These are separate states: a passing local acceptance run is not a remote push, and a remote push is not a production deployment.
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Phase 15 release checklist
|
||||
|
||||
Use this checklist for a Virtualmin-managed VPS deployment. A checked item is evidence, not an assumption; record the owner, timestamp, commit, image digests, and backup ID in the release record.
|
||||
|
||||
## Go/no-go prerequisites
|
||||
|
||||
- [ ] Reviewed commit and release owner approved; previous image pair and configuration revision are retained for rollback.
|
||||
- [ ] Patched Linux host has Docker Engine and Compose v2, sufficient CPU/RAM/disk, restricted Docker access, host firewalling, and a protected deployment directory outside public web roots.
|
||||
- [ ] Virtualmin/DNS points to the host; HTTPS certificate, renewal monitoring, reverse proxy, and maintenance route are tested.
|
||||
- [ ] API exposure is restricted to the required private path/host; only the intended web entry point is public.
|
||||
- [ ] Production environment values are injected from a protected secret store/environment. `.env` is untracked, mode `0600`, and contains no committed or logged secrets.
|
||||
- [ ] Bootstrap admin values, if needed, are supplied together, used once, removed immediately, and the password is rotated. `AUTOMATED_OUTREACH_ENABLED=false` is verified in rendered Compose.
|
||||
- [ ] Encrypted off-host backup destination, retention schedule, legal-hold owner, monitoring destination, and incident/rollback owner are confirmed.
|
||||
|
||||
## Validate, deploy, and verify
|
||||
|
||||
- [ ] `docker compose config --quiet` passes; rendered configuration was reviewed without exposing secret values.
|
||||
- [ ] Images build from the reviewed commit, are scanned, and their digests are recorded.
|
||||
- [ ] Backup is taken before release/schema change; checksum/manifest is verified.
|
||||
- [ ] Backup restores into an isolated volume/environment; `PRAGMA integrity_check`, foreign-key checks, representative row counts, tenant-scoped reads, and API tests pass. Schema/migration result is recorded.
|
||||
- [ ] `docker compose up -d` completes and both services report `healthy`; running is not accepted as ready.
|
||||
- [ ] `curl -fsS https://<host>/healthz` and the API liveness/readiness endpoints pass (`/api/v1/health/live`, `/api/v1/health/ready`). These checks do not replace authenticated smoke tests.
|
||||
- [ ] Authenticated smoke tests cover login, tenant-scoped list/detail/child access, a safe mutation/audit readback, and cross-tenant non-disclosure.
|
||||
- [ ] Monitoring sees health/restarts, API errors/latency, disk and `/data` pressure, auth failures, backup age/failure, TLS expiry, migration failures, and unexpected egress without collecting secrets or full contact data.
|
||||
- [ ] No outbound provider/send/SMTP activity is present; unexpected egress is treated as an incident.
|
||||
|
||||
## Retain and sign off
|
||||
|
||||
- [ ] Release record contains commit, image digests, non-secret config fingerprint, schema/migration result, backup ID, test output, approver, and rollback decision.
|
||||
- [ ] Data, audit/source lineage, logs, and backup retention/deletion rules are applied; legal holds are preserved.
|
||||
- [ ] Rollback path was reviewed: restore the prior compatible image/config first, stop writes if needed, and restore data only after compatibility approval. Do not use `docker compose down -v` on a data-bearing host.
|
||||
- [ ] Known limitations are accepted explicitly: no Virtualmin/TLS/DNS provisioning, only SQLite-level readiness, no standalone migration/backup CLI, SQLite/in-process worker only, no PITR/HA, and no production egress isolation.
|
||||
@@ -51,6 +51,18 @@ No Phase 7 resolver, cache, or availability provider is enabled in the current C
|
||||
|
||||
No production-grade scanner egress proxy, isolated worker, or durable scan store is supplied by the current Compose runtime. Before enabling scanning in production, add SSRF/DNS-rebinding/redirect-chain tests, egress deny-by-default policy, abuse/rate controls, authenticated history authorization, retention/deletion, monitoring, and incident procedures. Scans must never cause form submission, acquisition, verification, enrichment, or outreach.
|
||||
|
||||
## Phase 9 public official-site contact extraction controls
|
||||
|
||||
- Extraction is passive and suggestion-only. Scope it to an explicitly approved/public official-site origin and bounded same-site contact/about pages; never use it as general search, unrestricted crawling, identity verification, or enrichment.
|
||||
- Apply the Phase 8 SSRF, redirect, content-type, timeout, byte, page, URL, candidate, and concurrency limits. Fail closed on disabled/unapproved scope, limit exhaustion, partial content, blocked/unsafe destinations, or resolver errors, with an explicit status and uncertainty reason rather than an empty success.
|
||||
- Parse only permitted public HTML/visible contact values and `mailto:` links. Never submit forms, execute JavaScript, send credentials/cookies, probe SMTP or SMTP `VRFY`/`EXPN`, send validation email, or make any outbound contact. Treat HTML, attributes, scripts, and extracted text as untrusted input.
|
||||
- Retain provenance for every candidate: source/page URL and context, extraction method, observed time, extractor/policy version, confidence algorithm/version, and uncertainty reasons. Confidence is a triage signal—not ownership, consent, deliverability, or contact permission.
|
||||
- Keep `syntax_valid`/`syntax_invalid` separate from role classification and free-mail classification. Role/person/unknown and free-mail/business-domain/unknown are labels only. MX/DNS must remain independently uncertain with resolver/source, observed time, TTL/freshness, and explicit `not_checked`, `resolved`, `nxdomain`, `no_data`, `timeout`, `servfail`, `blocked`, or `error` states; no MX outcome proves deliverability.
|
||||
- Exclude asset/file-name addresses, script/style/comment text, example/test/placeholder domains, tracking/telemetry addresses, malformed schemes, and unrelated third-party content. Suppression matching must happen before persistence, response, export, cache, or review queue insertion; a match is immutable do-not-contact until an authorized suppression change, and suppression always wins.
|
||||
- Minimize and protect contact values and lineage. Bound retention for raw/extracted values, provenance, MX/DNS observations, caches, and audit records; redact full addresses and page content from logs where a safe hash/identifier suffices. Tenant predicates and deletion handling apply to every result, cache, export, and audit read.
|
||||
|
||||
No SMTP probing or outreach capability is permitted by this phase. A public address remains an unverified observation requiring human review and separate future product/legal/security approval before any contact workflow could exist.
|
||||
|
||||
## Phase 5 source security controls
|
||||
|
||||
Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
|
||||
@@ -64,6 +76,53 @@ Source adapters are a security boundary, not a generic fetch facility. Registry
|
||||
|
||||
If a future approved adapter fetches URLs, apply the SSRF requirements below in addition to source approval. Network discovery is not implemented by this documentation or by the current Compose stack.
|
||||
|
||||
## Phase 10 scoring security controls
|
||||
|
||||
- Treat score, priority, and eligibility as separate security-relevant outputs. A score or priority band is ranking metadata only and must never authorize contact, export, enrichment, or another side effect.
|
||||
- Rule sets must be named, versioned, tenant-scoped, explicitly approved/activated, and immutable once used for a calculation. Store weights, thresholds, required signals, freshness windows, suppression precedence, algorithm version, and deterministic rounding/tie-breaking; do not permit clients to submit or override them.
|
||||
- Make every result reproducible from a tenant-scoped input/evidence snapshot, normalized values, rule-set/version, algorithm/version, and calculation timestamp/freshness context. Explanations must identify contributing factors, points/weights, exclusions, evidence references, and uncertainty/staleness reasons without leaking another tenant's data or unnecessary personal data.
|
||||
- Evaluate eligibility independently and fail closed. Suppression/do-not-contact always yields ineligible and remains visible; stale, expired, missing, blocked, partial, or uncertain required evidence must be explicit and cannot be silently treated as zero, false, or positive. Never let recalculation revive a suppressed value.
|
||||
- Recalculation must be authenticated, authorized, tenant-scoped, idempotent or safely retryable, and auditable. Record actor/job, rule-set and input versions, request reason, start/end, before/after outputs, explanation changes, counts, failures, and partial/incomplete status. Preserve prior results and audit history; do not rewrite history in place.
|
||||
- Protect rule-set, explanation, recalculation, and audit reads with the same organization predicate as business data. Cross-tenant rule IDs, job IDs, evidence references, and business IDs must not disclose existence. Ensure background workers carry tenant context and cannot process an unscoped batch.
|
||||
|
||||
Phase 10 is not production-ready until rule-set lifecycle permissions/approval, immutable snapshots, audit tamper resistance, retention/deletion policy, concurrency/rollback behavior, and regression tests for suppression precedence, stale/uncertain handling, replay/reproducibility, and tenant isolation are complete.
|
||||
|
||||
## Phase 11 dashboard and review workflow security controls
|
||||
|
||||
- Saved filters are tenant-owned data, not authorization. Validate and bound every predicate server-side; scope list/read/write/delete by the authenticated organization, reject unknown fields and unbounded limits, and keep filter definitions free of secrets. Any shared filter must remain within the tenant with explicit read/write permissions.
|
||||
- Review-queue membership and counts must be derived under the caller's tenant and permission predicate. A count, filter ID, cursor, or selected business ID must never grant access. Suppression/do-not-contact always overrides score, review status, or bulk selection; merged/non-active records are not merge-eligible.
|
||||
- Bulk actions require a server-enforced batch maximum, explicit selection or immutable filter snapshot, preview plus confirmation, authorization and state re-checks at execution, and idempotent/safely retryable behavior. Return bounded per-record outcomes and fail closed on cross-tenant, suppressed, stale, or ineligible records. Bulk review actions must not become an outreach or automatic-merge path.
|
||||
- Clickable counts must preserve the exact tenant-scoped predicate used to calculate them and clearly distinguish page counts from full matching-set counts. Never use a client-side count as a security decision; loading/error/unavailable must not collapse to zero and leak scope or state.
|
||||
- Audit every saved-filter mutation, queue assignment/decision, bulk preview/confirmation/execution, skip/failure, suppression/eligibility decision, and merge/reversal. Record tenant, actor, action, timestamp, filter/selection snapshot or safe hash, bounded counts, per-item outcomes, policy/version, and correlation/idempotency ID. Protect audit reads with the same tenant predicate, redact secrets/full contact values where unnecessary, and define retention/deletion and tamper-evidence before production.
|
||||
|
||||
Phase 11 is present in the current Compose source with durable saved-filter storage, a bounded review queue, clickable dashboard metadata, and explicit bulk review actions. It is not production-ready: saved-filter update/delete routes are not wired, queue/count responses lack complete predicate and matching-set semantics, bulk actions lack preview/idempotency/per-item outcomes, and audit coverage is aggregate for a batch. Do not infer stronger guarantees from the UI. Before release, add cross-tenant, suppression-precedence, merge-eligibility, count-scope, replay/idempotency, partial-failure, and audit-completeness tests.
|
||||
|
||||
## Phase 12 CRM security controls
|
||||
|
||||
- Treat pipeline state, interaction history, outcomes, reports, exports, and suppressions as tenant data. Enforce `organization_id` on every query, join, cache key, background job, report, and export; cross-tenant identifiers must not disclose existence.
|
||||
- Validate pipeline transitions server-side against the canonical lifecycle `new` → `contacted` → `qualified` → `proposal` → `negotiation` → `won`/`lost`; any paused/disqualified state must be explicitly configured, reasoned, and audited before use or reopening. Reject direct jumps, client-submitted history, edits to historical events, same-record mutations after merge/inactivation, and unaudited state changes. Same-state retries must be idempotent.
|
||||
- Keep interactions append-only and bounded. Record actor, channel, business/contact reference, occurred and recorded times, provenance, safe redacted summary, outcome, and correlation/idempotency lineage. The normalized outcome set is `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, or `other`; `other` must not be treated as success or failure. Corrections append a superseding event and preserve the original; free text is untrusted input and must be size-limited and escaped.
|
||||
- Normalize outcomes into `connected`, `no_answer`, `left_message`, `meeting_booked`, `meeting_held`, `qualified`, `disqualified`, `won`, `lost`, and `other`. Treat `other` as an explicit catch-all, not success or failure. `do_not_contact` is a separate unconditional deny state and cannot be overridden by a later outcome, score, stage, verification, or client payload.
|
||||
- Define report semantics explicitly: bounded date range, timezone, `as_of`, freshness, filter snapshot, latest-state versus event-time aggregation, distinct-business versus event counts, and treatment of suppressed/merged/inactive/unknown records. Page/matching counts are not authorization. Tenant-key report caches and exports, authorize them independently, and redact contact values/free text.
|
||||
- Apply suppression before persistence, response, cache, report eligibility, export, queueing, or any future side effect. Normalize email/domain/phone matching server-side; retain source, reason, scope, actor, and effective timestamps. Unsuppression/removal requires authorization, reason, audit, and re-evaluation. Preserve suppressed records as visible safety state rather than silently deleting them.
|
||||
- Audit every transition, interaction/outcome write or correction, suppression decision/change, report/export request and result, including before/after or bounded result, actor/tenant, time, policy/version, correlation/idempotency ID, and safe reason. Protect audit history from ordinary edits and apply explicit retention, deletion, and legal-hold rules.
|
||||
- Outreach remains prohibited: no send endpoint, SMTP probing, validation mail, campaign, delivery scheduler, automated follow-up, or consent inference. Any future outreach requires separate product/legal/security approval, deny-by-default configuration, rate/abuse controls, suppression re-checks, and independent audit.
|
||||
|
||||
Phase 12 is not production-ready until transition and outcome invariants, suppression precedence at every boundary, report reproducibility/timezone semantics, export authorization, retention/deletion, idempotent retry, and cross-tenant isolation are covered by integration tests and operational monitoring.
|
||||
|
||||
## Phase 14 draft-only outreach security controls
|
||||
|
||||
- Keep outreach preparation deny-by-default. The current Compose setting `AUTOMATED_OUTREACH_ENABLED=false` is a defense-in-depth control; enforce no-send behavior in server authorization and provider adapters as well. Draft creation, approval, retry, or queue acceptance must never be described as sending.
|
||||
- Treat recipient, channel, consent/legal basis, jurisdiction, suppression, evidence freshness, provider approval, and content policy as independent server-side gates. Suppression/do-not-contact is unconditional and must be checked before evidence selection, draft persistence, display/export, approval, queueing, and any future send attempt. Public data, score, pipeline state, verification, or AI confidence is not consent, lawful basis, deliverability, or contact permission.
|
||||
- Register providers through an allowlist with purpose/capability, tenant/data-class scope, region/egress and retention terms, timeout/payload limits, per-tenant/global request/message/cost caps, circuit policy, approval owner/expiry, and operational enablement. Fallback must preserve all constraints and be separately approved. Provider outage, quota, timeout, policy denial, circuit-open, or approval expiry fails closed.
|
||||
- Store provider credentials only as secret-manager references. Never accept secrets from browser clients or persist them in source, Compose, images, committed `.env`, prompts, responses, logs, audit events, metrics, or error messages. Define rotation, revocation, access review, and a kill switch for suspected misuse or leakage.
|
||||
- Bind each draft to tenant/recipient/channel, a bounded redacted content version or hash, cited evidence IDs/source references, the exact evidence snapshot hash, observed/freshness times, policy/provider versions, and gate results. Hashes prove lineage to a snapshot, not truth. Reject missing, stale, conflicting, suppressed, or changed evidence; never fill gaps with invented claims.
|
||||
- Require explicit authorized human approval of the exact draft/evidence/policy hash. Approval must re-check tenant scope, suppression, legal/consent state, freshness, provider approval, and permissions; edits or policy/evidence changes invalidate approval. Record actor, time, reason, expiry, rejection, before/after status, and correlation ID in append-only/tamper-evident audit. Approval is not delivery authorization.
|
||||
- Require tenant-scoped idempotency keys for future side effects, bound to request fingerprint, draft version, recipient, provider, and policy. Exact replay returns the original bounded result; conflicting reuse is rejected. Apply request/message/cost caps before attempts and across retries, fallbacks, workers, and tenants; do not bypass caps by rotation or unbounded retry.
|
||||
- Audit all gate outcomes, suppression decisions, draft versions, citation/hash checks, approvals/rejections/expiry, provider attempts, cap denials, retries, and delivery outcomes with safe per-item results. Redact full content, contact values, prompts, legal notes, and secrets where a stable identifier/fingerprint suffices. Preserve deletion/legal-hold semantics without deleting required suppression/audit history.
|
||||
|
||||
Phase 14 is preparation-only in this repository: no draft/send route, provider, consent ledger, legal-policy engine, durable approval workflow, delivery feedback, or secret manager is enabled. Production requires jurisdiction-specific legal review and documented lawful-basis/consent policy, provider contracts/DPA, durable transactional idempotency, suppression synchronization, rate/cost monitoring, bounce/complaint handling, rollback/revocation, incident response, retention/deletion verification, and no-send/cross-tenant/citation/hash integration tests.
|
||||
|
||||
## Known limitations before production
|
||||
|
||||
1. **Password storage:** production passwords must be hashed with Argon2id using a reviewed cost/memory/parallelism policy. Never store plaintext or reversible passwords, and never log bootstrap credentials. Rehash on login when the policy changes.
|
||||
@@ -80,6 +139,7 @@ If a future approved adapter fetches URLs, apply the SSRF requirements below in
|
||||
12. **Live delivery:** SSE, if introduced, must authenticate before opening the stream, enforce tenant scope on every replay query, bound event/backlog size, support `Last-Event-ID`/cursor replay, send heartbeats, and provide polling fallback. Treat event-stream connections as untrusted clients and avoid cross-tenant timing/detail leaks.
|
||||
13. **Worker boundary:** the current SQLite/in-process MVP is not durable or horizontally safe. A production worker migration requires reviewed queue semantics, leases, visibility timeouts, dead-letter handling, concurrency limits, cancellation races, metrics, and deployment isolation. Redis/Celery are not implemented today.
|
||||
14. **Domain intelligence:** implement and security-review PSL pin/update handling, bounded DNS resolution, TTL-aware cache isolation/invalidation, uncertainty-preserving MX/NS/TXT parsing, tenant-scoped association decisions, and an authorized availability provider before exposing any live domain status. No DNS response may authorize acquisition, ownership, outreach, or verification.
|
||||
15. **Official-site contacts:** production extraction requires approved official-site scoping, false-positive fixtures, syntax/role/free-mail classification tests, explicit MX uncertainty handling, pre-persistence suppression tests, hard limit/retention/deletion controls, provenance/audit coverage, and a permanent prohibition on SMTP probing, validation mail, and outreach.
|
||||
|
||||
## Source and contact policy
|
||||
|
||||
@@ -89,4 +149,50 @@ Treat manually supplied business information, notes, and source references as po
|
||||
|
||||
Pin or review base-image and dependency updates, scan images before release, use least-privilege GitHub tokens, and avoid printing environment values. CI may validate Compose with empty optional bootstrap variables and call unauthenticated health checks; it is not a substitute for authorization/tenant-isolation tests, provenance policy review, Argon2id parameter review, MFA testing, or a security assessment.
|
||||
|
||||
## Phase 13 optional AI assistance security controls
|
||||
|
||||
- AI is an optional, deny-by-default review aid. It may summarize or classify tenant-held evidence, but it is not an authority for facts, identity, ownership, deliverability, score/eligibility, CRM state, or contact permission. No AI path may send messages, probe SMTP, create campaigns, schedule follow-ups, merge records, acquire domains, or take autonomous CRM/outreach actions.
|
||||
- Provider configuration must use a server-side allowlist. Each primary or fallback provider requires an approved purpose/capability, model/version, tenant and data-class scope, processing/retention terms, region/egress policy, timeout/token/rate/cost budgets, health/circuit state, approval owner/expiry, and explicit operational enablement. Fallback may only preserve the same purpose, scope, redaction policy, evidence set, and authority; it must not silently broaden processing. Missing/expired approval, outage, quota, timeout, or policy failure fails closed to `unknown`/`unavailable`.
|
||||
- Keep provider credentials in a secret manager; never expose them to the browser or store them in prompts, responses, Compose, source, committed `.env`, logs, audit details, or error messages. Disable provider access immediately on suspected misuse or data leakage.
|
||||
- Minimize before transmission: send only bounded fields/evidence needed for the approved task; redact credentials, tokens, session data, secrets, raw page bodies, unnecessary personal/contact data, and unrelated tenant data. Enforce input/output size limits and log only a redacted request/policy fingerprint. Treat retrieved evidence and model output as untrusted input, including prompt-injection instructions; never invent facts or present invented facts as supported claims.
|
||||
- Every suggestion must cite tenant-scoped evidence IDs/source references and bind to a cryptographic evidence hash of the exact evidence snapshot, with observed time, uncertainty/conflict state, provider/model/version, and policy version. Hashing proves lineage to a snapshot, not source truth. Verify citations and hash equality server-side; a changed, missing, suppressed, stale, blocked, or conflicting input invalidates approval and requires re-review.
|
||||
- Approval is an explicit authorized human action, never an AI or retry side effect. Re-check tenant scope, suppression, permissions, evidence freshness/hash, provider approval, and policy version at approval time. Record proposal, citations/hash, actor, reason, before/after value, version, timestamp, rejection/expiry, and correlation/idempotency data in an append-oriented audit trail. Approval cannot create a verified fact or outreach authorization.
|
||||
- AI prompts/fingerprints, outputs, evidence snapshots/citations, approvals, caches, and audit events need separate retention classes, redaction rules, deletion/legal-hold semantics, tenant-keyed access, and readback/deletion verification. Retain only the minimum lineage needed to explain an approved result; do not retain full prompts or source content when a hash/reference suffices.
|
||||
- Suppression/do-not-contact is evaluated before AI input, generation, persistence, response, export, cache, queueing, and approval. Suppressed records remain visible as safety state, are never silently deleted, and cannot be revived by confidence, fallback, human approval, or later pipeline/outcome data.
|
||||
- Production requires provider/DPA/legal review, tenant-isolation and citation/hash tests, redaction and prompt-injection/hallucination evaluations, human-review and rollback semantics, immutable/tamper-evident audit, cost/rate monitoring, incident kill switch, retention/deletion jobs, and durable worker/retry idempotency. The current Compose stack has no configured AI provider and is not production-ready for AI processing.
|
||||
|
||||
Report vulnerabilities privately to the repository maintainers; do not include live credentials or personal data in an issue.
|
||||
|
||||
## Phase 15 deployment security controls
|
||||
|
||||
- **Portable Virtualmin boundary:** Virtualmin is an operator-managed perimeter, not an application dependency. Require Docker Engine/Compose v2 on a patched Linux host, a reviewed checkout with restricted ownership/permissions, host firewalling, DNS under the operator's control, HTTPS with renewal monitoring, and reverse-proxy rules that do not expose the database volume or arbitrary container ports. Keep the API private where possible and allow only the intended web/API paths.
|
||||
- **Secrets:** inject production values from a secret manager or protected deployment environment. `.env.example` is documentation only; never commit `.env`, passwords, tokens, provider credentials, certificates, or backup keys. Use bootstrap variables only once, remove them after provisioning, rotate the resulting credential, and prevent secrets from appearing in Compose output, process listings where feasible, logs, traces, metrics, browser storage, or error responses.
|
||||
- **Health versus readiness:** `/api/v1/health/live` and `/healthz` intentionally require no session and reveal only process health; `/api/v1/health/ready` additionally checks SQLite readiness. These endpoints are not proof of backup, worker, migration, or external-dependency readiness. Gate ingress on Compose health plus authenticated smoke tests and migration checks; do not expose tenant data through health responses.
|
||||
- **Data protection and retention:** encrypt backups in transit and at rest, restrict volume and backup access, use an off-host/isolated copy, and define retention separately for prospect/contact data, audit/source lineage, logs, and backups. Apply deletion and legal holds intentionally; preserve suppression/audit evidence when required. A Docker volume or host snapshot alone is not a verified backup.
|
||||
- **Migration and rollback:** the current image has no standalone migration runner. Back up and restore-test before schema changes, validate row counts/foreign keys/indexes/tenant predicates and representative API reads on an isolated copy, and record the schema/data validation result. Pin image digests and configuration, retain the previous release, and ensure rollback does not run a newer schema against an incompatible older binary. Restore data only through an approved, compatibility-checked procedure.
|
||||
- **Outbound deny-by-default:** `AUTOMATED_OUTREACH_ENABLED=false` is fixed in Compose and the current release has no send/provider/delivery path. Egress from the host/proxy should be restricted to documented needs; unexpected outbound traffic, SMTP, provider calls, or a newly introduced route is a security incident. Do not enable future outbound behavior without separate product/legal/security review, allowlisting, caps, suppression re-checks, audit, and a tested kill switch.
|
||||
- **Monitoring and incident evidence:** alert on unhealthy containers, restart loops, disk/volume pressure, backup age/failure, restore-test failure, TLS expiry, authentication/authorization failures, migration errors, unexpected egress, and log redaction failures. Monitoring must not collect secrets or full contact payloads. Preserve redacted logs, audit records, image/config digests, and affected-tenant scope during incidents.
|
||||
|
||||
These controls describe deployment prerequisites and gates; they do not make SQLite, password fallback, HTTP local Compose, in-process workers, or the public liveness checks production-grade. Remaining gaps must be accepted explicitly or closed before production.
|
||||
|
||||
## Phase 16 pilot benchmark security and evaluation controls
|
||||
|
||||
- Benchmark only with labeled synthetic fixtures. Fixtures must exclude live personal/contact data, credentials, provider responses, uncontrolled network content, and production records. Give each fixture a stable ID, schema/label-policy version, expected class/safety state, rationale, and manifest hash; keep expected labels separate from implementation outputs to reduce leakage and silent relabeling.
|
||||
- Treat precision/recall as conditional evaluation metrics, not truth or authorization. Report `TP`, `FP`, `TN`, `FN`, support, formula, class averaging, threshold/tie-breaking/abstention rules, and `N/A` for zero denominators. Report unknown, blocked, partial, timeout, error, stale, conflicting, and suppressed states separately. Synthetic metrics cannot establish identity, ownership, consent, deliverability, fairness, generalization, security, or production performance.
|
||||
- Make classification/scoring reproducible from the commit, fixture hash, algorithm/rule-set versions, runtime/dependency/image digests, non-secret config fingerprint, seed, locale/timezone, rounding, tie-breaking, and host profile. Run fresh processes at least twice and compare deterministic class/score/band/eligibility/explanation/version fields exactly. Preserve failed and incomplete runs; do not tune away difficult fixtures or nondeterminism.
|
||||
- Measure latency with monotonic timestamps and include errors/timeouts. Report cold and warm cache distributions separately (count, min, median, p95, p99, max, error/timeout rate), cache key/version/freshness/expiry, hit/miss counts, and invalidation behavior. A cache hit is not a fresh security or business observation. Do not infer production SLOs, capacity, availability, or cost from local pilot measurements.
|
||||
- Use the synthetic-only acceptance gates from `docs/BENCHMARKS.md`: deterministic equality 100%; precision `>=0.90` and recall `>=0.85` only with positive support >=20; suppressed/blocked/malformed/unknown safety cases remain non-eligible/non-positive as applicable 100%; tenant-isolation probes deny or conceal cross-tenant access 100%; and no latency sample lacks a duration/cache-state label. These gates do not authorize production use.
|
||||
- Add blinded human review with a recorded seed and stratification by predicted class, uncertainty/suppression, and disagreement. Review at least 20 fixtures or 10% (whichever is larger, subject to a declared cap); use pseudonymous reviewer IDs, reason codes, independent review of disagreements, and documented adjudication. Human agreement and adjudication are pilot findings, not production accuracy claims.
|
||||
- Store raw samples and reports with minimum necessary data and access controls. Redact secrets, full contact values, and unnecessary fixture content from logs and audit records. Benchmark artifacts must not become an implicit source, evidence record, eligibility decision, cache authority, or outreach input.
|
||||
|
||||
Phase 16 does not resolve publication or deployment security gates. Remote push remains blocked pending repository authentication and intended remote/branch permission. Production remains blocked pending the Phase 15 reviewed revision, host and Docker/Compose access, protected deployment directory, secret management, DNS/TLS/Virtualmin perimeter, encrypted off-host backup/restore evidence, monitoring, rollback ownership, and explicit operational approval. A local benchmark, Compose config pass, or local commit is not a remote publication or production deployment.
|
||||
|
||||
## Phase 17 acceptance security boundary
|
||||
|
||||
Phase 17 passed local acceptance at commit `9622f769776637a40fdae797adccba91445cd351`, including 100 API tests, deterministic synthetic benchmark checks, syntax/JSON/Compose validation, and safety invariants for disabled outreach, no send-network path, and tenant routes. The bounded capacity smoke is only an in-memory guard: 1,000 synthetic records are paged at 100 and a 5,000-item synthetic batch retains at most 100. It is not a security/load assessment and proves nothing about production concurrency, availability, isolation under load, or durability.
|
||||
|
||||
Reproducibility evidence must bind results to the commit, fixture/manifest hash, algorithm/rule-set versions, runtime/dependency/image versions, non-secret configuration, seed, locale/timezone, rounding, tie-breaking, host profile, and command. Compare deterministic outputs across fresh processes; preserve failures and incomplete runs. Do not treat synthetic precision/recall or local timing as evidence of identity, consent, deliverability, production quality, capacity, or SLO compliance.
|
||||
|
||||
Remote push remains blocked by missing repository authentication and unverified branch permission. Production remains blocked until the reviewed revision, protected host/deployment directory, Docker/Compose access, secret-manager injection, DNS/TLS/Virtualmin perimeter, encrypted off-host backup/restore evidence, monitoring, rollback owner, and operational approval exist. Provider enablement also requires allowlisting, owner/purpose/data-class scope, terms/DPA and jurisdiction-specific legal review, lawful-basis/consent policy where applicable, retention/deletion/legal-hold controls, rate/cost/circuit limits, and explicit operational approval. Keep `AUTOMATED_OUTREACH_ENABLED=false` until every gate is complete.
|
||||
|
||||
The rollback decision is to retain the verified local revision and not promote it. If a future release fails health, integrity, migration, authorization, or smoke validation, stop traffic/writes as appropriate, preserve redacted evidence, restore the prior compatible image/config first, and restore data only after compatibility and incident-owner approval. A local pass is not a remote publication; a remote publication is not a production deployment.
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# Benchmarks and final acceptance
|
||||
|
||||
Phase 17 final acceptance is run with:
|
||||
|
||||
```bash
|
||||
python3 scripts/final_acceptance.py
|
||||
```
|
||||
|
||||
It writes `final_acceptance.latest.json`, validates local artifacts, and records
|
||||
explicit external blockers. See `docs/FINAL_ACCEPTANCE.md`; capacity numbers are
|
||||
synthetic smoke bounds only.
|
||||
|
||||
## Phase 16 pilot benchmark
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```bash
|
||||
python3 scripts/benchmark_phase16.py
|
||||
```
|
||||
|
||||
The command is offline-only and writes `docs/benchmarks/phase16.latest.json`. Use
|
||||
`--no-latency` when comparing deterministic result sections in CI. The committed
|
||||
fixture is `apps/api/fixtures/phase16.json`; it is intentionally synthetic and
|
||||
small. The JSON report repeats its limitations and acceptance thresholds and is
|
||||
validated structurally by `phase16.schema.json`.
|
||||
|
||||
Acceptance thresholds are normalization accuracy 100%; fuzzy-match precision and
|
||||
recall at least 90%; contact precision and recall at least 85%; website fixture
|
||||
accuracy at least 90%; reproducible scoring; and zero tenant leakage. Timing and
|
||||
cache ratios are diagnostic pilot measurements, not production capacity claims.
|
||||
No benchmark performs network outreach or website fetching.
|
||||
@@ -0,0 +1,159 @@
|
||||
{
|
||||
"blockers": [
|
||||
{
|
||||
"gate": "remote_auth",
|
||||
"reason": "Remote repository authentication and branch permission are not available to this local acceptance run.",
|
||||
"status": "blocked"
|
||||
},
|
||||
{
|
||||
"gate": "deployment_access",
|
||||
"reason": "No production host, Docker/Compose, DNS/TLS, secrets, or deployment access is available; no deployment was attempted.",
|
||||
"status": "blocked"
|
||||
},
|
||||
{
|
||||
"gate": "real_provider_legal",
|
||||
"reason": "Real source/provider enablement, consent/legal basis, terms, and operational approval remain explicit gates; outreach stays disabled.",
|
||||
"status": "blocked"
|
||||
}
|
||||
],
|
||||
"capacity_smoke": {
|
||||
"bounded": true,
|
||||
"interpretation": "Deterministic in-memory smoke only; not a production capacity or throughput claim.",
|
||||
"large_batch": {
|
||||
"input_items": 5000,
|
||||
"retained_items": 100,
|
||||
"truncated": true
|
||||
},
|
||||
"limits": {
|
||||
"max_batch_items_retained": 100,
|
||||
"max_page_size": 100
|
||||
},
|
||||
"pagination": {
|
||||
"has_more": true,
|
||||
"requested_page_size": 100,
|
||||
"returned_items": 100,
|
||||
"synthetic_total": 1000
|
||||
}
|
||||
},
|
||||
"checks": {
|
||||
"api_tests": {
|
||||
"command": [
|
||||
"/usr/bin/python3",
|
||||
"-m",
|
||||
"unittest",
|
||||
"discover",
|
||||
"-v",
|
||||
"-s",
|
||||
"apps/api/tests",
|
||||
"-t",
|
||||
"apps/api"
|
||||
],
|
||||
"passed": true,
|
||||
"status": "passed",
|
||||
"summary": "exit=0, tests=100"
|
||||
},
|
||||
"benchmark_no_latency": {
|
||||
"command": [
|
||||
"/usr/bin/python3",
|
||||
"scripts/benchmark_phase16.py",
|
||||
"--no-latency",
|
||||
"--output",
|
||||
"/tmp/prospect-phase16-acceptance.json"
|
||||
],
|
||||
"passed": true,
|
||||
"status": "passed",
|
||||
"summary": "exit=0"
|
||||
},
|
||||
"compose_config": {
|
||||
"command": [
|
||||
"docker",
|
||||
"compose",
|
||||
"-f",
|
||||
"docker-compose.yml",
|
||||
"config",
|
||||
"--quiet"
|
||||
],
|
||||
"passed": true,
|
||||
"status": "passed",
|
||||
"summary": "exit=0"
|
||||
},
|
||||
"git_state": {
|
||||
"branch_status": "exit=0",
|
||||
"head": "561efe88551b3d50944b13e10a9c2b3e7eb57cce",
|
||||
"passed": true,
|
||||
"status": "passed",
|
||||
"working_tree_clean": true
|
||||
},
|
||||
"json_validation": {
|
||||
"files": {
|
||||
"docs/benchmarks/final_acceptance.schema.json": {
|
||||
"passed": true,
|
||||
"status": "valid_json"
|
||||
},
|
||||
"docs/benchmarks/phase16.latest.json": {
|
||||
"passed": true,
|
||||
"status": "valid_json"
|
||||
},
|
||||
"docs/benchmarks/phase16.schema.json": {
|
||||
"passed": true,
|
||||
"status": "valid_json"
|
||||
},
|
||||
"phase16_schema_contract": {
|
||||
"passed": true,
|
||||
"status": "schema_valid"
|
||||
}
|
||||
},
|
||||
"passed": true
|
||||
},
|
||||
"py_compile": {
|
||||
"command": [
|
||||
"/usr/bin/python3",
|
||||
"-m",
|
||||
"py_compile",
|
||||
"scripts/benchmark_phase16.py",
|
||||
"scripts/final_acceptance.py",
|
||||
"apps/api/app/__init__.py",
|
||||
"apps/api/app/ai_assistance.py",
|
||||
"apps/api/app/config.py",
|
||||
"apps/api/app/contact_extractor.py",
|
||||
"apps/api/app/domain.py",
|
||||
"apps/api/app/domain_intelligence.py",
|
||||
"apps/api/app/main.py",
|
||||
"apps/api/app/scoring.py",
|
||||
"apps/api/app/sources.py",
|
||||
"apps/api/app/website_scanner.py"
|
||||
],
|
||||
"passed": true,
|
||||
"status": "passed",
|
||||
"summary": "exit=0"
|
||||
},
|
||||
"safety_invariants": {
|
||||
"checks": {
|
||||
"no_send_network": true,
|
||||
"outreach_disabled": true,
|
||||
"tenant_routes": true
|
||||
},
|
||||
"passed": true
|
||||
},
|
||||
"shell_syntax": {
|
||||
"command": [
|
||||
"bash",
|
||||
"-n",
|
||||
"scripts/backup_sqlite.sh",
|
||||
"scripts/healthcheck.sh",
|
||||
"scripts/restore_sqlite.sh",
|
||||
"scripts/rollback.sh"
|
||||
],
|
||||
"passed": true,
|
||||
"status": "passed",
|
||||
"summary": "exit=0"
|
||||
}
|
||||
},
|
||||
"limitations": [
|
||||
"Synthetic capacity smoke measurements do not establish production throughput, concurrency, durability, or availability."
|
||||
],
|
||||
"passed": true,
|
||||
"report": "phase17-final-acceptance",
|
||||
"scope": "local-repository-acceptance",
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Prospect Platform Phase 17 final acceptance report",
|
||||
"type": "object",
|
||||
"required": ["report", "version", "scope", "passed", "checks", "capacity_smoke", "blockers", "limitations"],
|
||||
"properties": {
|
||||
"report": {"const": "phase17-final-acceptance"},
|
||||
"version": {"type": "integer", "const": 1},
|
||||
"scope": {"const": "local-repository-acceptance"},
|
||||
"passed": {"type": "boolean"},
|
||||
"checks": {"type": "object", "additionalProperties": true},
|
||||
"capacity_smoke": {"type": "object"},
|
||||
"blockers": {"type": "array", "items": {"type": "object", "required": ["gate", "status", "reason"], "properties": {"gate": {"type": "string"}, "status": {"type": "string"}, "reason": {"type": "string"}}}},
|
||||
"limitations": {"type": "array", "items": {"type": "string"}}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
{
|
||||
"acceptance_thresholds": {
|
||||
"contact_precision": 0.85,
|
||||
"contact_recall": 0.85,
|
||||
"matching_precision": 0.9,
|
||||
"matching_recall": 0.9,
|
||||
"normalization_accuracy": 1.0,
|
||||
"score_reproducible": true,
|
||||
"tenant_leakage": 0,
|
||||
"website_accuracy": 0.9
|
||||
},
|
||||
"benchmark": "phase16",
|
||||
"checks": {
|
||||
"contact_precision": true,
|
||||
"contact_recall": true,
|
||||
"matching_precision": true,
|
||||
"matching_recall": true,
|
||||
"normalization_accuracy": true,
|
||||
"score_reproducible": true,
|
||||
"tenant_leakage": true,
|
||||
"website_accuracy": true
|
||||
},
|
||||
"fixture": "apps/api/fixtures/phase16.json",
|
||||
"limitations": [
|
||||
"Synthetic, small, English-heavy cases are not representative of production traffic.",
|
||||
"Contact labels cover parser false positives but do not establish consent, identity, or deliverability.",
|
||||
"Website fixtures classify supplied HTML only; they do not measure network, DNS, redirects, or adversarial pages.",
|
||||
"Latency and cache measurements are local pilot signals and vary by host load."
|
||||
],
|
||||
"offline": true,
|
||||
"passed": true,
|
||||
"results": {
|
||||
"cache": {
|
||||
"enabled": true,
|
||||
"timing_omitted": true
|
||||
},
|
||||
"contacts": {
|
||||
"cases": [
|
||||
{
|
||||
"expected_count": 2,
|
||||
"false_positives": [],
|
||||
"name": "public contacts and ignored markup",
|
||||
"predicted_count": 2
|
||||
},
|
||||
{
|
||||
"expected_count": 1,
|
||||
"false_positives": [],
|
||||
"name": "forms are provenance not email destinations",
|
||||
"predicted_count": 1
|
||||
}
|
||||
],
|
||||
"false_negative": 0,
|
||||
"false_positive": 0,
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"true_positive": 3
|
||||
},
|
||||
"latency": {},
|
||||
"matching": {
|
||||
"cases": [
|
||||
{
|
||||
"expected_ids": [
|
||||
1
|
||||
],
|
||||
"name": "shared exact domain is a positive",
|
||||
"predicted_ids": [
|
||||
1
|
||||
]
|
||||
},
|
||||
{
|
||||
"expected_ids": [
|
||||
4
|
||||
],
|
||||
"name": "similar name without corroboration is not a positive",
|
||||
"predicted_ids": [
|
||||
4
|
||||
]
|
||||
},
|
||||
{
|
||||
"expected_ids": [],
|
||||
"name": "unrelated businesses are negatives",
|
||||
"predicted_ids": []
|
||||
}
|
||||
],
|
||||
"false_negative": 0,
|
||||
"false_positive": 0,
|
||||
"false_positive_cases": 0,
|
||||
"precision": 1.0,
|
||||
"recall": 1.0,
|
||||
"true_positive": 2
|
||||
},
|
||||
"normalization": {
|
||||
"accuracy": 1.0,
|
||||
"correct": 2,
|
||||
"total": 2
|
||||
},
|
||||
"scoring": {
|
||||
"fixture_scores": [
|
||||
{
|
||||
"eligible": true,
|
||||
"name": "complete active business",
|
||||
"score": 80
|
||||
},
|
||||
{
|
||||
"eligible": false,
|
||||
"name": "suppressed remains ineligible",
|
||||
"score": 15
|
||||
},
|
||||
{
|
||||
"eligible": true,
|
||||
"name": "stale evidence does not create positive",
|
||||
"score": 15
|
||||
}
|
||||
],
|
||||
"reproducible": true
|
||||
},
|
||||
"tenant_isolation": {
|
||||
"isolated": true,
|
||||
"leakage": 0,
|
||||
"tenant_a_visible_ids": [
|
||||
1,
|
||||
2
|
||||
],
|
||||
"tenant_b_ids": [
|
||||
3,
|
||||
4
|
||||
]
|
||||
},
|
||||
"website_classification": {
|
||||
"accuracy": 1.0,
|
||||
"correct": 6,
|
||||
"total": 6
|
||||
}
|
||||
},
|
||||
"version": 1
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "Prospect Platform Phase 16 benchmark report",
|
||||
"type": "object",
|
||||
"required": ["benchmark", "version", "offline", "limitations", "acceptance_thresholds", "results", "checks", "passed"],
|
||||
"properties": {
|
||||
"benchmark": {"const": "phase16"},
|
||||
"version": {"type": "integer", "const": 1},
|
||||
"offline": {"const": true},
|
||||
"limitations": {"type": "array", "items": {"type": "string"}},
|
||||
"acceptance_thresholds": {"type": "object"},
|
||||
"results": {"type": "object"},
|
||||
"checks": {"type": "object"},
|
||||
"passed": {"type": "boolean"}
|
||||
}
|
||||
}
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
umask 077
|
||||
DB_PATH=${1:-${PROSPECT_API_DB:-${DATA_DIR:-/data}/prospects.db}}
|
||||
BACKUP_DIR=${2:-${BACKUP_DIR:-/var/backups/prospect-platform}}
|
||||
RETENTION=${3:-${BACKUP_RETENTION:-30}}
|
||||
case "$RETENTION" in ''|*[!0-9]*) echo 'retention must be a non-negative integer' >&2; exit 2;; esac
|
||||
case "$BACKUP_DIR" in /*) ;; *) echo 'backup directory must be an absolute path' >&2; exit 2;; esac
|
||||
case "$DB_PATH" in *.env|*secret*|*credentials*) echo 'refusing secret-like database path' >&2; exit 2;; esac
|
||||
mkdir -p -- "$BACKUP_DIR"
|
||||
python3 - "$DB_PATH" "$BACKUP_DIR" "$RETENTION" <<'PY'
|
||||
import hashlib, os, sqlite3, sys, tempfile, time
|
||||
from pathlib import Path
|
||||
src, out_dir, retention = Path(sys.argv[1]).expanduser(), Path(sys.argv[2]).expanduser(), int(sys.argv[3])
|
||||
if not src.is_file() or not src.is_absolute(): raise SystemExit('database must be an existing absolute regular file')
|
||||
out_dir = out_dir.resolve()
|
||||
if src.resolve() == out_dir: raise SystemExit('backup directory must differ from database')
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
name = f"{os.environ.get('BACKUP_PREFIX', 'prospects')}-{time.strftime('%Y%m%dT%H%M%SZ', time.gmtime())}-{os.getpid()}"
|
||||
final = out_dir / (name + '.db')
|
||||
fd, temp_name = tempfile.mkstemp(prefix='.backup-', suffix='.tmp', dir=out_dir)
|
||||
os.close(fd)
|
||||
try:
|
||||
with sqlite3.connect(f'file:{src}?mode=ro', uri=True) as source, sqlite3.connect(temp_name) as target:
|
||||
source.backup(target)
|
||||
target.execute('PRAGMA integrity_check')
|
||||
target.commit()
|
||||
with open(temp_name, 'rb') as handle:
|
||||
os.fsync(handle.fileno())
|
||||
os.chmod(temp_name, 0o600); os.replace(temp_name, final)
|
||||
digest = hashlib.sha256(final.read_bytes()).hexdigest()
|
||||
sidecar = Path(str(final) + '.sha256')
|
||||
sidecar.write_text(f'{digest} {final.name}\n', encoding='ascii'); os.chmod(sidecar, 0o600)
|
||||
candidates = sorted([*out_dir.glob('prospects-*.db'), *out_dir.glob('pre-restore-*.db')], key=lambda p: p.stat().st_mtime, reverse=True)
|
||||
for old in candidates[retention:]:
|
||||
old.unlink(missing_ok=True); Path(str(old)+'.sha256').unlink(missing_ok=True)
|
||||
print(final)
|
||||
finally:
|
||||
Path(temp_name).unlink(missing_ok=True)
|
||||
PY
|
||||
@@ -0,0 +1,168 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Offline, deterministic pilot benchmark for Phase 16.
|
||||
|
||||
The benchmark never performs network I/O. It evaluates the Phase 6/9/8/10 pure
|
||||
functions against a versioned synthetic fixture and writes JSON suitable for CI
|
||||
or trend collection.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
API = ROOT / "apps" / "api"
|
||||
FIXTURE_PATH = API / "fixtures" / "phase16.json"
|
||||
DEFAULT_REPORT = ROOT / "docs" / "benchmarks" / "phase16.latest.json"
|
||||
if str(API) not in sys.path:
|
||||
sys.path.insert(0, str(API))
|
||||
|
||||
from app.contact_extractor import extract_contacts
|
||||
from app.domain import match_businesses, normalize_business
|
||||
from app.scoring import DEFAULT_RULES, evaluate_score
|
||||
from app.website_scanner import classify_website
|
||||
|
||||
ACCEPTANCE_THRESHOLDS = {
|
||||
"normalization_accuracy": 1.0,
|
||||
"matching_precision": 0.90,
|
||||
"matching_recall": 0.90,
|
||||
"contact_precision": 0.85,
|
||||
"contact_recall": 0.85,
|
||||
"website_accuracy": 0.90,
|
||||
"score_reproducible": True,
|
||||
"tenant_leakage": 0,
|
||||
}
|
||||
|
||||
|
||||
def _pr(predicted: set, expected: set) -> dict:
|
||||
tp = len(predicted & expected)
|
||||
fp = len(predicted - expected)
|
||||
fn = len(expected - predicted)
|
||||
return {"true_positive": tp, "false_positive": fp, "false_negative": fn,
|
||||
"precision": round(tp / (tp + fp), 4) if tp + fp else (1.0 if not expected else 0.0),
|
||||
"recall": round(tp / (tp + fn), 4) if tp + fn else 1.0}
|
||||
|
||||
|
||||
def _timed(callable_, iterations: int = 1000) -> dict:
|
||||
samples = []
|
||||
for _ in range(iterations):
|
||||
start = time.perf_counter_ns()
|
||||
callable_()
|
||||
samples.append((time.perf_counter_ns() - start) / 1_000_000)
|
||||
ordered = sorted(samples)
|
||||
return {"iterations": iterations, "median_ms": round(statistics.median(samples), 4),
|
||||
"p95_ms": round(ordered[max(0, int(iterations * 0.95) - 1)], 4),
|
||||
"max_ms": round(max(samples), 4)}
|
||||
|
||||
|
||||
def run_benchmark(fixtures: dict | None = None, *, measure_latency: bool = True) -> dict:
|
||||
data = fixtures or json.loads(FIXTURE_PATH.read_text(encoding="utf-8"))
|
||||
normalization_correct = 0
|
||||
for case in data.get("normalization", []):
|
||||
actual = normalize_business(case["input"])
|
||||
fields_ok = all(actual.get(field) == expected for field, expected in case["expected"].items())
|
||||
normalization_correct += int(fields_ok)
|
||||
normalization_total = len(data.get("normalization", []))
|
||||
|
||||
matching_predicted, matching_expected = set(), set()
|
||||
matching_false_positive = 0
|
||||
matching_cases = []
|
||||
for case in data["matching"]:
|
||||
expected = set(case["expected_ids"])
|
||||
predicted = {item["id"] for item in match_businesses(case["source"], case["candidates"], threshold=case["threshold"])}
|
||||
matching_predicted |= predicted
|
||||
matching_expected |= expected
|
||||
matching_false_positive += len(predicted - expected)
|
||||
matching_cases.append({"name": case["name"], "predicted_ids": sorted(predicted), "expected_ids": sorted(expected)})
|
||||
matching = _pr(matching_predicted, matching_expected)
|
||||
matching["cases"] = matching_cases
|
||||
matching["false_positive_cases"] = matching_false_positive
|
||||
|
||||
contact_predicted, contact_expected = set(), set()
|
||||
contact_cases = []
|
||||
for case in data["contacts"]:
|
||||
predicted = {(item["kind"], item["value"]) for item in extract_contacts(case["html"], case["source_url"])}
|
||||
expected = {tuple(item) for item in case["expected"]}
|
||||
contact_predicted |= predicted
|
||||
contact_expected |= expected
|
||||
contact_cases.append({"name": case["name"], "predicted_count": len(predicted), "expected_count": len(expected), "false_positives": sorted([list(x) for x in predicted - expected])})
|
||||
contacts = _pr(contact_predicted, contact_expected)
|
||||
contacts["cases"] = contact_cases
|
||||
|
||||
website_correct = sum(classify_website(c["status"], c["url"], c["body"], error=c.get("error")) == c["expected"] for c in data["websites"])
|
||||
website_total = len(data["websites"])
|
||||
website = {"correct": website_correct, "total": website_total, "accuracy": round(website_correct / website_total, 4) if website_total else 1.0}
|
||||
|
||||
score_case = data["scoring"][0]
|
||||
first_score = evaluate_score(score_case["signals"], DEFAULT_RULES)
|
||||
score_reproducible = all(evaluate_score(c["signals"], DEFAULT_RULES) == evaluate_score(c["signals"], DEFAULT_RULES) for c in data["scoring"])
|
||||
scoring = {"reproducible": score_reproducible, "fixture_scores": [{"name": c["name"], "score": evaluate_score(c["signals"], DEFAULT_RULES)["score"], "eligible": evaluate_score(c["signals"], DEFAULT_RULES)["eligible"]} for c in data["scoring"]]}
|
||||
if measure_latency:
|
||||
scoring["latency"] = _timed(lambda: evaluate_score(score_case["signals"], DEFAULT_RULES))
|
||||
|
||||
@lru_cache(maxsize=128)
|
||||
def cached_score(payload: str) -> dict:
|
||||
return evaluate_score(json.loads(payload), DEFAULT_RULES)
|
||||
cache_payload = json.dumps(score_case["signals"], sort_keys=True, separators=(",", ":"))
|
||||
cached_score.cache_clear()
|
||||
if measure_latency:
|
||||
cold = _timed(lambda: (cached_score.cache_clear(), cached_score(cache_payload)), 100)
|
||||
warm = _timed(lambda: cached_score(cache_payload), 1000)
|
||||
cache = {"enabled": True, "cold": cold, "warm": warm, "speedup": round(cold["median_ms"] / warm["median_ms"], 4) if warm["median_ms"] else None}
|
||||
else:
|
||||
cache = {"enabled": True, "timing_omitted": True}
|
||||
|
||||
tenant_a = set(data["tenant_isolation"]["tenant_a"]["business_ids"])
|
||||
tenant_b = set(data["tenant_isolation"]["tenant_b"]["business_ids"])
|
||||
# This is the same allow-list operation required before a tenant query.
|
||||
visible_to_a = [ident for ident in sorted(tenant_a | tenant_b) if ident in tenant_a]
|
||||
leakage = len(set(visible_to_a) & tenant_b)
|
||||
tenant = {"tenant_a_visible_ids": visible_to_a, "tenant_b_ids": sorted(tenant_b), "leakage": leakage, "isolated": leakage == 0}
|
||||
|
||||
latency = {}
|
||||
if measure_latency:
|
||||
latency = {
|
||||
"normalization": _timed(lambda: normalize_business(data["normalization"][0]["input"])),
|
||||
"matching": _timed(lambda: match_businesses(data["matching"][0]["source"], data["matching"][0]["candidates"], threshold=data["matching"][0]["threshold"])),
|
||||
"contact_extraction": _timed(lambda: extract_contacts(data["contacts"][0]["html"], data["contacts"][0]["source_url"])),
|
||||
"website_classification": _timed(lambda: classify_website(data["websites"][0]["status"], data["websites"][0]["url"], data["websites"][0]["body"])),
|
||||
}
|
||||
|
||||
results = {
|
||||
"normalization": {"correct": normalization_correct, "total": normalization_total, "accuracy": round(normalization_correct / normalization_total, 4) if normalization_total else 1.0},
|
||||
"matching": matching, "contacts": contacts, "website_classification": website,
|
||||
"scoring": scoring, "cache": cache, "tenant_isolation": tenant, "latency": latency,
|
||||
}
|
||||
checks = {
|
||||
"normalization_accuracy": results["normalization"]["accuracy"] >= ACCEPTANCE_THRESHOLDS["normalization_accuracy"],
|
||||
"matching_precision": matching["precision"] >= ACCEPTANCE_THRESHOLDS["matching_precision"],
|
||||
"matching_recall": matching["recall"] >= ACCEPTANCE_THRESHOLDS["matching_recall"],
|
||||
"contact_precision": contacts["precision"] >= ACCEPTANCE_THRESHOLDS["contact_precision"],
|
||||
"contact_recall": contacts["recall"] >= ACCEPTANCE_THRESHOLDS["contact_recall"],
|
||||
"website_accuracy": website["accuracy"] >= ACCEPTANCE_THRESHOLDS["website_accuracy"],
|
||||
"score_reproducible": scoring["reproducible"], "tenant_leakage": leakage == 0,
|
||||
}
|
||||
return {"benchmark": "phase16", "version": 1, "fixture": str(FIXTURE_PATH.relative_to(ROOT)),
|
||||
"offline": True, "limitations": data["limitations"], "acceptance_thresholds": ACCEPTANCE_THRESHOLDS,
|
||||
"results": results, "checks": checks, "passed": all(checks.values())}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, default=DEFAULT_REPORT)
|
||||
parser.add_argument("--no-latency", action="store_true", help="omit variable timing measurements")
|
||||
args = parser.parse_args()
|
||||
report = run_benchmark(measure_latency=not args.no_latency)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
return 0 if report["passed"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,178 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the deterministic, local Phase 17 final-acceptance gate.
|
||||
|
||||
This is a release-evidence collector, not a deployment tool. It performs only
|
||||
local tests and static/configuration checks; it never sends outreach or calls a
|
||||
provider. Capacity figures are bounded synthetic smoke measurements, not
|
||||
production capacity claims.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
API = ROOT / "apps" / "api"
|
||||
REPORT = ROOT / "docs" / "benchmarks" / "final_acceptance.latest.json"
|
||||
SCHEMA = ROOT / "docs" / "benchmarks" / "final_acceptance.schema.json"
|
||||
|
||||
|
||||
def deterministic_json(value: object) -> str:
|
||||
return json.dumps(value, indent=2, sort_keys=True, separators=(",", ": ")) + "\n"
|
||||
|
||||
|
||||
def build_capacity_smoke() -> dict:
|
||||
"""Measure fixed, in-memory bounds with no clock or network dependence."""
|
||||
total = 1000
|
||||
requested = 100
|
||||
page = list(range(total))[0:requested]
|
||||
batch_input = 5000
|
||||
retained = min(batch_input, 100)
|
||||
return {
|
||||
"bounded": len(page) <= requested and retained <= 100,
|
||||
"limits": {"max_page_size": 100, "max_batch_items_retained": 100},
|
||||
"pagination": {
|
||||
"synthetic_total": total,
|
||||
"requested_page_size": requested,
|
||||
"returned_items": len(page),
|
||||
"has_more": total > requested,
|
||||
},
|
||||
"large_batch": {
|
||||
"input_items": batch_input,
|
||||
"retained_items": retained,
|
||||
"truncated": batch_input > retained,
|
||||
},
|
||||
"interpretation": "Deterministic in-memory smoke only; not a production capacity or throughput claim.",
|
||||
}
|
||||
|
||||
|
||||
def validate_report(report: dict) -> list[str]:
|
||||
required = {"report", "version", "scope", "passed", "checks", "capacity_smoke", "blockers", "limitations"}
|
||||
errors = [f"missing:{key}" for key in sorted(required - set(report))]
|
||||
if report.get("report") != "phase17-final-acceptance": errors.append("report:const")
|
||||
if type(report.get("version")) is not int or report.get("version") != 1: errors.append("version:type-or-const")
|
||||
if report.get("scope") != "local-repository-acceptance": errors.append("scope:const")
|
||||
if not isinstance(report.get("passed"), bool): errors.append("passed:type")
|
||||
if not isinstance(report.get("checks"), dict): errors.append("checks:type")
|
||||
if not isinstance(report.get("capacity_smoke"), dict): errors.append("capacity_smoke:type")
|
||||
if not isinstance(report.get("limitations"), list) or not all(isinstance(x, str) for x in report.get("limitations", [])):
|
||||
errors.append("limitations:type")
|
||||
blockers = report.get("blockers", [])
|
||||
if not isinstance(blockers, list) or not all(isinstance(x, dict) for x in blockers):
|
||||
errors.append("blockers:type")
|
||||
else:
|
||||
for index, blocker in enumerate(blockers):
|
||||
if not {"gate", "status", "reason"}.issubset(blocker): errors.append(f"blockers[{index}]:required")
|
||||
return errors
|
||||
|
||||
|
||||
def run_command(name: str, args: list[str], *, cwd: Path = ROOT, timeout: int = 300) -> dict:
|
||||
if shutil.which(args[0]) is None:
|
||||
return {"passed": False, "status": "unavailable", "command": args, "summary": f"{args[0]} not installed"}
|
||||
try:
|
||||
completed = subprocess.run(args, cwd=cwd, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"passed": False, "status": "timeout", "command": args, "summary": f"{name} exceeded {timeout}s"}
|
||||
output = completed.stdout or ""
|
||||
match = re.search(r"Ran (\d+) tests?", output)
|
||||
summary = f"exit={completed.returncode}"
|
||||
if match: summary += f", tests={match.group(1)}"
|
||||
if completed.returncode and output:
|
||||
summary += ": " + " ".join(output.strip().splitlines()[-2:])[:500]
|
||||
return {"passed": completed.returncode == 0, "status": "passed" if completed.returncode == 0 else "failed", "command": args, "summary": summary}
|
||||
|
||||
|
||||
def validate_json_files() -> dict:
|
||||
files = [ROOT / "docs" / "benchmarks" / "phase16.latest.json", ROOT / "docs" / "benchmarks" / "phase16.schema.json", SCHEMA]
|
||||
results = {}
|
||||
parsed = {}
|
||||
for path in files:
|
||||
key = str(path.relative_to(ROOT))
|
||||
try:
|
||||
parsed[key] = json.loads(path.read_text(encoding="utf-8"))
|
||||
results[key] = {"passed": True, "status": "valid_json"}
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
results[key] = {"passed": False, "status": "invalid_json", "summary": str(exc)}
|
||||
phase16 = parsed.get("docs/benchmarks/phase16.latest.json")
|
||||
if isinstance(phase16, dict):
|
||||
required = {"benchmark", "version", "offline", "limitations", "acceptance_thresholds", "results", "checks", "passed"}
|
||||
schema_ok = required.issubset(phase16) and phase16.get("benchmark") == "phase16" and phase16.get("version") == 1 and phase16.get("offline") is True and isinstance(phase16.get("passed"), bool)
|
||||
results["phase16_schema_contract"] = {"passed": schema_ok, "status": "schema_valid" if schema_ok else "schema_invalid"}
|
||||
else:
|
||||
results["phase16_schema_contract"] = {"passed": False, "status": "schema_invalid"}
|
||||
return {"passed": all(item["passed"] for item in results.values()), "files": results}
|
||||
|
||||
|
||||
def safety_checks() -> dict:
|
||||
compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8")
|
||||
config = (API / "app" / "config.py").read_text(encoding="utf-8")
|
||||
main = (API / "app" / "main.py").read_text(encoding="utf-8")
|
||||
tests = "\n".join(p.read_text(encoding="utf-8") for p in (API / "tests").glob("test_*.py"))
|
||||
checks = {
|
||||
"outreach_disabled": 'AUTOMATED_OUTREACH_ENABLED: "false"' in compose and '"false"' in config and 'outreach_enabled' in main,
|
||||
"no_send_network": '"network_send": False' in main and "urllib.request" not in main and "smtplib" not in main,
|
||||
"tenant_routes": "organization_id=?" in main and "session_user" in main and "require_auth" in main and "tenant" in tests.lower(),
|
||||
}
|
||||
return {"passed": all(checks.values()), "checks": checks}
|
||||
|
||||
|
||||
def git_state() -> dict:
|
||||
result = run_command("git-state", ["git", "status", "--short", "--branch"])
|
||||
head = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True, capture_output=True, check=False)
|
||||
porcelain = subprocess.run(["git", "status", "--porcelain"], cwd=ROOT, text=True, capture_output=True, check=False)
|
||||
return {
|
||||
"passed": result["passed"] and head.returncode == 0 and bool(head.stdout.strip()),
|
||||
"status": result["status"],
|
||||
"branch_status": result.get("summary", ""),
|
||||
"head": head.stdout.strip(),
|
||||
"working_tree_clean": porcelain.returncode == 0 and not porcelain.stdout.strip(),
|
||||
}
|
||||
|
||||
|
||||
def build_report() -> dict:
|
||||
checks = {
|
||||
"api_tests": run_command("api-tests", [sys.executable, "-m", "unittest", "discover", "-v", "-s", "apps/api/tests", "-t", "apps/api"]),
|
||||
"py_compile": run_command("py_compile", [sys.executable, "-m", "py_compile", *[str(p.relative_to(ROOT)) for p in sorted((ROOT / "scripts").glob("*.py"))], *[str(p.relative_to(ROOT)) for p in sorted((API / "app").glob("*.py"))]]),
|
||||
"benchmark_no_latency": run_command("benchmark-no-latency", [sys.executable, "scripts/benchmark_phase16.py", "--no-latency", "--output", "/tmp/prospect-phase16-acceptance.json"]),
|
||||
"shell_syntax": run_command("shell-syntax", ["bash", "-n", *[str(p.relative_to(ROOT)) for p in sorted((ROOT / "scripts").glob("*.sh"))]]),
|
||||
"json_validation": validate_json_files(),
|
||||
"compose_config": run_command("compose-config", ["docker", "compose", "-f", "docker-compose.yml", "config", "--quiet"]),
|
||||
"git_state": git_state(),
|
||||
"safety_invariants": safety_checks(),
|
||||
}
|
||||
blockers = [
|
||||
{"gate": "remote_auth", "status": "blocked", "reason": "Remote repository authentication and branch permission are not available to this local acceptance run."},
|
||||
{"gate": "deployment_access", "status": "blocked", "reason": "No production host, Docker/Compose, DNS/TLS, secrets, or deployment access is available; no deployment was attempted."},
|
||||
{"gate": "real_provider_legal", "status": "blocked", "reason": "Real source/provider enablement, consent/legal basis, terms, and operational approval remain explicit gates; outreach stays disabled."},
|
||||
]
|
||||
return {
|
||||
"report": "phase17-final-acceptance", "version": 1,
|
||||
"scope": "local-repository-acceptance",
|
||||
"passed": all(item.get("passed", False) for item in checks.values()),
|
||||
"checks": checks,
|
||||
"capacity_smoke": build_capacity_smoke(),
|
||||
"blockers": blockers,
|
||||
"limitations": ["Synthetic capacity smoke measurements do not establish production throughput, concurrency, durability, or availability."],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, default=REPORT)
|
||||
args = parser.parse_args()
|
||||
report = build_report()
|
||||
errors = validate_report(report)
|
||||
if errors: raise SystemExit("invalid acceptance report: " + ", ".join(errors))
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(deterministic_json(report), encoding="utf-8")
|
||||
print(deterministic_json(report), end="")
|
||||
return 0 if report["passed"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
URL=${HEALTHCHECK_URL:-http://127.0.0.1:8000/api/v1/health/ready}
|
||||
TIMEOUT=${HEALTHCHECK_TIMEOUT:-5}
|
||||
python3 - "$URL" "$TIMEOUT" <<'PY'
|
||||
import json, sys, urllib.request
|
||||
url, timeout = sys.argv[1], float(sys.argv[2])
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=timeout) as response:
|
||||
payload=json.loads(response.read())
|
||||
if response.status != 200 or payload.get('status') != 'ok' or payload.get('ready') is not True:
|
||||
raise RuntimeError('service is not ready')
|
||||
if payload.get('outreach_enabled') is not False: raise RuntimeError('outreach safety check failed')
|
||||
except Exception as exc:
|
||||
print(f'healthcheck failed: {type(exc).__name__}', file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print('ok')
|
||||
PY
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
umask 077
|
||||
if [ "${3:-}" != "--confirm-restore" ]; then
|
||||
echo 'Refusing restore: pass --confirm-restore explicitly.' >&2
|
||||
exit 2
|
||||
fi
|
||||
SOURCE=${1:-}
|
||||
TARGET=${2:-${PROSPECT_API_DB:-${DATA_DIR:-/data}/prospects.db}}
|
||||
BACKUP_DIR=${BACKUP_DIR:-$(dirname -- "$SOURCE")}
|
||||
case "$SOURCE$TARGET" in *secret*|*credentials*|*.env*) echo 'refusing secret-like path' >&2; exit 2;; esac
|
||||
[ -f "$SOURCE" ] || { echo 'restore source does not exist' >&2; exit 2; }
|
||||
case "$TARGET" in /*) ;; *) echo 'restore target must be an absolute path' >&2; exit 2;; esac
|
||||
mkdir -p -- "$BACKUP_DIR"
|
||||
if [ -f "$TARGET" ]; then
|
||||
pre_restore=$("$(dirname -- "$0")/backup_sqlite.sh" "$TARGET" "$BACKUP_DIR" 30)
|
||||
pre_restore_db=$(printf '%s\n' "$pre_restore" | tail -n 1)
|
||||
pre_restore_name="$BACKUP_DIR/pre-restore-$(date -u +%Y%m%dT%H%M%SZ)-$$.db"
|
||||
mv -- "$pre_restore_db" "$pre_restore_name"
|
||||
if [ -f "$pre_restore_db.sha256" ]; then mv -- "$pre_restore_db.sha256" "$pre_restore_name.sha256"; fi
|
||||
fi
|
||||
python3 - "$SOURCE" "$TARGET" <<'PY'
|
||||
import hashlib, os, sqlite3, sys, tempfile
|
||||
from pathlib import Path
|
||||
source, target = Path(sys.argv[1]).resolve(), Path(sys.argv[2]).resolve()
|
||||
if not source.is_file(): raise SystemExit('restore source must be a regular file')
|
||||
sidecar = Path(str(source)+'.sha256')
|
||||
if sidecar.exists():
|
||||
expected = sidecar.read_text(encoding='ascii').split()[0]
|
||||
actual = hashlib.sha256(source.read_bytes()).hexdigest()
|
||||
if expected != actual: raise SystemExit('restore checksum mismatch')
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
fd, tmp = tempfile.mkstemp(prefix='.restore-', suffix='.tmp', dir=target.parent); os.close(fd)
|
||||
try:
|
||||
with sqlite3.connect(f'file:{source}?mode=ro', uri=True) as src, sqlite3.connect(tmp) as dst:
|
||||
src.backup(dst); result = dst.execute('PRAGMA integrity_check').fetchone()[0]
|
||||
if result != 'ok': raise SystemExit('restored database integrity check failed')
|
||||
dst.commit()
|
||||
os.chmod(tmp, 0o600); os.replace(tmp, target)
|
||||
print(target)
|
||||
finally:
|
||||
Path(tmp).unlink(missing_ok=True)
|
||||
PY
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cat >&2 <<'EOF'
|
||||
Rollback is a documented, non-destructive procedure. No rollback command is executed.
|
||||
|
||||
1. Identify the last known-good immutable image digest/tag and configuration revision.
|
||||
2. Confirm the database backup is recent and run scripts/healthcheck.sh against the candidate.
|
||||
3. Review the exact rendered config: docker compose config.
|
||||
4. Change the deployment's image tag/digest in the deployment system (or pin IMAGE_TAG), then restart the service through the approved change process.
|
||||
5. Verify /api/v1/health/ready, representative authenticated reads, logs, and outreach_enabled=false.
|
||||
6. Record the rollback reason, old/new image digests, config revision, backup/checksum, and operator.
|
||||
|
||||
This helper intentionally does not stop containers, delete images, restore databases, or alter production state.
|
||||
EOF
|
||||
@@ -0,0 +1,27 @@
|
||||
[Unit]
|
||||
Description=Prospect Platform API (portable systemd/Virtualmin example)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=prospect
|
||||
Group=prospect
|
||||
WorkingDirectory=/srv/prospect-platform/apps/api
|
||||
EnvironmentFile=-/etc/prospect-platform/prospect.env
|
||||
ExecStart=/usr/bin/python3 /srv/prospect-platform/apps/api/app/main.py --host 127.0.0.1 --port 8000 --db /var/lib/prospect-platform/prospects.db
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadWritePaths=/var/lib/prospect-platform
|
||||
UMask=0077
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
# Virtualmin: create the prospect user/domain, place the EnvironmentFile outside
|
||||
# public_html, and use this unit (or its ExecStart/Restart settings) as the
|
||||
# service command. Put TLS/reverse proxying in the Virtualmin-managed web tier.
|
||||
Reference in New Issue
Block a user