rebuild source discovery workflows
CI / compose (push) Failing after 5m46s

This commit is contained in:
Marco0300
2026-09-04 20:52:37 +02:00
parent 3a9b553440
commit 6aefbdc1f3
18 changed files with 853 additions and 106 deletions
+49
View File
@@ -0,0 +1,49 @@
# Evidence-grounded AI opportunity assessment
## Route and safety boundary
`POST /api/v1/businesses/{business_id}/ai/opportunity-assessment`
The URL ID is an explicit operator selection and is resolved only in the caller's tenant. The request body must be `{}`. The business must first meet the deterministic opportunity-score threshold of `70`; otherwise the API returns `409` / `deterministic_threshold_not_met`. An unavailable approved AI/provider configuration returns `409` / `ai_provider_not_configured` with no assessment output.
This route is **review-only**. It creates no outreach draft, pipeline change, message, browser action, or network send. Every response records `network_send: false` and `automatic_outreach: false`.
## Exact assessment response
The assessment JSON has this exact shape:
```json
{
"opportunity_score": 0,
"confidence_score": 0,
"recommendation": "contact|review|low_priority|do_not_contact|insufficient_evidence",
"priority": "high|medium|low",
"reasons": [],
"missing_evidence": [],
"website_assessment": {
"status": "healthy|outdated|broken|missing|parked|unknown",
"broken": false,
"outdated": false,
"mobile_issue": false,
"https_issue": false,
"performance_issue": false
},
"domain_assessment": {
"status": "registered|missing|likely_available|unknown"
},
"contactability": {
"public_business_contact_found": false,
"contact_type": "none|general_business|named_business|free_mail|unknown"
},
"recommended_services": [],
"human_review_required": true,
"evidence_references": []
}
```
- Opportunity and confidence scores are independent integers from 0100.
- The endpoint is an internal decision surface: a `contact` recommendation is never permission to send outreach.
- Unknown/missing values normalize conservatively. Weak evidence (fewer than two stored evidence references), confidence below 70, or `insufficient_evidence` forces human review.
- Evidence references must be unique stored evidence IDs belonging to that exact business and tenant. Unknown IDs reject the provider output rather than being silently dropped. The run separately stores the schema version and evidence hashes in `ai_runs`.
- Active suppression always wins: recommendation becomes `do_not_contact`, public contactability becomes false/`none`, and human review remains required.
- Provider credentials remain write-only and server-side under the existing approved configuration path. The currently implemented deterministic assessment path makes no network request; any remote provider integration must use the same strict normalizer and evidence bundle.
+10 -2
View File
@@ -29,11 +29,11 @@ Phase 3 is a human-operated prospect workflow. Operators manually create a busin
- Notes may contain sensitive information. Limit access and avoid copying secrets, credentials, or unnecessary personal data into notes or audit details.
- Audit/activity records are operational evidence of changes, not a replacement for a production-grade immutable audit service.
There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.
The current stack includes bounded source-discovery jobs, passive website scanning, contact extraction, and tenant-scoped source records. These are review-oriented evidence workflows: no automated outreach worker exists and every optional network connector remains disabled until configured, approved, tested, and enabled.
## Phase 5 source operations boundary
Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.
Source controls run through the tenant-scoped registry and bounded worker. CSV/manual references can persist normalized source records; approved public connectors remain fail-closed until their source policy, limits, health, and runtime feature requirements are satisfied. Treat every run as tenant-scoped, bounded, and auditable.
Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.
@@ -41,6 +41,14 @@ Before enabling any adapter, verify the registry entry has a stable ID/version,
Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.
### Experimental Google Browser Search connector
`google_browser_search` is an experimental, **disabled-by-default** source. It becomes eligible for network access only when the runtime environment explicitly sets `GOOGLE_BROWSER_SEARCH_ENABLED=true`; leave the variable absent or false in every default Compose/deployment environment. The connector requires a reviewed source configuration with `approved=true`, `public_access=true`, `terms_accepted=true`, and an integer `rate_limit` of 112 requests per minute. The per-process connector enforces that rate without retrying or bypassing the wait window; run-level and source quota limits still apply.
Discovery text is built only from the passed, tenant-scoped discovery criteria (keywords/category/location), never from the source configuration. Do not store `query`, `keywords`, `city`, `location`, or other discovery criteria in this source configuration. The connector fetches only the public `https://www.google.com/search` result HTML with a bounded request/response and extracts visible result headings/links. It does not automate a browser, execute JavaScript, log in, send cookies, use a proxy, rotate identity, solve CAPTCHA, call private endpoints, or fall back to another search provider.
If the public page indicates CAPTCHA, bot detection, unusual traffic, access denial, a transport denial, an oversized response, or a rate-limit stop, the connector fails closed with structured `GOOGLE_BROWSER_BLOCKED` metadata. The source is marked blocked/circuit-open and the source-discovery job terminates with error code `GOOGLE_BROWSER_BLOCKED`; operators must stop and investigate/obtain approval rather than retrying, bypassing, or routing around the block.
## Phase 6 normalization and deduplication operations
Normalization and duplicate review are data-integrity operations, not discovery. Record the normalization and algorithm versions with every derived SA phone/location value and suggestion. Verify that local South African phone forms are interpreted only with explicit `+27` context, that original values remain available, and that ambiguous locations are flagged rather than guessed. Re-running the same input/version must produce the same canonical values, score, band, and reasons.