rebuild source discovery workflows
CI / compose (push) Failing after 5m46s

This commit is contained in:
Marco0300
2026-09-04 20:52:37 +02:00
parent 3a9b553440
commit 6aefbdc1f3
18 changed files with 853 additions and 106 deletions
+10 -2
View File
@@ -29,11 +29,11 @@ Phase 3 is a human-operated prospect workflow. Operators manually create a busin
- Notes may contain sensitive information. Limit access and avoid copying secrets, credentials, or unnecessary personal data into notes or audit details.
- Audit/activity records are operational evidence of changes, not a replacement for a production-grade immutable audit service.
There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.
The current stack includes bounded source-discovery jobs, passive website scanning, contact extraction, and tenant-scoped source records. These are review-oriented evidence workflows: no automated outreach worker exists and every optional network connector remains disabled until configured, approved, tested, and enabled.
## Phase 5 source operations boundary
Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.
Source controls run through the tenant-scoped registry and bounded worker. CSV/manual references can persist normalized source records; approved public connectors remain fail-closed until their source policy, limits, health, and runtime feature requirements are satisfied. Treat every run as tenant-scoped, bounded, and auditable.
Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.
@@ -41,6 +41,14 @@ Before enabling any adapter, verify the registry entry has a stable ID/version,
Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.
### Experimental Google Browser Search connector
`google_browser_search` is an experimental, **disabled-by-default** source. It becomes eligible for network access only when the runtime environment explicitly sets `GOOGLE_BROWSER_SEARCH_ENABLED=true`; leave the variable absent or false in every default Compose/deployment environment. The connector requires a reviewed source configuration with `approved=true`, `public_access=true`, `terms_accepted=true`, and an integer `rate_limit` of 112 requests per minute. The per-process connector enforces that rate without retrying or bypassing the wait window; run-level and source quota limits still apply.
Discovery text is built only from the passed, tenant-scoped discovery criteria (keywords/category/location), never from the source configuration. Do not store `query`, `keywords`, `city`, `location`, or other discovery criteria in this source configuration. The connector fetches only the public `https://www.google.com/search` result HTML with a bounded request/response and extracts visible result headings/links. It does not automate a browser, execute JavaScript, log in, send cookies, use a proxy, rotate identity, solve CAPTCHA, call private endpoints, or fall back to another search provider.
If the public page indicates CAPTCHA, bot detection, unusual traffic, access denial, a transport denial, an oversized response, or a rate-limit stop, the connector fails closed with structured `GOOGLE_BROWSER_BLOCKED` metadata. The source is marked blocked/circuit-open and the source-discovery job terminates with error code `GOOGLE_BROWSER_BLOCKED`; operators must stop and investigate/obtain approval rather than retrying, bypassing, or routing around the block.
## Phase 6 normalization and deduplication operations
Normalization and duplicate review are data-integrity operations, not discovery. Record the normalization and algorithm versions with every derived SA phone/location value and suggestion. Verify that local South African phone forms are interpreted only with explicit `+27` context, that original values remain available, and that ambiguous locations are flagged rather than guessed. Re-running the same input/version must produce the same canonical values, score, band, and reasons.