root/MarketingTool
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

add criteria-first discovery search provider

Browse Source
This commit is contained in:
Marco0300
2026-09-03 20:04:08 +02:00
parent e3257f1ce9
commit 725ef0db9f
8 changed files with 202 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4
View File
@@ -78,6 +78,10 @@ Compose passes the one-time `BOOTSTRAP_ADMIN_EMAIL` and `BOOTSTRAP_ADMIN_PASSWOR
Authenticated browser requests use a server-side session cookie; login creates a session and logout invalidates it. The liveness endpoints (`GET /api/v1/health/live` and `GET /healthz`) intentionally remain unauthenticated so Docker, ingress, and monitoring health checks can use them. Authentication is not a substitute for tenant/authorization checks: protected routes must enforce the session and organization boundary server-side.
### Criteria-first discovery
`POST /api/v1/discovery` accepts a bounded `criteria` object and may omit `seed_urls`. In that mode, the API calls the configured generic JSON search provider, bounds returned URLs to at most 50, and feeds them into the existing SSRF-checked, same-origin crawler and tenant-scoped persistence. Explicit `seed_urls` (1–5 public URLs) remain supported for controlled runs. Without a provider, the criteria-first request returns `503 {"error":"not_configured"}`. Check `GET /api/v1/discovery/provider-status` for redacted readiness. Configure `SEARCH_PROVIDER_URL` (HTTPS endpoint), `SEARCH_PROVIDER_ALLOWED_HOSTS` (comma-separated exact hostname allowlist containing the endpoint host), and optional `SEARCH_PROVIDER_API_KEY`; unsafe or missing configuration is never called. The endpoint must return JSON `{"results":[{"url":"https://example.test"}]}` (or `items`/`website`/`link` equivalents).
## Phase 5 source boundary and remaining limitations
Phase 5 defines a source adapter contract and registry; Phase 8 adds a bounded website-observation adapter, but it does not implement general network discovery, enrichment scheduling, or a live external-source adapter. A source adapter must declare its identity, terms owner, permitted purpose, rate limits, retention class, query/result schema, dry-run behavior, and health/circuit controls. CSV and manual reference adapters may be used for operator-supplied data; they must preserve source attribution and raw source records, and must not silently turn preview data into outreach or verified facts.