add policy-aware source adapter framework
This commit is contained in:
@@ -31,6 +31,16 @@ Phase 3 is a human-operated prospect workflow. Operators manually create a busin
|
||||
|
||||
There is no automated discovery job, DNS/website scanner, enrichment worker, or outreach worker to monitor in this release. CSV is preview-only; do not describe a preview as an import or assume that rows were persisted.
|
||||
|
||||
## Phase 5 source operations boundary
|
||||
|
||||
Phase 5 source controls are contract/runbook requirements; the current Compose stack has no network discovery worker or live external-source adapter. Operators may use CSV/manual reference workflows and dry-run discovery plans only. Treat every query as tenant-scoped, bounded, and auditable.
|
||||
|
||||
Before enabling any adapter, verify the registry entry has a stable ID/version, terms owner and review expiry, permitted purpose, tenant scope, rate/concurrency limits, timeout/size/retry policy, raw-record retention class, and health/circuit thresholds. Record product/legal/security approval and a separate operational enablement decision. If any item is missing or expired, keep the adapter disabled; do not substitute a URL or scrape command.
|
||||
|
||||
`dry_run` must perform validation/planning only: no network I/O, external adapter side effects, prospect-fact writes, or outreach. CSV and manual references may be previewed or recorded as operator-supplied observations with source attribution and capture time. A preview is not an import, verification, or discovery result.
|
||||
|
||||
Monitor per-source request counts, rate-limit responses, latency, errors, circuit state, and raw-record retention/deletion outcomes. On rate-limit, terms, approval, or circuit-open conditions, fail closed, preserve a safe audit event, and report deferred/unavailable rather than an empty result. Do not retry through another source or reset a circuit manually without an approved incident/change record. The current stack has no live source to monitor; these controls must precede any future implementation.
|
||||
|
||||
## Phase 4 jobs and live logging
|
||||
|
||||
The Phase 4 MVP provides SQLite-backed job status/detail/event routes and a browser monitor. A job moves `queued` → `running` → `succeeded`/`failed`/`cancelled`, retains its attempt and tenant identity, and appends per-job events with a monotonic sequence cursor. Operators inspect status and replay events by polling; SSE may provide lower-latency delivery but is not implemented and must replay from the persisted cursor and fall back to polling after disconnects.
|
||||
|
||||
@@ -3,6 +3,10 @@
|
||||
## Current safety boundary
|
||||
|
||||
- **Automated outreach is disabled.** The Compose file sets `AUTOMATED_OUTREACH_ENABLED=false` for both services. The MVP sends no email, SMS, or other outbound communication.
|
||||
- Phase 5 source handling is **registered, approval-gated, and fail-closed**. The current runtime has no live network source or network discovery implementation. CSV and manual reference adapters are operator-supplied only; neither proves a fact or grants contact permission.
|
||||
- Source registry entries must have an accountable terms owner, permitted purpose, approval status/expiry, rate/concurrency limits, retention class, and health/circuit policy before an adapter can be enabled.
|
||||
- Discovery queries must be tenant-scoped, bounded, auditable, and explicit about execution mode. `dry_run` validates/plans without contacting a source or writing prospect facts. No live source may run without recorded product/legal/security approval and explicit operational enablement.
|
||||
- Raw source records are sensitive lineage data: retain only the minimum needed to reproduce a normalized result, under the approved retention class, with tenant/source/query IDs, capture time, adapter version, and redaction metadata. Never store secrets or unnecessary personal/contact data.
|
||||
- Phase 3 intelligence is **manual and provenance-first**. Operators enter child intelligence/evidence records; the platform does not perform automated prospect discovery, DNS resolution, website/HTTP scanning, or external enrichment.
|
||||
- Every business, child record, note, pipeline transition, and audit/activity read or write must be constrained to the authenticated user's organization. A child identifier must never bypass the parent/tenant check. Cross-tenant misses should be indistinguishable from an absent record.
|
||||
- Evidence provenance (source reference/label, captured or observed time, actor, and confidence where supported) is data lineage, not proof that the platform independently verified the source. Do not fabricate provenance or silently upgrade an observation to a verified fact.
|
||||
@@ -14,6 +18,19 @@
|
||||
- Phase 4 job/live-log controls are not enabled in the current runtime. If added, job IDs, idempotency keys, status, cancellation, retries, and event cursors must all be authorized against the authenticated organization; never accept a job or child identifier as authorization by itself.
|
||||
- Persisted job events must be append-only, sequence-ordered per job, replayable from a cursor, and redacted to safe operational data. Never emit credentials, session cookies, API keys, full request bodies, or unnecessary contact/prospect data in polling responses, SSE frames, logs, or error details.
|
||||
|
||||
## Phase 5 source security controls
|
||||
|
||||
Source adapters are a security boundary, not a generic fetch facility. Registry review must verify the source identity, terms/robots and licensing owner, permitted collection purpose, approval expiry, tenant scope, rate/concurrency budget, raw-record retention/deletion policy, and circuit thresholds. Keep these controls server-side and auditable; a UI flag or client-supplied source ID is not authorization.
|
||||
|
||||
- `dry_run` is the safe default: validate a bounded discovery query and produce a plan without network I/O, adapter side effects, or prospect-fact writes.
|
||||
- CSV and manual reference adapters accept operator-supplied material only. Preserve citation/reference, actor, capture time, adapter/version, and normalization lineage; label it as supplied/observed rather than verified.
|
||||
- Raw source records must be access-controlled, tenant-scoped, minimally retained, immutable enough for replay/audit, and redacted for secrets and unnecessary personal data. Apply the approved retention class and deletion schedule.
|
||||
- Enforce per-source request, concurrency, byte, timeout, and retry limits. Rate-limit responses must not be bypassed by rotating identities or silently selecting another source.
|
||||
- Health controls must record success/failure/latency signals and use a circuit breaker with `closed`, `open`, and guarded `half-open` states. Open circuits fail closed, suppress live attempts, and surface a safe deferred/unavailable outcome.
|
||||
- No live network source is allowed without explicit product, legal, and security approval plus operational enablement of the registered adapter. Approval must be checked at execution time and expire safely.
|
||||
|
||||
If a future approved adapter fetches URLs, apply the SSRF requirements below in addition to source approval. Network discovery is not implemented by this documentation or by the current Compose stack.
|
||||
|
||||
## Known limitations before production
|
||||
|
||||
1. **Password storage:** production passwords must be hashed with Argon2id using a reviewed cost/memory/parallelism policy. Never store plaintext or reversible passwords, and never log bootstrap credentials. Rehash on login when the policy changes.
|
||||
|
||||
Reference in New Issue
Block a user