47 lines
3.4 KiB
Markdown
47 lines
3.4 KiB
Markdown
|
# Source and Discovery Rebuild Plan
|
||
|
|
|
||
|
|
## Audit snapshot
|
||
|
|
|
||
|
|
- **Frontend:** `apps/web/index.html`, `apps/web/app.js`, `apps/web/styles.css`; static same-origin delivery via `apps/web/server.py`.
|
||
|
|
- **Backend:** threaded Python/SQLite HTTP API in `apps/api/app/main.py`; adapters in `apps/api/app/sources.py`; bounded crawler/discovery in `apps/api/app/discovery.py`.
|
||
|
|
- **Persistence:** additive SQLite schema in `apps/api/schema.sql`, with compatibility upgrades in `connect()`.
|
||
|
|
- **Background work:** in-process `source_discovery` and `scoped_discovery` job worker plus schedule worker in `main.py`.
|
||
|
|
- **Existing boundaries to preserve:** tenant filtering, role checks, write-only provider credentials, suppression precedence, deterministic scoring, CRM, evidence, no automated outreach, and same-origin web/API delivery.
|
||
|
|
|
||
|
|
## Acceptance slices
|
||
|
|
|
||
|
|
1. **Reusable source registry**
|
||
|
|
- Separate connector configuration from discovery criteria.
|
||
|
|
- Add safe migration fields for owner, terms, rate/quota, credential status, and connector configuration.
|
||
|
|
- Make duplicate handling identity-based and return the existing registered source visibly.
|
||
|
|
- Keep previews separate from real persisted source IDs.
|
||
|
|
|
||
|
|
2. **Connector contract and safe adapters**
|
||
|
|
- Normalize connector methods around configuration validation, connection tests, criteria-based discovery, health, limits, normalized evidence, and structured errors.
|
||
|
|
- Preserve CSV/manual behavior and repair OSM/Wikidata/Common Crawl/CT/RDAP/DNS/public crawler connectors.
|
||
|
|
- Add a disabled-by-default Google Browser Search connector that reports challenge/blocked states without bypassing controls. Runtime/dependency requirements will be documented and feature-gated.
|
||
|
|
|
||
|
|
3. **Criteria-based discovery runs**
|
||
|
|
- Persist explicit criteria independently from source configuration.
|
||
|
|
- Select only enabled, valid sources; validate limits and dry-runs.
|
||
|
|
- Pass criteria and limits to connectors, retain source-level result/error status, record provenance, deduplicate candidates, and distinguish complete, partial, blocked, and failed outcomes.
|
||
|
|
|
||
|
|
4. **Evidence/enrichment and AI boundaries**
|
||
|
|
- Preserve bounded website/domain/contact checks and provenance.
|
||
|
|
- Keep AI evidence-driven, strict-schema, write-only-provider-configured, suppression-overridden, and review-only.
|
||
|
|
|
||
|
|
5. **Sources and Discovery UI**
|
||
|
|
- Rebuild the registry, source details, source setup, criteria builder, run history, live logs, per-source status, candidate provenance, and actionable empty/error states.
|
||
|
|
- Every action presents loading, success, validation, or API-error feedback; no fake IDs, silent actions, demo health, or fabricated records.
|
||
|
|
|
||
|
|
6. **Verification, documentation, and deployment**
|
||
|
|
- Add migration, source identity, criteria, connector mock, dedupe/provenance, blocked/partial, Google challenge, AI schema, and suppression tests.
|
||
|
|
- Run backend/full/frontend/desktop/Compose checks and live authenticated workflow checks where available.
|
||
|
|
- Update deployment/source architecture documentation, deploy only after data backup and verified live results.
|
||
|
|
|
||
|
|
## Known operational limits
|
||
|
|
|
||
|
|
- SQLite/in-process jobs remain a controlled single-node/pilot architecture.
|
||
|
|
- Public connectors must remain bounded and may return an honest blocked/network/rate-limit result.
|
||
|
|
- Google Browser Search remains disabled unless its reviewed browser runtime is installed and explicitly enabled; it will never bypass CAPTCHA, authentication, or anti-bot controls.
|