# Source and Discovery Rebuild Plan

## Audit snapshot

- **Frontend:** `apps/web/index.html`, `apps/web/app.js`, `apps/web/styles.css`; static same-origin delivery via `apps/web/server.py`.
- **Backend:** threaded Python/SQLite HTTP API in `apps/api/app/main.py`; adapters in `apps/api/app/sources.py`; bounded crawler/discovery in `apps/api/app/discovery.py`.
- **Persistence:** additive SQLite schema in `apps/api/schema.sql`, with compatibility upgrades in `connect()`.
- **Background work:** in-process `source_discovery` and `scoped_discovery` job worker plus schedule worker in `main.py`.
- **Existing boundaries to preserve:** tenant filtering, role checks, write-only provider credentials, suppression precedence, deterministic scoring, CRM, evidence, no automated outreach, and same-origin web/API delivery.

## Acceptance slices

1. **Reusable source registry**
   - Separate connector configuration from discovery criteria.
   - Add safe migration fields for owner, terms, rate/quota, credential status, and connector configuration.
   - Make duplicate handling identity-based and return the existing registered source visibly.
   - Keep previews separate from real persisted source IDs.

2. **Connector contract and safe adapters**
   - Normalize connector methods around configuration validation, connection tests, criteria-based discovery, health, limits, normalized evidence, and structured errors.
   - Preserve CSV/manual behavior and repair OSM/Wikidata/Common Crawl/CT/RDAP/DNS/public crawler connectors.
   - Add a disabled-by-default Google Browser Search connector that reports challenge/blocked states without bypassing controls. Runtime/dependency requirements will be documented and feature-gated.

3. **Criteria-based discovery runs**
   - Persist explicit criteria independently from source configuration.
   - Select only enabled, valid sources; validate limits and dry-runs.
   - Pass criteria and limits to connectors, retain source-level result/error status, record provenance, deduplicate candidates, and distinguish complete, partial, blocked, and failed outcomes.

4. **Evidence/enrichment and AI boundaries**
   - Preserve bounded website/domain/contact checks and provenance.
   - Keep AI evidence-driven, strict-schema, write-only-provider-configured, suppression-overridden, and review-only.

5. **Sources and Discovery UI**
   - Rebuild the registry, source details, source setup, criteria builder, run history, live logs, per-source status, candidate provenance, and actionable empty/error states.
   - Every action presents loading, success, validation, or API-error feedback; no fake IDs, silent actions, demo health, or fabricated records.

6. **Verification, documentation, and deployment**
   - Add migration, source identity, criteria, connector mock, dedupe/provenance, blocked/partial, Google challenge, AI schema, and suppression tests.
   - Run backend/full/frontend/desktop/Compose checks and live authenticated workflow checks where available.
   - Update deployment/source architecture documentation, deploy only after data backup and verified live results.

## Known operational limits

- SQLite/in-process jobs remain a controlled single-node/pilot architecture.
- Public connectors must remain bounded and may return an honest blocked/network/rate-limit result.
- Google Browser Search remains disabled unless its reviewed browser runtime is installed and explicitly enabled; it will never bypass CAPTCHA, authentication, or anti-bot controls.