MarketingTool/docs/source-discovery-rebuild-plan.md

Source and Discovery Rebuild Plan

Audit snapshot

• Frontend: apps/web/index.html, apps/web/app.js, apps/web/styles.css; static same-origin delivery via apps/web/server.py.
• Backend: threaded Python/SQLite HTTP API in apps/api/app/main.py; adapters in apps/api/app/sources.py; bounded crawler/discovery in apps/api/app/discovery.py.
• Persistence: additive SQLite schema in apps/api/schema.sql, with compatibility upgrades in connect().
• Background work: in-process source_discovery and scoped_discovery job worker plus schedule worker in main.py.
• Existing boundaries to preserve: tenant filtering, role checks, write-only provider credentials, suppression precedence, deterministic scoring, CRM, evidence, no automated outreach, and same-origin web/API delivery.

Acceptance slices

1. Reusable source registry

   • Separate connector configuration from discovery criteria.
   • Add safe migration fields for owner, terms, rate/quota, credential status, and connector configuration.
   • Make duplicate handling identity-based and return the existing registered source visibly.
   • Keep previews separate from real persisted source IDs.

2. Connector contract and safe adapters

   • Normalize connector methods around configuration validation, connection tests, criteria-based discovery, health, limits, normalized evidence, and structured errors.
   • Preserve CSV/manual behavior and repair OSM/Wikidata/Common Crawl/CT/RDAP/DNS/public crawler connectors.
   • Add a disabled-by-default Google Browser Search connector that reports challenge/blocked states without bypassing controls. Runtime/dependency requirements will be documented and feature-gated.

3. Criteria-based discovery runs

   • Persist explicit criteria independently from source configuration.
   • Select only enabled, valid sources; validate limits and dry-runs.
   • Pass criteria and limits to connectors, retain source-level result/error status, record provenance, deduplicate candidates, and distinguish complete, partial, blocked, and failed outcomes.

4. Evidence/enrichment and AI boundaries

   • Preserve bounded website/domain/contact checks and provenance.
   • Keep AI evidence-driven, strict-schema, write-only-provider-configured, suppression-overridden, and review-only.

5. Sources and Discovery UI

   • Rebuild the registry, source details, source setup, criteria builder, run history, live logs, per-source status, candidate provenance, and actionable empty/error states.
   • Every action presents loading, success, validation, or API-error feedback; no fake IDs, silent actions, demo health, or fabricated records.

6. Verification, documentation, and deployment

   • Add migration, source identity, criteria, connector mock, dedupe/provenance, blocked/partial, Google challenge, AI schema, and suppression tests.
   • Run backend/full/frontend/desktop/Compose checks and live authenticated workflow checks where available.
   • Update deployment/source architecture documentation, deploy only after data backup and verified live results.

Known operational limits

• SQLite/in-process jobs remain a controlled single-node/pilot architecture.
• Public connectors must remain bounded and may return an honest blocked/network/rate-limit result.
• Google Browser Search remains disabled unless its reviewed browser runtime is installed and explicitly enabled; it will never bypass CAPTCHA, authentication, or anti-bot controls.