deploy Nous v19 update

.env.example

@@ -16,6 +16,15 @@ GMAPS_POLL_SECONDS=2
 # Keep the default unless your provider asks for a different application name.
 OUTBOUND_USER_AGENT=ProspectOS/2.0 (+personal-use)
 
+# Optional Browser Use investigator for finding likely official domains that
+# Maps did not list. It uses your configured OpenAI-compatible AI provider and
+# a normal headless Chromium session; it does not rotate proxies, use stealth,
+# solve CAPTCHAs, or bypass access challenges.
+BROWSER_USE_ENABLED=true
+BROWSER_USE_MAX_STEPS=10
+BROWSER_USE_TIMEOUT_SECONDS=120
+ANONYMIZED_TELEMETRY=false
+
 # AI provider for analysis and tailored drafts: nous or custom.
 AI_PROVIDER=custom
 # For Nous, use https://portal.nousresearch.com to create an API key;

README.md

@@ -8,10 +8,21 @@ the local catalogue, and presents the remaining places for human review.
 The interface is split into Dashboard, Discover, Prospects, Runs, and Settings
 views. It includes website health and email evidence, explainable scoring,
 domain ideas, suppression, CSV export, scheduled runs, rescans, and human review. It never sends outreach
-automatically. Optional AI can analyse a prospect and draft a subject/email
+automatically. The Prospects explorer has an **All high potential** tab and
-from supplied evidence and your offer; you edit and approve the draft yourself.
+category tabs populated from each discovery run, with every tab sorted from
+highest to lowest opportunity score.
+
+Optional AI can analyse a prospect and draft a subject/email from supplied
+evidence and your offer; you edit and approve the draft yourself. Drafts now
+also return the reasoning, stored evidence IDs used, facts that still need
+confirmation, and a follow-up review note so a polished email is not mistaken
+for verified information.
 From a prospect's detail drawer, **Assess with AI** stores a contact probability,
 recommendation, and short evidence-grounded rationale for that specific company.
+The same drawer can optionally run a bounded Browser Use investigation to look
+for a likely official website when Maps did not list one. Candidate domains are
+shown with evidence and must be accepted by you before they replace the
+catalogue record.
 
 ## Start the stack
 
@@ -50,6 +61,10 @@ error rather than remaining stuck forever.
 | `GMAPS_POLL_SECONDS` | `2` | Poll interval while a job is running |
 | `OUTBOUND_USER_AGENT` | `ProspectOS/2.0 (+personal-use)` | Identifies provider HTTP requests; avoids Python's default client fingerprint |
 | `DISCOVERY_BACKEND` | `gmaps` when `GMAPS_URL` is set | Set to `browser` only for the bounded public-search fallback |
+| `BROWSER_USE_ENABLED` | `true` | Enables the optional website-candidate investigator |
+| `BROWSER_USE_MAX_STEPS` | `10` | Maximum Browser Use actions for one investigation (3–15) |
+| `BROWSER_USE_TIMEOUT_SECONDS` | `120` | Hard timeout for one investigation (30–300 seconds) |
+| `ANONYMIZED_TELEMETRY` | `false` | Keeps Browser Use telemetry disabled in this personal deployment |
 | `AI_PROVIDER` | `custom` | Set to `nous` for the Nous Research Portal, or `custom` for another OpenAI-compatible endpoint |
 | `AI_BASE_URL` | empty | Optional OpenAI-compatible `/chat/completions` endpoint |
 | `AI_API_KEY` | empty | Optional provider key; keep server-side |
@@ -82,6 +97,32 @@ Hermes refresh token and is not a portable credential for this standalone
 Docker app. ProspectOS therefore uses a server-side Nous API key here. The key
 is never returned by the Settings API or sent to the browser after saving.
 
+### Browser Use website investigator
+
+Browser Use is an optional local browser agent. It uses the AI provider already
+configured in Settings to navigate a small number of public search/result pages,
+then returns candidate URLs as structured data. It is intentionally limited to
+unauthenticated research: no proxy rotation, stealth fingerprints, CAPTCHA
+solving, login, or challenge bypass is configured. If a site blocks the browser,
+the action returns an explicit error and the rest of the prospect workflow is
+left intact. Browser Use itself is open source, but each investigation still
+uses your configured model/provider and may consume its quota.
+
+To use it:
+
+1. Configure and test an AI provider in **Settings** (a Nous key plus a selected
+   model also works).
+2. Open a prospect that has no detected website and choose **Find possible
+   websites**.
+3. Review the returned domains and evidence. Choose **Accept** only when the
+   domain is genuinely the business's official site; accepting runs the normal
+   website health and public-email scan.
+
+The API equivalents are `POST /api/prospects/{id}/website-candidates`,
+`GET /api/prospects/{id}/website-candidates`, and
+`PATCH /api/prospects/{id}/website-candidates/{candidate_id}` with
+`{"status":"accepted"}` or `{"status":"rejected"}`.
+
 The app stores its catalogue in the `prospect_data` Docker volume. The worker
 stores its own job data in `gmaps_data`; keep both volumes if you want to retain
 history. To inspect status:

api/Dockerfile

@@ -4,7 +4,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends chromium && rm
 WORKDIR /app
 COPY api/requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
-COPY api/server.py api/intelligence.py ./
+COPY api/server.py api/intelligence.py api/browser_investigator.py ./
 COPY web /app/web
 RUN useradd --system --create-home app && mkdir -p /data && chown -R app:app /app /data
 USER app

api/browser_investigator.py

@@ -0,0 +1,173 @@
+"""Optional Browser Use investigator for public website candidates.
+
+This module is deliberately isolated and imports browser-use only when called.
+It performs bounded, headless public research. It does not log in, solve
+CAPTCHAs, rotate proxies, or continue after an access challenge.
+"""
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import re
+from typing import Any
+from urllib.parse import urlparse
+
+
+_BLOCKED_HOSTS = {
+    "facebook.com", "instagram.com", "linkedin.com", "youtube.com", "x.com",
+    "twitter.com", "google.com", "google.co.za", "bing.com", "brave.com",
+    "duckduckgo.com", "startpage.com", "yelp.com", "yellowpages.com",
+}
+
+
+def _normalise_url(value: Any) -> str:
+    text = str(value or "").strip()
+    if not text:
+        return ""
+    if text.startswith("//"):
+        text = "https:" + text
+    if not re.match(r"^https?://", text, re.I):
+        text = "https://" + text
+    parsed = urlparse(text)
+    if parsed.scheme not in {"http", "https"} or not parsed.hostname:
+        return ""
+    host = parsed.hostname.lower().rstrip(".")
+    if host in {"localhost", "127.0.0.1", "::1"} or host.endswith(".local"):
+        return ""
+    if any(host == blocked or host.endswith("." + blocked) for blocked in _BLOCKED_HOSTS):
+        return ""
+    return text[:1000]
+
+
+def _parse_agent_result(raw: Any) -> list[dict[str, Any]]:
+    """Parse and normalise the agent's final JSON candidate list."""
+    text = str(raw or "").strip()
+    if text.startswith("```"):
+        text = re.sub(r"^```(?:json)?\s*|\s*```$", "", text, flags=re.I | re.S).strip()
+    try:
+        payload = json.loads(text)
+    except (TypeError, ValueError, json.JSONDecodeError):
+        start, end = text.find("{"), text.rfind("}")
+        if start < 0 or end <= start:
+            return []
+        try:
+            payload = json.loads(text[start:end + 1])
+        except (TypeError, ValueError, json.JSONDecodeError):
+            return []
+    if not isinstance(payload, dict):
+        return []
+    raw_candidates = payload.get("candidates")
+    if not isinstance(raw_candidates, list):
+        return []
+    output: list[dict[str, Any]] = []
+    seen: set[str] = set()
+    for item in raw_candidates[:10]:
+        if not isinstance(item, dict):
+            continue
+        url = _normalise_url(item.get("url") or item.get("website") or item.get("domain"))
+        if not url:
+            continue
+        domain = (urlparse(url).hostname or "").lower().removeprefix("www.")
+        if not domain or domain in seen:
+            continue
+        seen.add(domain)
+        try:
+            confidence = max(0, min(100, int(round(float(item.get("confidence", 0))))))
+        except (TypeError, ValueError):
+            confidence = 0
+        evidence = item.get("evidence")
+        if not isinstance(evidence, list):
+            evidence = []
+        output.append({
+            "url": url,
+            "domain": domain,
+            "confidence": confidence,
+            "evidence": [str(value).strip()[:240] for value in evidence if str(value).strip()][:8],
+            "status": str(item.get("status") or "candidate").strip()[:80],
+        })
+    return sorted(output, key=lambda item: item["confidence"], reverse=True)
+
+
+def investigate_websites(
+    prospect: dict[str, Any],
+    *,
+    base_url: str,
+    api_key: str,
+    model: str,
+    user_agent: str,
+) -> list[dict[str, Any]]:
+    """Run one bounded Browser Use investigation and return candidates."""
+    if os.environ.get("BROWSER_USE_ENABLED", "1").strip().lower() in {"0", "false", "no", "off"}:
+        raise RuntimeError("browser_use_disabled")
+    if not base_url or not model:
+        raise RuntimeError("browser_use_requires_ai_provider_and_model")
+    if not api_key:
+        raise RuntimeError("browser_use_requires_ai_api_key")
+    try:
+        from browser_use import Agent, BrowserProfile, ChatOpenAI
+    except ImportError as exc:
+        raise RuntimeError("browser_use_not_installed_rebuild_container") from exc
+
+    name = str(prospect.get("name") or "").strip()
+    address = str(prospect.get("address") or "").strip()
+    category = str(prospect.get("category") or "").strip()
+    phone = str(prospect.get("phone") or "").strip()
+    emails = str(prospect.get("emails") or "").strip()
+    task = f"""
+You are a bounded public-data website investigator. Find possible official websites for this one business:
+Business name: {name}
+Category/services: {category}
+Address/location: {address}
+Phone: {phone}
+Public email(s): {emails}
+
+Use only public, unauthenticated pages. Search at most three result pages and inspect at most five candidate domains.
+Try exact business-name and location/phone searches. Prefer the official business website over social networks,
+directories, review sites, and map pages. Do not log in, submit forms, solve CAPTCHAs, bypass access challenges,
+or use a proxy. If a site blocks the browser, record it and move on.
+
+Return ONLY JSON in this shape:
+{{"candidates":[{{"url":"https://example.co.za","confidence":0,"evidence":["name match","phone match"],"status":"candidate"}}]}}
+Confidence must be an integer from 0 to 100. Include only plausible official website URLs and never invent a URL.
+""".strip()
+
+    async def run() -> Any:
+        browser_profile = BrowserProfile(
+            headless=True,
+            executable_path=os.environ.get("PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH", "/usr/bin/chromium"),
+            minimum_wait_page_load_time=0.5,
+            wait_between_actions=0.7,
+            enable_default_extensions=False,
+            captcha_solver=False,
+        )
+        llm = ChatOpenAI(
+            model=model,
+            api_key=api_key,
+            base_url=base_url,
+            default_headers={"User-Agent": user_agent},
+            temperature=0.1,
+            frequency_penalty=None,
+            max_completion_tokens=900,
+            max_retries=1,
+            timeout=45,
+        )
+        agent = Agent(
+            task=task,
+            llm=llm,
+            browser_profile=browser_profile,
+            use_vision=False,
+            flash_mode=True,
+            use_judge=False,
+            enable_planning=False,
+            enable_signal_handler=False,
+        )
+        return await agent.run(max_steps=max(3, min(15, int(os.environ.get("BROWSER_USE_MAX_STEPS", "10")))))
+
+    timeout = max(30, min(300, int(os.environ.get("BROWSER_USE_TIMEOUT_SECONDS", "120"))))
+    try:
+        history = asyncio.run(asyncio.wait_for(run(), timeout=timeout))
+    except asyncio.TimeoutError as exc:
+        raise RuntimeError("browser_use_timeout") from exc
+    final = history.final_result() if history is not None and hasattr(history, "final_result") else ""
+    return _parse_agent_result(final)

api/requirements.txt

@@ -1 +1,2 @@
 playwright==1.52.0
+browser-use==0.13.10

api/server.py

@@ -27,8 +27,10 @@ from urllib.parse import parse_qs, quote_plus, unquote, urlparse
 
 try:
     from intelligence import domain_candidates, scan_website
+    from browser_investigator import investigate_websites
 except ImportError:  # tests import the module as api.server
     from api.intelligence import domain_candidates, scan_website
+    from api.browser_investigator import investigate_websites
 
 DB_PATH = os.environ.get("PROSPECT_DB", "/data/prospects.db")
 MAX_RESULTS = 50
@@ -52,6 +54,7 @@ AI_PROVIDER = os.environ.get("AI_PROVIDER", "custom").strip().lower()
 # reaches the provider. Identify this application consistently on provider
 # requests instead of relying on that default.
 OUTBOUND_USER_AGENT = os.environ.get("OUTBOUND_USER_AGENT", "ProspectOS/2.0 (+personal-use)").strip() or "ProspectOS/2.0 (+personal-use)"
+BROWSER_USE_ENABLED = os.environ.get("BROWSER_USE_ENABLED", "1").strip().lower() not in {"0", "false", "no", "off"}
 NOUS_BASE_URL = "https://inference-api.nousresearch.com/v1"
 NOUS_FREE_MODELS = [
     {"id": "meituan/longcat-2.0:free", "label": "Meituan · LongCat 2.0", "free": True},
@@ -70,6 +73,19 @@ _schema_lock = threading.Lock()
 _schema_path: str | None = None
 
 
+def browser_use_limits() -> tuple[int, int]:
+    """Return bounded Browser Use steps/timeout even if .env is malformed."""
+    try:
+        steps = int(os.environ.get("BROWSER_USE_MAX_STEPS", "10"))
+    except (TypeError, ValueError):
+        steps = 10
+    try:
+        timeout = int(os.environ.get("BROWSER_USE_TIMEOUT_SECONDS", "120"))
+    except (TypeError, ValueError):
+        timeout = 120
+    return max(3, min(15, steps)), max(30, min(300, timeout))
+
+
 def now() -> str:
     return datetime.now(timezone.utc).replace(microsecond=0).isoformat()
 
@@ -106,6 +122,7 @@ def db() -> sqlite3.Connection:
         source TEXT NOT NULL DEFAULT 'google_maps',
         source_id TEXT NOT NULL DEFAULT '',
         category TEXT NOT NULL DEFAULT '',
+        search_category TEXT NOT NULL DEFAULT '',
         address TEXT NOT NULL DEFAULT '',
         phone TEXT NOT NULL DEFAULT '',
         emails TEXT NOT NULL DEFAULT '',
@@ -135,6 +152,9 @@ def db() -> sqlite3.Connection:
         ai_subject TEXT NOT NULL DEFAULT '',
         ai_message TEXT NOT NULL DEFAULT '',
         ai_offer TEXT NOT NULL DEFAULT '',
+        ai_unknowns TEXT NOT NULL DEFAULT '[]',
+        ai_evidence_ids TEXT NOT NULL DEFAULT '[]',
+        ai_follow_up TEXT NOT NULL DEFAULT '',
         opportunity_score INTEGER NOT NULL DEFAULT 0,
         raw_json TEXT NOT NULL DEFAULT '{}'
     )""")
@@ -172,6 +192,19 @@ def db() -> sqlite3.Connection:
     conn.execute("""CREATE TABLE IF NOT EXISTS settings (
         key TEXT PRIMARY KEY, value TEXT NOT NULL DEFAULT '', updated_at TEXT NOT NULL
     )""")
+    conn.execute("""CREATE TABLE IF NOT EXISTS website_candidates (
+        id INTEGER PRIMARY KEY AUTOINCREMENT,
+        prospect_id INTEGER NOT NULL,
+        url TEXT NOT NULL,
+        domain TEXT NOT NULL,
+        confidence INTEGER NOT NULL DEFAULT 0,
+        evidence TEXT NOT NULL DEFAULT '[]',
+        source TEXT NOT NULL DEFAULT 'browser_use',
+        status TEXT NOT NULL DEFAULT 'candidate',
+        discovered_at TEXT NOT NULL,
+        raw_json TEXT NOT NULL DEFAULT '{}',
+        UNIQUE(prospect_id, domain)
+    )""")
 
     # Migrate databases created by v1-v5 without destroying the user's
     # catalogue. SQLite permits adding nullable/defaulted columns in place.
@@ -179,6 +212,7 @@ def db() -> sqlite3.Connection:
         "source": "TEXT NOT NULL DEFAULT 'google_maps'",
         "source_id": "TEXT NOT NULL DEFAULT ''",
         "category": "TEXT NOT NULL DEFAULT ''",
+        "search_category": "TEXT NOT NULL DEFAULT ''",
         "address": "TEXT NOT NULL DEFAULT ''",
         "phone": "TEXT NOT NULL DEFAULT ''",
         "emails": "TEXT NOT NULL DEFAULT ''",
@@ -208,6 +242,9 @@ def db() -> sqlite3.Connection:
         "ai_subject": "TEXT NOT NULL DEFAULT ''",
         "ai_message": "TEXT NOT NULL DEFAULT ''",
         "ai_offer": "TEXT NOT NULL DEFAULT ''",
+        "ai_unknowns": "TEXT NOT NULL DEFAULT '[]'",
+        "ai_evidence_ids": "TEXT NOT NULL DEFAULT '[]'",
+        "ai_follow_up": "TEXT NOT NULL DEFAULT ''",
         "opportunity_score": "INTEGER NOT NULL DEFAULT 0",
         "raw_json": "TEXT NOT NULL DEFAULT '{}'",
     }
@@ -230,6 +267,7 @@ def db() -> sqlite3.Connection:
     conn.execute("CREATE INDEX IF NOT EXISTS idx_prospects_review ON prospects(review_status)")
     conn.execute("CREATE INDEX IF NOT EXISTS idx_prospects_score ON prospects(opportunity_score DESC)")
     conn.execute("CREATE INDEX IF NOT EXISTS idx_runs_created ON runs(id DESC)")
+    conn.execute("CREATE INDEX IF NOT EXISTS idx_website_candidates_prospect ON website_candidates(prospect_id, confidence DESC)")
     conn.commit()
     _schema_path = DB_PATH
     _schema_lock.release()
@@ -380,7 +418,7 @@ def score_place(item: dict[str, Any], website: str, phone: str, rating: float |
     return score_opportunity(item)[0]
 
 
-def map_gmaps_result(item: dict[str, Any], query: str) -> dict[str, Any]:
+def map_gmaps_result(item: dict[str, Any], query: str, search_category: str = "") -> dict[str, Any]:
     """Normalize gosom/google-maps-scraper's place object to our catalogue."""
     name = as_text(item.get("title") or item.get("name") or item.get("business_name")) or "Unnamed place"
     website = normalize_url(item.get("website") or item.get("website_url"))
@@ -408,6 +446,7 @@ def map_gmaps_result(item: dict[str, Any], query: str) -> dict[str, Any]:
         "source": "google_maps",
         "source_id": source_id[:255],
         "category": category[:255],
+        "search_category": as_text(search_category)[:255],
         "address": address[:1000],
         "phone": phone[:120],
         "emails": emails[:500],
@@ -869,8 +908,9 @@ def enrich_place(item: dict[str, Any]) -> dict[str, Any]:
 
 
 def insert_catalogue(run_id: int, query: str, raw_results: list[dict[str, Any]], requested: int,
-                     progress: Callable[[int, int], None] | None = None) -> tuple[list[dict[str, Any]], int]:
+                     progress: Callable[[int, int], None] | None = None,
-    mapped = [map_gmaps_result(item, query) for item in raw_results[:requested]]
+                     search_category: str = "") -> tuple[list[dict[str, Any]], int]:
+    mapped = [map_gmaps_result(item, query, search_category) for item in raw_results[:requested]]
     conn = db()
     added = 0
     try:
@@ -899,14 +939,14 @@ def insert_catalogue(run_id: int, query: str, raw_results: list[dict[str, Any]],
             try:
                 cur = conn.execute("""INSERT OR IGNORE INTO prospects(
                 name,website,domain,description,query,discovered_at,source,source_id,
-                category,address,phone,emails,rating,review_count,maps_url,latitude,
+                category,search_category,address,phone,emails,rating,review_count,maps_url,latitude,
                 longitude,listing_status,website_status,website_http_status,website_final_url,
                 website_latency_ms,website_https,website_title,website_technology,website_has_viewport,
                 website_tls_expires_at,website_checked_at,website_error,email_evidence,score_reasons,
                 ai_probability,ai_recommendation,ai_reasoning,ai_checked_at,ai_subject,ai_message,ai_offer,opportunity_score,raw_json
-            ) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""", (
+            ) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)""", (
                 item["name"], item["website"], item["domain"], item["description"], item["query"], now(),
-                item["source"], item["source_id"], item["category"], item["address"], item["phone"], item["emails"],
+                item["source"], item["source_id"], item["category"], item.get("search_category", ""), item["address"], item["phone"], item["emails"],
                 item["rating"], item["review_count"], item["maps_url"], item["latitude"], item["longitude"],
                 item.get("listing_status", ""), item.get("website_status", "unknown"), item.get("website_http_status"),
                 item.get("website_final_url", ""), item.get("website_latency_ms", 0), item.get("website_https", 0),
@@ -952,7 +992,7 @@ def run_discovery(run_id: int, payload: dict[str, Any]) -> None:
                 if total:
                     update_run(run_id, stage="website_analysis", progress=76 + int(index * 19 / total), found=index)
 
-            results, added = insert_catalogue(run_id, query, raw_results, requested, enrichment_progress)
+            results, added = insert_catalogue(run_id, query, raw_results, requested, enrichment_progress, search_category=category)
             update_run(run_id, found=len(results), added=added, status="succeeded", stage="complete", progress=100, completed_at=now(), error="")
         except Exception as exc:
             message = str(exc).strip()[:500] or "discovery_failed"
@@ -998,7 +1038,7 @@ def scheduler_loop() -> None:
 
 
 def ai_generate_message(prospect: dict[str, Any], offer: str) -> dict[str, Any]:
-    """Generate an evidence-grounded draft through an OpenAI-compatible endpoint."""
+    """Generate an evidence-grounded, human-editable outreach draft."""
     runtime = ai_runtime_settings()
     ai_base_url = runtime["base_url"]
     ai_api_key = runtime["api_key"]
@@ -1007,24 +1047,57 @@ def ai_generate_message(prospect: dict[str, Any], offer: str) -> dict[str, Any]:
         raise RuntimeError("nous_api_key_required_for_standalone_app")
     if not ai_base_url or not ai_model:
         raise RuntimeError("ai_not_configured_set_AI_BASE_URL_and_AI_MODEL")
+    evidence_ids = ["prospect_record"]
+    if prospect.get("maps_url"):
+        evidence_ids.append("public_maps_listing")
+    if prospect.get("website"):
+        evidence_ids.append("website_scan")
+    if prospect.get("emails"):
+        evidence_ids.append("public_email_listing")
+    if prospect.get("phone"):
+        evidence_ids.append("public_phone_listing")
     evidence = {
-        "business": prospect.get("name"), "category": prospect.get("category"),
+        "evidence_ids": evidence_ids,
-        "location": prospect.get("address"), "website": prospect.get("website"),
+        "business_name": prospect.get("name"),
-        "website_status": prospect.get("website_status"), "website_title": prospect.get("website_title"),
+        "search_category": prospect.get("search_category") or prospect.get("category"),
-        "technology": prospect.get("website_technology"), "emails": prospect.get("emails"),
+        "address_or_location": prospect.get("address"),
-        "phone": prospect.get("phone"), "score": prospect.get("opportunity_score"),
+        "public_listing": {
+            "maps_url": prospect.get("maps_url"), "phone": prospect.get("phone"),
+            "rating": prospect.get("rating"), "review_count": prospect.get("review_count"),
+        },
+        "website": {
+            "url": prospect.get("website"), "status": prospect.get("website_status"),
+            "http_status": prospect.get("website_http_status"), "title": prospect.get("website_title"),
+            "technology": prospect.get("website_technology"), "https": bool(prospect.get("website_https")),
+            "latency_ms": prospect.get("website_latency_ms"),
+        },
+        "public_emails": prospect.get("emails"),
+        "opportunity_score": prospect.get("opportunity_score"),
         "score_reasons": _json_list(prospect.get("score_reasons")),
     }
-    system = ("You write concise, respectful B2B prospecting drafts for a South African web hosting and design business. "
+    system = (
-              "Use only the supplied evidence; never invent facts, prices, names, services, or claims. "
+        "You are a senior B2B copywriter for a South African web-hosting and website-design business. "
-              "Do not mention scraping or hidden analysis. Return strict JSON with keys subject, message, rationale. "
+        "Create a short, respectful, genuinely personalised first-contact email from the supplied evidence. "
-              "The message must be plain text, personalised, transparent, and include a polite opt-out line.")
+        "Use only stored facts and the user's offer; never invent a person's name, company activity, price, "
+        "service, problem, result, testimonial, urgency, or link. Do not mention scraping, scoring, hidden "
+        "analysis, or imply certainty where the evidence is uncertain. Do not insult the prospect or call a "
+        "site bad; use neutral wording such as 'may be having trouble loading' when appropriate. "
+        "If there is no website, do not say you reviewed their website. If there is no mockup URL in the offer, "
+        "do not invent one. Keep the email around 90-150 words, with a natural greeting using the business name, "
+        "one specific evidence-based observation, the exact offer/instructions supplied by the user, one simple "
+        "call to action, and a polite opt-out sentence. Avoid spammy hype, excessive emojis, all-caps, and fake "
+        "personal familiarity. Return ONLY strict JSON with exactly these keys: subject, message, rationale, "
+        "recommended_offer, unknowns, evidence_used, follow_up. subject is 5-10 natural words. message is plain "
+        "text. rationale is concise. recommended_offer is a short summary of the user's offer. unknowns is an "
+        "array of facts that should be confirmed. evidence_used is an array containing only the supplied evidence "
+        "IDs. follow_up is a short human review note, not part of the email."
+    )
     prompt = json.dumps({"evidence": evidence, "user_offer_or_instruction": offer[:2000]}, ensure_ascii=False)
     headers = {"Authorization": "Bearer " + ai_api_key} if ai_api_key else {}
     request_payload: dict[str, Any] = {
         "model": ai_model,
         "temperature": 0.4,
-        "max_tokens": 700,
+        "max_tokens": 900,
         "messages": [{"role": "system", "content": system}, {"role": "user", "content": prompt}],
     }
     # Nous exposes an OpenAI-compatible chat endpoint, but not every free model
@@ -1034,13 +1107,23 @@ def ai_generate_message(prospect: dict[str, Any], offer: str) -> dict[str, Any]:
         request_payload["response_format"] = {"type": "json_object"}
     result = http_json("POST", ai_base_url + "/chat/completions", request_payload, headers=headers, timeout=90, error_prefix="ai")
     content = (((result.get("choices") or [{}])[0].get("message") or {}).get("content") or "") if isinstance(result, dict) else ""
-    try:
+    parsed = _parse_ai_object(content)
-        parsed = json.loads(content)
-    except (TypeError, ValueError, json.JSONDecodeError) as exc:
-        raise RuntimeError("ai_invalid_json_response") from exc
     if not isinstance(parsed, dict) or not str(parsed.get("subject") or "").strip() or not str(parsed.get("message") or "").strip():
         raise RuntimeError("ai_incomplete_message")
-    return {"subject": str(parsed["subject"])[:200], "message": str(parsed["message"])[:8000], "rationale": str(parsed.get("rationale") or "")[:2000]}
+    def text_list(value: Any, limit: int = 6, size: int = 400) -> list[str]:
+        if not isinstance(value, list):
+            return []
+        return [str(item).strip()[:size] for item in value if str(item).strip()][:limit]
+    used = [item for item in text_list(parsed.get("evidence_used"), limit=10, size=80) if item in evidence_ids]
+    return {
+        "subject": str(parsed["subject"]).strip()[:200],
+        "message": str(parsed["message"]).strip()[:8000],
+        "rationale": str(parsed.get("rationale") or "").strip()[:2000],
+        "recommended_offer": str(parsed.get("recommended_offer") or offer).strip()[:2000],
+        "unknowns": text_list(parsed.get("unknowns")),
+        "evidence_used": used,
+        "follow_up": str(parsed.get("follow_up") or "Review the draft and confirm the offer before sending.").strip()[:1000],
+    }
 
 
 def _parse_ai_object(content: Any) -> dict[str, Any]:
@@ -1163,6 +1246,7 @@ class Handler(BaseHTTPRequestHandler):
                     "ready": ready,
                     "email_extraction_default": True,
                     "email_extraction_always": True,
+                    "browser_use_enabled": BROWSER_USE_ENABLED,
                     "ai_provider": ai_runtime["provider"],
                     "ai_configured": bool(ai_runtime["base_url"] and ai_runtime["model"] and (ai_runtime["provider"] != "nous" or ai_runtime["api_key"])),
                 })
@@ -1174,7 +1258,8 @@ class Handler(BaseHTTPRequestHandler):
             if parsed.path == "/api/settings":
                 ai_runtime = ai_runtime_settings()
                 models = nous_models() if ai_runtime["provider"] == "nous" else []
-                return self.send_json(200, {"maps": {"backend": backend_mode(), "worker_url": GMAPS_URL, "ready": backend_mode() == "browser" or bool(GMAPS_URL)}, "ai": {"provider": ai_runtime["provider"], "base_url": ai_runtime["base_url"], "model": ai_runtime["model"], "configured": bool(ai_runtime["base_url"] and ai_runtime["model"] and (ai_runtime["provider"] != "nous" or ai_runtime["api_key"])), "models": models}, "email_collection": {"always_on": True, "public_only": True}, "limits": {"max_results": MAX_RESULTS, "website_scan_timeout_seconds": 8}})
+                browser_steps, browser_timeout = browser_use_limits()
+                return self.send_json(200, {"maps": {"backend": backend_mode(), "worker_url": GMAPS_URL, "ready": backend_mode() == "browser" or bool(GMAPS_URL)}, "ai": {"provider": ai_runtime["provider"], "base_url": ai_runtime["base_url"], "model": ai_runtime["model"], "configured": bool(ai_runtime["base_url"] and ai_runtime["model"] and (ai_runtime["provider"] != "nous" or ai_runtime["api_key"])), "models": models}, "browser_use": {"enabled": BROWSER_USE_ENABLED, "headless": True, "public_only": True, "max_steps": browser_steps, "timeout_seconds": browser_timeout}, "email_collection": {"always_on": True, "public_only": True}, "limits": {"max_results": MAX_RESULTS, "website_scan_timeout_seconds": 8}})
             if parsed.path == "/api/prospects":
                 params = parse_qs(parsed.query)
                 q = (params.get("q", [""])[0]).strip().lower()
@@ -1182,7 +1267,7 @@ class Handler(BaseHTTPRequestHandler):
                 rows = conn.execute("SELECT * FROM prospects ORDER BY opportunity_score DESC, id DESC").fetchall()
                 items = []
                 for row in rows:
-                    haystack = " ".join(str(row[key] or "") for key in ("name", "domain", "description", "address", "category", "phone", "emails")).lower()
+                    haystack = " ".join(str(row[key] or "") for key in ("name", "domain", "description", "address", "category", "search_category", "phone", "emails")).lower()
                     if q and q not in haystack:
                         continue
                     if status and row["review_status"] != status:
@@ -1192,14 +1277,21 @@ class Handler(BaseHTTPRequestHandler):
             if parsed.path == "/api/export.csv":
                 params = parse_qs(parsed.query); q = params.get("q", [""])[0].strip().lower()
                 rows = [dict(r) for r in conn.execute("SELECT * FROM prospects ORDER BY opportunity_score DESC, id DESC")]
-                if q: rows = [r for r in rows if q in " ".join(str(r.get(k) or "") for k in ("name","domain","address","category","emails")).lower()]
+                if q: rows = [r for r in rows if q in " ".join(str(r.get(k) or "") for k in ("name","domain","address","category","search_category","emails")).lower()]
-                out = io.StringIO(); fields = ["id","name","category","address","phone","emails","domain","website","website_status","opportunity_score","score_reasons","review_status","last_seen"]
+                out = io.StringIO(); fields = ["id","name","search_category","category","address","phone","emails","domain","website","website_status","opportunity_score","score_reasons","review_status","last_seen"]
                 writer = csv.DictWriter(out, fieldnames=fields); writer.writeheader(); writer.writerows({k:r.get(k) for k in fields} for r in rows)
                 body = out.getvalue().encode(); self.send_response(200); self.send_header("Content-Type","text/csv; charset=utf-8"); self.send_header("Content-Disposition","attachment; filename=prospects.csv"); self.send_header("Content-Length",str(len(body))); self.end_headers(); self.wfile.write(body); return
             if parsed.path == "/api/suppressions":
                 return self.send_json(200, {"items": [dict(r) for r in conn.execute("SELECT * FROM suppressions ORDER BY id DESC")]})
             if parsed.path == "/api/schedules":
                 return self.send_json(200, {"items": [dict(r) for r in conn.execute("SELECT * FROM schedules ORDER BY id DESC")]})
+            match = re.fullmatch(r"/api/prospects/(\d+)/website-candidates", parsed.path)
+            if match:
+                prospect_id = int(match.group(1))
+                if not conn.execute("SELECT 1 FROM prospects WHERE id=?", (prospect_id,)).fetchone():
+                    return self.send_json(404, {"error": "not_found"})
+                rows = conn.execute("SELECT * FROM website_candidates WHERE prospect_id=? ORDER BY confidence DESC, id DESC", (prospect_id,)).fetchall()
+                return self.send_json(200, {"items": [dict(row) for row in rows]})
             match = re.fullmatch(r"/api/prospects/(\d+)/domains", parsed.path)
             if match:
                 row = conn.execute("SELECT * FROM prospects WHERE id=?", (int(match.group(1)),)).fetchone()
@@ -1283,6 +1375,50 @@ class Handler(BaseHTTPRequestHandler):
                 return self.send_json(502, {"error": str(exc)})
             except Exception as exc:
                 return self.send_json(502, {"error": "ai_test_failed:" + str(exc)[:200]})
+        candidates_match = re.fullmatch(r"/api/prospects/(\d+)/website-candidates", urlparse(self.path).path)
+        if candidates_match:
+            prospect_id = int(candidates_match.group(1))
+            conn = db()
+            try:
+                row = conn.execute("SELECT * FROM prospects WHERE id=?", (prospect_id,)).fetchone()
+            finally:
+                conn.close()
+            if not row:
+                return self.send_json(404, {"error": "not_found"})
+            if not BROWSER_USE_ENABLED:
+                return self.send_json(503, {"error": "browser_use_disabled"})
+            try:
+                runtime = ai_runtime_settings()
+                candidates = investigate_websites(
+                    dict(row), base_url=runtime["base_url"], api_key=runtime["api_key"],
+                    model=runtime["model"], user_agent=OUTBOUND_USER_AGENT,
+                )
+                conn = db()
+                for candidate in candidates:
+                    encoded_evidence = json.dumps(candidate.get("evidence") or [], ensure_ascii=False)
+                    encoded_raw = json.dumps(candidate, ensure_ascii=False)
+                    conn.execute(
+                        """INSERT INTO website_candidates(prospect_id,url,domain,confidence,evidence,source,status,discovered_at,raw_json)
+                           VALUES(?,?,?,?,?,?,?,?,?)
+                           ON CONFLICT(prospect_id,domain) DO UPDATE SET
+                             url=excluded.url, confidence=excluded.confidence, evidence=excluded.evidence,
+                             source=excluded.source, discovered_at=excluded.discovered_at, raw_json=excluded.raw_json,
+                             status=CASE WHEN website_candidates.status='accepted' THEN 'accepted' ELSE 'candidate' END""",
+                        (prospect_id, candidate["url"], candidate["domain"], candidate["confidence"], encoded_evidence,
+                         "browser_use", "candidate", now(), encoded_raw),
+                    )
+                conn.commit()
+                rows = conn.execute("SELECT * FROM website_candidates WHERE prospect_id=? ORDER BY confidence DESC, id DESC", (prospect_id,)).fetchall()
+                return self.send_json(200, {"items": [dict(item) for item in rows], "count": len(rows)})
+            except RuntimeError as exc:
+                return self.send_json(503, {"error": str(exc)})
+            except Exception as exc:
+                return self.send_json(502, {"error": "browser_use_failed:" + str(exc)[:240]})
+            finally:
+                try:
+                    conn.close()
+                except Exception:
+                    pass
         assess_match = re.fullmatch(r"/api/prospects/(\d+)/assess", urlparse(self.path).path)
         if assess_match:
             conn = None
@@ -1315,7 +1451,7 @@ class Handler(BaseHTTPRequestHandler):
                 payload = self.read_json(); offer = str(payload.get("offer") or "").strip()
                 if not offer: return self.send_json(400, {"error":"offer_required"})
                 draft = ai_generate_message(dict(row), offer)
-                stamp = now(); conn.execute("UPDATE prospects SET ai_subject=?,ai_message=?,ai_offer=?,ai_reasoning=?,ai_checked_at=? WHERE id=?", (draft["subject"],draft["message"],offer,draft["rationale"],stamp,int(message_match.group(1)))); conn.commit()
+                stamp = now(); conn.execute("UPDATE prospects SET ai_subject=?,ai_message=?,ai_offer=?,ai_reasoning=?,ai_unknowns=?,ai_evidence_ids=?,ai_follow_up=?,ai_checked_at=? WHERE id=?", (draft["subject"],draft["message"],offer,draft["rationale"],json.dumps(draft.get("unknowns") or [], ensure_ascii=False),json.dumps(draft.get("evidence_used") or [], ensure_ascii=False),draft.get("follow_up") or "",stamp,int(message_match.group(1)))); conn.commit()
                 return self.send_json(200, {**draft,"ai_checked_at":stamp,"prospect_id":int(message_match.group(1))})
             except RuntimeError as exc: return self.send_json(503, {"error":str(exc)})
             except (ValueError, TypeError, json.JSONDecodeError) as exc: return self.send_json(400, {"error":str(exc)})
@@ -1355,6 +1491,80 @@ class Handler(BaseHTTPRequestHandler):
             return self.send_json(400, {"error": str(exc)})
 
     def do_PATCH(self) -> None:
+        candidate_match = re.fullmatch(r"/api/prospects/(\d+)/website-candidates/(\d+)", urlparse(self.path).path)
+        if candidate_match:
+            prospect_id, candidate_id = (int(candidate_match.group(1)), int(candidate_match.group(2)))
+            conn: sqlite3.Connection | None = None
+            try:
+                payload = self.read_json()
+                status = str(payload.get("status", "")).strip().lower()
+                if status not in {"candidate", "accepted", "rejected"}:
+                    return self.send_json(400, {"error": "invalid_website_candidate_status"})
+                conn = db()
+                candidate = conn.execute(
+                    "SELECT * FROM website_candidates WHERE id=? AND prospect_id=?",
+                    (candidate_id, prospect_id),
+                ).fetchone()
+                prospect = conn.execute("SELECT * FROM prospects WHERE id=?", (prospect_id,)).fetchone()
+                if not candidate or not prospect:
+                    return self.send_json(404, {"error": "not_found"})
+                if status != "accepted":
+                    conn.execute("UPDATE website_candidates SET status=? WHERE id=?", (status, candidate_id))
+                    conn.commit()
+                    updated_candidate = conn.execute("SELECT * FROM website_candidates WHERE id=?", (candidate_id,)).fetchone()
+                    return self.send_json(200, {"candidate": dict(updated_candidate)})
+
+                # The candidate is now a real catalogue domain. Check the
+                # catalogue before scanning so SQLite's unique domain index
+                # produces an actionable response instead of a generic 500.
+                duplicate = conn.execute(
+                    "SELECT id FROM prospects WHERE domain=? AND id<>?",
+                    (str(candidate["domain"]).lower(), prospect_id),
+                ).fetchone()
+                if duplicate:
+                    return self.send_json(409, {"error": "domain_already_catalogued", "prospect_id": int(duplicate["id"])})
+                candidate_url = normalize_url(candidate["url"])
+                if not candidate_url:
+                    return self.send_json(400, {"error": "invalid_candidate_url"})
+
+                # Website probing is network-bound. Release the SQLite handle
+                # while it runs, then persist the selected URL and evidence in
+                # one short transaction.
+                base_item = {**dict(prospect), "website": candidate_url, "domain": str(candidate["domain"]).lower()}
+                conn.close()
+                conn = None
+                updated = enrich_place(base_item)
+                cols = [
+                    "website", "domain", "website_status", "website_http_status", "website_final_url",
+                    "website_latency_ms", "website_https", "website_title", "website_technology",
+                    "website_has_viewport", "website_tls_expires_at", "website_checked_at", "website_error",
+                    "emails", "email_evidence", "score_reasons", "opportunity_score",
+                ]
+                conn = db()
+                conn.execute(
+                    "UPDATE website_candidates SET status='candidate' WHERE prospect_id=? AND id<>? AND status='accepted'",
+                    (prospect_id, candidate_id),
+                )
+                conn.execute("UPDATE website_candidates SET status='accepted' WHERE id=? AND prospect_id=?", (candidate_id, prospect_id))
+                conn.execute(
+                    "UPDATE prospects SET " + ",".join(f"{column}=?" for column in cols) + " WHERE id=?",
+                    tuple(updated.get(column) for column in cols) + (prospect_id,),
+                )
+                conn.commit()
+                updated_candidate = conn.execute("SELECT * FROM website_candidates WHERE id=?", (candidate_id,)).fetchone()
+                updated_prospect = conn.execute("SELECT * FROM prospects WHERE id=?", (prospect_id,)).fetchone()
+                return self.send_json(200, {"candidate": dict(updated_candidate), "prospect": dict(updated_prospect)})
+            except sqlite3.IntegrityError:
+                if conn is not None:
+                    conn.rollback()
+                return self.send_json(409, {"error": "domain_already_catalogued"})
+            except (ValueError, TypeError, json.JSONDecodeError) as exc:
+                return self.send_json(400, {"error": str(exc)})
+            except Exception as exc:
+                return self.send_json(502, {"error": "website_candidate_accept_failed:" + str(exc)[:240]})
+            finally:
+                if conn is not None:
+                    conn.close()
         match = re.fullmatch(r"/api/prospects/(\d+)/review", urlparse(self.path).path)
         if not match:
             return self.send_json(404, {"error": "not_found"})

docker-compose.yml

@@ -25,6 +25,10 @@ services:
       GMAPS_REQUEST_TIMEOUT: ${GMAPS_REQUEST_TIMEOUT:-900}
       GMAPS_POLL_SECONDS: ${GMAPS_POLL_SECONDS:-2}
       OUTBOUND_USER_AGENT: ${OUTBOUND_USER_AGENT:-ProspectOS/2.0 (+personal-use)}
+      BROWSER_USE_ENABLED: ${BROWSER_USE_ENABLED:-true}
+      BROWSER_USE_MAX_STEPS: ${BROWSER_USE_MAX_STEPS:-10}
+      BROWSER_USE_TIMEOUT_SECONDS: ${BROWSER_USE_TIMEOUT_SECONDS:-120}
+      ANONYMIZED_TELEMETRY: ${ANONYMIZED_TELEMETRY:-false}
       AI_PROVIDER: ${AI_PROVIDER:-custom}
       AI_BASE_URL: ${AI_BASE_URL:-}
       AI_API_KEY: ${AI_API_KEY:-}

tests/test_app.py

@@ -9,6 +9,7 @@ from http.client import HTTPConnection
 from unittest.mock import patch
 
 from api import server
+from api import browser_investigator
 
 
 class BrowserOnlyAppTests(unittest.TestCase):
@@ -56,6 +57,8 @@ class BrowserOnlyAppTests(unittest.TestCase):
         self.assertEqual(status, 202)
         first = self.wait_for_run(first["id"])
         self.assertEqual(first["added"], 2)
+        saved = self.request("GET", "/api/prospects")[1]["items"]
+        self.assertTrue(all(item["search_category"] == "plumbers" for item in saved))
         status, second = self.request("POST", "/api/discover", {"category": "plumbers", "keywords": "plumbing", "location": "Cape Town", "max_results": 5})
         self.assertEqual(status, 202)
         second = self.wait_for_run(second["id"])
@@ -153,6 +156,48 @@ class BrowserOnlyAppTests(unittest.TestCase):
         saved = server.db().execute("SELECT ai_probability,ai_recommendation,ai_reasoning FROM prospects WHERE id=1").fetchone()
         self.assertEqual(tuple(saved), (84.0, "contact", "Active business with a clear website opportunity."))
 
+    def test_browser_investigator_normalises_and_filters_candidates(self):
+        raw = '{"candidates":[{"url":"https://www.acme.co.za/about","confidence":88,"evidence":["name match"]},{"url":"https://facebook.com/acme","confidence":99},{"domain":"other.co.za","confidence":"42","evidence":"not a list"}]}'
+        candidates = browser_investigator._parse_agent_result(raw)
+        self.assertEqual([item["domain"] for item in candidates], ["acme.co.za", "other.co.za"])
+        self.assertEqual(candidates[0]["confidence"], 88)
+        self.assertEqual(candidates[1]["evidence"], [])
+
+    def test_browser_website_candidates_can_be_found_and_accepted(self):
+        conn = server.db()
+        conn.execute("INSERT INTO prospects(name,domain,category,address,discovered_at) VALUES(?,?,?,?,?)", ("Acme Plumbing", "maps:abc", "plumber", "Cape Town", server.now()))
+        conn.commit(); conn.close()
+        found = [{"url": "https://acmeplumbing.co.za", "domain": "acmeplumbing.co.za", "confidence": 91, "evidence": ["name match", "location match"]}]
+        with patch("api.server.investigate_websites", return_value=found):
+            status, body = self.request("POST", "/api/prospects/1/website-candidates", {})
+        self.assertEqual(status, 200)
+        self.assertEqual(body["count"], 1)
+        candidate_id = body["items"][0]["id"]
+        status, body = self.request("GET", "/api/prospects/1/website-candidates")
+        self.assertEqual(status, 200)
+        self.assertEqual(body["items"][0]["domain"], "acmeplumbing.co.za")
+        status, body = self.request("PATCH", f"/api/prospects/1/website-candidates/{candidate_id}", {"status": "accepted"})
+        self.assertEqual(status, 200)
+        self.assertEqual(body["candidate"]["status"], "accepted")
+        self.assertEqual(body["prospect"]["domain"], "acmeplumbing.co.za")
+        self.assertEqual(body["prospect"]["website"], "https://acmeplumbing.co.za")
+
+    def test_email_draft_returns_structured_review_notes(self):
+        conn = server.db()
+        conn.execute("INSERT INTO prospects(name,domain,search_category,category,address,maps_url,website_status,emails,opportunity_score,score_reasons,discovered_at) VALUES(?,?,?,?,?,?,?,?,?,?,?)", ("Acme Plumbing", "maps:abc", "plumbers", "Plumber", "Cape Town", "https://maps.example/acme", "no_detected_website", "acme@gmail.com", 88, '["No detected website","Public email found"]', server.now()))
+        for key, value in (("AI_PROVIDER", "custom"), ("AI_BASE_URL", "https://ai.example/v1"), ("AI_MODEL", "test-model"), ("AI_API_KEY", "test-key")):
+            conn.execute("INSERT INTO settings(key,value,updated_at) VALUES(?,?,?)", (key, value, server.now()))
+        conn.commit(); conn.close()
+        content = json.dumps({"subject": "A website idea for Acme Plumbing", "message": "Good day Acme Plumbing,\n\nI noticed your public listing does not show a dedicated website. We can help with a professional online presence. Would you like me to send a few details?\n\nKind regards", "rationale": "Uses the missing-website signal and the supplied offer.", "recommended_offer": "Gold Plus hosting and a free one-page website", "unknowns": ["Confirm the preferred contact person"], "evidence_used": ["prospect_record", "public_maps_listing", "public_email_listing"], "follow_up": "Confirm the offer and recipient before sending."})
+        with patch("api.server.http_json", return_value={"choices": [{"message": {"content": content}}]}):
+            status, body = self.request("POST", "/api/prospects/1/message", {"offer": "Gold Plus hosting at R119/month with a free one-page website."})
+        self.assertEqual(status, 200)
+        self.assertEqual(body["unknowns"], ["Confirm the preferred contact person"])
+        self.assertEqual(body["evidence_used"], ["prospect_record", "public_maps_listing", "public_email_listing"])
+        saved = server.db().execute("SELECT ai_unknowns,ai_evidence_ids,ai_follow_up FROM prospects WHERE id=1").fetchone()
+        self.assertEqual(json.loads(saved[0]), body["unknowns"])
+        self.assertEqual(json.loads(saved[1]), body["evidence_used"])
+
 
 if __name__ == "__main__":
     unittest.main()

web/app.js

@@ -1,28 +1,41 @@
 const $=id=>document.getElementById(id); const esc=v=>String(v??"").replace(/[&<>\"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','\"':'&quot;',"'":'&#39;'}[c]));
 let prospects=[],runs=[],settings={},selected=null,pollTimer=null;
+let activeProspectCategory='all';
+const HIGH_POTENTIAL_SCORE=70;
 async function api(path,opt={}){const r=await fetch(path,{...opt,headers:{'Content-Type':'application/json',...(opt.headers||{})}});const b=await r.json().catch(()=>({}));if(!r.ok)throw Error(b.error||`Request failed (${r.status})`);return b}
 function date(v){if(!v)return'—';const d=new Date(v);return isNaN(d)?String(v):d.toLocaleString(undefined,{day:'numeric',month:'short',hour:'2-digit',minute:'2-digit'})}
 function layout(title,sub,body,actions=''){ $('pageName').textContent=title; return `<section class="page-head"><div><span class="eyebrow">PROSPECTOS</span><h1>${title}</h1><p>${sub}</p></div><div class="head-actions">${actions}</div></section>${body}` }
 function empty(text){return `<div class="empty"><span>⌕</span><b>${text}</b></div>`}
 function scoreClass(n){return n>=75?'hot':n>=55?'warm':'cool'}
-function prospectRows(){const q=($('prospectSearch')?.value||'').toLowerCase();const st=$('prospectStatus')?.value||'';return prospects.filter(p=>{const hay=[p.name,p.domain,p.category,p.address,p.emails].join(' ').toLowerCase();return(!q||hay.includes(q))&&(!st||p.review_status===st)})}
+function prospectCategory(p){const raw=String(p.search_category||p.category||'').trim();return raw?raw.replace(/\s+/g,' ').replace(/^./,c=>c.toUpperCase()):'Other'}
-function prospectTable(){const rows=prospectRows();return `<div class="table-wrap"><table><thead><tr><th>Prospect</th><th>Website</th><th>Email</th><th>Score</th><th>Review</th><th></th></tr></thead><tbody>${rows.map(p=>`<tr data-open="${p.id}"><td><b>${esc(p.name)}</b><small>${esc(p.category||'Business')} · ${esc(p.address||'')}</small></td><td><span class="badge ${p.website_status==='healthy'?'good':'warn'}">${esc((p.website_status||'unknown').replaceAll('_',' '))}</span>${p.website?`<a href="${esc(p.website)}" target="_blank" rel="noopener">Open ↗</a>`:'<small>No detected website</small>'}</td><td>${p.emails?`<span class="email">✉ ${esc(p.emails)}</span>`:'<small>Not found</small>'}</td><td><span class="score ${scoreClass(Number(p.opportunity_score||0))}">${p.opportunity_score||0}</span></td><td><select data-review="${p.id}"><option value="pending" ${p.review_status==='pending'?'selected':''}>Needs review</option><option value="approved" ${p.review_status==='approved'?'selected':''}>Approved</option><option value="rejected" ${p.review_status==='rejected'?'selected':''}>Rejected</option><option value="contacted" ${p.review_status==='contacted'?'selected':''}>Contacted</option></select></td><td><button class="icon-action" data-message="${p.id}" title="Draft AI email">✦</button></td></tr>`).join('')}</tbody></table></div>${rows.length?'':empty('No prospects match these filters')}`}
+function prospectTabs(){const categories=[...new Set(prospects.map(prospectCategory))].sort((a,b)=>a.localeCompare(b));if(activeProspectCategory!=='all'&&!categories.includes(activeProspectCategory))activeProspectCategory='all';const count=category=>category==='all'?prospects.filter(p=>Number(p.opportunity_score||0)>=HIGH_POTENTIAL_SCORE).length:prospects.filter(p=>prospectCategory(p)===category).length;return `<div class="prospect-tab-list"><button class="prospect-tab ${activeProspectCategory==='all'?'active':''}" data-category-tab="all">All high potential <span>${count('all')}</span></button>${categories.map(category=>`<button class="prospect-tab ${activeProspectCategory===category?'active':''}" data-category-tab="${esc(category)}">${esc(category)} <span>${count(category)}</span></button>`).join('')}</div><p class="tab-hint">All shows scores ${HIGH_POTENTIAL_SCORE}+; category tabs keep every result grouped by the search category.</p>`}
+function prospectRows(){const q=($('prospectSearch')?.value||'').toLowerCase();const st=$('prospectStatus')?.value||'';return prospects.filter(p=>{const hay=[p.name,p.domain,p.search_category,p.category,p.address,p.emails].join(' ').toLowerCase();const tabMatch=activeProspectCategory==='all'?Number(p.opportunity_score||0)>=HIGH_POTENTIAL_SCORE:prospectCategory(p)===activeProspectCategory;return tabMatch&&(!q||hay.includes(q))&&(!st||p.review_status===st)}).sort((a,b)=>Number(b.opportunity_score||0)-Number(a.opportunity_score||0)||String(a.name||'').localeCompare(String(b.name||'')))}
+function prospectTable(){const rows=prospectRows();return `<div class="table-wrap"><table><thead><tr><th>Prospect</th><th>Website</th><th>Email</th><th>Potential</th><th>Review</th><th></th></tr></thead><tbody>${rows.map(p=>`<tr data-open="${p.id}"><td><b>${esc(p.name)}</b><small>${esc(prospectCategory(p))} · ${esc(p.address||'')}</small></td><td><span class="badge ${p.website_status==='healthy'?'good':'warn'}">${esc((p.website_status||'unknown').replaceAll('_',' '))}</span>${p.website?`<a href="${esc(p.website)}" target="_blank" rel="noopener">Open ↗</a>`:'<small>No detected website</small>'}</td><td>${p.emails?`<span class="email">✉ ${esc(p.emails)}</span>`:'<small>Not found</small>'}</td><td><span class="score ${scoreClass(Number(p.opportunity_score||0))}">${p.opportunity_score||0}</span><small>${Number(p.opportunity_score||0)>=85?'Hot':Number(p.opportunity_score||0)>=70?'High':Number(p.opportunity_score||0)>=45?'Medium':'Low'}</small></td><td><select data-review="${p.id}"><option value="pending" ${p.review_status==='pending'?'selected':''}>Needs review</option><option value="approved" ${p.review_status==='approved'?'selected':''}>Approved</option><option value="rejected" ${p.review_status==='rejected'?'selected':''}>Rejected</option><option value="contacted" ${p.review_status==='contacted'?'selected':''}>Contacted</option></select></td><td><button class="icon-action" data-message="${p.id}" title="Draft AI email">✦</button></td></tr>`).join('')}</tbody></table></div>${rows.length?'':empty('No prospects match this tab and filter')}`}
 function renderDashboard(){const total=prospects.length,pending=prospects.filter(p=>p.review_status==='pending').length,hot=prospects.filter(p=>p.opportunity_score>=75).length,withEmail=prospects.filter(p=>p.emails).length;const latest=runs[0];$('view').innerHTML=layout('Dashboard','A clear view of what was found, what needs review, and where the best opportunities are.',`<div class="metric-grid"><div class="metric"><small>CATALOGUE</small><b>${total}</b><span>saved prospects</span></div><div class="metric"><small>NEEDS REVIEW</small><b>${pending}</b><span>awaiting your decision</span></div><div class="metric"><small>HIGH OPPORTUNITY</small><b>${hot}</b><span>score 75 or higher</span></div><div class="metric"><small>PUBLIC EMAILS</small><b>${withEmail}</b><span>evidence-backed contacts</span></div></div><div class="dashboard-grid"><article class="panel"><div class="panel-title"><div><span class="eyebrow">NEXT ACTION</span><h2>Review your strongest leads</h2><p>Inspect the evidence before deciding whether to contact them.</p></div><a class="primary" href="#prospects">Open explorer →</a></div><div class="spotlight">${prospects.slice(0,5).map(p=>`<button class="spot" data-open="${p.id}"><span class="score ${scoreClass(p.opportunity_score)}">${p.opportunity_score}</span><span><b>${esc(p.name)}</b><small>${esc(p.website_status?.replaceAll('_',' ')||'unknown')} · ${esc(p.emails||'no public email')}</small></span><span>›</span></button>`).join('')||empty('Run a discovery to build your catalogue')}</div></article><article class="panel"><div class="panel-title"><div><span class="eyebrow">RECENT ACTIVITY</span><h2>Runs</h2></div><a href="#runs">View all</a></div><div class="activity">${runs.slice(0,6).map(r=>`<div><span class="run-dot ${r.status}"></span><span><b>#${r.id} · ${esc(r.query)}</b><small>${esc(r.status)} · ${r.added||0} added</small></span><time>${date(r.created_at)}</time></div>`).join('')||empty('No runs yet')}</div></article></div>`)}
 function renderDiscover(){ $('view').innerHTML=layout('Discover','Set the criteria for this run. Source configuration and provider settings live in Settings.',`<div class="two-col"><article class="panel form-panel"><div class="panel-title"><div><span class="eyebrow">NEW RUN</span><h2>Find potential customers</h2><p>Search Google Maps, deduplicate the catalogue, then enrich each result.</p></div><span class="pill">Email collection always on</span></div><form id="discoverForm"><label>Keywords<input id="keywords" required placeholder="plumbing services"></label><label>Category<input id="category" required placeholder="plumbers"></label><label>Location<input id="location" value="South Africa" required></label><label>Maximum results<select id="max-results"><option>10</option><option selected>25</option><option>50</option></select></label><label class="checkline"><input type="checkbox" id="extra-reviews"> Collect extra reviews <small>May increase run time</small></label><div class="form-row"><button class="primary" type="submit" id="discoverBtn">Run discovery →</button><span id="discoverMessage" class="notice"></span></div></form></article><article class="panel"><div class="panel-title"><div><span class="eyebrow">HOW IT WORKS</span><h2>Evidence-first pipeline</h2></div></div><ol class="steps"><li><b>Search</b><span>Maps worker finds public business listings.</span></li><li><b>Enrich</b><span>Website, HTTPS, DNS hints and public email checks run safely.</span></li><li><b>Review</b><span>You decide what is relevant before any contact.</span></li></ol></article></div>`)}
-function renderProspects(){ $('view').innerHTML=layout('Prospects','Search, filter and review your catalogue. Click a row to inspect the evidence.',`<article class="panel"><div class="toolbar"><div class="search"><span>⌕</span><input id="prospectSearch" placeholder="Search name, address, domain or email"></div><select id="prospectStatus"><option value="">All review statuses</option><option value="pending">Needs review</option><option value="approved">Approved</option><option value="rejected">Rejected</option><option value="contacted">Contacted</option></select><button class="secondary" id="exportCsv">Export CSV</button><button class="secondary" id="refreshProspects">Refresh</button></div><div id="prospectTable">${prospectTable()}</div></article><aside id="detail" class="detail-panel" hidden></aside>`)}
+function renderProspects(){ $('view').innerHTML=layout('Prospects','Browse the strongest opportunities first, then switch to a search category when you want a focused list.',`<article class="panel"><div class="toolbar"><div class="search"><span>⌕</span><input id="prospectSearch" placeholder="Search name, address, domain or email"></div><select id="prospectStatus"><option value="">All review statuses</option><option value="pending">Needs review</option><option value="approved">Approved</option><option value="rejected">Rejected</option><option value="contacted">Contacted</option></select><button class="secondary" id="exportCsv">Export CSV</button><button class="secondary" id="refreshProspects">Refresh</button></div><div id="prospectTabs" class="prospect-tabs">${prospectTabs()}</div><div id="prospectTable">${prospectTable()}</div></article><aside id="detail" class="detail-panel" hidden></aside>`)}
 function renderRuns(){ $('view').innerHTML=layout('Runs','Monitor discovery jobs and inspect errors without leaving the run history.',`<article class="panel"><div class="table-wrap"><table><thead><tr><th>Run</th><th>Status</th><th>Results</th><th>Started</th><th>Source</th></tr></thead><tbody>${runs.map(r=>`<tr><td><b>#${r.id}</b><small>${esc(r.query)}</small></td><td><span class="status ${r.status}">${esc(r.status)}</span><small>${esc(r.stage||'')}</small></td><td>${r.added||0} added / ${r.found||0} found</td><td>${date(r.created_at)}</td><td>${esc(r.source||'google_maps')}</td></tr>`).join('')}</tbody></table></div>${runs.length?'':empty('No discovery runs yet')}</article>`)}
 function renderSettings(){const ai=settings.ai||{};const sch=settings.schedules||[];const sup=settings.suppressions||[];const provider=ai.provider||'custom';const modelList=Array.isArray(ai.models)?ai.models:[];const selectedModel=ai.model||'';const options=modelList.map(m=>`<option value="${esc(m.id)}">${esc(m.label||m.id)}${m.free?' · free':''}</option>`).join('')+(selectedModel&&!modelList.some(m=>m.id===selectedModel)?`<option value="${esc(selectedModel)}">${esc(selectedModel)}</option>`:'');$('view').innerHTML=layout('Settings','Configure providers, automation and safety controls here. These settings are separate from daily discovery.',`<div class="settings-grid"><article class="panel"><div class="panel-title"><div><span class="eyebrow">SOURCE</span><h2>Google Maps worker</h2><p>The bundled standard gosom worker runs locally and does not require an API key.</p></div><span class="status ${settings.maps?.ready?'succeeded':'failed'}">${settings.maps?.ready?'Ready':'Not connected'}</span></div><div class="setting-line"><span>Connection</span><b>${esc(settings.maps?.worker_url||'Not configured')}</b></div><div class="setting-line"><span>Email extraction</span><b class="good-text">Always on · public pages only</b></div><div class="setting-line"><span>Website scan</span><b>8 second bounded timeout</b></div></article><article class="panel"><div class="panel-title"><div><span class="eyebrow">AI PROVIDER</span><h2>Draft assistant</h2><p>Used only when you click “Draft AI email”.</p></div><span class="status ${ai.configured?'succeeded':'failed'}">${ai.configured?'Configured':'Not configured'}</span></div><form id="settingsForm"><label>Provider<select id="aiProvider"><option value="nous" ${provider==='nous'?'selected':''}>Nous Portal · free models</option><option value="custom" ${provider!=='nous'?'selected':''}>Custom OpenAI-compatible</option></select></label><p class="field-help" id="aiProviderNote">${provider==='nous'?'Nous uses the official inference endpoint. OAuth is supported by Hermes CLI; this standalone app uses a server-side API key.':'Use any OpenAI-compatible /v1 endpoint.'}</p><label>Base URL<input id="aiBase" value="${esc(ai.base_url||'')}" ${provider==='nous'?'readonly':''} placeholder="https://provider.example/v1"></label><label>API key<input id="aiKey" type="password" placeholder="Leave blank to keep current key"></label><label>Model<input id="aiModel" list="nousModelList" value="${esc(selectedModel)}" placeholder="model-name"><datalist id="nousModelList">${options}</datalist></label><p class="field-help">Nous model availability can change. Refresh Settings to update the free catalog.</p><div class="form-row"><button class="primary" type="submit">Save AI settings</button><button class="secondary" id="testAi" type="button">Test connection</button><span id="settingsMessage" class="notice"></span></div></form></article><article class="panel wide-panel"><div class="panel-title"><div><span class="eyebrow">SCHEDULES</span><h2>Recurring discovery</h2><p>Run bounded searches automatically; all results still arrive as needs-review.</p></div></div><form id="scheduleForm" class="inline-form"><input id="sCat" placeholder="Category"><input id="sKey" placeholder="Keywords"><input id="sLoc" value="South Africa" placeholder="Location"><select id="sHours"><option value="24">Daily</option><option value="48">Every 2 days</option><option value="168">Weekly</option></select><button class="secondary">Add schedule</button></form><div class="list">${sch.map(s=>`<div><b>${esc(s.keywords)} · ${esc(s.category)}</b><span>${esc(s.location)} · every ${s.interval_hours}h · next ${date(s.next_run_at)}</span></div>`).join('')||'<p class="muted">No schedules configured.</p>'}</div></article><article class="panel wide-panel"><div class="panel-title"><div><span class="eyebrow">SAFETY</span><h2>Suppression centre</h2><p>Suppressed domains and emails are excluded before new prospects are saved.</p></div></div><form id="suppressionForm" class="inline-form"><select id="supKind"><option value="domain">Domain</option><option value="email">Email</option></select><input id="supValue" placeholder="example.co.za or info@example.co.za"><input id="supReason" placeholder="Reason"><button class="secondary">Suppress</button></form><div class="list">${sup.map(s=>`<div><b>${esc(s.value)}</b><span>${esc(s.kind)} · ${esc(s.reason||'manual')} · ${date(s.created_at)}</span></div>`).join('')||'<p class="muted">No suppressions configured.</p>'}</div></article></div>`)}
 async function load(){[prospects,runs,settings]=await Promise.all([api('/api/prospects').then(x=>x.items||[]),api('/api/runs').then(x=>x.items||[]),Promise.all([api('/api/settings'),api('/api/schedules'),api('/api/suppressions')]).then(([s,sch,sup])=>({...s,schedules:sch.items||[],suppressions:sup.items||[]}))]);}
 async function route(){const r=location.hash.slice(1)||'dashboard';document.querySelectorAll('[data-route]').forEach(a=>a.classList.toggle('active',a.dataset.route===r));try{await load();({dashboard:renderDashboard,discover:renderDiscover,prospects:renderProspects,runs:renderRuns,settings:renderSettings}[r]||renderDashboard)();$('updated').textContent='Updated '+new Date().toLocaleTimeString([], {hour:'2-digit',minute:'2-digit'});}catch(e){$('view').innerHTML=empty(e.message)}}
-async function openMessage(id){selected=prospects.find(p=>String(p.id)===String(id));if(!selected)return;$('messageTitle').textContent='Draft for '+selected.name;$('offerInput').value='';$('subjectOutput').value=selected.ai_subject||'';$('bodyOutput').value=selected.ai_message||'';$('messageStatus').textContent='';$('messageDialog').showModal()}
+function suggestedOffer(p){if(p.ai_offer)return p.ai_offer;if(['broken','unreachable','error','slow'].includes(String(p.website_status||'')))return'Move their hosting to our Gold Plus package at R119/month and rebuild the website at no extra design charge.';if(!p.website)return'Offer a professional .co.za domain, reliable hosting from R119/month, professional email, and a free one-page website.';return'Offer a friendly website and hosting review; only suggest a change that is supported by the evidence.'}
+function jsonArray(value){try{const parsed=JSON.parse(value||'[]');return Array.isArray(parsed)?parsed:[]}catch{return[]}}
+function showDraftNotes(d){let notes=$('draftNotes');if(!notes){notes=document.createElement('div');notes.id='draftNotes';notes.className='draft-notes';const status=$('messageStatus');if(status)status.parentNode.insertBefore(notes,status)}if(!notes)return;const unknowns=Array.isArray(d?.unknowns)?d.unknowns:[];const used=Array.isArray(d?.evidence_used)?d.evidence_used:[];notes.innerHTML=`<b>AI review notes</b><p>${esc(d?.rationale||'The draft uses the prospect evidence and your instructions.')}</p>${unknowns.length?`<p><b>Confirm before sending:</b> ${esc(unknowns.join(' · '))}</p>`:''}${used.length?`<p class="muted">Evidence used: ${esc(used.join(', '))}</p>`:''}${d?.follow_up?`<p class="muted">${esc(d.follow_up)}</p>`:''}`}
+async function openMessage(id){selected=prospects.find(p=>String(p.id)===String(id));if(!selected)return;$('messageTitle').textContent='Draft for '+selected.name;$('offerInput').value=suggestedOffer(selected);$('subjectOutput').value=selected.ai_subject||'';$('bodyOutput').value=selected.ai_message||'';$('messageStatus').textContent='';showDraftNotes({rationale:selected.ai_reasoning,unknowns:jsonArray(selected.ai_unknowns),evidence_used:jsonArray(selected.ai_evidence_ids),follow_up:selected.ai_follow_up});$('messageDialog').showModal()}
 function openDrawer(id){const p=prospects.find(x=>String(x.id)===String(id));if(!p)return;selected=p;$('drawerName').textContent=p.name;let reasons=[];try{reasons=JSON.parse(p.score_reasons||'[]')}catch{};$('drawerBody').innerHTML=`<div class="drawer-score"><span class="score ${scoreClass(Number(p.opportunity_score||0))}">${p.opportunity_score||0}</span><div><b>Opportunity score</b><small>Explainable triage signal, not an AI verdict</small></div></div><div class="drawer-section"><h3>Contact</h3><p>${esc(p.emails||'No public email found')}</p><p>${esc(p.phone||'No phone listed')}</p></div><div class="drawer-section"><h3>Website health</h3><div class="detail-grid"><span>Status<b>${esc((p.website_status||'unknown').replaceAll('_',' '))}</b></span><span>HTTPS<b>${p.website_https?'Yes':'No / unknown'}</b></span><span>Latency<b>${p.website_latency_ms?p.website_latency_ms+' ms':'—'}</b></span><span>Checked<b>${date(p.website_checked_at)}</b></span></div></div><div class="drawer-section"><h3>Why this scored</h3><ul>${reasons.map(r=>`<li>${esc(r)}</li>`).join('')||'<li>No score reasons recorded</li>'}</ul></div><div class="drawer-actions"><button class="primary" data-message="${p.id}">Draft AI email</button>${p.website?`<a class="secondary" href="${esc(p.website)}" target="_blank" rel="noopener">Open website ↗</a>`:''}</div>`;$('detailDrawer').classList.add('open');$('detailDrawer').setAttribute('aria-hidden','false')}
 document.addEventListener('click',async e=>{const open=e.target.closest('[data-open]');if(open)openDrawer(open.dataset.open);const msg=e.target.closest('[data-message]');if(msg)openMessage(msg.dataset.message);if(e.target.id==='exportCsv')location.href='/api/export.csv';if(e.target.id==='refreshProspects')route();if(e.target.id==='closeDrawer'||!e.target.closest('#detailDrawer')&&$('detailDrawer').classList.contains('open')&&!e.target.closest('[data-open]')){$('detailDrawer').classList.remove('open');$('detailDrawer').setAttribute('aria-hidden','true')}});
 document.addEventListener('change',async e=>{if(e.target.dataset.review){try{const p=await api(`/api/prospects/${e.target.dataset.review}/review`,{method:'PATCH',body:JSON.stringify({status:e.target.value})});const i=prospects.findIndex(x=>x.id===p.id);if(i>=0)prospects[i]=p}catch(err){alert(err.message)}}if(e.target.id==='prospectSearch'||e.target.id==='prospectStatus')$('prospectTable').innerHTML=prospectTable();if(e.target.id==='aiProvider'){const nous=e.target.value==='nous';$('aiBase').readOnly=nous;$('aiBase').value=nous?'https://inference-api.nousresearch.com/v1':$('aiBase').value;const note=$('aiProviderNote');if(note)note.textContent=nous?'Nous uses the official inference endpoint. OAuth is supported by Hermes CLI; this standalone app uses a server-side API key.':'Use any OpenAI-compatible /v1 endpoint.';if(nous){try{const data=await api('/api/ai/models?provider=nous');const model=$('aiModel');const list=$('nousModelList');const models=data.models||[];if(list)list.innerHTML=models.map(m=>`<option value="${esc(m.id)}" label="${esc(m.label||m.id)}"></option>`).join('');if(model&&models.length&&!models.some(m=>m.id===model.value))model.value=models[0].id;}catch(err){}}}});
 document.addEventListener('submit',async e=>{if(e.target.id==='discoverForm'){e.preventDefault();const m=$('discoverMessage');$('discoverBtn').disabled=true;m.textContent='Queueing…';try{const r=await api('/api/discover',{method:'POST',body:JSON.stringify({keywords:$('keywords').value,category:$('category').value,location:$('location').value,max_results:Number($('max-results').value),extra_reviews:$('extra-reviews').checked,include_email:true})});m.textContent=`Run #${r.id} started. View it in Runs.`;setTimeout(route,500)}catch(x){m.textContent=x.message}finally{$('discoverBtn').disabled=false}}if(e.target.id==='settingsForm'){e.preventDefault();const m=$('settingsMessage');try{const ai={AI_PROVIDER:$('aiProvider').value,AI_BASE_URL:$('aiBase').value,AI_MODEL:$('aiModel').value};if($('aiKey').value)ai.AI_API_KEY=$('aiKey').value;await api('/api/settings',{method:'POST',body:JSON.stringify({ai})});m.textContent='Settings saved.';m.className='notice ok';}catch(x){m.textContent=x.message}}if(e.target.id==='scheduleForm'){e.preventDefault();try{await api('/api/schedules',{method:'POST',body:JSON.stringify({category:$('sCat').value,keywords:$('sKey').value,location:$('sLoc').value,interval_hours:Number($('sHours').value)})});route()}catch(x){alert(x.message)}}if(e.target.id==='suppressionForm'){e.preventDefault();try{await api('/api/suppressions',{method:'POST',body:JSON.stringify({kind:$('supKind').value,value:$('supValue').value,reason:$('supReason').value})});route()}catch(x){alert(x.message)}}});
-document.addEventListener('click',async e=>{if(e.target.id==='generateMessage'){if(!selected)return;const s=$('messageStatus');s.textContent='Generating…';try{const d=await api(`/api/prospects/${selected.id}/message`,{method:'POST',body:JSON.stringify({offer:$('offerInput').value})});$('subjectOutput').value=d.subject;$('bodyOutput').value=d.message;s.textContent='Draft ready. Review and edit before sending.';s.className='notice ok'}catch(x){s.textContent=x.message;s.className='notice error'}}if(e.target.id==='copyMessage'){await navigator.clipboard.writeText(`Subject: ${$('subjectOutput').value}\n\n${$('bodyOutput').value}`);$('messageStatus').textContent='Copied to clipboard.'}if(e.target.id==='testAi'){const s=$('settingsMessage');s.textContent='Testing…';s.className='notice';try{const d=await api('/api/ai/test',{method:'POST',body:'{}'});s.textContent=`Connection works · ${d.provider} · ${d.model}`;s.className='notice ok'}catch(x){s.textContent=x.message;s.className='notice error'}}});
+document.addEventListener('click',async e=>{if(e.target.id==='generateMessage'){if(!selected)return;const s=$('messageStatus');const button=e.target;button.disabled=true;button.textContent='Generating…';s.textContent='The AI is comparing the evidence with your offer…';try{const d=await api(`/api/prospects/${selected.id}/message`,{method:'POST',body:JSON.stringify({offer:$('offerInput').value})});$('subjectOutput').value=d.subject;$('bodyOutput').value=d.message;selected={...selected,ai_subject:d.subject,ai_message:d.message,ai_offer:$('offerInput').value,ai_reasoning:d.rationale||'',ai_unknowns:JSON.stringify(d.unknowns||[]),ai_evidence_ids:JSON.stringify(d.evidence_used||[]),ai_follow_up:d.follow_up||''};const i=prospects.findIndex(p=>String(p.id)===String(selected.id));if(i>=0)prospects[i]=selected;showDraftNotes(d);s.textContent='Draft ready. Review, edit, and confirm the facts before sending.';s.className='notice ok'}catch(x){s.textContent=x.message;s.className='notice error'}finally{button.disabled=false;button.textContent='Regenerate draft'}}if(e.target.id==='copyMessage'){await navigator.clipboard.writeText(`Subject: ${$('subjectOutput').value}\n\n${$('bodyOutput').value}`);$('messageStatus').textContent='Copied to clipboard.'}if(e.target.id==='testAi'){const s=$('settingsMessage');s.textContent='Testing…';s.className='notice';try{const d=await api('/api/ai/test',{method:'POST',body:'{}'});s.textContent=`Connection works · ${d.provider} · ${d.model}`;s.className='notice ok'}catch(x){s.textContent=x.message;s.className='notice error'}}});
 function aiRecommendationLabel(value){return value==='contact'?'Contact recommended':value==='review'?'Review first':'Do not contact yet'}
 function addAssessmentPanel(id){const p=prospects.find(x=>String(x.id)===String(id));const body=$('drawerBody');if(!p||!body)return;const has=Number.isFinite(Number(p.ai_probability))&&p.ai_checked_at;const section=document.createElement('section');section.className='drawer-section';section.id='aiAssessmentPanel';section.innerHTML=has?`<h3>AI assessment</h3><div class="detail-grid"><span>Contact probability<b>${Math.round(Number(p.ai_probability))}%</b></span><span>Recommendation<b>${esc(aiRecommendationLabel(p.ai_recommendation))}</b></span></div><p>${esc(p.ai_reasoning||'No rationale returned.')}</p><p class="muted">Assessed ${date(p.ai_checked_at)}</p><button class="secondary" data-assess="${p.id}">Reassess with AI</button>`:`<h3>AI assessment</h3><p>Use the collected evidence to estimate whether this company is worth contacting.</p><button class="secondary" data-assess="${p.id}">Assess with AI</button>`;const actions=body.querySelector('.drawer-actions');if(actions)body.insertBefore(section,actions);else body.append(section)}
-const _openDrawerWithAssessment=openDrawer;openDrawer=function(id){_openDrawerWithAssessment(id);addAssessmentPanel(id)}
+function candidateEvidence(candidate){try{const value=JSON.parse(candidate.evidence||'[]');return Array.isArray(value)?value.filter(Boolean).map(String):[]}catch{return[]}}
+function renderWebsiteCandidates(items,id){const section=$('websiteCandidatesPanel');if(!section)return;const rows=Array.isArray(items)?items:[];section.innerHTML=`<h3>Possible official websites</h3><p class="muted">Review the evidence before accepting a domain. Accepting runs the normal website and public-email scan.</p><button class="secondary" data-find-websites="${id}">Search again</button>${rows.length?`<div class="candidate-list">${rows.map(c=>{const evidence=candidateEvidence(c);const status=c.status||'candidate';return `<div class="candidate-card"><div><a href="${esc(c.url)}" target="_blank" rel="noopener">${esc(c.domain)} ↗</a><span class="badge ${status==='accepted'?'good':status==='rejected'?'warn':'info'}">${esc(status)}</span></div><p>Confidence <b>${Number(c.confidence||0)}%</b></p>${evidence.length?`<ul>${evidence.map(item=>`<li>${esc(item)}</li>`).join('')}</ul>`:'<p class="muted">No evidence returned.</p>'}${status==='candidate'?`<div class="candidate-actions"><button class="primary" data-accept-website="${c.id}" data-prospect-id="${id}">Accept domain</button><button class="secondary" data-reject-website="${c.id}" data-prospect-id="${id}">Reject</button></div>`:''}</div>`}).join('')}</div>`:'<p class="muted">No plausible official website candidates found yet.</p>'}`}
+function addWebsitePanel(id){const body=$('drawerBody');const p=prospects.find(x=>String(x.id)===String(id));if(!p||!body)return;const section=document.createElement('section');section.className='drawer-section';section.id='websiteCandidatesPanel';section.innerHTML=`<h3>Find a possible website</h3><p>Searches a few permitted public pages for a likely official domain when Maps did not list one. It never logs in or bypasses access challenges.</p><button class="secondary" data-find-websites="${p.id}">Find possible websites</button><p class="muted">Requires a tested AI provider in Settings.</p>`;const actions=body.querySelector('.drawer-actions');if(actions)body.insertBefore(section,actions);else body.append(section);api(`/api/prospects/${p.id}/website-candidates`).then(d=>{if((d.items||[]).length)renderWebsiteCandidates(d.items,p.id)}).catch(()=>{})}
+const _openDrawerWithAssessment=openDrawer;openDrawer=function(id){_openDrawerWithAssessment(id);addAssessmentPanel(id);addWebsitePanel(id)}
 async function assessProspect(id,button){if(button){button.disabled=true;button.textContent='Assessing…'}try{const d=await api(`/api/prospects/${id}/assess`,{method:'POST',body:'{}'});const index=prospects.findIndex(p=>String(p.id)===String(id));if(index>=0)prospects[index]={...prospects[index],...d};selected=prospects[index]||selected;openDrawer(id)}catch(error){const panel=$('aiAssessmentPanel');if(panel){const notice=document.createElement('p');notice.className='notice error';notice.textContent=error.message;panel.append(notice)}if(button){button.disabled=false;button.textContent='Assess with AI'}}}
 document.addEventListener('click',async e=>{const button=e.target.closest('[data-assess]');if(button){e.preventDefault();e.stopPropagation();await assessProspect(button.dataset.assess,button)}});
+async function findWebsiteCandidates(id,button){if(button){button.disabled=true;button.textContent='Searching…'}const section=$('websiteCandidatesPanel');try{const d=await api(`/api/prospects/${id}/website-candidates`,{method:'POST',body:'{}'});renderWebsiteCandidates(d.items||[],id)}catch(error){if(section){const notice=document.createElement('p');notice.className='notice error';notice.textContent=error.message;section.append(notice)}}finally{if(button){button.disabled=false;button.textContent='Search again'}}}
+document.addEventListener('click',async e=>{const find=e.target.closest('[data-find-websites]');if(find){e.preventDefault();e.stopPropagation();await findWebsiteCandidates(find.dataset.findWebsites,find);return}const accept=e.target.closest('[data-accept-website]');if(accept){e.preventDefault();e.stopPropagation();accept.disabled=true;accept.textContent='Checking…';try{const d=await api(`/api/prospects/${accept.dataset.prospectId}/website-candidates/${accept.dataset.acceptWebsite}`,{method:'PATCH',body:JSON.stringify({status:'accepted'})});if(d.prospect){const i=prospects.findIndex(p=>String(p.id)===String(d.prospect.id));if(i>=0)prospects[i]={...prospects[i],...d.prospect};selected=d.prospect}openDrawer(accept.dataset.prospectId);if($('prospectTable'))$('prospectTable').innerHTML=prospectTable()}catch(error){accept.disabled=false;accept.textContent='Accept domain';const panel=$('websiteCandidatesPanel');if(panel){const notice=document.createElement('p');notice.className='notice error';notice.textContent=error.message;panel.append(notice)}}return}const reject=e.target.closest('[data-reject-website]');if(reject){e.preventDefault();e.stopPropagation();reject.disabled=true;try{const d=await api(`/api/prospects/${reject.dataset.prospectId}/website-candidates/${reject.dataset.rejectWebsite}`,{method:'PATCH',body:JSON.stringify({status:'rejected'})});const current=await api(`/api/prospects/${reject.dataset.prospectId}/website-candidates`);renderWebsiteCandidates(current.items||[],reject.dataset.prospectId)}catch(error){reject.disabled=false;const panel=$('websiteCandidatesPanel');if(panel){const notice=document.createElement('p');notice.className='notice error';notice.textContent=error.message;panel.append(notice)}}}});
 window.addEventListener('hashchange',route);$('refreshAll').addEventListener('click',route);route();
+document.addEventListener('click',e=>{const tab=e.target.closest('[data-category-tab]');if(!tab)return;e.preventDefault();e.stopPropagation();activeProspectCategory=tab.dataset.categoryTab||'all';if($('prospectTabs'))$('prospectTabs').innerHTML=prospectTabs();if($('prospectTable'))$('prospectTable').innerHTML=prospectTable()});

web/index.html

@@ -1,3 +1,3 @@
 <!doctype html>
-<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="theme-color" content="#0d1726"><title>ProspectOS</title><link rel="stylesheet" href="/style.css?v=ui-v14"></head>
+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><meta name="theme-color" content="#0d1726"><title>ProspectOS</title><link rel="stylesheet" href="/style.css?v=ui-v16"></head>
-<body><div class="app-shell"><aside class="sidebar"><div class="brand"><span class="brand-mark">P</span><div><b>ProspectOS</b><small>Growth workspace</small></div></div><nav id="nav"><a href="#dashboard" data-route="dashboard"><span>⌂</span>Dashboard</a><a href="#discover" data-route="discover"><span>⌁</span>Discover</a><a href="#prospects" data-route="prospects"><span>▦</span>Prospects</a><a href="#runs" data-route="runs"><span>◷</span>Runs</a><a href="#settings" data-route="settings"><span>⚙</span>Settings</a></nav><div class="sidebar-foot"><span class="status-dot" id="sourceDot"></span><div><b id="sourceLabel">Checking worker</b><small id="sourceSub">Connecting…</small></div></div></aside><main class="main"><header class="topbar"><div><span class="crumb">Workspace</span><span class="slash">/</span><b id="pageName">Dashboard</b></div><div class="top-actions"><span id="updated">—</span><button class="icon-btn" id="refreshAll" title="Refresh">↻</button></div></header><div id="view"></div><footer>ProspectOS · public data only · manual review before contact</footer></main></div><aside id="detailDrawer" class="detail-drawer" aria-hidden="true"><div class="drawer-head"><div><span class="eyebrow">PROSPECT DETAIL</span><h2 id="drawerName">Prospect</h2></div><button class="icon-btn" id="closeDrawer">×</button></div><div id="drawerBody"></div></aside><dialog id="messageDialog"><form method="dialog" id="messageForm"><div class="dialog-head"><div><label class="eyebrow">AI ASSISTED DRAFT</label><h2 id="messageTitle">Tailor a message</h2><p>Use the evidence and your offer to create an editable draft.</p></div><button class="icon-btn" value="cancel">×</button></div><label for="offerInput">Your offer or instruction</label><textarea id="offerInput" rows="4" placeholder="Move hosting to Gold Plus for R119/month and rebuild the website free."></textarea><div id="messageStatus" class="notice"></div><button class="primary" id="generateMessage" type="button">Generate subject and email</button><label for="subjectOutput">Subject</label><input id="subjectOutput"><label for="bodyOutput">Email draft</label><textarea id="bodyOutput" rows="12"></textarea><div class="dialog-actions"><button class="secondary" id="copyMessage" type="button">Copy draft</button><button class="secondary" value="cancel">Close</button></div></form></dialog><script src="/app.js?v=ui-v15"></script></body></html>
+<body><div class="app-shell"><aside class="sidebar"><div class="brand"><span class="brand-mark">P</span><div><b>ProspectOS</b><small>Growth workspace</small></div></div><nav id="nav"><a href="#dashboard" data-route="dashboard"><span>⌂</span>Dashboard</a><a href="#discover" data-route="discover"><span>⌁</span>Discover</a><a href="#prospects" data-route="prospects"><span>▦</span>Prospects</a><a href="#runs" data-route="runs"><span>◷</span>Runs</a><a href="#settings" data-route="settings"><span>⚙</span>Settings</a></nav><div class="sidebar-foot"><span class="status-dot" id="sourceDot"></span><div><b id="sourceLabel">Checking worker</b><small id="sourceSub">Connecting…</small></div></div></aside><main class="main"><header class="topbar"><div><span class="crumb">Workspace</span><span class="slash">/</span><b id="pageName">Dashboard</b></div><div class="top-actions"><span id="updated">—</span><button class="icon-btn" id="refreshAll" title="Refresh">↻</button></div></header><div id="view"></div><footer>ProspectOS · public data only · manual review before contact</footer></main></div><aside id="detailDrawer" class="detail-drawer" aria-hidden="true"><div class="drawer-head"><div><span class="eyebrow">PROSPECT DETAIL</span><h2 id="drawerName">Prospect</h2></div><button class="icon-btn" id="closeDrawer">×</button></div><div id="drawerBody"></div></aside><dialog id="messageDialog"><form method="dialog" id="messageForm"><div class="dialog-head"><div><label class="eyebrow">AI ASSISTED DRAFT</label><h2 id="messageTitle">Tailor a message</h2><p>Use the evidence and your offer to create an editable draft.</p></div><button class="icon-btn" value="cancel">×</button></div><label for="offerInput">Your offer or instruction</label><textarea id="offerInput" rows="4" placeholder="Move hosting to Gold Plus for R119/month and rebuild the website free."></textarea><div id="messageStatus" class="notice"></div><button class="primary" id="generateMessage" type="button">Generate subject and email</button><label for="subjectOutput">Subject</label><input id="subjectOutput"><label for="bodyOutput">Email draft</label><textarea id="bodyOutput" rows="12"></textarea><div class="dialog-actions"><button class="secondary" id="copyMessage" type="button">Copy draft</button><button class="secondary" value="cancel">Close</button></div></form></dialog><script src="/app.js?v=prospects-tabs-v1"></script></body></html>

web/style.css

@@ -106,6 +106,8 @@ input::placeholder { color: #aeb6c5; } input:focus, select:focus { border-color:
 
 .setup-section { max-width: 1160px; display: grid; grid-template-columns: 1.2fr 1fr; gap: 30px; margin-top: 17px; padding: 22px 24px; border: 1px solid #e3e6ee; border-radius: 13px; background: #fbfcfe; }.setup-section h2 { margin: 5px 0; color: #38445a; font-size: 15px; }.setup-section p, .setup-section ol { margin: 6px 0 0; color: #8792a5; font-size: 11px; }.setup-section ol { padding-left: 17px; line-height: 1.9; }.setup-section code { padding: 2px 4px; border-radius: 4px; background: #eff1f6; color: #5d687d; font-size: 10px; }.footer { max-width: 1160px; display: flex; justify-content: space-between; gap: 20px; padding: 23px 2px 0; color: #a3acbb; font-size: 10px; }
 .muted { color: #9ca5b5; font-size: 11px; }
+.draft-notes{padding:12px;border:1px solid #e6eaf2;border-radius:9px;background:#fafbff;color:#536178;font-size:11px;line-height:1.5}.draft-notes b{color:#334158}.draft-notes p{margin:4px 0}.draft-notes .muted{font-size:10px}
+.prospect-tabs{margin:4px 0 22px;padding:14px 0 0;border-top:1px solid #edf0f5}.prospect-tab-list{display:flex;gap:8px;overflow-x:auto;padding:2px 1px 3px}.prospect-tab{border:1px solid #dfe4ed;background:#fff;color:#536178;border-radius:999px;padding:9px 13px;white-space:nowrap;font:inherit;font-size:12px;font-weight:750;cursor:pointer;transition:.18s}.prospect-tab:hover{border-color:#b9b5ff;background:#f7f6ff;color:#4f47df}.prospect-tab.active{background:#635bff;border-color:#635bff;color:#fff;box-shadow:0 6px 14px #635bff2e}.prospect-tab span{display:inline-grid;place-items:center;min-width:20px;height:20px;margin-left:5px;padding:0 5px;border-radius:99px;background:#eef1f7;color:#647188;font-size:10px}.prospect-tab.active span{background:#fff3;color:#fff}.tab-hint{margin:10px 0 0;color:#8a96a8;font-size:11px}.candidate-list{display:grid;gap:10px;margin-top:15px}.candidate-card{padding:13px;border:1px solid #e6eaf2;border-radius:10px;background:#fbfcff}.candidate-card>div:first-child{display:flex;align-items:center;gap:8px}.candidate-card a{color:#4f47df;font-weight:750;text-decoration:none;font-size:13px}.candidate-card p{font-size:12px!important;color:#68758a!important}.candidate-card ul{padding-left:18px;margin:7px 0}.candidate-card li{font-size:11px!important;padding:2px 0!important}.candidate-actions{display:flex;gap:7px;margin-top:10px}.candidate-actions button{padding:8px 10px;font-size:11px}
 
 @media (max-width: 980px) { .app-shell { grid-template-columns: 72px minmax(0, 1fr); }.sidebar { padding: 22px 10px 16px; }.brand { justify-content: center; padding: 0; }.brand > div, .side-nav a:not(.active) { display: none; }.side-nav { margin-top: 44px; }.side-nav a { justify-content: center; padding: 12px 0; }.side-nav .nav-icon { width: auto; }.source-card { justify-content: center; padding: 11px 0; }.source-card > div, .sidebar-foot p { display: none; }.main-content { padding: 0 25px 34px; }.workspace-grid { grid-template-columns: 1fr; }.run-panel { order: -1; }.run-panel .mini-history { display: none; } }
 @media (max-width: 680px) { .app-shell { display: block; }.sidebar { position: static; height: auto; flex-direction: row; align-items: center; justify-content: space-between; padding: 13px 16px; }.side-nav { display: flex; gap: 3px; margin: 0; }.side-nav a, .side-nav a:not(.active) { display: flex; padding: 8px 10px; }.sidebar-foot { display: none; }.main-content { padding: 0 14px 30px; }.topbar { height: 53px; }.updated { display: none; }.page-heading { display: block; padding: 27px 0 20px; }.heading-badge { display: inline-grid; margin-top: 17px; }.stats-grid { grid-template-columns: repeat(2, 1fr); gap: 9px; }.stat-card { min-height: 105px; padding: 14px; }.stat-card strong { font-size: 24px; }.discovery-panel, .run-panel, .explorer-panel { padding: 17px; }.discovery-form { grid-template-columns: 1fr; gap: 13px; }.field.wide, .options.wide, .form-footer.wide { grid-column: auto; }.options { display: grid; gap: 12px; }.form-footer { display: block; }.form-message { display: block; margin-top: 10px; }.panel-heading, .explorer-heading { display: block; }.source-tag { display: inline-flex; margin-top: 14px; }.explorer-actions { display: grid; grid-template-columns: 1fr 1fr; margin-top: 15px; }.search-box { width: auto; grid-column: 1 / -1; }.explorer-actions select, .explorer-actions .quiet-btn { width: 100%; }.prospect-card { grid-template-columns: 1fr; gap: 12px; }.review-control { align-items: flex-start; }.review-control select { width: 100%; }.prospect-meta { flex-wrap: wrap; }.meta-spacer { display: none; }.setup-section { display: block; padding: 17px; }.setup-section ol { margin-top: 14px; }.footer { display: block; line-height: 1.8; } }