MarketingTool/README.md

# ProspectOS · Maps Scout

A small, personal-use prospect workspace. It sends one search at a time to
the bundled [gosom/google-maps-scraper](https://github.com/gosom/google-maps-scraper)
worker, waits for its asynchronous job to finish, filters places already in
the local catalogue, and presents the remaining places for human review.

The interface is split into Dashboard, Discover, Prospects, Runs, Logs, Teach AI,
Promotions, and Settings
views. It includes website health and email evidence, explainable scoring,
domain ideas, suppression, CSV export, scheduled runs, rescans, and human review. It never sends outreach
automatically. The Prospects explorer has an **All high potential** tab and
category tabs populated from each discovery run, with every tab sorted from
highest to lowest opportunity score. It also provides search, potential-score,
website-health, public-email, and review-status filters; review status itself
is changed inside the prospect drawer so list rows stay focused on evidence.

Optional AI can analyse a prospect and draft a subject/email from supplied
evidence and your offer; you edit and approve the draft yourself. Drafts now
also return the reasoning, stored evidence IDs used, facts that still need
confirmation, and a follow-up review note so a polished email is not mistaken
for verified information.
The draft dialog also includes a revision chat: after generating a draft, ask
for changes such as “make this warmer”, “shorten the opening”, or “use the
selected promotion more naturally”. The chat sends the current editable
subject/body and applies the returned revision in place; it never sends an
email. Every revision is saved on the prospect and recorded in the AI/audit
logs for review.
Draft instructions explicitly ask the model to choose a prospect-specific
angle, use one or two real facts, vary its opening and close, avoid stock
greetings and marketing filler, and read naturally aloud. The strategy note
and variation hint are private planning context rather than copy to paste, so
each run can use a different natural phrasing without inventing claims.
The **Teach AI** view stores the sender company's website, services, packages,
prices, and writing guidance. Its bounded public-page reader captures a
reviewable homepage text snapshot that is included in AI assessments and email
drafts. The **Promotions** view stores dated, active/inactive offers; an active
promotion can be selected in the email draft dialog and is passed to the AI
with its exact description and terms.
From a prospect's detail drawer, **Generate modern mockup** creates a
shareable, mobile-first website concept using the prospect's public details
and (when available) a fresh snapshot of its current homepage. The preview is
a standalone HTML document (or a four-page HTML site) ready to place in a
PHP/HTML hosting folder; it does not require PHP, JavaScript, forms, tracking,
external fonts, or unapproved remote assets to render. If AI is configured it can produce a tailored design; when
AI is unavailable the deterministic fallback still produces a polished
preview. Each tokenised link is read-only, marked as a concept, sent with
`noindex`, and can be included once in a human-reviewed email draft. It is not
a production site and should be checked and edited before sending.
The design prompt receives a small category-matched library of Unsplash stock
images; generated pages may use only those supplied image URLs, and the
fallback template uses the same curated set.
Each saved concept also has an **Edit template** action. It opens the shareable
site beside a chat-style editor where you can ask for copy, emphasis, palette,
imagery, or layout changes. The request is sent to the configured AI with the
prospect's public evidence and rendered through the same deterministic,
sanitised template renderer. Every successful edit creates a new version and
share token; earlier versions remain in the prospect drawer and are never
overwritten. If AI is unavailable, safe visual hints such as “make it darker”
still select one of the built-in palettes.
The **Logs** view separates API calls, user actions, AI requests, searches,
Browser Use/tool steps, and errors. AI request payloads and responses are
truncated and credentials such as API keys, authorization headers, tokens, and
passwords are redacted. Logs are retained up to `LOG_RETENTION` entries (20,000
by default).
From a prospect's detail drawer, **Assess with AI** stores a contact probability,
recommendation, and short evidence-grounded rationale for that specific company.
The same drawer can optionally run a bounded website investigation to look for
a likely official website when Maps did not list one. The investigator first
opens public Bing/alternative result pages with headless Playwright and reads
the visible result links and snippets directly from the DOM; it does not rely
on unstable Browser Use element indexes. Search links are persisted immediately
and then checked for reachability/title identity evidence. Browser Use remains
as a fallback when no parseable result links are found. Candidate domains are
shown with evidence and must be accepted by you before they replace the
catalogue record.

Prospect review status is changed inside the detail drawer (In review,
Accepted, Rejected, or Contacted), keeping the explorer focused on discovery
categories and opportunity scores. The AI draft dialog shows the prospect
evidence, lets you choose a voice and call-to-action, and produces an editable
subject and plain-text email using a stricter evidence-grounded prompt.

## Start the stack

```bash
cp .env.example .env
docker compose up -d --build
```

Open `http://SERVER:8000` for ProspectOS. The Maps worker is available on
`http://SERVER:8081` if you want to inspect its native Web UI.

### How the worker connects

No Google Cloud API key is required. The bundled standard worker exposes a
local REST API on the Docker network, and ProspectOS calls it directly.

The app uses `/api/v1/jobs` to queue a search, polls `/api/v1/jobs/{id}`, and
downloads `/api/v1/jobs/{id}/download` when the job finishes. Jobs are
deliberately asynchronous because a Maps scrape can take several minutes.

Only set `GMAPS_API_MODE=saas` and `GMAPS_API_KEY=...` if you deliberately
replace the bundled worker with gosom's optional SaaS edition.

A run that loses its worker connection is shown as failed with a specific
error rather than remaining stuck forever.

## Configuration

| Variable | Default | Purpose |
|---|---:|---|
| `HOST_PORT` | `8000` | Host port for ProspectOS |
| `PUBLIC_BASE_URL` | empty | Public origin used for client-facing mockup share links; set it behind a reverse proxy if forwarded host headers are unavailable |
| `GMAPS_HOST_PORT` | `8081` | Host port for the Maps worker admin UI |
| `GMAPS_API_MODE` | `standard` | `standard` for the bundled worker; `saas` only for the optional SaaS API |
| `GMAPS_API_KEY` | empty | Not required in standard mode; only used in SaaS mode |
| `GMAPS_REQUEST_TIMEOUT` | `900` | Maximum wait for the Maps worker in seconds; raise it for very large searches |
| `GMAPS_POLL_SECONDS` | `2` | Poll interval while a job is running |
| `OUTBOUND_USER_AGENT` | `ProspectOS/2.0 (+personal-use)` | Identifies provider HTTP requests; avoids Python's default client fingerprint |
| `DISCOVERY_BACKEND` | `gmaps` when `GMAPS_URL` is set | Set to `browser` only for the bounded public-search fallback |
| `BROWSER_USE_ENABLED` | `true` | Enables the optional website-candidate investigator |
| `BROWSER_USE_MAX_STEPS` | `10` | Maximum Browser Use actions for one investigation (3–50); can be changed in Settings |
| `BROWSER_USE_TIMEOUT_SECONDS` | `600` | Hard timeout for one investigation (30–1800 seconds); can be changed in Settings |
| `BROWSER_USE_LLM_TIMEOUT_SECONDS` | `90` | Timeout for one Browser Use model call (10–300 seconds); can be changed in Settings |
| `WEBSITE_SCAN_TIMEOUT` | `8` | Per-page public website/email scan timeout (2–60 seconds); can be changed in Settings |
| `ANONYMIZED_TELEMETRY` | `false` | Keeps Browser Use telemetry disabled in this personal deployment |
| `LOG_RETENTION` | `20000` | Maximum operational log entries retained in SQLite |
| `AI_PROVIDER` | `custom` | Set to `nous` for the Nous Research Portal, or `custom` for another OpenAI-compatible endpoint |
| `AI_BASE_URL` | empty | Optional OpenAI-compatible `/chat/completions` endpoint |
| `AI_API_KEY` | empty | Optional provider key; keep server-side |
| `AI_MODEL` | empty | Model name used for evidence-grounded drafts |
| `AI_MOCKUP_TIMEOUT_SECONDS` | `45` | Maximum AI wait for mockup copy before the deterministic premium renderer is used (20–300 seconds) |

### Nous Research Portal

Choose **Nous Portal · free models** in Settings, paste a Nous API key, select a
free model, save, and use **Test connection** before drafting a message. The
app uses `https://inference-api.nousresearch.com/v1` automatically and tries
the live `/models` catalog; if the catalog is unavailable it shows a stable
fallback list of the free models currently published by Nous. Availability and
model IDs can change, so refresh Settings periodically.

If a Nous request returns `ai_cloudflare_1010`, rebuild the app from the latest
source so provider requests include the explicit `ProspectOS/2.0` client
identity. The old build used Python's default `Python-urllib` fingerprint, which
can be rejected by the Portal's Cloudflare policy. This is not a Maps error and
it cannot be fixed by changing the model name. If the rebuilt app still receives
1010, test the same key from the server with the provider's documented
OpenAI-compatible endpoint and contact Nous support if the endpoint continues to
block that client; do not try to evade the security challenge.

The bundled fallback entries are LongCat 2.0, Solar Pro 4, Step 3.7 Flash,
Laguna S 2.1, Laguna XS 2.1, Ling 3.0 Flash Fin, and Ling 3.0 Flash Sante
(using the Portal's `:free` model suffix).

Hermes Agent has its own browser-based OAuth flow, but that flow stores a
Hermes refresh token and is not a portable credential for this standalone
Docker app. ProspectOS therefore uses a server-side Nous API key here. The key
is never returned by the Settings API or sent to the browser after saving.

### Browser Use website investigator

Browser Use is an optional local browser agent. The website finder first uses
deterministic Playwright DOM extraction so a search result cannot be lost just
because an AI click target became stale. It uses the AI provider only as a
bounded fallback when no usable result link was extracted. Both paths are
limited to unauthenticated public research: no proxy rotation, stealth
fingerprints, CAPTCHA solving, login, or challenge bypass is configured. If a
site blocks the browser, the action records the block and leaves any partial
candidate links intact. Browser Use itself is open source, but fallback
investigations still use your configured model/provider and may consume quota.

To use it:

1. Configure and test an AI provider in **Settings** (a Nous key plus a selected
   model also works).
2. Open a prospect that has no detected website and choose **Find possible
   websites**.
3. Review the returned domains and evidence. Choose **Accept** only when the
   domain is genuinely the business's official site; accepting runs the normal
   website health and public-email scan.

The API equivalents are `POST /api/prospects/{id}/website-candidates`,
`GET /api/prospects/{id}/website-candidates`, and
`PATCH /api/prospects/{id}/website-candidates/{candidate_id}` with
`{"status":"accepted"}` or `{"status":"rejected"}`.

### Runtime settings

Open **Settings → Browser and discovery timing** to change Browser Use's
enabled state, total investigation timeout, maximum agent steps, and per-call
AI timeout. The same panel controls the Maps worker wait timeout/poll interval,
search depth, website health/email scan timeout, and the maximum wait for
AI-written mockup copy. Values are stored in the local SQLite database, survive
container restarts, and override `.env` defaults for new runs. The safe UI
ranges prevent a malformed value from creating an unbounded worker; a slow
mockup provider automatically falls back to the complete local renderer.

The app stores its catalogue in the `prospect_data` Docker volume. The worker
stores its own job data in `gmaps_data`; keep both volumes if you want to retain
history. To inspect status:

```bash
docker compose ps
docker compose logs --tail=100 app
docker compose logs --tail=100 gmaps
```

## Local fallback mode

For a quick local smoke test without the Maps worker:

```bash
DISCOVERY_BACKEND=browser GMAPS_URL= docker compose up -d --build
```

This uses Playwright against public search engines, keeps the same catalogue
and review UI, and does not bypass CAPTCHAs or other access controls.

## Data and review behavior

Each saved place includes its name, Maps link, website (when present), root
domain, address, phone, category, rating, review count, public emails returned
by the worker when available, source ID, and raw result JSON. A place
with no website is still retained using a stable `maps:<place_id>` catalogue
key, so it can be reviewed as a potential website opportunity. Existing
websites are deduplicated by root domain.

Email extraction is always enabled. The worker may visit each public business
website found by Maps; expect longer runs and review the resulting provenance
before using any contact information. It does not invent email addresses when
none are published.

Website scans are bounded and SSRF-safe. A scan records HTTP status, final URL,
latency, HTTPS/TLS, title, technology hints, contact-page emails, and errors.
Use `/api/prospects/{id}/scan` for a manual rescan. The explorer's “Draft AI
email” action calls `/api/prospects/{id}/message`, and **Assess with AI** calls
`/api/prospects/{id}/assess`; both require the optional AI
variables above and accepts your exact offer/instructions (for example, the
Gold Plus R119/month plan and a free rebuild). Suppressions are enforced before
new records are saved and can be managed through `/api/suppressions`.

The Teach AI page saves a `company_profile` object through `POST /api/settings`.
Its **Read website now** action calls `POST /api/company-profile/learn` and
stores a bounded, reviewable public homepage snapshot. Promotions are managed
with `GET/POST /api/promotions`, `PATCH /api/promotions/{id}`, and
`DELETE /api/promotions/{id}`. An active promotion selected in the draft dialog
is sent as `promotion_id` and retained on that prospect's draft.
The initial email draft is generated with `POST /api/prospects/{id}/message`.
The revision chat calls `POST /api/prospects/{id}/message/edit` with the current
`subject`, `message`, and a bounded wording instruction. The endpoint reuses
the same evidence and offer boundary, preserves a selected mockup preview URL,
and persists the revised subject, body, rationale, unknowns, and evidence IDs.
Mockups are created with `POST /api/prospects/{id}/mockup`, listed with
`GET /api/prospects/{id}/mockups`, and opened by their public
`/mockups/{token}` share URL. Choose `mode=onepage` for a single scrollable
landing page or `mode=multipage` for real Home, Services, About, and Contact
routes. Both modes use the same deterministic premium renderer: responsive
sticky navigation, layered hero imagery, editorial service cards, accessible
contact calls-to-action, a dark structured footer, hover states, and CSS-only
entrance/ambient animations with reduced-motion support. AI is asked only for
bounded copy and curated image choices; malformed, truncated, unavailable, or
slow AI responses fall back to the same complete renderer instead of failing
the request. Set `PUBLIC_BASE_URL` when the reverse proxy does not forward the
public host; otherwise the link is built from the request's forwarded host or
`Host` header.
Use `GET /api/mockups/{id}` to load editor metadata and
`POST /api/mockups/{id}/edit` with `{"instruction":"...","page":"all"}` to
create the next version. The edit endpoint requires reviewer access, limits
instructions to 3,000 characters, and accepts `all`, `home`, `services`,
`about`, or `contact` as the page scope.

## Tests

```bash
python3 -m py_compile api/server.py
node --check web/app.js
python3 -m unittest discover -s tests -q
```

This project is packaged for personal research. Public availability does not
override a website's terms, robots controls, or applicable privacy law; keep
searches small, use the worker only on data you are permitted to collect, and
do not turn the review screen into unattended outreach.

## Workspace accounts and operational controls

Authentication is enabled by default. On a fresh database, open the app and
complete the first-run owner setup. For unattended deployments, set
`OWNER_EMAIL`, `OWNER_PASSWORD` (8+ characters), `OWNER_NAME`, and
`WORKSPACE_NAME` in `.env`; the owner is created automatically only when no
user exists. Sessions are server-side, cookies are HTTP-only, and mutating
browser requests require a per-session CSRF token. Failed sign-ins are
temporarily rate-limited after five consecutive failures.

Roles are deliberately small:

| Role | Capabilities |
|---|---|
| Viewer | Read catalogue and runs |
| Reviewer | Viewer + review status, notes, and follow-up tasks |
| Researcher | Reviewer + discovery runs, retries, cancellation, and assignments |
| Admin | Researcher + users, workspace name, backups, and audit trail |
| Owner | Admin, and cannot be removed as the last active owner |

The **Users** page manages workspace members. A prospect drawer contains its
assignee, notes, tasks, review status, and score/AI assessment history. Every
workspace-scoped API query checks the signed-in organisation before returning
data.

## Backups and recovery

The app keeps the existing SQLite catalogue and performs additive schema
migrations; it does not delete or rewrite existing prospects. Use **Users →
Verified backups → Create backup** or run this inside the app container:

```bash
python scripts/backup_sqlite.py
```

Backups use SQLite's online backup API, run `PRAGMA integrity_check`, and keep
the newest `BACKUP_RETENTION` files in `BACKUP_DIR` (default `/data/backups`).
The background scheduler creates one automatically every `BACKUP_INTERVAL_HOURS`
(default 24; set it to `0` to disable automatic creation).
Include the `prospect_data` volume in server backups. A scheduled or interrupted
run is marked `recovery_required` after 30 minutes without a heartbeat. The
Runs page can show its persisted event timeline and retry a failed/recoverable
run as a new attempt, preserving the original run. Active jobs can be cancelled
and the worker records the cancellation in the event stream.

The audit trail records sign-ins, user and workspace changes, assignments,
review decisions, notes, tasks, AI assessments, message generation, run
retries/cancellations, settings, suppressions, and backup actions. Operational
logs remain available under **Logs**; credentials and authorization values are
redacted and retention is bounded by `LOG_RETENTION`.

This release intentionally keeps SQLite for safety. PostgreSQL migration can
be added later as a separate, tested data-migration project once the current
workspace and backup workflow is stable.