283 lines
15 KiB
Markdown
283 lines
15 KiB
Markdown
# ProspectOS · Maps Scout
|
||
|
||
A small, personal-use prospect workspace. It sends one search at a time to
|
||
the bundled [gosom/google-maps-scraper](https://github.com/gosom/google-maps-scraper)
|
||
worker, waits for its asynchronous job to finish, filters places already in
|
||
the local catalogue, and presents the remaining places for human review.
|
||
|
||
The interface is split into Dashboard, Discover, Prospects, Runs, Logs, Teach AI,
|
||
Promotions, and Settings
|
||
views. It includes website health and email evidence, explainable scoring,
|
||
domain ideas, suppression, CSV export, scheduled runs, rescans, and human review. It never sends outreach
|
||
automatically. The Prospects explorer has an **All high potential** tab and
|
||
category tabs populated from each discovery run, with every tab sorted from
|
||
highest to lowest opportunity score. It also provides search, potential-score,
|
||
website-health, public-email, and review-status filters; review status itself
|
||
is changed inside the prospect drawer so list rows stay focused on evidence.
|
||
|
||
Optional AI can analyse a prospect and draft a subject/email from supplied
|
||
evidence and your offer; you edit and approve the draft yourself. Drafts now
|
||
also return the reasoning, stored evidence IDs used, facts that still need
|
||
confirmation, and a follow-up review note so a polished email is not mistaken
|
||
for verified information.
|
||
The **Teach AI** view stores the sender company's website, services, packages,
|
||
prices, and writing guidance. Its bounded public-page reader captures a
|
||
reviewable homepage text snapshot that is included in AI assessments and email
|
||
drafts. The **Promotions** view stores dated, active/inactive offers; an active
|
||
promotion can be selected in the email draft dialog and is passed to the AI
|
||
with its exact description and terms.
|
||
The **Logs** view separates API calls, user actions, AI requests, searches,
|
||
Browser Use/tool steps, and errors. AI request payloads and responses are
|
||
truncated and credentials such as API keys, authorization headers, tokens, and
|
||
passwords are redacted. Logs are retained up to `LOG_RETENTION` entries (20,000
|
||
by default).
|
||
From a prospect's detail drawer, **Assess with AI** stores a contact probability,
|
||
recommendation, and short evidence-grounded rationale for that specific company.
|
||
The same drawer can optionally run a bounded website investigation to look for
|
||
a likely official website when Maps did not list one. The investigator first
|
||
opens public Bing/alternative result pages with headless Playwright and reads
|
||
the visible result links and snippets directly from the DOM; it does not rely
|
||
on unstable Browser Use element indexes. Search links are persisted immediately
|
||
and then checked for reachability/title identity evidence. Browser Use remains
|
||
as a fallback when no parseable result links are found. Candidate domains are
|
||
shown with evidence and must be accepted by you before they replace the
|
||
catalogue record.
|
||
|
||
Prospect review status is changed inside the detail drawer (In review,
|
||
Accepted, Rejected, or Contacted), keeping the explorer focused on discovery
|
||
categories and opportunity scores. The AI draft dialog shows the prospect
|
||
evidence, lets you choose a voice and call-to-action, and produces an editable
|
||
subject and plain-text email using a stricter evidence-grounded prompt.
|
||
|
||
## Start the stack
|
||
|
||
```bash
|
||
cp .env.example .env
|
||
docker compose up -d --build
|
||
```
|
||
|
||
Open `http://SERVER:8000` for ProspectOS. The Maps worker is available on
|
||
`http://SERVER:8081` if you want to inspect its native Web UI.
|
||
|
||
### How the worker connects
|
||
|
||
No Google Cloud API key is required. The bundled standard worker exposes a
|
||
local REST API on the Docker network, and ProspectOS calls it directly.
|
||
|
||
The app uses `/api/v1/jobs` to queue a search, polls `/api/v1/jobs/{id}`, and
|
||
downloads `/api/v1/jobs/{id}/download` when the job finishes. Jobs are
|
||
deliberately asynchronous because a Maps scrape can take several minutes.
|
||
|
||
Only set `GMAPS_API_MODE=saas` and `GMAPS_API_KEY=...` if you deliberately
|
||
replace the bundled worker with gosom's optional SaaS edition.
|
||
|
||
A run that loses its worker connection is shown as failed with a specific
|
||
error rather than remaining stuck forever.
|
||
|
||
## Configuration
|
||
|
||
| Variable | Default | Purpose |
|
||
|---|---:|---|
|
||
| `HOST_PORT` | `8000` | Host port for ProspectOS |
|
||
| `GMAPS_HOST_PORT` | `8081` | Host port for the Maps worker admin UI |
|
||
| `GMAPS_API_MODE` | `standard` | `standard` for the bundled worker; `saas` only for the optional SaaS API |
|
||
| `GMAPS_API_KEY` | empty | Not required in standard mode; only used in SaaS mode |
|
||
| `GMAPS_REQUEST_TIMEOUT` | `900` | Maximum wait for the Maps worker in seconds; raise it for very large searches |
|
||
| `GMAPS_POLL_SECONDS` | `2` | Poll interval while a job is running |
|
||
| `OUTBOUND_USER_AGENT` | `ProspectOS/2.0 (+personal-use)` | Identifies provider HTTP requests; avoids Python's default client fingerprint |
|
||
| `DISCOVERY_BACKEND` | `gmaps` when `GMAPS_URL` is set | Set to `browser` only for the bounded public-search fallback |
|
||
| `BROWSER_USE_ENABLED` | `true` | Enables the optional website-candidate investigator |
|
||
| `BROWSER_USE_MAX_STEPS` | `10` | Maximum Browser Use actions for one investigation (3–50); can be changed in Settings |
|
||
| `BROWSER_USE_TIMEOUT_SECONDS` | `600` | Hard timeout for one investigation (30–1800 seconds); can be changed in Settings |
|
||
| `BROWSER_USE_LLM_TIMEOUT_SECONDS` | `90` | Timeout for one Browser Use model call (10–300 seconds); can be changed in Settings |
|
||
| `WEBSITE_SCAN_TIMEOUT` | `8` | Per-page public website/email scan timeout (2–60 seconds); can be changed in Settings |
|
||
| `ANONYMIZED_TELEMETRY` | `false` | Keeps Browser Use telemetry disabled in this personal deployment |
|
||
| `LOG_RETENTION` | `20000` | Maximum operational log entries retained in SQLite |
|
||
| `AI_PROVIDER` | `custom` | Set to `nous` for the Nous Research Portal, or `custom` for another OpenAI-compatible endpoint |
|
||
| `AI_BASE_URL` | empty | Optional OpenAI-compatible `/chat/completions` endpoint |
|
||
| `AI_API_KEY` | empty | Optional provider key; keep server-side |
|
||
| `AI_MODEL` | empty | Model name used for evidence-grounded drafts |
|
||
|
||
### Nous Research Portal
|
||
|
||
Choose **Nous Portal · free models** in Settings, paste a Nous API key, select a
|
||
free model, save, and use **Test connection** before drafting a message. The
|
||
app uses `https://inference-api.nousresearch.com/v1` automatically and tries
|
||
the live `/models` catalog; if the catalog is unavailable it shows a stable
|
||
fallback list of the free models currently published by Nous. Availability and
|
||
model IDs can change, so refresh Settings periodically.
|
||
|
||
If a Nous request returns `ai_cloudflare_1010`, rebuild the app from the latest
|
||
source so provider requests include the explicit `ProspectOS/2.0` client
|
||
identity. The old build used Python's default `Python-urllib` fingerprint, which
|
||
can be rejected by the Portal's Cloudflare policy. This is not a Maps error and
|
||
it cannot be fixed by changing the model name. If the rebuilt app still receives
|
||
1010, test the same key from the server with the provider's documented
|
||
OpenAI-compatible endpoint and contact Nous support if the endpoint continues to
|
||
block that client; do not try to evade the security challenge.
|
||
|
||
The bundled fallback entries are LongCat 2.0, Solar Pro 4, Step 3.7 Flash,
|
||
Laguna S 2.1, Laguna XS 2.1, Ling 3.0 Flash Fin, and Ling 3.0 Flash Sante
|
||
(using the Portal's `:free` model suffix).
|
||
|
||
Hermes Agent has its own browser-based OAuth flow, but that flow stores a
|
||
Hermes refresh token and is not a portable credential for this standalone
|
||
Docker app. ProspectOS therefore uses a server-side Nous API key here. The key
|
||
is never returned by the Settings API or sent to the browser after saving.
|
||
|
||
### Browser Use website investigator
|
||
|
||
Browser Use is an optional local browser agent. The website finder first uses
|
||
deterministic Playwright DOM extraction so a search result cannot be lost just
|
||
because an AI click target became stale. It uses the AI provider only as a
|
||
bounded fallback when no usable result link was extracted. Both paths are
|
||
limited to unauthenticated public research: no proxy rotation, stealth
|
||
fingerprints, CAPTCHA solving, login, or challenge bypass is configured. If a
|
||
site blocks the browser, the action records the block and leaves any partial
|
||
candidate links intact. Browser Use itself is open source, but fallback
|
||
investigations still use your configured model/provider and may consume quota.
|
||
|
||
To use it:
|
||
|
||
1. Configure and test an AI provider in **Settings** (a Nous key plus a selected
|
||
model also works).
|
||
2. Open a prospect that has no detected website and choose **Find possible
|
||
websites**.
|
||
3. Review the returned domains and evidence. Choose **Accept** only when the
|
||
domain is genuinely the business's official site; accepting runs the normal
|
||
website health and public-email scan.
|
||
|
||
The API equivalents are `POST /api/prospects/{id}/website-candidates`,
|
||
`GET /api/prospects/{id}/website-candidates`, and
|
||
`PATCH /api/prospects/{id}/website-candidates/{candidate_id}` with
|
||
`{"status":"accepted"}` or `{"status":"rejected"}`.
|
||
|
||
### Runtime settings
|
||
|
||
Open **Settings → Browser and discovery timing** to change Browser Use's
|
||
enabled state, total investigation timeout, maximum agent steps, and per-call
|
||
AI timeout. The same panel controls the Maps worker wait timeout/poll interval,
|
||
search depth, and website health/email scan timeout. Values are stored in the
|
||
local SQLite database, survive container restarts, and override `.env` defaults
|
||
for new runs. The safe UI ranges prevent a malformed value from creating an
|
||
unbounded worker.
|
||
|
||
The app stores its catalogue in the `prospect_data` Docker volume. The worker
|
||
stores its own job data in `gmaps_data`; keep both volumes if you want to retain
|
||
history. To inspect status:
|
||
|
||
```bash
|
||
docker compose ps
|
||
docker compose logs --tail=100 app
|
||
docker compose logs --tail=100 gmaps
|
||
```
|
||
|
||
## Local fallback mode
|
||
|
||
For a quick local smoke test without the Maps worker:
|
||
|
||
```bash
|
||
DISCOVERY_BACKEND=browser GMAPS_URL= docker compose up -d --build
|
||
```
|
||
|
||
This uses Playwright against public search engines, keeps the same catalogue
|
||
and review UI, and does not bypass CAPTCHAs or other access controls.
|
||
|
||
## Data and review behavior
|
||
|
||
Each saved place includes its name, Maps link, website (when present), root
|
||
domain, address, phone, category, rating, review count, public emails returned
|
||
by the worker when available, source ID, and raw result JSON. A place
|
||
with no website is still retained using a stable `maps:<place_id>` catalogue
|
||
key, so it can be reviewed as a potential website opportunity. Existing
|
||
websites are deduplicated by root domain.
|
||
|
||
Email extraction is always enabled. The worker may visit each public business
|
||
website found by Maps; expect longer runs and review the resulting provenance
|
||
before using any contact information. It does not invent email addresses when
|
||
none are published.
|
||
|
||
Website scans are bounded and SSRF-safe. A scan records HTTP status, final URL,
|
||
latency, HTTPS/TLS, title, technology hints, contact-page emails, and errors.
|
||
Use `/api/prospects/{id}/scan` for a manual rescan. The explorer's “Draft AI
|
||
email” action calls `/api/prospects/{id}/message`, and **Assess with AI** calls
|
||
`/api/prospects/{id}/assess`; both require the optional AI
|
||
variables above and accepts your exact offer/instructions (for example, the
|
||
Gold Plus R119/month plan and a free rebuild). Suppressions are enforced before
|
||
new records are saved and can be managed through `/api/suppressions`.
|
||
|
||
The Teach AI page saves a `company_profile` object through `POST /api/settings`.
|
||
Its **Read website now** action calls `POST /api/company-profile/learn` and
|
||
stores a bounded, reviewable public homepage snapshot. Promotions are managed
|
||
with `GET/POST /api/promotions`, `PATCH /api/promotions/{id}`, and
|
||
`DELETE /api/promotions/{id}`. An active promotion selected in the draft dialog
|
||
is sent as `promotion_id` and retained on that prospect's draft.
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
python3 -m py_compile api/server.py
|
||
node --check web/app.js
|
||
python3 -m unittest discover -s tests -q
|
||
```
|
||
|
||
This project is packaged for personal research. Public availability does not
|
||
override a website's terms, robots controls, or applicable privacy law; keep
|
||
searches small, use the worker only on data you are permitted to collect, and
|
||
do not turn the review screen into unattended outreach.
|
||
|
||
## Workspace accounts and operational controls
|
||
|
||
Authentication is enabled by default. On a fresh database, open the app and
|
||
complete the first-run owner setup. For unattended deployments, set
|
||
`OWNER_EMAIL`, `OWNER_PASSWORD` (8+ characters), `OWNER_NAME`, and
|
||
`WORKSPACE_NAME` in `.env`; the owner is created automatically only when no
|
||
user exists. Sessions are server-side, cookies are HTTP-only, and mutating
|
||
browser requests require a per-session CSRF token. Failed sign-ins are
|
||
temporarily rate-limited after five consecutive failures.
|
||
|
||
Roles are deliberately small:
|
||
|
||
| Role | Capabilities |
|
||
|---|---|
|
||
| Viewer | Read catalogue and runs |
|
||
| Reviewer | Viewer + review status, notes, and follow-up tasks |
|
||
| Researcher | Reviewer + discovery runs, retries, cancellation, and assignments |
|
||
| Admin | Researcher + users, workspace name, backups, and audit trail |
|
||
| Owner | Admin, and cannot be removed as the last active owner |
|
||
|
||
The **Users** page manages workspace members. A prospect drawer contains its
|
||
assignee, notes, tasks, review status, and score/AI assessment history. Every
|
||
workspace-scoped API query checks the signed-in organisation before returning
|
||
data.
|
||
|
||
## Backups and recovery
|
||
|
||
The app keeps the existing SQLite catalogue and performs additive schema
|
||
migrations; it does not delete or rewrite existing prospects. Use **Users →
|
||
Verified backups → Create backup** or run this inside the app container:
|
||
|
||
```bash
|
||
python scripts/backup_sqlite.py
|
||
```
|
||
|
||
Backups use SQLite's online backup API, run `PRAGMA integrity_check`, and keep
|
||
the newest `BACKUP_RETENTION` files in `BACKUP_DIR` (default `/data/backups`).
|
||
The background scheduler creates one automatically every `BACKUP_INTERVAL_HOURS`
|
||
(default 24; set it to `0` to disable automatic creation).
|
||
Include the `prospect_data` volume in server backups. A scheduled or interrupted
|
||
run is marked `recovery_required` after 30 minutes without a heartbeat. The
|
||
Runs page can show its persisted event timeline and retry a failed/recoverable
|
||
run as a new attempt, preserving the original run. Active jobs can be cancelled
|
||
and the worker records the cancellation in the event stream.
|
||
|
||
The audit trail records sign-ins, user and workspace changes, assignments,
|
||
review decisions, notes, tasks, AI assessments, message generation, run
|
||
retries/cancellations, settings, suppressions, and backup actions. Operational
|
||
logs remain available under **Logs**; credentials and authorization values are
|
||
redacted and retention is bounded by `LOG_RETENTION`.
|
||
|
||
This release intentionally keeps SQLite for safety. PostgreSQL migration can
|
||
be added later as a separate, tested data-migration project once the current
|
||
workspace and backup workflow is stable.
|