add resilient multi-engine browser discovery

This commit is contained in:
Marco0300
2026-09-05 14:59:28 +02:00
parent 6debef6015
commit 521d666ecf
4 changed files with 113 additions and 30 deletions
+17 -5
View File
@@ -1,8 +1,10 @@
# Browser-only ProspectOS
This is the stripped-down replacement for the larger ProspectOS application.
It searches public engines (Google first, then Bing and DuckDuckGo if Google
challenges the server) through a headless Playwright Chromium browser. Each
It searches public engines (Google, Bing, DuckDuckGo, Brave Search, and
Startpage) through a headless Playwright Chromium browser, moving to the next
engine when a server-side challenge, timeout, or empty/changed result page is
encountered. Each
discovery run builds a query from category, keywords, and location, filters
results already present in the local catalogue by root domain, and stores only
new prospects. The explorer supports search and human review states
@@ -19,9 +21,19 @@ docker compose up -d --build
Open `http://SERVER:8000`.
If Google presents a CAPTCHA or bot challenge, the run fails safely. The app
does not bypass challenges or rotate identities. Keep runs small and spaced
out. The implementation uses a fresh headless browser context for each run.
If another service already owns port 8000, choose the host port without
changing the container:
```bash
HOST_PORT=8080 docker compose up -d --build
```
If an engine presents a CAPTCHA or bot challenge, the run moves on to the next
engine. The app does not bypass challenges or rotate identities. Keep runs
small and spaced out. The implementation uses a fresh headless browser
context for each engine and unwraps normal search redirect links before
deduplication. If every public engine blocks the server IP, the run reports a
clear `search_engines_unavailable` error instead of inventing prospects.
## Run locally