add resilient multi-engine browser discovery
This commit is contained in:
@@ -1,8 +1,10 @@
|
||||
# Browser-only ProspectOS
|
||||
|
||||
This is the stripped-down replacement for the larger ProspectOS application.
|
||||
It searches public engines (Google first, then Bing and DuckDuckGo if Google
|
||||
challenges the server) through a headless Playwright Chromium browser. Each
|
||||
It searches public engines (Google, Bing, DuckDuckGo, Brave Search, and
|
||||
Startpage) through a headless Playwright Chromium browser, moving to the next
|
||||
engine when a server-side challenge, timeout, or empty/changed result page is
|
||||
encountered. Each
|
||||
discovery run builds a query from category, keywords, and location, filters
|
||||
results already present in the local catalogue by root domain, and stores only
|
||||
new prospects. The explorer supports search and human review states
|
||||
@@ -19,9 +21,19 @@ docker compose up -d --build
|
||||
|
||||
Open `http://SERVER:8000`.
|
||||
|
||||
If Google presents a CAPTCHA or bot challenge, the run fails safely. The app
|
||||
does not bypass challenges or rotate identities. Keep runs small and spaced
|
||||
out. The implementation uses a fresh headless browser context for each run.
|
||||
If another service already owns port 8000, choose the host port without
|
||||
changing the container:
|
||||
|
||||
```bash
|
||||
HOST_PORT=8080 docker compose up -d --build
|
||||
```
|
||||
|
||||
If an engine presents a CAPTCHA or bot challenge, the run moves on to the next
|
||||
engine. The app does not bypass challenges or rotate identities. Keep runs
|
||||
small and spaced out. The implementation uses a fresh headless browser
|
||||
context for each engine and unwraps normal search redirect links before
|
||||
deduplication. If every public engine blocks the server IP, the run reports a
|
||||
clear `search_engines_unavailable` error instead of inventing prospects.
|
||||
|
||||
## Run locally
|
||||
|
||||
|
||||
Reference in New Issue
Block a user