add durable jobs and live job monitor

apps/api/README.md

@@ -1,6 +1,6 @@
-# Prospect Platform API — Phase 3
+# Prospect Platform API — Phase 4 boundary
 
-Dependency-light JSON API for tenant-scoped, **manual** prospect workflows. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context. It never performs automated discovery, DNS/website scanning, or outreach.
+Dependency-light JSON API for tenant-scoped, **manual** prospect workflows. Core domain rules use Python's standard library and persistence is SQLite. The API stores businesses plus child intelligence/evidence records, pipeline state, notes, and audit context, with the Phase 4 job/live-log contract described below. It never performs automated discovery, DNS/website scanning, or outreach.
 
 ## Run
 
@@ -18,6 +18,22 @@ Set `PROSPECT_API_PORT` or pass `--port`; set `PROSPECT_API_DB` or pass `--db` t
 
 All protected endpoints require the server-side session cookie. Every query is constrained by the authenticated user's `organization_id`; IDs from another tenant behave as not found and must not disclose whether a record exists.
 
+## Phase 4 jobs and live logging (target contract)
+
+The intended job resource has a stable `job_id`, tenant/creator metadata, operation/payload fingerprint, `status`, `attempt`, timestamps, cancellation state, and terminal error/result metadata. Its lifecycle is `queued` → `running` → exactly one terminal state: `succeeded`, `failed`, or `cancelled`. State transitions and worker messages must be persisted transactionally with tenant and job identifiers; terminal jobs are immutable except for controlled retention/redaction.
+
+### Idempotency and events
+
+Mutating job creation should require an idempotency key (for example, an `Idempotency-Key` header). The key must be scoped to the authenticated tenant and operation, stored with a request fingerprint, and return the original job/result for an exact replay. Reuse with a different payload must be rejected rather than creating a second job. Idempotency must cover side effects, not merely the HTTP response.
+
+Live events should be append-only and ordered per job with a durable integer `sequence`/cursor. A consumer can resume from the last acknowledged sequence, tolerate duplicate delivery, and detect gaps. Events must contain only safe operational detail; do not persist passwords, session cookies, API keys, or unnecessary prospect/contact data.
+
+### Control and delivery semantics
+
+The MVP job routes are `POST /api/v1/jobs`, `GET /api/v1/jobs`, `GET /api/v1/jobs/{id}`, `GET /api/v1/jobs/{id}/events`, `POST /api/v1/jobs/{id}/cancel`, and `POST /api/v1/jobs/{id}/retry`. They are tenant-scoped and backed by SQLite persistence. Cancellation is best effort and must be race-safe with workers; retry creates a new attempt/lineage and must not repeat completed side effects. Polling supports a bounded cursor and backoff in the web monitor. An SSE endpoint may stream the same persisted sequence events with `Last-Event-ID`, heartbeats, disconnect/reconnect support, and polling fallback, but SSE is not implemented in this MVP.
+
+The current runtime has no durable job queue and no Redis/Celery integration. Its in-process/SQLite worker is single-instance, non-durable across process loss, and unsuitable for horizontal scaling or guaranteed execution; it is pilot-only.
+
 ### Health and workspace
 
 - `GET /api/v1/health/live` — unauthenticated liveness check.
@@ -58,6 +74,6 @@ All SQL uses parameters and all responses are JSON. Scores include `score_versio
 
 List and child-record endpoints are deliberately bounded. For business lists, use `page` (starting at 1) and `page_size` within the server-enforced maximum; invalid values are rejected rather than allowing an unbounded query. Supported filters are applied inside the tenant-scoped query before pagination: `q`, `score_min`, `score_max`, `website_class`, and `pipeline_stage`. The UI's page and filter controls are convenience clients, not authorization controls. A filtered page is not a count of the entire unfiltered tenant unless the response explicitly says so.
 
-## Remaining limitations
+## Remaining limitations and production migration work
 
-SQLite is a pilot store with no production migration runner, queue, scheduler, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope.
+SQLite is a pilot store with an MVP job/event schema but no production migration runner, durable queue, scheduler, worker lease/recovery, durable backup command, or tested restore workflow. Authentication currently uses a development password fallback and does not by itself provide production Argon2id, MFA, CSRF protection, rate limiting, or a complete retention-grade audit system. Automated discovery, DNS/HTTP scanning, and outreach remain explicitly out of scope. Production migration work must add schema/index hardening and retention policy, tenant-scoped authorization tests, transactional event sequencing, cancellation/retry semantics, observability, and a reviewed Postgres plus durable queue/worker design. Redis/Celery remain future options, not implemented dependencies.