# Worker activity and incidents — Phase 6 milestone 11

System health now shows durable activity for mailbox polling, AI extraction and explicit push passes on web and native mobile. Every pass started through the shipped worker commands records its start and completion, including idle passes with empty queues. Handled error outcomes and unexpected exceptions record a generic error phase. Exception text, tokens, hostnames, user identifiers and message content are not copied into monitoring records.

These are coarse observations of each logical worker kind, not per-process inventory. Multiple replicas share one stream. The newest started pass owns a random completion token; older or duplicate completions cannot overwrite its result. A recent pass in one replica cannot certify that every replica is running. Long passes are only observed at their boundaries and can become overdue after 15 minutes. No provider request is made by reading the screen or running the incident monitor.

## Expectations and freshness

- AI extraction is expected when its provider/model configuration is enabled and complete.
- Mailbox polling is expected when it is enabled, configured, initialized to the current identity, unpaused and owned by an active administrator.
- Push runs are manual by default. `PB_PUSH_MONITOR_EXPECTED=false` avoids reporting an intentionally unscheduled push pass as missing. Set it true only if an approved regular push schedule is actually installed and enabled. This flag does not enable sending, create a schedule or bypass `run-push.py --send`.
- WhatsApp is webhook-driven and has no polling-worker heartbeat.

An expected worker with no recorded activity is **No activity recorded**. Activity at least 900 seconds old is **Activity overdue**, including the exact boundary. Future timestamps produce **Clock needs review**. A started pass is **completion pending**, not recovered. A finished pass means the wrapper returned, not that every job or remote delivery succeeded. AI/provider failures continue to appear in the integration counters from milestone 10.

Disabled, paused, uninitialized or incomplete connectors do not require continuous worker activity. Their existing records remain visible. The screen separately shows the latest monitor observation and warns when none is recent; refreshing a page never records or resolves an incident.

## Incident recorder

From `platform`, with the intended private server environment and explicit `PB_DATABASE_URL` already loaded:

```sh
apps/api/.venv/bin/python scripts/monitor-workers.py run-once
```

For a deliberately supervised service, the API image includes the equivalent module:

```sh
python -m phonebuzz.worker_monitor watch
```

Watch mode observes every 30 seconds. It is not installed or scheduled by this feature. One local preview observation is recorded during validation; continuous monitoring and external alert delivery remain inactive. Use the same configuration as the API and workers, or expected-worker decisions may differ. Do not enable an integration just to satisfy monitoring.

The monitor serializes concurrent passes in the database and records only state transitions: incident opened, condition changed, recovery observed, or monitoring requirement removed. Repeated identical observations add no rows. Changing a requirement to disabled/paused is recorded as suspension, never successful recovery. A fresh start alone cannot resolve an earlier incident; a completed pass must be observed. Out-of-order monitor timestamps are rejected. Short failures between monitor passes may not produce incidents; this is observation history, not a complete execution log.

The newest 100 transitions per worker are retained, with monotonic per-worker sequences to preserve ordering even within the same second. Older monitoring transitions are pruned only by the monitor. This retention does not affect business audit events or messages. Administrators page through history 20 items at a time using `GET /v1/system-health?incident_offset=0`; the offset is bounded to 0–300. History and current activity may differ until the next monitor observation.

No automatic retry, reset, recovery, message delivery or notification is performed. Unknown push delivery outcomes retain their existing no-blind-resend behavior. Database monitoring writes fail the worker pass before business work starts if the start cannot be recorded. If a completion write fails after work, existing job idempotency/leases still apply; an operator must review the failure rather than blindly repeat a commercial send.

## Migration and validation

Migration `b92eb06cf643` follows `a81da95be532`, adding `worker_heartbeats` and `worker_incidents`. There are no fabricated heartbeats or migration-seeded healthy states. Downgrade refuses to delete populated worker evidence. Back up before migration. Restart existing workers after upgrading so they use the wrappers; direct calls to the underlying job helpers do not report pass evidence.

Local validation on 2026-10-12: 67 distinct backend tests passed across the regression and final command checks, with one PostgreSQL-only skip in the regression run. PostgreSQL validation passed 25 tests, including 20 PostgreSQL-backed cases plus isolated migration/command checks; covered concurrent monitors, runtime grants, migration/schema matching and restore. Eight relevant browser tests, 21 contract tests, both TypeScript checks, production web build and iOS/Android exports passed. Physical-device and hosted failure drills remain pending.

Both local preview/development databases were backed up before migration; all 49 existing table digests remained unchanged and the two new tables started empty. Existing passwords, sessions, catalogue, stock and sales records were preserved. Live email, WhatsApp, push and formal invoice activation remain deferred.

## Administrator event acknowledgement (Phase 6 milestone 12)

Administrators can review each retained transition on web and native clients. `POST /v1/system-health/incidents/{id}/acknowledge` accepts only a trimmed 10–1,000-character `note` and strict boolean `reviewed: true`. The server records the administrator ID/name and its timestamp; notes are returned only through the administrator health API. Do not enter credentials or customer data. Reviews are one-time and cannot be overwritten or edited.

Acknowledgement refers to that immutable historical event ID, including events followed by recovery or another condition. It does not acknowledge later events, mark a worker healthy, clear a condition, change monitor timing, retry jobs, restart workers or send messages. Recovery still requires worker completion and a monitor observation. Review of a recovered/suspended transition records that the transition was seen; it makes no service guarantee.

Authority and session validity are checked again after acquiring the same account locks used by access changes. An atomic conditional update admits one review; competing or repeated requests return 409 without overwriting the saved note. A pruned/missing event returns 404. Web/native keep draft text on failed saves; 403/404/409 require refresh before another attempt. Refresh/navigation discards the local draft. Session expiry signs the user out. Following an uncertain network result, refresh to check whether the review saved before retrying.

Notes and reviewer snapshots share the existing retention limit of 100 transitions per worker, pruned by the monitor. They are not a permanent incident case log. A generic attributed business audit entry remains after pruning, without note contents or condition details. No alert is suppressed by acknowledgement. This milestone does not add ongoing comments, assignments, notification delivery or automatic recovery.

Migration `ca3fc17d0754` follows `b92eb06cf643`, adding four nullable review columns without inventing historical reviews. Downgrade refuses to remove acknowledged events. Local preview/development backups are under `.local/backups/before-incident-review-20261012T171013Z/`; all 51 existing table digests (original columns) remained unchanged after migration. No real preview incident was acknowledged during testing.

Validation on 2026-10-12: 36 focused SQLite/backend cases; 37 PostgreSQL runner cases (32 used disposable PostgreSQL schemas), including concurrent review, restricted runtime permissions and schema checks; 10 distinct browser scenarios across focused runs; 21 contract cases; web/mobile TypeScript; production web build; and iOS/Android bundle exports. Browser checks include actual API writes through the origin-protected proxy, mobile-width layout, persisted notes, conflict drafts and expiry. Native device interaction, hosted failure drills and live alert delivery remain unverified.

Phase 6 milestone 13 now provides a [repeatable offline failure-and-recovery rehearsal](WORKER-RECOVERY-DRILL.md), with 56 checkpoints across AI, mailbox and push evidence, acknowledgement and database-write failures. Hosted alerting, supervisor restart policy and escalation recipients still require operational decisions and deployment.