# Private server supplier extraction comparison

Phase 6 milestone 19, 2026-10-15 (Asia/Karachi). The API image now includes `phonebuzz.extraction_benchmark`, sharing the same evaluator with the existing laptop script. It compares extraction from reviewed one-item supplier excerpts with and without approved examples. This is measurement of prompt memory, not weight training or automatic model promotion.

## Run explicitly

Prepare separate administrator-reviewed teaching examples and holdouts as described in [LEARNING-EVALUATION.md](LEARNING-EVALUATION.md). Provision the private model using [SERVER-AI.md](SERVER-AI.md). From the verified release, use its `pbmodel` helper and rebuilt API image:

```sh
umask 077
set -C
pbmodel run --rm --no-deps -T api \
  python -m phonebuzz.extraction_benchmark --run --limit 5 \
  > extraction-comparison-001.json
```

Choose a new private output filename each time. Inspect the command exit status and JSON status; an unsuccessful command can leave incomplete output. This does not start dependencies, add examples, initialize a database or enable AI. The operator must already have started the database and private model. `--run` is mandatory for the server module; limit is 1–20, default five. Each case makes two extraction requests plus provenance checks. AI/teaching flags may remain off; explicit opt-in authorizes only this evaluation's requests.

Both identity checks and extraction use the configured fixed loopback or private Compose destination. The shared transport rejects arbitrary destinations, redirects and cloud-backed models. No cloud fallback is used. The server module reads environment configuration; packaged images exclude `.env` files. Keep unrelated `.env` files out of the operator working directory because application dependency initialization may read them.

The laptop entry point remains `scripts/evaluate-learning.py --run --limit 5`, run from `apps/api` with its intended private `.env`. It delegates to this implementation and saves a new mode-0600 report under `.local`; empty holdouts/examples print a status without creating a report. Without `--run`, that legacy wrapper remains a no-op. It now returns nonzero for empty datasets or inference failures.

## Read the result

| Status | Meaning |
| --- | --- |
| `completed` | Both arms were measured with stable provenance; exit 0. This is not quality approval, and inaccurate valid outputs can still complete. |
| `completed_with_failures` | Inference or validation failed in at least one arm. Failed outputs remain in missing-value/item-count metrics and denominators; nonzero exit. |
| `no_holdouts` | No eligible active-administrator-reviewed holdouts; no model request, metrics or success claim; nonzero exit. |
| `no_examples` | None of the selected holdouts has an eligible independent teaching example; no model request or metrics; nonzero exit. |
| `failed` | Configuration, database, model identity or provenance could not be verified. All partial scores are discarded; nonzero exit. |

Reports retain model/prompt identity, dataset digests, case/example IDs and versions, field/item-count metrics, failures, timings and number of cases with examples. They omit supplier excerpts, expected values, raw model output, credentials and raw exception details. IDs and digests remain private operational metadata. See the metric definitions and evaluation limits in [LEARNING-EVALUATION.md](LEARNING-EVALUATION.md).

## Safeguards and limits

Each database snapshot is enforced read-only using PostgreSQL read-only transactions or SQLite query-only mode. The CLI additionally opens SQLite with `mode=ro`, requiring an existing file and preventing accidental creation. No application record, approval, account, credential or feature flag is changed.

The entire holdout partition is excluded from teaching: all holdout conversations and normalized exact duplicate excerpts, including cases outside the current limit. Existing active-administrator review and example-size limits remain in force. Dataset/example fingerprints and model digests are rechecked before and after every inference arm and once before reporting. Changed or unavailable provenance discards the comparison. A later revocation cannot recall text already submitted during an in-flight request; there is no automatic retry. These are boundary checks, not locks preventing edits during a request.

Baseline/assisted order alternates per case. Timing covers extraction/scoring, excluding surrounding identity/database checks; this is not a load test. Cases with no selected examples are counted explicitly when other cases have examples. Missing and wrong fields, extra items and unsupported values are measured separately; null-to-null matches never inflate accuracy.

Local automated checks use synthetic fixtures and mocked transport. No new real-supplier accuracy claim or live provider activation follows from this milestone. Representative supplier samples, independent final acceptance data, agreed thresholds, actual Linux image execution and hosted performance remain pending. Keep the final acceptance set outside repeated prompt tuning.