# Private server assistant comparison

Phase 6 milestone 18, 2026-10-15 (Asia/Karachi). `phonebuzz.assistant_benchmark` is included in the API image and supports the same fixed loopback/private Compose model destinations as the application. It replaces the assistant laptop evaluator's indirect dependency on an extraction script with the shared validated model identity/transport. Supplier extraction now has the corresponding shared evaluator in [EXTRACTION-BENCHMARK.md](EXTRACTION-BENCHMARK.md), added in milestone 19.

The operator explicitly submits administrator-reviewed holdout questions and selected approved examples to the configured private model. The command measures routing with and without examples; it never enables teaching, trains weights, promotes a model, executes a proposed lookup or writes business records. It can run while the application's AI/teaching flags remain off. No eligible holdouts means no provider request and no accuracy score.

## On the private VPS

First provision and verify the local model using [SERVER-AI.md](SERVER-AI.md). Run from the verified release with the `pbmodel` Compose helper defined there. The API image must be rebuilt to contain this module. Use its existing environment and database; do not copy private configuration into the source package.

```sh
umask 077
set -C
pbmodel run --rm --no-deps -T api \
  python -m phonebuzz.assistant_benchmark --run --limit 20 \
  > assistant-comparison-001.json
```

Choose a new private output filename for each run. `set -C` prevents overwriting an existing file. Inspect the command exit status and JSON `status`; a failed command can leave an incomplete output file. The example does not start dependent services, initialize a database, add examples or activate AI. It assumes the authorized operator has already started the private database and model.

`--run` is required. `--limit` accepts 1–50 holdouts, with two inference calls per case and additional model/dataset checks. Metadata and eligibility reads add overhead; this is not a load benchmark. No arbitrary model URL is accepted. Provider redirects, cloud-backed models and invalid identities are rejected through the shared transport.

For the existing laptop workflow, `scripts/evaluate-assistant-feedback.py --run --limit 20` now delegates to the same implementation and retains its private mode-0600 report under `.local`. That wrapper intentionally retains its documented private `.env` behavior. The container module constructs configuration from environment values; the packaged image excludes `.env` files. Do not place unrelated `.env` files in an operator working directory where application dependency initialization could read them.

## What the result means

- `completed`: both arms were measured with stable provenance. Exit 0 means measurement completed, not that business accuracy or pilot acceptance passed.
- `completed_with_failures`: at least one inference/validation failed. Failures remain in the denominator, scores are retained and exit status is nonzero.
- `no_holdouts`: an active administrator must reserve separate evaluation questions first. No model call or score is produced; exit status is nonzero.
- `failed`: configuration, database, model identity or evaluation provenance could not be verified. Partial scores are discarded and errors are sanitized; exit status is nonzero.

Reports include model digest, prompt version, dataset fingerprint, case/example IDs and versions, exact/scope/query/filter match counts, failure counts, timings and paired improvements/regressions. They omit questions, answers, source content, explanations, database URLs and raw exception text. Treat IDs/metrics as private operational evidence. Different model/dataset/prompt identities describe different experiments; compare them deliberately.

## Consistency and database safeguards

The selected holdout snapshot and teaching examples are frozen before evaluation. Existing family separation, active-administrator review, duplicate exclusion and example-size limits remain authoritative. Before and after every inference arm, the evaluator rechecks the eligible dataset and model digest. Revocation, changed labels/examples or replaced weights discard the whole comparison and stop subsequent calls. A change during a request cannot recall content already submitted to the private model; no automatic retry occurs.

Baseline/assisted call order alternates across cases to reduce systematic first-call timing bias. Timing covers interpretation, not the surrounding identity/database checks. Cases with no selected teaching examples are counted explicitly, and their lack of improvement must not be presented as learning.

Every database snapshot uses a read-only transaction: `SET TRANSACTION READ ONLY` on PostgreSQL and query-only mode on SQLite. The SQLite CLI connection additionally uses `mode=ro` and requires an existing file, so a missing target is not silently created. Reports are printed or written separately from business storage. No feature flag, credential, review state or model weight is changed.

## Evaluation limits

The existing four synthetic integration questions scored baseline 3/4 and assisted 4/4 in the 2026-10-15 run, with one improved case and no regressions. That small, previously used fixture is not an independent staff acceptance set or a production accuracy claim. The model/dataset identity and private metrics report were retained; the temporary database and model process were removed afterward.

Representative staff questions, abbreviations/Roman Urdu, supplier material and agreed thresholds remain pending. Reserve an untouched final acceptance set; repeated use turns a holdout into a development set. Keep assistant teaching disabled until those requirements and the hosted checks are reviewed. See [ASSISTANT-FEEDBACK.md](ASSISTANT-FEEDBACK.md) and [RELEASE-READINESS.md](RELEASE-READINESS.md).