# Private staff staging

Phase 6 milestone 4: deployment files and local checks are prepared. **No hosted staging server is deployed.** The existing PHP website and preview remain separate. The user confirmed that the current hosting provides only a control panel, with no VPS/SSH access. This Docker package requires a Linux server with Docker Engine, Compose v2.24+ and SSH administration, so it cannot be installed through the confirmed access alone. Do not upload the platform to `public_html`. A managed application hosting alternative would require a separate deployment configuration and verification of Node.js, Python, PostgreSQL, worker and persistent-storage support; a VPS is not the only possible hosting model. The owner selected preparation for a Linux VPS on 2026-10-15; no provider/server has been selected or purchased. A reproducible upload package and read-only host preflight are available in [PILOT-PACKAGE.md](PILOT-PACKAGE.md).

## What is included

- Separate web/API images, PostgreSQL 17, one-off migrations, private attachment and certificate volumes, and a Caddy HTTPS entry point.
- Only ports 80/443 are published. Web, API and PostgreSQL have no direct host ports. API and worker run as UID 10001 with a read-only image; web runs as the Node user.
- Distinct database administrator, migration-owner and runtime passwords. The runtime role can read/write application records, but cannot create/drop tables, truncate records, create roles or change the migration version. Only the migration service receives owner credentials.
- A fresh MFA encryption key, generated per installation and mounted separately from application code. Losing this key prevents existing MFA credentials from being used; replacing it with a new key is not recovery.
- `/health` checks the API process; `/ready` also checks database access and the exact migration head expected by that release. It returns a generic 503 for missing, old, unexpected or unavailable schema state. It does not certify attachment storage, external integrations or full business functionality.
- IP/CIDR access restrictions for both HTTPS hosts, no search indexing and blocked public API documentation. Staff still authenticate inside the application. The proxy uses the actual connecting IP, not untrusted forwarded headers. DNS must point directly at this server; do not put a CDN or another proxy in front without redesigning and testing this restriction.
- A deliberately optional extraction worker (`ai` profile). Initial staging has AI/learning, IMAP, WhatsApp and invoice issuance disabled. Starting the default stack does not launch the disabled worker.

Compose secrets are file mounts, not an encrypted secret vault. Their source files are readable inside only the services receiving them. Source files use mode 0444 inside a host directory with mode 0700 because file-backed Compose secrets retain source ownership/mode. Preserve that directory boundary; never place individual secret files in a shared directory or loosen directory permissions. Docker/server administrators are trusted operators. Never paste resolved Compose configuration into tickets once it contains integration credentials; use `config --quiet` for routine validation.

The Docker context has an allowlist. Local databases, preview credentials, `.env` files, `.local`, virtual environments and build outputs are excluded. Images are built from source plus the existing dependency locks. Base image tags still need recorded digests and a rebuild/security-patch policy before pilot operation.

## Hosting information needed

1. A staging server with SSH administration and permission to run Docker. The hosting control panel alone does not establish this capability.
2. Two unused DNS hostnames, for example `staff-staging.phonebuzz.me` and `api-staging.phonebuzz.me`, plus DNS management access. These names are proposals, not active endpoints.
3. The public office/VPN IP ranges allowed to reach staging and the email for certificate notices. Physical phone testing must use an allowed network/VPN; mobile carrier IPs can change.
4. The responsible server operator and the staff administrator's email/name. The first password is entered privately on the server; no default account is created.

SSH passwords/private keys and account passwords should not be pasted into chat. Installation and DNS changes wait for an identified, authorized server. Use synthetic business data until hosted backups, monitoring and security checks are complete.

## File protection before pilot access

Production file operations now require a configured local scanner. Follow [PILOT-HARDENING.md](PILOT-HARDENING.md), provision the host daemon, and add `-f deploy/staging/compose.scan.yml` to the Compose examples below whenever operating API/worker services. The base stack can start without the scanner, but file uploads/imports/downloads stay blocked; `/ready` does not certify scanning. The guide also explains the new shared sign-in limits and migration.

## First installation on the staging server

Prefer the verified server-source ZIP described in [PILOT-PACKAGE.md](PILOT-PACKAGE.md). Extract into a new private server directory outside all public web roots. If copying source manually, copy only the `platform` source. Exclude `.local`, `.env*`, `*.db*`, `node_modules`, `.venv`, `.next`, test artifacts and existing credentials. Run commands below from that directory. The example domains/email/IP are documentation placeholders: replace them with the approved staging details before use.

```sh
python3 scripts/prepare-staging.py \
  --web-host staff.example.com \
  --api-host api.example.com \
  --tls-email operator@example.com \
  --allow-cidr 203.0.113.7/32 \
  --release staging-001

docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml config --quiet
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml build api web
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml up -d --wait db api web proxy
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml ps -a
```

The generator refuses an existing `.local/staging` directory, preserving passwords and the MFA key. Do not rerun it to update an installation. DNS A/AAAA records must resolve to the correct server and ports 80/443 must be reachable for HTTPS certificate provisioning. Remove stale AAAA records if IPv6 is not configured. Caddy obtains real certificates only when actually started on the configured server. Database initialization runs only for a new database volume; editing password files later does not rotate existing PostgreSQL roles.

The dependency sequence is PostgreSQL → migrations/grants → API readiness → web health → proxy. A failed migration or unhealthy API blocks initial startup of dependent services. Health checks expose unhealthy containers but do not themselves restart or notify an operator; configure external alerting before a pilot.

Create the first administrator explicitly once the services are ready:

```sh
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml run --rm --no-deps api \
  python -m phonebuzz.cli create-admin --email admin@example.com --name "Staging administrator"
```

The CLI prompts for the new password twice. Do not use legacy website, mailbox or preview passwords. Existing live accounts are not imported. Visit the approved HTTPS web hostname from an allowed IP, enroll MFA, and verify sign-in/sign-out on web and native clients. See ACCOUNT-SECURITY.md for operator recovery and key handling. Use a second separately named administrator for approval workflows.

## Optional worker and integrations

`.local/staging/integrations.env` is a private, mode-0600 environment file shared by the API and worker. It starts with `PB_AI_ENABLED=false` and `PB_AI_LEARNING_ENABLED=false`. Later AI activation needs an approved model/provider configuration in that file, followed by recreating the API and explicitly starting the `ai` profile. Do not enable it just to satisfy a health check. An unconfigured worker exits with an explanation and bounded restart attempts.

The optional [private model overlay](SERVER-AI.md) now connects API/worker to a fixed Ollama service on an internal Docker network. It keeps AI disabled until explicitly selected and supplies metadata/synthetic evaluation probes. The base stack and laptop loopback default are unchanged. Linux startup, resource sizing and real staff evaluation remain pending. Do not silently substitute a paid hosted provider. The web API's manual trading and review features work with AI disabled.

IMAP, WhatsApp and invoice issuance are explicitly forced off in this staging recipe. Their activation requires the outstanding receiving/onboarding/tax work, reviewed configuration and dedicated tests. Live email testing remains deferred as requested.

## Backup and restore procedure

Back up the database, private attachments and original MFA key together. Use encrypted off-server storage with operator access control and an agreed retention policy; Docker volumes on the same server are not a disaster-recovery backup. No scheduled/off-site backup or alert service has been configured yet.

For a consistent initial backup, enter a maintenance window and stop all application writers (API/worker and any administrative CLI/migration jobs). From `platform/`, use a new private directory and the following commands. `BACKUP_DIR` must be a new absolute directory on protected storage.

```sh
umask 077
BACKUP_DIR=/private-backups/phonebuzz-staging-001
mkdir -m 700 "$BACKUP_DIR"
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml stop proxy web worker api
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml exec -T db \
  pg_dump -U postgres -d phonebuzz --format=custom --no-owner --no-acl > "$BACKUP_DIR/database.dump"
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml run --rm --no-deps -T --entrypoint tar api \
  -C /data/attachments -cf - . > "$BACKUP_DIR/attachments.tar"
cp .local/staging/secrets/mfa_key "$BACKUP_DIR/mfa_key"
chmod 600 "$BACKUP_DIR/mfa_key"
cp .local/staging/compose.env "$BACKUP_DIR/release.env"
docker compose --env-file .local/staging/compose.env -f deploy/staging/compose.yml up -d --wait api web proxy
```

Check every command's exit status; stop on failures and mark incomplete backups unusable. If the worker was previously enabled, separately restart its approved profile after the API is healthy. Preserve the matching source release and resolved image digests beside each backup. An operator must encrypt/copy the completed backup off-server; do not attach it to a ticket or include it in a source upload.

Test restore into a **fresh, isolated staging installation**, never over the live database:

1. Prepare fresh database credentials using the generator in a separate checkout and use different staging hostnames. Run the drill on a separate server, or pass `-p phonebuzz-restore-check` on **every** restore Compose command so it cannot reuse the original `phonebuzz-staging` volumes. On the same server, do not start the restore proxy while the original owns ports 80/443. Keep the new database passwords; restore the backup's MFA key into `.local/staging/secrets/mfa_key` with mode 0444 before starting the API.
2. Start only `db`. Its initialization creates empty `phonebuzz`, `pb_migrator` and `phonebuzz` runtime role. Restore the archive as `pb_migrator` with `--no-owner --no-acl --exit-on-error`; this ensures restored tables belong to the migration owner, not the PostgreSQL superuser. For example:

   ```sh
   docker compose -p phonebuzz-restore-check --env-file .local/staging/compose.env -f deploy/staging/compose.yml up -d --wait db
   docker compose -p phonebuzz-restore-check --env-file .local/staging/compose.env -f deploy/staging/compose.yml exec -T db \
     sh -ec 'export PGPASSWORD="$(cat /run/secrets/db_migrator_password)"; exec pg_restore -h 127.0.0.1 -U pb_migrator -d phonebuzz --no-owner --no-acl --exit-on-error' < "$BACKUP_DIR/database.dump"
   docker compose -p phonebuzz-restore-check --env-file .local/staging/compose.env -f deploy/staging/compose.yml run --rm --no-deps -T --entrypoint tar api \
     -C /data/attachments -xf - < "$BACKUP_DIR/attachments.tar"
   ```

3. Run migrations/grants using the backed-up release (or an explicitly tested upgrade), then start API/web and, on a separate server, proxy. Keep using `-p phonebuzz-restore-check`. Inspect exit status and readiness before admitting staff. Test a restored MFA sign-in/backup code, sample quotation/order reservations and attachment downloads; compare stored hashes/counts with the backup evidence. Revoke restored sessions before admitting staff if incident recovery requires it.
4. Retain the source backup until the restore drill is signed off. No automated restore against existing volumes is supplied. `docker compose down -v` destroys these volumes and is not an update procedure.

The PostgreSQL logical restore test in POSTGRES-VALIDATION.md has passed locally. A full container-volume/attachment/key restore on the actual host remains an acceptance gate.

## Authenticator key maintenance

Use the offline procedure in [MFA-KEY-ROTATION.md](MFA-KEY-ROTATION.md). The runtime and container entrypoint accept one optional previous-key file and reject invalid/duplicate pairs. The default Compose configuration continues to mount only its original primary key; no preview/staging key is rotated automatically. A real installation must stop all writers, mount both keys for maintenance and update every service to the replacement-only secret after verification. Container mount and hosted restore drills remain pending.

## Updating an existing staging release

Build/tag the new API and web images first with a new `STAGING_RELEASE` in the existing private `compose.env`. Preserve all secrets and volumes. Record previous image digests, take and verify a backup, then stop application writers. Run the migration service explicitly with `run --rm --no-deps migrate` and check successful exit before `up -d --wait api web proxy`. Recreate the approved worker separately if enabled. Do not merely restart an old migration container and assume new migrations ran.

On migration failure, leave application writers stopped for investigation. An older image may be incompatible with an upgraded schema; rollback means a reviewed compatible image/schema plan or restoration of the complete pre-upgrade backup into a separate installation. Do not automatically downgrade or delete data.

## Local validation and remaining gates

```sh
npm run build:web
apps/api/.venv/bin/python scripts/check-staging.py
apps/api/.venv/bin/python scripts/check-postgres.py \
  apps/api/tests/test_readiness.py apps/api/tests/test_staging_roles.py \
  apps/api/tests/test_staging_setup.py apps/api/tests/test_workspace.py
```

`check-staging.py` requires Caddy, Docker Compose (a daemon is not needed) and Node. It uses disposable secrets/configuration and a synthetic loopback API; it verifies Compose/Caddy syntax plus the standalone page, static assets, secure cookie attributes, exact-origin checks and authenticated API proxy. It never contacts the preview or live site and stops its temporary server. The PostgreSQL checks exercise actual migrations, shipped runtime grants, permitted staff login/company writes, denied DDL/truncation/schema-version writes and readiness failures.

Docker Engine is unavailable on this development machine, so the Linux images and full Compose startup have **not** been run here. Initial Linux image builds, container networking/volume permissions, restart behavior, real DNS/TLS, allowed/denied external access, disk monitoring, alerts, off-site backups/restore, representative load and physical iPhone/Android checks remain pending. Passing local checks does not make the application production-ready.

References: [Docker Compose secrets](https://docs.docker.com/reference/compose-file/secrets/), [Compose service configuration](https://docs.docker.com/reference/compose-file/services/), [Caddy IP matchers](https://caddyserver.com/docs/caddyfile/matchers#remote-ip), [Caddy automatic HTTPS](https://caddyserver.com/docs/automatic-https). Next standalone packaging follows the documentation bundled with the installed Next version.

The [offline worker recovery rehearsal](WORKER-RECOVERY-DRILL.md) verifies local observation/review behavior before hosted failure drills. It creates its own disposable SQLite database and is independent of staging credentials; it does not exercise supervisor restart, container outages or alert delivery.

## Offline staff workflow precheck

The server source package includes `scripts/check-pilot.py`, a disposable 15-checkpoint API workflow and coordinated SQLite/files/MFA recovery check. Run it with the API virtualenv from the extracted source directory; no running services or actual server credentials are used. See [PILOT-REHEARSAL.md](PILOT-REHEARSAL.md). This complements the hosted PostgreSQL/files/key backup and restore procedure above; it does not validate the live deployment.