Production and Staging Deploy and Recovery

This runbook is for maintainers bringing a pinned Git tag up on staging or production, and for recovering the stack without wiping data. It does not replace version-to-version upgrades.

Companion guides

• Upgrading InfoConnect — jump from <from-tag> to <to-tag> with backup, .env merge, and rollback.
• Backup, Restore, and Rollback — complete durable-state backup and recovery.
• Local development: testing a release with git pull — follow a release branch on a development machine; this path is not for staging or production.
• Justfile Commands — recipe reference.
• Troubleshooting / FAQ — short answers for the failure modes below.

Production deploys are Docker Compose plus the justfile prod recipes. Pin a release tag. Do not deploy from a moving main.

Examples below use <configured-api-key> as a placeholder. Never paste a live key into docs, tickets, or screenshots.

Stage-dark rollout (#534)

Ship stage-dark alone in a quiesced deployment. Keep ingestion quiesced through replacement of every API replica and all workers together; do not run a mixed old/new publisher fleet. Run the report-only orphan sweep (#523) before and after deployment, and follow its manual repair guidance for existing eligible orphan records. Resume ingestion after the post-deploy sweep and acceptance checks complete. Use the tagged rebuild procedure below.

Tagged release: just prod-rebuild

Run the manual procedure in a persistent operator session. These recipes run attached and do not provide SSH-disconnect recovery. Take the complete backup from the checkout matching the serving release before switching tags, following Backup, Restore, and Rollback.

Review production runtime planning before deployment for the default API memory limit, insufficient-capacity failures, and explicit overrides for small QA hosts. prod-up exports the checkout's Git revision to child commands, including the worker guard and container recreation. It also prepares and probes the trace directory before starting services; see search trace permissions.

just prod-rebuild builds every image (API, both embedding workers, preprocessing, models, Console) before any container is created, stopped, or recreated, recreates the operators Console with a backend key, then runs the image smoke checks. The image build is the slow part and no longer sits inside the outage window: if the build fails, the recipe stops with the running stack untouched.

# 1. Pin the release (detached HEAD is expected)
git fetch --tags
git checkout <release-tag>

# 2. Ensure .env exists and API_KEY_CLIENTS is valid JSON with at least one key
#    Never cp .env.example over a live .env (that wipes keys, CORS, and tuning).
test -f .env || cp .env.example .env

# 3. API + workers + Console key injection + image smoke
just prod-rebuild

just prod-rebuild runs just prod-up rebuild, just prod-ui-up, then just prod-smoke. It runs the license audit, writes .runtime/prod.env, runs just prod-build (all images, first), pre-downloads models, waits for Redis/Qdrant/API, and gates on just prod-verify-version plus Console readiness. The whole-stack compose up -d is called with --force-recreate and no --build, so each container is recreated from the pre-built image rather than restarted or rebuilt in place — that is what makes the outage window only stop-swap-start, and it is also what applies Docker's json-file log-rotation options (see Rotation and retention). If key resolution, the pre-build, or the authenticated Console check fails, the rebuild stops before smoke.

Bare just prod-up reuses worker images

Only the rebuild path runs just prod-build. A bare just prod-up still builds just the API image (the Celery worker guard runs with --build) and recreates the rest from their existing worker-embedding-* / worker-preprocessing-* images. The Celery node names still carry the new PROD_DEPLOYMENT_REVISION label from the checkout, so the revision guard passes — it verifies the label, not the code. Workers then report the new revision while running the old image (observed on the 2026-09-23 0.5.8 redeploy: readiness tasks absent, process_document.py predating the deploy). Always deploy new code with just prod-rebuild.

To verify worker code after a deploy, compare image age and a code marker inside the container, not just the Celery node name:

docker ps --format '{{.Names}} {{.Image}}' | grep worker
docker inspect <worker-container> --format '{{.Image}} {{.Created}}'
docker exec <worker-container> grep -c reconcile_discovery_readiness /app/app/tasks/process_document.py

A worker image Created timestamp older than the deploy, or a missing symbol the release added, means the image is stale — run just prod-rebuild.

just prod-ui-up requires the API container to be running and a configured key in API_KEY_CLIENTS. It resolves the effective key (an explicit INFOCONNECT_API_KEY is honored only when it is configured), recreates console with that value, and then runs the readiness plus authenticated-read gate. The Console is operators-only; treat http://127.0.0.1:3000 as the access URL and keep it behind an external auth boundary.

Console access: SSH tunnel, not the LAN

The prod Console publishes on loopback only: docker-compose.console.yml binds ${CONSOLE_PUBLISH_ADDR:-127.0.0.1}:3000:3000, so port 3000 never reaches the LAN unless an operator explicitly opts in. Reach it from your workstation through an SSH tunnel (substitute the deployment host; nothing below is a real hostname):

ssh -L 3000:127.0.0.1:3000 <prod-host>
# then open http://127.0.0.1:3000 in a local browser

LAN publishing is an explicit per-invocation opt-in, still keyed through the recipe:

CONSOLE_PUBLISH_ADDR=0.0.0.0 just prod-ui-up

The Console service is defined only in docker-compose.console.yml, never in docker-compose.prod.yml, so a plain docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d neither starts nor restarts a Console. The overlay requires INFOCONNECT_API_KEY and aborts with a message naming just prod-ui-up when it is unset or empty; just prod-up and just prod-ui-up include the overlay and supply the resolved key.

Staging isolation

Rehearse on staging with a distinct COMPOSE_PROJECT_NAME and remapped host ports. Do not bind the shared default stack ports while another environment is using them. See Upgrading InfoConnect.

Stop without destroying data:

just prod-ui-down    # Console only
just prod-down       # API, workers, Redis, Qdrant — volumes preserved

Console readiness: /api/ready, not the homepage

Three different things used to be treated as the same success: the HTML shell rendering, the backend being reachable, and the backend accepting the console's server credential. Production once ran healthy with an empty INFOCONNECT_API_KEY because the container healthcheck only GETed /.

The console now exposes GET /api/ready, and the Docker healthcheck uses it. It returns 200 only when all three hold, and 503 with a stable code otherwise:

code	Meaning	Fix
server_key_missing	INFOCONNECT_API_KEY is unset or blank in the console container	just prod-ui-up (resolves a key from API_KEY_CLIENTS) or set a configured key
server_key_rejected	The key is set but the backend refuses it (401/403 on a protected read)	The key must be a member of the running API's API_KEY_CLIENTS; if the backend .env changed, run just prod-up then just prod-ui-up
backend_unreachable	The console cannot reach INFOCONNECT_API_URL	Check the API container is up (just prod-logs api) and the compose network
backend_degraded	/health answered but is not semantically healthy — including a 200 body with "status": "degraded"	Inspect the component list from GET /api/health; a degraded backend is not ready
capabilities_unexpected	The protected credential check answered with an unexpected shape/status	Check API logs; the readiness contract expects GET /web-search/capabilities

/api/health remains the diagnostic proxy to backend /health: it proves a key was injected, not that the key is valid — backend /health is unauthenticated, so a wrong nonempty key still passes it. /api/ready is the readiness signal; / is only the renderable shell.

Every recipe that starts the console now resolves the key first: prod-up, prod-ui-up, and prod-rebuild all call just prod-console-key, which fails before any Compose mutation when the effective key is missing, blank, or not a configured member of API_KEY_CLIENTS, and passes the validated key to the whole-stack up. An explicit INFOCONNECT_API_KEY is honored only when it is a configured member of API_KEY_CLIENTS; unset it to use the first configured key.

.runtime/prod.env is generated resource planning (sizing, memory, worker counts) — it is not a credential source and contains no API keys. Do not copy secrets into it.

Confirm readiness after any console change:

curl -fsS http://127.0.0.1:3000/api/ready        # {"ready":true} or 503 {"code": "..."}

Readiness logs and alerting

The console emits one console_not_ready reason=<code> line per failure transition and one console_ready line on recovery — never env values, headers, bodies, or keys. docker logs <console-container> or your log collector sees exactly one event per flap, not one per healthcheck poll.

The container going unhealthy is a signal, not an alert delivery. To route it: point your deployment's existing monitor (container-health event watcher, uptime probe, or log alert) at console_not_ready/console_ready log events, or at a periodic external GET http://<host>:3000/api/ready check that pages when it stays 503 past its grace window. This repository ships no Prometheus series for console readiness; wire the receiver your deployment already uses.

Before an approved recreation, follow the monitoring runbook for Docker log rotation and disk budgeting. It also owns API-to-worker request lookup, optional off-host shipping and opt-in tracing.

Why docker compose restart does not pick up .env

Compose interpolates the project-root .env into each service's environment: when the container is created. The .env file is not mounted into the API, workers, or Console. docker compose restart (and docker restart) only restarts the existing process; it does not re-read .env or recreate the container.

After you change .env (API keys, WEB_SEARCH_ENABLED, CORS, model settings, and so on):

# Apply API / worker env — recreates containers from the new interpolation
just prod-up

# Apply Console env, resolve the key, and verify readiness
just prod-ui-up

A tagged image or Dockerfile change needs just prod-rebuild, which includes Console key injection. If running the steps separately, use just prod-up rebuild followed by just prod-ui-up and just prod-smoke.

Recreate, not restart, for log rotation

Docker applies json-file driver options when a container is created, not on restart. The prod and Console overlays set max-size=100m / max-file=20 (DOCKER_LOG_MAX_SIZE / DOCKER_LOG_MAX_FILE), so those options only take effect once containers are recreated. just prod-up therefore calls the whole-stack compose up -d with --force-recreate (and just prod-ui-up recreates the Console) instead of restarting existing containers. That includes redis and qdrant: their data lives in the redis_data / qdrant_data named volumes, so a deploy restarts those services for a moment but never deletes their data (just prod-down clean and just wipe are the destructive commands). Verify after a deploy:

docker inspect <api-container> --format '{{json .HostConfig.LogConfig}}'
# {"Type":"json-file","Config":{"max-size":"100m","max-file":"20"}}

A container whose LogConfig.Config is empty predates the rotation policy and needs recreation (just prod-rebuild). See Rotation and retention.

caution

just stop / just dev is the local hot-reload loop. It is not how production Compose picks up .env. See Configuration.

Celery worker revision conflict — recover with just prod-down

Before starting API and workers, just prod-up pings the shared Celery broker (scripts/check_celery_workers.py) in a one-shot API container on the Compose network. It builds that image first so the checker matches the checkout, publishes no service ports, and starts no dependencies. It uses the same authenticated broker configuration as the workers, so the host need not resolve redis. Any worker from another Git revision or without a revision-bearing node name (preprocessing-<revision>@<host> / embedding-<n>-<revision>@<host>) blocks startup.

The checker prefixes the exception text with ERROR: and prints no traceback and no exception class name. Match these strings from format_revision_conflict — they are the diagnostic:

ERROR: Celery worker revision conflict detected before startup.
Expected deployment revision: <revision>
Offending worker node(s): <worker-node>, ...

Look for Celery worker revision conflict detected before startup., Expected deployment revision:, and Offending worker node(s):. A Python class name does not appear in this output.

Broker inspection failures also block startup, but are not revision conflicts:

Diagnostic	Action
broker DNS resolution failed	Check the execution network. Compose names such as redis resolve inside Compose, not on the host. For a manual host check, set CELERY_BROKER_URL to a reachable endpoint for the same broker.
broker is unreachable	Check address, port, network, and TLS settings.
broker authentication failed	Check the Celery broker username and password.
broker authorization failed	Check the Celery user's Redis ACL permissions.
worker inspection failed	Check transport configuration and broker logs; the cause was not classified.

The checker does not print broker credentials. Do not tear down workers to fix a DNS or authentication error.

Typical causes:

• leftover Compose workers from a previous tag or commit
• a local just dev worker still connected to the same Redis
• another Compose project sharing the broker

Recover without deleting the index:

# 1. Stop Compose workers and orphans. Volumes (qdrant_data, redis_data) stay.
just prod-down

# 2. If a local/dev worker is still on the broker (the error also suggests this):
just stop

# 3. Bring the intended tag back up
just prod-rebuild

Confirm just prod-down printed ✅ Production stopped (data preserved).

Do not use these for revision-conflict recovery

• just prod-down clean — deletes volumes, images, and build cache
• just wipe / just wipe -y — destroys Qdrant and Redis data

Those commands are for decommissioning or a deliberate empty-corpus reset, not for unblocking startup. See Justfile Commands.

Scoped API recreate must carry PROD_DEPLOYMENT_REVISION

docker compose ... up -d --no-deps api recreates only the API, but the container's CELERY_WORKER_REVISION interpolates ${PROD_DEPLOYMENT_REVISION:-unknown}. Without it the API starts as revision unknown and assert_no_conflicting_workers crash-loops it against the running workers. Source .runtime/prod.env and export PROD_DEPLOYMENT_REVISION=<git rev-parse --short=12 HEAD> (or the workers' existing revision) for any scoped recreate.

The same recreate must not let a stale .runtime/prod.env override the live Redis budget: the publication startup assert compares PUBLICATION_REDIS_MEMORY_MB (from PROD_REDIS_MEMORY_MB) against the running container's maxmemory. just prod-runtime-plan pins the emitted Redis budget to the live redis container's cgroup memory limit (docker inspect <redis-container> --format '{{.HostConfig.Memory}}'), so a Redis container created by a one-off compose override keeps its real budget in .runtime/prod.env instead of the host-derived plan. If the file is already stale, set PROD_REDIS_MEMORY, PROD_REDIS_MEMORY_MB, and REDIS_MAXMEMORY to match the live container before recreating.

Web search enablement vs Chat Answer Mode

These are two different controls. Mixing them up looks like a deploy bug.

Control	Where it lives	What it does
WEB_SEARCH_ENABLED	Server .env (Compose-injected)	Deployment capability gate. Default false. Changing it requires recreating API containers (just prod-up), not docker compose restart.
Chat Answer Mode	Console UI, per turn (tab preference)	Auto / Docs-only / Web for the next Chat turn. Cannot override the gate.

When the gate is off (WEB_SEARCH_ENABLED=false):

• Auto runs as Docs-only
• an explicit Web turn is rejected
• the Console mode control collapses to Docs-only (the stored preference is not overwritten)

When the gate is on:

• Auto may use web search only for live, time-varying questions (a corpus gap is not enough)
• Docs-only never offers the web tool
• Web requires a public web-search pass before final synthesis

Enabling WEB_SEARCH_ENABLED=true does not put Chat into Web mode. Operators still choose Answer Mode per turn. Search has no web-search control.

Read the gate without running a search. Run the curl on the deployment host or through the approved SSH tunnel (see Post-deploy acceptance checks):

curl -fsS http://localhost:8000/web-search/capabilities \
  -H "X-API-Key: <configured-api-key>"

What you see	Meaning	What to do
{"enabled": true} or {"enabled": false} matching .env	The API gate is live as configured	Continue to Chat Answer Mode
JSON enabled does not match .env	The API container still has the old interpolation	Recreate with just prod-up, not docker compose restart
HTTP 401	<configured-api-key> is missing from API_KEY_CLIENTS	Fix the mapping in .env and recreate (just prod-up)
Unreachable (connection refused, timeout, empty reply)	The API is down, still starting, or you are not on the deployment host	On that host (or the approved tunnel): just status and just prod-logs api
"enabled": true but the Console has no Web Answer Mode	Backend gate is on; the Console UI gate is stale or unconfigured	just prod-ui-up, hard-refresh the browser, then curl -fsS http://127.0.0.1:3000/api/ready. This is the separate Console UI gate — the same missing/rejected-key path as failure mode 4 — not an .env mismatch

See Chat and Web Search API.

Post-deploy acceptance checks

Run every check on the deployment host (the private-LAN staging or production machine running Compose) or through the approved SSH tunnel to that host. localhost and 127.0.0.1 below mean that host, not your laptop.

Redis (6379) and Qdrant (6333) are internal Compose services. They need not be publicly exposed. A laptop-side curl to localhost:8000 that fails is not a failed deploy — you are curling the wrong machine.

Do not declare the tagged deploy successful until all of these pass. Use <configured-api-key> from API_KEY_CLIENTS, not an example demo key.

1. Product Version and image smoke

just version                 # Product Version of the checked-out tag (may differ from the tag string)
just prod-verify-version     # running GET /health version matches pyproject.toml
just prod-smoke              # already part of just prod-rebuild

Compare /health to just version; both must report the same Product Version for the checked-out tag. See Health API.

2. Container and component health

just status docker --iterations 1
curl -fsS http://localhost:8000/health

GET /health must report "status": "healthy" with api, redis, qdrant, and celery ok. A degraded payload is not a pass.

3. Auth fail-closed and a real search

# Missing key is 401
curl -sS -o /dev/null -w "%{http_code}\n" \
  -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query":"machine learning","scope":"Content","limit":5}'
# expect 401

# Configured key reaches the index (empty results are ok; 401 is not)
curl -fsS -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <configured-api-key>" \
  -d '{"query":"machine learning","scope":"Content","limit":5}'

The successful response is an Evidence Result page: rows are Document Versions, and pagination counts Document Versions rather than chunks. A 409 METADATA_DISCOVERY_NOT_READY is a client readiness conflict for Metadata or Both; a 503 indicates that active publication authority or the registry could not be verified and is retryable. In just prod-logs api, the startup banner No API keys are configured — EVERY request will be rejected with 401 Unauthorized must be absent.

just search-smoke default key

just search-smoke sends X-API-Key: ${INFOCONNECT_API_KEY:-test-key-123}. Production API_KEY_CLIENTS often does not include that demo key. Export INFOCONNECT_API_KEY=<configured-api-key> first, or use the curl above.

4. Console readiness (not just the homepage)

curl -fsS -o /dev/null http://127.0.0.1:3000          # HTML shell — necessary but not sufficient
curl -fsS http://127.0.0.1:3000/api/ready            # key injected, valid, backend healthy
curl -fsS http://127.0.0.1:3000/api/documents?limit=1\&offset=0   # authenticated read; empty library ok

just prod-ui-check runs exactly this against the deployment container — the readiness wait plus one read-only authenticated request — and already gates prod-up / prod-ui-up / prod-smoke. A 503 from /api/ready carries a code; use the table in Console readiness. /api/health alone does not validate credentials. Open Search or Status in the Console and confirm a backend-backed view, not the shell's service notice.

5. Web-search gate vs Answer Mode

curl -fsS http://localhost:8000/web-search/capabilities \
  -H "X-API-Key: <configured-api-key>"

Interpret 401, unreachable, .env mismatch, and "enabled": true with no Console Web Answer Mode using the table in Web search enablement vs Chat Answer Mode. A matching JSON payload is not enough if the Console UI still lacks Web mode — that is failure mode 4 (just prod-ui-up, hard-refresh, /api/ready).

Expected .env	Capabilities	Chat check
WEB_SEARCH_ENABLED=false	"enabled": false	Auto behaves as Docs-only; Web is rejected
WEB_SEARCH_ENABLED=true	"enabled": true	Auto / Docs-only / Web match Chat; Search still has no web control

6. Worker revision

git rev-parse --short=12 HEAD
just prod-logs worker-embedding-1

Worker hostnames must include that 12-character revision (embedding-1-<revision>@...). If just prod-up prints Celery worker revision conflict detected before startup. with Expected deployment revision: and Offending worker node(s):, that is a failed deploy — recover with just prod-down, not clean.

Summary checklist

• Checked out a release tag (not main)
• just prod-rebuild completed, including its prod-ui-up step
• /health healthy; just prod-verify-version matches just version
• Authenticated search with <configured-api-key> is not 401; fail-closed banner absent
• http://127.0.0.1:3000/api/ready returns {"ready":true} (key injected and accepted)
• /web-search/capabilities matches WEB_SEARCH_ENABLED; Answer Mode is not mistaken for the gate
• Workers registered on the current Git revision
• .env changes applied by recreating containers, not docker compose restart
• docker inspect <container> --format '{{json .HostConfig.LogConfig}}' shows max-size=100m / max-file=20 (rotation applied by recreation, not restart)
• Recovery used just prod-down (data preserved), never clean / wipe, unless you intend to destroy the index