Production and Staging Deploy and Recovery

This runbook is for maintainers bringing a pinned Git tag up on staging or production, and for recovering the stack without wiping data. It does not replace version-to-version upgrades.

Companion guides

Upgrading InfoConnect — jump from <from-tag> to <to-tag> with backup, .env merge, and rollback.
Backup, Restore, and Rollback — Qdrant snapshots and Redis AOF.
Testing New Releases — local git pull on a development machine.
Justfile Commands — recipe reference.
Troubleshooting / FAQ — short answers for the failure modes below.

Production deploys are Docker Compose plus the justfile prod recipes. Pin a release tag. Do not deploy from a moving main.

Examples below use <configured-api-key> as a placeholder. Never paste a live key into docs, tickets, or screenshots.

Tagged release: `just prod-rebuild` then `just prod-ui-up`

A tagged deploy has two layers. just prod-rebuild rebuilds and smokes the API and workers. just prod-ui-up recreates the operators Console with a real backend key.

# 1. Pin the release (detached HEAD is expected)
git fetch --tags
git checkout <release-tag>

# 2. Ensure .env exists and API_KEY_CLIENTS is valid JSON with at least one key
#    Never cp .env.example over a live .env (that wipes keys, CORS, and tuning).
test -f .env || cp .env.example .env

# 3. API + workers + image smoke (shorthand for: just prod-up rebuild && just prod-smoke)
just prod-rebuild

# 4. Console with an injected API key (required even if the console container is already up)
just prod-ui-up

just prod-rebuild is just prod-up rebuild followed by just prod-smoke. It runs the license audit, writes .runtime/prod.env, pre-downloads models, rebuilds images, waits for Redis/Qdrant/API, then gates on just prod-verify-version.

just prod-ui-up requires the API container to be running and at least one key in API_KEY_CLIENTS. It takes the first configured key and recreates console with INFOCONNECT_API_KEY set. The Console is operators-only; treat http://127.0.0.1:3000 as the access URL and keep it behind an external auth boundary.

Staging isolation

Rehearse on staging with a distinct COMPOSE_PROJECT_NAME and remapped host ports. Do not bind the shared default stack ports while another environment is using them. See Upgrading InfoConnect.

Stop without destroying data:

just prod-ui-down    # Console only
just prod-down       # API, workers, Redis, Qdrant — volumes preserved

Why the Console can start with an empty `INFOCONNECT_API_KEY`

just prod-up / just prod-rebuild starts every service in docker-compose.yml + docker-compose.prod.yml, including console. The Console service interpolates:

INFOCONNECT_API_KEY=${INFOCONNECT_API_KEY:-}

The project-root .env.example does not define INFOCONNECT_API_KEY (that variable belongs to the Console, not the Python API). The default is therefore an empty string.

The Console container can still report healthy: its healthcheck only GETs http://127.0.0.1:3000 (the HTML shell). Same-origin /api/* proxy routes read INFOCONNECT_API_KEY at request time and fail closed when it is empty (500 — The console is not configured. Set INFOCONNECT_API_KEY on the server.). The homepage loading is not proof that Search, Library, Status, or Chat can reach the API.

just prod-ui-up is the supported fix: it recreates console with a key taken from API_KEY_CLIENTS. Do not put a live key in the client bundle, and never prefix it with NEXT_PUBLIC_.

Confirm the proxy is configured:

curl -fsS http://127.0.0.1:3000/api/health

A JSON health payload means the server key was injected. A 500 about INFOCONNECT_API_KEY means run just prod-ui-up (and confirm API_KEY_CLIENTS is non-empty).

Why `docker compose restart` does not pick up `.env`

Compose interpolates the project-root .env into each service's environment: when the container is created. The .env file is not mounted into the API, workers, or Console. docker compose restart (and docker restart) only restarts the existing process; it does not re-read .env or recreate the container.

After you change .env (API keys, WEB_SEARCH_ENABLED, CORS, model settings, and so on):

# Apply API / worker env — recreates containers from the new interpolation
just prod-up

# Apply Console env / inject API_KEY_CLIENTS into INFOCONNECT_API_KEY
just prod-ui-up

A tagged image or Dockerfile change still needs just prod-rebuild (or just prod-up rebuild), then just prod-ui-up.

caution

just stop / just dev is the local hot-reload loop. It is not how production Compose picks up .env. See Configuration.

Celery worker revision conflict — recover with `just prod-down`

Before starting API and workers, just prod-up pings the shared Celery broker (scripts/check_celery_workers.py) and refuses to continue if any registered worker is from another Git revision or lacks a revision-bearing node name (preprocessing-<revision>@<host> / embedding-<n>-<revision>@<host>).

The checker prefixes the exception text with ERROR: and prints no traceback and no exception class name. Match these strings from format_revision_conflict — they are the diagnostic:

ERROR: Celery worker revision conflict detected before startup.
Expected deployment revision: <revision>
Offending worker node(s): <worker-node>, ...

Look for Celery worker revision conflict detected before startup., Expected deployment revision:, and Offending worker node(s):. A Python class name does not appear in this output.

Typical causes:

leftover Compose workers from a previous tag or commit
a local just dev worker still connected to the same Redis
another Compose project sharing the broker

Recover without deleting the index:

# 1. Stop Compose workers and orphans. Volumes (qdrant_data, redis_data) stay.
just prod-down

# 2. If a local/dev worker is still on the broker (the error also suggests this):
just stop

# 3. Bring the intended tag back up
just prod-rebuild
just prod-ui-up

Confirm just prod-down printed ✅ Production stopped (data preserved).

Do not use these for revision-conflict recovery

just prod-down clean — deletes volumes, images, and build cache
just wipe / just wipe -y — destroys Qdrant and Redis data

Those commands are for decommissioning or a deliberate empty-corpus reset, not for unblocking startup. See Justfile Commands.

Web search enablement vs Chat Answer Mode

These are two different controls. Mixing them up looks like a deploy bug.

Control	Where it lives	What it does
`WEB_SEARCH_ENABLED`	Server `.env` (Compose-injected)	Deployment capability gate. Default `false`. Changing it requires recreating API containers (`just prod-up`), not `docker compose restart`.
Chat Answer Mode	Console UI, per turn (tab preference)	Auto / Docs-only / Web for the next Chat turn. Cannot override the gate.

When the gate is off (WEB_SEARCH_ENABLED=false):

Auto runs as Docs-only
an explicit Web turn is rejected
the Console mode control collapses to Docs-only (the stored preference is not overwritten)

When the gate is on:

Auto may use web search only for live, time-varying questions (a corpus gap is not enough)
Docs-only never offers the web tool
Web requires a public web-search pass before final synthesis

Enabling WEB_SEARCH_ENABLED=true does not put Chat into Web mode. Operators still choose Answer Mode per turn. Search has no web-search control.

Read the gate without running a search. Run the curl on the deployment host or through the approved SSH tunnel (see Post-deploy acceptance checks):

curl -fsS http://localhost:8000/web-search/capabilities \
  -H "X-API-Key: <configured-api-key>"

What you see	Meaning	What to do
`{"enabled": true}` or `{"enabled": false}` matching `.env`	The API gate is live as configured	Continue to Chat Answer Mode
JSON `enabled` does not match `.env`	The API container still has the old interpolation	Recreate with `just prod-up`, not `docker compose restart`
HTTP 401	`<configured-api-key>` is missing from `API_KEY_CLIENTS`	Fix the mapping in `.env` and recreate (`just prod-up`)
Unreachable (connection refused, timeout, empty reply)	The API is down, still starting, or you are not on the deployment host	On that host (or the approved tunnel): `just status` and `just prod-logs api`
`"enabled": true` but the Console has no Web Answer Mode	Backend gate is on; the Console UI gate is stale or unconfigured	`just prod-ui-up`, hard-refresh the browser, then `curl -fsS http://127.0.0.1:3000/api/health`. This is the separate Console UI gate — the same empty-`INFOCONNECT_API_KEY` path as failure mode 4 — not an `.env` mismatch

See Chat and Web Search API.

Post-deploy acceptance checks

Run every check on the deployment host (the private-LAN staging or production machine running Compose) or through the approved SSH tunnel to that host. localhost and 127.0.0.1 below mean that host, not your laptop.

Redis (6379) and Qdrant (6333) are internal Compose services. They need not be publicly exposed. A laptop-side curl to localhost:8000 that fails is not a failed deploy — you are curling the wrong machine.

Do not declare the tagged deploy successful until all of these pass. Use <configured-api-key> from API_KEY_CLIENTS, not an example demo key.

1. Product Version and image smoke

just version                 # Product Version of the checked-out tag (may differ from the tag string)
just prod-verify-version     # running GET /health version matches pyproject.toml
just prod-smoke              # already part of just prod-rebuild

v0.5.1 ships Product Version 0.5.0. Compare /health to just version, not to the Git tag string. See Health API.

2. Container and component health

just status docker --iterations 1
curl -fsS http://localhost:8000/health

GET /health must report "status": "healthy" with api, redis, qdrant, and celery ok. A degraded payload is not a pass.

3. Auth fail-closed and a real search

# Missing key is 401
curl -sS -o /dev/null -w "%{http_code}\n" \
  -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query":"machine learning","limit":5}'
# expect 401

# Configured key reaches the index (empty results are ok; 401 is not)
curl -fsS -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: <configured-api-key>" \
  -d '{"query":"machine learning","limit":5}'

In just prod-logs api, the startup banner No API keys are configured — EVERY request will be rejected with 401 Unauthorized must be absent.

just search-smoke default key

just search-smoke sends X-API-Key: ${INFOCONNECT_API_KEY:-test-key-123}. Production API_KEY_CLIENTS often does not include that demo key. Export INFOCONNECT_API_KEY=<configured-api-key> first, or use the curl above.

4. Console proxy (not just the homepage)

curl -fsS -o /dev/null http://127.0.0.1:3000          # HTML shell — necessary but not sufficient
curl -fsS http://127.0.0.1:3000/api/health            # proves INFOCONNECT_API_KEY was injected

Open Search or Status in the Console and confirm a backend-backed view, not the unconfigured 500.

5. Web-search gate vs Answer Mode

curl -fsS http://localhost:8000/web-search/capabilities \
  -H "X-API-Key: <configured-api-key>"

Interpret 401, unreachable, .env mismatch, and "enabled": true with no Console Web Answer Mode using the table in Web search enablement vs Chat Answer Mode. A matching JSON payload is not enough if the Console UI still lacks Web mode — that is failure mode 4 (just prod-ui-up, hard-refresh, /api/health).

Expected `.env`	Capabilities	Chat check
`WEB_SEARCH_ENABLED=false`	`"enabled": false`	Auto behaves as Docs-only; Web is rejected
`WEB_SEARCH_ENABLED=true`	`"enabled": true`	Auto / Docs-only / Web match Chat; Search still has no web control

6. Worker revision

git rev-parse --short=12 HEAD
just prod-logs worker-embedding-1

Worker hostnames must include that 12-character revision (embedding-1-<revision>@...). If just prod-up prints Celery worker revision conflict detected before startup. with Expected deployment revision: and Offending worker node(s):, that is a failed deploy — recover with just prod-down, not clean.

Summary checklist

Checked out a release tag (not main)
just prod-rebuild then just prod-ui-up
/health healthy; just prod-verify-version matches just version
Authenticated search with <configured-api-key> is not 401; fail-closed banner absent
http://127.0.0.1:3000/api/health returns JSON (Console key injected)
/web-search/capabilities matches WEB_SEARCH_ENABLED; Answer Mode is not mistaken for the gate
Workers registered on the current Git revision
.env changes applied by recreating containers, not docker compose restart
Recovery used just prod-down (data preserved), never clean / wipe, unless you intend to destroy the index

Tagged release: just prod-rebuild then just prod-ui-up​

Why the Console can start with an empty INFOCONNECT_API_KEY​

Why docker compose restart does not pick up .env​

Celery worker revision conflict — recover with just prod-down​

Web search enablement vs Chat Answer Mode​

Post-deploy acceptance checks​

1. Product Version and image smoke​

2. Container and component health​

3. Auth fail-closed and a real search​

4. Console proxy (not just the homepage)​

5. Web-search gate vs Answer Mode​

6. Worker revision​

Summary checklist​