Upgrading InfoConnect (version to version)

This runbook upgrades a running production server from any release <from-tag> to any newer <to-tag>. It has two halves:

1. Compute the delta for your exact jump (.env variables, service topology, reindex trigger).
2. Execute a reversible procedure: record state, back up, stop, switch tag, reconcile config, rebuild, verify, and roll back if anything fails.

Production deploys are Docker Compose plus the justfile prod recipes. Release tags are pinned Git tags — never deploy from a moving main.

Companion guides

• Production and Staging Deploy and Recovery — tagged just prod-rebuild with Console key injection, Compose .env recreate, worker-revision recovery, and post-deploy checks.
• Backup, Restore, and Rollback covers the snapshot mechanics this page references.
• On a development machine you can track a branch with git pull instead of pinning a tag — see Local development: testing a release with git pull at the end of this page.

What an upgrade touches — and what it must not

State	Where it lives	Behavior during upgrade
Search index (all vectors and payloads)	Docker volume qdrant_data → /qdrant/storage (Qdrant v1.17.0)	Durable. just prod-down preserves it. Never run just prod-down clean during an upgrade — that deletes volumes.
Redis broker state	Docker volume redis_data	Transient Celery queues, job status, and rate-limit counters. No backup required for an upgrade; re-check any in-flight uploads after the swap.
.env configuration	Repo root, gitignored	Untouched by a tag checkout. You reconcile it manually (Step 6).
ML models	model_cache volume + ./models	Reused across versions; new models are pre-downloaded during just prod-up.

Step 0: Dry-run on staging first

Strongly recommended: restore the complete backup (Step 3) into a staging Compose stack — a separate COMPOSE_PROJECT_NAME with remapped ports — and run this entire runbook against it before touching the live server. The only difference between a rehearsed upgrade and an outage is where the rollback runs.

Step 1: Compute the delta for your exact jump

Every version jump has a different delta. Do not skip this: it decides which env vars, ports, and (critically) reindex work your jump needs.

1a. .env changes

git diff <from-tag> <to-tag> -- .env.example

For every variable the diff adds, merge it into your live .env by hand:

• Never cp .env.example .env — your live file carries your API keys, CORS origins, categories, and tuned settings; the example carries demo defaults.
• Handle renamed or removed variables the same way: retire the old name so it cannot silently shadow a new one.
• Check any default that is an operator gate. Example: WEB_SEARCH_ENABLED=false arrived defaulting to off — an operator who merged it could later enable web egress by choice, not by accident.

Then re-verify the two things that break loudest:

API_KEY_CLIENTS is valid JSON. It is a JSON object mapping API keys to client ids. An empty object ({}) fails closed: every request is rejected with 401 until keys are restored. Validate it parses:

python3 -c "import json; val=[l.split('=',1)[1] for l in open('.env') if l.startswith('API_KEY_CLIENTS=')][0]; json.loads(val); print('API_KEY_CLIENTS is valid JSON')"

Never export API_KEY_CLIENTS=... from a shell: shell export strips the quotes and breaks the JSON. The justfile deliberately avoids dotenv-load for the same reason — always edit the .env file itself.

Model IDs are still allowlisted. Model identifiers are validated against a fixed commercial-use allowlist (APPROVED_MODEL_IDS in app/config.py): DENSE_MODEL_NAME=BAAI/bge-small-en-v1.5, SPARSE_MODEL_NAME=Qdrant/bm25, RERANK_MODEL_NAME=Xenova/ms-marco-MiniLM-L-6-v2, SPACY_MODEL_NAME=en_core_web_sm, and the ModernBERT ONNX repo/file. A non-approved value is a settings validation error — containers crash-loop at startup. See OCR and Classification Settings before changing any model ID.

1b. Service topology changes

git diff <from-tag> <to-tag> -- docker-compose.yml docker-compose.prod.yml

Look for new services. Each new service brings a host port that must be free and an image that must build. Example: the console service (operators Console, built from ./frontend, host port 3000) appeared in the prod Compose files after v0.2.0 — an operator upgrading past that point needs port 3000 free and the frontend/ build context present. The Console is managed with just prod-ui-up / just prod-ui-down; prod-ui-up resolves the validated effective API key for it.

Also note changed scale variables (e.g. PROD_EMBEDDING_REPLICA_*) — just prod-up passes them automatically from the generated runtime plan.

1c. The reindex trigger

Decide now whether your jump requires re-embedding the corpus. Plan a full reindex with just embed if the delta touches any of:

• the dense or sparse embedding model ID (DENSE_MODEL_NAME, SPARSE_MODEL_NAME),
• the vector dimension (VECTOR_SIZE — must match the dense model; currently 384),
• the Qdrant collection/payload schema (e.g. the metadata v2 payload groups),
• chunking parameters (CHUNK_SIZE, CHUNK_OVERLAP) or extraction/OCR behavior that changes source text,
• EMBEDDING_PIPELINE_VERSION in app/utils/metadata_v2.py — bumped by the project whenever stored vectors or their source text can change.

If none of those moved, smoke the existing index first (Step 8) — an application-only release does not invalidate vectors. Stored points carry their own embedding provenance (dense_model, dense_dimensions, sparse_model, embedded_at, embedding_version) captured at embed time, so points embedded by an older release keep truthful metadata even after the upgrade. If a big jump added provenance metadata your old points lack, see the legacy metadata v2 payload backfill before reindexing everything.

Step 2: Record the current state

git rev-parse HEAD            # exact commit you are leaving
git describe --tags           # release you are leaving
just version                  # Product Version the running app resolves
curl -fsS http://localhost:6333/collections/documents   # point count you expect to still see

Record both the Product Version (pyproject.toml) and tag string; the rollback and version gates validate the Product Version directly.

Step 3: Back up durable state

Copy .env aside (it is gitignored, and a checkout will not bring it back):

cp .env ".env.bak.$(git describe --tags).$(date +%Y%m%d-%H%M%S)"

The .env.bak.* pattern is gitignored on purpose — backups carry the same secrets as .env and must never be committable.

Take the complete quiesced backup from the checkout matching the serving release, following Backup, Restore, and Rollback. Verify the complete backup set as that guide requires before proceeding. Run the remaining commands in a persistent operator session: the manual recipes do not provide SSH-disconnect recovery.

Step 4: Stop production (data preserved)

just prod-down

just prod-down stops and removes containers but preserves volumes. It prints ✅ Production stopped (data preserved). — confirm you see that, not the clean path. just prod-down clean deletes qdrant_data and is for decommissioning, never for upgrades.

Step 5: Check out the target tag

git fetch --tags
git checkout <to-tag>

Pin the release tag. Do not git pull on main in production — you lose reproducibility and the rollback target. git checkout prints a detached-HEAD warning; that is expected and correct for a pinned deploy.

Step 6: Reconcile .env from your Step 1a delta

Apply the merged additions/renames/removals to the live .env. Re-run the API_KEY_CLIENTS JSON check and the model-allowlist check from Step 1a — this is the last cheap place to catch them.

Step 7: Bring the new version up

just setup-models      # ensure ./models cache dirs exist with permissive permissions
just prod-up rebuild   # full path: deps → license audit → runtime plan → image rebuild → model pre-download → trace checks → scale → wait → version gate
just prod-ui-up        # recreate Console and recheck readiness

just prod-up rebuild rebuilds every service image from the new tag, resolves the Console key before Compose mutation, and gates startup with readiness plus an authenticated read. Bare just prod-up builds the API image for its in-network revision check but may reuse old worker and Console images, so it is not a release upgrade command. Startup finishes with just prod-wait-for-services (Redis ping, Qdrant healthz, API /health), just prod-verify-version, and just prod-ui-check; if you bring services up manually, run those checks yourself. First boot after a tag switch can spend minutes loading models.

Shorthand: just prod-rebuild = just prod-up rebuild + just prod-ui-up + just prod-smoke.

Step 8: Verification gate

Do not declare success until all of these pass:

just prod-verify-version   # running API reports the Product Version of the checked-out tag
just prod-smoke            # API + embedding worker images import the app and load reranker + spaCy
just status docker         # containers healthy, workers registered
just search "machine learning" -- --scope Content  # real query against the surviving index
just prod-logs api         # scan startup logs

• Version gate: just prod-verify-version compares the running API's /health version against pyproject.toml at the checked-out tag. A mismatch means stale images — rerun just prod-up rebuild. Compare against just version, remembering the tag-vs-Product-Version gap (Step 2).
• Real search: confirms the index survived and auth works end to end. Then exercise ingestion once — embed a small sample (e.g. just embed test-documents/malaysia-wikipedia.txt) and search for content from it.
• Fail-closed banner: in just prod-logs api, confirm the startup banner No API keys are configured — EVERY request will be rejected with 401 Unauthorized is absent. If it appears, API_KEY_CLIENTS did not survive the .env reconciliation — fix it before anything else.

If (and only if) Step 1c flagged a reindex, run it now: re-ingest each client's corpus with just embed <path> per client API key (see Backup, Restore, and Rollback for the drop-and-rebuild variant), then repeat the search checks.

Rollback

Copy-pasteable. Run it whole, in order, whenever the verification gate fails and the cause is not a one-line .env fix:

# 1. Stop the failed release (volumes preserved)
just prod-down

# 2. Return to the recorded pre-upgrade commit (Step 2)
git checkout <old-commit>

# 3. Restore the pre-upgrade .env
cp .env.bak.<from-tag>.<timestamp> .env

# 4. If the failed release changed data, restore the complete pre-upgrade
#    backup using Backup, Restore, and Rollback before starting the old release.

# 5. Rebuild and re-verify the old version
just prod-up rebuild
just prod-verify-version
just search "machine learning" -- --scope Content

Restore data only if the failed release changed it, following Backup, Restore, and Rollback. End every rollback by repeating the Step 8 checks.

Worked example (historical, as of v0.5.1): v0.2.0 → v0.5.1

Worked example — a real jump through this runbook

This example reflects the v0.5.1 release and is retained as a point-in-time reference; the current release is v0.5.15. Delta computed with git diff v0.2.0 v0.5.1 -- .env.example docker-compose.yml docker-compose.prod.yml:

• .env additions (merge into live .env): HYBRID_ALPHA=0.5, SEARCH_DENSE_SCORE_FLOOR=0.70, SEARCH_SPARSE_SCORE_FLOOR=1.0, SEARCH_TOTAL_COUNT_CAP=1000, WEB_SEARCH_ENABLED=false, WEB_SEARCH_PROVIDER=ddgs, WEB_SEARCH_DDGS_BACKEND=bing, WEB_SEARCH_DDGS_TIMEOUT_SEC=5.0, WEB_SEARCH_TIMEOUT_SEC=5.5, CELERY_WORKER_REVISION=unknown. No variables were removed or renamed. WEB_SEARCH_ENABLED is an egress gate — it arrived off; keep it off unless you opt in.
• Topology: one new service, console in docker-compose.console.yml (built from ./frontend, host port 3000); embedding worker topology unchanged otherwise. Port 3000 must be free; the Console is optional and started via just prod-ui-up.
• Reindex trigger: not fired. DENSE_MODEL_NAME, SPARSE_MODEL_NAME, RERANK_MODEL_NAME, and VECTOR_SIZE=384 are identical in both tags, and EMBEDDING_PIPELINE_VERSION stayed 1 — the release added embedding-provenance metadata for new points but did not change what a stored vector is. Existing index survives; smoke it instead of reindexing.
• Version gate expectation: the v0.5.1 tag reports Product Version 0.5.1 (just version) — just prod-verify-version requires /health to match.

Upgrade: Steps 2–8 as written. Rollback target: the commit recorded in Step 2 (the pre-upgrade tag, v0.2.0).

Local development: testing a release with git pull

On a development machine — not staging or production — you can follow a moving branch with git pull instead of pinning a release tag. The production steps above still apply to any real deployment; this section is the lighter local loop.

cd infoconnect-search-engine
git pull
diff .env .env.example      # review new/renamed variables (see Step 1a for the rules)
just prod-up rebuild        # use plain `just prod-up` only for .env-only changes

Sync .env before restarting

New releases often add configuration. Compare your live .env against the example and merge new variables by hand — never cp .env.example .env, which wipes your API_KEY_CLIENTS, CORS, and tuned settings:

diff .env .env.example

Lines the diff adds are new variables to consider. Re-run the API_KEY_CLIENTS JSON check and the model-allowlist check from Step 1a after editing.

Rebuild or plain restart?

What changed in the git pull output	Command
Only .py / application code	just prod-up rebuild (workers have no source bind mount)
.env values only	just prod-up (recreate containers to apply interpolation)
pyproject.toml, docker/Dockerfile, or docker-compose*.yml	just prod-up rebuild (new deps / image)
Major jump or anything unclear	just prod-down then just prod-up rebuild (clean, no cached layers)

Verify

just status         # every service green
just search-smoke   # sends an explicit content-scope query; checks the Evidence Result contract

Then open http://localhost:8000/docs to confirm the endpoints — and any new ones — are present. If services fail to start, the cause is almost always a missing new .env variable (diff .env .env.example) or uninstalled dependencies (just prod-up rebuild).

Summary checklist

• Delta computed for the exact jump (.env.example, docker-compose*.yml)
• New vars merged by hand; API_KEY_CLIENTS re-validated as JSON; model IDs allowlisted
• Reindex decision made from the Step 1c triggers, not from habit
• Current commit + Product Version recorded; .env and the complete Step 3 backup set verified
• just prod-down (never clean) → git checkout <to-tag> → .env reconciled
• just setup-models → just prod-up rebuild → gates all green (version, smoke, real search, no fail-closed banner)
• Rollback rehearsed on staging before the live window