Skip to main content

Upgrading InfoConnect (version to version)

This runbook upgrades a running production server from any release <from-tag> to any newer <to-tag>. It has two halves:

  1. Compute the delta for your exact jump (.env variables, service topology, reindex trigger).
  2. Execute a reversible procedure: record state, back up, stop, switch tag, reconcile config, rebuild, verify, and roll back if anything fails.

Production deploys are Docker Compose plus the justfile prod recipes. Release tags are pinned Git tags — never deploy from a moving main.

Companion guides

What an upgrade touches — and what it must not​

StateWhere it livesBehavior during upgrade
Search index (all vectors and payloads)Docker volume qdrant_data → /qdrant/storage (Qdrant v1.17.0)Durable. just prod-down preserves it. Never run just prod-down clean during an upgrade — that deletes volumes.
Redis broker stateDocker volume redis_dataTransient Celery queues, job status, and rate-limit counters. No backup required for an upgrade; re-check any in-flight uploads after the swap.
.env configurationRepo root, gitignoredUntouched by a tag checkout. You reconcile it manually (Step 6).
ML modelsmodel_cache volume + ./modelsReused across versions; new models are pre-downloaded during just prod-up.

Step 0: Dry-run on staging first​

Strongly recommended: restore the Qdrant backup (Step 3) into a staging Compose stack — a separate COMPOSE_PROJECT_NAME with remapped ports — and run this entire runbook against it before touching the live server. The only difference between a rehearsed upgrade and an outage is where the rollback runs.

Step 1: Compute the delta for your exact jump​

Every version jump has a different delta. Do not skip this: it decides which env vars, ports, and (critically) reindex work your jump needs.

1a. .env changes​

git diff <from-tag> <to-tag> -- .env.example

For every variable the diff adds, merge it into your live .env by hand:

  • Never cp .env.example .env — your live file carries your API keys, CORS origins, categories, and tuned settings; the example carries demo defaults.
  • Handle renamed or removed variables the same way: retire the old name so it cannot silently shadow a new one.
  • Check any default that is an operator gate. Example: WEB_SEARCH_ENABLED=false arrived defaulting to off — an operator who merged it could later enable web egress by choice, not by accident.

Then re-verify the two things that break loudest:

API_KEY_CLIENTS is valid JSON. It is a JSON object mapping API keys to client ids. An empty object ({}) fails closed: every request is rejected with 401 until keys are restored. Validate it parses:

python3 -c "import json; val=[l.split('=',1)[1] for l in open('.env') if l.startswith('API_KEY_CLIENTS=')][0]; json.loads(val); print('API_KEY_CLIENTS is valid JSON')"

Never export API_KEY_CLIENTS=... from a shell: shell export strips the quotes and breaks the JSON. The justfile deliberately avoids dotenv-load for the same reason — always edit the .env file itself.

Model IDs are still allowlisted. Model identifiers are validated against a fixed commercial-use allowlist (APPROVED_MODEL_IDS in app/config.py): DENSE_MODEL_NAME=BAAI/bge-small-en-v1.5, SPARSE_MODEL_NAME=Qdrant/bm25, RERANK_MODEL_NAME=Xenova/ms-marco-MiniLM-L-6-v2, SPACY_MODEL_NAME=en_core_web_sm, and the ModernBERT ONNX repo/file. A non-approved value is a settings validation error — containers crash-loop at startup. See OCR and Classification Settings before changing any model ID.

1b. Service topology changes​

git diff <from-tag> <to-tag> -- docker-compose.yml docker-compose.prod.yml

Look for new services. Each new service brings a host port that must be free and an image that must build. Example: the console service (operators Console, built from ./frontend, host port 3000) appeared in docker-compose.prod.yml after v0.2.0 — an operator upgrading past that point needs port 3000 free and the frontend/ build context present. The Console is managed with just prod-ui-up / just prod-ui-down, which resolve an API key from API_KEY_CLIENTS for it.

Also note changed scale variables (e.g. PROD_EMBEDDING_REPLICA_*) — just prod-up passes them automatically from the generated runtime plan.

1c. The reindex trigger​

Decide now whether your jump requires re-embedding the corpus. Plan a full reindex with just embed if the delta touches any of:

  • the dense or sparse embedding model ID (DENSE_MODEL_NAME, SPARSE_MODEL_NAME),
  • the vector dimension (VECTOR_SIZE — must match the dense model; currently 384),
  • the Qdrant collection/payload schema (e.g. the metadata v2 payload groups),
  • chunking parameters (CHUNK_SIZE, CHUNK_OVERLAP) or extraction/OCR behavior that changes source text,
  • EMBEDDING_PIPELINE_VERSION in app/utils/metadata_v2.py — bumped by the project whenever stored vectors or their source text can change.

If none of those moved, smoke the existing index first (Step 8) — an application-only release does not invalidate vectors. Stored points carry their own embedding provenance (dense_model, dense_dimensions, sparse_model, embedded_at, embedding_version) captured at embed time, so points embedded by an older release keep truthful metadata even after the upgrade. If a big jump added provenance metadata your old points lack, see Qdrant metadata v2 backfill before reindexing everything.

Step 2: Record the current state​

git rev-parse HEAD            # exact commit you are leaving
git describe --tags           # release you are leaving
just version                  # Product Version the running app resolves
curl -fsS http://localhost:6333/collections/documents   # point count you expect to still see

Note the Product Version (pyproject.toml) is not always the tag string — v0.5.1 shipped Product Version 0.5.0. Record both; the rollback and the version gate reference them.

Step 3: Back up durable state​

Copy .env aside (it is gitignored, and a checkout will not bring it back):

cp .env ".env.bak.$(git describe --tags).$(date +%Y%m%d-%H%M%S)"

The .env.bak.* pattern is gitignored on purpose — backups carry the same secrets as .env and must never be committable.

Snapshot the qdrant_data volume. Either tar the volume through a throwaway container (runs while prod is up or down):

# The volume is named <compose-project>_qdrant_data; confirm yours:
docker volume ls | grep qdrant_data

docker run --rm \
  -v <compose-project>_qdrant_data:/qdrant/storage:ro \
  -v "$(pwd)":/backup \
  alpine tar czf "/backup/qdrant_data-$(git describe --tags)-$(date +%Y%m%d-%H%M%S).tar.gz" \
  -C /qdrant/storage .

…or use the Qdrant snapshot API and copy the artifact out, as detailed in Backup, Restore, and Rollback:

curl -X POST http://localhost:6333/collections/documents/snapshots
curl -fsS http://localhost:6333/collections/documents/snapshots

Verify the backup file exists and is non-trivially sized before proceeding.

Step 4: Stop production (data preserved)​

just prod-down

just prod-down stops and removes containers but preserves volumes. It prints ✅ Production stopped (data preserved). — confirm you see that, not the clean path. just prod-down clean deletes qdrant_data and is for decommissioning, never for upgrades.

Step 5: Check out the target tag​

git fetch --tags
git checkout <to-tag>

Pin the release tag. Do not git pull on main in production — you lose reproducibility and the rollback target. git checkout prints a detached-HEAD warning; that is expected and correct for a pinned deploy.

Step 6: Reconcile .env from your Step 1a delta​

Apply the merged additions/renames/removals to the live .env. Re-run the API_KEY_CLIENTS JSON check and the model-allowlist check from Step 1a — this is the last cheap place to catch them.

Step 7: Bring the new version up​

just setup-models      # ensure ./models cache dirs exist with permissive permissions
just prod-up rebuild   # full path: deps → license audit → runtime plan → model pre-download → image rebuild → scale → wait → version gate

just prod-up rebuild rebuilds images from the new tag (a bare just prod-up would reuse the old images — the version gate in Step 8 exists to catch exactly that). It finishes with just prod-wait-for-services (Redis ping, Qdrant healthz, API /health) and just prod-verify-version on its own; if you ever bring services up manually, run just prod-wait-for-services yourself and do not skip it — first boot after a tag switch can spend minutes loading models.

Shorthand: just prod-rebuild = just prod-up rebuild + just prod-smoke.

Step 8: Verification gate​

Do not declare success until all of these pass:

just prod-verify-version   # running API reports the Product Version of the checked-out tag
just prod-smoke            # API + embedding worker images import the app and load reranker + spaCy
just status docker         # containers healthy, workers registered
just search "machine learning"   # real query against the surviving index
just prod-logs api         # scan startup logs
  • Version gate: just prod-verify-version compares the running API's /health version against pyproject.toml at the checked-out tag. A mismatch means stale images — rerun just prod-up rebuild. Compare against just version, remembering the tag-vs-Product-Version gap (Step 2).
  • Real search: confirms the index survived and auth works end to end. Then exercise ingestion once — embed a small sample (e.g. just embed test-documents/malaysia-wikipedia.txt) and search for content from it.
  • Fail-closed banner: in just prod-logs api, confirm the startup banner No API keys are configured — EVERY request will be rejected with 401 Unauthorized is absent. If it appears, API_KEY_CLIENTS did not survive the .env reconciliation — fix it before anything else.

If (and only if) Step 1c flagged a reindex, run it now: re-ingest each client's corpus with just embed <path> per client API key (see Backup, Restore, and Rollback for the drop-and-rebuild variant), then repeat the search checks.

Rollback​

Copy-pasteable. Run it whole, in order, whenever the verification gate fails and the cause is not a one-line .env fix:

# 1. Stop the failed release (volumes preserved)
just prod-down

# 2. Return to the recorded pre-upgrade commit (Step 2)
git checkout <old-commit>

# 3. Restore the pre-upgrade .env
cp .env.bak.<from-tag>.<timestamp> .env

# 4. Restore the Qdrant volume from the tar backup (destructive to current volume contents)
docker run --rm \
  -v <compose-project>_qdrant_data:/qdrant/storage \
  -v "$(pwd)":/backup \
  alpine sh -c "rm -rf /qdrant/storage/* && tar xzf /backup/qdrant_data-<from-tag>-<timestamp>.tar.gz -C /qdrant/storage"

# 5. Rebuild and re-verify the old version
just prod-up rebuild
just prod-verify-version
just search "machine learning"

Restore the volume only if the failed release wrote to the index (embedding ran, documents were ingested, a reindex started). If the failure happened before any write, the surviving volume is already the pre-upgrade state and Step 4 only wastes the maintenance window. End every rollback by re-checking just prod-logs api for the fail-closed banner, exactly as in Step 8.

Worked example: v0.2.0 → v0.5.1​

Worked example — a real jump through this runbook

Delta computed with git diff v0.2.0 v0.5.1 -- .env.example docker-compose.yml docker-compose.prod.yml:

  • .env additions (merge into live .env): HYBRID_ALPHA=0.5, SEARCH_DENSE_SCORE_FLOOR=0.70, SEARCH_SPARSE_SCORE_FLOOR=1.0, SEARCH_TOTAL_COUNT_CAP=1000, WEB_SEARCH_ENABLED=false, WEB_SEARCH_PROVIDER=ddgs, WEB_SEARCH_DDGS_BACKEND=bing, WEB_SEARCH_DDGS_TIMEOUT_SEC=5.0, WEB_SEARCH_TIMEOUT_SEC=5.5, CELERY_WORKER_REVISION=unknown. No variables were removed or renamed. WEB_SEARCH_ENABLED is an egress gate — it arrived off; keep it off unless you opt in.
  • Topology: one new service, console in docker-compose.prod.yml (built from ./frontend, host port 3000); embedding worker topology unchanged otherwise. Port 3000 must be free; the Console is optional and started via just prod-ui-up.
  • Reindex trigger: not fired. DENSE_MODEL_NAME, SPARSE_MODEL_NAME, RERANK_MODEL_NAME, and VECTOR_SIZE=384 are identical in both tags, and EMBEDDING_PIPELINE_VERSION stayed 1 — the release added embedding-provenance metadata for new points but did not change what a stored vector is. Existing index survives; smoke it instead of reindexing.
  • Version gate expectation: the v0.5.1 tag reports Product Version 0.5.0 (just version) — just prod-verify-version compares against 0.5.0, not 0.5.1.

Upgrade: Steps 2–8 as written. Rollback target: the commit recorded in Step 2 (the pre-upgrade tag, v0.2.0).

Summary checklist​

  • Delta computed for the exact jump (.env.example, docker-compose*.yml)
  • New vars merged by hand; API_KEY_CLIENTS re-validated as JSON; model IDs allowlisted
  • Reindex decision made from the Step 1c triggers, not from habit
  • Current commit + Product Version recorded; .env and qdrant_data backed up and verified
  • just prod-down (never clean) → git checkout <to-tag> → .env reconciled
  • just setup-models → just prod-up rebuild → gates all green (version, smoke, real search, no fail-closed banner)
  • Rollback rehearsed on staging before the live window