Skip to main content

Backup, Restore, and Rollback

Use this runbook before UAT, production changes, collection rebuilds, or release rollback. Commands that change live data are separated from dry-run checks and must be run only during a maintenance window.

Dry-run Evidence​

Capture this evidence before making changes:

just status docker --iterations 1
curl -fsS http://localhost:6333/collections/documents
docker compose exec -T redis redis-cli INFO persistence
docker compose ps

The dry-run proves API, Qdrant, Redis, and workers are reachable, and records the current collection and Redis persistence state.

Qdrant snapshot​

Qdrant data lives in the qdrant_data volume at /qdrant/storage. The preferred backup is a collection snapshot:

curl -X POST http://localhost:6333/collections/documents/snapshots
curl -fsS http://localhost:6333/collections/documents/snapshots

Copy the generated snapshot out of the Qdrant container or volume and store it with the release ID, environment, collection name, and timestamp. Verify the copied file exists before proceeding.

Redis AOF​

Redis is configured with append-only persistence: appendonly yes, appendfsync everysec, and noeviction. Redis stores queues, job state, rate-limit counters, and transient coordination state. Back up the AOF before maintenance:

docker compose exec -T redis redis-cli SAVE
docker compose cp redis:/data ./backups/redis-data

The acceptable data loss window for Redis is one second of acknowledged AOF writes plus any in-flight Celery task state not yet persisted. If Redis is lost, clients may need to re-submit uploads whose status cannot be recovered.

pause indexing workers​

Pause indexing workers before restoring snapshots or rebuilding a collection:

docker compose up -d --scale worker-embedding-1=0 --scale worker-embedding-2=0 --scale worker-preprocessing-1=0
just status docker --iterations 1

Resume workers after Qdrant and Redis are healthy:

just prod-up

Restore and rebuild​

Restore Qdrant from the snapshot artifact matching the environment and release. If a full rebuild is safer, drop the bad collection during the maintenance window and re-ingest source documents per Client with the correct API key mapping.

After restore or rebuild, verify:

curl -fsS http://localhost:6333/collections/documents
just status docker --iterations 1
just search-smoke

Rollback​

Rollback has two parts: application image rollback and data rollback. First deploy the previous pinned image or commit using the release process in the testing-new-releases guide. Then restore Qdrant and Redis only if the data was changed by the failed release.

Record the rollback owner, previous image or commit, Qdrant snapshot name, Redis AOF backup path, verification output, and decision to resume workers.