Skip to main content

Backup, Restore, and Publication Recovery

Use this runbook for retained-volume Redis process crashes, publication fences, Redis AOF incidents, and release rollback. Publication state is safety-critical: Qdrant Manifests are the sole read-time generation authority, while Redis owns the publication journal, fences, worker roster, and cached registry.

Tagged deploys, Console key injection, Compose .env recreate, and Celery worker revision-conflict recovery are in Production and Staging Deploy and Recovery. Version-to-version jumps are in Upgrading InfoConnect.

Guarantee boundary​

The retained-volume guarantee requires Redis, Qdrant, and shared UPLOAD_DIR durable media to survive and honor fsync. The upload volume contains the dispatch outbox needed to recover in-flight publication classification. This does not add high availability, replication, Sentinel, automatic failover, or off-host backup scheduling.

The maintained scripts/publication_backup.py command creates a checksummed, named backup set for the four Qdrant publication stores. It does not back up Redis or UPLOAD_DIR, and it does not define an RPO. Capture their complete durable media under the same global write quiescence. Never treat a live-volume copy as a consistent backup.

A named volume is persistence, not backup. If the only Redis publication volume is lost or corrupted beyond repair, keep search fail-closed. An authoritative Qdrant Manifest can prove read-time generation eligibility, but it cannot reconstruct the lost Redis journal, fences, or worker state. Never fabricate those records or choose a generation from timestamps or the greatest generation number.

First response​

  1. Stop new publishers and leave affected Clients fenced.
  2. Preserve the Redis volume and complete multipart AOF directory, Qdrant volume, and shared UPLOAD_DIR volume before attempting repair.
  3. Capture Redis INFO server, INFO persistence, CONFIG GET, the authority health diagnostics, and the four Qdrant collection schemas.
  4. Confirm the running Redis image is the qualified digest-pinned 7.4.10 build.
  5. Do not remove a captured worker, lower the required ACK count, change a registry pointer, or return partial search results to restore availability.

Publication metrics and logs use bounded state categories. They must not include Client IDs or Custom Metadata values.

The operator metrics are:

  • infoconnect_publication_store_operations_total{store,operation,outcome} for fixed Content, Fuzzy, Metadata, and Manifest initialization, backup, validation, restore, and cleanup outcomes;
  • infoconnect_publication_store_records{store} for current per-store capacity;
  • infoconnect_publication_transitions_total{transition,outcome} for stage, verify, fence, activation, readiness, acknowledgement, cleanup, and purge.
  • infoconnect_publication_runtime{metric="cleanup_intents"} for the bounded count of obsolete generations durably waiting for Celery delivery or retry.

All label values come from closed enums. Use job diagnostics and bounded logs for identity-level repair; never add a Client, document, generation, or Custom Metadata value as a metric label.

Dry-run evidence​

Capture evidence before making a live change:

just status docker --iterations 1
curl -fsS http://localhost:6333/collections
docker compose exec -T redis redis-cli INFO persistence
docker compose ps

This dry-run records API, Qdrant, Redis, and worker health without mutating publication state. Keep the output with the incident or release record.

To pause indexing workers during a maintenance window, scale every embedding and preprocessing worker to zero, then confirm the stopped replicas with just status docker --iterations 1. Do not treat a worker pause as a write quiescence proof while API or other publisher processes remain live.

There is no acceptable data loss window for the Redis publication authority: if its durable journal and registry cannot be recovered, search stays fail-closed. If a compatible Qdrant snapshot cannot be restored, rebuild from source under each Client's own API-key scope, then complete the normal publication proof before search opens.

Retained generation writers​

For manual orphan cleanup, confirm that the reported generation is still abandoned. Use the reported deterministic cleanup_id to correlate the cleanup outcome. If cleanup diagnostics identify retained writer admission, follow the quiescence and manual-clear steps below before re-driving cleanup. An upsert transport timeout can leave a Qdrant request running after the client stops waiting; a successful retry does not clear the earlier admission. A killed writer or an unconfirmed Redis admission/release can also require this recovery.

  1. Quiesce affected writers and prove their outstanding Qdrant calls have completed. A retry, elapsed client deadline, or stopped worker alone is insufficient proof.
  2. Identify the affected Client, document, version, and generation from diagnostics. Derive its deterministic cleanup_id using app.services.publication.cleanup_intent_id(PublicationIdentity(...), generation).
  3. In the configured publication Redis keyspace, clear only that generation's generation-writers:<cleanup_id> set. Keep its cleanup-tombstones hash entry; never remove a tombstone to admit retries.
  4. Re-drive the Celery task named app.tasks.process_document.cleanup_publication_generation on the preprocessing queue with client_id, document_id, version_id, and obsolete_generation. Cleanup preserves a currently selected generation.
  5. For an obsolete generation, verify zero points across Content, Fuzzy, Metadata, and Manifest, and no Manifest-less pair for that identity and generation.

Generation cleanup does not take a global read fence. Retained writers block cleanup with PublicationBusy and leave the intent for retry. The cleanup safety contract is owned by ADR-0011.

Lost classification dispatch reply​

If Celery accepted a publication classification dispatch but the caller lost its reply, do not manually delete or edit the staged upload, node batches, or adjacent *.outbox.json and lock files. The outbox records a stable task ID before broker publication and owns the selector, classification-completion, and publication steps through atomic accepted, running, and completed states. A retry resolves that same entry: it either replays the accepted task ID, waits for its running owner, or returns its completed result. It must not start another classification or publication state machine.

Before the outbox expiry, an accepted or running dispatch keeps this retry behavior. After expiry, the parent-delivery reconciler consults the publication authority operation named by parent_result.publication_operation_id: a READY operation is recorded as a synthetic completed result and cleaned up, while any other or missing operation state is abandoned and the outbox is removed. This terminal cleanup is automatic and remains under the outbox lock; operators should not perform it by hand.

Keep every API, preprocessing, and embedding container mounted to the same UPLOAD_DIR volume. The bounded outbox cleanup may remove completed delivery evidence, or evidence whose staged node manifest is already gone, after its renewed Celery delivery lifetime; it is not an operator recovery command.

Retained-volume Redis SIGKILL or process restart​

Restart Redis against the same retained volume. Startup must refuse readiness unless all of these remain true:

  • Redis is 7.4.10 and reports the expected new run_id;
  • AOF is enabled and healthy with appendfsync everysec, no-appendfsync-on-rewrite no, and aof-load-truncated no;
  • maxmemory-policy is noeviction;
  • the authority enters RECOVERING durably before any worker becomes ready.

Every search process must discard its local mirrors, register its stable deployment slot with a new random incarnation, and load a complete immutable registry snapshot. The control process verifies every selected identity against exactly one authoritative Qdrant Manifest and its committed lane evidence. Workers may serve only after loading the verified immutable snapshot and confirming a new safe serving lease. Until then, every affected search returns one retryable 503.

Incomplete pre-activation fence​

An operation in FENCED with an unchanged registry pointer is not published.

  1. Read the exact operation journal, immutable captured set, and fence ACK map.
  2. Reject ACKs from missing, joining, replaced, lease-expired, or stale incarnations.
  3. Resume only when every originally captured member has a durable ACK. There is no quorum and no member removal within that attempt.
  4. If the operation is abandoned before activation, use the coordinator's idempotent abort/reconcile operation, verify that the prior generation's Manifest remains authoritative, then publish the old ready snapshot durably. The abort is permitted only after every missing captured incarnation is lease-expired or replaced; it leaves the verified staged generation retryable with a fresh immutable roster.

Do not clear the fence with an ad hoc Redis command.

Lost activation reply or WAITAOF timeout​

A timeout, connection loss, or local AOF count of zero is not success.

  1. Reconnect and read the exact operation ID, request digest, identity, generation, and journal state.
  2. If it remains pre-activation, retry the versioned activation script idempotently.
  3. If it is exactly ACTIVATED, issue a later monotonic durability-probe write and then WAITAOF 1 0 <timeout> on that same new connection.
  4. Expose no ready marker until the probe confirms local count 1.
  5. Any conflict leaves the Client fenced and raises an invariant alarm.

The activation script leaves the Client fenced by design. A durable registry pointer is not yet permission for a worker to serve.

Post-activation convergence failure​

Keep the activated Client fenced. Re-read the exact Redis registry and verify that each selected identity has exactly one authoritative Qdrant Manifest with the committed lane evidence. Captured workers load the same complete snapshot, recheck its marker and epoch, install it atomically, and durably ACK its digest. Admit joining workers only after all captured workers converge.

The worker control tick automatically detects a FENCED or ACTIVATED operation whose recovery_owner lease has expired and drives the existing takeover/recovery path. A recapture that briefly captured no live workers therefore does not require an API restart; keep the Client fenced while the takeover completes and verify that the authority returns to READY.

If a missing captured incarnation is lease-expired or replaced, durably start a new convergence recovery epoch, capture the current live roster, and require that complete new roster to ACK the already-activated snapshot. Never reuse an ACK from the abandoned roster.

Never roll the registry pointer backward or choose another generation from Qdrant.

AOF truncation, corruption, or unhealthy persistence​

aof-load-truncated no intentionally turns a malformed tail into failed startup.

  1. Copy the complete Redis 7 multipart AOF directory and manifest before inspection.
  2. Keep API readiness false and all publication state fenced.
  3. Do not run redis-check-aof --fix as an availability shortcut.
  4. Repair or restore only through an incident-approved procedure, then start in RECOVERING and verify every selected registry identity against Qdrant Manifest authority.

If Redis durable publication media cannot be recovered, keep search fail-closed. Never fabricate the journal or registry from Qdrant timestamps or generation numbers.

Four-store Qdrant backup and restore​

The four Qdrant publication stores are Content, Fuzzy, Metadata, and Manifest. The Manifest lane is the sole read-time generation authority; the other three lanes are immutable evidence validated against it.

Qdrant snapshots are per collection, so publication and document writes must be quiesced for the whole operation. Create an immutable named set:

uv run python scripts/publication_backup.py create \
  --name release-2026-09-01 \
  --backup-dir /srv/infoconnect/backups/publication \
  --quiesced-ingestion \
  --json

The directory contains exactly four snapshots—Content, Fuzzy, Metadata, and Manifest—plus manifest.json. The manifest records each file's SHA-256 digest and size, the Product Version, publication layout version, Metadata Discovery signature, vector dimensions and model signatures, collection names and schema digest, and the stable Redis registry revision. Copy the complete directory to durable off-host media together with the quiesced Redis and UPLOAD_DIR backup.

Validate the set before an incident or restore:

uv run python scripts/publication_backup.py validate \
  --name release-2026-09-01 \
  --backup-dir /srv/infoconnect/backups/publication \
  --json

Validation rejects a missing, extra, empty, damaged, path-escaping, or release-incompatible lane artifact before changing Qdrant. Restore Redis and UPLOAD_DIR first, keep all publication writes quiesced, then restore the four Qdrant snapshots:

uv run python scripts/publication_backup.py restore \
  --name release-2026-09-01 \
  --backup-dir /srv/infoconnect/backups/publication \
  --quiesced-ingestion \
  --json

The restore command acquires a leased, single-writer maintenance fence and atomically moves the global authority into MAINTENANCE before the first Qdrant mutation. A concurrent contender is rejected; an expired owner can be replaced only with the explicit quiescence acknowledgement. Workers invalidate local request gates, staging and activation stop, and the command waits one safe-serving deadline before restoring any lane. It returns the authority to READY only after all four snapshots and active Manifest evidence pass. On restore failure, the exact maintenance fence is abandoned atomically before the bounded failure is returned, so a later operation can retry without inheriting a stale owner.

Restore succeeds only when the Redis registry revision remains stable, its active identity count exactly matches Qdrant authority, and every authoritative Manifest re-verifies all required lane IDs, counts, digests, and Metadata vectors. Otherwise the command returns a bounded reason code and repair guidance without Client IDs or Custom Metadata values. Application startup independently verifies every selected Redis identity against Qdrant before search opens.

Application rollback is safe before the first Redis-only activation under a global fence. After an activation, deploying code that cannot interpret the authority schema is unsafe. Remain fenced and follow an explicit schema-compatible rollback or restore plan.

Record the incident owner, application revision, Redis image digest and run ID, authority boot/schema/script versions, Qdrant version and collection-schema digests, operation IDs, durability probe result, and the evidence used to resume serving.