Skip to main content

Qdrant metadata v2 backfill

Use this runbook when a collection contains pre-v2 point payload metadata and the application has already been upgraded to write schema v2 metadata for new points.

The backfill only sets the top-level Qdrant payload key metadata to its schema v2 shape. It does not delete points, recreate collections, drop old indexes, or remove old payload fields outside metadata.

What changes​

The migration transforms legacy metadata groups into the v2 public contract:

  • metadata.processing.source_filename → metadata.source.filename, with extension and mime_type inferred from the filename when possible.
  • metadata.parsing.page_count → metadata.document.page_count as an integer.
  • metadata.processing.page_number, metadata.processing.page_label, and the point payload chunk_index → metadata.location.*.
  • metadata.classification.categories/status → metadata.classification.categories/status; legacy category_model and error are intentionally not exposed in public v2 metadata.
  • OCR/native extraction signals from metadata.ocr, extraction_method, and ocr_used → metadata.quality.text_extraction, ocr_used, and structured warnings when OCR or mixed extraction is meaningful.
  • Non-empty metadata.parsing.title/author and source-created date fields → metadata.extracted.*; empty title/author noise is dropped.
  • metadata.custom and other arbitrary non-system JSON metadata are preserved under metadata.custom.

Already-normalized v2 metadata is idempotent: repeated dry-runs or apply runs should report no additional updates after the first successful apply.

1. Dry-run first​

Dry-run is the default and performs metadata-only scans with with_payload=True and with_vectors=False.

uv run python scripts/backfill_qdrant_metadata_v2.py \
  --qdrant-url "$QDRANT_URL" \
  --collection-name "$COLLECTION_NAME"

For a smaller canary scan:

uv run python scripts/backfill_qdrant_metadata_v2.py --limit 100

Dry-run output from the 2026-06-11 guide run:

Metadata v2 backfill mode: dry-run
Collection: documents
Checked: 35
Would update: 0
Updated: 0
Skipped: 35
Errors: 0
No Qdrant writes were performed. Re-run with --apply to update metadata payloads.

Expected summary fields:

  • checked: points read from Qdrant.
  • would_update: points whose metadata would change.
  • updated: points written; always 0 in dry-run mode.
  • skipped: points with no metadata or already-normalized metadata.
  • errors: points or update batches that failed transformation/write handling.

2. Apply explicitly​

Only run apply after the dry-run count is expected and the application version that reads/writes metadata v2 is deployed.

uv run python scripts/backfill_qdrant_metadata_v2.py --apply

The script writes batches with Qdrant SetPayloadOperation, setting only the metadata key for selected point IDs. By default it waits for Qdrant write completion. Use --no-wait only during a controlled maintenance window where deferred write acknowledgement is acceptable.

3. Verify after apply​

Run the dry-run again:

uv run python scripts/backfill_qdrant_metadata_v2.py

Success criteria:

  • errors is 0.
  • would_update is 0 or only reflects new legacy writes from an older worker still running during the migration.
  • Search and document-list endpoints continue to return results with v2 metadata paths such as metadata.source.filename, metadata.location.page_number, and metadata.quality.ocr_used.

Qdrant index implications​

Issue #71 moved filter paths to v2 payload keys. This backfill updates payload values so those v2 indexes can match older points. It does not rebuild or delete indexes.

If the collection was created before v2 indexes existed, run the collection setup or verification path that creates the current payload indexes before relying on filters over v2 fields. Keeping old v1 indexes temporarily is safe; they can be removed later only after:

  1. all workers and API instances write v2 metadata,
  2. this backfill reports no pending updates,
  3. no dashboards, scripts, or saved filters query old paths such as page_number, ocr_used, or extraction_method, and
  4. a rollback plan exists for restoring old indexes if an external consumer was missed.

Do not combine old path removal with this backfill. Treat index cleanup as a separate maintenance task so data transformation and query/index cleanup can be verified independently.