Data Backfills

Two backfill paths bring documents onto a current schema. Metadata Discovery converges automatically after new ingest; its operator command remains available for historical corpora and repairs. The legacy metadata v2 migration is still deliberate and operator-only.

Backfill	When you need it	Script
Metadata Discovery backfill	A client's Metadata and Both search scopes stay closed because its documents predate Metadata Discovery, or the Metadata Discovery signature changed	scripts/backfill_metadata_discovery.py
Legacy metadata v2 payload backfill	A collection still holds pre-v2 point payload metadata after the app was upgraded to write v2	scripts/backfill_qdrant_metadata_v2.py

Most current deployments only ever run the Metadata Discovery backfill. The v2 payload backfill is a historical one-time migration for collections created before the metadata v2 schema existed.

Metadata Discovery backfill

Metadata Discovery searches the Custom Metadata values attached to your documents. Documents ingested before the feature shipped carry no Metadata Leaves, so Metadata and Both search scopes stay closed for a client until its corpus is backfilled.

Automatic convergence after ingest

New ingests converge readiness automatically. When process_document finishes a document — including a completed-with-no-text document — it links a converge_discovery_readiness task onto the publication finalize step. That task is the Metadata Backfill invoked automatically for that one client: it takes the same per-client claim, plans, republishes pending versions, and commits terminal readiness, so a successful embed leaves the client ready with no operator step.

While a converge is pending, Metadata and Both searches answer 409 METADATA_DISCOVERY_NOT_READY with Retry-After, and GET /status/{job_id} reports metadata_ready: false with readiness_state: "backfilling". The field flips to metadata_ready: true only after the converge commits ready. Concurrent embeds serialize on the backfill claim. If another run holds it (backfill_already_running), converge reschedules after the claim's bounded retry delay without consuming its error retry budget. The claim's renewal and expiry rules are described below. Lost claims and genuine backfill errors still consume retries and fail when the configured retry limit is exhausted. A violating document leaves the client blocked with its violations surfaced, never silently ready.

A successful Hard Delete under publication also enqueues the same task for the client, because it changes the accepted publication set without an ingest chain, and the sweep skips clients already ready. The enqueue is best-effort: a broker failure is logged and the delete still succeeds, and the cached counts refresh on the client's next converge.

Clients that never ingest again — corpora published before this feature, or a client whose converge was lost to a worker kill — are covered by the reconcile_discovery_readiness sweep. The preprocessing worker starts it on worker_ready and it reschedules itself every 60 seconds: it enumerates clients that own an accepted publication manifest and enqueues the same converge_discovery_readiness task for each one that is not ready. Clients with an in-flight ingest are skipped (their own chain converges them), and clients blocked on contract violations are skipped until the data is repaired. An in-flight ingest is proven by a live job: the sweep and a successful converge both reap ingest-pending markers whose jobs are gone, so a marker orphaned by a lost converge cannot hold a client backfilling until the marker TTL lapses.

The manual command below remains the repair path: use it to recover a client stuck backfilling after repeated converge failures, or to --force a full re-embed.

Before you start

Run the Legacy Publication bootstrap first. The backfill only touches Document Versions the publication registry already accepted; anything the bootstrap blocked stays blocked and is never backfilled into visibility.

Dense Metadata retrieval uses a new Metadata collection generation with both dense and sparse value vectors. The service never adds a dense vector name to the old sparse-only collection or pretends its points are current. After deploying this release, the advanced Metadata Discovery signature makes earlier readiness stale; complete this backfill before reopening Metadata or Both for a client.

Dry run first

uv run python scripts/backfill_metadata_discovery.py --client-id acme

A dry run writes nothing. It reports, per client:

• versions_total — accepted Document Versions in the corpus
• up_to_date — versions already carrying current Metadata Leaves
• pending — versions the backfill would republish
• violating — versions whose Custom Metadata breaks the contract, with the offending path

Dry runs exit 0 even when work remains, so they are safe in a pipeline.

Apply

uv run python scripts/backfill_metadata_discovery.py \
  --client-id acme --apply --max-versions 200

Each targeted Document Version is republished as a new immutable generation carrying its existing content chunks, fuzzy terms, visibility, and manifest plus the derived Metadata Leaves and their dense and sparse value vectors. Both vectors are generated from the lossless rendered value, not its path. The new generation is activated and the superseded one is purged. Committed evidence is never mutated in place, and a Soft Deleted version stays deleted.

--apply exits non-zero while any work is outstanding — deferred batches, failed versions, or blocked versions — so a runbook cannot mistake a partial rollout for a completed one.

Bounded batches and resuming

--max-versions caps how many Document Versions one run republishes per client. Anything beyond it is reported as deferred and the client stays backfilling. Re-run the same command to continue:

until uv run python scripts/backfill_metadata_discovery.py \
        --client-id acme --apply --max-versions 200; do sleep 5; done

Resuming is safe because progress is derived from published evidence, not from a cursor. A version whose active generation already carries the leaves the current signature derives is up_to_date, so a re-run after a crash, a bounded batch, or a completed rollout converges instead of republishing. If a run dies after staging a generation but before activation, the next run replays that stranded generation and preserves the original manifest provenance before activating it; a per-run job id cannot make an otherwise identical immutable generation fail forever. Running a completed backfill again publishes nothing; pass --force to re-embed and republish every accepted version anyway.

Throughput: batch embedding and workers

--batch-size controls how many versions' Metadata Leaf texts one worker embeds in a single call (default 32). Batching amortizes model overhead and is the largest speedup on CPU-only hosts. --workers runs that many concurrent workers per client (default 1); each drains the shared pending queue, embeds a batch, and republishes independently. The publication fence/ACK protocol keeps concurrent republishes of distinct versions safe. On a 16-thread host, --workers 4 with the default batch size is a reasonable starting point.

Only one apply invocation may work on a client at a time. A second invocation exits non-zero with backfill_already_running and a bounded retry delay; it does not plan, publish, or change readiness. The claim is renewed while the first invocation runs and expires automatically if its process dies.

publication_authority_inconsistent is different from an empty client or a metadata violation. It means accepted publication evidence exists without usable client authority state; repair the publication authority before retrying. The client is never marked ready from that state.

Rolling out to every client

uv run python scripts/backfill_metadata_discovery.py --all-clients --apply

--all-clients targets the union of every client in API_KEY_CLIENTS and every client that owns a published manifest in Qdrant. It also compares the number of manifest clients with the publication authority catalogue. If those counts disagree, the command continues with the clients it can enumerate, warns that coverage may be partial, and exits non-zero. With --json, the client_discovery object reports coverage_partial, manifest_clients, and authority_catalogue_clients; logs go to stderr so stdout remains one parseable JSON document.

Violations block readiness

A Document Version whose Custom Metadata exceeds a Metadata Discovery cap is never truncated, split, skipped, or lexically downgraded. It is reported and it blocks readiness:

{
  "client_id": "acme",
  "readiness_state": "blocked",
  "violations": [
    {
      "document_id": "contract-2024-118",
      "version_id": "v3",
      "code": "metadata_rendered_leaf_bytes_limit",
      "path": ["customer", "notes"],
      "observed": 4096,
      "limit": 2048,
      "unit": "utf8_bytes"
    }
  ]
}

Repair the document with PATCH /update, then re-run the backfill. Reports carry paths, counts, and document identities — never a Custom Metadata value.

Watching progress

Each applied run binds one job id per client and publishes progress through the ordinary status endpoint:

curl -H "X-API-Key: $CLIENT_KEY" http://localhost:8000/status/$JOB_ID

{
  "job_id": "6f1f0d0c8b4b4bd2a0d8f0d1a3c7e5b9",
  "operation": "metadata_backfill",
  "status": "processing",
  "readiness_state": "backfilling",
  "versions_total": 1284,
  "versions_completed": 400,
  "versions_pending": 884
}

The command also streams one JSON progress line per settled version — versions_done, versions_total, elapsed, eta, plus backfilled and failed counts — so a watcher can tail the run without polling. Plain runs emit these lines on stdout; with --json they move to stderr so stdout stays a single report document.

Embed, metadata update, and metadata backfill jobs share this one client-bound language, including the 404 a job belonging to another client returns.

The client's own view is GET /discovery/readiness. Until it reports ready, Metadata and Both requests fail with 409 METADATA_DISCOVERY_NOT_READY and Content search keeps working.

Model, vector, and version changes

Each Metadata Leaf records the Metadata Discovery version and the dense/sparse model signature it was built under. Changing any of them, or moving to a new Metadata vector schema, makes published leaves incompatible: readiness drops back to not_started with stale_signature: true, Metadata and Both are gated again, and the backfill has to be re-run. The apply path embeds every leaf before staging, verifies both named vectors with the complete publication generation, and marks the client ready only after Qdrant proves the authoritative Manifest for every accepted Document Version. Run a dry run after any embedding-model or Metadata collection-generation change to see the scope of the work.

Operator metrics

/metrics exposes aggregate rollout progress with no client identity attached:

infoconnect_metadata_discovery_readiness_clients{state="ready"} 12
infoconnect_metadata_discovery_readiness_clients{state="backfilling"} 1
infoconnect_metadata_discovery_readiness_clients{state="blocked"} 2

Blocked clients never fail global service health.

Command reference

Flag	Default	Description
--client-id	—	Target one client; repeat for several
--all-clients	—	Target every configured and every published client
--apply	off (dry run)	Publish leaves and record readiness
--batch-size	32	Versions embedded per batch call per worker
--max-versions	100	Maximum Document Versions republished per client per run
--workers	1	Concurrent workers draining the pending queue per client
--force	off	Re-embed and republish even up-to-date versions
--max-violations	20	Maximum violating identities reported per client
--json	off	Emit one machine-readable report on stdout; send logs to stderr
--qdrant-url	QDRANT_URL	Qdrant endpoint
--publication-redis-url	PUBLICATION_REDIS_URL	Publication coordinator endpoint

Exit code is 0 only when every targeted client converged with nothing blocked, failed, or deferred.

Legacy metadata v2 payload backfill

Historical migration

This backfill is for collections created before the metadata v2 payload schema existed. Deployments first stood up on a v2-writing release never need it. Keep it only if you are carrying a corpus older than the metadata v2 cutover.

Use this runbook when a collection contains pre-v2 point payload metadata and the application has already been upgraded to write schema v2 metadata for new points.

The backfill only sets the top-level Qdrant payload key metadata to its schema v2 shape. It does not delete points, recreate collections, drop old indexes, or remove old payload fields outside metadata.

What changes

The migration transforms legacy metadata groups into the v2 public contract:

• metadata.processing.source_filename → metadata.source.filename, with extension and mime_type inferred from the filename when possible.
• metadata.parsing.page_count → metadata.document.page_count as an integer.
• metadata.processing.page_number, metadata.processing.page_label, and the point payload chunk_index → metadata.location.*.
• metadata.classification.categories/status → metadata.classification.categories/status; legacy category_model and error are intentionally not exposed in public v2 metadata.
• OCR/native extraction signals from metadata.ocr, extraction_method, and ocr_used → metadata.quality.text_extraction, ocr_used, and structured warnings when OCR or mixed extraction is meaningful.
• Non-empty metadata.parsing.title/author and source-created date fields → metadata.extracted.*; empty title/author noise is dropped.
• metadata.custom and other arbitrary non-system JSON metadata are preserved under metadata.custom.

Already-normalized v2 metadata is idempotent: repeated dry-runs or apply runs should report no additional updates after the first successful apply.

Dry-run first

Dry-run is the default and performs metadata-only scans with with_payload=True and with_vectors=False.

uv run python scripts/backfill_qdrant_metadata_v2.py \
  --qdrant-url "$QDRANT_URL" \
  --collection-name "$COLLECTION_NAME"

For a smaller canary scan:

uv run python scripts/backfill_qdrant_metadata_v2.py --limit 100

Expected summary fields:

• checked: points read from Qdrant.
• would_update: points whose metadata would change.
• updated: points written; always 0 in dry-run mode.
• skipped: points with no metadata or already-normalized metadata.
• errors: points or update batches that failed transformation/write handling.

Apply explicitly

Only run apply after the dry-run count is expected and the application version that reads/writes metadata v2 is deployed.

uv run python scripts/backfill_qdrant_metadata_v2.py --apply

The script writes batches with Qdrant SetPayloadOperation, setting only the metadata key for selected point IDs. By default it waits for Qdrant write completion. Use --no-wait only during a controlled maintenance window where deferred write acknowledgement is acceptable.

Verify after apply

Run the dry-run again:

uv run python scripts/backfill_qdrant_metadata_v2.py

Success criteria:

• errors is 0.
• would_update is 0 or only reflects new legacy writes from an older worker still running during the migration.
• Search and document-list endpoints continue to return results with v2 metadata paths such as metadata.source.filename, metadata.location.page_number, and metadata.quality.ocr_used.

Qdrant index implications

Filter paths live on v2 payload keys. This backfill updates payload values so those v2 indexes can match older points. It does not rebuild or delete indexes.

If the collection was created before v2 indexes existed, run the collection setup or verification path that creates the current payload indexes before relying on filters over v2 fields. Keeping old v1 indexes temporarily is safe; they can be removed later only after:

1. all workers and API instances write v2 metadata,
2. this backfill reports no pending updates,
3. no dashboards, scripts, or saved filters query old paths such as page_number, ocr_used, or extraction_method, and
4. a rollback plan exists for restoring old indexes if an external consumer was missed.

Do not combine old path removal with this backfill. Treat index cleanup as a separate maintenance task so data transformation and query/index cleanup can be verified independently.