Metadata and Entities Reference

The ingestion pipeline stores Metadata v2 and extracted entities on each indexed chunk. System metadata is written by the pipeline. Custom metadata is supplied by users through /embed and, when publication is enabled, /update. Search exposes this information in Evidence Results for each Document Version; the documents listing exposes it in its metadata response.

Metadata v2 groups

Group	Fields	Owner	Notes
metadata.schema_version	"2"	System	Identifies the public metadata schema version
metadata.source.*	filename, mime_type, extension	System	File identity and type. filename comes from the upload name and can be renamed with top-level filename on PATCH /update.
metadata.document.*	page_count	System	Document-level parse facts
metadata.location.*	page_number, page_label, chunk_index	System	Chunk location inside the source document
metadata.classification.*	status, categories	System	ModernBERT classification output
metadata.quality.*	text_extraction, ocr_used, warnings	System	Extraction method and OCR warnings
metadata.extracted.*	title, author, created_at	System	Optional parser-derived document properties
metadata.embedding.*	dense_model, dense_dimensions, sparse_model, embedded_at, embedding_version, dense_runtime, chunker, extraction	System	Optional search-vector provenance captured at embed time
metadata.custom	Any allowed user keys	User	Custom metadata from upload and update requests

System groups are read-only from the public API, with one documented exception: PATCH /update accepts a top-level filename field that writes metadata.source.filename on live points for that document_id and Client. Putting metadata.source.filename (or the source group) inside the custom metadata object is still rejected. User metadata also rejects the legacy parsing, processing, and ocr groups; keys starting with metadata. or entities.; and the server-owned payload roots metadata, entities, client_id, version_id, deleted, deleted_at, and deleted_by.

The optional embedding group records the model IDs, dense dimensions, FastEmbed runtime, chunker settings, extraction path, and pipeline version that produced the stored dense and sparse vectors. The indexed top-level embedding_version mirrors the detail group's version for maintenance selection. The query-time reranker and classification-only ModernBERT model are excluded because they do not produce the stored vectors. Documents indexed before pipeline versioning may expose only the historical dense_model, dense_dimensions, sparse_model, and embedded_at fields until they are re-embedded; missing detail is not inferred during reads.

Custom metadata

Custom metadata is nested under metadata.custom.* in Qdrant and must be filtered using the explicit metadata.custom. prefix. For example, uploading {"department":"Finance"} lets you search with "filters": {"metadata.custom.department": "Finance"}. Keys containing spaces or other supported special characters use the same prefix; see Keys with spaces (or other special characters) for the GET encoding example. To search eligible Custom Metadata values, see Searching Metadata.

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@test-documents/malaysia-wikipedia.txt" \
  -F "document_id=metadata-entities-001" \
  -F 'metadata={"department":"Engineering","team":"Backend","author":["wanjia","adlin"],"tenant_id":"tenant-a","project":"alpha"}'

Discoverable Metadata Value contract

Every Custom Metadata write through /embed or /update is checked against the current versioned Metadata Discovery contract before it is admitted. /embed validates the complete object before staging. /update applies its JSON Merge Patch to the stored Custom Metadata object, then validates that complete result in the asynchronous worker before republishing. A violation rejects the entire write; values and paths are never truncated, split, coerced, skipped to satisfy a limit, or downgraded to lexical-only discovery.

The canonical Custom Metadata remains unchanged. The contract separately derives typed discovery leaves as follows:

• A nested string or finite JSON number is one eligible leaf.
• Every item in a homogeneous string list or homogeneous finite-number list is an independent leaf at the list's path.
• Objects are traversed recursively, but object containers do not produce leaves themselves. Booleans, null, empty lists, and mixed or otherwise unsupported lists produce no discovery text; their canonical values remain stored.
• Paths are arrays of exact JSON object-key strings. Dots, slashes, empty strings, and other key characters stay literal; there is no path escape syntax.
• Repeated occurrences with the same path, type, and value count once. Numerically equivalent JSON numbers at one path also count once.
• Derived leaves are deterministic and carry discovery version 2, a dense model signature for BAAI/bge-small-en-v1.5 with 384 dimensions, and the sparse metadata model ifc:metadata:trigram:v1.

These rules and limits determine whether values are eligible for Metadata Discovery. They do not change the existing exact-match Custom Metadata filter behavior.

Current hard limits

The current Metadata Discovery contract uses version 2. These limits are fixed across clients and deployments. Byte counts use UTF-8; canonical and path sizes use compact JSON. The caps are registered as METADATA_DISCOVERY_V2_LIMITS in app/utils/metadata_discovery.py; a future limit-set revision registers a new version instead of changing the bounds already used for published leaves.

Limit	Maximum
Complete canonical Custom Metadata object	65,536 bytes
Nested object path depth	16 string segments
Deduplicated discovery leaves	64 leaves
One path segment	128 bytes
Compact JSON string-segment-array path	768 bytes
One rendered string or number leaf	2,048 bytes

Non-finite numbers and text or path segments that cannot be encoded as UTF-8 are also rejected.

Contract errors

An /embed contract violation returns HTTP 422 with error_code: "metadata_discovery_limit". The details object contains only a stable code, the offending string-segment path, numeric observed and limit fields, and a unit. It never contains a Custom Metadata value.

{
  "detail": "Metadata Discovery contract violation",
  "error_code": "metadata_discovery_limit",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e",
  "details": {
    "code": "metadata_rendered_leaf_bytes_limit",
    "path": ["profile", "display_name"],
    "observed": 2049,
    "limit": 2048,
    "unit": "utf8_bytes"
  }
}

For /update, a violation in the merged result is reported by the asynchronous job on GET /status/{job_id} in the value-free error_details fields (see the Status API).

The six hard-limit detail codes are metadata_canonical_json_bytes_limit, metadata_path_depth_limit, metadata_leaf_count_limit, metadata_path_segment_bytes_limit, metadata_path_json_bytes_limit, and metadata_rendered_leaf_bytes_limit. Canonical-value failures use metadata_non_finite_number or metadata_utf8_encoding.

Entity fields

Entity extraction runs during embedding. spaCy supplies people, organizations, dates, and locations; regex patterns add finance-oriented fields.

SPACY_NER_ENABLED defaults to true. When it is disabled, the embedding task emits the configured entity keys with empty lists.

Entity key	Source	Example values
persons	spaCy PERSON	"Jane Doe"
organizations	spaCy ORG	"Acme Bank"
dates	spaCy DATE / TIME	"January 2026"
locations	spaCy GPE / LOC	"Kuala Lumpur"
monetary_amounts	Regex	"MYR 10,000", "$500.00"
account_numbers	Regex	IBANs, card-like numbers, account-number phrases
transaction_refs	Regex	"TXN-ABC123456", SWIFT message refs
account_types	Regex	"savings account", "mortgage", "line of credit"

Entity filters use the entity key directly and support a single string or a list of strings. Lists match any value within that field; multiple filter keys are combined with AND logic.

curl -X POST http://localhost:8000/search \
  -H "X-API-Key: super-secret-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "capital city",
    "scope": "content",
    "filters": {
      "locations": ["Malaysia", "Kuala Lumpur"],
      "metadata.custom.department": "Engineering"
    }
  }'

The exact filters constrain the candidate set but do not affect rank. The response uses the Evidence Result contract: rows are Document Versions with canonical metadata, lane-local retrieval contributions, and bounded Content/Metadata Evidence Sets. meta.total counts Document Versions rather than chunks or metadata leaves.

Illustrative response metadata:

{
  "meta": {
    "query": "capital city",
    "total": 10,
    "offset": 0,
    "limit": 5,
    "has_more": true,
    "query_time_ms": 30,
    "retrieval": {
      "alpha": 0.5,
      "reranked": false,
      "reranked_count": 0
    },
    "applied_filters": {
      "locations": [
        "Malaysia",
        "Kuala Lumpur"
      ],
      "metadata.custom.department": "Engineering"
    }
  }
}

Related docs

• Search Filters explains filter logic and supported paths.
• Search API shows request and response shapes.
• Update API explains how to change custom metadata after upload and how to rename metadata.source.filename.