Document Upload API

The /embed endpoint accepts document uploads and processes them asynchronously through a Celery pipeline.

Endpoint

POST /embed

Request

Headers

Header	Required	Description
X-API-Key	Yes	Valid API key from API_KEY_CLIENTS env var

Form Data

Field	Type	Required	Description
file	File	Yes	Document file (PDF, DOCX, PPTX, PPT, XLSX, XLS, TXT, MD, TEXT, PNG, JPG, JPEG, TIF, TIFF)
document_id	string	Yes	Unique identifier for the document
metadata	JSON string	No	Custom metadata fields (e.g., {"department":"Engineering","author":["wanjia","adlin"]}). Most keys are stored under metadata.custom.* in Qdrant and are filtered using the explicit metadata.custom. prefix. version_id is a reserved ingestion key: it is stored as a top-level indexed payload field for version-scoped deletion, not under metadata.custom.*.

Supported File Types

Extension	MIME Type	Parser
.pdf	application/pdf	pypdfium2
.docx	application/vnd.openxmlformats-officedocument.wordprocessingml.document	python-docx
.pptx	application/vnd.openxmlformats-officedocument.presentationml.presentation	python-pptx
.ppt	application/vnd.ms-powerpoint	LibreOffice → python-pptx (opt-in)
.xlsx	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	openpyxl
.xls	application/vnd.ms-excel	LibreOffice → openpyxl (opt-in)
.txt	text/plain	Plain text
.md	text/markdown	Plain text
.text	text/plain	Plain text
.png	image/png	RapidOCR
.jpg	image/jpeg	RapidOCR
.jpeg	image/jpeg	RapidOCR
.tif	image/tiff	RapidOCR
.tiff	image/tiff	RapidOCR

Presentations emit one page per slide (slide number) and spreadsheets emit one page per visible worksheet (worksheet ordinal). Legacy binary .ppt/.xls uploads require OFFICE_LEGACY_CONVERSION_ENABLED=true and a LibreOffice (soffice) binary on the worker; when conversion is disabled they are rejected at accept time with HTTP 400 and error_code: "legacy_office_disabled", and when enabled but the binary is unavailable the job fails with a clear ingestion error.

Image uploads (.png/.jpg/.jpeg/.tif/.tiff) are parsed by OCR only. When OCR is disabled (RAPIDOCR_OCR_ENABLED=false) an image upload is rejected at accept time with HTTP 400 and error_code: "ocr_disabled" rather than being queued and later failing. When OCR is enabled but an image yields no text, the job completes with zero chunks (see the pipeline note below) instead of failing.

File Size Limits

• Maximum file size: 200MB
• Files exceeding this limit are rejected with HTTP 400 and error_code: "file_too_large"

Response

{
  "job_id": "abc123-def456-ghi789",
  "document_id": "my-document-001",
  "status": "pending"
}

Field	Description
job_id	Unique job identifier for status polling
document_id	The document ID you provided
status	Current status: pending

Processing Pipeline

When a document is uploaded:

1. Validation - File size and type validated
2. Parsing - Document parsed based on file extension; PNG/JPG/JPEG uploads always use RapidOCR, and PDF pages with sparse text fall back to OCR. A standalone image whose OCR yields no text has no native-text fallback, so the job completes with zero chunks (chunks_created: 0) rather than failing.
3. Chunking - Text split into overlapping chunks (default: 512 tokens, 50 overlap)
4. Classification selection - Publication mode selects and scores representative staged chunks before generation embedding; legacy mode does this after embedding
5. Entity Extraction - spaCy NER + regex-enhanced extraction captures persons, organizations, dates, locations, monetary amounts, account numbers, transaction references, and account types
6. Hybrid Embedding - FastEmbed generates dense (384-dim) and sparse (BM25) vectors
7. Storage/publication - Legacy mode stores vectors before its asynchronous category update; publication mode embeds classified chunks and activates only the verified immutable generation

Classification

Classification typically takes 2-4 seconds per document. It runs asynchronously after embedding in legacy mode and before generation embedding in publication mode. You can customize categories or disable classification through environment variables. See Configuration, Architecture: Post-Embedding Classification, and the Auto-Categorizing Your Documents guide.

Examples

Basic Upload

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@document.pdf" \
  -F "document_id=doc-001"

With Metadata

Custom metadata fields are preserved under metadata.custom.* in Qdrant and can be filtered in search using the explicit metadata.custom. prefix (e.g. metadata.custom.department). System metadata uses explicit dotted paths such as metadata.source.filename.

Accepted metadata must also satisfy the current versioned Metadata Discovery type and size contract. Validation happens before the upload is staged or queued, and a violation rejects the whole request with HTTP 422. See the Metadata and Entities Reference for eligible values, exact limits, path semantics, and the value-free diagnostic shape.

Use metadata.version_id to index a specific version of a document. Version ids are scoped to the caller's client and document_id; they do not need to be globally unique. If omitted or blank, the pipeline uses document_id as the default version id. The version_id value is stored as top-level payload version_id and is not copied into metadata.custom. It is validated like document_id — only alphanumeric characters, dash, underscore, and dot are allowed; any other value is rejected with 400 Bad Request.

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@resume.docx" \
  -F "document_id=candidate-123" \
  -F 'metadata={"department":"Engineering","team":"Backend","author":["wanjia","adlin"],"version_id":"v2"}'

Batch Embed Folder with Metadata

Use the just embed command or the embed_folder.py script to embed all documents in a folder with the same metadata:

# Using just
just embed test-documents/resume-dataset \
  --metadata '{"department":"Engineering","team":"Backend"}'

# Using the script directly
uv run python scripts/embed_folder.py test-documents/resume-dataset \
  --metadata '{"department":"Engineering","team":"Backend"}'

Shell Quoting

just embed now preserves quoted JSON arguments, so standard single-quoted JSON works the same way as the direct script command.

Multiple Files

for file in documents/*.pdf; do
  doc_id=$(basename "$file" .pdf)
  curl -X POST http://localhost:8000/embed \
    -H "X-API-Key: super-secret-key" \
    -F "file=@$file" \
    -F "document_id=$doc_id"
done

From Python

import requests

url = "http://localhost:8000/embed"
headers = {"X-API-Key": "super-secret-key"}

with open("document.pdf", "rb") as f:
    files = {"file": f}
    data = {
        "document_id": "my-doc-001",
        "metadata": '{"author": "Jane Smith", "department": "Engineering"}'
    }
    response = requests.post(url, headers=headers, files=files, data=data)
    print(response.json())
    # {'job_id': '...', 'document_id': 'my-doc-001', 'status': 'pending'}

Response Codes

Code	Description
200	Success - Document queued for processing
400	Bad Request - Invalid file type, file size, empty upload, document id, or metadata
401	Unauthorized - Invalid API key
422	Validation Error - Missing required form fields, invalid request shape, or a Metadata Discovery contract violation
429	Rate Limited - Too many requests

Error Responses

{
  "detail": "Invalid API key",
  "error_code": "auth_failed",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "File size exceeds maximum allowed (200MB)",
  "error_code": "file_too_large",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Unsupported file type",
  "error_code": "unsupported_format",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Image uploads require OCR, but OCR is disabled on this server (RAPIDOCR_OCR_ENABLED=false). Upload a text-based document instead, or ask the operator to enable OCR.",
  "error_code": "ocr_disabled",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Invalid metadata JSON",
  "error_code": "validation_error",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "metadata must be a JSON object",
  "error_code": "validation_error",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Reserved metadata keys are not allowed: classification, metadata.created_at, processing",
  "error_code": "validation_error",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

System and legacy metadata paths are read-only and rejected here. See the Metadata and Entities Reference for the authoritative group and reserved-key contract.

Checking Status

After upload, poll the status endpoint to track processing:

curl http://localhost:8000/status/JOB_ID \
  -H "X-API-Key: super-secret-key"

See Status API for details.

Best Practices

1. Use meaningful document IDs - Include version numbers if content changes: report-v1, report-v2
2. Add relevant metadata - Include fields you'll want to filter by in searches
3. Check status before searching - Follow the Status API and wait for search_ready: true on publication-backed jobs before including results in search.
4. Handle duplicates - Delete old versions before uploading new ones