Document Upload API

The /embed endpoint accepts document uploads and processes them asynchronously through a Celery pipeline.

Endpoint

POST /embed

Request

Headers

Header	Required	Description
X-API-Key	Yes	Valid API key from API_KEY_CLIENTS env var

Form Data

Field	Type	Required	Description
file	File	Yes	Document file (PDF, DOCX, PPTX, PPT, XLSX, XLS, TXT, MD, TEXT, PNG, JPG, JPEG, TIF, TIFF)
document_id	string	Yes	Unique identifier for the document
metadata	JSON string	No	Custom metadata fields (e.g., {"department":"Engineering","author":["wanjia","adlin"]}). Most keys are stored under metadata.custom.* in Qdrant and support filtering. version_id is a reserved ingestion key: it is stored as a top-level indexed payload field for version-scoped deletion, not under metadata.custom.*.

Supported File Types

Extension	MIME Type	Parser
.pdf	application/pdf	PyMuPDF
.docx	application/vnd.openxmlformats-officedocument.wordprocessingml.document	python-docx
.pptx	application/vnd.openxmlformats-officedocument.presentationml.presentation	python-pptx
.ppt	application/vnd.ms-powerpoint	LibreOffice → python-pptx (opt-in)
.xlsx	application/vnd.openxmlformats-officedocument.spreadsheetml.sheet	openpyxl
.xls	application/vnd.ms-excel	LibreOffice → openpyxl (opt-in)
.txt	text/plain	Plain text
.md	text/markdown	Plain text
.text	text/plain	Plain text
.png	image/png	RapidOCR
.jpg	image/jpeg	RapidOCR
.jpeg	image/jpeg	RapidOCR
.tif	image/tiff	RapidOCR
.tiff	image/tiff	RapidOCR

Presentations emit one page per slide (slide number) and spreadsheets emit one page per visible worksheet (worksheet ordinal). Legacy binary .ppt/.xls uploads require OFFICE_LEGACY_CONVERSION_ENABLED=true and a LibreOffice (soffice) binary on the worker; when disabled or unavailable, they are rejected with a clear error.

File Size Limits

• Maximum file size: 200MB
• Files exceeding this limit are rejected with HTTP 400 and error_code: "file_too_large"

Response

{
  "job_id": "abc123-def456-ghi789",
  "document_id": "my-document-001",
  "status": "pending"
}

Field	Description
job_id	Unique job identifier for status polling
document_id	The document ID you provided
status	Current status: pending

Processing Pipeline

When a document is uploaded:

1. Validation - File size and type validated
2. Parsing - Document parsed based on file extension; PNG/JPG/JPEG uploads always use RapidOCR, and PDF pages with sparse text fall back to OCR
3. Chunking - Text split into overlapping chunks (default: 512 tokens, 50 overlap)
4. Entity Extraction - spaCy NER + regex-enhanced extraction captures persons, organizations, dates, locations, monetary amounts, account numbers, transaction references, and account types
5. Hybrid Embedding - FastEmbed generates dense (384-dim) and sparse (BM25) vectors, stored in Qdrant
6. Classification - After all embedding batches complete, ModernBERT ONNX selects representative chunks via MMR and assigns categories
7. Category Update - The uploaded document version's chunks in Qdrant are updated with classification results

Classification

Document classification runs asynchronously after embedding completes and typically takes 2-4 seconds per document. You can customize categories or disable it entirely via environment variables. See Configuration, Architecture: Post-Embedding Classification, and the Auto-Categorizing Your Documents guide.

Examples

Basic Upload

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@document.pdf" \
  -F "document_id=doc-001"

With Metadata

Custom metadata fields are preserved under metadata.custom.* in Qdrant and can be filtered in search. Short keys belong to user metadata; system metadata uses explicit dotted paths such as metadata.source.filename.

Use metadata.version_id to index a specific version of a document. Version ids are scoped to the caller's client and document_id; they do not need to be globally unique. If omitted or blank, the pipeline uses document_id as the default version id. The version_id value is stored as top-level payload version_id and is not copied into metadata.custom. It is validated like document_id — only alphanumeric characters, dash, underscore, and dot are allowed; any other value is rejected with 400 Bad Request.

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@resume.docx" \
  -F "document_id=candidate-123" \
  -F 'metadata={"department":"Engineering","team":"Backend","author":["wanjia","adlin"],"version_id":"v2"}'

Batch Embed Folder with Metadata

Use the just embed command or the embed_folder.py script to embed all documents in a folder with the same metadata:

# Using just
just embed test-documents/resume-dataset \
  --metadata '{"department":"Engineering","team":"Backend"}'

# Using the script directly
uv run python scripts/embed_folder.py test-documents/resume-dataset \
  --metadata '{"department":"Engineering","team":"Backend"}'

Shell Quoting

just embed now preserves quoted JSON arguments, so standard single-quoted JSON works the same way as the direct script command.

Multiple Files

for file in documents/*.pdf; do
  doc_id=$(basename "$file" .pdf)
  curl -X POST http://localhost:8000/embed \
    -H "X-API-Key: super-secret-key" \
    -F "file=@$file" \
    -F "document_id=$doc_id"
done

From Python

import requests

url = "http://localhost:8000/embed"
headers = {"X-API-Key": "super-secret-key"}

with open("document.pdf", "rb") as f:
    files = {"file": f}
    data = {
        "document_id": "my-doc-001",
        "metadata": '{"author": "Jane Smith", "department": "Engineering"}'
    }
    response = requests.post(url, headers=headers, files=files, data=data)
    print(response.json())
    # {'job_id': '...', 'document_id': 'my-doc-001', 'status': 'pending'}

Response Codes

Code	Description
200	Success - Document queued for processing
400	Bad Request - Invalid file type, file size, empty upload, document id, or metadata
401	Unauthorized - Invalid API key
422	Validation Error - Missing required form fields or invalid request shape
429	Rate Limited - Too many requests

Error Responses

{
  "detail": "Invalid API key",
  "error_code": "auth_failed",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "File size exceeds maximum allowed (200MB)",
  "error_code": "file_too_large",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Unsupported file type",
  "error_code": "unsupported_format",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Invalid metadata JSON",
  "error_code": "validation_error",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "metadata must be a JSON object",
  "error_code": "validation_error",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

{
  "detail": "Reserved metadata keys are not allowed: classification, metadata.created_at, processing",
  "error_code": "validation_error",
  "request_id": "5d7f4c1a2b9e4b3e8f6f0d2a4c9b8a1e"
}

System and legacy metadata paths are read-only and rejected here. See the Metadata and Entities Reference for the authoritative group and reserved-key contract.

Checking Status

After upload, poll the status endpoint to track processing:

curl http://localhost:8000/status/JOB_ID \
  -H "X-API-Key: super-secret-key"

See Status API for details.

Best Practices

1. Use meaningful document IDs - Include version numbers if content changes: report-v1, report-v2
2. Add relevant metadata - Include fields you'll want to filter by in searches
3. Check status before searching - Wait for completed status before including in search results
4. Handle duplicates - Delete old versions before uploading new ones