Skip to main content

Embedding Throughput

Use just bench-run against a running API to measure ingestion throughput for a document corpus:

just bench-run baseline test-documents/banking

The recipe writes bench-<label>.json with documents processed, chunks created, elapsed seconds, documents per minute, chunks per minute, and per-document upload/processing timings.

Compare two saved runs with:

just bench-compare bench-baseline.json bench-current.json

The production runtime generator emits PROD_EMBEDDING_OMP_THREADS, and the embedding workers pass it to OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, and MKL_NUM_THREADS. Keep Celery embedding concurrency at 1; scale replicas and confirm memory headroom before raising total embedding slots.

For go-live evidence, attach the JSON report, the .runtime/prod.env used for the run, and host CPU/RAM details from docs/benchmarks/environment.