Search & Retrieval Benchmarks
Test Dataset
6 banking SEC 10-K filings (2024) embedded with --max-pages 5, producing 76 chunks across 30 pages.
Command: just embed test-documents/banking --max-pages 5
Top-1 Query Accuracy
Each query targets a specific expected document. The benchmark records the top-ranked Evidence Result's Document Version from the live API; rank is the caller-facing ordering signal.
| Test Query | Expected Doc | Actual Document Version | Result | Latency |
|---|---|---|---|---|
| "item part financial related bank america" | bank_of_america_2024.pdf | bank_of_america_2024.pdf (unversioned) | PASS | 21ms |
| "registrant indicate check mark whether report" | citigroup_2024.pdf | citigroup_2024.pdf (unversioned) | PASS | 19ms |
| "banking global markets activities fees business" | goldman_sachs_2024.pdf | goldman_sachs_2024.pdf (unversioned) | PASS | 22ms |
| "securities exchange commission united states washington" | morgan_stanley_2024.pdf | morgan_stanley_2024.pdf (unversioned) | PASS | 22ms |
| "financial activities nature subsidiaries insurance securities" | wells_fargo_2024.pdf | wells_fargo_2024.pdf (unversioned) | PASS | 21ms |
| "employees compensation development performance management risk" | wells_fargo_2024.pdf | wells_fargo_2024.pdf (unversioned) | PASS | 22ms |
Accuracy: 6/6 (100.0%) top-1 recall
Search Latency
The alpha=0.7 hybrid row is a historical measurement from before the default changed to
0.5; it is retained for reproducibility, not as the current default.
| Metric | Mean | Median | Range |
|---|---|---|---|
| Hybrid (alpha=0.7) | 19.6ms | 16ms | 16-27ms |
| Dense-only (alpha=1.0) | 19.2ms | 17ms | 16-26ms |
| Sparse-only (alpha=0.0) | 8.8ms | 9ms | 7-11ms |
| Top-1 query set | 21.2ms | 21.5ms | 19-22ms |
| Total matches available | 20 per query | - | - |
Observations
- Sparse search was fastest at 8.8ms mean query time.
- Dense and hybrid search stayed under the 50ms target on the 76-chunk banking corpus.
- The formerly failing regulatory-language query now resolves to
morgan_stanley_2024.pdfin this fresh run. - All measured search modes remain suitable for interactive search without reranking.
Reproduce these benchmarks
# Seed data
just embed test-documents/banking --max-pages 5
# Run top-ranked Content queries
just search "item part financial related bank america" -- --scope Content
just search "banking global markets activities fees business" -- --scope Content
# Latency test with different alpha values
just search "financial results" -- --scope Content --alpha 0.7
just search "financial results" -- --scope Content --alpha 1.0
just search "financial results" -- --scope Content --alpha 0.0