Load Performance Evidence

This page records the repeatable methodology for issue #87. Run it against a UAT-like stack, not a developer hot-reload process.

Targets

Area	Target	Evidence command
Search without rerank	p95 < 100ms	scripts/benchmark_publication_default.py or measured /search samples
Search with rerank	p95 < 300ms when enabled	/search with rerank=true samples
Indexing throughput	documents/hour recorded	test_load.sh and load_performance_report.py
Large-document handling	large-document result recorded	Upload near configured max size in UAT
Resource sizing	CPU, memory, disk recorded	Measurements from the owned isolated deployment

Procedure

Start an isolated CPU-only deployment through scripts.qa_stack; do not use just prod-up or the shared development ports for this procedure. See the QA execution contract for resource admission, allocated ports, and credential ownership.

uv run python -m scripts.qa_stack up --clients load --max-file-size-mb 200

# Replace <project> with the project printed by qa_stack up.
DEPLOYMENT=.qa/<project>/deployment.json
export BASE_URL="$(jq -r '.api_base' "$DEPLOYMENT")"
API_KEY="$(cat "$(dirname "$DEPLOYMENT")/keys/load.key")"

uv run python -m scripts.qa_stack preflight --deployment "$DEPLOYMENT"
./.agents/skills/search-engine-test/scripts/test_load.sh "$API_KEY" test-documents/banking 5 3

Capture actual latency samples, throughput, large-document results, and resource measurements from that deployment. The report command below illustrates the input format; its sample numbers are not evidence from your run.

python scripts/load_performance_report.py \
  --search-latencies-ms 20,30,40 \
  --rerank-latencies-ms 120,180,240 \
  --documents-processed 6 \
  --elapsed-seconds 120 \
  --large-document-result pass \
  --cpu-percent 42.5 \
  --memory-percent 61.0 \
  --disk-percent 30.0

Replace the sample latency values with measured values from the run. Link the JSON output from the pull request or release checklist for go-live sign-off.

After retaining the run's evidence, remove only its owned deployment:

uv run python -m scripts.qa_stack down --deployment "$DEPLOYMENT"

The report is also a readiness gate. It exits 0 only when all required evidence is present and passing; otherwise it exits 1 and sets:

{
  "go_live_ready": false,
  "evidence_status": "fail",
  "evidence_failures": [
    "search latency samples missing",
    "rerank latency samples missing",
    "throughput evidence missing",
    "large document evidence missing or failed",
    "resource sizing evidence missing"
  ]
}

Both search and rerank latency sections fail when samples are missing, not just when p95 exceeds the target. Treat a missing-evidence failure the same as a measured target failure until the UAT packet includes real samples.

Deviation Handling

If p95 search exceeds target, record whether rerank was enabled, current Qdrant collection size, worker counts, CPU, memory, disk, queue depth, and resource sizing assumptions. If rerank p95 exceeds the target, keep rerank opt-in and document the decision.

Reclaimed publication-mode default budget

PR #256 shipped default publication Content search with deferred Manifest validation and a bounded 4x raw-candidate prefetch. On the representative seven-Document-Version, 78-active-chunk corpus, 20 warm-up requests followed by 100 measured requests produced p95 130ms with explicit Content scope and reranking disabled. The original implementation measured p95 192ms; the focused optimization prototype reached p95 123ms.

Follow-up ic-256-followup-default-path-sub100ms adds a per-Client cache of Qdrant-derived active-generation Manifest authority. Publication activation invalidates and refreshes the affected identity after the strong, waited Qdrant selection succeeds. A missing, mismatched, or malformed cache entry is never eligible evidence: the request falls back to the existing deterministic Manifest retrieval and identity-scoped uniqueness proof in Qdrant.

The same seven-Document-Version, 78-active-chunk default-path harness, with publications serialized through active settlement, produced server-reported p50 54ms, p95 69ms, p99 74ms, and max 78ms after 20 warm-up requests and 100 measured requests. Rerank was disabled and every request used explicit Content scope; wall-clock p95 was 77.995ms. The default publication-search budget is therefore restored to p95 below 100ms. Metadata and Both remain explicit per-query opt-ins.

The executable harness is scripts/benchmark_publication_default.py. Its raw 100-sample transcript records the code head, isolated Compose project and ports, serialized setup, request body, corpus shape, every server and wall-clock sample, and derived nearest-rank percentiles. To reproduce against an already-started isolated stack, set COMPOSE_PROJECT_NAME, QDRANT_PORT, and REDIS_PORT to its recorded project identity and allocated host ports. Use its BASE_URL and owned API_KEY; write new evidence locally rather than overwriting the historical transcript:

SEARCH_TEST_API_KEY="${API_KEY:?set the isolated Client key}" uv run python \
  scripts/benchmark_publication_default.py \
  --compose-project "${COMPOSE_PROJECT_NAME:?set the isolated project}" \
  --api-base "${BASE_URL:?set the isolated API URL}" \
  --qdrant-port "${QDRANT_PORT:?set the allocated Qdrant port}" \
  --redis-port "${REDIS_PORT:?set the allocated Redis port}" \
  --setup \
  --output .qa/evidence/publication-default.json

The harness rejects the shared project name and shared host ports before it uploads or measures anything.