Publication Load Release Gate
Issue #195 adds the hooks for this test. It does not claim production performance. Issue #203 must execute this fixed plan against the production adapter before cutover.
Generate the machine-readable plan without running or fabricating measurements:
uv run python scripts/publication_load_plan.py \
--output publication-load-plan.json
The output is marked evidence_status: not_run until the release-gate runner records
real evidence.
When the production-adapter runner has recorded the complete measurements, validate them against the exact release-candidate commit:
uv run python scripts/validate_publication_release.py \
--input publication-load-evidence.json \
--output publication-release-verdict.json \
--git-head "$(git rev-parse HEAD)"
The input must use schema version 1, set evidence_status to pass, and name
the same Git head. It must contain all 324 unique primary runs, all 632 quality
outcomes, positive per-tier Qdrant point/byte and Redis byte measurements, the
fixed runtime conditions, and every hard-safety count. A missing, stale,
incomplete, below-threshold, or over-budget field exits nonzero.
The atomic verdict records only the input SHA-256, evaluated and requested Git heads, bounded gate results, bounded failure reasons, and a timestamp. It never copies Client IDs, Custom Metadata, raw samples, or query results from the evidence file. Keep both files as immutable release artifacts.
Fixed corpus and quality contract
Use seed 19219083. Four equal tiers contain 1,000 active target Document Versions
with 8, 16, 32, and 64 discoverable Metadata Leaves per version: 4,000 target versions,
120,000 Metadata Leaf records, and 8,000 Content chunks. Preserve the 632 query outcomes:
| Query contract | Outcomes | Required pass rate |
|---|---|---|
| Exact Metadata lookup at rank 1 | 120 | 100% |
| Lexical Metadata lookup in top 3 | 120 | 95% |
| Semantic Metadata lookup in top 10 | 120 | 90% |
| Content retrieval | 120 | 100% |
| Both-scope retrieval | 120 | 100% |
| Eight four-tier adversarial controls | 32 | 100% |
The adversarial controls cover path-only matches, wrong Client, non-authoritative generation, soft-delete defaults and include-deleted behavior, and cross-version isolation. Keep 100 staged or superseded generations per tier and the established wrong-Client and soft-deleted distractors.
Primary 108-cell matrix
Run all combinations three times in randomized order:
| Axis | Values |
|---|---|
| Metadata Leaves per Document Version | 8, 16, 32, 64 |
| Search scope | Content, Metadata, Both |
| Deployment and load | 2 workers / 16 concurrent; 4 / 64; 8 / 256 |
| Publication churn | idle 0/min; steady 12/min across 16 Clients; burst 60/min across 64 Clients for 2 minutes, then 8 minutes steady |
| Repetition | 5-minute warmup plus 15-minute measurement, repeated 3 times |
This is $4 \times 3 \times 3 \times 3 = 108$ measured cells and 324 runs. Each cell requires at least 10,000 completed search attempts. Use equal request counts per leaf tier so pooled results are not traffic-mix weighted.
Run with at least two independent ready search workers and third-worker join/replacement churn. Include Redis AOF rewrite load and ordinary Celery broker/result traffic. Report successful latency separately from expected affected-Client fail-closed 503 responses.
For each topology, concurrency, and churn profile, run a benchmark-only same-tier Content baseline with publication validation disabled. This baseline path must be unreachable in production.
Record raw samples and derive:
- per-tier p50, p95, p99, and p95 ratio to the same-tier/profile baseline;
- pooled p95 ratio from equal-sized raw samples across all four tiers and the matching pooled baseline, never an average of p95s;
- throughput, CPU, RSS, Redis and Qdrant utilization, pool wait, and event-loop lag;
- fence capture-to-all-ACK, activation, activation-to-all-converged, and affected-Client 503-window durations;
- ACK retries, lease expiries, join wait, reconciliation age, and stale/half result count;
- Both-scope branch skew and whole-request failure count.
Hard safety gates
Any run fails immediately if it observes:
- stale, non-authoritative, cross-Client, or half-published evidence in a successful response;
- anything other than one Redis selected-generation pointer and exactly one authoritative Qdrant Manifest for an identity;
- activation without every member of the immutable captured live-worker set ACKed;
- success from an expired or stale incarnation, or a changing, invalid, or boot-mismatched snapshot; apply the Search API publication contract when assessing pinned reads;
- candidate-lane payloads, Redis state, or process memory creating eligibility without an authoritative Qdrant Manifest.
Latency budgets
| Scope | Pooled and diagnostic per-tier p95 ratio budget |
|---|---|
| Content | <= 1.10x |
| Metadata | <= 1.25x |
| Both | <= 1.75x |
Pooled p95 remains a release gate. A pooled pass never overrides the per-tier rule.
The focused local rerun on the current implementation head exceeded all three budgets:
Content 1.10x, Metadata 1.25x, and Both 1.75x. That focused run is diagnostic rather
than production release evidence, but it leaves the latency gate open.
Per-tier service objectives become required when either condition occurs in a production-representative 4-worker/64-concurrency or 8-worker/256-concurrency profile:
- the same tier/scope ratio exceeds its budget in at least 2 of 3 runs and the bootstrap 95% confidence-interval lower bound is above that budget; or
- any run exceeds 1.05 times its budget: Content
>1.155x, Metadata>1.3125x, or Both>1.8375x.
The prior synthetic 64-leaf Both result was 1.9097x. It exceeds the 1.8375x
single-run magnitude threshold. Production evidence must disprove that magnitude or the
release contract must add explicit objectives by leaf tier and scope. Do not hide this
caveat behind a pooled pass.
After the primary matrix, run saturation sweeps at 2 workers/32 concurrency, 4/128, and 8/512. They locate the capacity knee; they do not relax any safety or latency gate.