Dedicated Client Storage Specification

Status: accepted design; not implemented. This is the repository copy of issue #100, published for implementation handoff. The agreed glossary and ADR-0012 are part of this handoff.

Problem Statement

Clients need separate document storage, indexes, backups, and recovery boundaries. Today they share physical publication collections, with access restricted by server-injected Client filters. This supports logical isolation but does not give operators an independent storage and recovery unit for each Client.

Operators need to migrate existing data without losing document versions, metadata, deletion state, vectors, or publication evidence. They also need daily off-host backups, complete Client-state restore, bounded backup retention, and a clear distinction between Operational Client Deletion and Permanent Client Erasure.

This specification replaces the deferred design sketch previously recorded in this issue. The revisit requirement is now confirmed, and ADR-0012 is accepted. The earlier proposals to remove Client filters, collapse document identity, provision on first upload, or retain shared placement as a default do not apply.

Solution

Give every Client four dedicated publication collections—Content, Fuzzy, Metadata, and Manifest—on shared Qdrant infrastructure. Preserve existing API behavior, server-resolved Client identity, and mandatory Client filters. Operators provision a Client before accepting its traffic and perform migration and restore during maintenance windows.

Provide operator-managed daily off-host Client Backups with 30-day retention. A Client Restore recovers complete application-visible document state and reconciles publication state before traffic resumes, without changing another Client's data. Operational Client Deletion removes live data and access while backups expire normally. Permanent Client Erasure also removes retained backups and requires explicit confirmation.

All Clients continue using the same embedding models and dimensions. Separate collections do not promise separate compute resources, disk bandwidth, credentials, or availability.

User Stories

1. As a Client, I want my documents stored separately from other Clients' documents, so that my storage boundary is independent of their data.
2. As a Client, I want all four publication lanes dedicated to me, so that Content, Fuzzy, Metadata, and Manifest evidence share the same ownership boundary.
3. As a Client, I want my API keys to resolve to the same Client store, so that issuing another key does not create another tenant.
4. As an operator, I want departments requiring isolation represented as separate Clients, so that document ownership remains unambiguous.
5. As a Client, I want to use a document ID also used by another Client, so that document identifiers do not require global coordination.
6. As a Client, I want existing upload and job-status behavior preserved, so that storage migration does not require consumer API changes.
7. As a Client, I want Content, Metadata, Both, and fuzzy-assisted retrieval confined to my documents, so that no search mode crosses Client boundaries.
8. As a Client, I want document listing, detail, version access, and custom metadata discovery confined to my documents, so that non-search reads remain isolated.
9. As a Client, I want metadata updates confined to my documents, so that a shared document ID cannot modify another Client's data.
10. As a Client, I want re-embedding confined to my documents, so that maintenance cannot replace another Client's vectors.
11. As a Client, I want Soft Delete and Restore to preserve version ownership and lifecycle state, so that storage placement does not change visibility rules.
12. As a Client, I want document Hard Delete confined to my document identity, so that deleting a document never deletes another Client's version.
13. As a Client, I want publication recovery to select only verified committed evidence, so that incomplete writes cannot become searchable.
14. As a Client, I want Metadata Discovery Readiness preserved through migration and restore, so that metadata search is neither silently degraded nor admitted prematurely.
15. As an operator, I want explicit Client placement configuration, so that I can determine which collections belong to a Client.
16. As an operator, I want collection naming to avoid collisions between Client identifiers, so that two configured Clients cannot share a store accidentally.
17. As an operator, I want provisioning to create and verify all four lanes and their indexes, so that a Client is ready before its traffic is admitted.
18. As an operator, I want unknown or incomplete placement to fail closed, so that configuration errors never route traffic to shared collections or another Client.
19. As an operator, I want repeated provisioning to preserve existing data and detect incompatible configuration, so that a rerun is safe.
20. As an operator, I want configured collection and index settings applied consistently, so that independent indexes remain compatible with the common embedding models.
21. As an operator, I want API and workers to use the same placement after coordinated restart, so that ingestion and retrieval agree on the storage boundary.
22. As an operator, I want global evidence verification and metadata backfill to cover all configured Clients, so that splitting collections does not hide incomplete or stranded state.
23. As an operator, I want to pause traffic and work before migration, so that copied evidence represents a consistent source state.
24. As an operator, I want migration to preserve document identities, versions, vectors, metadata, deletion state, and publication evidence, so that existing behavior survives cutover without re-ingestion.
25. As an operator, I want migration integrity checks beyond point counts, so that matching counts cannot conceal incorrect vectors or generation selection.
26. As an operator, I want a rehearsed rollback before production cutover, so that a failed migration can return to a verified source state.
27. As an operator, I want interrupted migration to have a safe restart or explicit recovery procedure, so that partial copies cannot be mistaken for a completed cutover.
28. As an operator, I want a Client Backup produced daily, so that the target recoverable data-loss window is no more than 24 hours.
29. As an operator, I want completed backups stored off-host, so that loss of the Qdrant host does not destroy the recovery copy.
30. As an operator, I want backup contents validated before they are advertised as recoverable, so that incomplete transfers or mismatched evidence cannot count as successful backups.
31. As an operator, I want each backup set associated with exactly one Client, so that restore and erasure have an explicit ownership boundary.
32. As an operator, I want complete Client-state restore rather than collection-only restore, so that restored documents and publication state agree.
33. As an operator, I want incompatible, corrupt, incomplete, or wrong-Client backups rejected before destructive restore steps, so that invalid inputs cannot replace good data.
34. As an operator, I want queued and running work handled during restore, so that old tasks cannot overwrite restored state after traffic resumes.
35. As a Client, I want another Client's restore to leave my data unchanged, so that recovery does not roll back unrelated documents.
36. As an operator, I want failed restore to leave traffic fail-closed with a recovery procedure, so that partially restored data cannot be served as complete.
37. As an operator, I want a representative restore rehearsal to measure recovery time, so that the one-business-day target is supported by evidence.
38. As an operator, I want backups to expire after 30 days, so that historical data does not remain indefinitely.
39. As an operator, I want incomplete or failed backup attempts distinguished from valid recovery sets, so that retention and restore cannot treat them as successful backups.
40. As an operator, I want Operational Client Deletion to revoke access and remove live data while retaining backups until expiry, so that normal offboarding preserves the agreed recovery window.
41. As an operator, I want Permanent Client Erasure to require explicit Client-scoped confirmation, so that an irreversible action cannot be triggered accidentally.
42. As an operator, I want Permanent Client Erasure to remove retained backup copies as well as live data, so that deleting collections alone is not reported as complete erasure.
43. As an operator, I want old keys and queued tasks unable to recreate deleted Client data, so that completed deletion remains effective.
44. As an operator, I want deletion, erasure, and retention to leave other Clients' live data and backups unchanged, so that destructive operations remain scoped.
45. As an operator, I want commands to identify incomplete work and required recovery steps, so that partial operational failures do not produce a false success.
46. As an operator, I want documented provisioning, backup, restore, migration, rollback, and deletion procedures, so that another operator can safely run the system.

Implementation Decisions

• Client is the document ownership and isolation boundary defined by the domain glossary. API keys identify Clients; they do not define new tenants. Document identity remains the pair of Client and document ID.
• Use dedicated collections for every Client, not an opt-in hybrid shared/dedicated policy. Keep one shared Qdrant infrastructure deployment. Collection count grows by four per Client.
• Evolve the existing centralized publication placement resolver rather than introduce separate routing conventions. All ingestion, retrieval, lifecycle, maintenance, recovery, and operator paths must resolve placement consistently.
• Placement is server-owned configuration. User requests and metadata cannot supply Client identity or choose collection names. Preserve reserved-key validation, job-to-Client binding, and mandatory Client filters as defense in depth.
• Use stable, collision-resistant collection placement. Configuration must reject overlapping collection ownership and incompatible lane schemas before traffic admission. No unknown or missing Client placement may fall back to a shared collection.
• Provision through operator commands before accepting traffic. Verify all four lane collections, named vector schemas, and required payload indexes. Provisioning is repeatable and must not silently replace incompatible collections or existing data.
• Operators manage configuration and coordinate API/worker restarts. Dynamic onboarding and customer self-service are not required. Existing public endpoint contracts stay unchanged.
• All Clients use the same configured dense and sparse embedding models and dimensions. Separate collection/index settings must remain compatible with that common model configuration; document and query embeddings continue using the same model space.
• Make startup checks, global publication verification, recovery, index management, metadata backfill, readiness convergence, and operational scans aware of all configured Client placements. A single shared Manifest scan or global provisioning flag cannot remain the authority for all stores.
• Preserve the publication architecture: Qdrant Manifest evidence is the sole read-time admission authority; Redis journals and fences publication. Preserve immutable generations, source visibility when republishing, authority-before-store deletion ordering, and pinned-snapshot request semantics from ADR-0011.
• Discovery Readiness remains worker/operator-owned. Requests and startup do not write readiness. Non-ready Metadata/Both requests remain explicit readiness failures; unreadable operational state remains an operational failure.
• Migrate existing data in a maintenance window with producers, ingestion, mutations, queued/running work, and publication operations brought to a verified consistent boundary. A brief search outage is acceptable. Preserve existing vectors rather than require re-ingestion.
• Verify migrated lane contents and publication commitments, not just counts. Exercise API-visible versions, search, and lifecycle state before cutover. Retain a verified rollback source through the cutover decision; the final runtime must not retain a shared-placement fallback. Remove obsolete data according to the accepted erasure and retention contract, not an indefinite compatibility path.
• Extend the existing publication backup and recovery facilities instead of building a parallel backup stack. Existing facilities provide checksummed four-lane backup validation and maintenance acknowledgements, but complete Client-state restore must reconcile the restored document evidence with publication and readiness state.
• Backup sets must be independently attributable and removable per Client. Avoid mixed-Client backup packaging that would prevent one Client's Permanent Client Erasure without destroying another Client's recovery data.
• Operator configuration supplies the off-host destination and credentials. A local working copy or another volume on the same host is not the disaster-recovery copy. Daily scheduling and 30-day retention are part of this build, not deferred runbook suggestions.
• A completed Client Backup must represent a consistent document/publication state and pass integrity verification. Protect its control-plane contents and credentials through existing security conventions; do not expose them in command output.
• Client Restore may use a deployment-wide maintenance window. Prevalidate Client ownership, integrity, and compatibility before destructive steps. Reconcile publication state and readiness, exclude stale queued/running work, and prove restored evidence before reopening traffic. Restore must not roll back or mutate another Client's data.
• The restore contract concerns application-visible document state, not historical Celery task execution or byte-for-byte restoration of the entire deployment's Redis databases. Restore must nevertheless preserve isolation and prevent stale jobs from admitting or recreating obsolete document state.
• Operational Client Deletion removes access and live document/control state while retained backups expire normally. Permanent Client Erasure also removes all retained copies under the configured backup destination and requires explicit operator confirmation. Neither operation may allow stale work or old credentials to recreate removed data.
• Permanent erasure has no rollback after completion. Do not report completion while known retained copies remain or an underlying deletion failed. Backup destination policies must permit the agreed retention and erasure behavior; inaccessible or immutable copies are a reported prerequisite/failure, not silently excluded.
• Update deployment configuration, active user/operator documentation, and ADR status relationships with the cutover. ADR-0012 revisits ADR-0002's placement decision; it does not remove the logical Client isolation invariants.

Testing Decisions

• The user approved two highest-level acceptance seams: authenticated HTTP API and operator CLI against an isolated real stack. Read-only storage and backup-destination inspection supplements them where external responses cannot prove physical separation or removal. No new public testing endpoints are needed.
• Good tests assert observable results, isolation boundaries, state transitions, error behavior, and destructive-operation precedence. Do not assert private helper calls, hard-coded collection-name spelling, configuration forwarding, incidental wording, or mock echoes.
• API coverage exercises authentication, upload and job polling, every applicable search mode, listing/detail/version access, metadata update, re-embedding, Soft Delete, Restore, and document Hard Delete. Use at least two Clients with the same document ID and distinct content; use two keys belonging to one Client as well.
• Read-only evidence proves each Client owns four distinct physical collections and that no other Client's records enter them. Reject missing placement, collisions, incompatible provisioning, and unprovisioned Clients without shared-store fallback or partial admission.
• Exercise asynchronous work through real workers and poll public job outcomes. Restart API/workers and interrupt publication work to prove placement consistency, recovery, and immutable-generation visibility. Retain existing fencing, Client isolation, and readiness behavior rather than testing only happy-path search.
• Operator acceptance exercises provisioning, migration, verified cutover, and rollback, with old and new versions, metadata, soft-deleted records, and existing vectors. Compare application-visible outcomes and immutable evidence, not only point counts.
• Backup/restore acceptance uses real Redis and Qdrant plus an isolated off-host-equivalent destination. Create Client A and B, back up A, change both, restore A, and verify A's backed-up document state while B retains its newer state. Prove search and lifecycle behavior after reopening traffic, not merely successful snapshot upload.
• Negative restore cases include wrong Client, missing lane/control evidence, checksum corruption, incompatible vector schema, interrupted restore, and stale queued work. Preconditions must reject before destructive mutation where possible; interrupted mutation must remain fail-closed until recovery completes.
• Scheduling and retention tests use controlled time rather than waiting for days. Verify daily invocation, valid versus incomplete backup sets, 30-day expiry boundaries, and isolation of another Client's backups. Verify a backup becomes recoverable only after off-host transfer and validation complete.
• Deletion acceptance proves revoked access and removal of live state. Operational deletion retains recoverable backups until expiry. Permanent erasure without explicit confirmation must not mutate data; confirmed erasure removes all owned retained copies and prevents stale tasks from recreating data. Inject a failed removal and require failure rather than false success.
• Prefer existing Client-isolation API scenarios, publication stack fault harness, real-stack recoverability coverage, and backup checksum/compatibility/maintenance tests as prior art. Adapt broken behavioral contracts to dedicated placement; do not duplicate suites at every internal layer. Add narrow lower-level tests only for boundaries not deterministically provable at the acceptance seams.
• Every stack run uses a distinct Compose project, separate volumes, and non-conflicting host ports; never mutate or bind the shared default stack. Acceptance includes running the actual API and operator commands, not only mocked tests.
• Record a representative restore rehearsal's corpus size, backup age, duration, and success criteria. At most 24 hours of recoverable data loss and restoration within one business day are targets, not proven guarantees until the rehearsal supports them.

Out of Scope

• Different embedding models or vector dimensions per Client, dynamic model loading, and model migration.
• Dedicated Qdrant instances, separate resource/availability guarantees, or hard noisy-neighbor isolation.
• Online migration or restore without deployment maintenance windows.
• Customer-facing provisioning, backup, restore, retention, or deletion APIs/UI.
• Dynamic Client onboarding without coordinated configuration and restart.
• A parent/child customer or workspace domain model.
• Automatic expiry of live documents; 30-day retention applies to backups.
• Hybrid placement retaining shared collections as the normal runtime default.
• Rewriting search relevance, scoring, classification, OCR, or parsing behavior.
• Treating collection-only snapshots as complete recovery, or treating live collection deletion alone as permanent erasure.

Further Notes

• Accepted scope comes from the design interview and ADR-0012. Use the glossary terms Client, Client Backup, Client Restore, Operational Client Deletion, and Permanent Client Erasure throughout implementation and operator documentation.
• Source understanding was refreshed after fast-forwarding main to d4948b2. The accepted glossary and ADR were written locally during this session and must be included in the implementation change; their presence in the remote branch is not assumed.
• Preliminary estimate: 15–22 engineer-days of work; planning budget 18–25 engineer-days including contingency, approximately 4–5 working weeks for one engineer familiar with the repository. This is an estimate, not measured implementation time.
• Production corpus size, storage throughput, and the configured off-host destination determine actual copy and restore duration. Operators supply destination credentials at deployment; no secrets belong in this issue.
• Daily backup consistency must account for publication fencing and ingestion quiescence using the existing maintenance contract. Do not advertise the agreed recovery objectives while backup failures or an inconsistent transfer leave the newest usable set outside the target window.
• Issue #203 contains related recoverability work and existing proof patterns. Reuse that work; this issue adds dedicated Client placement and the complete scoped operations contract, not a second independent recovery architecture.
• Done means every story's externally observable behavior is delivered and the API/CLI acceptance evidence is recorded. A routing-only change or an untested backup/restore scaffold is not completion.