Scaling for High Traffic

This guide explains how the InfoConnect Hybrid Search Engine handles increasing load. Whether you're onboarding 10 users or 10,000, understanding scaling helps you plan infrastructure costs, predict performance, and know when to invest in more capacity.

Think of scaling as staffing a restaurant. You need enough waiters (API workers) to take orders, enough cooks (embedding workers) to prepare the food, a big enough pantry (Qdrant) to store ingredients, and a reliable ticket system (Redis) to coordinate everything. This guide shows you how to size each part of the kitchen for your expected crowd.

When Should You Read This Guide?

You should review this guide if any of these situations apply to your deployment:

• You are planning a production launch and need to know what server size to provision
• Users are reporting slow searches or timeouts during peak usage hours
• Document uploads are backing up and taking longer than expected to process
• You are growing from a pilot to full rollout and need to estimate infrastructure costs
• Your team wants to understand the trade-offs between speed, capacity, and hardware cost

Scaling is not just an engineering concern. It directly affects user experience, operational cost, and the pace at which you can onboard new documents. A PM who understands these trade-offs can make better decisions about feature prioritization and rollout timing. You do not need to understand every technical detail, but you should know which questions to ask and what the answers mean for your product roadmap.

Architecture Overview for Scaling

Think of the system as a restaurant kitchen. The API is the waiter taking orders, workers are the cooks, Redis is the order ticket system, and Qdrant is the pantry. Each can be staffed up independently. When you understand this flow, you can identify which part of the system is the bottleneck and scale exactly that component rather than throwing more hardware at everything.

The InfoConnect Hybrid Search Engine runs as a multi-service Docker Compose deployment with dedicated workers for CPU-bound and I/O-bound tasks.

The seven services and their scaling characteristics:

Service	Role	Bottleneck	Scales By
api	HTTP requests	CPU / connections	Uvicorn workers
worker-embedding-1	NER, dense+sparse embedding, upsert	CPU / memory	Horizontal replicas
worker-embedding-2	Same as embedding-1	CPU / memory	Horizontal replicas
worker-preprocessing-1	Parse, chunk, classify	CPU / memory	Horizontal replicas
redis	Broker, results, state	Memory	Vertical only (single node)
qdrant	Vector storage	CPU / disk I/O	Vertical only (single node)
models	Volume mount for ONNX/spaCy artifacts	Disk	Pre-warmed at build time

What This Means in Practice

The system has two scaling levers: add more workers (for document processing throughput) and add more API workers (for handling more simultaneous users). The database (Qdrant) and task queue (Redis) currently scale vertically — meaning you give them a bigger machine rather than adding more machines. See Future Directions for plans to change this.

How to read the table: The "Scales By" column tells you how to grow each service. "Horizontal replicas" means you can add more identical copies. "Uvicorn workers" means you can configure more processes inside the API container. "Vertical only" means the only option is a bigger server — for now. The "Bottleneck" column tells you what resource runs out first, which helps you decide what size server to provision. If you are unsure which service to scale, start with the one whose bottleneck matches your observed problem: slow searches point to Qdrant or API workers, and slow uploads point to embedding or preprocessing workers.

Why Two Embedding Worker Services?

The docker-compose.yml declares two identical embedding worker services so that docker compose up --scale can distribute replicas across Docker's default scheduler. Both pull from the same embedding queue.

Production Auto-Tuning

The InfoConnect Hybrid Search Engine includes a built-in sizing tool that reads your server's hardware and automatically configures the right number of workers. You don't need to guess — run one command and the system adapts. This means your engineering team can deploy to a new server without manually tuning configuration files.

The just prod-up command automatically sizes workers to your host hardware before starting containers.

How Auto-Tuning Works

Terminal

just prod-up

This runs scripts/generate_prod_runtime.py, which inspects host CPU and RAM, then writes .runtime/prod.env with tuned values. Inspect the generated plan:

Terminal

cat .runtime/prod.env

Auto-Tuning Formulas

Parameter	Formula	Bounds
api_workers	cpu_count // 8	min 1, max 4
embedding_total_slots	min(cpu//2, memory_budget//3072)	min 1, max 8
embedding_concurrency	1 (fixed)	always 1
embedding_omp_threads	1 (fixed)	passed to OpenMP/BLAS thread caps
preprocessing_replicas	1 (fixed)	always 1
preprocessing_concurrency	2 (fixed)	always 2

Where memory_budget = total_ram_mb - 1024 (1GB reserved for the OS).

Practical Impact

On a typical 8-core / 16GB server, auto-tuning produces 1 API worker and 2 embedding workers with ~5GB each. On a 32-core / 64GB server, it produces 4 API workers and 8 embedding workers. The key takeaway: bigger servers process documents faster, and the system configures itself accordingly.

Example Generated Plan

On the 2026-06-11 local production-stack run, the host reported 16 CPU cores and 80,279 MB RAM:

.runtime/prod.env

PROD_RUNTIME_GENERATED_AT=2026-06-11T08:32:06+00:00
PROD_HOST_CPU_COUNT=16
PROD_HOST_MEMORY_MB=80279
PROD_API_WORKERS=2
PROD_EMBEDDING_TOTAL_SLOTS=8
PROD_EMBEDDING_CONCURRENCY=1
PROD_EMBEDDING_REPLICA_1_SCALE=4
PROD_EMBEDDING_REPLICA_2_SCALE=4
PROD_EMBEDDING_CPU_PER_WORKER=1
PROD_EMBEDDING_OMP_THREADS=1
PROD_EMBEDDING_MEMORY_PER_WORKER=2560M
PROD_PREPROCESSING_TOTAL_SLOTS=2
PROD_PREPROCESSING_CONCURRENCY=2
PROD_PREPROCESSING_REPLICAS=1

Validate Before Deploying

Always run cat .runtime/prod.env after just prod-up to confirm the auto-tuner did not oversubscribe your host. If it did, override with explicit environment variables in .env.

Scaling Workers Horizontally

Workers are the engine that processes documents. When you need to ingest more documents faster, you add more workers — like hiring more cooks in a kitchen. Each worker type handles a different stage of the pipeline. Understanding the difference between embedding and preprocessing workers helps you decide where to invest when document processing is the bottleneck.

Embedding Workers

Embedding workers are the throughput bottleneck for document ingestion. Scale them with Docker Compose:

Terminal

docker compose up -d --scale worker-embedding-1=4 --scale worker-embedding-2=4

Each embedding worker is configured with:

Setting	Value	Why
Concurrency	1	Prevents model memory duplication in the same process
Memory	2GB	Sufficient for FastEmbed + spaCy model resident set
CPU	1	One core per embedding worker in the generated production runtime plan
max_tasks_per_child	250	Recycles worker after 250 tasks to control memory growth
max_memory_per_child	1.5GB	Hard recycle threshold
PROD_EMBEDDING_OMP_THREADS	1	Caps OpenMP, OpenBLAS, and MKL threads inside each embedding worker

What This Means

Each embedding worker uses about 2GB of RAM because it loads AI models into memory. You can't run multiple tasks in one worker (they'd fight over memory), but you can run many workers in parallel. Think of it as hiring specialists — each one works on one document at a time, but they all work simultaneously.

Capacity planning: A single embedding worker processes roughly 2-5 documents per minute depending on size. Four workers quadruple that throughput.

Do Not Increase Embedding Concurrency

Keep PROD_EMBEDDING_CONCURRENCY=1. Each worker loads the dense embedding model, sparse embedding model, and spaCy NER model into RAM. Running multiple concurrent tasks in the same process would duplicate this memory per task, quickly exhausting the container's 2GB limit.

The runtime plan also sets PROD_EMBEDDING_OMP_THREADS=1, which Docker Compose forwards as OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, and MKL_NUM_THREADS. Raising these thread caps can oversubscribe CPU when multiple embedding replicas run at once; change them only with benchmark evidence.

Preprocessing Workers

Preprocessing handles parsing, OCR fallback, chunking, and the post-embedding ModernBERT ONNX classification callback.

Terminal

docker compose up -d --scale worker-preprocessing-1=3

Setting	Value	Why
Concurrency	2	Parsing and chunking are I/O-bound; two tasks fill CPU wait time
Memory	6GB	Accommodates OCR model, PyMuPDF, and ModernBERT ONNX
max_tasks_per_child	500	Preprocessing is less memory-leaky than embedding
max_memory_per_child	4GB	Hard recycle threshold for preprocessing

Task Routing

Celery routes tasks to separate queues so you can scale each stage independently:

Queue	Tasks	Worker Type
preprocessing	process_document, parse, chunk, classify_document_task, reconcile_pending_classification	preprocessing workers
embedding	embed_nodes_task, ner_extraction	embedding workers

The pipeline starts on preprocessing, then enqueues embedding batches to the embedding queue. Classification callbacks and delayed reconciliation run on preprocessing; the authoritative trigger and recovery flow is documented in Architecture: Post-Embedding Classification.

Worker Lifecycle Settings

Celery worker recycling prevents memory leaks from long-running processes:

Worker Type	max_tasks_per_child	max_memory_per_child
Embedding	250	1.5GB
Preprocessing	500	4GB

These are set in app/tasks/worker.py and overridden by production environment variables where applicable.

API Server Tuning

The API server handles all incoming requests — searches, uploads, and status checks. Tuning it means ensuring it can handle enough simultaneous users without becoming a bottleneck. If users report slow searches or timeouts, this is usually the first place to look.

Uvicorn Workers

The API container runs Uvicorn with multiple workers:

Setting	Default	Max	Control
PROD_API_WORKERS	2	4	.runtime/prod.env or .env

The auto-tuner sets this to cpu_count // 8 with a ceiling of 4. For most workloads, 2 workers handle search traffic while Celery workers handle document processing.

No Resource Limits on API by Default

The api service in docker-compose.yml does not set mem_limit or cpus. For production, add explicit limits to prevent a runaway process from starving workers:

docker-compose.yml

services:
  api:
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 1G

CORS Configuration

CORS origins are controlled with CORS_ORIGINS:

.env

CORS_ORIGINS=["https://app.yourdomain.com"]

The API always enables credentialed CORS requests (allow_credentials=True) in app/main.py. There is no separate CORS_ALLOW_CREDENTIALS environment variable today.

Rate Limiting

The built-in rate limiter is a Redis-backed sliding window with a default of 30 requests per minute per resolved client id for protected routes. POST /search also consumes a search-specific per-client counter. See the Rate Limiting guide for details on algorithm behavior and scope.

Qdrant Optimization

Qdrant is the database that stores document embeddings — the mathematical representations that power semantic search. Its configuration affects how fast searches return results and how many documents you can store. If search slows down as your document collection grows, Qdrant is the component that needs attention.

Current Configuration

The Qdrant collection is configured as follows:

Parameter	Value	Notes
Mode	Single node	No clustering or replication
Shards	1	All data on one shard
On-disk payload	Enabled	Payload stored on disk while vectors stay in memory
Named vectors	text-dense (384-dim, COSINE) + text-sparse (BM25)	Hybrid retrieval

What This Means

Qdrant currently runs as a single instance with vectors in memory and payload stored on disk (on_disk_payload: true in the 2026-06-11 collection response). This is fast for search but means:

• Collection size is limited by available RAM — roughly 1M chunks per 4GB of RAM
• There is no redundancy — if the Qdrant container crashes, search is unavailable until it restarts
• Search stayed fast in the guide run — observed hybrid search examples returned in 12-30ms without reranking

Payload Indexes

By default, payload fields are indexed for filtering. The exact count depends on NER_ENTITY_KEYS configuration.

Filter Key	Indexed Payload Path	Type	Use Case
document_id	document_id	keyword	Exact document lookups
created_at	created_at	datetime	Time-range filtering
metadata.classification.categories	metadata.classification.categories	keyword	Category filtering
metadata.source.filename	metadata.source.filename	keyword	Filename lookup
metadata.location.page_number	metadata.location.page_number	integer	Page-level search
metadata.quality.text_extraction	metadata.quality.text_extraction	keyword	Extraction method tracking
metadata.quality.ocr_used	metadata.quality.ocr_used	bool	OCR audit
persons	entities.persons	keyword	Person entity filtering
organizations	entities.organizations	keyword	Organization entity filtering
dates	entities.dates	keyword	Date/time entity filtering
locations	entities.locations	keyword	Location entity filtering
monetary_amounts	entities.monetary_amounts	keyword	Money entity filtering
account_numbers	entities.account_numbers	keyword	Account-number filtering
transaction_refs	entities.transaction_refs	keyword	Transaction reference filtering
account_types	entities.account_types	keyword	Account-type filtering

Custom metadata fields are still filterable by key. When the hybrid collection is ensured, the service scans existing metadata.custom.* keys across the shared collection and creates payload indexes for discovered paths; brand-new custom keys become indexed after a later collection-ensure or verification pass.

Client Concurrency

QdrantService uses ThreadPoolExecutor(max_workers=4) for concurrent Qdrant operations. This means up to 4 Qdrant API calls can run in parallel per request context.

Scroll Pagination

Batch reads use scroll with a batch size of 256 points. This is used during classification when reading a document's embeddings back from Qdrant.

Redis Configuration

Redis is the coordination hub that connects the API to the workers. It holds task queues, processing status, and rate-limit counters. Its configuration affects system reliability. If Redis becomes unavailable, new document uploads cannot be queued and workers have nothing to process.

Memory and Persistence

Setting	Value	Effect
Version	7-alpine	Latest stable Redis
maxmemory	512mb	Hard memory ceiling
maxmemory-policy	noeviction	Keys are never evicted; write fails at limit
AOF	appendfsync everysec	Durable but not synchronous

What This Means

Redis is configured to use at most 512MB and will reject writes if it fills up. For context, 512MB can hold roughly 50,000 pending task results. If you process very large batches of documents simultaneously, you may need to increase this limit.

Noeviction Policy

With noeviction, Redis rejects writes once 512MB is exhausted. Monitor memory with INFO memory. If you see OOM errors, increase maxmemory or add result expiry.

Database Separation

Database	Purpose	Key Patterns
/0	Application state	Rate-limit counters, job status, embedding batch counters
/1	Celery broker	Task queues, routing
/2	Celery results	Task return values

Result Expiry

Celery results expire after 86400 seconds (24 hours). This prevents unbounded growth in DB /2.

Embedding Pipeline Tuning

The embedding pipeline converts raw text into searchable vectors. Tuning batch sizes and concurrency here determines how fast documents move through the system. When your users are waiting for documents to finish processing, these settings control how efficiently the system chews through the work.

Batch Sizes

Setting	Default	Effect
EMBEDDING_BATCH_SIZE	32	Texts fed to FastEmbed per batch
MAX_NODES_PER_TASK	50	Chunks grouped into one Celery task

A document with 200 chunks produces 4 embedding tasks (200 / 50 = 4), each processing up to 50 chunks. Within each task, FastEmbed processes texts in batches of 32.

Parallel Dense + Sparse

The embedder uses asyncio.gather() to run dense and sparse embedding in parallel:

# From app/services/embedder.py
dense_future = self._dense_model.embed(texts)
sparse_future = self._sparse_model.embed(texts)
dense_vectors, sparse_vectors = await asyncio.gather(dense_future, sparse_future)

NER Batching

spaCy processes chunks via nlp.pipe() with batch_size=50. This is more efficient than per-chunk processing because it amortizes pipeline overhead.

ONNX Classifier

ModernBERT ONNX inference uses:

Setting	Default	Purpose
Micro-batch	8	Hypotheses scored per ONNX batch
MODERNBERT_ONNX_INTRA_OP_THREADS	1	Threads within a single operator
MODERNBERT_ONNX_INTER_OP_THREADS	1	Threads between operators

Thread Settings for Containers

The classifier defaults to single-threaded ONNX execution. For bare-metal deployments with many cores, raise MODERNBERT_ONNX_INTRA_OP_THREADS cautiously. In containers with CPU limits, keep both ONNX thread settings at 1 to avoid oversubscription.

Category Prefilter

Before ONNX scoring, a Jaccard-based prefilter reduces the number of category hypotheses. This avoids scoring every category against every document when the vocabulary overlap is low.

Search Performance

Search performance is what your users directly experience. These settings control how fast results come back and how relevant they are. When someone types a query and waits for results, every millisecond counts toward their perception of the system.

Hybrid Fusion

Search uses alpha blending between dense and sparse scores:

Parameter	Default	Meaning
alpha	0.5	Equal dense and sparse weights; configurable with HYBRID_ALPHA
retrieval_top_k	10	Results before reranking

The retrieval limit is doubled internally before fusion to ensure the final limit results are high quality after score blending.

Reranking

Parameter	Default	Behavior
rerank	false	Disabled by default
RERANK_TIMEOUT_SEC	2.0	Hard timeout for cross-encoder
top_k_rerank	20	Candidates sent to reranker

If reranking exceeds 2.0 seconds, the request falls back to hybrid scores without reranking. This prevents a slow reranker from timing out the entire search.

What This Means for Users

• Without reranking (default): Searches are fast (~15-20ms) and results are good
• With reranking enabled: Searches take longer (~200-500ms) but the top results are more relevant
• The 2-second timeout is a safety net: if reranking is slow, the system returns results anyway rather than making the user wait

For most use cases, the default (no reranking) is the right choice. Enable it when precision of the top 5 results matters more than speed.

Stateless Design

SearchService is stateless per request. No session state is held between searches, so any API worker can handle any search request.

Monitoring and Observability

Monitoring tells you whether the system is healthy and helps you catch problems before users notice. These tools give you visibility into every component. Without monitoring, you are flying blind — you won't know a worker is stuck or Redis is full until users start complaining.

Container Health

Check all services with:

Terminal

just status

For a live-updating dashboard:

Terminal

just status live

Structured Logging

All services emit structured JSON logs via structlog. The log level is controlled by LOG_LEVEL (default INFO). In production, set:

.env

LOG_LEVEL=WARNING

Health Endpoint

GET /health returns the status of all dependencies:

{
  "status": "healthy",
  "components": {
    "api": "ok",
    "redis": "ok",
    "qdrant": "ok",
    "celery": "ok"
  }
}

Use this for load balancer health checks and uptime monitoring. If any component shows "error," that is the service you need to investigate first.

Docker Compose Health Checks

All production services define healthcheck blocks in docker-compose.yml with appropriate intervals and retries. The API health check polls /health. Workers use Celery's built-in inspect command.

Quick Reference: Capacity Planning

Use this table to estimate what hardware you need based on your expected load.

By User Count

Concurrent Users	Recommended Hardware	Expected Search Latency	Document Ingestion Rate
1-10	4 cores / 8GB RAM	< 50ms	~5 docs/min
10-50	8 cores / 16GB RAM	< 50ms	~15 docs/min
50-200	16 cores / 32GB RAM	< 100ms	~40 docs/min
200+	32 cores / 64GB RAM	< 100ms	~80 docs/min

About These Numbers

Search latency assumes hybrid search without reranking. Document ingestion rate depends heavily on document size, OCR usage, and whether classification is enabled. Scanned PDFs with OCR are 5-10x slower than plain text. These are guidelines — run your own benchmarks using just eval-retrieval.

Document Types and Scaling Impact

Not all documents are equal when it comes to processing time. Here is how different document types affect your capacity planning:

Document Type	Relative Processing Time	Why
Plain text (TXT, MD)	1x baseline	No parsing overhead, no OCR
DOCX with native text	1.2x baseline	Slight parsing overhead from python-docx
PPTX / XLSX (native)	1.2x baseline	Native python-pptx/openpyxl parsing
Legacy PPT / XLS (converted)	10-60x baseline	Requires LibreOffice conversion (up to 120s timeout); very slow
PDF with native text	1.5x baseline	PyMuPDF extraction plus per-page analysis
Scanned PDF with OCR	5-10x baseline	RapidOCR runs on every page with fewer than 80 characters
Image uploads (PNG, JPG)	8-12x baseline	Always processed through OCR

If 50% of your uploads are scanned PDFs, your effective document ingestion rate is roughly half what the capacity table suggests. Plan accordingly by either adding more preprocessing workers or accepting longer processing times. Communicate this to stakeholders so they understand why scanned documents take longer.

This table helps you set expectations with users and justify additional worker capacity when your document mix is heavy on scanned content.

Scanned documents are a common source of surprise for product teams because users expect all uploads to process at the same speed. Being explicit about this upfront prevents support tickets later.

By Document Volume

Total Documents	Estimated Chunks	Qdrant RAM Needed	Recommended Setup
Up to 1,000	~5,000	< 1GB	Single node, default config
1,000 - 10,000	~50,000	1-2GB	Single node, 16GB server
10,000 - 100,000	~500,000	2-8GB	Single node, 32GB server
100,000+	500,000+	8GB+	Consider Qdrant clustering (see Future Directions)

Estimating Chunks

Each document produces approximately 1 chunk per page for text-heavy documents, or 3-5 chunks per page when chunk size is 512 tokens. A typical 10-page PDF produces 20-50 chunks. Use the /documents endpoint to check actual chunk counts for your documents.

Scaling Decision Flowchart

Scaling Checklist

Before you scale, run through this checklist to make sure you are addressing the right bottleneck:

• Confirm the bottleneck — Use just status to see which service is at capacity. Is it the API, the embedding workers, or something else?
• Check document volume — Are you ingesting more documents than expected? Scanned PDFs with OCR are 5-10x slower than plain text.
• Review search patterns — Are users doing many searches, or are searches complex with reranking enabled?
• Verify hardware sizing — Does your server match the recommended hardware for your user count in the table above?
• Monitor Redis memory — Run docker exec infoconnect-search-engine-redis-1 redis-cli INFO memory and check usage against the 512MB limit.
• Check Qdrant collection size — Query /collections/documents to see point count and estimate RAM usage.
• Test after changes — Always run just eval-retrieval after scaling to confirm search quality has not degraded.

Start with Workers

If you are unsure where to begin, start by adding embedding workers. Document ingestion is usually the first bottleneck, and adding workers is safer than resizing the server because it does not require downtime.

Key Takeaway for Capacity Planning

The two numbers that matter most are concurrent users (drives API worker needs) and documents per day (drives embedding worker needs). Search speed is primarily determined by Qdrant RAM and whether reranking is enabled. Focus your planning on these three variables and you will cover 90% of scaling decisions.

Common Questions from Product Teams

Q: Do I need to scale if I only have 5 users?

No. The default configuration with a single embedding worker and 2 API workers handles up to 10 concurrent users comfortably. Scale when you notice slowdowns or when you plan to grow beyond that range.

Q: Why does document processing slow down when I upload a scanned PDF?

Scanned PDFs require OCR (Optical Character Recognition) to extract text from images. OCR is computationally expensive and runs inside the preprocessing worker. A 10-page scanned PDF can take as long as 50 plain-text pages. If you process many scanned documents, consider adding preprocessing workers or accepting longer processing times.

Q: Can I just use a bigger server instead of adding workers?

Yes, up to a point. A bigger server lets you run more workers on the same machine, which is the simplest scaling strategy. However, there is a practical limit to how large a single server can be. Once you outgrow a 32-core / 64GB server, you need horizontal scaling — multiple servers — which requires the future capabilities described below.

Q: What happens if Qdrant runs out of memory?

Search becomes slow or fails. Qdrant stores all vectors in memory for speed. When memory is exhausted, the operating system starts swapping to disk, which makes searches take seconds instead of milliseconds. Monitor collection size and plan to move to a larger server or enable on-disk mode before you hit the limit.

Q: Will adding more API workers make searches faster?

Not directly. API workers handle more simultaneous requests, which helps when many users search at the same time. Individual search speed is determined by Qdrant performance and whether reranking is enabled. If one user reports slow searches, the fix is usually Qdrant tuning or disabling reranking — not adding API workers.

Q: How do I know if Redis is the bottleneck?

Redis bottlenecks are rare but show up as "Redis OOM" errors in logs or stalled document processing where uploads succeed but never complete. Check Redis memory with INFO memory. If used memory is near 512MB, increase the limit or reduce result expiry time.

Future Directions

These are scaling improvements that are not yet built but represent the roadmap for handling significantly larger workloads. They require engineering investment but unlock new levels of performance. Consider these when your use case outgrows the single-host Docker Compose model.

These capabilities are not yet implemented but represent the natural next steps for scaling beyond single-host Docker Compose.

Reverse Proxy / Load Balancer

Currently there is no reverse proxy in front of the API. For multiple API replicas, you need a load balancer to distribute traffic and terminate TLS.

Business impact: Enables running multiple API instances behind a single URL, which means zero downtime during deployments and the ability to handle thousands of simultaneous users.

Recommended approach: Add Nginx or Traefik as a service in docker-compose.yml:

docker-compose.yml (future)

services:
  proxy:
    image: traefik:v3.0
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

A proxy also lets you add network-level rate limiting before requests reach the Redis-backed application limiter.

Qdrant Clustering

Current deployment runs Qdrant as a single node with one shard and no replication. For high availability and larger collections:

Business impact: Removes the single-point-of-failure for search, enables storing millions of documents, and keeps search fast as the collection grows.

Future Setting	Current	Target
Shard count	1	3-6 (based on collection size)
Replication factor	1	2 (each shard on 2 nodes)
On-disk mode	disabled	enabled for collections > 1M points
Read/write split	none	separate read and write endpoints

Distributed Rate Limiting

The built-in limiter now stores counters in Redis, so all API workers share one sliding window per protected-route, search, or auth-failure key.

Business impact: Enforces consistent usage limits across API instances, which is essential for multi-tenant or SaaS deployments where fair usage must be global.

Additional layers:

1. Built-in Redis-backed rate limiting for per-client API quotas
2. External rate limiter at the proxy level (Nginx limit_req, Traefik middleware, or Cloudflare) for edge abuse and DDoS controls

Redis High Availability

Current deployment uses a single Redis container. For production HA:

Business impact: Prevents task queue loss during Redis failures. Without this, a Redis crash means in-flight document processing tasks are lost and must be resubmitted.

1. Redis Sentinel — automatic failover with 1 master + 2 replicas + 3 sentinels
2. Redis Cluster — sharded data for larger state requirements
3. Connection pooling — add redis-py connection pool configuration in app/services/redis_client.py

Kubernetes / Container Orchestration

Docker Compose is sufficient for single-host deployments. For multi-host or cloud-native scaling:

Business impact: Enables auto-scaling based on actual traffic — the system automatically provisions more capacity during peak hours and scales down during quiet periods, reducing infrastructure costs.

Component	Kubernetes Equivalent
API	Deployment + HPA (Horizontal Pod Autoscaler)
Embedding workers	Deployment + HPA or KEDA (event-driven scaling)
Preprocessing workers	Deployment + HDA
Redis	Redis Operator or managed service
Qdrant	Qdrant Helm chart with StatefulSet
Models	Init container or shared PVC

A Helm chart would encapsulate all of this, with values.yaml for resource tuning per environment.

Asynchronous Search

Currently search is synchronous: the client blocks until results return. For very large collections or complex reranking:

Business impact: Enables searching across very large document collections without timeout issues, and supports progressive result loading in the UI for a better user experience.

1. Polling pattern — accept the query, return a search_job_id, and poll GET /search/status/{id} (similar to /embed + /status)
2. WebSocket streaming — stream results as they are retrieved and scored, useful for progressive UI rendering

Caching Layer

No query or embedding cache exists today. Adding caching would reduce redundant work:

Business impact: Reduces infrastructure load and improves response times for popular queries. If 100 users search for the same thing, only 1 search actually hits the database.

Cache Type	Key	TTL	Benefit
Query results	Hash of query + filters + alpha	5 minutes	Identical searches return instantly
Embedding vectors	Hash of text content	1 hour	Re-ingestion of similar documents skips embedding
Search metadata	Collection stats	30 seconds	Reduces Qdrant info calls

Redis (DB /0) already has capacity for this. A cache check before Qdrant would add minimal latency.

How to Talk to Engineering About Scaling

When you need to discuss scaling with your engineering team, here is how to frame the conversation productively:

Instead of: "The system is slow. Make it faster."

Try: "Users report searches taking over 2 seconds during peak hours. Based on the capacity planning table, we are in the 50-200 user range on a 16GB server. Should we add API workers or move to a 32GB server?"

Key questions to bring to engineering:

1. What is the current bottleneck? — Ask them to run just status and identify which service is at capacity.
2. What is our document growth forecast? — Share your expected document volume for the next 6-12 months so they can plan Qdrant sizing.
3. What is our user growth forecast? — Concurrent users drive API worker needs; total users drive document processing needs.
4. Are we using OCR heavily? — If you process scanned documents, flag this. It changes the worker math significantly.
5. What is our infrastructure budget? — Bigger servers cost more, but so does engineering time for complex multi-host setups. Discuss the trade-off.

Information to gather before the meeting:

• Average daily active users and peak concurrent users
• Average documents uploaded per day and expected growth
• Types of documents (scanned PDFs vs. text files)
• Current server specifications (CPU cores, RAM)
• Any user complaints about speed or timeouts

Having this context lets engineering give you specific recommendations instead of generic advice.

What's Next

• Tune individual parameters in the Performance Tuning guide
• Understand request throttling in the Rate Limiting guide
• Review the Architecture Guide for component interaction details