Architecture

Executive Summary

The InfoConnect Hybrid Search Engine is like a smart filing system that reads your documents, understands what they mean, and finds them when you need them.

Think of it like a restaurant: you place an order (upload a document), the kitchen prepares it (workers process it), and everything is stored in the pantry (database) ready to serve when requested.

The system has four internal parts. The console's optional Chat view also calls the operator-selected LLM provider as an external model service:

• The Operator Console (Web UI): Gives trusted operators cited chat, search, and document-management workflows. Backend calls use a server-only key; Chat uses an operator-supplied provider key per request. The console must remain behind an external auth boundary
• The Front Desk (API): Takes your requests and gives you answers
• The Kitchen (Workers): Processes documents behind the scenes
• The Pantry (Storage): Keeps everything organized and ready to find

How a Document Moves Through the System

When you upload a document, it goes through four stages before it is ready to search:

Step 1: Upload

You send a file to the system through the API. The front desk checks that the file is valid (right type, not too big) and gives you a ticket number to track progress.

Step 2: Parse

Workers read the document and pull out the text. If it is a PDF, Word doc, or plain text, they extract the words. If it is an image or scanned page, they use AI to read it like a person would.

Step 3: Understand

The system breaks the text into small chunks and, during embedding, uses AI to:

• Identify names, dates, and organizations mentioned in the text
• Turn each chunk into vectors that capture both meaning and keywords

Step 4: Store and Categorize

After all chunks are embedded, the system selects representative chunks and classifies the document. Then all the information is saved in a special database that understands meaning, not just exact words. This lets the system find documents even when you search using different words than what is written.

Upload → Parse → Understand → Store & Categorize
  ↓        ↓         ↓               ↓
 API    Workers   Workers         Database

How Search Works

Searching works like asking a librarian who knows every book in the library:

1. You type a question in plain English
2. The system converts your question into a mathematical representation of its meaning
3. It compares your question against all stored documents
4. Returns the best matches ranked by relevance, showing you the most useful documents first

The system uses two methods at once:

• Semantic search: Finds documents about the same concept, even with different words
• Keyword search: Finds exact word matches for technical terms or names

Results are combined so you get the best of both approaches.

Main Services

API Service (The Front Desk)

The API is the public face of the system. It handles:

• Receiving document uploads
• Answering search queries
• Checking processing status
• Managing your documents (list, delete)
• Health monitoring

The API sits on port 8000 and speaks HTTP, the same language web browsers use.

Preprocessing Workers (The Prep Cooks)

These workers handle the first phase of document processing and the final classification callback:

• Read different file formats (PDF, Word, text, images)
• Split long documents into manageable chunks
• Run post-embedding document classification on the preprocessing queue

Docker Compose runs one preprocessing worker by default (worker-preprocessing-1) with --concurrency=2 and a 6GB container memory limit, while local development (just dev) runs one preprocessing worker with --concurrency=4. This work is I/O bound, so higher concurrency is useful because workers spend time waiting for disk/network operations.

Embedding Workers (The Sous Chefs)

These workers handle the heavy AI processing:

• Extract named entities (people, organizations, dates, places)
• Create dense vector embeddings (mathematical meaning representations)
• Create sparse vector embeddings (keyword importance scores)
• Save everything to the database

Two embedding workers process embedding tasks in parallel, doubling throughput for large document batches. Each worker uses 2 CPU cores and 2GB memory.

Post-Embedding Classification (The Specialist)

ModernBERT ONNX handles document classification after all embedding batches complete.

• Triggering: Preprocessing sets a version-scoped Redis atomic counter at classification_pending:{client_id}:{document_id}:{version_id} (TTL: 1 hour), and each embedding batch decrements it in a finally block. It also schedules a reconciliation check for the counter TTL boundary.
• Callback and recovery: The last embedding task to decrement the counter to zero triggers classify_document_task on the preprocessing queue. If that final Redis signal fails, the embedding task schedules a delayed classification fallback. Reconciliation later re-enqueues an embedding-complete document that is still pending, or records failed after its bounded attempts expire.
• Terminal failures: Classification retries retryable Qdrant and model failures. After retries are exhausted, it writes classification.status: "failed" for the same client/document version and propagates the task failure.
• Chunk selection: The classifier reads chunk embeddings back from Qdrant, computes a centroid, and uses MMR to select CLASSIFICATION_SELECT_TOP_K representative chunks (default: 3).
• Scoring: ModernBERT ONNX NLI scores the selected chunks against the configured category list.
• Latency: End-to-end classification typically takes 2-4 seconds per document.

The classifier still runs locally with ONNX Runtime, so no separate model-serving tier is required.

Storage Services

Redis: The task queue that holds jobs waiting for workers, plus a cache for quick lookups.

Qdrant: The vector database that stores documents in a way that supports semantic search. It keeps both the mathematical meaning (dense vectors) and keyword information (sparse vectors) for each document chunk.

Why There Are Two Worker Queues

Imagine a kitchen with two stations: one for quick prep work and one for careful cooking.

Preprocessing Queue: Like washing and chopping vegetables. It is quick work that happens in bursts while waiting for the fridge to open. Multiple people can chop vegetables at the same station without getting in each other"s way.

Embedding Queue: Like carefully preparing a sauce that needs constant attention. It requires focus and resources. If too many chefs try to make sauces at once, they will run out of burners and pots.

Separating them lets us run preprocessing and embedding with different concurrency profiles (Docker default: preprocessing 1×2, embedding 2×1; local dev: preprocessing 1×4, embedding 2×1), matching their resource needs.

Storage at a Glance

Documents live in Qdrant, organized like a card catalog that understands meaning:

• Each document is split into chunks (about 512 words each)
• Every chunk gets stored with:
  • A dense vector (captures semantic meaning)
  • A sparse vector (captures keyword importance)
  • Metadata (document ID, page number, categories)
  • Entities (people, organizations, dates found in the text)

The database supports hybrid search, combining semantic and keyword matching to find the best results.

Redis handles the task queue and temporary caching. It keeps track of:

• Jobs waiting to be processed
• Jobs currently being worked on
• Recent results for quick repeat access

Advanced Technical Details

Click to expand technical specifications and architecture diagrams

System Overview

The InfoConnect Hybrid Search Engine consists of the following components:

• Operator Console (Next.js): Internal-only search, cited BYOK chat, and document operations through same-origin server routes
• API Service (FastAPI): HTTP endpoints for document upload, search, status, and management
• Task Queue (Redis + Celery): Asynchronous document processing with dedicated queues
• Vector Store (Qdrant): Hybrid vector database with dense and sparse indices
• Workers: Prefork-based Celery workers for preprocessing and embedding
• Embedding Service (FastEmbed): Generates dense and sparse embeddings
• RapidOCR Service: RAPIDOCR_MODEL_NAME (rapidocr-onnxruntime) extracts text from images and low-text PDF pages
• Classification (post-embedding callback): ModernBERT ONNX categorizes documents after the last embedding batch finishes
• Entity Extractor (spaCy): NER for persons, organizations, dates, locations

Architecture Diagram

Document Processing Pipeline

When a document is uploaded via POST /embed:

Processing Stages

Stage	Queue	Description
Validation	API	File size, type, auth
Parsing	Preprocessing	Extract text from PDF/DOCX/PPTX/XLSX/TXT/Images; OCR fallback uses RAPIDOCR_MODEL_NAME (rapidocr-onnxruntime) for images and low-text PDF pages. Legacy .ppt/.xls converted via LibreOffice when OFFICE_LEGACY_CONVERSION_ENABLED=true
Chunking	Preprocessing	Split into overlapping chunks
Entity Extraction	Embedding	spaCy NER for persons, orgs, dates, locations
Dense Embedding	Embedding	BAAI/bge-small-en-v1.5 (384-dim)
Sparse Embedding	Embedding	Qdrant/bm25 sparse vectors
Storage	Embedding	Upsert chunk vectors and metadata to Qdrant
Chunk Selection	Embedding / Qdrant	Centroid + MMR selects CLASSIFICATION_SELECT_TOP_K representative chunks from stored embeddings
Classification	Preprocessing	classify_document_task runs after embedding finishes and updates document categories; reconcile_pending_classification recovers missed callbacks and terminalizes stranded work

Search Pipeline

When a search query is received via POST /search:

Celery Task Queues

Queue	Purpose	Workers	Concurrency	Container Memory Limit
preprocessing	I/O-bound parsing/chunking + post-embedding classification callback	1	2 on worker-preprocessing-1	6G
embedding	CPU-bound NER/embeddings	2	1 per worker (2 total)	2G per worker

Note: These memory figures are Docker container limits, not Celery child recycle settings. In production, Docker Compose runs 1 preprocessing worker with --concurrency=2 and 2 embedding workers with --concurrency=1 each. Classification and its delayed reconciliation task run on the preprocessing queue; see Post-Embedding Classification for the authoritative trigger and recovery behavior.

Task Routing

# preprocessing tasks
process_document.apply_async(queue='preprocessing')
classify_document.apply_async(queue='preprocessing')

# embedding tasks (automatic from preprocessing chain)
extract_entities_and_embed.apply_async(queue='embedding')

Qdrant Collection Schema

Collection: documents

Vectors:
  - text-dense: 384-dim float32 (cosine similarity)
  - text-sparse: BM25 sparse vector

Payload:
  - document_id: string
  - chunk_id: string (format: "doc_id:index")
  - chunk_text: string
  - chunk_index: integer
  - page_number: integer
  - categories: string[] (classification results)
  - metadata: object (user-provided + classification)
  - entities: object (NER results)
    - persons: string[]
    - organizations: string[]
    - dates: string[]
    - locations: string[]
    - monetary_amounts: string[]
    - account_numbers: string[]
    - transaction_refs: string[]
    - account_types: string[]

System Components

Operator Console

• Next.js App Router application under frontend/
• Scope: Trusted-operator cited chat, search, document management, upload, job status, health, and metrics; destructive operations are intentionally available
• Access boundary: Must not be public; deployment requires external SSO, a gateway, or an internal-only network because the app adds no application-level authentication
• Credential boundary: Backend proxy routes add the backend API key from server-only environment variables. Chat instead accepts an operator-supplied key for the selected LLM provider per request, keeps no server-side history, and never passes that key to the Python API
• Backend contract: Shared proxy transport fails closed, search forwards an allow-list of POST /search fields, and upstream failures map to safe client messages

API Layer

• FastAPI application on port 8000
• Endpoints: See the search, ingestion, document management, status, health, and metadata update references
• Rate limiting: 30 requests/minute per resolved client id for protected routes, with an additional /search counter (configurable via RATE_LIMIT)
• Health checks: Monitors Redis, Qdrant, and Celery status
• Lifecycle management: Async context managers for proper service startup/shutdown

Task Processing Layer

Celery Workers: Two separate worker pools for different workload types:

• Preprocessing Queue: I/O-bound tasks

  • 1 worker in Docker Compose by default (prefork pool, --concurrency=2)
  • 1 worker in local dev with --concurrency=4
  • 6G Docker container memory limit
  • Tasks: PDF parsing, DOCX parsing, text chunking, OCR fallback, and the post-embedding classify_document_task callback

• Embedding Queue: CPU/GPU-bound tasks

  • 2 workers in Docker Compose by default (prefork pool, --concurrency=1 each)
  • 2G memory limit per worker
  • Tasks: NER extraction, dense embedding, sparse embedding, Qdrant upsert

Task Chain:

Reliability:

• Exponential backoff retry (max 3 retries)
• Embedding workers recycle child processes after 250 tasks or ~1.5GiB per child; the preprocessing worker recycles after 500 tasks
• Docker container memory limits are separate: 2G per embedding worker and 6G for worker-preprocessing-1
• PyTorch cache cleared on worker startup
• Model warm-up on service initialization

Data Layer

Redis (redis:7-alpine):

• DB 0: API response caching
• DB 1: Celery task broker
• DB 2: Celery result backend (24h TTL)
• Append-only persistence enabled

Qdrant (v1.17.0):

• Primary collection: documents with text-dense (384-dim cosine) and text-sparse (BM25) vectors
• Companion collection: documents-fuzzy-terms with client- and chunk-scoped text-fuzzy spelling-vocabulary vectors
• HTTP port: 6333, GRPC port: 6334
• Persistent volume: qdrant_data

Core Services

Service	Purpose	Technology
EmbedderService	Dense/sparse embedding generation	FastEmbed (BAAI/bge-small-en-v1.5)
QdrantService	Vector storage and hybrid search	Qdrant client with raw-score gates and RRF fusion
EntityExtractor	Named entity recognition	spaCy (en_core_web_sm)
SearchService	Hybrid retrieval + reranking	FastEmbed + cross-encoder
ModernBERTClassifierService	Document classification	onnx-community/ModernBERT-base-nli-ONNX

Hybrid Retrieval Architecture

The system uses a hybrid retrieval approach combining multiple signals:

Method	Strengths	Best For
Dense (FastEmbed)	Semantic understanding	Conceptual queries, paraphrases
Sparse (BM25)	Exact keyword matching	Technical terms, proper nouns
Hybrid (Fusion)	Best of both worlds	General search, high recall
+ Reranking	Precision at top-k	When result quality matters most

Retrieval Components

0. Query normalization and typo correction:

   • See the Search API contract

1. Dense Embeddings (BAAI/bge-small-en-v1.5, 384-dim):

   • Generated by EmbedderService.embed()
   • Cosine similarity matching
   • Captures conceptual meaning

2. Sparse Embeddings (BM25 via Qdrant/bm25):

   • Generated by EmbedderService.embed_sparse()
   • Traditional keyword matching
   • Better for exact term matches

3. Pre-fusion confidence gates:

   • Apply model-specific raw-score floors to each active retrieval stream
   • Admit the union of candidates that clear either active stream's floor
   • See the Search API reference for the authoritative thresholds and calibration

4. Fusion (Reciprocal Rank Fusion):

   • Combines dense and sparse results
   • RRF formula: score = Σ 1/(k + rank)
   • Balances semantic and lexical relevance

5. Optional Reranking (Xenova/ms-marco-MiniLM-L-6-v2 via FastEmbed):

   • 2-second timeout to prevent blocking
   • More accurate but slower
   • Enabled via rerank=true in search request

Deployment Options

Running the System

# Start all services (API, workers, Redis, Qdrant)
just prod-up

# Verify everything is running
just status

# Stop when done
just prod-down

Production Docker Compose

All services containerized with health checks and resource limits:

Service	Image	Memory	Health Check
api	Custom multi-stage	-	HTTP /health
worker-embedding-1	embedding-runtime	2GB limit	-
worker-embedding-2	embedding-runtime	2GB limit	-
worker-preprocessing-1	api-runtime	6GB limit	-
redis	redis:7-alpine	-	redis-cli ping
qdrant	qdrant/v1.17.0	Persistent volume	HTTP /healthz

Shared Volumes:

• ./models: Model cache for FastEmbed, spaCy, and ONNX artifacts
• redis_data: Redis persistence
• qdrant_data: Qdrant storage

Scaling Considerations

Bottleneck	Solution
Search latency	Add Qdrant replicas
Document backlog	Scale embedding workers (1 per GPU)
Memory pressure	Reduce batch size, restart workers frequently
API throughput	Run multiple API instances behind load balancer

Data Flow Summary

Operation	Data Flow
Upload	Client → API → Redis → Celery → Qdrant
Browser search	Browser → Next.js server proxy → API → FastEmbed → Qdrant → browser
Browser chat	Browser → Next.js chat route → selected LLM provider; search tool → API → Qdrant; streamed cited answer → browser
Direct API search	API client → API → FastEmbed → Qdrant → API client
Status	Client → API → Redis → Client
Delete	Client → API → Qdrant

Security & Reliability

Rate Limiting

• Algorithm: Sliding window (60-second window)
• Default limit: 30 requests/minute per resolved client id
• Configuration: RATE_LIMIT environment variable
• Scope: Applied to protected routes per client id; POST /search also has a search-specific per-client counter

Client Isolation (Multi-Tenancy)

• API key → client mapping: API_KEY_CLIENTS is a dict of api_key → client_id. Auth is resolved by resolve_client(x_api_key) in app/utils/security.py; a missing key or an empty client value returns None → 401.
• Server-injected filter: Every Qdrant read and write passes a mandatory client_id filter. There is a single shared documents collection; the filter is applied server-side and cannot be bypassed by the caller.
• Cross-client access returns 404: GET /status/{job_id} checks a jobmeta:{job_id} Redis binding. If the job belongs to a different client, the response is 404 Job not found — not 403 — to avoid leaking the existence of other clients' jobs. The same principle applies to delete and update operations on another client's document.
• Fail-closed when unconfigured: If API_KEY_CLIENTS is empty at startup, the application logs a CRITICAL banner and every request to a protected endpoint returns 401 until keys are configured.

Error Handling

• Celery retries: Exponential backoff with max 3 retries
• Rerank timeout: Graceful degradation (returns unranked results after 2 seconds)
• Memory protection: Workers restart before OOM (2GB container limit for embedding, 1.5GB Celery child limit in code; preprocessing uses 6GB container / 4GB Celery child limit)
• Structured logging: All failures logged with structlog

Observability

• Logging: Structured JSON logs via structlog
• Health checks: /health endpoint monitors all dependencies and reports degraded status when required services are unavailable
• Job tracking: Celery result backend tracks job status
• Performance metrics: Query timing included in search responses
• Container health: Docker health checks for all services

What's Next

Now that you understand how the system works, you might want to:

• Set up the system: Follow the Installation Guide to get running
• Process your first document: Check out the First Document Guide
• Configure the system: See Configuration for customization options
• Learn the API: Reference the Embed API or Search API for endpoint details
• Explore retrieval settings: Visit Embeddings and Retrieval for search tuning