Skip to main content

Setting Up

Before you can upload documents and search them, you need to tell the system how to run. This is done through a configuration file called .env. Think of it like setting up your preferences before using an app.

What You Need​

  • The repository downloaded to your computer
  • A text editor (like Notepad, VS Code, or any code editor)

Step 1: Find the Example File​

In the main folder of the project, you'll see a file called .env.example. This file contains all the settings with sensible defaults.

# Navigate to your project folder (if you're not already there)
cd infoconnect-search-engine

# List files and look for .env.example
ls -la

Step 2: Create Your Own .env File​

The easiest way to get started is to copy the example file:

cp .env.example .env

This creates a new .env file with all the default settings:

ls -la .env

Or you can do it manually:

  1. Open .env.example in your text editor
  2. Save it as a new file called .env (just .env, no extra name before the dot)
tip

The .env file is special — it holds your configuration but is kept private. It won't be shared if you upload your code somewhere (it's listed in .gitignore).

Step 3: Understanding the Settings​

Open your new .env file. It is organized into seven sections. Here's what the important ones do:

1. Identity & Access​

API_KEY_CLIENTS={"test-key-123":"demo-client","another-key":"demo-client","super-secret-key":"demo-client","finance-demo-key":"finance-demo-client","ops-demo-key":"ops-demo-client"}
RATE_LIMIT=300
AUTH_FAILURE_RATE_LIMIT=10

API_KEY_CLIENTS maps each API key to the client id that owns its data. Include one configured key in the X-API-Key header for every request. The copied .env.example sets RATE_LIMIT=300 for local testing; if RATE_LIMIT is omitted, the application default is 30 requests per minute per resolved client. /search also has its own per-client sliding-window counter. AUTH_FAILURE_RATE_LIMIT defaults to 10 failed authentication attempts per minute per source. Set RATE_LIMIT=0 to disable rate limiting (not recommended for production).

qa-client is a QA-only client for release live checks and isolation runs (owner: the captain; never real users). Provision its production key out of band and keep it last in the mapping; see the QA-Only API Key procedure.

warning

If API_KEY_CLIENTS is empty ({}) or unset, startup prints a FAIL-CLOSED banner and every request is rejected with 401 Unauthorized until you add a key-to-client mapping. If you change the configured keys, remember what you set them to—you'll need one for every request.

2. Core Behavior​

DEBUG=true
CORS_ORIGINS=["http://localhost:3000","http://localhost:8080"]
LOG_LEVEL=INFO
RETRIEVAL_TOP_K=10
RERANK_TOP_N=5
MAX_FILE_SIZE_MB=200
ALLOWED_EXTENSIONS=[".pdf",".docx",".txt",".md",".text",".pptx",".ppt",".xlsx",".xls",".png",".jpg",".jpeg",".tif",".tiff"]

These settings control the API surface, upload limits, CORS origins, logging, and default search result counts. ALLOWED_MIME_TYPES sits beside ALLOWED_EXTENSIONS and validates the uploaded file's content type.

3. Domain Taxonomy​

ALLOWED_CATEGORIES=["financial reporting","risk management","corporate governance","regulatory compliance","business operations","market analysis","legal","human capital","technology","sustainability","KYC","Credit","Collateral","Ledger","other"]
NER_ENTITY_KEYS=["persons","organizations","dates","locations","monetary_amounts","account_numbers","transaction_refs","account_types"]

ALLOWED_CATEGORIES controls the labels ModernBERT can assign. NER_ENTITY_KEYS controls which extracted entity groups appear in API responses. Both values must stay in JSON array format.

4. Pipeline Tuning​

CHUNK_SIZE=512
CHUNK_OVERLAP=50
EMBEDDING_BATCH_SIZE=32
MAX_NODES_PER_TASK=50
RAPIDOCR_OCR_ENABLED=true
OCR_PAGE_TEXT_THRESHOLD=80

These settings tune document chunking, embedding batch sizes, and OCR fallback behavior. Keep them unchanged for your first run.

5. Classification & Entity Extraction​

CLASSIFICATION_ENABLED=true
SPACY_NER_ENABLED=true
CLASSIFIER_BACKEND=modernbert-onnx
SPACY_MODEL_NAME=en_core_web_sm
CLASSIFICATION_SELECT_TOP_K=3
MODERNBERT_PREFILTER_ENABLED=false

These settings enable automatic categorization and named-entity extraction. The ModernBERT options below this section control the ONNX model file, threshold, token limits, and thread counts.

6. AI / ML Models​

DENSE_MODEL_NAME=BAAI/bge-small-en-v1.5
SPARSE_MODEL_NAME=Qdrant/bm25
RERANK_MODEL_NAME=Xenova/ms-marco-MiniLM-L-6-v2
FASTEMBED_CACHE_DIR=./models/fastembed
HF_HOME=./models/huggingface

These are the embedding, sparse retrieval, reranking, and model-cache settings used by the processing and search services. Leave the model identifiers at their approved values.

The configuration enforces a seven-value commercial-use model allowlist. The approved setting/value pairs are:

  • SPACY_MODEL_NAME=en_core_web_sm
  • DENSE_MODEL_NAME=BAAI/bge-small-en-v1.5
  • SPARSE_MODEL_NAME=Qdrant/bm25
  • RERANK_MODEL_NAME=Xenova/ms-marco-MiniLM-L-6-v2
  • RAPIDOCR_MODEL_NAME=rapidocr-onnxruntime
  • MODERNBERT_ONNX_MODEL_REPO=onnx-community/ModernBERT-base-nli-ONNX
  • MODERNBERT_ONNX_MODEL_FILE=onnx/model_quantized.onnx

If any of these identifiers is changed to an unapproved value, settings validation fails during import, so the API and workers cannot start. Keep the approved defaults unless you have a source-approved change.

7. Infrastructure​

REDIS_APP_PASSWORD=replace-with-app-password
REDIS_CELERY_PASSWORD=replace-with-celery-password
REDIS_PUBLICATION_PASSWORD=replace-with-publication-password
REDIS_URL=redis://app:replace-with-app-password@localhost:6379/0
CELERY_BROKER_URL=redis://celery:replace-with-celery-password@localhost:6379/1
CELERY_RESULT_BACKEND=redis://celery:replace-with-celery-password@localhost:6379/2
PUBLICATION_ENABLED=true
PUBLICATION_REDIS_URL=redis://publication:replace-with-publication-password@localhost:6379/3
PUBLICATION_REDIS_MEMORY_MB=512
QDRANT_URL=http://localhost:6333
COLLECTION_NAME=documents

These tell the system where to find Redis, Celery, and Qdrant. Replace all three Redis password placeholders with distinct URL-safe values. Keep PUBLICATION_ENABLED=true: search serves exclusively through the publication authority, and the API refuses to start when publication is disabled or its Redis contract is unusable, instead of booting and failing every search with 503. PUBLICATION_REDIS_MEMORY_MB is the Redis memory budget in MB; the Compose Redis derives its maxmemory from that budget, so change them together.

Quick Start Configuration​

For your first time, you only need to check a few things:

  1. API_KEY_CLIENTS: Keep the defaults or map each key to your client id
  2. Redis passwords: Replace all three placeholder values with distinct URL-safe secrets
  3. CLASSIFICATION_ENABLED: Leave enabled if you want automatic categories
  4. Everything else can stay as-is

Next Steps​

Once your .env file is ready:

  1. Install the system if you haven't already
  2. Upload your first document

Common Problems​

ProblemCauseSolution
"File not found" when copyingWrong directoryMake sure you're in the project root where .env.example lives
Changes not taking effectForgot to restartStop and restart the services after changing .env
Permission deniedFile ownershipCheck that your user can read and write .env, then restart the services after correcting ownership.

What's Next​