Embeddings and Retrieval

The search engine combines dense and sparse vector representations to find documents that match both semantic meaning and exact keywords.

Dense embeddings

• Default model: BAAI/bge-small-en-v1.5
• Vector size: 384
• Best for: semantic meaning

Sparse embeddings

• Model: Qdrant/bm25
• Best for: exact keyword matching

Reranking

• Model: Xenova/ms-marco-MiniLM-L-6-v2
• When to use: when top-result precision matters more than raw latency

Cache location

Models download on first use and are stored in ./models/fastembed/.

Chunking Strategy

• Chunk size: 512
• Overlap: 50

The live ingestion pipeline uses LlamaIndex's SentenceSplitter, which balances context preservation with retrieval accuracy.

Search Behavior: How alpha Works

alpha controls the dense (semantic) and sparse (keyword) retrieval weights:

• 1.0 → dense-only search
• 0.0 → sparse-only search
• 0.0 < alpha < 1.0 → weighted hybrid search, with dense weight alpha and sparse weight 1 - alpha

When to adjust alpha

Use case	Suggested alpha	Why
Conceptual / semantic queries	0.8 - 1.0	Prioritize meaning over exact terms
Keyword-heavy queries	0.0 - 0.3	Match specific terms precisely
General search	0.5	Balanced hybrid; current default

Start with the default alpha=0.5. Lower values favor sparse/lexical matches; higher values favor dense/semantic matches. The default floors are 0.70 for dense cosine similarity and 1.0 for sparse BM25 score. They gate raw candidates before fusion, and a hybrid candidate is retained when either active stream clears its floor. Raising a floor can remove noise, but a floor that is too high over-filters genuine matches and can produce no results. Queries with gibberish or no confident match still return no results. Recalibrate the floors when the embedding model or corpus distribution changes.

See Low-confidence filtering for the authoritative mechanism and configuration names.

Response Structure

Search results return grouped fields:

{
  "results": [
    {
      "id": "71d35bf9-4c69-bf23-b969-34e3af48b85d",
      "score": {
        "value": 1.0,
        "rerank": null,
        "strategy": "hybrid"
      },
      "document": {
        "id": "metadata-entities-001",
        "created_at": "2026-06-11T16:38:00.679223+08:00"
      },
      "chunk": {
        "id": "metadata-entities-001:0",
        "index": 0,
        "text": "MALAYSIA ======== From Wikipedia, the free encyclopedia Not to be confused with Malaisia, Malesia, or Malaya. --------------------------------------------------...",
        "page": null
      },
      "classification": {
        "categories": [
          "sustainability"
        ],
        "status": "completed"
      },
      "entities": {
        "persons": [
          "Malaisia",
          "Malesia",
          "Borneo",
          "Sarawak"
        ],
        "organizations": [
          "Wikipedia",
          "the Malayan Union",
          "the Federation of Malaya"
        ],
        "dates": [
          "the 18th century",
          "three years",
          "1946",
          "1948",
          "31 August 1957",
          "16 September 1963",
          "August 1965"
        ],
        "locations": [
          "Malaya",
          "Malaysia",
          "Southeast Asia",
          "South China Sea",
          "Peninsular",
          "East Malaysia",
          "Thailand",
          "Singapore",
          "Vietnam",
          "Indonesia",
          "Brunei",
          "Philippines",
          "Kuala Lumpur",
          "Putrajaya",
          "the British Empire",
          "Straits",
          "British Malaya",
          "Japan",
          "North Borneo"
        ],
        "monetary_amounts": [
          "34 million"
        ],
        "account_numbers": [],
        "transaction_refs": [],
        "account_types": []
      },
      "metadata": {
        "schema_version": "2",
        "source": {
          "filename": "malaysia-wikipedia.txt",
          "mime_type": "text/plain",
          "extension": "txt"
        },
        "document": {},
        "location": {
          "page_number": 1,
          "page_label": "1",
          "chunk_index": 0
        },
        "classification": {
          "status": "completed",
          "categories": [
            "sustainability"
          ]
        },
        "quality": {
          "text_extraction": "native",
          "ocr_used": false,
          "warnings": []
        },
        "custom": {
          "department": "Engineering",
          "team": "Backend",
          "author": [
            "wanjia",
            "adlin"
          ],
          "tenant_id": "tenant-a",
          "project": "alpha"
        }
      }
    }
  ],
  "meta": {
    "query": "capital of Malaysia",
    "total": 10,
    "offset": 0,
    "limit": 5,
    "has_more": true,
    "query_time_ms": 12,
    "retrieval": {
      "alpha": 0.5,
      "reranked": false
    },
    "applied_filters": {}
  }
}

Field groups

Group	Fields	Description
score	value, rerank, strategy	Ranking scores and retrieval mode. strategy stays dense, sparse, or hybrid; reranking is reported by rerank and meta.retrieval.reranked.
document	id, created_at	Document identifier
chunk	id, index, text, page	Chunk content and source page
classification	categories	Document classification
metadata	Custom fields	Uploaded metadata
entities	Extracted data	Named entities from NER
highlights	start, end, term	Matched Term spans ([start, end) offsets into chunk.text) when highlight: true is requested; otherwise an empty list

Entity Extraction

Every chunk gets analyzed for named entities during the embedding phase.

Stored entity groups

Category	Example
Persons	"Geoffrey Hinton"
Organizations	"Microsoft"
Dates / Times	"January 15, 2024"
Locations	"Seattle"
Monetary Amounts	"$10,000"
Account Numbers	"ACCT-998877"
Transaction References	"TXN-2024-001"
Account Types	"checking"