Skip to main content

CLI Tools

Most command-line workflows should go through just. The recipes wrap the maintained scripts and preserve shell quoting for JSON arguments.

Authentication​

The search and embed scripts read INFOCONNECT_API_KEY when no explicit API key is passed.

export INFOCONNECT_API_KEY=super-secret-key

Use this for local terminals and CI jobs where the API key is already provided by the environment.

Search from the terminal​

just search "quarterly risk report"
just search "quarterly risk report" -- --json
just search "compliance" -- --filter document_id=doc-123 --rerank
just search "ledger" -- --field document --field chunk --field score

just search calls scripts/search.py and supports the same query controls as POST /search: filters, pagination, alpha, reranking, score thresholds, and field projection.

Observed just search output from a historical 2026-06-11 run (before v0.3.1; the current default is alpha=0.5):

Query: quarterly risk report
Results: 10
Total Matches: 20
Query Time: 16 ms
Alpha: 0.70
Top result: malaysia-wiki-001, score 0.500, categories sustainability

Embed files and folders​

just embed ./documents
just embed ./documents/report.pdf
just embed ./documents --max-pages 5
just embed ./documents --metadata '{"department":"Finance"}'

just embed calls scripts/embed_folder.py, uploads supported files (PDF, DOCX, PPTX, XLSX, legacy PPT/XLS, TXT, MD, images) through the public /embed API, polls /status/{job_id}, and reports successes and failures. It uses runtime concurrency from .runtime/prod.env when available; pass --concurrency to override it for a one-off run.

Monitor services​

just status
just status live
just status docker
just status local

Use just status live while a batch ingest is running. It shows API health, Redis/Qdrant readiness, worker state, queue counts, resource pressure, and recent alerts in a Rich terminal dashboard.

Operational scripts​

ScriptPurpose
scripts/download_models.pyPre-cache FastEmbed, spaCy, reranker, and ModernBERT assets before first boot or for offline deployments
scripts/verify_hybrid_collection.pyVerify Qdrant hybrid collection setup with a small upsert/search check
scripts/backfill_qdrant_fuzzy_vectors.pyBackfill the client-scoped spelling vocabulary for documents embedded before typo correction was enabled
scripts/backfill_embedding_version.pyBackfill filterable version stamps for legacy and v1-signature embedding points
scripts/reembed_documents.pySelect stale live chunks and enqueue vectors-only re-embedding from stored chunk text
scripts/evaluate_retrieval.pyCompare BM25, hybrid, and hybrid+rerank retrieval quality
scripts/generate_ndcg_toy_dataset.pyCreate a small graded retrieval dataset from existing chunks
scripts/benchmark_pr_classification.pyBenchmark ModernBERT classification behavior

Prefer the matching just recipe when one exists. Use scripts directly only for specialized diagnostics or automation.

Backfill embedding versions​

New /embed writes carry an indexed top-level embedding_version and detailed metadata.embedding provenance. Classify older points for one client with the read-only default:

uv run python scripts/backfill_embedding_version.py \
  --client-id client-a \
  --json

Apply mode creates and verifies the INTEGER payload index first. It stamps points without captured provenance as legacy version 0, stamps the known v1 signature as version 1, and leaves non-default captured signatures un-stamped for operator review:

uv run python scripts/backfill_embedding_version.py \
  --client-id client-a \
  --apply \
  --json

Re-embed stale documents​

The targeted driver selects live points where embedding_version is below the running pipeline version, including points with no stamp. It groups them by client, document, and document version. Dry-run is the default:

uv run python scripts/reembed_documents.py \
  --client-id client-a \
  --document-id doc-123 \
  --json

Use --apply to enqueue one embedding-queue task per matching document. The task reads stored chunk_text, updates dense and sparse vectors, then writes detailed provenance and the indexed root stamp. It does not rebuild payloads, rerun classification, or touch the fuzzy-terms collection. Historical chunks without captured chunker or extraction lineage are stamped with explicit unknown values. Known lineage must match the running chunker and extraction/OCR pipeline; incompatible documents must be re-ingested. Automatic runs exclude documents with captured non-default model or chunker signatures, including stale stamped versions, and log their identities as re-ingest-only. An explicit --document-id selection also refuses these documents with a clear instruction to re-ingest them from the original source; they cannot be repaired truthfully from stored chunk text.

uv run python scripts/reembed_documents.py \
  --client-id client-a \
  --document-id doc-123 \
  --apply \
  --json

One embedding model is active per collection. A model swap requires a full re-embed of every document from stored chunk text before model-swap cutover; targeted runs are for per-document or same-model pipeline fixes.

Backfill typo-correction vocabulary​

Newly embedded chunks maintain their spelling vocabulary automatically. For existing Qdrant data, run the backfill once per client. Start with its read-only dry run:

uv run python scripts/backfill_qdrant_fuzzy_vectors.py \
  --client-id client-a \
  --json

Before applying, quiesce ingestion and document delete/restore operations for that client. The apply command reconciles the entire client vocabulary, so --limit is intentionally available only for dry runs:

uv run python scripts/backfill_qdrant_fuzzy_vectors.py \
  --client-id client-a \
  --apply \
  --quiesced-ingestion \
  --json

--quiesced-ingestion is an explicit maintenance-window acknowledgement; the script does not pause writers itself. Use --qdrant-url and --collection-name when the deployment differs from the configured defaults. A nonzero exit reports one or more skipped write batches as errors.