Skip to main content

Your First Document

This guide walks you through uploading your first document and searching it. You will use the sample Wikipedia article about Malaysia included in the repository.

Before You Begin​

Make sure you have:

  • Your .env file set up
  • A terminal open
  • An API key
  • The sample file at test-documents/malaysia-wikipedia.txt

Step 1: Start the Services​

just prod-up
just status

Wait for all services to become healthy before continuing.

Step 2: Upload a Sample Document​

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@test-documents/malaysia-wikipedia.txt" \
  -F "document_id=malaysia-wiki-001"

Example response:

{
  "job_id": "<job_id>",
  "document_id": "malaysia-wiki-001",
  "status": "pending"
}
Developer Shortcut

For interactive terminal use, you can run just embed test-documents/malaysia-wikipedia.txt instead of the curl command above. The just command handles upload, polling, and displays a summary automatically.

Note: The just commands are developer convenience tools for terminal access only. For programmatic API calls from applications, use the curl/HTTP examples shown in this guide.

Step 3: Wait for Processing​

The upload is asynchronous. Substitute the job_id returned by /embed in the status request:

JOB_ID="<job_id>"
curl "http://localhost:8000/status/${JOB_ID}" \
  -H "X-API-Key: super-secret-key"

Poll until the status is terminal and search_ready is true. A completed response has this shape; optional fields such as chunks_created, categories, classification_status, and entities_extracted may also be present:

{
  "job_id": "<job_id>",
  "document_id": "malaysia-wiki-001",
  "status": "completed",
  "search_ready": true,
  "snapshot_generation": 7
}

The status endpoint can also report pending, processing, retrying, or a failure status while the job is running or if processing fails.

tip

If you do not want to poll manually, run a loop until the status becomes completed and search_ready is true (see the Status API).

Step 4: Search Your Document​

Use an explicit lowercase content scope for ordinary text retrieval:

curl -X POST http://localhost:8000/search \
  -H "X-API-Key: super-secret-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "capital of Malaysia",
    "scope": "content",
    "limit": 5
  }'

The response is a page of Evidence Results. Each row represents one Document Version, not an individual chunk, and includes its canonical metadata, lane-local retrieval contributions, and bounded Content and Metadata Evidence Sets. Each metadata evidence item carries a path, value, and match_kind (exact, lexical, or semantic).

{
  "results": [
    {
      "rank": 1,
      "document_version": {
        "document_id": "malaysia-wiki-001",
        "version_id": null
      },
      "canonical_metadata": {
        "schema_version": "2",
        "source": {
          "filename": "malaysia-wikipedia.txt",
          "mime_type": "text/plain",
          "extension": "txt"
        },
        "document": {},
        "location": {
          "page_number": 1,
          "page_label": "1",
          "chunk_index": 0
        },
        "classification": {
          "status": "pending",
          "categories": []
        },
        "quality": {
          "text_extraction": "native",
          "ocr_used": false,
          "warnings": []
        },
        "custom": {}
      },
      "retrieval_contributions": [
        {"lane": "content", "rank": 1}
      ],
      "content_evidence": {
        "items": [
          {
            "chunk_id": "malaysia-wiki-001:0",
            "chunk_index": 0,
            "text": "Malaysia is a country in Southeast Asia...",
            "page": 1,
            "literal_spans": []
          }
        ],
        "total": 1,
        "truncated": false
      },
      "metadata_evidence": {
        "items": [],
        "total": 0,
        "truncated": false
      }
    }
  ],
  "meta": {
    "query": "capital of Malaysia",
    "total": 1,
    "total_is_lower_bound": false,
    "offset": 0,
    "limit": 5,
    "has_more": false,
    "query_time_ms": 15,
    "retrieval": {
      "alpha": 0.5,
      "reranked": false,
      "reranked_count": 0
    },
    "applied_filters": {}
  }
}

meta.total, pagination, and the results rows count Document Versions, not chunks or leaves. The retrieval window is capped at 200; a request where offset + limit exceeds 200 returns 422. meta.total_is_lower_bound can be true when a retrieval lane reaches its scan cap. Chunk identifiers and indexes appear only inside content_evidence; they do not change the Document Version identity of the row.

Try Two More Searches​

curl -X POST http://localhost:8000/search \
  -H "X-API-Key: super-secret-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "When did Malaysia gain independence?",
    "scope": "content",
    "limit": 5
  }'

curl -X POST http://localhost:8000/search \
  -H "X-API-Key: super-secret-key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What languages are spoken in Malaysia?",
    "scope": "content",
    "limit": 5
  }'

Common Problems​

ProblemCauseSolution
connection refusedServices not runningRun just prod-up
401 UnauthorizedWrong API keyCheck API_KEY_CLIENTS in .env, for example API_KEY_CLIENTS={"super-secret-key":"demo-client"}
No search resultsDocument is still processing or its serving snapshot is catching upWait for status: "completed" and search_ready: true (see Status API)
Document stuck on pendingWorkers not runningCheck with just status
409 METADATA_DISCOVERY_NOT_READYMetadata or both requested before Client readinessRead Searching Metadata and /discovery/readiness; use explicit content while backfill runs
503 Service UnavailableActive publication authority cannot be verifiedRetry after Retry-After; do not use an unverified or partial response. A coherently empty pre-bootstrap GET /documents returns an empty 200 instead.
File not foundWrong directoryRun from project root

What's Next​