Working with Images and Scanned Documents

Upload a screenshot, scanned PDF, or photo and the search engine extracts the text automatically using OCR (Optical Character Recognition). No manual transcription needed.

What You Can Do

Find text inside screenshots and photos
Search scanned PDFs without retyping
Process documents from any source — digital or physical

How it works

The AI reads text from images using RapidOCR, then processes it like any other document. Upload an image and the extracted text becomes fully searchable.

This guide uses the Harry Potter Wikipedia screenshot included in the repo (test-documents/harry-potter-wiki-frontpage.png).

Before You Begin

Make sure these services are running:

just prod-up
just status

Also check that the API is responding:

curl http://localhost:8000/health
# Output from the 2026-06-11 run:
# {"status":"healthy","components":{"api":"ok","redis":"ok","qdrant":"ok","celery":"ok","modernbert-onnx":"ok"}}

Step 1: Make Sure OCR Is Ready

Check that OCR is enabled in your environment:

grep '^RAPIDOCR_OCR_ENABLED=' .env
# Output:
# RAPIDOCR_OCR_ENABLED=true

You should see RAPIDOCR_OCR_ENABLED=true. The default OCR backend is RAPIDOCR_MODEL_NAME=rapidocr-onnxruntime.

Step 2: Upload an Image or Scanned PDF

Use the Harry Potter screenshot included in the repo:

curl -X POST http://localhost:8000/embed \
  -H "X-API-Key: super-secret-key" \
  -F "file=@test-documents/harry-potter-wiki-frontpage.png" \
  -F "document_id=harry-potter-ocr-001"
# Output:
# {"job_id":"cb643480586f4ce08cbb5a3ac8e49fd0","document_id":"harry-potter-ocr-001","status":"pending"}

Save the job_id from the response. You'll need it in the next step.

info

What happens: The AI reads the image and extracts text, then processes it like any other document.

Step 3: Wait for OCR to Finish

Images take longer than text files because of OCR. On the 2026-06-11 run, this image completed in about 10 seconds on the local production stack:

curl http://localhost:8000/status/cb643480586f4ce08cbb5a3ac8e49fd0 \
  -H "X-API-Key: super-secret-key"

Output excerpt from the 2026-06-11 run:

{
  "job_id": "cb643480586f4ce08cbb5a3ac8e49fd0",
  "document_id": "harry-potter-ocr-001",
  "status": "completed",
  "chunks_created": 2,
  "categories": [],
  "classification_status": "pending",
  "entities_extracted": {
    "persons": [
      "Harry Potteris",
      "J. K.Rowling",
      "Ron Hary Potter Weasley",
      "Hermione Granger",
      "knownin-universeasMuggles",
      "J. K. Rowling",
      "Thomas Taylor",
      "Cliff Wright",
      "Giles Greenfield",
      "Jason",
      "on26 Cockcroft",
      "Genre Fantasy"
    ],
    "organizations": [
      "Wikipedia",
      "Hogwarts School of Witchcraft andWizardry",
      "the Ministry of Magic",
      "Philosopher's Stone",
      "Chamber of Secrets",
      "Azkaban",
      "Scholastic Press",
      "Deathly Hallows",
      "the Guinness World Record"
    ],
    "dates": [
      "1997",
      "1998",
      "1999",
      "2000",
      "2003",
      "2007",
      "June",
      "February 2023",
      "26 June 1997-21July 2007",
      "twenty-four hours"
    ],
    "locations": [
      "the United States",
      "United Kingdom"
    ],
    "monetary_amounts": [
      "0million"
    ],
    "account_numbers": [],
    "transaction_refs": [],
    "account_types": []
  },
  "error": null,
  "retry_count": null,
  "max_retries": null,
  "next_retry_in_seconds": null,
  "created_at": "2026-06-11T16:37:40.184086+08:00",
  "completed_at": "2026-06-11T16:37:49.757938+08:00"
}

Wait for the status to show "completed".

## Step 4: Search the Extracted Text

Now you can search the text that was inside the image:

```bash
curl -X POST http://localhost:8000/search \
  -H "X-API-Key: super-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"query": "Harry Potter characters"}'

Output excerpt from the same pre-calibration run, which used the former alpha=0.7 default:

{
  "results": [
    {
      "id": "d912ce8d-4f5c-232e-23dd-b18791ba0208",
      "score": {
        "value": 1.0,
        "rerank": null,
        "strategy": "hybrid"
      },
      "document": {
        "id": "harry-potter-ocr-001",
        "created_at": "2026-06-11T16:37:40.184086+08:00"
      },
      "chunk": {
        "id": "harry-potter-ocr-001:0",
        "index": 0,
        "text": "HarryPotter 文A 133languages Article Talk Read Viewsource ViewhistoryTools From Wikipedia,the free encyclopedia This articleis about thenovel series.For thecharacter,seeHarryPotter(character).Forthefilmseries,seeHarryPotter(film series).Forthetelevisionseries,seeHarryPotter(TVseries).Forthefranchise,seeWizardingWorld.Forthe universe,see Fictional universeof HarryPotter.Forotheruses,seeHarryPotter(disambiguation). Harr...",
        "page": null
      },
      "classification": {
        "categories": [
          "other"
        ],
        "status": "completed"
      },
      "entities": {
        "persons": [
          "Harry Potteris",
          "J. K.Rowling",
          "Ron Hary Potter Weasley",
          "Hermione Granger",
          "knownin-universeasMuggles"
        ],
        "organizations": [
          "Wikipedia",
          "Hogwarts School of Witchcraft andWizardry",
          "the Ministry of Magic",
          "Philosopher's Stone",
          "Chamber of Secrets",
          "Azkaban",
          "Scholastic Press",
          "Deathly Hallows"
        ],
        "dates": [
          "1997",
          "1998",
          "1999",
          "2000",
          "2003",
          "2007"
        ],
        "locations": [
          "the United States"
        ],
        "monetary_amounts": [],
        "account_numbers": [],
        "transaction_refs": [],
        "account_types": []
      },
      "metadata": {
        "schema_version": "2",
        "source": {
          "filename": "harry-potter-wiki-frontpage.png",
          "mime_type": "image/png",
          "extension": "png"
        },
        "document": {},
        "location": {
          "page_number": 1,
          "page_label": "1",
          "chunk_index": 0
        },
        "classification": {
          "status": "completed",
          "categories": [
            "other"
          ]
        },
        "quality": {
          "text_extraction": "ocr",
          "ocr_used": true,
          "warnings": [
            {
              "code": "ocr_applied",
              "message": "Text was extracted with OCR because no native text was available."
            }
          ]
        },
        "custom": {}
      }
    }
  ],
  "meta": {
    "query": "Harry Potter characters",
    "total": 20,
    "offset": 0,
    "limit": 10,
    "has_more": true,
    "query_time_ms": 13,
    "retrieval": {
      "alpha": 0.7,
      "reranked": false
    },
    "applied_filters": {}
  }
}

## How OCR Decides When to Run

The system decides when to use OCR based on what you upload:

| File Type | What Happens |
|-----------|--------------|
| PNG, JPG, JPEG | Always uses OCR - every pixel gets read |
| PDF with lots of text | Uses native text extraction (fast) |
| PDF with scanned pages | Uses OCR only on pages with little text |

You can adjust the threshold for PDFs. By default, pages with fewer than 80 characters trigger OCR. Change this with `OCR_PAGE_TEXT_THRESHOLD` in your configuration.

### OCR Pipeline Flowchart

```mermaid
flowchart TB
    subgraph Upload["📤 File Upload"]
        File["File: PNG/JPG/PDF/DOCX/PPTX/XLSX/TXT"]
    end

    subgraph Detection["🔍 File Type Detection"]
        CheckType{"What file type?"}
    end

    subgraph ImagePath["🖼️ Image Processing"]
        ImageOCR["RapidOCR<br/>rapidocr-onnxruntime"]
        ImageMeta["Metadata:<br/>ocr_used: true<br/>text_extraction: 'ocr'"]
    end

    subgraph PDFPath["📄 PDF Processing"]
        PDFParse["PyMuPDF<br/>Extract native text"]
        PerPage["For each page:"]
        ThresholdCheck{"Native text chars < 80?"}
        NativeText["Use native text"]
        NativeMeta["Metadata:<br/>ocr_used: false<br/>text_extraction: 'native'"]
        PDFOCR["Convert to image<br/>→ RapidOCR"]
        PDFOCRMeta["Metadata:<br/>ocr_used: true<br/>text_extraction: 'ocr'"]
    end

    subgraph OtherPath["📃 Other Documents"]
        DOCX["DOCX: python-docx"]
        PPTX["PPTX: python-pptx (slides)"]
        XLSX["XLSX: openpyxl (worksheets)"]
        TXT["TXT/MD: Plain text"]
        NoOCR["No OCR needed<br/>(native text extraction)"]
    end

    subgraph Output["📦 Output"]
        Documents["List of documents<br/>with OCR metadata"]
        Pipeline["→ Continue to chunking<br/>→ Classification<br/>→ Embedding"]
    end

    File --> CheckType
    
    CheckType -->|"PNG, JPG, JPEG"| ImagePath
    CheckType -->|"PDF"| PDFPath
    CheckType -->|"DOCX, PPTX, XLSX, TXT, MD"| OtherPath
    
    ImagePath --> ImageMeta --> Documents
    
    PDFParse --> PerPage
    PerPage --> ThresholdCheck
    ThresholdCheck -->|"No (≥80 chars)<br/>or OCR disabled"| NativeText --> NativeMeta --> Documents
    ThresholdCheck -->|"Yes (<80 chars)<br/>+ OCR enabled"| PDFOCR --> PDFOCRMeta --> Documents
    
    DOCX --> NoOCR --> Documents
    PPTX --> NoOCR --> Documents
    XLSX --> NoOCR --> Documents
    TXT --> NoOCR --> Documents
    
    Documents --> Pipeline

    style ImagePath fill:#e3f2fd
    style PDFPath fill:#fff3e0
    style OtherPath fill:#f3e5f5
    style Detection fill:#e8f5e9

Key Details:

Configuration	Default	Description
`OCR_PAGE_TEXT_THRESHOLD`	80	Minimum native text characters to skip OCR for PDF pages
`RAPIDOCR_OCR_ENABLED`	true	Master switch to enable/disable OCR feature
`RAPIDOCR_MODEL_NAME`	rapidocr-onnxruntime	OCR engine identifier

Where this happens in code:

File: app/tasks/process_document.py
Function: load_documents_task() (lines 379-615)
OCR Service: app/services/rapidocr.py

Tips for Better Results

Clear images work better than blurry ones
Text should be reasonably sized (not tiny)
Screenshots, photos, and scans all work
Resize very large images before uploading to speed things up

Common Problems

Problem	Cause	Solution
OCR very slow	Large image file	Resize image before upload
No text extracted	Image too blurry	Use clearer image
Wrong characters	Handwriting or stylized text	Use printed text when possible
Timeout errors	Image too large or complex	Reduce image size or increase timeout

What's Next

Learn about categories Auto-Categorizing Your Documents
Try advanced search options Embeddings and Retrieval
Upload your first regular document Your First Document

What You Can Do​

Before You Begin​

Step 1: Make Sure OCR Is Ready​

Step 2: Upload an Image or Scanned PDF​

Step 3: Wait for OCR to Finish​

Tips for Better Results​

Common Problems​

What's Next​