Benchmarking Modern Document Intelligence: OCR, VLMs, and Layout Parsers
Which extraction architecture yields the highest fidelity across diverse document types? We ran six engines in parallel to find out.
Abstract: This study evaluates six document extraction methodologies (PaddleOCR, LlamaParse, Gemini Vision, PyMuPDF4LLM, MinerU+Qwen, Azure Document Intelligence) across five document types (digital text, scanned images, tables, charts, handwriting). Our findings indicate that no single engine dominates; rather, document type is the primary determinant for optimal engine selection.
The Extraction Challenge
While extracting text from well-formed PDFs is largely solved, real-world documents contain scanned images, complex tables, charts, and handwriting. Poor extraction directly degrades downstream systems like Retrieval-Augmented Generation (RAG) pipelines.
We built the DocExtract Framework—a parallel orchestration engine—to empirically compare these architectures and provide actionable guidance for developers.
The 6 Engines Under Review
PaddleOCR
Traditional Local OCR
Deep learning CNN/CRNN. Fast, private, but lacks layout understanding.
PyMuPDF4LLM
Heuristic Layout-Aware
Uses PDF metadata. Great for digital text, fails on scanned images.
LlamaParse
Cloud RAG Parser
Optimized for LLM-ready Markdown and structured table extraction.
Gemini Vision
Vision-Language Model
Multimodal reasoning. Excellent zero-shot handwriting recognition.
MinerU + Qwen
Hybrid Local AI
Structural analysis with VLM refinement. Dominates chart interpretation.
Azure Doc Intel
Enterprise Cloud
Prebuilt models offer the highest accuracy for scanned and handwritten text.
System Architecture
FastAPI backend dispatching requests to 6 engines in parallel via asyncio.gather().
Evaluation Dataset & Methodology
Digital PDFs
Selectable text layer
Scanned Image
No embedded text
Tables
Multi-column bordered
Charts
Bar, line, pie visual data
Handwriting
Cursive & printed notes
Weighted Scoring
0.20 × 1-page + 0.30 × 5-page + 0.50 × 10-page (penalizes compounding errors).
Dimensions
Text completeness, structural fidelity, table quality, and semantic coherence (0-10 scale).
Benchmark Results
| Document Type | PaddleOCR | LlamaParse | Gemini Vision | PyMuPDF4LLM | MinerU+Qwen | Azure DI |
|---|---|---|---|---|---|---|
| Digital PDF | 10.23 | 10.33 | 10.33 | 10.33 | 10.33 | 10.23 |
| Scanned Image | 3.33 | 4.90 | 8.17 | 8.60 | 0.00 | 8.80 |
| Structured Tables | 5.20 | 7.80 | 4.45 | 6.20 | 5.70 | 5.80 |
| Charts | 3.10 | 8.02 | 5.77 | 0.00 | 8.92 | 3.40 |
| Handwriting | 2.50 | 3.13 | 8.40 | 5.13 | 7.35 | 8.85 |
Category Champions
No single engine dominates all categories. Document type is the primary determinant of optimal engine selection.
Engine Selection Guide
High-Volume Digital PDFs
Fastest extraction at zero API cost. Reconstructs layout from PDF metadata perfectly.
Complex Tables for RAG
Optimized specifically for structured Markdown output, preserving row/col alignment.
Charts & Visual Data
Two-stage pipeline uniquely capable of local visual data interpretation.
Enterprise Intake (Scans/Handwriting)
Best OCR infrastructure with automatic fallback for cursive and printed handwritten notes.
Privacy vs. Performance
Local models (PaddleOCR, PyMuPDF4LLM, MinerU) are viable for sensitive PII/legal documents without external network calls.
Cost-Benefit
Cloud services incur API costs but solve complex edge cases. MinerU+Qwen offers a compelling zero-cost alternative for charts if you can manage deployment complexity.
Conclusion
This study proves that the assumption of a "one-size-fits-all" extraction tool is flawed. While cloud APIs like Azure DI and LlamaParse offer high fidelity for unstructured or heavily tabular data, local models like PyMuPDF4LLM and MinerU combined with VLMs provide competitive, cost-effective, and privacy-preserving alternatives for specific document types.
Future Work
- Automated document-type classification for intelligent engine routing before extraction.
- Benchmarking next-gen Gemini 2.5 Pro capabilities on charts/tables.
- Quantitative cost-per-page scalability testing under concurrent loads.