Vision-RAG Mining Intelligence System
Multimodal RAG system for mining intelligence with vision connectors and social data ingestion.
Problem Statement
What was being solved
Mining companies produce vast quantities of scanned reports — geological surveys, environmental assessments, reserve estimates — that exist only as PDF scans. Analysts needing to query across a 300+ document corpus face hours of manual search per question.
Why it matters
Mining investment decisions depend on rapid synthesis of historical report data. A single mis-read reserve estimate can mean a nine-figure investment error. The cost of slow or incorrect retrieval is not academic.
Why it's non-trivial
Standard OCR (Tesseract, Adobe PDF parser) fails on scanned mining reports because they contain: multi-column layouts, embedded geological tables with merged cells, hand-annotated diagrams, and non-standard unit notation. The retrieval problem is compounded by jargon-dense language that breaks generic embedding models.
Why Naive Fails
The obvious first approach
The obvious approach: run Tesseract OCR on the PDFs, chunk the extracted text, embed with OpenAI, and retrieve with cosine similarity.
What breaks
Tesseract on 300 DPI scans achieves ~60–70% character accuracy on mining tables — table headers get misread, numbers in adjacent columns get merged, and row structure collapses entirely. The resulting text is too corrupted to embed meaningfully. Cosine similarity on generic embeddings fails to distinguish 'proven reserves' from 'probable reserves' — a legally and financially significant distinction.
Architecture
Stack
Gemini 2.5 Flash
Multimodal vision reconstructs table structure from scanned PDFs that traditional OCR can't parse. Flash tier balances quality and cost for a 300+ document ingestion pipeline.
FAISS
Sub-millisecond retrieval with no standalone vector DB to operate — the 300+ document corpus fits in memory on the deployment server.
BM25
Exact-match retrieval for mining terminology (formation names, chemical notations) that dense embeddings miss. Complements FAISS via Reciprocal Rank Fusion.
LangChain
[TO BE ADDED]
FastAPI
[TO BE ADDED]
How
- 01
Gemini 2.5 Flash over traditional OCR
Gemini's vision capability understands table structure semantically — it can reconstruct a merged-cell table by understanding the document's visual layout, not just extracting characters. Traditional OCR returns character streams that destroy spatial structure.
- 02
FAISS + BM25 hybrid via Reciprocal Rank Fusion
Mining terminology (formation names, chemical symbols, regulatory codes) is exact-match sensitive. Dense embeddings alone lose these. BM25 exact-match retrieval is combined via RRF for complementary coverage.
- 03
Zero-downtime incremental indexing
New reports arrive weekly. Re-indexing the full corpus on each addition would create downtime windows. The pipeline maintains an append-only FAISS index with periodic consolidation, keeping the system queryable during additions.
Metrics / Outcomes
Results
93%+ query accuracy on 300+ document corpus. Sub-3s retrieval latency. Zero-downtime incremental indexing on weekly report additions.
Solving a similar problem?
I'm open to conversations about production AI systems — agentic workflows, RAG pipelines, or messy integration problems like this one.