Document-to-Vector RAG Pipeline
Clean, metadata-rich chunks ready for embedding, with a human-check view.
The challenge
Naive PDF and DOCX extraction is noisy: split letters, broken acronyms, stray page numbers, mid-sentence breaks, no chapter structure. Character-count chunkers then cut mid-sentence and wreck retrieval.
What we built
Python tooling that cleans those artifacts deterministically, rebuilds chapter and section structure, and splits on sentence boundaries into embedding-sized chunks (~400-800 chars) with optional overlap. Every chunk carries a stable, readable ID and metadata, and we compared pdfplumber, PyMuPDF and unstructured.io to find the best quality-for-effort backend.
The result
Clean, metadata-rich JSON chunks ready for embedding and search, with a JSON-to-Markdown view for human checking and traceable, filterable retrieval downstream.