The Library That Reads Itself
My Literature Share on the Synology holds hundreds of documents and answers nothing. I rebuilt retrieval around a reasoning-based idea instead of a vector index: no embeddings, no chunking, one model endpoint on the box. And I checked the benchmark everyone quotes, because the table behind it says something the headline does not.
The Library That Reads Itself
There is a share on my Synology called the Literature Share. Books, manuals, sales documentation, slide decks, everything worth having twice. It is the kind of folder you want to interrogate: "which maturity model does this vendor actually describe?", "what does the 2019 whitepaper say about integration effort?". It is also the kind that resists every tool. The files are PDFs, PPTX, DOCX, markdown. The answers sit in tables, figures, slide footers. A full-text search returns strings; it does not answer questions.
So I built a question-answering path for it. The interesting part is not that it works. It is what fell out of the recipe on the way: no vector database, no chunking. This post is about the idea behind that, PageIndex, about why the classic RAG recipe is wrong for this job, about what the benchmark numbers really say once you check them, and about what I run instead.
Every figure in this post is sourced. The full list is at the bottom, and I have marked where a number is a vendor's own claim rather than an independent measurement.
What the classic recipe does
The conventional RAG pipeline is mechanical. Read the file into text, cut it into fixed-size chunks, embed every chunk, push the vectors into a store, answer a question by pulling the chunks whose vectors sit closest to it.
Three properties follow from that design, and they are where the argument starts.
None of that is a bug. It is the price of making retrieval fast, dense, and blind to structure.
The idea behind it: PageIndex
The team behind PageIndex makes the opposite bet. Their phrase, and I think it is the correct one: similarity is not relevance, and relevance requires reasoning. You replace the vector index with a tree index: a hierarchy of nodes built from the actual structure of the document, sections, subsections, tables, captions. At question time, a model reads that tree the way a human turns to the right section of a report, with the full context of the question in hand, and pulls the nodes it reasons are relevant.
| Vector RAG | PageIndex | |
|---|---|---|
| Index | vector index | tree index |
| Unit | fixed-size chunks | natural sections |
| Retrieval | semantic similarity search | LLM reasoning over the tree |
| Result | opaque, "vibe retrieval" | traceable to explicit references |
| Context | query embedding only | full context: history, domain knowledge |
That table is theirs, quoted from the project README [1]. It is a good summary of the bet. What it does not do is prove it, and the proof is where most write-ups stop reading.
What the benchmark actually says
The number everyone repeats is 98.7% on FinanceBench. I went and checked what sits behind it, because the version circulating on LinkedIn is missing the two facts that matter most.
FinanceBench is a document-QA benchmark over SEC filings: 10,231 questions across 40 US-listed companies and 361 filings from 2015 to 2023. From that pool the authors drew a 150-case sample, had experts manually review every model answer, and open-sourced those 150 cases [3]. Every number below is on that 150-case set, PageIndex's included.
Read that chart from the bottom and three things happen.
First, the "vector RAG scores about 50%" line is real, and it is not a strawman. It is GPT-4-Turbo with one vector store per filing, 75 of 150 correct, straight out of the benchmark paper's own results table [3]. Give the same model one shared store across all filings instead and it collapses to 19%. Retrieval design, not model quality, is doing that damage. That part of the PageIndex argument survives contact with the source.
Second, and this is the fact that gets left out: vector RAG is not the strongest classical baseline. Putting the whole filing in the context window scores 79% with the same GPT-4-Turbo [3]. Hand the model the exact evidence pages and it reaches 85%, which is the practical ceiling for that generation of model. So the honest comparison for a reasoning-based retriever is against 79%, not against 50%. The 49-point gap that gets posted around is a comparison against the weaker of the two available baselines.
Third, three years of model progress are baked into the gap. The FinanceBench baselines were measured in late 2023 on GPT-4-Turbo, Llama2 and Claude2. Mafin 2.5 is a 2026 system running 2026-era models. I could not find a single published run that holds the model fixed and varies only the retrieval architecture. Until someone does that, nobody knows how much of the gap is the tree and how much is simply three years of better models.
Two more things worth stating plainly. The 98.7% is self-reported by VectifyAI, although to their credit the per-question results are published and checkable [2], which is more than the competing vendors in their own comparison table offer. And the widely quoted 31% for "GPT-4o" is not what people think it is: it is "ChatGPT 4o + Search", measured on 66.7% of the benchmark, by Fintool, a commercial competitor grading a rival [2]. I would not put weight on that number, and I am only mentioning it because it circulates so widely.
None of this makes PageIndex a bad idea. It makes it an idea with one good, narrow, well-structured benchmark behind it. SEC filings are the friendliest possible corpus for a tree index: deeply structured, consistently formatted, machine-generated. My Literature Share is the opposite of that, which is exactly why I found the architecture interesting and the benchmark unpersuasive as a promise.
What it costs
The cost argument is stronger than the accuracy argument, and it is the part that actually decided my design.
Building the tree is cheap and you pay it once: about $0.001 per page, so a 1,000-page textbook is a little over a dollar and a few minutes, and every later question reuses the index [1].
The alternative to retrieval is handing the model the whole PDF on every question, which is exactly the 79% baseline above. That works, and it gets expensive fast, because the cost grows with the document while the tree's does not.
Two point one times at 52 pages, sixteen point six times at 420, and at 805 pages the document stops fitting the context window entirely [1]. One honest caveat that the README states and most write-ups drop: prompt caching is excluded from that measurement. On a document you query repeatedly with caching on, the multiplier narrows considerably. It is a fair upper bound, not a universal one.
So the real trade is not "accuracy versus accuracy". It is: long context is accurate and expensive and eventually impossible; retrieval is cheap and constant; and the question is how much accuracy the retrieval costs you.
Why I did not simply run it
Two reasons, both honest.
Cost structure at library scale. One tree is cheap. A library is hundreds of documents, and every node wants its own LLM call, per document, once at build and again on every question for navigation. On my machine everything runs through one llama-swap endpoint that must never be displaced by batch work, and the library-scale attempt hit exactly that wall: 429, rate limited, after ten retries, measured on 3 September 2026. The endpoint serves the models. A job that starves it is the wrong architecture.
Duplication. Before that decision, the pipeline had already converted every document through docling: structure and table recognition with OCR, page-image render, per-figure crop and caption. That export is the structural index a PageIndex tree tries to restate: headings, tables with per-cell data, figures with page and bounding box, in one file. Generating a second tree per document doubles the work, and the second bill buys nothing the parse did not already answer.
So I kept the idea and swapped the engine.
What runs instead
Four stages, one pass per document.
The query step is deliberately thin: pick a document, ask it, get the passages with the answer plus their page context. The retrieval reasoning is whatever you bring, which is the point.
The tradeoff, honestly
| Classic vector RAG | PageIndex (upstream) | This pipeline | |
|---|---|---|---|
| Indexing cost | embedding per chunk | per-section LLM calls | one parse pass, no second tree |
| Infrastructure | vector DB, embedding model | LLM calls per question | SMB, docling, one local model |
| Answer path | fast, opaque | traceable, slower, costlier per query | cheap, deterministic queries |
| Good for | huge homogeneous corpora | long structured single documents | mixed personal corpora, offline |
What you give up is the instant feel of a vector hit. That is fine here. A library is read, not filtered.
Where it stands
The code is public: github.com/pauloesterwitz/PageIndex-Embedding. It is early, and the honest finding is the interesting part: for document collections, structure is the index. The vector store is optional, and sometimes the right engineering move is not to fight your tool's limits but to accept the tool's gift: in this case, the export that already is the index.
The other finding is smaller and more useful. When a benchmark number is doing a lot of persuading, go and read the table it came from. In this case the table had a better argument in it than the headline did.
Sources
Paul Oesterwitz
AI & SAP Consultant · PhD Researcher