PageIndex Reality Check
PageIndex promises retrieval without embeddings: index a document as a tree of its own sections and let an LLM reason its way to the right one. I measured it against six conventional pipelines on the same corpus, the same questions and the same judge. It lost, and the way it lost is more interesting than the fact that it did.
PageIndex Reality Check
PageIndex promises retrieval without embeddings. Instead of chunking a document and scoring vectors, you index it as a tree of its own sections, and at query time an LLM reasons its way down that tree, the way a person turns to the right chapter instead of running a search. The pitch is good. The published numbers are striking.
I put it into my RAG Lab next to six conventional pipelines and measured it for three days. It lost. This is what the measurements say, what closed part of the gap, and what did not.
The setup
Where it landed
| Approach | Composite | Gold-chunk hit |
|---|---|---|
| RAG + Reranker | 0.840 | 94% |
| Corrective RAG | 0.827 | 92% |
| HyDE | 0.826 | 89% |
| Plain hybrid RAG | 0.824 | 90% |
| Agentic RAG | 0.822 | 90% |
| GraphRAG | 0.821 | 90% |
| PageIndex Hybrid | 0.706 | 66% |
| PageIndex | 0.680 | 57% |
The six vector pipelines sit within 0.019 of each other. PageIndex sits 0.14 below all of them. That is not a tuning gap, that is a different league.
The gap, decomposed
The useful question is not how far behind it is, but which of its two mechanisms loses. So I measured three configurations on the same questions, retrieval only.
| Configuration | Gold chunks found, of 100 |
|---|---|
| Plain hybrid search over the whole corpus | 91 |
| The same search, restricted to the documents the model chose | 78 |
| Full PageIndex: tree navigation inside those documents | 59 |
Routing to documents costs 13 hits. Navigating to sections costs 19 more. The second number is the method's core mechanism, and it is the expensive one.
The failure trace says the same thing from another angle: of the misses, the large majority are the right document with the wrong section. Candidates that only the tree surfaced contributed exactly one gold chunk across 100 questions.
Round 2: the obvious fixes
Some of the gap was my own fault, and worth fixing before drawing conclusions.
Result: 50% to 57% gold-chunk hit, 0.651 to 0.680 composite. Real, and nowhere near enough.
Round 3: does walking the tree twice help?
My implementation descends greedily: two sections per hop, no backtracking, no second look. Upstream's own documented prompt is less greedy than that, it puts the whole tree in one call and asks for every section that might hold the answer, and their production service combines that with a value-based Monte Carlo tree search. So iterating is not a deviation from the method. It is closer to it.
| Navigation strategy | Gold chunks found |
|---|---|
| Greedy descent, two sections per hop | 59 |
| Whole tree in one call | 65 |
| Beam search with backtracking | 54 |
| Navigate, grade the result, re-enter rejected branches | 59 |
| Tree as a re-ranking prior, excluding nothing | 89 |
Re-entry fired on 16 of 100 questions and changed no result, for 35% more model calls. Beam search scored below the greedy descent it replaced. Seeing the whole tree at once is a genuine improvement, and still lands 13 hits below simply searching those same documents without a tree.
The only configuration that got close to plain search was the one that stopped excluding anything: keep corpus-wide candidates and let the tree merely add a bonus to chunks in chosen sections. That is not navigation any more, and at 89 it still does not beat 91.
Round 3b: the one change that worked
If the tree cannot pick sections well, maybe the problem is what it gets to look at. During navigation the model sees a section's title and a summary. A summary compresses thousands of characters into two or three sentences and throws away exactly the rare token the question turns on.
So I showed it each candidate section's best-scoring sentence instead.
| Composite | Gold-chunk hit | Latency | |
|---|---|---|---|
| PageIndex baseline | 0.680 | 57% | 60.6 s |
| Evidence-augmented navigation | 0.742 | 68% | 43.6 s |
Better on every axis, and faster, because navigation stops sooner when the evidence is in front of the model. It also beats my tree-plus-vector hybrid variant without doing any vector search at all.
Two other ideas from the same round failed cleanly. Ranking documents by score instead of asking the model never fails to pick a document and finds both gold documents on 35 multi-hop questions against 14, better even than plain search, but it retrieves no better chunks at any budget. And splitting sections into one node per chunk, from 516 nodes to 2,710, made the model choose worse among near-identical snippets: retrieval fell to 46.
Measuring this honestly
The pinned model is not deterministic under concurrent load. Two identical runs of the same code flipped 11 of 100 questions. Most of the differences in this article are smaller than that.
So the harness replays model calls from a cache: identical prompts return the identical answer they returned before, and only genuinely new prompts hit the model. Two runs then differ by the code alone. The first change I measured looked like a regression of 4 questions until I did this, and turned out to be noise in both directions.
Every claim above is a paired comparison with that replay in place, and the kill criterion was written down before the runs, not after.
What this is not
It is not a refutation of PageIndex. The corpus is 11 thematically overlapping AI papers, and the questions were synthesized from individual chunks, which makes them lookup questions that share rare tokens with their answer. That is the ideal case for keyword and vector matching and close to the worst case for reasoning over structure. The published headline number for the method comes from financial filings, where a section heading really is the answer's address.
It is also not a tree-quality problem. I rebuilt the trees: rejected two unusable tables of contents, cut a 482-page book's root from 99 children to 41, and fixed summaries that described the previous section. The result moved by one hit.
Over to you
The honest summary is that on this corpus, structure-first retrieval cannot usefully exclude and cannot usefully re-rank, and the one thing that helped was giving the model real evidence at the moment it decides.
If you have run PageIndex or another structure-first retriever on real documents, I would like to hear what actually helped you. Did evidence in the navigation prompt move your numbers too? Did you find a document type where the tree genuinely wins? Or did you hit the same ceiling and stop?
The whole lab is public, including the harness, the variants, the negative results and the commit messages that record what was reverted and why: github.com/pauloesterwitz/RAG-Lab
Paul Oesterwitz
AI & SAP Consultant · PhD Researcher