What Your Vector Index Never Saw

There is a moment in almost every RAG project where someone asks a question the system should obviously be able to answer, and it cannot. The document is in the index. The answer is on page 14. And the system says nothing useful.

When that happens, the instinct is to blame retrieval. Tune the chunk size. Add a reranker. Switch the embedding model. I have watched a lot of teams spend a lot of time in that loop.

My honest opinion after doing this in real projects: blunt embedding is often not enough, and the reason usually has nothing to do with retrieval at all. If you go back and look at what actually entered the index, you find that the answer was never there in the first place. It was in a bar chart. It was in a screenshot of a transaction. It was in a process diagram with six boxes and five arrows. The pipeline read the page, found no sentences worth keeping, and moved on.

That is the failure mode I want to talk about. Not "the model retrieved the wrong chunk", but "the information was never embedded". And especially in business documentation, where a large share of the real content sits in images, traditional embedding runs into a wall that no amount of tuning gets you past.

The loss happens before retrieval, and it is permanent

A conventional pipeline turns a PDF into a string. The string gets chunked, the chunks get embedded, and the vectors go into a store. Along the way:

The important part is the sequence. This loss happens at parsing time, before a single vector exists. Whatever the parser discarded is simply not in the index. You can tune retrieval forever and you will not recover it, because there is nothing to recover. It is a ceiling, not a knob.

How much it costs you depends entirely on your corpus. For press releases, very little. For engineering documentation, financial filings, audit reports and system landscape documentation, quite a lot. Which is to say: exactly the document classes that SAP and enterprise projects run on.

This is not just my impression. The FATHOMS-RAG framework paper found that text-only evaluation setups retrieve only textual content, with images, figures and tables in PDFs ignored during ingestion, and connected that directly to elevated hallucination rates on multimodal question categories.

I find that last point underrated. A blind index does not fail loudly. It fails confidently.

Docling: getting the parsing step right first

Before you can ask how a chart enters a vector space, you have to stop throwing charts away. That is what Docling does.

Docling started at IBM Research Zurich, is MIT licensed, and is hosted in the LF AI and Data Foundation. It ingests PDF, DOCX, PPTX, XLSX, HTML, EPUB, Apple Pages, images, LaTeX, email formats, audio and video, and emits Markdown, HTML, DocTags, or a lossless JSON representation called DoclingDocument.

Structure preservation. Layout analysis and reading order, plus table structure recognition that captures cell topology rather than just text, using the OTSL format. A table survives as a table, not as a scrambled paragraph.

Enrichments. These are the features that decide whether a figure becomes content or a placeholder. All of them are disabled by default, because they cost processing time:

Worth internalising: if you installed Docling, ran it with defaults, and concluded that it does not handle your figures, you did not actually test the feature. You have to switch it on.

Granite-Docling-258M

IBM's compact document vision-language model, Apache 2.0, roughly 258 million parameters. A SigLIP2 vision encoder, a Granite 165M language backbone, and an Idefics3-style pixel shuffle connector. It outputs DocTags, a markup that captures layout and element types in a single pass, and it does something specifically relevant here: it converts charts into tables, formulas into LaTeX, and tables into OTSL.

Against SmolDocling-256M it reports table recognition TEDS of 0.96 versus 0.76, OCRBench 500 versus 338, code F1 0.988 versus 0.915, and equation F1 0.968 versus 0.947. It runs on vLLM, SGLang, MLX and ONNX, which means it fits comfortably on local hardware. I run it on a DGX Spark without it getting in the way of anything else.

Here is the thing people miss about Docling, though. Docling does not answer the vector space question. It is the step before. What it gives you is a chart that has become one of three things: a table, a text description, or a cleanly cropped image with its surrounding context intact. Only then do you get to decide how that enters the index.

Three routes into the vector space

This is the actual question, and there are three answers. They are not competing philosophies. They are three points on a cost curve.

Route A: describe, then embed

Turn the chart into text, either a vision-model caption or a chart-to-table conversion, then embed that text with whatever text embedding model you already use.

In favour: no new infrastructure, no new vector store, works with any embedding model, and the intermediate representation is human readable. When retrieval goes wrong you can open the index and see exactly what the system thought the chart said. That auditability is worth more than people expect.

Against: the description is a bottleneck representation. Whatever the vision model did not mention is gone, permanently, a second time. A chart with forty data points becomes three sentences. If the question is "what was the Q3 dip", and the caption said "revenue grew over the period", you are back to square one.

This is precisely Docling's do_picture_description plus Granite-Docling's chart-to-table conversion.

Route B: one shared vector space

Image and text are embedded by the same model into the same space. One vector per chart, one vector per text chunk, one search across both.

In favour: architecturally the simplest multimodal option. Cross-modal search comes for free, so "show me the slide with the revenue dip" works without any special handling. Your existing vector store does not change shape.

Against: a single vector has to compress an entire page or figure. Fine-grained hits get washed out. It will tell you that page 14 is relevant. It will not tell you which row of the table did it.

Models: Cohere Embed v4, Gemini Embedding 2, Voyage Multimodal, WeMM-Embedding, Jina Embeddings v4 in dense mode.

Route C: late interaction on the page image

Skip parsing entirely. Render the page as an image, cut it into patches, embed every patch separately, and score relevance with MaxSim, the ColBERT mechanism: for each query token, take the maximum similarity across all document patches, then sum.

In favour: the highest retrieval accuracy on visually complex documents, and no OCR errors, because there is no OCR. You also get interpretability the other routes cannot offer, since you can highlight which region of the page answered the question.

Against: storage. ColPali produces 1,024 patch vectors of 128 dimensions per page. That is orders of magnitude more index than one vector per chunk. There is an entire research strand devoted to making this affordable, including Structural Anchor Pruning and the Visual RAG Toolkit's training-free pooling.

Models: ColPali and ColQwen, Nemotron ColEmbed V2, Jina v4 in multi-vector mode.

The honest conclusion

There is no winner. There is a cost curve. Route A is cheap and lossy, Route C is expensive and precise, Route B sits in between. And in practice the right answer is usually a combination: let Docling parse everything that is genuinely parseable, meaning text, tables and structure, and send only the actually visual elements down the image route.

Why the obvious answer is the wrong one

When you tell someone that charts need to go into the vector space, the reflex answer is "use CLIP". CLIP was the model that made joint image-text embedding a normal thing: 2021, 400 million image-text pairs, contrastive training. It remains a solid baseline for cross-modal search and zero-shot classification.

For documents, it breaks, and the reason is refreshingly concrete. CLIP encoders operate at 224 or 336 pixels of input resolution. For a page with small type, that is not enough. CLIP recognises large text and misses small text, as the TextMonkey work documents. Crop strategies raise the effective resolution but split words across crops and destroy semantics in the process.

Put plainly: CLIP was built to match "a dog on a beach". It was not built to read an axis label. To CLIP, a bar chart is a pattern of rectangles. This is the single most useful thing to understand about why the naive answer does not hold, and it is a resolution problem, not an intelligence problem.

The model landscape

WeMM-Embedding, from Tencent's WeChat Vision team, is the most interesting recent entry and the one I would look at first for a self-hosted stack. Apache 2.0, published as a technical report with code and weights, in 2B, 4B and 9B variants, with the 9B built on Qwen3.5-9B. It covers text, images, video, visual documents and arbitrarily interleaved inputs, but not audio. Embeddings are 4,096 dimensions with Matryoshka truncation. On MMEB-v2 the 9B scores 80.6 overall, 81.9 on image, 74.3 on video and 83.3 on visual documents. The 2B variant already surpasses the previously leading 8B open-source baseline. It is deployed in production across WeChat Channels, Official Accounts, Moments and e-commerce.

That visual documents score of 83.3 is exactly the number this article is about, and it comes with a permissive licence and downloadable weights. That combination is rare.

Gemini Embedding 2 is Google's first natively multimodal embedding model, generally available through the Gemini API and Vertex. Text, image, video, audio and documents map into one shared space. Per call it takes up to 8,192 text tokens, 6 images, 120 seconds of video, 180 seconds of audio and 6 PDF pages, with output dimensions from 128 to 3,072 through Matryoshka Representation Learning. The best all-rounder on paper. API only, which for a lot of my client work is where the conversation ends.

Cohere Embed v4 offers 128k context and images up to 2 million pixels, with Matryoshka dimensions at 256, 512, 1,024 and 1,536. It takes text, images and mixed payloads in a single call and returns one vector, so a PDF page screenshot or a slide can be indexed directly without being converted to text first. It is available on the Cohere platform, AWS SageMaker and Azure AI Foundry. Pragmatically this is the most deployable commercial Route B option for enterprise work, mostly because the Azure AI Foundry path makes European region deployment a real conversation rather than a hypothetical one.

Voyage Multimodal, from Voyage AI, part of MongoDB since early 2025, covers text, images and video. API only, no open weights. One property worth stealing regardless of whether you use it: Voyage Multimodal and Jina v4 both lose under 1 percent quality when truncated to 256 dimensions, while several competitors degrade sharply. For a large chart index, that is a direct and significant storage lever.

Open source options

If the weights have to live on your own hardware, the field is smaller but genuinely good.

One licence warning, because it catches people: Jina Embeddings v4 ships under the Qwen Research License, inherited from its base model. It is not the drop-in permissive option its popularity suggests. Check it before it goes into a client deployment.

On leaderboards, ViDoRe V3 is the benchmark to watch for visual document retrieval, and the ranking moves monthly. Cite these with a date attached, or not at all.

What I would actually build

For a locally hosted stack, in order:

One, parse properly. Docling with Granite-Docling-258M. Small enough to run alongside everything else, and the chart-to-table conversion alone changes what reaches your index.

Two, one vector per page. WeMM-Embedding-2B or Qwen3-VL-Embedding-2B as the single-vector baseline. Route B, one index, no architectural surprises. Both Apache 2.0, both self-hostable, both fit on modest hardware.

Three, rerank rather than index. Only if that measurably proves too coarse, add ColQwen or Nemotron ColEmbed as a multi-vector reranking stage over the top 50 candidates.

Step three is the recommendation I rarely see stated explicitly, so let me be blunt about it: use late interaction as a reranker, not as your index. The precision of Route C is real and worth having. The storage cost of Route C is also real and mostly avoidable, because you do not need patch-level vectors for the entire corpus. You need them for the fifty pages you are about to choose between. Retrieve cheaply and broadly with one vector per page, then rerank the shortlist with the expensive method.

That single change turns a storage problem into a compute problem, and compute you can schedule.

One caveat that matters more in enterprise projects

Docling's remote picture description requires enable_remote_services=True. That flag sends document crops to an external endpoint.

For public documents, fine. For client documentation, engineering drawings, contracts or anything under a non-disclosure agreement, that flag is the one you do not flip. The alternative is not hypothetical: Granite Vision 3.1 and SmolVLM both run locally, and Granite-Docling-258M is small enough that local inference is not a compromise, it is just the default.

The same logic decides the embedding model. Gemini Embedding 2 is arguably the best all-rounder available and it is API only. Cohere Embed v4 at least offers an Azure AI Foundry deployment path. WeMM-Embedding and Qwen3-VL-Embedding you can simply download. For a lot of the work I do, that last property outranks every benchmark number in this article.

Conclusion

The thing I would want someone to take away is not a model recommendation. Models change monthly, and half the numbers in this article will be stale within a year.

It is this: go and look at what your index actually contains. Not what your pipeline was supposed to ingest, what it did ingest. Take twenty pages from your most figure-heavy document, run them through your current pipeline, and read the output. Count what survived.

In my experience that exercise is uncomfortable and extremely fast. You do not need a benchmark to discover that a system landscape diagram became a bare placeholder, or that a financial table became a run of unlabelled numbers. It takes ten minutes and it tells you whether your retrieval problem is a retrieval problem at all.

Business documentation is where this bites hardest, because business documentation is where people communicate deliberately in pictures. Architecture diagrams, process flows, screenshots of the actual transaction, quarterly charts. Those are not decoration around the real content. Very often they are the content, and the surrounding text is the decoration.

Blunt embedding treats them as noise. That is the limit, and it is worth knowing exactly where it sits before you spend another sprint tuning chunk sizes.

Sources