Traditional RAG vs Vectorless RAG: How PageIndex Changes Document Retrieval

Imagine you’re building an AI assistant that answers questions about a company’s 120-page annual report.

A user asks, “What risks did the company identify, and how do those risks affect its revenue projections?”

Your application retrieves a few text chunks containing the words risks and revenue. The language model combines them into an answer, but it misses the relationship between a risk discussed on page 12 and a financial projection explained on page 47.

The answer sounds reasonable. The problem is that the retrieved context is incomplete.

This is one of the challenges developers face when building retrieval-augmented generation (RAG) systems. Traditional vector-based retrieval is useful for finding semantically similar text, but document structure and relationships between sections can be difficult to preserve through chunking alone.

An alternative approach, often called Vectorless RAG, uses document structure to guide retrieval. PageIndex is one implementation of this idea: it builds a hierarchical index and uses an LLM to navigate relevant sections.

Both approaches can help an LLM answer questions using external documents. They differ in how they organize information, locate relevant context, and trade retrieval cost against precision.

Let me explain how they work, when each makes sense, and how to choose the right approach for a real application.

Retrieval-augmented generation connects a language model to external information. Instead of relying entirely on its trained knowledge, the model retrieves relevant content from a data source and uses that content to formulate an answer.

Traditional RAG commonly uses vector embeddings to retrieve that content.

Consider a collection of PDF reports, internal documentation, support tickets, and technical manuals.

A typical pipeline has seven stages:

For example, a report might be divided into 512-token chunks with overlapping text between adjacent chunks. Each chunk gets an embedding, and the system stores its content and metadata.

When a user asks about revenue growth, the retriever searches for vectors close to the query embedding and returns the top matching chunks.

The LLM then generates an answer from that context.

Traditional RAG has a mature ecosystem and works well across many types of content.

For a support assistant searching millions of product documentation chunks, vector retrieval can be a sensible starting point.

The quality of a RAG answer depends heavily on the context retrieved before generation.

Chunking creates one recurring problem: the boundaries chosen for retrieval may not match the boundaries that make sense to a human reader.

Suppose a financial report contains:

A fixed-size chunk might contain the end of one section and the beginning of another. Alternatively, a critical explanation may be split across two chunks.

The retriever can return passages that look relevant individually while missing the relationship between them.

Other common challenges include:

These problems are not inevitable. Structure-aware chunking, hybrid retrieval, reranking, metadata, and contextual retrieval can improve traditional RAG considerably.

However, they introduce additional design decisions and operational work.

Vectorless RAG takes a different approach to document retrieval. Instead of converting every text chunk into an embedding and searching for nearby vectors, it builds a structured representation of a document and uses an LLM to navigate that structure.

PageIndex is a prominent example of this approach. It creates a tree-like index of document sections, then uses LLM reasoning to identify the relevant parts of the document. The PageIndex repository describes the method in detail.

Think about how you would read a financial report.

You probably wouldn’t split every paragraph into 512-token blocks, assign each block a numerical representation, and calculate which blocks are closest to your question.

You would open the table of contents, locate the relevant chapter, examine its subsections, and read the passages that answer your question.

Vectorless RAG tries to reproduce that style of navigation.

The process has two major phases: indexing and retrieval.

1. Document processing

The system reads a document and identifies its logical structure. Depending on the source, that might include chapters, headings, subsections, page ranges, and summaries.

For example:

This hierarchy becomes a navigable index. Each node can contain metadata such as a section title, page range, summary, or child nodes.

2. Query processing

When a user asks a question, the LLM examines the index and decides which branches deserve attention.

For example, the question about risks and revenue might lead it to inspect both 2. Risk Factors and 3.2 Revenue.

3. Section retrieval

The system retrieves the relevant sections from the original document. It can preserve their surrounding context and page references rather than relying exclusively on independently retrieved text chunks.

4. Answer generation

The LLM uses the retrieved passages to construct an answer grounded in the source material.

The exact implementation varies by system. PageIndex uses hierarchical indexing and reasoning-based retrieval, while other vectorless implementations may add summaries, document-level filtering, or other components.

Its biggest advantage is that it makes document organization part of the retrieval process.

These benefits are especially relevant for financial reports, legal contracts, regulatory documents, academic papers, and technical manuals.

Vectorless retrieval introduces its own costs.

First, LLM reasoning is part of the retrieval process. Depending on the implementation, a query may require several model calls to inspect branches and select relevant sections. That can increase latency and API costs.

Second, the index depends on the quality of the document structure. A badly formatted PDF, a document without meaningful headings, or a collection of unrelated chat messages may not provide a useful hierarchy.

Third, indexing has a cost. The system must extract the structure and may need to generate summaries or verify page mappings. Frequently changing documents can require index updates.

Finally, an LLM can navigate to the wrong branch or overlook a relevant section. Page references improve traceability, but they do not guarantee that the answer is correct.

Vectorless RAG is therefore a retrieval strategy with specific strengths, not an automatic solution to hallucinations.

One distinction deserves attention: traditional RAG is a broad architectural pattern, while Vectorless RAG describes a family of retrieval techniques. Traditional systems can also use full-text search, reranking, structural metadata, and hierarchical retrieval. The comparison above focuses on conventional vector-based RAG and PageIndex-style tree retrieval.

This question needs a little nuance.

Traditional vector retrieval is a natural fit when an application searches a large, heterogeneous collection. A vector index can narrow the search space efficiently, although performance depends on the index, hardware, filters, and workload.

A tree-based system must also decide which documents to inspect before navigating their internal sections. Searching a separate tree for every document would become expensive as the collection grows.

PageIndex has described a file-system indexing layer intended to support reasoning across large document collections. That adds a document-level navigation layer to the approach. See the official PageIndex repository for its current architecture and capabilities.

For developers, the important distinction is between two retrieval problems:

Vector search can be effective for the first problem. Hierarchical navigation can be effective for the second.

You do not have to use the same technique for both.

Yes. A hybrid architecture can use both approaches where each is useful.

Imagine building an enterprise assistant that searches thousands of documents, including policy pages, engineering documentation, financial reports, and legal agreements.

A practical pipeline might look like this:

The vector retriever narrows the collection to likely candidates. For structured reports, the system then navigates the document tree to retrieve the appropriate sections.

This architecture introduces more components to maintain, so it is worth using only when the retrieval quality justifies the added complexity.

A simpler hybrid is to use vector retrieval as the default and route selected document types to a structure-aware retriever.

Use these practical scenarios when designing a RAG application.

For example, a customer support assistant searching product FAQs, help articles, and historical support tickets is a reasonable candidate for vector-based RAG.

For example, an assistant analyzing an annual report may need to connect a risk factor with a financial result and a mitigation strategy. Tree-based retrieval gives it an explicit way to navigate those sections.

Do not select an architecture based on a promising demo alone. Build a representative evaluation set containing questions your users actually ask.

Measure:

Test questions that require information from several sections, questions involving exact numbers, and questions where similar terminology appears in unrelated sections.

A system that retrieves relevant text is not necessarily a system that retrieves sufficient evidence to answer correctly.

Traditional RAG remains a practical choice for applications that need to search large and varied collections of information. Vectorless RAG offers a different retrieval strategy for long documents whose structure carries meaning.

PageIndex demonstrates how an LLM can navigate a document tree to locate relevant sections, preserve page references, and support answers that are easier to trace back to their sources.

For developers building production RAG systems, the decision should come down to the shape of the data and the questions users ask.

If users need to find relevant information across a broad knowledge base, start with vector retrieval. If they need to analyze complex reports and connect information across sections, evaluate tree-based retrieval. If your application has both needs, consider a hybrid architecture.

The best retrieval system is the one that consistently finds the evidence your application needs, at a cost and latency your users can accept.

I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.