RAG or large context: what to choose for PDF, tables and long documents

A large context is convenient when you need to work with a limited set of documents: one contract, several reports, a package of files for a one-time check. RAG helps manage the search of a library that is constantly changing and does not fit into a single query. Research confirms the trade-off between quality and cost, but does not provide a single winner for all PDFs.

The main condition for the text pipeline: both approaches depend on the quality of the extracted text. OCR errors and lost table structure should be checked separately. Multimodal mode can also transmit page images to the model; for example, OpenAI supports text and image PDFs for vision models. This is a separate option for testing, and not a guarantee that the document will be read correctly.

After the article, you will be able to group tasks by scenario, decide where full context is sufficient, where search is needed, and where to combine the two, and compare the options using your own files. We describe the approaches without tying them to a specific brand. The research underlying this article used models from 2023 and 2024; it is not a ranking of current models.

Two ways to provide documents to a model

In the text pipeline, the model receives the prepared text, and not independent access to the entire folder. The difference between the approaches is how much material and which fragments are included in the request. If page images are also transferred, this must be taken into account separately in the evaluation of quality and cost.

Full context. A document or multiple documents are passed entirely to the model along with the question. The model sees everything at the same time and looks for the right places herself. This approach became practical when models with a large context window appeared, that is, with a large maximum amount of text that they accept for one request.

Selection of fragments (RAG). Documents are divided into fragments in advance and indexed. When a question comes in, the system looks for the most relevant fragments and passes only those to the model. RAG (Retrieval Augmented Generation) stands for pre-retrieval generation.

Each approach has its price:

• Full context is simpler in architecture, does not depend on the quality of the search, but each query processes the entire amount of text. This affects cost and response time, and the scope is limited by the window size.
• RAG ​​passes selected fragments, which can reduce the number of input tokens, and scales to libraries that won't fit in a single window. The full cost also depends on index preparation, search and re-ranking. But it adds components that can also go wrong: slicing, searching, context building. How to diagnose them, we discussed in the article about why RAG invents answers.

The size of the context window alone does not determine the quality. Just because a model can accept the whole document does not mean that it will use every part of it equally well.

What the researchers were actually comparing

Two papers are often cited in the RAG or long context debate. It is worth knowing what exactly was checked in them.

Lost in the Middle (2023). The authors tested how the models use information in a long context on two tasks: answering a question on several documents and searching for meaning by key. The result was U-shaped: quality was highest when the desired information was at the beginning or end of the context, and significantly decreased when it was in the middle. The decline also persisted for models specially designed for the long context. This is the result for the 2023 models on two specific tasks. It does not describe all current models, but it shows that the use of long context should be tested, not assumed.

RAG or Long Context (2024). The paper compared RAG and long context on several public datasets with three models up to date in 2024. The full text compared GPT-4o, GPT-3.5-Turbo and Gemini-1.5-Pro. Authors' conclusion: Given sufficient resources, long context consistently outperformed RAG on average, and RAG maintained a significant cost advantage. The authors proposed a hybrid Self-Route method: the model itself evaluates whether it has enough found fragments, and if necessary, the request is redirected to the full context. In the conditions studied, this reduced costs at a quality close to full context. The work was accepted for EMNLP 2024 (industry track).

What does not follow from these works:

• that a particular modern model works better or worse in one of the approaches;
• that the long context is always more accurate: the result "on average" and "with sufficient resources" refers to the data sets studied and the RAG implementation studied;
• that RAG is always cheaper in your case: the cost depends on the volume, frequency of requests and how many times the same document is used;
• that either approach correctly calculates the tables or corrects recognition errors.

The practical conclusion of both works is different: quality and cost must be measured by their tasks, and the hybrid can retain most of the quality at a lower price.

Why PDF is more complicated than plain text

Most of the RAG or greater context arguments tacitly assume that the text of the document already exists and is correct. For PDF, this assumption is often false.

Scans. In the text pipeline, OCR converts a page image into text. Confidence scores depend on the product and API version: for example, in Microsoft Document Intelligence custom models starting with API 2024-11-30 GA, a field's confidence score takes word confidence into account. Do not apply the earlier response schema's rule about independent scores to every version. Check the actual confidence scores for words, fields, and tables against manually labeled examples.

Columns and reading order. Text in two columns when extracted may merge into lines where half of the sentence is from the left column and half from the right.

Signatures, seals, marginal notes. They are either lost or inserted in the middle of the main text.

Tables split across pages. A table spanning multiple pages may be split, with or without repeated headers. Microsoft's documentation separately explains that confidence for rows split across pages may be lower than for intact rows, even when individual cells are recognized with high confidence.

Units of measurement. Thousands of hryvnias in the header of the table and numbers in the cells may appear in different fragments of the text. The model sees "1,250" and doesn't know it's thousands.

For critical processes, the confidence score is used to determine which results to report to the human. The confidence score is not a measured accuracy on all future documents: the acceptance threshold should be checked on a representative sample. Financial or medical records require additional key field validations and exception handling.

Conclusion for the text pipeline: wrong text from the PDF will damage both the full context and the RAG. Therefore, the quality of text extraction is checked first, separately from the choice of architecture.

Choice scenarios

Instead of a general rule, it is more useful to divide your tasks into scenarios. Below is a matrix of editorial scenarios. These are guidelines for what to check first, not the results of testing the specified volumes.

Pay attention to the third line. A large window does not guarantee correct arithmetic. If the answer requires a sum, difference, or quotient to be calculated from table data, the choice between RAG and the full context is secondary: first, you need to make sure that the tables are extracted correctly, and the calculation itself is checked by code.

A hybrid approach

In practice, approaches are often combined. Here are some common ways.

Question Routing. Different types of questions follow different paths. Questions about a specific term in one contract are handled by the full context of that contract. Questions "where in our regulations are described..." go through the search. Calculations according to the tables go through the code.

Document search, then full document. RAG first determines which documents are relevant, and then the model is fed not a set of small fragments, but a large coherent section or the entire document found. This reduces the risk that the condition and its exception will end up in different fragments, of which only one will reach the model.

Self-sufficiency check. Self-Route idea from the 2024 study: the model first matches the found fragments or evaluates whether they are sufficient, and only if not, the query is repeated with the full context. This is a specific method for a specific job. Whether it will give a similar effect in your case, only a test will show.

Hybrid adds complexity: more routes, more room for error, more settings to test. Therefore, it is worth starting with the simplest option, which closes the main scenarios, and add routes when measurements show that they are needed.

How to compare on your data

The comparison makes sense only under the same conditions. Here's how to build it.

Identical files. Take real process documents, including problematic ones: scans, multi-page spreadsheets, column documents. Fix the version and the text extraction method is the same for both approaches. Manually check the extracted text quality on multiple pages separately.

Reference Answers. Prepare a set of questions with answers that have been checked by people who know the subject. Include questions that are answered in the middle of a long document, questions that are based on a table, and questions that are not answered in the documents.

What to count:

• separately, the proportion of correct answers and the proportion of correct refusals;
• the cost of one correct answer for the same period: costs for OCR, embeddings, index, search, reordering and generation, divided by the number of correct meaningful answers. Spread the one-time costs over a reasonable amount of usage, and make the caching terms and traffic comparable. Account for correct rejections separately; if there are no correct answers, this metric is not defined;
• p95 delays, that is, the time for which 95% of requests are answered;
• errors caused by extracting text separately so as not to attribute them to the architecture.

Same model. Compare approaches on the same model. Otherwise, you are comparing models, not approaches.

What to do in the near future:

1. Arrange your tasks according to scenarios from the matrix: a one-time document, a large library, tables with calculations, frequently changing data.
2. Test the quality of extracting text from your PDFs on several representative documents, separately for scans and tables.
3. Prepare a set of questions with reference answers, including questions from the middle of the papers and calculations.
4. Run full context and RAG on same files and same model.
5. Compare the proportion of correct answers, the cost of one correct answer and p95 delay.
6. For calculations based on tables, check separately the option with the calculation code on the extracted data.

Frequently asked questions