Retrieval-augmented generation looks simple on a whiteboard. Index your documents, retrieve the relevant passages, hand them to a model, get a grounded answer. A demo over a few hundred clean files proves the idea in an afternoon.

Then the project meets real company data. RAG for enterprise is a different problem: millions of documents, dozens of formats, years of duplicates and contradictions, and questions where a wrong answer has a cost. The model rarely turns out to be the weak link. The data underneath it usually is.

Key takeaways

• Most RAG failures are data failures. Retrieval can only return what ingestion prepared, so messy inputs produce confident, wrong answers.
• The data foundation is a pipeline, not a parser. Extraction is one step among many: cleaning, deduplication, normalization, enrichment and keeping it all current.
• Format and duplication problems compound at scale. A small error rate across millions of documents becomes a constant stream of bad retrievals.
• Build the data foundation once and share it. When every team rebuilds its own preparation, quality drifts and costs multiply.

Why enterprise RAG keeps failing in production

The pattern is familiar. The pilot works, usage grows, then users report answers that are outdated, half right, or drawn from the wrong version of a document.

The instinct is to blame the model, or to tune prompts and retrieval settings. Those changes help at the margins, but they treat symptoms. The common cause sits further upstream: the system is retrieving from data that was never prepared for retrieval.

In practice, a typical RAG pipeline is assembled quickly: a parser, a splitter, an embedding model and a vector store. That works when the source material is uniform. Enterprise data is not. It spans scanned PDFs, email threads with attachments, call transcripts, spreadsheets and wiki pages, written by thousands of people over many years. Feed that in raw and the retriever has no way to tell the current policy from the 2019 draft, or a signed contract from a redlined copy.

We made a related argument in why unstructured context is the new moat: the model is no longer the bottleneck. The context it can draw on is.

What the data foundation actually does in a RAG system

A lot happens between a raw document and a useful search result. A solid data foundation handles each step.

From raw files to clean text

• Connect and ingest: pull content from every source (file shares, email, CRM, call recordings, ticketing systems) through one data ingestion pipeline rather than a script per source.
• Parse and extract: turn each format into clean text and structure, keeping tables, headings, reading order and the link between an email and its attachments.
• Clean and deduplicate: strip boilerplate, signatures and repeated disclaimers, and collapse near-identical copies so one fact is stored once.

From clean text to retrievable context

• Normalize: make the same thing look the same everywhere. “USD”, “$” and “dollars” should mean one currency, and a customer named five different ways should resolve to one entity.
• Enrich: add the metadata retrieval depends on: source, owner, date, document type, product line, region and access rights.
• Segment: break content into units that keep their meaning. Good document chunking follows the structure of the document, not an arbitrary character count.
• Keep it current: detect what changed and reprocess only that, so the index reflects the business today rather than the day it was built.

This is the work context engineering describes: deciding what information an AI system has available, not just how you ask it a question.

The specific data problems that break enterprise RAG

Most retrieval failures trace back to a short list of data issues.

• Lost structure. A table flattened into a stream of numbers, or a two-column PDF read straight across, produces text that embeds well and means nothing.
• Duplicates and versions. When ten copies of a document exist, the retriever may return the oldest one, or three copies of the same passage that crowd out the answer.
• Chunks without context. A passage that says “the limit is 30 days” is useless if the chunk lost the heading that says which policy it belongs to.
• Inconsistent terminology. Teams, regions and systems name the same thing differently. Without normalization, a query in one vocabulary misses documents written in another.
• Missing metadata. Without dates, owners and permissions, the system cannot prefer recent sources or respect who is allowed to see what.
• Conversational data treated like documents. Emails and call transcripts carry meaning across many turns and speakers. Split them like a PDF and the answer to a question ends up separated from the question.

At enterprise volume these arrive together, every day, which is why one-off fixes do not hold.

How to build a data foundation that supports RAG at scale

Organizations that run RAG across many sources tend to converge on the same principles.

Treat preparation as shared infrastructure

Prepare data once and let every application use it. When each team builds its own pipeline, the same documents get parsed five different ways and every assistant gives a slightly different answer. A shared layer gives you one place to fix quality and one consistent view of the business. We describe this pattern in more depth in before and after managed context.

Automate the work that does not scale by hand

• Use extraction that handles complex documents and conversations, not just clean text files.
• Run deduplication and normalization as standard steps, not a cleanup project.
• Process changes incrementally, so no document is processed twice and the index never goes stale.
• Track lineage, so every answer can be traced back to the source passage it came from.

Keep data where it already lives

Moving sensitive data into yet another system creates governance work. Prefer preparation that runs alongside your existing data platform and writes results back to it.

This is the approach behind Flexor ACE, our AI Context Engine. ACE automates the preparation of any type of unstructured data, documents and conversations alike: it parses, extracts, classifies and normalizes content, adds business context and domain terminology, and writes the result back to your data platform in Iceberg or Delta Lake. Change data capture means no document is processed twice, and the same prepared data serves analytics and agents, so moving between them needs no extra work.

How different enterprise teams use RAG once the data foundation works

With a reliable foundation, use cases that were too risky become practical.

• Customer support: agents answer from current product documentation, past tickets and call transcripts, with sources cited, instead of from whatever was indexed at launch.
• Legal and procurement: teams compare contract terms across thousands of agreements and find every clause that conflicts with a new policy.
• Finance: analysts query invoices, statements and email approvals together, and trace each figure back to its document.
• Sales: account teams get a grounded view of a customer from CRM notes, emails and meeting recordings before every call.
• Analytics: the same prepared data feeds dashboards, so leaders can measure trends in calls or tickets that used to be invisible.

The pattern is the same in every case. The value comes from AI that understands the business, and that understanding is built in the data foundation. If your agents keep stumbling on business data, unstructured data killed my AI agent covers the agent side of the same problem.

Want to see what a shared data foundation looks like on your own documents? Book a demo of Flexor ACE.

FAQs

What is the difference between RAG and fine-tuning?

Fine-tuning changes a model’s weights by training it on examples, which shapes its style and behavior but bakes knowledge in at a point in time. RAG leaves the model unchanged and supplies relevant information at question time. For company knowledge that changes often and must be traceable to a source, RAG is usually the better fit. Many teams combine both: fine-tuning for tone and format, retrieval for facts.

How many documents can an enterprise RAG system handle?

There is no hard ceiling: vector databases and search engines routinely index hundreds of millions of passages. The practical limit is preparation and governance, not storage. As volume grows, duplicates, outdated versions and permission errors grow with it, and retrieval quality drops. Systems that scale well invest in incremental processing, deduplication and metadata, so the index stays accurate as it grows rather than just getting bigger.

Does RAG work with real-time or only static data?

RAG works with changing data, provided the pipeline keeps up. The retriever can only find what has been processed and indexed, so freshness depends on how quickly new and edited content flows through. Change data capture helps here: the system detects what changed and reprocesses only that. For truly live values such as stock levels or account balances, many teams query the system of record directly alongside retrieval.

What is chunking and why does it affect retrieval?

Chunking splits documents into smaller passages that can be embedded and retrieved individually. Chunks that are too large dilute relevance; chunks that are too small lose the surrounding meaning. Splitting by a fixed number of characters often cuts tables, lists and arguments in half. Structure-aware chunking follows headings, sections and conversation turns, and attaches context such as the document title, which makes each passage understandable on its own.

How do you measure whether a RAG system works?

Measure retrieval and answers separately. For retrieval, check whether the right passages appear in the top results for a set of real questions with known answers. For answers, score correctness, completeness and whether each claim is supported by a cited source. Track the share of questions the system declines to answer, too. Rerun the same evaluation set after every data or model change so you catch regressions early.