The RAG pipeline we deploy when clients say ‘our docs are a mess’

“Our docs are a mess” is the most common sentence in our discovery calls, and it is usually an understatement. The corpus that prompted this post was typical: 3,874 files across a shared drive, two wikis that disagreed with each other, and a PDF folder named FINAL_v2_ACTUAL. Here is the pipeline we deploy on that kind of material, with the numbers from the deployment, because the interesting decisions are all upstream of the model.

Start with an inventory, not an index

The first deliverable is a spreadsheet, not an embedding. We crawl everything, record owner, last-modified date and document type, and sit down with the client for an hour of rulings: which wiki wins when they conflict, which folders are dead, what must never be retrievable (HR investigations, unsigned contracts). Skipping this step does not save time; it converts a one-hour meeting into six weeks of “why did the bot cite the 2019 policy” tickets.

Dedupe before you embed

Of the 3,874 files, 14% were exact duplicates (hash match) and another 9% were near-duplicates — old versions of policies, “copy of copy of” drafts — caught with shingle-based similarity. After dedupe and the exclusion rulings, 2,090 documents went forward. This matters for retrieval quality, not just storage: if five near-identical versions of the leave policy exist, they will fill all five retrieval slots and crowd out the answer to the second half of the question.

Chunking: 400 tokens, headers attached

Chunking is the part that decides everything, and our default is deliberately specific: 400-token chunks, 60 tokens of overlap, split at heading boundaries, with the full heading path prepended to every chunk. The config we ship:

{
  "chunker": {
    "target_tokens": 400,
    "overlap_tokens": 60,
    "split_on": ["h1", "h2", "h3", "table"],
    "prepend_heading_path": true,
    "min_tokens": 80,
    "tables": "keep_whole"
  },
  "metadata": ["source", "department", "doc_type", "updated_at", "superseded_by"]
}

Why 400? At 1,000-plus tokens, a chunk about parental leave also contains sick leave and public holidays, similarity scores blur, and answers drown in adjacent policy. At 150, chunks lose their subject: a fragment that reads “This does not apply to contractors” is unanswerable on its own. The prepended heading path — HR Handbook > Leave > Parental leave — fixes the pronoun problem for both the embedding and the model reading the chunk later. Tables are never split; a table sliced in half is worse than no table.

Metadata filters do the heavy lifting

Before any vector math, we filter: department, document type, and a hard rule that superseded documents are excluded from retrieval (not deleted — auditors like history). Half of the “RAG accuracy problems” we get called in to fix are actually staleness problems, and no reranker rescues you from confidently citing a policy that was replaced in 2023.

An eval set of 60 real questions

Before tuning anything, we collected 60 real questions by asking staff one thing: “what did you look up last week?” Each question got a source-of-truth page agreed with the client. Sixty real questions beat six hundred synthetic ones, because staff ask questions whose words do not appear in the documents — the exact failure synthetic sets never contain. That set became the yardstick for every choice below.

What reranking changed

Retrieval quality against the 60-question eval set

Setup	Recall@5	Answer cites correct doc
Embeddings only	63%	71%
+ metadata filters	74%	80%
+ reranker (top 40 → keep 8)	88%	92%

The reranker — a cross-encoder scoring the top 40 candidates and keeping 8 — added about 180 ms and $0.0004 per query. Its gains concentrate exactly where embeddings struggle: questions phrased in words that never appear in the right chunk. At these prices there is no cost argument against it; the argument is only ever latency, and 180 ms is invisible next to generation time.

The remaining 8% of misses were almost all corpus problems — the answer genuinely was not written down anywhere. That is useful output too: we hand the client a list of questions their documentation cannot answer, and it becomes their writing backlog.

What we'd tell you to do

• Fix the corpus before you touch a model: inventory, rulings, dedupe. It is unglamorous and it is most of the result.
• Build the eval set first, from real questions. Every tuning decision after that is a measurement instead of an argument.
• Default to ~400-token chunks with heading paths attached, keep tables whole, and adjust only when the eval set tells you to.
• Exclude superseded documents with metadata, not hope.
• Log the retrieved chunks, not just the answers — when something goes wrong, the retrieval log is where you find out why.