RAG Pipeline Architecture: Every Stage, From Ingestion to Citations

Most RAG explainers stop at the three-box diagram: documents go in, a vector search happens, the model answers. That diagram is true and almost useless if you are the person who has to build the thing.

A production RAG pipeline is eight distinct stages, and only two of them are the ones people talk about. The interesting engineering is in the boring parts: keeping connectors alive, detecting what changed, re-embedding only what needs it, and making retrieval return the right chunk instead of a plausible one.

This guide walks the whole pipeline, stage by stage, with a real worked example rather than a generic diagram. Context Link is itself a RAG pipeline in production, so the numbers here are the actual numbers: chunk sizes, vector dimensions, the rerank formula, the lot.

If you already know what RAG is and you want the architecture, this is the piece. If you want the concept explained in plain English first, read RAG for non-developers and come back.

At the end there is an honest build-vs-buy section. Building a RAG pipeline is very doable. Keeping one running is a different job.

The Eight Stages of a RAG Pipeline

Here is the whole thing before we take it apart:

1. Ingestion fetch content from a source, on a schedule
2. Parsing turn PDFs, HTML, docs, and email into clean text
3. Chunking split that text into retrievable units
4. Embedding turn each chunk into a vector
5. Storage write vectors and metadata to a vector index
6. Retrieval embed the query, find nearest chunks
7. Reranking reorder results using signals similarity misses
8. Grounding hand the chunks to an LLM, return an answer with citations

Stages 1 to 5 run continuously in the background. Stages 6 to 8 run in the ~800ms after someone asks a question. That split is the single most important thing about RAG architecture: the expensive, fragile, ongoing work happens when nobody is watching, and the part users experience is comparatively simple.

Teams that build their own pipeline usually get 6, 7, and 8 working in a weekend, then spend the next year on 1 through 5.

Stage 1: Ingestion and Sync

Ingestion is where content enters the pipeline. It is also where most RAG pipeline projects quietly die.

There are two ways in.

Pull. Your pipeline holds credentials and fetches on a schedule. This is how you handle Notion, Google Drive, OneDrive, Basecamp, Monday, IMAP inboxes, and website crawls. Every one of those is a different auth model, a different pagination scheme, and a different set of rate limits.

Push. Something else sends you content over an API. This is far simpler to operate because you do not own the fetching, the auth, or the retries. Context Link supports both: pre-built connectors pull on a schedule, and custom connections let an AI agent fetch from any tool it can reach and push the result in as markdown. The custom connections docs cover the push shape.

Change Detection Is the Real Work

A first sync is easy. The second one is where the design decisions bite.

You need a stable identifier per document so re-syncing updates the existing record rather than creating a duplicate. Context Link keeps a per-source unique ID on every post, unique within the connection, so a page that gets renamed still maps back to the same record.

You also need to know whether the content actually changed. Re-embedding an unchanged document is pure waste: an API call, a bill, and a write for no new information. The cheap fix is a content hash. Store a SHA of each chunk's body, and on re-sync compare hashes before you spend anything. If they match, do nothing.

Scheduling matters too. Context Link re-syncs connected sources every 24 hours by default. That is a deliberate trade-off, not a limitation of the tech. Real-time sync means webhooks for every provider that supports them, polling for the ones that do not, and a queue that can absorb a Notion workspace-wide edit without falling over.

Scoping and Permissions

Ingestion is also where access control gets decided, and retrofitting it later is painful.

Two questions to answer before you write any code. Which subset of a source gets indexed? Nobody wants their entire Google Drive in a vector database. And who can see the resulting chunks?

Context Link denormalises a user ID onto every chunk at write time, copied down from the connection. Personal connections produce private chunks; organisation connections produce shared ones. The search query then filters on user_id IS NULL OR user_id =? in the same statement as the vector comparison. Doing it at write time means the filter is one predicate at read time rather than a permissions join.

Stage 2: Parsing to Clean Text

Embeddings are computed over text. Almost nothing arrives as text.

You are converting PDFs, Word documents, HTML, spreadsheets, Markdown, RTF, and email into something a model can read. Every format has a trap:

HTML carries navigation, cookie banners, and footers that will happily become your most-retrieved chunks if you index them. Strip chrome before chunking.
PDFs have no reading order. Multi-column layouts, tables, and headers come out interleaved unless the parser is layout-aware.
Spreadsheets are not prose. Context Link renders them into Markdown tables, one section per sheet, with a preamble naming the file, so a retrieved chunk still says what it is a row of.
Email is threads, quoting, and signatures. Without dedupe, one long thread produces twenty near-identical chunks.

Normalising everything to Markdown early is worth the effort. It keeps headings, lists, and tables intact, it is compact in tokens, and it is what LLMs handle best downstream. One text format after parsing means every later stage has one code path.

Stage 3: Chunking Strategy

Chunking is the highest-leverage decision in the pipeline and the one most often made by copying a default from a tutorial.

The tension is simple. Small chunks give precise retrieval but arrive without context, so the model gets a sentence with no idea what document it came from. Large chunks carry context but dilute the embedding: a 4,000-word chunk about six topics has a vector that is near-average for all of them and close to none of them.

What Context Link Does

Chunks land between roughly 800 and 3,200 characters, split on structural boundaries rather than a fixed character count. That is a wide band on purpose. A short FAQ answer should stay whole; a long policy document should split at its headings.

Two rules that matter more than the exact numbers:

Split on structure first, length second. Headings, paragraphs, and list boundaries are semantic joins that the author already put there. Use them, and fall back to length only inside an oversized section.
Filter for substance. Not every chunk deserves an embedding. Navigation lists, tables of contents, and boilerplate footers pass a length check and fail a usefulness check. Context Link runs a text-quality filter before embedding, which keeps the index smaller and the retrieval cleaner.

Chunking Strategy Choices Worth Knowing

Fixed-size with overlap is the tutorial default. It works, it is trivially implementable, and the overlap wastes 10 to 20% of your embedding spend re-encoding the same sentences.
Recursive structural splitting is what most production systems settle on. Split on the biggest boundary that fits, recurse into what does not.
Semantic chunking uses embedding distance between adjacent sentences to find topic shifts. It genuinely improves retrieval on long unstructured prose, and it costs an extra embedding pass over everything you ingest.
Parent-document retrieval embeds small chunks but returns their larger parent. Best of both, and more moving parts.

Pick one, then measure. Chunking changes are the fastest way to move retrieval quality, and the only way to know if it worked is a set of test questions with known-good answers.

Stage 4: Choosing an Embedding Model

An embedding model turns text into a vector, so that texts with similar meaning end up close together in that vector space. That is the whole trick behind semantic search: "how do we handle refunds" can retrieve a document titled "returns policy" with no shared keywords.

Context Link uses 1,536-dimension embeddings, which is the standard output size for OpenAI's text-embedding-3-small. Three things to weigh when you pick:

Dimensions. More dimensions capture more nuance and cost more to store and search. At 1,536 dimensions a million chunks is around 6 GB of raw vector data before indexes. Some newer models support truncation, letting you trade accuracy for footprint after the fact.

Hosted or self-hosted. A hosted API is one HTTP call and a per-token bill. A self-hosted open model removes the per-token cost and the data-leaves-your-network question, and adds a GPU to your infrastructure. The MTEB leaderboard is the standard reference for comparing retrieval quality across both.

Switching cost. This is the one people miss. Vectors from different models are not comparable. Changing your embedding model means re-embedding the entire corpus, every chunk, at full cost. Choose as if you will be stuck with it, because for a while you will be.

One non-negotiable: embed the query with exactly the same model you embedded the documents with. Mismatched models return confident nonsense.

Stage 5: Vector Storage

You need somewhere to keep vectors that can answer "which of these million rows is nearest to this one" quickly.

The genuine decision is whether to add a dedicated vector database or use the one you already run. Context Link uses pgvector on PostgreSQL, and for most teams that is the right call. Your chunks already need a row with a post ID, a position, a user ID, a content hash, and a searchable flag. Keeping the vector in the same row means your permission filters, your joins, and your metadata queries are all just SQL, in one transaction, with one backup story.

Dedicated vector databases earn their place at serious scale or when you need index types Postgres does not offer. Below tens of millions of vectors, a separate store mostly buys you a second system to keep in sync with the first.

Two schema details worth stealing:

Upsert on a composite key. Context Link upserts chunks keyed on document plus position, so a re-sync rewrites in place instead of deleting and re-inserting.
Keep a soft-delete flag. A searchable boolean lets you archive old versions of a document without discarding the embeddings you already paid for. Context Link uses exactly this for version history: when a document changes, the old chunks are moved onto a version record and flagged unsearchable rather than deleted.

Stage 6: Retrieval

Now the fast path. A question arrives and you have a few hundred milliseconds.

Embed the query with the same model as the corpus, then run a nearest-neighbour search. Context Link uses cosine similarity across the organisation's chunks, filtered to searchable rows and scoped to what that user is allowed to see. One query, one index scan, no fan-out per source.

That last point is the difference between a real pipeline and a wrapper. Native AI connectors search one tool at a time: search my email, search my Drive. A proper RAG pipeline searches everything you have connected in a single query, which means the person asking does not have to know where the answer lives before they ask.

Pull back more candidates than you intend to use. Similarity alone is a decent first filter and a poor final ranking, which is what the next stage exists to fix.

Stage 7: Reranking

Raw cosine similarity has known failure modes. It favours short text, because a 12-word fragment can sit very close to a 12-word query while saying nothing useful. It ignores which source a chunk came from. And it has no idea that the document title is a near-exact match for what was asked.

Context Link runs a blended rerank over the candidate set rather than a second model pass. The score is:

similarity × log(text_length) × source_weight

Each term earns its place. The log(text_length) factor pushes substantive passages above one-line fragments, with a log rather than a linear term so long documents do not simply win. The source_weight is a per-connection dial: your product docs can outrank a scraped competitor blog, and Modes let the same source carry a different weight per use case, so a "customer-support" profile prioritises help docs while a "sales" profile prioritises positioning.

Two boosts run on top of that:

Fuzzy tag boosting. Documents carry generated tags. A fuzzy match between the query and a document's tags multiplies its score, which catches topical relevance that a single chunk's text misses.
Title matching. A strong reorder pulls chunks whose document title matches the query to the front. If someone asks about the refund policy and you have a document called "Refund Policy", it should be result one, and pure vector similarity will not reliably put it there.

Finally, full-document expansion. If the top results cluster into only one or two source documents, Context Link pulls in the neighbouring chunks from those documents. A question answered across three consecutive paragraphs gets all three, not just the one that happened to score highest.

A cross-encoder reranker model is the other standard option here. It is more accurate than a scoring formula and adds a model call, latency, and cost to every query. Formula first, cross-encoder when you have measured that you need it.

Stage 8: Grounding the Answer With Citations

The last stage is the one that decides whether anyone trusts the output.

You have your ranked chunks. Two things to do with them.

Return them raw. Hand the model the numbered chunks as clean markdown and let it reason. This is the primary mode for agent use, and it is what Context Link's Get Context returns to Claude, ChatGPT, and any MCP-aware agent.

Or compose an answer. Layer a small, fast LLM on top: numbered context blocks, a strict instruction to answer only from those blocks, and a JSON response containing the answer plus the indices it actually used. Context Link's Ask Question does this and returns a paragraph with citations pointing back at the source documents. Any citation index outside the supplied range gets dropped rather than shown.

Two hardening details that are easy to skip and expensive to skip.

Tell the model the context is untrusted. Your chunks came from scraped websites and inbound email. Anything in them could be an instruction aimed at your prompt. The system prompt has to state that context blocks are reference material, never instructions.

Fail honestly. When retrieval returns nothing relevant, the pipeline should say so. An LLM handed thin context will produce a fluent, confident, wrong answer. Distinguishing "no context found" from "answered" is a two-line change that removes an entire category of trust problem.

Build vs Buy: Where the Cost Actually Lives

Here is the honest version, because the internet is full of people telling you a RAG pipeline is a weekend project and people telling you it needs a platform team. Both are describing different stages.

What Is Genuinely Cheap to Build

Stages 4 through 8. Embedding, storage, retrieval, reranking, and grounding are a few hundred lines. The libraries are mature, pgvector is a Postgres extension, and a competent developer will have working semantic search over a folder of documents in a day. If your corpus is one static document set that rarely changes, build it. Seriously. You do not need a vendor for that.

What Is Expensive to Keep Running

Stages 1 through 3, forever.

Connectors. Every source is a separate integration with its own OAuth flow, pagination, rate limits, and error semantics. Six sources is six integrations, and none of them are interesting to maintain.
Token refresh and auth drift. Refresh tokens expire, scopes change, users revoke access, providers deprecate endpoints. This produces a steady trickle of silent failures where the sync stops and nobody notices until an answer is three months stale.
Re-sync and freshness. Scheduling, change detection, incremental re-embedding, and backfills when something breaks. Getting this right is why the content hash and the stable per-source ID matter so much.
Parsing edge cases. A PDF with two columns. A spreadsheet with merged cells. An email thread with an inline reply. Each one is a small fix, and they never stop arriving.
Access control. Personal versus shared content, revocation, and making sure a chunk from someone's private inbox never surfaces in a colleague's answer.

None of that is hard. All of it is ongoing. The realistic cost of a self-built pipeline is not the initial build, it is that someone now owns a set of integrations as a permanent side-job.

How to Choose

Build when the corpus is yours and static, retrieval is deeply coupled to your product's core logic, you have a compliance requirement that rules out third parties, or you need control over retrieval behaviour that no vendor exposes.

Buy when the content lives in six SaaS tools you do not control, freshness matters, more than one person needs access with different permissions, and the pipeline is infrastructure for your business rather than the business itself.

There is a middle option too. Self-hosted open-source RAG platforms give you the connectors without the vendor, at the price of running the stack yourself, which is a real trade-off worth reading about in the Onyx comparison. At the other end, enterprise platforms bundle everything with enterprise pricing and enterprise onboarding, covered in the Glean comparison. The RAG as a service buyer's guide walks the pricing tiers across the whole market.

A RAG Pipeline Without Coding

If the stages above read as "I want all of that and none of the maintenance", that is what a managed pipeline is for.

Context Link runs every stage described here. Connect Notion, Google Drive, OneDrive, Basecamp, Monday, websites, IMAP inboxes, or uploaded files, and the parsing, chunking, embedding, indexing, re-sync, reranking, and citation handling all happen server-side. If your tool has no pre-built connector, the custom-connections skill lets your AI fetch it and push it in as markdown. If Claude can read it, you can push it to Context Link.

Then you query it from wherever you already work. There is a REST API, a ChatGPT connector, a Claude skill, and a hosted RAG MCP server that any MCP-aware agent can call as a tool. No new chat app to adopt.

For developers specifically, that last option is the interesting one. You skip stages 1 through 5 entirely and still get to control how retrieval is used in your own agent stack.

Wrapping Up

Five things worth keeping from this:

A RAG pipeline is eight stages, split between a continuous background path (ingestion, parsing, chunking, embedding, storage) and a fast query path (retrieval, reranking, grounding).
Chunking strategy moves retrieval quality more than model choice does. Split on structure, filter out boilerplate, and test against known-good questions.
Your embedding model is a long-term commitment, because changing it means re-embedding everything.
Similarity alone is not a ranking. Blend in length, source weight, tag matches, and title matches, then expand to neighbouring chunks.
The build-vs-buy line does not sit where people expect. Retrieval code is cheap. Connectors, auth, and freshness are the recurring cost.

If you are building, start with the query path over a single folder of documents, get your test questions passing, and only then take on ingestion. You will learn more from twenty real questions than from a month of architecture diagrams.

If you would rather not own the connectors, connect one source to Context Link and ask it something you already know the answer to. It takes about ten minutes, and it is the fastest way to see what a maintained pipeline returns before you commit to maintaining your own.

← Back to Blog