What is RAG and how it works: embedding to vector search

RAG (Retrieval-Augmented Generation) is a technique where a language model searches for and finds relevant documents before generating an answer, adds them to the prompt, and lets the model base its answer not only on training data but on fresh, verifiable sources. That's the short answer to what RAG is and how it works: semantic search via embedding, document splitting via chunking, and adding that context to the prompt. This article walks through all three with real code examples, building a minimal system from scratch.

💡 Pro Tip: When setting up RAG, pick your chunk size first, then your model — if you go in the wrong order, changing your embedding model later means recomputing the entire vector store.

Table of Contents

• Understanding RAG in one sentence
• What embedding is, what a vector represents
• text-embedding-3 models and dimensions
• Chunking: splitting a document into pieces
• Avoiding context loss with Contextual Retrieval
• Vector store and similarity search
• Hybrid search: dense + sparse
• Placing the retrieval result into the prompt
• A minimal end-to-end example
• 5 common mistakes
• How do you evaluate a RAG system
• Keep the evaluation set alive
• FAQ
• What is RAG and how does it work?
• What's the difference between embedding and classic keyword search?
• Why do vector search results come back irrelevant?
• What's the simplest way to set up RAG in a small project?
• Update (September 2026)
• Conclusion
• Sources

Understanding RAG in one sentence

In Pinecone's definition, RAG is "a technique that uses authoritative, external data to improve the accuracy, relevance, and usefulness of a model's output." In practice this means: instead of asking a language model a question directly, you first pull documents relevant to the question from a data source, then add those documents to the model's prompt, and the model grounds its answer in that context.

RAG consists of four components: ingestion (bringing data into the system), retrieval (finding documents relevant to the query), augmentation (combining the found documents with the prompt), and generation (the model producing the final answer). Without all of these components a system doesn't count as RAG — merely searching and listing results is retrieval, not RAG.

Why do models get things wrong without RAG? Because when asked about something not in the training data or something outdated, the model produces, in Pinecone's words, "confident but wrong and irrelevant output" — this is called hallucination. RAG is a way of telling the model "answer by looking at this document" instead of "take your best guess."

It's worth thinking about these four components separately, because each can break in a different place. If ingestion breaks (you never got the document into the system, or you got it in the wrong format), retrieval has nothing to find. If retrieval breaks (the wrong chunks get found), augmentation isn't fed the right information. If augmentation breaks (even if the retrieved chunks are correct, they get placed wrong in the prompt), the model can ignore them. If generation breaks (the model ignores the context), the whole chain is wasted. When debugging a RAG system, checking them in order — ingestion, then retrieval, then augmentation, then generation — speeds up finding where things broke; I generally prefer to follow this order, because an error at the start of the chain breaks every step that follows.

What embedding is, what a vector represents

An embedding is the semantic content of a piece of text converted into a numerical vector (a list of decimal numbers). As OpenAI's embeddings guide puts it, the distance between two vectors measures how related those two pieces of text are. So the vectors for "cat" and "kitten" end up close together, while the vectors for "cat" and "tractor" end up far apart — this is what lets you capture the synonymy and context that classic keyword search misses.

text-embedding-3 models and dimensions

In January 2024, OpenAI announced two new embedding models, "text-embedding-3-small" and "text-embedding-3-large", a noticeable upgrade over the prior "text-embedding-ada-002" from December 2022; OpenAI didn't retire ada-002 — it recommends the new models but still allows the old one. By OpenAI's own measurements, the MIRACL multilingual average went from 31.4 on ada-002 to 44.0 on -3-small and 54.9 on -3-large; the English-heavy MTEB average rose from 61.0 to 62.3 and 64.6.

According to OpenAI's documentation, the default vector length is 1536 for text-embedding-3-small and 3072 for text-embedding-3-large. You can shorten this dimension via the dimensions parameter in the API: even when text-embedding-3-large's vector is shortened to 256 dimensions, it still outperforms the 1536-dimension ada-002. This is useful when you want to reduce storage cost — a shorter vector means less disk and faster similarity computation.

The table below summarizes the embedding models by dimension and MTEB score (OpenAI's official announcement and platform docs):

Chunking: splitting a document into pieces

Chunking is the process of splitting a large piece of text into smaller pieces called "chunks." Why split it? Because embedding models have a limited context window, and squeezing an entire document into a single vector loses track of which part of the document is relevant to the question.

Chunk size is a trade-off: according to Pinecone's chunking guide, "if chunks are too small or too large, it can lead to imprecise search results or missed opportunities." Chunks that are too small lose context; chunks that are too large drag in irrelevant information, lowering search precision.

A simple word-based, overlapping chunking function looks like this:

1def chunk_text(text, chunk_size=500, overlap=50):
2    words = text.split()
3    chunks = []
4    start = 0
5    while start < len(words):
6        end = start + chunk_size
7        chunks.append(" ".join(words[start:end]))
8        start = end - overlap
9    return chunks

The overlap parameter here reduces the risk of a sentence being split in half and losing its meaning, by leaving shared words between consecutive chunks — you need to try chunk size and overlap values on a small test set specific to your document type.

Avoiding context loss with Contextual Retrieval

Anthropic's engineering post from September 19, 2024 makes a common problem with classic chunking concrete: when a chunk on its own contains a sentence like "The company's revenue grew 3% compared to the previous quarter," which company and which quarter it's referring to becomes ambiguous. Their proposed solution is "Contextual Embeddings": before each chunk, a short explanatory context specific to that chunk is prepended; this context is typically "50-100 tokens" long.

According to Anthropic's measurements, using Contextual Embeddings alone reduces the retrieval failure rate over the top 20 chunks by 35% (from 5.7% to 3.7%). Combining Contextual Embeddings with BM25 pushes that reduction to 49% (from 5.7% to 2.9%), and adding reranking pushes it to 67%. Thanks to prompt caching, the one-time cost of generating this context is $1.02 per million document tokens.

Vector store and similarity search

After converting chunks to embeddings, you store them in a vector database. When a query comes in, cosine similarity is computed between the query's embedding and every chunk's embedding in the database, and the documents with the highest scores are returned. In the RAG flow, this step, in Pinecone's description, is the retrieval step that "finds the true meaning of the user's query by using semantic search from a vector database."

Hybrid search: dense + sparse

Embedding-based (dense) search alone isn't always enough. Pinecone's recommendation is to improve search results "by using hybrid search, which combines semantic search using dense vectors with lexical search using sparse vectors." The most commonly used algorithm on the lexical side is BM25 (Best Matching 25) — as Anthropic defines it, "a ranking function that uses lexical matching to find precise word or phrase matches."

As a final step, results are usually reordered by a reranking model — in Pinecone's words, "reranked according to a combined relevance score" — meaning the candidates gathered by a fast but coarse first-pass search are filtered down using a more expensive but more accurate model.

In a small project (up to a few hundred thousand chunks), a managed vector database requires far less maintenance than building your own ANN (approximate nearest neighbor) index. I prefer starting with a quick prototype on a managed store, and only evaluating my own infrastructure once scale grows and indexing cost or query latency visibly increase — this lets you answer "does it work?" first, instead of spending time on premature optimization.

Placing the retrieval result into the prompt

How you hand the found chunks to the model directly determines the quality of the answer. Pinecone's suggested augmented prompt template looks like this:

1QUESTION:
2<the user's question>
3CONTEXT:
4<the retrieved chunks>
5Using the CONTEXT provided, answer the QUESTION. Keep your answer grounded in the facts of the CONTEXT. If the CONTEXT doesn't contain the answer to the QUESTION, say you don't know.

The critical part is the last two sentences: you're explicitly asking the model to ground its answer in the CONTEXT, and to say it doesn't know if the CONTEXT lacks the answer. Without this, the model can ignore your chunks and hallucinate from its training data anyway.

The traditional hybrid flow Anthropic describes has six steps: split the text into chunks, generate TF-IDF encodings and semantic embeddings, find lexical matches with BM25, find semantically similar chunks with embeddings, combine and de-duplicate the results, then add the top K chunks to the prompt — the production-grade version of the chunk → embed → retrieve → augment flow above.

A minimal end-to-end example

Below is a minimal example based on the approach in OpenAI's embeddings guide: we embed the text, compute cosine similarity against the user's query, and find the closest chunk.

1from openai import OpenAI
2 
3client = OpenAI()
4 
5def get_embedding(text, model="text-embedding-3-small"):
6    text = text.replace("\n", " ")
7    response = client.embeddings.create(input=[text], model=model)
8    return response.data[0].embedding

This function wraps OpenAI's embeddings API; you can pass text-embedding-3-small or text-embedding-3-large as the model parameter.

1import numpy as np
2 
3def cosine_similarity(a, b):
4    a, b = np.array(a), np.array(b)
5    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
6 
7chunks = [
8    "RAG grounds the model's answer in external sources.",
9    "Chunking splits a large text into small pieces.",
10    "Cosine similarity measures the angle between two vectors.",
11]
12chunk_embeddings = [get_embedding(c) for c in chunks]
13 
14query = "What is RAG?"
15query_embedding = get_embedding(query)
16 
17scores = [cosine_similarity(query_embedding, ce) for ce in chunk_embeddings]
18best_index = int(np.argmax(scores))
19print(chunks[best_index])

This scaffold is a three-line vector store — in production you'd use Pinecone, Qdrant, or pgvector instead, since looping a Python list is slow for thousands of chunks, but the logic stays identical: embed, compute similarity, return the highest score.

Combining chunk_text above with this scaffold completes the end-to-end flow: split the document into chunks, embed each chunk, embed the query with the same model, and find the closest chunk.

1document = "... (long document text) ..."
2doc_chunks = chunk_text(document, chunk_size=200, overlap=20)
3doc_embeddings = [get_embedding(c) for c in doc_chunks]
4 
5def retrieve(query, top_k=3):
6    q_emb = get_embedding(query)
7    scored = [(cosine_similarity(q_emb, e), c) for e, c in zip(doc_embeddings, doc_chunks)]
8    scored.sort(key=lambda x: x[0], reverse=True)
9    return [chunk for _, chunk in scored[:top_k]]

The retrieve function returns the top top_k chunks; placing them into the "QUESTION / CONTEXT" template above and sending it to the model completes a minimal RAG loop. Passing a list to embeddings.create to batch chunks together, instead of one call per chunk, is a practical way to cut request counts on documents with many chunks; I prefer embedding chunks in groups rather than one at a time during ingestion.

5 common mistakes

• Picking the wrong chunk size: chunks that are too small fragment context, chunks that are too large carry irrelevant content and lower search precision.
• Leaving a chunk without context: if references inside a chunk like "the company" or "this quarter" are ambiguous about which entity they point to, the model misinterprets it even if retrieval finds the right chunk.
• Relying only on dense search: embedding search catches paraphrasing but can miss exact-matching terms (product codes, error messages); skipping hybrid search loses these kinds of queries.
• Underestimating hallucination risk without RAG: asking a question without adding an external source raises the risk of the model producing a "confident but wrong" answer.
• Not adding a grounding instruction to the prompt: adding context without telling the model "ground your answer only in the CONTEXT" still lets the model fall back on its own knowledge.

What these five mistakes share: they all let the system quietly produce a wrong answer while "appearing to work." It's easy to spot the error when a RAG system gives no answer at all; but a "plausible" answer built on wrong or incomplete context needs regular manual testing with real user questions to catch — automated tests can confirm the right chunks were found, but not always whether the answer is grounded in the CONTEXT or in the model's own knowledge.

How do you evaluate a RAG system

Building a RAG system is easy; proving it works is hard. Pinecone asks this directly: do you really need RAG, and how will you know it's working? The answer lies in "ground truth" evaluation: any application needs to be provably working before it ships, and AI is no different — "determining a set of queries and the expected answers to those queries is critical to knowing whether the application is working."

In practice this means writing down real user questions and each one's expected answer (or which document the answer appears in) before you start writing code. This small set is what makes every decision that follows — chunk size, overlap, embedding model, top_k value — measurable. Without a set, every adjustment you make is a guess.

Keep the evaluation set alive

Pinecone's second point of emphasis is that maintaining this set over time is also critical to knowing where to improve and whether your improvements actually worked. So the set isn't a one-time file; every new type of question that comes in from users gets added to it.

Splitting measurement into two layers makes this easier. At the retrieval layer: is the chunk with the expected answer among the top K results? At the generation layer: did the model actually use that chunk, or fall back on its own knowledge? The first layer is measured automatically, the second needs a human read. I generally prefer to fix retrieval first and only tune the prompt afterward — the best prompt can't fix a wrong chunk.

One more note: as Pinecone reminds us, RAG is just one optimization; there are others too, like query rewriting, chunk expansion, and knowledge graphs. If you have a solid baseline measurement, you can see which of these works on your own data; without it, they all look equally plausible.

FAQ

What is RAG and how does it work?

RAG is a technique where a language model finds relevant documents from an external source before generating an answer, and adds them to the prompt. It works in four steps: bringing data into the system (ingestion), finding documents relevant to the query (retrieval), combining those documents with the prompt (augmentation), and the model generating the answer (generation).

What's the difference between embedding and classic keyword search?

Classic keyword search only finds exactly matching words. Embedding-based search converts the semantic content of text into a vector and finds texts that are close in meaning, because the distance between two vectors measures relatedness. This is why a search for "car" can also find a document that mentions "automobile."

Why do vector search results come back irrelevant?

One of the most common reasons is using only dense (embedding-based) search — this method catches paraphrasing but can miss words or identifiers that need an exact match; hybrid search (dense + sparse) closes this gap. Another reason is the wrong chunk size: chunks that are too large or too small lower search precision.

What's the simplest way to set up RAG in a small project?

Split your documents into chunks, convert each chunk into a vector with an embedding model (e.g. text-embedding-3-small), store them in a simple list or a small vector database, find the closest chunks with cosine similarity when a query comes in, and hand them to the model with a prompt template that includes a grounding instruction.

Update (September 2026)

This article was originally written with the APIs and models available on November 12, 2024. Since then, RAG's core mechanics (chunk → embed → retrieve → prompt) haven't changed, but it has matured along three axes.

On standardization, Anthropic announced the Model Context Protocol (MCP) on November 25, 2024 — an open protocol standardizing how LLMs connect to data sources; the biggest infrastructure shift, moving retrieval from custom integrations toward a multi-step structure interwoven with tool calling. OpenAI also shipped a built-in file_search tool with the Responses API on March 11, 2025; retrieval is now usable as a "hosted" tool without setting up a separate vector store.

On verifiability, Anthropic introduced the Citations API, made available on January 23, 2025 (expanded to Bedrock on June 30, 2025); it automatically adds citations to exact passages in source documents. In practice: instead of hand-writing the "QUESTION / CONTEXT" template above and hoping the model follows the grounding instruction, you can track at the API level which answer came from which passage — partly automating the "5 common mistakes" problem of verifying whether an answer is truly grounded in the CONTEXT.

On retrieval quality, a new generation of embedding models arrived: Cohere released Embed 4 on April 15, 2025, unifying text and image in one embedding space; Google shipped gemini-embedding-001 experimentally in March 2025, reaching general availability on July 14, 2025 — it has held a top spot on the MTEB multilingual leaderboard since that March launch. Hybrid (dense+sparse) search also became a common default: dense retrieval catches paraphrasing but can miss an exact-matching word or identifier, and sparse search closes that gap.

Conclusion

RAG's foundation boils down to three steps: split the text into chunks, convert each chunk to a vector with an embedding model, run a similarity search on the query, and hand the found chunks to the model with a grounding instruction. This mechanic hasn't changed; what's changed is the tooling around it — standard protocols like MCP, covered in the Update section above, and vector databases with hybrid search support.

If you're starting a small RAG project, validate your chunk size and embedding model with a small test set first, then move on to agentic, multi-step retrieval flows. Not skipping the grounding instruction in your prompt template is probably the single biggest step against hallucination — more on prompt templates in our archive piece. Build the habit of actually verifying retrieval results early, before production; on how retrieval connects to agent frameworks, see that article.

Sources

• Retrieval-Augmented Generation (RAG) — Pinecone — RAG definition, its four components, and the augmented prompt template.
• Chunking Strategies — Pinecone — the chunk size trade-off and chunking definition.
• Embeddings — OpenAI Platform Docs — embedding definition, dimension, and cosine similarity usage.
• New embedding models and API updates — OpenAI — the text-embedding-3 announcement and MTEB/MIRACL scores.
• Introducing Contextual Retrieval — Anthropic — Contextual Embeddings, the BM25 combination, and performance gains.
• Model Context Protocol — Anthropic — the MCP announcement, November 25, 2024.
• API release notes — Anthropic — the date Citations capability shipped in the API (January 23, 2025).
• Gemini Embedding available in the Gemini API — Google Developers Blog — the gemini-embedding-001 GA announcement, July 14, 2025.
• Changelog — OpenAI Developers — the release date of the Responses API and the built-in file search tool (March 11, 2025).
• Introducing Embed 4 — Cohere — the multimodal Embed 4 announcement, April 15, 2025.
• Introducing Citations on the Anthropic API — Anthropic — the Citations API announcement and its June 30, 2025 Amazon Bedrock update.