Home / Learn / Data & AI / Offline AI / Lesson 12.1
All lessons

Module 12: Private document search (RAG), fully offline · Lesson 12.1 · about 20 min

Why the model cannot read your files by itself

A model only knows what it learned in training and what is in its context window. Learn why that leaves your own files out, and how retrieval (RAG) fixes it without uploading anything.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 12 · about 20 minutes · read first: 3.3 and 11.2. You’ll be able to explain why a model cannot see your files, and what each step of a private document search pipeline does.

A model only knows its training and the text in front of it, so to answer from your files you first look up the few right passages and hand it just those.

1. Why this matters

The most common reason people want local AI is private documents: contracts, patient letters, research notes, policies. “Ask my documents” is also the most asked-for feature at work. This module builds it, fully offline, in about 150 lines of Python.

2. What a model actually knows

A model knows two things:

  1. What it learned in training. Frozen on the day training ended (its knowledge cut-off). Your files were not in it.
  2. What is in its context window right now (Lesson 3.3): the system prompt, the chat so far, and anything you paste.

That is all. It cannot open a folder, and it does not remember last week’s chat unless the app pastes it back in. So “make the model know my files” really means “get the right text into the context window at the right moment”.

Picture a very well-read consultant locked in a room with no filing cabinet. They know a lot in general, but about your company they know only the papers you slide under the door.

“If I chat with it about my files, it learns them.” Chatting never changes the model’s weights. It “remembers” only what is pasted into the current conversation. Close the chat and that knowledge is gone, unless an app stores it and pastes it back in next time.

3. Why not paste everything?

A 300-page policy manual is about 150,000 tokens. On a laptop with an 8,192-token context, it does not fit. Even on a big machine with a 128K window, every question would make the model read the whole manual again (prefill, Lesson 3.5), which can take minutes, and the KV cache (Lesson 8.4) would eat gigabytes. Models also pay less attention to facts buried in the middle of a very long context (“lost in the middle”).

4. The fix: look it up first (RAG)

Retrieval-augmented generation (RAG) works like a good librarian:

  1. Once, ahead of time: read every file, cut it into short pieces (chunks, Lesson 12.3), turn each chunk into a list of numbers that captures its meaning (an embedding, Lesson 12.2), and store them (Lesson 12.4).
  2. Each question: turn the question into numbers the same way, find the 5 or so chunks whose numbers are closest, and paste only those into the prompt with an instruction: “answer from these sources and cite them” (Lesson 12.5).

The question. Someone asks: "How many days of leave do new staff get?" The handbook has 300 pages, about 150,000 tokens, far too many to paste.

Turn the question into numbers. The embedding model turns the question into a vector of 768 numbers that captures its meaning.

Find the closest chunks. The index holds 400 chunks of the handbook, each already turned into numbers. The 5 closest to the question win, including "page 12: Staff get 20 days of paid vacation a year".

Build a small prompt. Instructions (100 tokens) + 5 chunks of 400 tokens + the question (12 tokens) is about 2,100 tokens: roughly 70 times less than the whole handbook.

Answer with a source. The model reads only those chunks and writes: "New staff get 20 days of paid vacation a year [1]", with "[1] handbook.pdf, page 12" underneath.

The model now answers from your text, with page numbers, and the prompt stays small. Every step runs on your machine: the embedding model and the chat model both run in Ollama, and the index is a file on your disk.

Maths and principle

Principle: retrieval shrinks the text the model must read from the whole library to the few pieces that matter for this question.

Equation:

prompt tokens≈instructions+k×chunk size+question

With k = 5 chunks of 400 tokens: about 2,000 + 150 tokens, whatever the size of the library.

Analogy: an open-book exam. You do not reread the whole book for each question; you use the index to find two pages, then read those carefully. The index costs effort once; every lookup after that is cheap. Where it breaks: if the index points to the wrong pages, the best reader in the world still answers wrongly. Retrieval quality limits answer quality.

Try it: a library of 4,000 chunks; you send the top 5. What fraction of the library does the model read per question? (5 ÷ 4,000 = 0.125%.)

k
how many chunks the search hands to the model for each question, often 3 to 8
chunk size
how long each piece of a document is, often 250 to 650 tokens
instructions + question
the fixed part of the prompt: "answer only from these sources and cite them", plus what was asked
Physical meaning
RAG is an open-book exam with a good index. The prompt stays the size of a few pages whether your library is a hundred pages or a million, so a laptop can answer from a whole office's files, as long as the index points to the right pages.

5. Worked example

Made-up numbers. A small accounting office has 1,200 PDF pages of client letters and tax notes. Pasting it all: about 600,000 tokens, impossible on their 16 GB laptop. With RAG: about 2,400 chunks indexed once (12 minutes overnight on the laptop), then each question sends about 2,300 tokens and gets an answer in 8 to 15 seconds, with “[2] client-letters/2025-03.pdf, page 4” under it. No page ever leaves the office.

6. Practice

  1. Estimate how many tokens your own largest document has (pages × about 500).
  2. Would it fit in your model’s context setting from Lesson 10.3?
  3. Think of three questions you would ask your own files. Write them down: you will test with them in Lesson 12.7.
Answers
  1. A 40-page report is about 20,000 tokens.
  2. Not in 8,192; just in 32,768 if your memory allows it (check with the KV cache calculator in 8.4). RAG avoids the problem.
  3. Keep them specific (“What notice period does the Smith lease need?”); vague questions are hard for any search.

New words: RAG or retrieval-augmented generation (look up passages, then answer from them), chunk (a short piece of a document), knowledge cut-off (the date a model’s training data ends).

  • A model knows only its training data (frozen at its cut-off) and what is in its context window right now.
  • Pasting whole libraries does not fit, is slow to read every time, and facts in the middle get less attention.
  • RAG: index once (read, chunk, embed, store); for each question, find the closest chunks and answer from them with citations.
  • The prompt stays small (instructions + k × chunk size + question) however big the library grows.
  • Every step can run on your own machine, and answer quality is limited by retrieval quality.

Check yourself

1Your laptop model has never seen your company handbook. Where can it get the handbook facts from?
2Why not paste all 2,000 pages into every question?
3In RAG, what does the R (retrieval) step do?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Prices and model names change fast: check the official source before you buy.