Build a Local RAG Notes Search With Ollama and Python

Notes pile up faster than anyone can search them. A folder of Markdown files from three years of projects holds the answer to nearly any question you could ask, and plain keyword search fails the moment you cannot remember the exact words you used.

Retrieval-augmented generation, introduced in a 2020 paper by Patrick Lewis and colleagues (Lewis et al.), fixes this by splitting the job in two. A search step pulls the relevant passages; a language model then writes an answer from those passages alone. Everything below runs on one machine through Ollama, so nothing leaves it. The setup follows the Ollama Python library and the embeddings API, and the Ollama install itself is covered in x9m8’s guide to running a model locally with Ollama.

What you need

Install Ollama, then pull two models: one that turns text into vectors and one that writes answers. The Ollama docs list embeddinggemma, qwen3-embedding and all-minilm as recommended embedding models. The embeddinggemma library page describes a 300M-parameter model from Google, a 622MB download with a 2K context window, and it requires Ollama v0.11.10 or later. For the answering model, llama3.2 matches the earlier guide.

ollama pull embeddinggemma
ollama pull llama3.2
pip install ollama numpy

The Python library needs Python 3.8 or newer and a running Ollama server. One rule from the embeddings docs matters more than it looks: use the same embedding model for indexing and for querying. Vectors from different models live in different spaces, so mixing them produces scores that mean nothing.

Chunking, embedding and search in one file

The flow has two phases. Indexing runs once per batch of notes: read files, cut chunks, embed each chunk, save the vectors beside the text. Querying runs on every question: embed the question, score it against every stored vector, hand the best few chunks to the chat model. For a personal notes folder, a JSON file and a NumPy array are enough; skipping a vector database is a judgment call, not a benchmarked claim.

Chunking comes first. Whole files are too coarse, because one vector for a ten-page note blurs every topic in it together. The script below cuts text into overlapping character windows so a sentence that straddles a boundary still appears whole in at least one chunk. The 1,000-character size and 200-character overlap are starting points chosen for this example, not figures from any documentation.

import json
from pathlib import Path

import numpy as np
import ollama

EMBED_MODEL = "embeddinggemma"
CHAT_MODEL = "llama3.2"
INDEX = Path("index.json")


def chunk(text, size=1000, overlap=200):
    step = size - overlap
    return [text[i:i + size] for i in range(0, max(len(text) - overlap, 1), step)]


def build_index(folder):
    chunks, sources = [], []
    for path in Path(folder).rglob("*.md"):
        for piece in chunk(path.read_text(encoding="utf-8")):
            chunks.append(piece)
            sources.append(str(path))
    vectors = []
    for i in range(0, len(chunks), 32):
        batch = chunks[i:i + 32]
        vectors += ollama.embed(model=EMBED_MODEL, input=batch)["embeddings"]
    INDEX.write_text(json.dumps(
        {"chunks": chunks, "sources": sources, "vectors": vectors}))


def ask(question, k=4):
    idx = json.loads(INDEX.read_text())
    q = np.array(ollama.embed(model=EMBED_MODEL, input=question)["embeddings"][0])
    scores = np.array(idx["vectors"]) @ q
    top = np.argsort(scores)[::-1][:k]
    context = "\n\n".join(f"[{idx['sources'][i]}]\n{idx['chunks'][i]}" for i in top)
    reply = ollama.chat(model=CHAT_MODEL, messages=[
        {"role": "system", "content": "Answer only from the context. "
         "If the context does not contain the answer, say you don't know. "
         "Name the files you used."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ])
    return reply.message.content, [(idx["sources"][i], float(scores[i])) for i in top]

Three API details carry the design. ollama.embed accepts a single string or a list, and returns an embeddings field holding one vector per input; the batching loop above relies on that. The /api/embed endpoint returns L2-normalized, unit-length vectors, which makes a plain dot product equal to cosine similarity, the measure the Ollama docs recommend for most semantic search. And ollama.chat takes a list of messages with system, user, assistant or tool roles, with the reply readable as reply.message.content.

Call build_index("notes") once, then ask("What did I decide about the router firmware?") as often as you like. The function returns the answer plus the source files and scores behind it. Print both, every time.

Why it fails

A retrieval pipeline can look healthy while returning confident nonsense. Most failures trace back to a few places, and each leaves a visible trace if you look at the retrieved chunks instead of only the final answer.

Chunks that cut the answer in half

A fixed window ignores structure. A table, a code block or a numbered procedure can split so that the question lands on one half and the answer sits in the other. Chunks that start mid-sentence also embed poorly, since the vector reflects a fragment. Splitting on Markdown headings first, then falling back to windows for long sections, usually keeps related lines together; that is common practice rather than a documented rule.

Chunks that silently exceed the model

The embed endpoint has a truncate parameter that defaults to true, cutting the end of each input to fit the context length. With false, an oversize input returns an error instead. The default means a chunk that is too long loses its tail without a warning, and the lost part can never be retrieved. Even at one token per character, 1,000 characters fits inside a 2K window, so the script above stays safe, but a larger chunk size needs a check.

Retrieval that misses

Embeddings capture topic similarity, not exact strings. A question containing an error code, a part number or a name may rank unrelated prose above the one note that contains it. Fixes include adding keyword search alongside vectors and raising k. Raising k has a cost, covered next.

A model that ignores the middle

Stuffing in more context does not guarantee use of it. Liu and colleagues found that model performance is often highest when relevant information sits at the beginning or end of the input, and degrades when it sits in the middle, even for models built for long contexts (Lost in the Middle). Four short chunks beat twenty long ones.

Hallucination despite good retrieval

Even with the right passage in front of it, a model can blend in what it learned in training or round a detail into something plausible. The mechanisms behind that are laid out in x9m8’s piece on why AI chatbots hallucinate. Retrieval narrows the problem. It does not remove it.

A quick triage table

When an answer looks wrong, work through the pipeline in order. Checking the last stage first wastes time, because an error upstream guarantees one downstream.

Making failures visible

Opinion, not documentation: the cheapest defenses are the ones that expose the evidence. Print the file names and scores returned with every answer, so a score far below the others flags a weak match. Keep the instruction to say “I don’t know” in the system message, and read answers against the cited chunk, not against memory. Keep a short list of questions whose answers you already know, and rerun them after any change to chunk size, overlap or models. If scores drop, the change hurt retrieval.

The advice on size is judgment, so test it on your own notes: short, dense reference files tolerate small chunks, while narrative writing needs larger ones to keep an argument intact. Re-embedding is required whenever the embedding model changes, and the index file should record which model built it. A mismatch produces no error; it just degrades results.

Sources

1. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”: https://arxiv.org/abs/2005.11401
2. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”: https://arxiv.org/abs/2307.03172
3. Ollama documentation, Embeddings: https://docs.ollama.com/capabilities/embeddings
4. Ollama API reference (embed, chat): https://github.com/ollama/ollama/blob/main/docs/api.md
5. Ollama Python library: https://github.com/ollama/ollama-python
6. Ollama library, embeddinggemma: https://ollama.com/library/embeddinggemma