I take a ton of notes and have a lot of agent sessions, as many of you probably do.
You’ve probably also done the whole “Second Brain” thing, set up Obsidian, and let your notes pile up with no connections.
I finally had enough and got tired of trying all of the off-the-shelf and open source projects… I knew I needed embeddings and a system to use them.
Of course I started with the various embedding leaderboards, top blog posts, and the AI experts recommendations.
The model choice looked settled. Papers, tutorials, and best-practice posts kept pointing to all-MiniLM-L6-v2 on Hugging Face, also packaged as all-minilm in Ollama: small, fast, 384 dimensions, and common enough to become the default.
That is where a sensible builder stops. The leaderboard has a winner. The blog posts agree. Ship it.
But one question was still sitting there:
Whose data picked the winner?
The leaderboard used someone else’s corpus, someone else’s questions, and someone else’s definition of a good result. My notes had their own vocabulary, structure, and answer patterns. A best practice can be correct and still be wrong for the pile in front of you.
Or, we could do the work.
Let’s run the models against the notes they actually have to search.
What the leaderboard said
Most open-source retrieval guides point at the same handful of small models as the safe defaults. all-MiniLM-L6-v2 from SBERT is the most common pick.
The Hugging Face Massive Text Embedding Benchmark (MTEB) ranks it well on the categories it covers.
What the eval said
A small harness (embed-eval-on-your-vault.py) ran seven candidates on the actual note pile.
Round 1 on fifty-four notes, twenty-nine queries: all seven models looked great. Recall@5 above 90% for every candidate.
The leaderboard picked nomic-embed-text as the winner.
Five of seven hit a perfect recall@10. If the decision had been made there, that is what would have shipped.
Stage 1 on nine hundred and ninety-five notes, a hundred and fifty questions. The ranking did not shift.
It inverted.
The Bar
A saturated benchmark does not return an error. It returns a confident, specific, useless ranking, which is far more dangerous than no ranking at all. The leaderboard you copied was someone running a real test on their corpus. Copying their winner is copying their corpus, and you do not have their corpus. You have yours.
There were not enough documents for a wrong answer to have anywhere to hide.
The test was not measuring quality. It was measuring nothing.
And here is a detail worth a beat: all-minilm was the fastest model in the field, six to eight milliseconds a query, and nearly the worst at finding anything.
The fix costs almost nothing. The whole harness is a small script:
Run it on the real document set, not a test slice. Every candidate here ran through Ollama’s local embedding API, so the entire experiment cost zero dollars in API fees.
The boring input, more of your own real data, is what made the decision trustworthy.
There was no clever trick. There was just enough material for the truth to have somewhere to show up.
Four runs. One week. $23 dollars.
The full eval took roughly a week of evenings, four runs, and a Modal bill that came out to under twenty dollars in metered cost, all of it covered by free tier credits.
Modal Platform & Costs
Modal was a good fit for this experiment: serverless GPU access made the large batch practical, the service scaled to zero between runs, and the Starter plan’s monthly free credits absorbed the test.
A larger model on a bigger GPU would have been different. Use the right model on the right hardware for the right job.
The chunked A/B ran on Modal’s L4 GPU service to compare the same model across backends and the workload was over in minutes per run.
The entire embedding-service account across June and July, which includes the eval and some additional experiments, totaled $22.73 in metered cost.
Run it yourself
The full harness is a single Python file, no external dependencies beyond the embedding models you already have or can run locally.
It loads your notes, embeds them with each candidate, scores recall against a hand-built ground-truth set, and prints the ranking.
# Clone the harness, point it at your vault, run the eval
git clone https://github.com/kylebrodeur/embed-eval-on-your-vault.git
cd embed-eval-on-your-vault
python embed-eval-on-your-vault.py \
--corpus ~/path/to/your/notes \
--queries queries.example.json \
--models embeddinggemma,nomic-embed-text,bge-large,all-minilm \
--out results.csv
Ground truth is the part that does not scale. You need real questions a real person would ask, and a real answer in the corpus.
The numbers in this post are committed. The four eval runs that produced them are a public dataset:
The dataset contains one CSV for each run:
the 54-note baseline,
the 995-note model comparison that flipped the ranking,
the 996-note validation run that confirmed the new order,
and the chunked pass used for the whole-note versus chunked A/B.
Clone the dataset, run the simple harness against it, and the inverted ranking reproduces.
The data is the receipt.
What this means
The lesson is not “embeddinggemma in Ollama is great.” The model will get superseded in a year.
The lesson is the shape of the decision: copying their winner is copying their corpus, and you do not have their corpus.
Do your own evals.
If you build agents on top of retrieval, the stakes rise.
A bad retrieval does not stay a search error. It becomes an answer grounded in the wrong source, delivered with the same confidence as the truth.
A benchmark is only as honest as it is hard.