The Course

Eight lessons. Each one is: a claim → a command that tests it → the number you should see → what it means → what to do about it.

Rules:

1. Predict before you run. Write the number down. Being wrong on record is the entire mechanism by which this becomes skill. Reading the answer teaches nothing.
2. Change one thing. The harness is built so the context strategy is the only variable. Change two, learn zero.
3. Lessons 1–6 cost no API calls. That's deliberate — the bottleneck is loop speed, not model access.

Time: ~4 hours to work through. Lesson 5 is the one that matters most.

Lesson 0 — Orientation (5 min) done

cd lab
python3 llm.py gemini          # smoke test
python3 corpus_hard.py         # what the task is

144 docs, ~7.7k tokens if you dump all of it. 24 questions of two kinds:

• trap — "What is the current deploy region for Project Peregrine?" A superseded

2023 doc exists that is lexically better than the current one: it repeats the project name, says "current", and has no filler diluting its term frequency.

• multihop — "What is the budget code charged for Project Peregrine?" The answer

lives in a doc that never mentions Peregrine. You must learn the portfolio name first, then look up the portfolio.

Everything downstream is about those two failure shapes.

Lesson 1 — Accuracy is a compound number. Decompose it. (15 min) done

Claim: accuracy = (fact entered the window) × (model used it). Only the first factor is yours. Measure it separately or you'll spend days tuning prompts against a ceiling you can't see.

Predict: at budget 800, what fraction of questions have the gold string present in S1_topk's assembled context?

python3 recall.py --budget 800

  strategy              recall  multihop    trap
  S0_dump_all              46%        0%     92%
  S1_topk                  71%       42%    100%
  S0b_dump_full           100%      100%    100%

What it means: S1's accuracy can never exceed 71%. It measured 75% end-to-end earlier — which is above its own recall, meaning the model guessed right a couple of times. That's not a win, it's noise you'd mistake for signal.

The habit: measure assembly recall before you touch a prompt. Zero API calls, one second. If recall is 40%, prompt engineering is a waste of your afternoon.

Lesson 2 — Truncation is the #1 silent killer (20 min) done

Predict: S0 dumps docs in corpus order up to the budget. At 800 tokens its multihop recall is 0%. What is it at 1600?

python3 recall.py
python3 inspect_ctx.py --kind multihop --i 1 --rank 5

Q (multihop): What is the budget code charged for Project Vireo?
GOLD: BC-2781

retrieval ranking:
   2023-04-11  Vireo-region-old        <- stale doc ranks FIRST
   2026-02-19  Vireo-ownership
   2026-02-19  portfolio-Everglade-finance
   ...

strategy              gold in ctx  ctx_tok
S0_dump_all                    NO      780
S1_topk                       YES      790

What it means: S0 scored 8% end-to-end. Not because the model is weak — because the answer was never in the window. It correctly said UNKNOWN.

The habit: the first question on any context bug is "was the fact even there?" inspect_ctx.py answers it in one second with zero API calls. In production this is the majority of "the model is dumb" tickets.

Lesson 3 — A broken component can hide behind a good score (30 min) done

Predict: on trap questions, how often does the stale doc rank above the current one in S1_topk? Trap accuracy was 100%, so presumably rarely.

python3 recall.py --rank

trap questions where BOTH docs retrieved : 12/12
  ...and the STALE one ranks HIGHER      : 12/12

Every single time. Trap accuracy is 100% by luck: both docs fit in an 800-token budget, and the model resolves the conflict using the dates. The ranker is completely wrong and the metric says everything is fine.

Now shrink the budget so only one doc fits:

python3 recall.py --budget 300

Multihop recall collapses to 0% for every retrieval strategy.

What it means: end-to-end accuracy is a lagging indicator. It hides broken components whenever slack elsewhere in the system compensates. Remove the slack — tighten the budget, add distractors — and the latent bug surfaces.

This is why you need component metrics, not just end-to-end scores.

→ Do exercise E1 in exercises.py. Target: 0/12 stale-above-current at budget 300. Verify with python3 check.py E1.

Lesson 4 — Your obvious improvements probably do nothing (20 min) done

S1 → S2 (recency sort) → S3 (explicit "prefer later dates" framing). Both changes are things every RAG guide tells you to do.

Predict: how many accuracy points does S3 buy over S1?

S1_topk                75%       50%      100%
S2_topk_recency        75%       50%      100%
S3_framed              75%       50%      100%

Zero. All three. The framing targets stale-vs-current confusion, and traps were already at 100% — there was no headroom in the failure mode it addresses.

What it means: an intervention only helps where the corresponding failure mode is actually costing you. Without the per-kind breakdown, the aggregate would have looked plausible and you'd have shipped three no-ops and believed in them.

The habit: before building a fix, find the failure it targets in your breakdown and confirm it's non-zero. If you can't point at the number it will move, don't build it.

Lesson 5 — The wall that a bigger window cannot fix (45 min) ★ done

This is the most important lesson in the lab.

python3 inspect_ctx.py S1_topk --kind multihop --i 1 --show

The context contains: "Project Vireo is owned by portfolio Pinecrest." It does not contain the Pinecrest finance doc — because the word "Pinecrest" does not appear in the question. BM25 cannot retrieve on a term that doesn't exist yet.

Nor can embeddings. Nor can a 10M-token window at real corpus scale. The missing information is not relevance, it's ordering: hop B's query is a function of hop A's result.

The only fix is another turn:

S1_topk (1 call,  785 ctx tok)   multihop 50%
S5_iterative (3 calls, 345 tok)  multihop 67%

What it means: this is the actual boundary between "RAG" and "agent". Not a buzzword — a structural property of the task. If the next query term is unknowable until you read the current result, you need a loop. If it isn't, a loop is wasted money.

Diagnostic you can apply at work: can I write down every search I'll need before seeing any results? Yes → one-shot retrieval. No → you need turns.

→ Do exercise E2. Beat S1's 42% multihop recall with zero LLM calls — extract the entity with a regex and issue a second retrieval. Target ≥90% at ≤800 tokens. python3 check.py E2

The lesson inside the lesson: S5 spends 2 extra LLM calls on a planner. E2 gets the same structural win for free. Reach for a loop when you need one; don't reach for a model when a regex does it.

Lesson 6 — Compression is the highest-leverage move, and the easiest to botch (30 min) done

S4_compressed    ctx 173 tok    multihop 33%    (S1: 785 tok, 42%)

S4 is 4.5× cheaper and worse — it compressed away the very sentence carrying the second hop.

What it means: compression that drops recall isn't compression, it's deletion. The correct order is recall first, then squeeze. Lossy compression is where agents die silently, because the loss shows up three steps later as an inexplicable wrong answer.

→ Do exercise E3. Take your E2 solution to ≤200 context tokens while keeping ≥90% multihop recall. Reference solution: 100% at 109 tokens — 7× smaller than S1 and 2.4× more accurate. python3 check.py E3

Lesson 7 — Your grader is a measuring instrument, and it's wrong (30 min) done

The original grader used gold in pred. It scored this correct:

gold: 'Luca Moretti'
pred: 'Yuki Tanaka, Sana Iqbal, Nils Eriksen, Mira Castellanos, Luca Moretti, Lene Brandt, Emre Demir'

A grader that rewards listing every candidate rewards exactly the behaviour context engineering exists to prevent. Fixed grader, replayed from disk:

python3 regrade.py --diff

S4_compressed   67% → 58%      multihop 33% → 17%

No re-run, no new API calls. That's what JSONL-logging every trial buys you: when you discover the instrument was miscalibrated — and you will — you replay history instead of re-arguing it from memory.

What it means: every metric you write will be gameable, usually by your own optimization, silently. Attacking your own metric is not paranoia; it is eval design.

→ Do exercise E4. Write a strategy that scores well while doing something you'd never ship. Then fix evaluate_hard.grade() so it fails. Repeat until you can't cheat it.

Lesson 8 — Which of your conclusions are real? (30 min, uses API) done

python3 evaluate_hard.py --provider groq --model openai/gpt-oss-120b --n 12

Compare per-kind numbers against the Gemini run in README.md.

Conclusions that survive a model swap were about context engineering. Ones that don't were about one model's quirks, and you should stop believing them. Structural findings (Lessons 2, 5, 6) will transfer. Margin-level findings (a 3-point difference between S2 and S3) will not.

The habit: before you generalize a result, change something you believe is irrelevant and check the result survives.

The eight rules, compressed

Context engineering

1. Assembly recall is your accuracy ceiling. Measure it first, separately, for free.
2. When something looks broken, ask "was the fact even in the window?" before anything else.
3. Compress only after recall is high. Deletion masquerading as compression is the classic silent failure.
4. An intervention only pays where its failure mode is currently costing you.
5. If the next query term depends on the current result, you need a turn, not a bigger window.
6. Budget is a policy about what deserves the window, not a limit on what fits.

Evals

1. Never report one aggregate. Break down by failure kind; grade into mechanisms (correct/stale/hedged/unknown/wrong/error), never binary.
2. Log every trial to disk. Verify the eval can fail before trusting that it passed. A saturated benchmark measures nothing.

Doing this on your own system

The transferable artifact is the shape, not the corpus:

Start with 30 labelled examples. That's enough to catch the failures in Lessons 2, 3, and 5 — which is most of what actually breaks in production.