Building AI Agents: tools, memory and evals
For software engineers who already call an LLM API. In 10 weeks, hand-roll the agent loop, add retrieval, memory, an MCP server, evals and safety, and ship a deployed agent with tracing, cost caps and a kill switch.
Following gives you your own copy, dated from today. Free to start.

About this roadmap
Who this is for
You are a software engineer who has called an LLM API and watched a demo agent work once. You want agents that work every time: tested, observable, cheap enough to run, and safe to give real tools. This is a ten-week, framework-agnostic path. You hand-roll the loop first, so that when you meet a framework you can tell what it saves you and what it hides.
What you will have built
- Week 3: a command-line agent with three tools, a step limit and a budget, in about 150 lines of plain Python or TypeScript.
- Week 6: an MCP server that wraps your retrieval and notes, tested with the Inspector and used by your own agent.
- Week 8: an eval suite built from real traces, with code graders, one calibrated LLM judge, and a CI gate that blocks regressions.
- Week 10: a deployed agent with tracing, per-run and per-day cost caps, approval gates for risky actions, and a kill switch you have actually pulled.
How the weeks work
Each week has one idea, one thing to build and a definition of done. Read the sources in the order given, then build before you read more. The notes attached to weeks are short essays with runnable code; the flashcards are the questions you should still be able to answer in a month.
Evals start in week 1, not week 8. You write a 15-case golden set for your first prompt and you keep adding to it. By week 8 you are not starting an eval practice, you are making an existing one rigorous.
Failure modes to watch
- Framework first. If you cannot write the loop in 60 lines, a framework will hide the bug you are about to have.
- More agent than you need. A fixed workflow with one model call per step beats an autonomous agent on cost, latency and predictability. Earn each degree of autonomy.
- Vibes instead of numbers. "It seems better" is not a result. Every prompt, tool and model change gets a score on the golden set.
- Trusting tool output. Anything the agent reads from outside your system is untrusted input, and it can carry instructions. Week 9 is not optional.
- No ceiling. An agent without a step limit, a token budget and a kill switch is a production incident waiting for a trigger.
Map
Every week on one canvas. Open a week to see its goals and resources.
What’s inside
10 weeks · 49 goals
Phase 1 · Model, prompt, loopWeeks 1–3 · 15 goals
WK 1LLM APIs done right5 goals
Treat the model as an unreliable remote function: typed in, typed out, metered and retried.
Goals
- Compute by hand the cost of one call with 2,000 input and 500 output tokens, then confirm it against the usage field
- Build llm.py with streaming, a timeout, and retry with backoff and jitter
- Return a schema-validated object and handle the case where validation still fails
- Write 15 golden cases for one extraction task and score them with a script
- Explain in one paragraph why output tokens dominate latency and input tokens dominate loop cost
WK 2Prompting as engineering5 goals
Prompts are code: versioned, scored against a golden set and priced.
Goals
- Write four prompt versions and score each on the golden set, keeping a results table
- Add 3 few-shot examples chosen from your errors, then check whether they helped or just memorized
- Order the prompt stable-first, enable caching and measure the hit rate across 20 calls
- Put prompts in versioned files and log the prompt version with every call
- Write down two changes that would silently break your cache
WK 3Project: tool use and a CLI agent5 goals
Build the agent loop by hand, with three tools, a step limit and a budget.
Goals
- Write the loop: call, run tool_use blocks, return tool_result blocks, stop on a final answer
- Implement three tools with strict schemas and instructive error messages
- Add MAX_STEPS, a dollar budget and a per-tool timeout, and prove each one triggers
- Write 10 tasks with expected outcomes and a runner that reports pass rate, steps and cost
- Read 3 failing traces and rewrite one tool description or error message that caused a failure
Phase 2 · What the agent knowsWeeks 4–6 · 15 goals
WK 4Retrieval for agents5 goals
Put the right few thousand tokens in front of the model, and prove you did.
Goals
- Chunk the corpus on structure (headings, code blocks) with a small overlap and keep the source for every chunk
- Write 30 questions with their answer chunk ids
- Implement BM25, dense search and RRF, and report recall@5 for each
- Compute RRF by hand for two documents and match your code
- Give the agent search_docs and read_doc tools and cap the number of searches per task
WK 5Memory and context management5 goals
The window is a budget: trim, clear, summarize and write notes outside it.
Goals
- Compute when compaction fires for your window size and tokens per step, then confirm in a run
- Implement a scratchpad note tool and re-inject its contents at the end of the prompt
- Clear stale tool results while keeping every tool_use and tool_result pair valid
- Implement summary compaction with fixed headings and a cheap model
- Run a 40-step task with compaction on and off and compare success and cost
WK 6Project: an MCP server5 goals
Expose your tools through the Model Context Protocol and use them from a real client.
Goals
- Build the server with three tools and one resource, with bounded outputs and strict schemas
- Call every tool from the Inspector and save a transcript of the raw JSON-RPC messages
- Add an MCP client to your CLI agent and run your 10 tasks through the server
- Connect the server to a second client and fix one description the model misread
- Write down what the server must never do on a user's machine and enforce it in code
Phase 3 · Patterns and proofWeeks 7–8 · 9 goals
WK 7Workflows, agents, multi-agent4 goals
Choose the least autonomy that works: chains, routers and orchestrators before open-ended agents.
Goals
- Implement prompt chaining, routing, orchestrator-workers and evaluator-optimizer in under 60 lines each
- Run all four designs on 20 cases and tabulate accuracy, cost and latency
- Port one design to a framework and write down what it saved and what it hid
- Write one paragraph on when a single agent beats several, using your numbers
WK 8Project: evals and tracing5 goals
Turn vibes into a suite you can trust: traces, error analysis, graders and a CI gate.
Goals
- Log a full trace for every run: messages, tool calls with arguments, latency, tokens, cost, prompt version
- Read 50 traces, write a one-line failure note on each and group them into a taxonomy
- Build 40 cases from the taxonomy with code graders for tool use, budget and content
- Hand-label 30 outputs, run a judge on them and compute TPR and TNR
- Add a CI job with a confidence interval and a regression rule, then prove it fails on a broken prompt
Resources
- A Field Guide to Rapidly Improving AI Productsblog
- Using LLM-as-a-Judge For Evaluation: A Complete Guideblog
- Evaluating the Effectiveness of LLM-Evaluatorsblog
- Demystifying evals for AI agentsarticle
- tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainspaper
- Semantic conventions for generative AIarticle
Phase 4 · Safe and shippedWeeks 9–10 · 10 goals
WK 9Safety and permissions5 goals
Assume the model will be fooled and limit what a fooled model can do.
Goals
- Draw your agent's private data, untrusted content and external communication legs, and break one
- Write 10 injection attacks and run them against the agent, recording which succeed
- Add permission tiers, an egress allowlist and approval that shows the exact arguments
- Run code tools in a container with no network, no secrets and a read-only mount
- Add the 10 attacks to the eval suite and make a failure block the CI job
WK 10Capstone: ship an agent5 goals
Deploy one agent that does real work, with evals, tracing, cost caps and a kill switch.
Goals
- Choose one narrow job and write its success criteria and 20 eval cases before coding
- Deploy it with tracing, a per-run cap, a per-day cap and a step limit
- Add a kill switch an operator can flip without a deploy and approval for irreversible actions
- Run a kill-switch drill and an injection drill, and record what you fixed
- Write the runbook and a one-page write-up of what failed in the first week
Notes
8 published notes
- Nov 2, 2026
- Nov 2, 2026
- Nov 2, 2026
When retrieval beats a longer context
Nov 2, 2026Designing tools a model can use
Nov 2, 2026- Nov 2, 2026
- Nov 2, 2026