@omarUpdated Nov 3, 2026

Building AI Agents: tools, memory and evals

For software engineers who already call an LLM API. In 10 weeks, hand-roll the agent loop, add retrieval, memory, an MCP server, evals and safety, and ship a deployed agent with tracing, cost caps and a kill switch.

Following gives you your own copy, dated from today. Free to start.

4 phases · 10 weeks

About this roadmap

Who this is for

You are a software engineer who has called an LLM API and watched a demo agent work once. You want agents that work every time: tested, observable, cheap enough to run, and safe to give real tools. This is a ten-week, framework-agnostic path. You hand-roll the loop first, so that when you meet a framework you can tell what it saves you and what it hides.

What you will have built

  • Week 3: a command-line agent with three tools, a step limit and a budget, in about 150 lines of plain Python or TypeScript.
  • Week 6: an MCP server that wraps your retrieval and notes, tested with the Inspector and used by your own agent.
  • Week 8: an eval suite built from real traces, with code graders, one calibrated LLM judge, and a CI gate that blocks regressions.
  • Week 10: a deployed agent with tracing, per-run and per-day cost caps, approval gates for risky actions, and a kill switch you have actually pulled.

How the weeks work

Each week has one idea, one thing to build and a definition of done. Read the sources in the order given, then build before you read more. The notes attached to weeks are short essays with runnable code; the flashcards are the questions you should still be able to answer in a month.

Evals start in week 1, not week 8. You write a 15-case golden set for your first prompt and you keep adding to it. By week 8 you are not starting an eval practice, you are making an existing one rigorous.

Failure modes to watch

  • Framework first. If you cannot write the loop in 60 lines, a framework will hide the bug you are about to have.
  • More agent than you need. A fixed workflow with one model call per step beats an autonomous agent on cost, latency and predictability. Earn each degree of autonomy.
  • Vibes instead of numbers. "It seems better" is not a result. Every prompt, tool and model change gets a score on the golden set.
  • Trusting tool output. Anything the agent reads from outside your system is untrusted input, and it can carry instructions. Week 9 is not optional.
  • No ceiling. An agent without a step limit, a token budget and a kill switch is a production incident waiting for a trigger.

Map

Every week on one canvas. Open a week to see its goals and resources.

What’s inside

10 weeks · 49 goals

Phase 1 · Model, prompt, loopWeeks 1–3 · 15 goals
WK 1LLM APIs done right5 goals

Treat the model as an unreliable remote function: typed in, typed out, metered and retried.

Goals

  • Compute by hand the cost of one call with 2,000 input and 500 output tokens, then confirm it against the usage field
  • Build llm.py with streaming, a timeout, and retry with backoff and jitter
  • Return a schema-validated object and handle the case where validation still fails
  • Write 15 golden cases for one extraction task and score them with a script
  • Explain in one paragraph why output tokens dominate latency and input tokens dominate loop cost
WK 2Prompting as engineering5 goals

Prompts are code: versioned, scored against a golden set and priced.

Goals

  • Write four prompt versions and score each on the golden set, keeping a results table
  • Add 3 few-shot examples chosen from your errors, then check whether they helped or just memorized
  • Order the prompt stable-first, enable caching and measure the hit rate across 20 calls
  • Put prompts in versioned files and log the prompt version with every call
  • Write down two changes that would silently break your cache
WK 3Project: tool use and a CLI agent5 goals

Build the agent loop by hand, with three tools, a step limit and a budget.

Goals

  • Write the loop: call, run tool_use blocks, return tool_result blocks, stop on a final answer
  • Implement three tools with strict schemas and instructive error messages
  • Add MAX_STEPS, a dollar budget and a per-tool timeout, and prove each one triggers
  • Write 10 tasks with expected outcomes and a runner that reports pass rate, steps and cost
  • Read 3 failing traces and rewrite one tool description or error message that caused a failure
Phase 2 · What the agent knowsWeeks 4–6 · 15 goals
WK 4Retrieval for agents5 goals

Put the right few thousand tokens in front of the model, and prove you did.

Goals

  • Chunk the corpus on structure (headings, code blocks) with a small overlap and keep the source for every chunk
  • Write 30 questions with their answer chunk ids
  • Implement BM25, dense search and RRF, and report recall@5 for each
  • Compute RRF by hand for two documents and match your code
  • Give the agent search_docs and read_doc tools and cap the number of searches per task
WK 5Memory and context management5 goals

The window is a budget: trim, clear, summarize and write notes outside it.

Goals

  • Compute when compaction fires for your window size and tokens per step, then confirm in a run
  • Implement a scratchpad note tool and re-inject its contents at the end of the prompt
  • Clear stale tool results while keeping every tool_use and tool_result pair valid
  • Implement summary compaction with fixed headings and a cheap model
  • Run a 40-step task with compaction on and off and compare success and cost
WK 6Project: an MCP server5 goals

Expose your tools through the Model Context Protocol and use them from a real client.

Goals

  • Build the server with three tools and one resource, with bounded outputs and strict schemas
  • Call every tool from the Inspector and save a transcript of the raw JSON-RPC messages
  • Add an MCP client to your CLI agent and run your 10 tasks through the server
  • Connect the server to a second client and fix one description the model misread
  • Write down what the server must never do on a user's machine and enforce it in code
Phase 3 · Patterns and proofWeeks 7–8 · 9 goals
WK 7Workflows, agents, multi-agent4 goals

Choose the least autonomy that works: chains, routers and orchestrators before open-ended agents.

Goals

  • Implement prompt chaining, routing, orchestrator-workers and evaluator-optimizer in under 60 lines each
  • Run all four designs on 20 cases and tabulate accuracy, cost and latency
  • Port one design to a framework and write down what it saved and what it hid
  • Write one paragraph on when a single agent beats several, using your numbers
WK 8Project: evals and tracing5 goals

Turn vibes into a suite you can trust: traces, error analysis, graders and a CI gate.

Goals

  • Log a full trace for every run: messages, tool calls with arguments, latency, tokens, cost, prompt version
  • Read 50 traces, write a one-line failure note on each and group them into a taxonomy
  • Build 40 cases from the taxonomy with code graders for tool use, budget and content
  • Hand-label 30 outputs, run a judge on them and compute TPR and TNR
  • Add a CI job with a confidence interval and a regression rule, then prove it fails on a broken prompt
Phase 4 · Safe and shippedWeeks 9–10 · 10 goals
WK 9Safety and permissions5 goals

Assume the model will be fooled and limit what a fooled model can do.

Goals

  • Draw your agent's private data, untrusted content and external communication legs, and break one
  • Write 10 injection attacks and run them against the agent, recording which succeed
  • Add permission tiers, an egress allowlist and approval that shows the exact arguments
  • Run code tools in a container with no network, no secrets and a read-only mount
  • Add the 10 attacks to the eval suite and make a failure block the CI job
WK 10Capstone: ship an agent5 goals

Deploy one agent that does real work, with evals, tracing, cost caps and a kill switch.

Goals

  • Choose one narrow job and write its success criteria and 20 eval cases before coding
  • Deploy it with tracing, a per-run cap, a per-day cap and a step limit
  • Add a kill switch an operator can flip without a deploy and approval for irreversible actions
  • Run a kill-switch drill and an injection drill, and record what you fixed
  • Write the runbook and a one-page write-up of what failed in the first week