# Building AI Agents: tools, memory and evals

By @omar · 10 weeks

> For software engineers who already call an LLM API. In 10 weeks, hand-roll the agent loop, add retrieval, memory, an MCP server, evals and safety, and ship a deployed agent with tracing, cost caps and a kill switch.

## About

### Who this is for

You are a software engineer who has called an LLM API and watched a demo agent work once. You want agents that work every time: tested, observable, cheap enough to run, and safe to give real tools. This is a ten-week, framework-agnostic path. You hand-roll the loop first, so that when you meet a framework you can tell what it saves you and what it hides.

### What you will have built

- **Week 3:** a command-line agent with three tools, a step limit and a budget, in about 150 lines of plain Python or TypeScript.
- **Week 6:** an MCP server that wraps your retrieval and notes, tested with the Inspector and used by your own agent.
- **Week 8:** an eval suite built from real traces, with code graders, one calibrated LLM judge, and a CI gate that blocks regressions.
- **Week 10:** a deployed agent with tracing, per-run and per-day cost caps, approval gates for risky actions, and a kill switch you have actually pulled.

### How the weeks work

Each week has one idea, one thing to build and a definition of done. Read the sources in the order given, then build before you read more. The notes attached to weeks are short essays with runnable code; the flashcards are the questions you should still be able to answer in a month.

Evals start in week 1, not week 8. You write a 15-case golden set for your first prompt and you keep adding to it. By week 8 you are not starting an eval practice, you are making an existing one rigorous.

### Failure modes to watch

- **Framework first.** If you cannot write the loop in 60 lines, a framework will hide the bug you are about to have.
- **More agent than you need.** A fixed workflow with one model call per step beats an autonomous agent on cost, latency and predictability. Earn each degree of autonomy.
- **Vibes instead of numbers.** "It seems better" is not a result. Every prompt, tool and model change gets a score on the golden set.
- **Trusting tool output.** Anything the agent reads from outside your system is untrusted input, and it can carry instructions. Week 9 is not optional.
- **No ceiling.** An agent without a step limit, a token budget and a kill switch is a production incident waiting for a trigger.

## Model, prompt, loop

### Week 1: LLM APIs done right

Focus: Treat the model as an unreliable remote function: typed in, typed out, metered and retried.

Goals:

- Compute by hand the cost of one call with 2,000 input and 500 output tokens, then confirm it against the usage field
- Build llm.py with streaming, a timeout, and retry with backoff and jitter
- Return a schema-validated object and handle the case where validation still fails
- Write 15 golden cases for one extraction task and score them with a script
- Explain in one paragraph why output tokens dominate latency and input tokens dominate loop cost

Resources:

- [Messages API reference](https://platform.claude.com/docs/en/api/messages) (article, Request fields, response content blocks, usage)
- [Structured outputs (Claude)](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) (article)
- [Structured model outputs (OpenAI)](https://developers.openai.com/api/docs/guides/structured-outputs) (article)
- [Streaming messages](https://platform.claude.com/docs/en/build-with-claude/streaming) (article, Event types and the event flow)
- [Let's build the GPT Tokenizer](https://www.youtube.com/watch?v=zduSFxRajkE) (video, Skim: the opening and the section on why tokenization causes odd behavior)
- [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/) (blog)

### Week 2: Prompting as engineering

Focus: Prompts are code: versioned, scored against a golden set and priced.

Goals:

- Write four prompt versions and score each on the golden set, keeping a results table
- Add 3 few-shot examples chosen from your errors, then check whether they helped or just memorized
- Order the prompt stable-first, enable caching and measure the hit rate across 20 calls
- Put prompts in versioned files and log the prompt version with every call
- Write down two changes that would silently break your cache

Resources:

- [Prompt engineering overview (Claude)](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/overview) (article)
- [Prompt engineering (OpenAI)](https://developers.openai.com/api/docs/guides/prompt-engineering) (article)
- [Prompting 101](https://www.youtube.com/watch?v=ysPbXH0LpIE) (video)
- [Prompting Fundamentals and How to Apply them Effectively](https://eugeneyan.com/writing/prompting/) (blog)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903) (paper, Sections 1–3)
- [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) (article, How it works, pricing, cache invalidation)

### Week 3: Project: tool use and a CLI agent

Focus: Build the agent loop by hand, with three tools, a step limit and a budget.

Goals:

- Write the loop: call, run tool_use blocks, return tool_result blocks, stop on a final answer
- Implement three tools with strict schemas and instructive error messages
- Add MAX_STEPS, a dollar budget and a per-tool timeout, and prove each one triggers
- Write 10 tasks with expected outcomes and a runner that reports pass rate, steps and cost
- Read 3 failing traces and rewrite one tool description or error message that caused a failure

Deliverable: A public repo with a one-command CLI agent (about 150 lines of loop and tools), three tools with strict schemas, a step limit and a dollar budget, plus tasks.jsonl with 10 tasks and a runner that prints pass rate, average steps and cost per task. Done when a stranger can run it from the README and at least 8 of the 10 tasks pass.

Resources:

- [Building effective agents](https://www.anthropic.com/research/building-effective-agents) (article)
- [How to Build an Agent](https://ampcode.com/notes/how-to-build-an-agent) (blog)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629) (paper, Sections 1–3)
- [Tool use with Claude](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview) (article)
- [Function calling (OpenAI)](https://developers.openai.com/api/docs/guides/function-calling) (article)
- [Writing effective tools for AI agents](https://www.anthropic.com/research/writing-tools-for-agents) (article)

## What the agent knows

### Week 4: Retrieval for agents

Focus: Put the right few thousand tokens in front of the model, and prove you did.

Goals:

- Chunk the corpus on structure (headings, code blocks) with a small overlap and keep the source for every chunk
- Write 30 questions with their answer chunk ids
- Implement BM25, dense search and RRF, and report recall@5 for each
- Compute RRF by hand for two documents and match your code
- Give the agent search_docs and read_doc tools and cap the number of searches per task

Resources:

- [Contextual retrieval in AI systems](https://www.anthropic.com/research/contextual-retrieval) (article)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401) (paper, Skim sections 1–3)
- [Semantic Search (Sentence Transformers)](https://www.sbert.net/examples/sentence_transformer/applications/semantic-search/README.html) (article)
- [Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf) (paper)
- [dorianbrown/rank_bm25](https://github.com/dorianbrown/rank_bm25) (code)
- [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) (tool, Filter by retrieval and by model size)

### Week 5: Memory and context management

Focus: The window is a budget: trim, clear, summarize and write notes outside it.

Goals:

- Compute when compaction fires for your window size and tokens per step, then confirm in a run
- Implement a scratchpad note tool and re-inject its contents at the end of the prompt
- Clear stale tool results while keeping every tool_use and tool_result pair valid
- Implement summary compaction with fixed headings and a cheap model
- Run a 40-step task with compaction on and off and compare success and cost

Resources:

- [Effective context engineering for AI agents](https://www.anthropic.com/research/effective-context-engineering-for-ai-agents) (article)
- [Context Engineering for AI Agents: Lessons from Building Manus](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus) (blog)
- [Context Engineering for Agents](https://www.youtube.com/watch?v=_IlTcWciEC4) (video)
- [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) (paper)
- [MemGPT: Towards LLMs as Operating Systems](https://arxiv.org/abs/2310.08560) (paper, Sections 1–3)
- [Memory tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/memory-tool) (article)

### Week 6: Project: an MCP server

Focus: Expose your tools through the Model Context Protocol and use them from a real client.

Goals:

- Build the server with three tools and one resource, with bounded outputs and strict schemas
- Call every tool from the Inspector and save a transcript of the raw JSON-RPC messages
- Add an MCP client to your CLI agent and run your 10 tasks through the server
- Connect the server to a second client and fix one description the model misread
- Write down what the server must never do on a user's machine and enforce it in code

Deliverable: A repo with an MCP server exposing three tools and one resource, runnable over stdio, with a README showing the Inspector session. A second client (a chat app or editor) and your own agent both call the tools unchanged. Done when a stranger can add your server to a client with one config block and run a tool.

Resources:

- [What is the Model Context Protocol?](https://modelcontextprotocol.io/docs/getting-started/intro) (article)
- [Specification: key changes](https://modelcontextprotocol.io/specification/2026-07-28/changelog) (article)
- [Build an MCP server](https://modelcontextprotocol.io/docs/develop/build-server) (course)
- [Building Agents with Model Context Protocol (workshop)](https://www.youtube.com/watch?v=kQmXtrmQ5Zg) (video)
- [MCP Inspector](https://modelcontextprotocol.io/docs/tools/inspector) (tool)
- [Security best practices](https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices) (article)

## Patterns and proof

### Week 7: Workflows, agents, multi-agent

Focus: Choose the least autonomy that works: chains, routers and orchestrators before open-ended agents.

Goals:

- Implement prompt chaining, routing, orchestrator-workers and evaluator-optimizer in under 60 lines each
- Run all four designs on 20 cases and tabulate accuracy, cost and latency
- Port one design to a framework and write down what it saved and what it hid
- Write one paragraph on when a single agent beats several, using your numbers

Resources:

- [How We Build Effective Agents](https://www.youtube.com/watch?v=D7_ipDqhtwk) (video)
- [Agent patterns (claude-cookbooks)](https://github.com/anthropics/claude-cookbooks/tree/main/patterns/agents) (code)
- [How we built our multi-agent research system](https://www.anthropic.com/research/multi-agent-research-system) (article)
- [Don't Build Multi-Agents](https://cognition.com/blog/dont-build-multi-agents) (blog)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://arxiv.org/abs/2303.17651) (paper, Sections 1–3)
- [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) (tool, Agents, handoffs, guardrails, tracing)

### Week 8: Project: evals and tracing

Focus: Turn vibes into a suite you can trust: traces, error analysis, graders and a CI gate.

Goals:

- Log a full trace for every run: messages, tool calls with arguments, latency, tokens, cost, prompt version
- Read 50 traces, write a one-line failure note on each and group them into a taxonomy
- Build 40 cases from the taxonomy with code graders for tool use, budget and content
- Hand-label 30 outputs, run a judge on them and compute TPR and TNR
- Add a CI job with a confidence interval and a regression rule, then prove it fails on a broken prompt

Deliverable: An evals/ directory with 40 cases, code graders, one judge with measured TPR and TNR on 30 hand labels, trace logging, and a CI job that reports pass rate with a confidence interval and fails on a significant drop. Done when deliberately breaking a prompt turns the CI job red.

Resources:

- [A Field Guide to Rapidly Improving AI Products](https://hamel.dev/blog/posts/field-guide/) (blog)
- [Using LLM-as-a-Judge For Evaluation: A Complete Guide](https://hamel.dev/blog/posts/llm-judge/) (blog)
- [Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) (blog)
- [Demystifying evals for AI agents](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) (article)
- [tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://arxiv.org/abs/2406.12045) (paper, Abstract, the pass^k metric, and the results section)
- [Semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/) (article)

## Safe and shipped

### Week 9: Safety and permissions

Focus: Assume the model will be fooled and limit what a fooled model can do.

Goals:

- Draw your agent's private data, untrusted content and external communication legs, and break one
- Write 10 injection attacks and run them against the agent, recording which succeed
- Add permission tiers, an egress allowlist and approval that shows the exact arguments
- Run code tools in a container with no network, no secrets and a read-only mount
- Add the 10 attacks to the eval suite and make a failure block the CI job

Resources:

- [The lethal trifecta for AI agents](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) (blog)
- [Prompt Injection, explained](https://www.youtube.com/watch?v=FgxwCaL6UTA) (video)
- [Agents Rule of Two: A Practical Approach to AI Agent Security](https://ai.meta.com/blog/practical-ai-agent-security/) (article)
- [OWASP Top 10 for LLM Applications](https://genai.owasp.org/llm-top-10/) (article, Prompt injection, sensitive information disclosure and excessive agency)
- [Defeating Prompt Injections by Design](https://arxiv.org/abs/2503.18813) (paper, Introduction and design overview)
- [Making Claude Code safer with sandboxing](https://www.anthropic.com/research/claude-code-sandboxing) (article)

### Week 10: Capstone: ship an agent

Focus: Deploy one agent that does real work, with evals, tracing, cost caps and a kill switch.

Goals:

- Choose one narrow job and write its success criteria and 20 eval cases before coding
- Deploy it with tracing, a per-run cap, a per-day cap and a step limit
- Add a kill switch an operator can flip without a deploy and approval for irreversible actions
- Run a kill-switch drill and an injection drill, and record what you fixed
- Write the runbook and a one-page write-up of what failed in the first week

Deliverable: A deployed agent (a URL or an installable CLI) with evals in CI, tracing, per-run and per-day cost caps, approval for irreversible actions, a kill switch, a one-page runbook and a short write-up of what failed in your first week of use. Done when you have run the kill-switch drill and the agent stopped within one step.

Resources:

- [12-factor agents](https://github.com/humanlayer/12-factor-agents) (code)
- [Agents](https://huyenchip.com/2025/01/07/agents.html) (blog)
- [Effective harnesses for long-running agents](https://www.anthropic.com/research/effective-harnesses-for-long-running-agents) (article)
- [Langfuse documentation](https://langfuse.com/docs) (tool, Tracing and evaluation)
- [LiteLLM: budgets and rate limits](https://docs.litellm.ai/docs/proxy/users) (tool)
- [Andrej Karpathy: Software Is Changing (Again)](https://www.youtube.com/watch?v=LCEmiRjPEtQ) (video)

## Notes

- [Prompt injection is a confused-deputy problem](https://ornix.io/u/omar/building-ai-agents/prompt-injection-is-a-confused-deputy-problem.md)
- [Evals you can trust](https://ornix.io/u/omar/building-ai-agents/evals-you-can-trust.md)
- [MCP from the wire up](https://ornix.io/u/omar/building-ai-agents/mcp-from-the-wire-up.md)
- [Context is a budget](https://ornix.io/u/omar/building-ai-agents/context-is-a-budget.md)
- [When retrieval beats a longer context](https://ornix.io/u/omar/building-ai-agents/when-retrieval-beats-a-longer-context.md)
- [Designing tools a model can use](https://ornix.io/u/omar/building-ai-agents/designing-tools-a-model-can-use.md)
- [The agent loop in 60 lines](https://ornix.io/u/omar/building-ai-agents/the-agent-loop-in-60-lines.md)
- [Budgeting tokens and latency](https://ornix.io/u/omar/building-ai-agents/budgeting-tokens-and-latency.md)

Follow this roadmap on Ornix: https://ornix.io/u/omar/building-ai-agents
