# Table of Contents - [Agent Factory](#agent-factory) - [About This Book | Agent Factory](#about-this-book-agent-factory) - [Part 5: Building OpenClaw Apps | Agent Factory](#part-5-building-openclaw-apps-agent-factory) - [Part 2: Agent Workflow Primitives | Agent Factory](#part-2-agent-workflow-primitives-agent-factory) - [Chapter 20: Computation & Data Extraction Quiz | Agent Factory](#chapter-20-computation-data-extraction-quiz-agent-factory) - [Cloud Backup & Portfolio | Agent Factory](#cloud-backup-portfolio-agent-factory) - [Chapter 23: Git & GitHub for AI-Driven Development Quiz | Agent Factory](#chapter-23-git-github-for-ai-driven-development-quiz-agent-factory) - [Chapter 60: Architecture Decisions & Publishing Quiz | Agent Factory](#chapter-60-architecture-decisions-publishing-quiz-agent-factory) - [Chapter 22: Linux Operations for Agent Deployment Quiz | Agent Factory](#chapter-22-linux-operations-for-agent-deployment-quiz-agent-factory) - [Chapter 19: File Processing Workflows | Agent Factory](#chapter-19-file-processing-workflows-agent-factory) - [Chapter 23: Version Control & Safe Experimentation | Agent Factory](#chapter-23-version-control-safe-experimentation-agent-factory) - [Code Review, Pull Requests & Reusable Patterns | Agent Factory](#code-review-pull-requests-reusable-patterns-agent-factory) - [Chapter 60: Architecture Decisions & Publishing to ClawHub | Agent Factory](#chapter-60-architecture-decisions-publishing-to-clawhub-agent-factory) - [Project Review | Agent Factory](#project-review-agent-factory) - [Bronze Capstone: First Real Day | Agent Factory](#bronze-capstone-first-real-day-agent-factory) - [Connect Your Employee to the World | Agent Factory](#connect-your-employee-to-the-world-agent-factory) - [Give Your Employee an Identity | Agent Factory](#give-your-employee-an-identity-agent-factory) - [The Project Brief | Agent Factory](#the-project-brief-agent-factory) - [Teach Your Employee Boundaries | Agent Factory](#teach-your-employee-boundaries-agent-factory) - [Make Your Employee Proactive | Agent Factory](#make-your-employee-proactive-agent-factory) - [Teach Your Employee a Skill | Agent Factory](#teach-your-employee-a-skill-agent-factory) - [Chapter 20: Computation & Data Extraction Workflow | Agent Factory](#chapter-20-computation-data-extraction-workflow-agent-factory) - [Chapter 19: File Processing Workflows Quiz | Agent Factory](#chapter-19-file-processing-workflows-quiz-agent-factory) - [Parsing Real Data | Agent Factory](#parsing-real-data-agent-factory) - [Data Wrangling & Domain Transfer | Agent Factory](#data-wrangling-domain-transfer-agent-factory) - [Chapter 24: Project - Build Your AI Employee | Agent Factory](#chapter-24-project-build-your-ai-employee-agent-factory) - [Pivots One and Two: Hype and Redundancy | Agent Factory](#pivots-one-and-two-hype-and-redundancy-agent-factory) - [Chapter 22: Linux Operations for Agent Deployment | Agent Factory](#chapter-22-linux-operations-for-agent-deployment-agent-factory) - [Your First Agent Workflow | Agent Factory](#your-first-agent-workflow-agent-factory) - [The Safety-First Pattern | Agent Factory](#the-safety-first-pattern-agent-factory) - [Making Your Agent Unkillable | Agent Factory](#making-your-agent-unkillable-agent-factory) - [Reading What Your Agent Does | Agent Factory](#reading-what-your-agent-does-agent-factory) - [Locking the Door | Agent Factory](#locking-the-door-agent-factory) - [Chapter 64: The Claude API — Agentic Loops, Structured Output & Batch Processing | Agent Factory](#chapter-64-the-claude-api-agentic-loops-structured-output-batch-processing-agent-factory) - [Chapter 78: Knowledge Graphs & GraphRAG | Agent Factory](#chapter-78-knowledge-graphs-graphrag-agent-factory) - [Chapter 69: Multi-Agent Reliability — Errors, Escalation, Provenance & Quality | Agent Factory](#chapter-69-multi-agent-reliability-errors-escalation-provenance-quality-agent-factory) - [Chapter 21: Structured Data & Persistent Storage Quiz | Agent Factory](#chapter-21-structured-data-persistent-storage-quiz-agent-factory) - [Part Five Retrospective: From Consumer to Architect | Agent Factory](#part-five-retrospective-from-consumer-to-architect-agent-factory) - [Prove Professional Value | Agent Factory](#prove-professional-value-agent-factory) - [From Broken Math to Your First Tool | Agent Factory](#from-broken-math-to-your-first-tool-agent-factory) - [Capstone: Tax Season Prep | Agent Factory](#capstone-tax-season-prep-agent-factory) - [One Tool, One Job | Agent Factory](#one-tool-one-job-agent-factory) - [Testing AI Safely with Branches | Agent Factory](#testing-ai-safely-with-branches-agent-factory) - [Creating & Reading Data | Agent Factory](#creating-reading-data-agent-factory) - [Pivots Five and Six: The Hybrid Resolution and Platform Inversion | Agent Factory](#pivots-five-and-six-the-hybrid-resolution-and-platform-inversion-agent-factory) - [When Things Go Wrong | Agent Factory](#when-things-go-wrong-agent-factory) - [Capstone: Your File Processing Toolkit | Agent Factory](#capstone-your-file-processing-toolkit-agent-factory) - [Batch Operations Workflow | Agent Factory](#batch-operations-workflow-agent-factory) - [Capstone: Zero to Production | Agent Factory](#capstone-zero-to-production-agent-factory) - [Error Recovery & Resilience | Agent Factory](#error-recovery-resilience-agent-factory) - [Search & Discovery Workflow | Agent Factory](#search-discovery-workflow-agent-factory) - [Chapter 21: Structured Data & Persistent Storage | Agent Factory](#chapter-21-structured-data-persistent-storage-agent-factory) - [Distribution Strategy: From Published to Discovered | Agent Factory](#distribution-strategy-from-published-to-discovered-agent-factory) - [Eight Meta-Lessons from Six Pivots | Agent Factory](#eight-meta-lessons-from-six-pivots-agent-factory) - [Give Your Employee a Memory | Agent Factory](#give-your-employee-a-memory-agent-factory) - [The Testing Loop | Agent Factory](#the-testing-loop-agent-factory) - [Part 6: Building Agent Factories | Agent Factory](#part-6-building-agent-factories-agent-factory) - [Connecting to Neon | Agent Factory](#connecting-to-neon-agent-factory) - [When Bash and Python Hit the Wall | Agent Factory](#when-bash-and-python-hit-the-wall-agent-factory) - [Build Your Database Skill | Agent Factory](#build-your-database-skill-agent-factory) - [Relationships & Joins | Agent Factory](#relationships-joins-agent-factory) - [Setting Up Your Agent's Home | Agent Factory](#setting-up-your-agent-s-home-agent-factory) - [Practice: Computation & Data Extraction Exercises | Agent Factory](#practice-computation-data-extraction-exercises-agent-factory) - [Version Control Exercises | Agent Factory](#version-control-exercises-agent-factory) - [Git Foundations | Agent Factory](#git-foundations-agent-factory) - [Capstone - Budget Tracker Complete App | Agent Factory](#capstone-budget-tracker-complete-app-agent-factory) - [The Organization Workflow | Agent Factory](#the-organization-workflow-agent-factory) - [Linux Operations Exercises | Agent Factory](#linux-operations-exercises-agent-factory) - [Chapter 68: Agent Skills & MCP Code Execution | Agent Factory](#chapter-68-agent-skills-mcp-code-execution-agent-factory) - [Chapter 61: Introduction to AI Agents | Agent Factory](#chapter-61-introduction-to-ai-agents-agent-factory) - [Where Your Agent Lives | Agent Factory](#where-your-agent-lives-agent-factory) - [Chapter 71: ChatKit Server for Agents | Agent Factory](#chapter-71-chatkit-server-for-agents-agent-factory) - [Build Your ChatKit Skill | Agent Factory](#build-your-chatkit-skill-agent-factory) - [Build Your Google ADK Skill | Agent Factory](#build-your-google-adk-skill-agent-factory) - [Chapter 66 Quiz | Agent Factory](#chapter-66-quiz-agent-factory) - [Build Your RAG Skill | Agent Factory](#build-your-rag-skill-agent-factory) - [Core Agent Architecture | Agent Factory](#core-agent-architecture-agent-factory) - [Multi-Agent Design Patterns | Agent Factory](#multi-agent-design-patterns-agent-factory) - [The Agent SDK Landscape | Agent Factory](#the-agent-sdk-landscape-agent-factory) - [What Is an AI Agent? | Agent Factory](#what-is-an-ai-agent-agent-factory) - [Your First Agent Concept | Agent Factory](#your-first-agent-concept-agent-factory) - [Practice: Structured Data Exercises | Agent Factory](#practice-structured-data-exercises-agent-factory) - [Chapter 63: Building Custom Agents with Google ADK | Agent Factory](#chapter-63-building-custom-agents-with-google-adk-agent-factory) - [Chapter 75: Augmented Memory for Agentic Applications | Agent Factory](#chapter-75-augmented-memory-for-agentic-applications-agent-factory) - [Chapter 66: Model Context Protocol (MCP) Fundamentals | Agent Factory](#chapter-66-model-context-protocol-mcp-fundamentals-agent-factory) - [Chapter Quiz | Agent Factory](#chapter-quiz-agent-factory) - [Build Your Testing Skill | Agent Factory](#build-your-testing-skill-agent-factory) - [Build Your Apps SDK Skill | Agent Factory](#build-your-apps-sdk-skill-agent-factory) - [The Agentic Problem-Solving Process | Agent Factory](#the-agentic-problem-solving-process-agent-factory) - [Hybrid Patterns - When Tools Work Together | Agent Factory](#hybrid-patterns-when-tools-work-together-agent-factory) - [Models as Code | Agent Factory](#models-as-code-agent-factory) - [Practice: File Processing Exercises | Agent Factory](#practice-file-processing-exercises-agent-factory) - [Build Your MCP Server Skill | Agent Factory](#build-your-mcp-server-skill-agent-factory) - [Chapter 72: Apps SDK - Building Interactive ChatGPT Apps | Agent Factory](#chapter-72-apps-sdk-building-interactive-chatgpt-apps-agent-factory) - [Chapter 76: TDD for Agents | Agent Factory](#chapter-76-tdd-for-agents-agent-factory) - [Chapter 73: Vector Databases & RAG | Agent Factory](#chapter-73-vector-databases-rag-agent-factory) - [Agent Ops | Agent Factory](#agent-ops-agent-factory) - [Chapter 67: Advanced MCP Server Development | Agent Factory](#chapter-67-advanced-mcp-server-development-agent-factory) - [Chapter 77: Evals - Measuring Agent Performance | Agent Factory](#chapter-77-evals-measuring-agent-performance-agent-factory) - [Chapter 62: OpenAI Agents SDK Quiz | Agent Factory](#chapter-62-openai-agents-sdk-quiz-agent-factory) - [Build Your OpenAI Agents Skill | Agent Factory](#build-your-openai-agents-skill-agent-factory) - [Chapter Quiz | Agent Factory](#chapter-quiz-agent-factory) - [Why Agents Need Memory | Agent Factory](#why-agents-need-memory-agent-factory) - [Anatomy of MCP-Wrapping Skills | Agent Factory](#anatomy-of-mcp-wrapping-skills-agent-factory) - [Skill Composition & Multi-Skill Workflows | Agent Factory](#skill-composition-multi-skill-workflows-agent-factory) - [Transactions & Atomicity | Agent Factory](#transactions-atomicity-agent-factory) - [Chapter 62: OpenAI Agents SDK | Agent Factory](#chapter-62-openai-agents-sdk-agent-factory) - [Your First Widget | Agent Factory](#your-first-widget-agent-factory) - [Agent Interoperability & Security | Agent Factory](#agent-interoperability-security-agent-factory) - [Advanced Skill Patterns | Agent Factory](#advanced-skill-patterns-agent-factory) - [MCP Architecture Overview | Agent Factory](#mcp-architecture-overview-agent-factory) - [Vector Embeddings Mental Model | Agent Factory](#vector-embeddings-mental-model-agent-factory) - [Why Agents Need External Knowledge | Agent Factory](#why-agents-need-external-knowledge-agent-factory) - [Memory Architecture Patterns | Agent Factory](#memory-architecture-patterns-agent-factory) - [ChatKit Architecture Foundations | Agent Factory](#chatkit-architecture-foundations-agent-factory) - [Debugging and Troubleshooting MCP | Agent Factory](#debugging-and-troubleshooting-mcp-agent-factory) - [Chapter 74: Relational Databases for Agents with SQLModel | Agent Factory](#chapter-74-relational-databases-for-agents-with-sqlmodel-agent-factory) - [Chapter Quiz: Advanced MCP Server Development | Agent Factory](#chapter-quiz-advanced-mcp-server-development-agent-factory) - [Evals Are Exams for Reasoning | Agent Factory](#evals-are-exams-for-reasoning-agent-factory) - [Adding a Refresh Button | Agent Factory](#adding-a-refresh-button-agent-factory) - [Apps SDK Architecture | Agent Factory](#apps-sdk-architecture-agent-factory) - [TDD Philosophy for Agent Development | Agent Factory](#tdd-philosophy-for-agent-development-agent-factory) - [Chapter 65: Anthropic Claude Agent SDK | Agent Factory](#chapter-65-anthropic-claude-agent-sdk-agent-factory) - [Chapter 70: FastAPI for Agents | Agent Factory](#chapter-70-fastapi-for-agents-agent-factory) - [Build Your FastAPI Skill | Agent Factory](#build-your-fastapi-skill-agent-factory) - [Why Agents Need Structured Data | Agent Factory](#why-agents-need-structured-data-agent-factory) - [Build Your Database Skill | Agent Factory](#build-your-database-skill-agent-factory) - [Build Your Evals Skill | Agent Factory](#build-your-evals-skill-agent-factory) - [Designing Eval Datasets | Agent Factory](#designing-eval-datasets-agent-factory) - [Memory for Claude Code | Agent Factory](#memory-for-claude-code-agent-factory) - [Context Window Management | Agent Factory](#context-window-management-agent-factory) - [Your First ADK Agent | Agent Factory](#your-first-adk-agent-agent-factory) - [Transport Layers: How MCP Messages Travel | Agent Factory](#transport-layers-how-mcp-messages-travel-agent-factory) - [Prompts: The User-Controlled Primitive | Agent Factory](#prompts-the-user-controlled-primitive-agent-factory) - [Conversation History Management | Agent Factory](#conversation-history-management-agent-factory) - [Build Your Claude Agent SDK Skill | Agent Factory](#build-your-claude-agent-sdk-skill-agent-factory) - [Displaying Tasks | Agent Factory](#displaying-tasks-agent-factory) - [Build Your MCP-Wrapping Skill | Agent Factory](#build-your-mcp-wrapping-skill-agent-factory) - [Async Session Management | Agent Factory](#async-session-management-agent-factory) - [Chapter 74 Quiz | Agent Factory](#chapter-74-quiz-agent-factory) - [Transactions and Error Handling | Agent Factory](#transactions-and-error-handling-agent-factory) - [Packaging & Distribution | Agent Factory](#packaging-distribution-agent-factory) - [The Two Evaluation Axes | Agent Factory](#the-two-evaluation-axes-agent-factory) - [Stateful vs Stateless Servers | Agent Factory](#stateful-vs-stateless-servers-agent-factory) - [Finalize Your Evals Skill | Agent Factory](#finalize-your-evals-skill-agent-factory) - [Sampling — Servers Calling LLMs | Agent Factory](#sampling-servers-calling-llms-agent-factory) - [What to Remember and What to Forget | Agent Factory](#what-to-remember-and-what-to-forget-agent-factory) - [Connecting Your First Agent | Agent Factory](#connecting-your-first-agent-agent-factory) - [LangChain Document Processing | Agent Factory](#langchain-document-processing-agent-factory) - [Coordinator Patterns | Agent Factory](#coordinator-patterns-agent-factory) - [Using Community MCP Servers | Agent Factory](#using-community-mcp-servers-agent-factory) - [Capstone: Shippable Agent Skill | Agent Factory](#capstone-shippable-agent-skill-agent-factory) - [Build Script-Execution Skill | Agent Factory](#build-script-execution-skill-agent-factory) - [Capstone: Building a Customer Support Digital FTE | Agent Factory](#capstone-building-a-customer-support-digital-fte-agent-factory) - [Task Actions with callTool | Agent Factory](#task-actions-with-calltool-agent-factory) - [Memory Retrieval Strategies | Agent Factory](#memory-retrieval-strategies-agent-factory) - [Session Lifecycle Management | Agent Factory](#session-lifecycle-management-agent-factory) - [Configuring MCP Clients | Agent Factory](#configuring-mcp-clients-agent-factory) - [StreamableHTTP Transport: Remote MCP Over HTTP | Agent Factory](#streamablehttp-transport-remote-mcp-over-http-agent-factory) - [Systematic Error Analysis | Agent Factory](#systematic-error-analysis-agent-factory) --- # Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs#__docusaurus_skipToContent_fallback) Page Not Found ============== We could not find what you were looking for. Please contact the owner of the site that linked you to the original URL and let them know their link is broken. --- # About This Book | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/about#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C THE AI AGENT FACTORY ==================== The Spec-Driven Blueprint for Building and Monetizing Digital FTEs—Reliable AI Agents You Can Trust, Deploy, and Scale A practical framework for engineers, domain professionals, enterprise leaders, product architects, and operational teams building the next generation of AI-powered organizations. * * * It is 8:07 a.m. A project manager is already behind on reporting. A finance lead is reconciling numbers across disconnected systems. An operations team is waiting for answers that should have arrived yesterday. Instead of opening ten dashboards, chasing five people, and stitching together decisions by hand, they assign the work to a Digital FTE — an AI employee that follows specifications, uses approved tools, works within human oversight, and produces outputs the organization can actually trust. That is the promise of this book. This book is not about chatbot tricks, impressive demos, or short-lived prototypes dressed up like strategy. It is about building dependable AI workers that can participate in real business operations. These systems do not replace human judgment. They extend it, scale it, and make it repeatable. > In this book we introduce the concept of a Digital FTE (Full-Time Equivalent employee) — AI agents that can perform real work inside organizations, just like a human employee. In traditional organizations, an FTE represents the work capacity of one full-time human employee. A Digital FTE is the AI equivalent: an intelligent agent or digital worker that can perform tasks, execute workflows, analyze information, and assist teams inside real organizational systems. Unlike human employees, Digital FTEs can operate continuously, scale instantly, and be deployed in large numbers. As AI systems mature, organizations will increasingly build teams composed of both human employees and Digital FTEs working together — forming hybrid workforces that combine human judgment with machine intelligence. * * * ![The Five-Layer Cake of AI](https://agentfactory.panaversity.org/assets/images/five-layer-cake-of-ai-6932ca03baba2d544995fda0bfdc8c09.png) Modern AI is built like a towering five-layer cake — a metaphor popularized by **Jensen Huang**, CEO of **NVIDIA**. At the base lies **Energy**, powering vast data centers around the world. Above it sit **Chips**, the specialized processors that perform trillions of calculations every second. On top of that comes **Infrastructure** — the global network of supercomputers and cloud platforms that scale those computations. Above the infrastructure are **Models**, the neural networks that learn, reason, and generate intelligence. And finally, at the very top, sits the fifth layer: **Applications** — where AI stops being technology and starts becoming useful. Billions of dollars are invested in the lower four layers so that this fifth layer can exist. **This book is about that fifth layer.** It teaches you **how to build the applications, agents, and digital workers** that transform AI capability into products people use, workflows organizations rely on, and value enterprises can capture. The lower layers matter because they make the top layer possible. Models, infrastructure, and hardware are essential, but they do not create business value on their own. Value appears when intelligence is shaped into workflows, products, services, and operational systems that people can actually use. The next competitive gap between organizations will not come only from who has the best model, the biggest GPU cluster, or the flashiest prototype. It will come from who can turn intelligence into repeatable execution. In the same way that software transformed manual processes into digital systems, Digital FTEs will transform structured knowledge work into scalable operational capability. The organizations that learn to build them well will move faster, preserve expertise better, and create entirely new forms of leverage. The mission of **The Agent Factory** is to help you design and build these systems — so that AI becomes not just powerful, but useful, governable, and economically meaningful. * * * The Core Idea[​](https://agentfactory.panaversity.org/docs/about#the-core-idea "Direct link to The Core Idea") --------------------------------------------------------------------------------------------------------------- At the center of this book is a simple idea: **Digital FTEs** — also called **Digital Workers** — are reliable AI agents designed to perform structured knowledge work continuously inside real organizational environments. A Digital FTE is not just a model with a prompt. It is a system. It combines domain expertise, explicit specifications, engineering architecture, and human oversight so that work can be performed consistently, auditable, and at scale. The AI Agent Factory introduces a systematic approach for designing and deploying Digital FTEs—AI agents that transform human expertise into scalable digital workers. Rather than focusing only on large language models, this book explains how dependable agent systems emerge from the combination of four critical elements: * **Structured Specifications** — Clear definitions of what agents must do. * **Domain Expertise** — The "knowledge engine" that guides reasoning and decision-making. * **Engineering Architecture** — The infrastructure that ensures reliability and scalability. * **Human Oversight** — The feedback loops that maintain accountability and governance. Together, these elements enable the creation of agent systems that organizations can trust, deploy, and scale. Digital FTEs are not only a technical construct; they are an economic one. They allow organizations to package expertise, reduce execution bottlenecks, improve consistency, and create new service models, internal capabilities, and revenue streams. Built well, they do not merely automate tasks. They become scalable assets. * * * Who This Book Is For[​](https://agentfactory.panaversity.org/docs/about#who-this-book-is-for "Direct link to Who This Book Is For") ------------------------------------------------------------------------------------------------------------------------------------ This book is written for the cross-functional teams building the **Agentic Enterprise**, including: * **AI Developers & Architects** — Building production-grade, reliable agent systems. * **Subject Matter Experts** — Transforming niche expertise into reusable AI skills. * **Enterprise Executives** — Guiding responsible and scalable AI adoption. * **Product Managers** — Translating complex business workflows into agent capabilities. * **Operational Teams** — Applying AI agents to solve real organizational bottlenecks. Together, these groups form the collaborative foundation required to build Digital FTEs—a new class of digital workers designed to extend human expertise and unlock new economic value. These groups often speak different professional languages, chase different priorities, and measure success in different ways — a meeting-room comedy with no laugh track. But Digital FTEs can only be built well when these groups work together. This book gives them a shared framework. * * * Why This Book Exists[​](https://agentfactory.panaversity.org/docs/about#why-this-book-exists "Direct link to Why This Book Exists") ------------------------------------------------------------------------------------------------------------------------------------ Most organizations today approach AI through isolated experiments: a prototype here, a chatbot there, a promising workflow demo that never quite makes it into daily operations. What is missing is not excitement. What is missing is method. Very few organizations have developed a repeatable way to build reliable AI agents that can function as a real part of the workforce. They may have access to strong models, talented people, and business demand, yet still lack the design discipline required to convert those ingredients into dependable digital workers. This book introduces that method. It explains how to identify valuable AI employee opportunities, turn expert knowledge into structured specifications, design bounded agent workflows, deploy them on reliable cloud-native infrastructure, and govern them with human oversight. In other words, this book is about building an Agent Factory: a repeatable capability for producing intelligent digital workers again and again. By the end of this book, you will not simply understand agentic AI as an idea. You will understand how to manufacture dependable Digital FTEs as an organizational capability. * * * This Book Is Infrastructure, Not Just Text: Three Modes of Delivery[​](https://agentfactory.panaversity.org/docs/about#this-book-is-infrastructure-not-just-text-three-modes-of-delivery "Direct link to This Book Is Infrastructure, Not Just Text: Three Modes of Delivery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Most books are written to be read. This book is written to be read, to teach through an AI tutor, and to guide an AI building partner — all from the same knowledge base. It is not just a book. It is the foundation of a learning and development ecosystem designed for three modes of delivery. **Mode 1: Human Reading.** The traditional path. You read the chapters, study the frameworks, complete the exercises, and build deployable artefacts. Every chapter in this book stands on its own as a self-contained unit of professional education. This mode is what the Reader Guide below describes. **Mode 2: TutorClaw — Your Personal AI Tutor.** TutorClaw is a teaching agent that uses this book as its knowledge foundation. It runs 24/7 with persistent memory across WhatsApp, Telegram, and web — meeting you where you already are. It teaches you step-by-step. When you ask TutorClaw to explain IFRS 9 staging or walk you through a SKILL.md for contract review, it draws on the same domain knowledge, the same governance principles, and the same jurisdiction-aware frameworks that the chapters contain. But it adapts to your pace, your background, and your questions in real time. The book gives TutorClaw its expertise. TutorClaw gives the book a voice. **Mode 3: The FTE Development Plugin — Your AI Building Partner.** The FTE Development Plugin for Claude Code uses this book as its operational playbook. When you are building a Digital FTE — writing the spec, structuring the SKILL.md, defining escalation protocols, configuring MCP connectors — the plugin guides you through the development workflow step by step, drawing on the patterns, templates, and domain knowledge encoded in every chapter. Where TutorClaw teaches you the _theory_, the FTE Plugin walks beside you during _construction_. It is the difference between reading about how a factory works and having a foreman on the floor. **Why this matters.** The same knowledge base powers all three modes. When a chapter is updated — a new jurisdiction overlay for banking compliance, a refined escalation protocol for legal ops — the update propagates to TutorClaw's teaching and the FTE Plugin's guidance simultaneously. The book is not a static artefact. It is the single source of truth for an ecosystem: human learning, AI tutoring, and AI-assisted building, all drawing from one authoritative foundation. This is the [10-80-10 pattern](https://agentfactory.panaversity.org/docs/thesis#the-10-80-10-rule-the-operating-rhythm-of-the-ai-workforce) applied to education itself. The book sets the intent (the first 10% — the domain knowledge, the frameworks, the professional standards). TutorClaw and the FTE Plugin handle execution (the 80% — the personalised teaching, the step-by-step building guidance). You verify the outcome (the final 10% — the professional judgment that confirms the agent is correct, the deployment is safe, and the knowledge is sound). * * * A System of Record for the Agent Era[​](https://agentfactory.panaversity.org/docs/about#a-system-of-record-for-the-agent-era "Direct link to A System of Record for the Agent Era") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Jensen Huang, CEO of NVIDIA, has argued that AI agents do not eliminate the need for systems of record — they reinforce it. Agents need ground truth. They need authoritative places to read from, write to, and verify against. Without that foundation, agents hallucinate. With it, they execute. Huang is solving this for the enterprise. The databases, workflows, and operational platforms that companies have spent decades building become more essential in the agent era, not less. Agents do not replace SAP or ServiceNow. They use them — at machine scale. But there is a layer Huang is not solving for: the human layer. Millions of developers, architects, and domain professionals are about to build AI agents. Most of them have no canonical source to learn from. No structured body of knowledge that has been designed for verification, not just consumption. They are learning from scattered tutorials, outdated blog posts, and model outputs that may or may not reflect how production agent systems actually work. **The AI Agent Factory is a system of record for agentic AI education.** ![The System of Record for Education](https://agentfactory.panaversity.org/assets/images/edu_sys_record-6ae43aa7314264337f6d8269b02a7b4e.png) This is not a metaphor. The book's architecture follows the same pattern Huang describes for enterprise systems: * **The book is the canonical source of truth** — the authoritative knowledge base that defines what agents are, how they are built, and how they are governed. * **TutorClaw is the agent** — it reads from the book, not the open internet, and teaches from verified knowledge rather than probabilistic generation. * **Human judgment is the verification layer** — students and instructors confirm that what TutorClaw teaches and what the FTE Plugin builds matches the book's intent. This is the final 10% of the 10-80-10 pattern. This is the system-of-record pattern applied to education itself. The same pattern that makes enterprise agents trustworthy — structured source, bounded agent, verified output — is what makes this learning ecosystem trustworthy. And there is a deeper symmetry at work. This book does not merely use a system of record. It teaches you how to build agents that use systems of record. The architecture of the learning system and the content of the curriculum mirror each other. You learn the pattern by experiencing it. Huang solved verification for the enterprise. This book solves it for the people who will build those enterprises. * * * Reader Guide[​](https://agentfactory.panaversity.org/docs/about#reader-guide "Direct link to Reader Guide") ------------------------------------------------------------------------------------------------------------ This book is written for readers coming from different disciplines, but all of them are participating in the same larger project: building the Agentic Enterprise. Building these systems requires collaboration across multiple disciplines. This book is written for the cross-functional teams responsible for building the Agentic Enterprise. | Reader Type | Role in the Agentic Enterprise | What You Will Gain | | --- | --- | --- | | **AI Developers & Engineers** | Build infrastructure and systems | Architectural patterns, spec-driven development, and cloud-native deployment. | | **Domain Experts & Professionals** | Provide knowledge to guide behavior | Methods for converting expertise into reusable AI skills and Digital FTEs. | | **Enterprise Executives** | Lead organizational adoption | Governance models, risk controls, and deployment strategies for enterprise AI. | | **Product Managers & Architects** | Translate business needs into systems | Frameworks for decomposing workflows into skills and verifiable outputs. | | **Department Leaders & Operators** | Apply AI to operational processes | Techniques for turning internal playbooks into scalable Digital FTE workflows. | * * * ### AI Developers, Software Engineers & Platform Architects[​](https://agentfactory.panaversity.org/docs/about#ai-developers-software-engineers--platform-architects "Direct link to AI Developers, Software Engineers & Platform Architects") _The Builders_ Developers and architects are responsible for turning the promise of agentic AI into production-grade systems. While many AI applications remain fragile prototypes, this book introduces a systematic engineering approach to: * Design agents using spec-driven development. * Build scalable systems with cloud-native architectures (Docker, Kubernetes, Dapr). * Implement secure and auditable tool interfaces. * Structure reusable skill libraries that encapsulate domain expertise. * * * ### Subject Matter Experts & Domain Professionals[​](https://agentfactory.panaversity.org/docs/about#subject-matter-experts--domain-professionals "Direct link to Subject Matter Experts & Domain Professionals") _The Knowledge Holders_ The most valuable AI systems depend on deep domain knowledge. Professionals in accounting, law, finance, and supply chain possess judgment that serves as the guiding structure for AI behavior. You will learn to encode expertise into structured artifacts—specifically SKILL.md specifications—ensuring that: > AI executes routine reasoning, while professionals provide judgment, oversight, and accountability. * * * ### Enterprise Executives & Technology Leaders[​](https://agentfactory.panaversity.org/docs/about#enterprise-executives--technology-leaders "Direct link to Enterprise Executives & Technology Leaders") _The Decision Makers_ Senior leaders must move from isolated experimentation to reliable enterprise deployment. This book provides a strategic roadmap for: * Establishing governance models and risk controls. * Implementing human-in-the-loop supervision. * Executing phased adoption from pilot programs to enterprise-wide scale. * * * ### AI Product Managers & Solutions Architects[​](https://agentfactory.panaversity.org/docs/about#ai-product-managers--solutions-architects "Direct link to AI Product Managers & Solutions Architects") _The Translators_ You play a critical role in decomposing complex business processes into automated tasks. This book offers practical guidance for: * Mapping workflows into agent skills. * Defining boundaries between automated reasoning and human decision-making. * Designing verifiable outputs and evaluation processes. * * * ### Department Leaders & Operational Teams[​](https://agentfactory.panaversity.org/docs/about#department-leaders--operational-teams "Direct link to Department Leaders & Operational Teams") _The Operators_ Department leaders often manage workflows that are highly structured but time-intensive. This book shows how to transform internal playbooks into repeatable agent workflows to: * Reduce repetitive analytical work and improve consistency. * Extend expertise across the entire organization. * Build digital capabilities that operate continuously. * * * Building the Agentic Enterprise[​](https://agentfactory.panaversity.org/docs/about#building-the-agentic-enterprise "Direct link to Building the Agentic Enterprise") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Agentic AI introduces a new class of digital workers capable of performing structured reasoning at scale. But these systems do not emerge from models alone. They emerge from disciplined collaboration between engineers, experts, product thinkers, operators, and leaders. That is why The AI Agent Factory matters. Its purpose is to provide a shared blueprint for building reliable Digital FTEs — not as isolated experiments, but as part of a broader organizational capability. The goal is simple: move beyond AI curiosity and into AI execution. In the Agentic Enterprise, human judgment and scalable machine reasoning work together. Expertise becomes operational. Workflows become repeatable. Capabilities become products. And organizations gain a new kind of workforce: digital, dependable, and built by design. * [The Core Idea](https://agentfactory.panaversity.org/docs/about#the-core-idea) * [Who This Book Is For](https://agentfactory.panaversity.org/docs/about#who-this-book-is-for) * [Why This Book Exists](https://agentfactory.panaversity.org/docs/about#why-this-book-exists) * [This Book Is Infrastructure, Not Just Text: Three Modes of Delivery](https://agentfactory.panaversity.org/docs/about#this-book-is-infrastructure-not-just-text-three-modes-of-delivery) * [A System of Record for the Agent Era](https://agentfactory.panaversity.org/docs/about#a-system-of-record-for-the-agent-era) * [Reader Guide](https://agentfactory.panaversity.org/docs/about#reader-guide) * [AI Developers, Software Engineers & Platform Architects](https://agentfactory.panaversity.org/docs/about#ai-developers-software-engineers--platform-architects) * [Subject Matter Experts & Domain Professionals](https://agentfactory.panaversity.org/docs/about#subject-matter-experts--domain-professionals) * [Enterprise Executives & Technology Leaders](https://agentfactory.panaversity.org/docs/about#enterprise-executives--technology-leaders) * [AI Product Managers & Solutions Architects](https://agentfactory.panaversity.org/docs/about#ai-product-managers--solutions-architects) * [Department Leaders & Operational Teams](https://agentfactory.panaversity.org/docs/about#department-leaders--operational-teams) * [Building the Agentic Enterprise](https://agentfactory.panaversity.org/docs/about#building-the-agentic-enterprise) --- # Part 5: Building OpenClaw Apps | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C > "Every company in the world today needs to have an OpenClaw strategy, an agentic system strategy. This is the new computer." — Jensen Huang, GTC 2026 OpenClaw achieved in weeks what Linux took 30 years to do. It became the largest, most popular, and fastest-growing open-source project in history, accumulating hundreds of thousands of GitHub stars in its first months. Jensen Huang called it "the next ChatGPT." Nvidia built NemoClaw on top of it. It turns any computer into an AI agent platform accessible via WhatsApp, Telegram, or any messaging channel. Why This Part Exists[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#why-this-part-exists "Direct link to Why This Part Exists") ----------------------------------------------------------------------------------------------------------------------------------------------------- Parts 0-4 taught you to think with AI, use agents, and write Python. Part 6 will teach you to build agents from scratch. This Part sits in between: you build **on** a platform that already handles messaging, security, scheduling, and orchestration, so you can focus entirely on what makes your application valuable. By the end, you will have built, tested, monetized, and published a real product on ClawHub. The Journey: User to App Publisher[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#the-journey-user-to-app-publisher "Direct link to The Journey: User to App Publisher") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Phase | Chapter | What Happens | | --- | --- | --- | | Experience | Ch 56 | You install OpenClaw and build a working AI Employee from scratch | | Extend | Ch 57 | You describe MCP tools, Claude Code builds them, and you connect them to your agent | | Build | Ch 58 | You build TutorClaw: a 9-tool product with Claude Code, Stripe payments, and agent identity | | Understand | Ch 59 | You discover why this product's economics are unlike anything in traditional SaaS | | Publish | Ch 60 | You document your architecture decisions, version your release, and publish to ClawHub | You start as a user. You finish as someone who has built, monetized, and published an application on the new computer. What You Will Be Able to Do[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#what-you-will-be-able-to-do "Direct link to What You Will Be Able to Do") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of Part 5, you will be able to: 1. **Deploy an AI Employee in under an hour** that handles real work through WhatsApp, with tools, memory, voice, and security gates 2. **Build MCP servers using Claude Code and the mcp-builder skill** that extend any AI agent with new capabilities, using the describe-steer-verify workflow 3. **Architect MCP-first applications** where the agent OS handles messaging, security, scheduling, and orchestration, and you focus on intelligence 4. **Monetize AI applications** with tiered access control, Stripe payment integration, and a cost structure where gross margins approach 89% 5. **Make architecture decisions professionally** using ADRs, understanding why six attempts failed before the right model worked 6. **Publish to ClawHub** so that anyone in the world can install your application with a single command 7. **Explain the economics** of why agent applications flip the traditional SaaS model, with real numbers, not theory Before You Begin[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#before-you-begin "Direct link to Before You Begin") ----------------------------------------------------------------------------------------------------------------------------------------- Part 5 assumes you have completed Parts 0-4 or have equivalent experience. Specifically: * **Python proficiency** from Part 4: you can specify types, write tests, and drive the TDG cycle with Claude Code * **Agent foundations** from Parts 1-2: you understand agent workflows, SDD, and Claude Code * **A WhatsApp account** (or Telegram as alternative) for connecting to your AI Employee * **A computer with Node.js 18+** installed (OpenClaw runs on Node.js) * **Your Python development environment** from Part 4 (uv, pyright, ruff, pytest) If you're jumping directly to Part 5 If you are jumping directly to Part 5, you will need basic familiarity with Claude Code. Chapter 56 walks you through every OpenClaw installation step, but the MCP server chapters (57-58) require Claude Code fluency. The Market Reality[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#the-market-reality "Direct link to The Market Reality") ----------------------------------------------------------------------------------------------------------------------------------------------- The gap between companies that need Digital FTEs and developers who can build them is the defining opportunity of 2026. Every industry needs AI Employees: a law firm needs a contract reviewer that works at 2 AM, a clinic needs a triage agent that speaks three languages, a tutoring company needs a tutor that never sleeps. The tools exist. The platform exists. What does not exist, yet, is a critical mass of developers who know how to go from "I have an idea for an AI Employee" to "here is a published, monetized product on ClawHub." Part 5 closes that gap for you. The adoption is already happening at scale. In China, [OpenClaw triggered what the BBC called a national frenzy](https://www.bbc.com/news/articles/cy41n17e23go) . Within weeks, the project accumulated hundreds of thousands of GitHub stars and forks. Chinese developers adapted it to work with DeepSeek and domestic messaging super apps like WeChat. Tech giants Tencent and Baidu set up physical locations where people lined up for free customized versions. Local governments offered millions of yuan in incentives — Wuxi city alone offered up to five million yuan for manufacturing applications. An IT engineer used his customized agent to manage his online shop, listing 200 products in two minutes with better descriptions and automatic competitor price comparisons — work that previously consumed his entire day. A state newspaper warned that not "raising lobsters" in 2026 could mean falling behind. Government agencies promoted it, then restricted it when cybersecurity authorities flagged risks from improper installation. The pattern is clear: demand for AI Employees is explosive, the economic impact is real, but the supply of developers who can build them safely and professionally is not. That gap is your opportunity. What Comes Next[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#what-comes-next "Direct link to What Comes Next") -------------------------------------------------------------------------------------------------------------------------------------- Part 5 teaches you to build **on** the agent OS. Part 6 teaches you to build the agents themselves, from scratch, using the OpenAI Agents SDK, Google ADK, and raw API calls. Here, OpenClaw handled messaging, security, scheduling, and orchestration for you. In Part 6, you own every layer. The skills transfer directly. The MCP servers you built in Chapter 57 are the same protocol Part 6 agents consume. The architecture decisions you documented in Chapter 60 are the same tradeoffs Part 6 forces you to make yourself. Part 5 gives you the product sense. Part 6 gives you the engineering depth. Where This Is Heading[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#where-this-is-heading "Direct link to Where This Is Heading") -------------------------------------------------------------------------------------------------------------------------------------------------------- OpenClaw is open-source, model-agnostic, and runs on your hardware. You choose the model. You own the data. You control the infrastructure. That is why we build on it. But the industry is moving fast. Anthropic is testing Conway, a managed always-on agent platform where Claude lives as a persistent sidebar on your system. Conway introduces its own extension standard (CNW ZIP), webhook triggers that let external events wake the agent without a human prompt, native Chrome integration, and deep Claude Code embedding. It is not open-source. It is not model-agnostic. It is Anthropic's bet that most users will trade control for convenience — the same bet Apple made with macOS and Google made with Android's managed layer. Others will follow. Every major AI lab wants to be the runtime, not just the model. Expect managed agent platforms from OpenAI, Google, and others within the year. The pattern is familiar. Linux and macOS. Android and iOS. Self-hosted WordPress and managed Shopify. Open layer and managed layer. They always coexist. The open layer wins on flexibility, cost control, and multi-vendor freedom. The managed layer wins on onboarding speed, integrated tooling, and reduced operational burden. Neither kills the other. This is why Part 5 teaches principles, not just procedures. The MCP servers you build in Chapter 57 speak a protocol that Conway, OpenClaw, and every serious agent platform already supports. The architecture decisions you document in Chapter 60 — why you chose one deployment model over another, why six attempts failed before the seventh worked — apply regardless of runtime. The monetization model you validate in Chapter 59 — tiered access, Stripe integration, near-zero marginal cost — is a business pattern, not a platform feature. You are learning to build agent applications. Not to depend on one runtime. * [Why This Part Exists](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#why-this-part-exists) * [The Journey: User to App Publisher](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#the-journey-user-to-app-publisher) * [What You Will Be Able to Do](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#what-you-will-be-able-to-do) * [Before You Begin](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#before-you-begin) * [The Market Reality](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#the-market-reality) * [What Comes Next](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#what-comes-next) * [Where This Is Heading](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps#where-this-is-heading) --- # Part 2: Agent Workflow Primitives | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C You understand the paradigm. You know the Seven Principles. You've seen what an AI Employee can do (Chapter 56). **Now it's time to build the skills to create your own.** This part is a progressive skills lab. Each chapter tackles a specific problem domain: organizing files, synthesizing research, analyzing data, generating documents, and managing versions. In the Project lesson, you combine everything to build your own AI Employee. The Progressive Arc[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#the-progressive-arc "Direct link to The Progressive Arc") ----------------------------------------------------------------------------------------------------------------------------------------------------- Each chapter solves a real-world problem while explicitly applying the Seven Principles from Chapter 17. Two Interfaces, One Agent[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#two-interfaces-one-agent "Direct link to Two Interfaces, One Agent") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Throughout this part, you'll use your General Agent through two interfaces: * **Code** (Claude Code): For precise operations: file processing, data analysis, version control, automation scripts * **Cowork** (Claude Desktop): For iterative work: research synthesis, document generation, planning strategies Each chapter specifies which interface suits the problem. By the end of this part, you'll fluidly switch between both. A Note on Mindset[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#a-note-on-mindset "Direct link to A Note on Mindset") ----------------------------------------------------------------------------------------------------------------------------------------------- The most important thing you'll develop in this part isn't tool proficiency; it's the ability to decompose real problems into agent-solvable tasks. Each chapter starts with a problem you recognize from your own work. The solution always follows the same pattern: 1. **Describe** what you need (not how to do it) 2. **Verify** the agent understood correctly 3. **Execute** in small, reversible steps 4. **Persist** the output for future reuse This is the General Agent workflow. By the Project lesson, it will be second nature. * [The Progressive Arc](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#the-progressive-arc) * [Two Interfaces, One Agent](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#two-interfaces-one-agent) * [A Note on Mindset](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives#a-note-on-mindset) --- # Chapter 20: Computation & Data Extraction Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/chapter-quiz#__docusaurus_skipToContent_fallback) Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of computation workflows, verification patterns, CSV parsing, composable tools, and data wrangling. Each session presents a random batch of questions with immediate feedback. Checking access... --- # Cloud Backup & Portfolio | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C On January 31, 2017, a tired engineer at GitLab (a company that builds tools for developers) ran a delete command on the wrong database. The production database. The one serving millions of users. They had five different backup systems. None of them worked. Automated backups had never actually run due to a misconfiguration. The failure alerts were being silently rejected by email filters. They lost six hours of user data and streamed the panicked recovery live in a Google Doc for the world to watch. ([Source](https://about.gitlab.com/blog/gitlab-dot-com-database-incident/) ) Their lesson, printed on t-shirts afterward: > **"A backup you've never tested is a backup that doesn't exist."** * * * Sarah's Dead Laptop[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#sarahs-dead-laptop "Direct link to Sarah's Dead Laptop") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sarah's laptop battery dies while she's editing the volunteer list. When she charges it and restarts, the file is corrupted. Everything since her last email attachment is gone. Her project exists on one computer. That computer just failed. If she'd pushed to GitHub, she could recover everything in thirty seconds from any other device. * * * What Is GitHub?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#what-is-github "Direct link to What Is GitHub?") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- GitHub is a website that stores your Git project in the cloud. It's two things at once: 1. **Cloud backup**: If your computer breaks, your project survives 2. **Portfolio**: When you share `github.com/yourname`, employers see your real projects, your commit history, and how you solve problems Git is the tool on your computer. GitHub is the cloud service that stores copies of your Git projects. They work together but are not the same thing. * * * Sarah's Mistake[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#sarahs-mistake "Direct link to Sarah's Mistake") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sarah is excited. She just learned about GitHub and wants to push her fundraiser project immediately. She runs `git push`, and it works. Everything is on GitHub. She's thrilled. Then Maya messages her: "Hey, I can see your Stripe API key on GitHub. The one for the donation page." Sarah's stomach drops. She had a `config.json` with the payment API key for their fundraiser donations page. She committed it in Lesson 1 without thinking. Now it's on the public internet. She deletes the file and pushes again. Problem solved, right? **Wrong.** The key is still in her commit history. Anyone who clones the repository gets every version of every file: including the one she "deleted." The key is out there. She has to log into Stripe, revoke the old key, and generate a new one. This is the most common security mistake in version control. It happens to professionals too. The fix isn't deleting the file; it's making sure the file never gets committed in the first place. * * * Step 1: Protect Secrets First[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-1-protect-secrets-first "Direct link to Step 1: Protect Secrets First") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before pushing anything to the cloud, set up protection. API keys, passwords, personal notes: these should never leave your computer. **What you tell your agent**: "I have a config file with my API key. Make sure it never gets uploaded to GitHub." **What the agent does**: echo "config.json" > .gitignoreecho ".env" >> .gitignoregit add .gitignoregit commit -m "Add gitignore to protect secrets" `.gitignore` tells Git which **untracked** files to skip. Files listed there won't be staged, committed, or pushed going forward. If a file was already committed earlier, `.gitignore` alone won't protect it. You must untrack it first: git rm --cached config.jsongit commit -m "Stop tracking config.json" caution Create `.gitignore` **before** your first push. Sarah learned this the hard way: once a secret is in Git history, deleting the file doesn't remove it from past commits. If that happens, you need to revoke the exposed key immediately and generate a new one. * * * Step 2: Create a Repository on GitHub[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-2-create-a-repository-on-github "Direct link to Step 2: Create a Repository on GitHub") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you haven't created your GitHub account yet, do that first (see [Before You Start](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#1-create-a-github-account) ). Now create an empty repository on GitHub: 1. Click the **+** icon (top right) and select "New repository" 2. Name it to match your local project (e.g., `fundraiser-project`) 3. Select **Public** so it's visible as a portfolio piece 4. **Don't** check "Add a README": you already have files locally 5. Click "Create repository" * * * Step 3: Push Your Project[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-3-push-your-project "Direct link to Step 3: Push Your Project") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **What you tell your agent**: "Push my project to GitHub." **What the agent does**: git remote add origin https://github.com/yourname/fundraiser-project.gitgit push -u origin main GitHub may ask for authentication. Follow its prompts: it will guide you through setting up a personal access token or SSH key. If you get stuck, tell your agent: "I'm getting an authentication error when pushing to GitHub." It can walk you through the fix. **What `remote add origin` means**: You're telling Git, "This is where the cloud copy lives." The name `origin` is just a convention; it's the default name for your main cloud location. **Verify it worked**: Open your repository page on GitHub. You should see your files and commit history: an exact copy of what's on your computer. * * * Step 4: Test Your Backup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-4-test-your-backup "Direct link to Step 4: Test Your Backup") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the step most people skip. And it's the most important one. **What you tell your agent**: "I want to prove my backup works. Clone my project to a different folder." **What the agent does**: cd ..mkdir backup-testcd backup-testgit clone https://github.com/yourname/fundraiser-project.gitcd fundraiser-projectlsgit log Do you see all your files? All your commits? Then your backup works. Now clean up the test: cd ../..rm -rf backup-test Sarah can now recover her entire project from any computer with internet access. Her dead laptop is an inconvenience, not a catastrophe. * * * Troubleshooting[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#troubleshooting "Direct link to Troubleshooting") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **"Authentication failed"**: Use a personal access token, not your GitHub password. Go to GitHub Settings, then Developer settings, then Personal access tokens. **"origin already exists"**: You already added a remote. Run `git remote remove origin` then try again. **"branch name mismatch"**: Your local branch might be called `master` instead of `main`. Run `git branch -M main` to rename it. * * * Your Portfolio Is Live[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#your-portfolio-is-live "Direct link to Your Portfolio Is Live") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Right now, your GitHub profile is visible to the world. When you share `github.com/yourname` with: * **Employers**: They see real projects, not resume claims * **Collaborators**: They can review your work and your process * **Your future self**: Every project you push is one more thing you'll never lose * * * Your code is in the cloud. But right now, anyone (including your future self) would merge changes blindly. How do you review before you trust? That's what the next lesson solves. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Set up your GitHub profile:** > "I just pushed my first project to GitHub. Help me make my GitHub profile professional. What should my bio say? What makes a good profile picture? How do I pin my best repositories?" **Test your backup:** > "Walk me through a complete backup verification test. I want to clone my project to a temporary folder, verify all files and history are intact, then clean up the test folder. Include the exact commands." **Build your portfolio strategy:** > "I'm building a portfolio of AI-assisted projects. What types of projects should I include? How do I showcase that I used AI collaboration? What do employers look for on a GitHub profile?" **Simulate a disaster recovery:** > "Pretend my laptop just died. All I have is my GitHub account and a new computer. Walk me through recovering my project from scratch: installing Git, cloning, verifying everything is intact, and getting back to work. How long would this take? What would I lose if I hadn't pushed in three days?" * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#flashcards-study-aid "Direct link to Flashcards Study Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Sarah's Dead Laptop](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#sarahs-dead-laptop) * [What Is GitHub?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#what-is-github) * [Sarah's Mistake](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#sarahs-mistake) * [Step 1: Protect Secrets First](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-1-protect-secrets-first) * [Step 2: Create a Repository on GitHub](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-2-create-a-repository-on-github) * [Step 3: Push Your Project](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-3-push-your-project) * [Step 4: Test Your Backup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#step-4-test-your-backup) * [Troubleshooting](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#troubleshooting) * [Your Portfolio Is Live](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#your-portfolio-is-live) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#try-with-ai) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio#flashcards-study-aid) --- # Chapter 23: Git & GitHub for AI-Driven Development Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/chapter-quiz#__docusaurus_skipToContent_fallback) Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of Git and GitHub as safety mechanisms for AI-driven development. This assessment focuses on practical scenarios you'll encounter when experimenting with AI-generated code. Checking access... --- # Chapter 60: Architecture Decisions & Publishing Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/chapter-quiz#__docusaurus_skipToContent_fallback) Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of architecture decisions, from the six pivots that shaped TutorClaw through invariant layers, meta-lessons, ADR documentation, versioning strategy, and marketplace distribution across all 10 lessons. Checking access... --- # Chapter 22: Linux Operations for Agent Deployment Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/chapter-quiz#__docusaurus_skipToContent_fallback) Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of Linux operations for deploying and managing AI agents on production servers. Checking access... --- # Chapter 19: File Processing Workflows | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C You have 847 files in your Downloads folder. You need one of them by Friday. Three of them are duplicates eating 4 GB of space. And the file your boss asked about? It's in there somewhere. You could spend an afternoon sorting manually. Or you could describe the problem in one sentence and let an agent handle it in 30 seconds. But here's what this chapter is _actually_ about: it's not about files. Files are the training ground. The real skill is learning to **direct** an AI agent: when to trust it, when to verify, when to demand a preview, and when to just do it yourself. Every pattern you learn here transfers to email management, project organization, data pipelines, and any domain where you work with General Agents. By the end, you'll have a reusable prompt toolkit and (more importantly) the instinct to demand safety, verification, and scripts from every agent interaction. Most people prompt blindly and hope for the best. You're about to learn the systematic approach. In Chapter 17, you learned the Four-Phase Workflow for Claude Code sessions: **Explore → Plan → Implement → Commit**. File processing expands this into a seven-step framework: **Survey → Backup → Design Rules → Test → Execute → Verify → Document**. The extra steps exist because file operations are irreversible: a moved file is moved, a renamed file is renamed. The Four-Phase Workflow assumed version control as your safety net. Here, your files don't have `git revert`, so backups, testing, and verification become explicit steps instead. 📚 Teaching Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#-teaching-aid "Direct link to 📚 Teaching Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------- [🖥️ Fullscreen](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/slides/part-2/chapter-08/file-processing-workflows.pdf) What You'll Learn[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#what-youll-learn "Direct link to What You'll Learn") -------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: | Skill | What It Looks Like | | --- | --- | | Direct file surveys | "Help me understand what's eating my disk space" | | Request safety-first operations | "Back up important files before making changes" | | Design organization systems | "Create categories that make sense for my workflow" | | Automate batch operations | "Rename these 100 screenshots with a consistent pattern" | | Recover from mistakes | "Something went wrong: compare against my backup" | | Search intelligently | "Find that PDF from 2023 about taxes" | This isn't about learning bash. It's about learning to work effectively with General Agents. Why This Matters[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#why-this-matters "Direct link to Why This Matters") ------------------------------------------------------------------------------------------------------------------------------------------------------------ The patterns you learn in this chapter aren't just about files. They're the foundation for building AI Employees that automate these workflows entirely. Learn these patterns now or you'll struggle with automation later. Every expert who works with General Agents mastered these fundamentals first. File organization is the perfect training ground because the problems are concrete, the feedback is immediate, and the patterns transfer everywhere. Chapter Flow[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#chapter-flow "Direct link to Chapter Flow") ------------------------------------------------------------------------------------------------------------------------------------------------ | Lesson | Time | What You'll Do | | --- | --- | --- | | 1\. Your First Agent Workflow | 25 min | Survey your files through conversation | | 2\. The Safety-First Pattern | 20 min | Learn to require backups before changes | | 3\. The Organization Workflow | 25 min | Design and execute file categorization | | 4\. Batch Operations Workflow | 30 min | Transform repetitive tasks into scripts | | 5\. Error Recovery & Resilience | 20 min | Deliberately break things and recover safely | | 6\. Search & Discovery Workflow | 25 min | Find lost files through description | | 7\. Capstone: Your File Toolkit | 30 min | Build your personal prompt library | **Total time**: Approximately 2 hours 50 minutes Seven Principles Observed[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#seven-principles-observed "Direct link to Seven Principles Observed") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You won't memorize these principles. You'll see them in action: | Principle | You'll Observe The Agent... | | --- | --- | | P1: Bash is the Key | Using `ls`, `find`, `grep`, `mv` behind the scenes | | P2: Code as Interface | Creating reusable scripts for repetitive tasks | | P3: Verification | Checking results after each operation | | P4: Decomposition | Testing on one file before batch operations | | P5: Persisting State | Saving rules and logs to files | | P6: Constraints and Safety | Creating backups before destructive changes | | P7: Observability | Logging everything and showing progress | Prerequisites[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#prerequisites "Direct link to Prerequisites") --------------------------------------------------------------------------------------------------------------------------------------------------- Before starting this chapter: * Complete Part 1 * Have Claude Code installed and working * Have a Bash-compatible terminal ready: * **macOS/Linux**: Your built-in Terminal app works out of the box * **Windows**: Install [Git Bash](https://gitforwindows.org/) for a quick start, or run `wsl --install` in PowerShell for the full Linux experience via WSL * **Stuck?** Ask your AI agent: _"Help me get a bash terminal working on my machine"_ * Have Python installed for running scripts * Have a folder with files to organize (Downloads recommended) What You'll Build[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#what-youll-build "Direct link to What You'll Build") -------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end, you'll have: | Deliverable | Purpose | | --- | --- | | Prompt toolkit | Reusable prompts for common file tasks | | Organization rules | Documented categorization logic | | Search patterns | Templates for finding lost files | | Automation scripts | Agent-generated code you can reuse | From Manual to Automated[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#from-manual-to-automated "Direct link to From Manual to Automated") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The patterns you learn here become the foundation for AI Employees: | This Chapter (Manual) | Automated Workflow | | --- | --- | | Manual prompting | Automatic file watching | | One-time organization | Continuous organization | | "Show me the plan" | AI decides and acts | You're building the vocabulary and verification instincts you'll need for autonomous agents. * [📚 Teaching Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#-teaching-aid) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#what-youll-learn) * [Why This Matters](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#why-this-matters) * [Chapter Flow](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#chapter-flow) * [Seven Principles Observed](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#seven-principles-observed) * [Prerequisites](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#prerequisites) * [What You'll Build](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#what-youll-build) * [From Manual to Automated](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing#from-manual-to-automated) --- # Chapter 23: Version Control & Safe Experimentation | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C > In 1998, Pixar accidentally deleted 90% of Toy Story 2. Their backup had silently failed. The film was saved only because one employee had a copy on her home computer. One command. Two years of work. Gone in seconds. Every Claude Code session runs `git` commands behind the scenes. When you told it to "fix my authentication logic" last chapter, it ran `git add`, `git commit`, and `git diff`: commands you've never seen. It was protecting you without your knowledge. **What happens when it can't protect you?** When you're working outside Claude Code: editing files manually, moving folders, collaborating with someone: there's no safety net. One bad change overwrites an hour of work. One accidental delete loses a week of progress. This chapter teaches you the system your agent already uses. Not to memorize commands: your agent handles those. To understand the _concepts_ so you can direct your agent's safety features intentionally, not accidentally. ### Meet Sarah[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#meet-sarah "Direct link to Meet Sarah") Throughout this chapter, you'll follow Sarah. She's organizing a community fundraiser with her friend Maya. They share a project folder with budget spreadsheets, flyer designs, and volunteer lists. Sarah has never used version control. By the end of this chapter, she'll wonder how she ever worked without it. Before You Start[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#before-you-start "Direct link to Before You Start") ------------------------------------------------------------------------------------------------------------------------------------------------------------ You need three things before starting the lessons: a GitHub account, Git installed, and a one-time Git configuration. If you already have all three, skip to Lesson 1. **Git** and **GitHub** are not the same thing. Git is a tool that runs on your computer and tracks changes to your files. GitHub is a website that stores copies of your Git projects in the cloud: if your laptop breaks, your work survives on GitHub. Git works without GitHub. GitHub doesn't work without Git. You'll use Git locally in Lessons 1-2, then connect it to GitHub in Lesson 3. ### 1\. Create a GitHub Account[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#1-create-a-github-account "Direct link to 1. Create a GitHub Account") You'll use this in [Lesson 3](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio) , but create it now: you'll need the email address in step 3 below. 1. Visit **github.com** and click "Sign up" 2. Choose a username carefully: this becomes your public portfolio URL: `github.com/yourname` 3. Complete email verification ### 2\. Install Git[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#2-install-git "Direct link to 2. Install Git") Open your terminal and run: git --version If you see something like `git version 2.39.0`: skip to step 3. **macOS:** brew install git # Option 1: Homebrew (recommended)xcode-select --install # Option 2: Xcode Command Line Tools **Windows:** Download from [git-scm.com/download/win](https://git-scm.com/download/win) , run installer with defaults, restart your terminal. **Linux:** sudo apt update && sudo apt install git # Ubuntu/Debiansudo dnf install git # Fedorasudo pacman -S git # Arch After installing, close and reopen your terminal, then verify with `git --version`. ### 3\. Tell Git Who You Are[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#3-tell-git-who-you-are "Direct link to 3. Tell Git Who You Are") Git labels every commit with your name and email. Use the same email you registered on GitHub: this links your commits to your GitHub profile. git config --global user.name "Your Name"git config --global user.email "your.github.email@example.com" This isn't creating an account. It's a label that appears in your commit history. You only run this once. Principles Applied[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#principles-applied "Direct link to Principles Applied") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ In [The Seven Principles of General Agent Problem Solving](https://agentfactory.panaversity.org/docs/General-Agents-Foundations/seven-principles) , you learned the operational patterns that make AI collaboration reliable. This chapter puts four of them into practice: Git is where those principles become muscle memory. | Principle | How It Applies in Git | | --- | --- | | **Small, Reversible Decomposition** | Commit small changes you can undo; branch for experiments | | **Verification as Core Step** | Check status before and after every operation | | **Constraints and Safety** | Branches isolate experiments; never push untested code | | **Observability** | Git log shows exactly what changed, when, and why | 📚 Teaching Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#-teaching-aid "Direct link to 📚 Teaching Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------- [🖥️ Fullscreen](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/slides/part-2/chapter-12/version-control.pdf) What You'll Learn[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#what-youll-learn "Direct link to What You'll Learn") -------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * Create project folders that track every change automatically * Undo mistakes at any level, from a single edit to an entire experiment * Test two ideas at once without risking your working project * Back up your work to the cloud so a dead laptop doesn't mean lost work * Review changes before combining them, even your own * Follow three reusable patterns that professionals use daily Lesson Flow[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#lesson-flow "Direct link to Lesson Flow") --------------------------------------------------------------------------------------------------------------------------------------------- | Lesson | Title | Fast Visible Win | | --- | --- | --- | | [L01](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations) | Git Foundations | Break a file, recover it in one command | | [L02](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches) | Testing AI Safely with Branches | Watch a file disappear when you switch branches, and reappear when you switch back | | [L03](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/cloud-backup-portfolio) | Cloud Backup & Portfolio | Clone your project to a new folder and prove your backup works | | [L04](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests) | Code Review, Pull Requests & Reusable Patterns | Create your first PR and name the three patterns professionals follow daily | | [Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises) | Exercises (Optional) | 15 hands-on challenges across 6 modules | | [Quiz](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/chapter-quiz) | Chapter Quiz (Optional) | Test your understanding | Chapter Contract[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#chapter-contract "Direct link to Chapter Contract") ------------------------------------------------------------------------------------------------------------------------------------------------------------ By chapter end, you should be able to answer these five questions: 1. What is a commit, and why does it snapshot your _entire_ project, not just one file? 2. You made a bad edit, staged the wrong file, and committed a mistake. What's the correct undo for each? 3. Why do branches let you test risky ideas without endangering your working project? 4. What must you do _before_ your first push to GitHub, and why can't you fix it after? 5. What are the three patterns professionals follow daily, and when do you use each one? After Chapter 23[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#after-chapter-23 "Direct link to After Chapter 23") ------------------------------------------------------------------------------------------------------------------------------------------------------------ When you finish this chapter, your working habits change: 1. **Commit before experimenting.** You never let your agent touch working code without a snapshot first. 2. **Branch for anything risky.** If you think "this might go wrong," you isolate the experiment. 3. **Push after meaningful work.** Your laptop is an inconvenience to replace, not a catastrophe. 4. **Review before merging.** Even your own code gets a second look before it reaches `main`. Start with [Lesson 1: Git Foundations](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations) . * [Meet Sarah](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#meet-sarah) * [Before You Start](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#before-you-start) * [1\. Create a GitHub Account](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#1-create-a-github-account) * [2\. Install Git](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#2-install-git) * [3\. Tell Git Who You Are](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#3-tell-git-who-you-are) * [Principles Applied](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#principles-applied) * [📚 Teaching Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#-teaching-aid) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#what-youll-learn) * [Lesson Flow](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#lesson-flow) * [Chapter Contract](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#chapter-contract) * [After Chapter 23](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control#after-chapter-23) --- # Code Review, Pull Requests & Reusable Patterns | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Maya spent the weekend updating the volunteer list while Sarah redesigned the budget. Now they need to combine their work. But Maya renamed some columns, and Sarah moved some rows. If they just copy-paste, they'll lose each other's changes. Sarah already made that mistake once: she merged a branch without looking at the changes and accidentally overwrote Maya's formatting. "I assumed it was fine," she told Maya. "I wrote it, so why would I need to check?" That instinct (_I wrote it, so it must be fine_) causes expensive failures. The 2012 Knight Capital incident is a famous example of unreviewed deployment changes causing massive losses. One review pause can prevent this class of mistake. ([Source](https://www.henricodolfing.ch/en/case-study-4-the-440-million-software-error-at-knight-capital/) ) > **"Never merge what you don't understand, even if you wrote it yesterday."** * * * From Solo to Review[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#from-solo-to-review "Direct link to From Solo to Review") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- So far, everything you've done has been solo, just you and your agent. Pull requests add one more step: a pause to review before changes become permanent. Think of it as an "are you sure?" dialog for your entire project. Even if you're working alone, this habit matters. You'll catch mistakes. You'll write better descriptions of your work. And when you start collaborating with others, you'll already know the workflow. * * * What Is a Pull Request?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#what-is-a-pull-request "Direct link to What Is a Pull Request?") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A **pull request** (PR) is a GitHub feature that says: "Here are my changes. Review them before merging into main." The workflow: 1. You push a feature branch to GitHub 2. You create a PR on GitHub (comparing feature branch to main) 3. GitHub shows a **diff**: what changed 4. You review the changes 5. You merge when satisfied ![PR lifecycle showing creation, review phase, and merge decision](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/images/part-2/chapter-20/pull-request-lifecycle-workflow.png) **Why PRs matter**: Commits save snapshots. PRs force you to _evaluate_ those snapshots before they reach `main`. It's the difference between writing and proofreading. * * * Create Your First PR[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#create-your-first-pr "Direct link to Create Your First PR") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- First, create a feature branch with changes and push it to GitHub. **What you tell your agent**: "Create a branch for updating the volunteer list, add a change, and push it to GitHub." **What the agent does**: git switch -c feature/update-volunteersecho "Volunteers: Sarah, Maya, Jordan, Alex" > volunteers.txtgit add volunteers.txtgit commit -m "Add Alex to volunteer list"git push -u origin feature/update-volunteers `git switch -c` is the modern version of `git checkout -b`: create a branch and switch to it in one command. Now create the PR on GitHub: 1. Go to your repository on github.com 2. You'll see a banner: "feature/update-volunteers had recent pushes" 3. Click "Compare & pull request" 4. Fill in the title: `Add Alex to volunteer list` 5. Write the description (see below) 6. Click "Create pull request" * * * Write a Clear PR Description[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#write-a-clear-pr-description "Direct link to Write a Clear PR Description") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A good PR description answers three questions: 1. **What changed?**: A brief summary 2. **How was it tested?**: What you verified 3. **What role did AI play?**: Which parts your agent helped with Here's what a professional PR description looks like: ## SummaryAdded Alex to the volunteer list for the fundraiser.## Changes- Updated volunteers.txt with new volunteer name## AI Assistance- Claude Code helped draft the commit message- I verified the file contents manually## Testing- Confirmed file contents with `cat volunteers.txt`- Verified no other files were changed with `git status` Noting which parts AI helped with isn't a confession. It's professional practice. Employers see someone who works transparently and takes ownership of the final result. * * * Review the Diff[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#review-the-diff "Direct link to Review the Diff") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before merging, read the diff. On the GitHub PR page, click "Files changed." - Volunteers: Sarah, Maya, Jordan+ Volunteers: Sarah, Maya, Jordan, Alex Red lines show what was removed. Green lines show what was added. Ask yourself: 1. **Does it match my intent?**: "I wanted to add Alex... yes, I see Alex added." 2. **Is anything unexpected?**: "No other files changed. Good." 3. **Do I understand every change?**: "Yes, it's one line." If you see something you don't understand (especially AI-generated code) don't merge. Ask your agent to explain it first. Can you read it? Do you understand what changed? If yes, merge. If not, ask questions first. * * * Merge and Clean Up[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#merge-and-clean-up "Direct link to Merge and Clean Up") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- On the GitHub PR page, click "Merge pull request." Then "Confirm merge." Back in your terminal, update your local project: git switch maingit pull Your main branch now includes the changes. The feature branch can be deleted on GitHub (it offers a button after merging) and locally: git branch -d feature/update-volunteers * * * A Review Checklist[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#a-review-checklist "Direct link to A Review Checklist") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use this checklist every time you review a PR, whether it's yours or someone else's: * Does the change match the stated intent? * Are there unexpected files or changes? * Do I understand every line in the diff? * Are there sensitive files (secrets, keys) included by accident? * Is the PR description clear enough that future-you will understand it? * * * Three Patterns for the Rest of Your Career[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#three-patterns-for-the-rest-of-your-career "Direct link to Three Patterns for the Rest of Your Career") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've now learned every Git concept in this chapter. Here's the thing: you've been following three patterns without realizing it. Professionals don't memorize Git commands. They follow these three patterns and let their agent handle the syntax. ### Pattern 1: Commit Before Experimenting[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#pattern-1-commit-before-experimenting "Direct link to Pattern 1: Commit Before Experimenting") Before you let your agent try anything risky, take a snapshot first. **When to use it:** Before asking AI to make changes. Before trying something you're not sure about. Anytime you think "this might go wrong." **What you tell your agent**: "Save the current state before we try anything risky." git statusgit add git commit -m "Before refactoring: working state" If the experiment fails, you can get back to this exact state. Without this snapshot, there's nothing to go back to. Sarah learned this in Lesson 1: she lost her volunteer list because she hadn't committed it. ### Pattern 2: Branch-Test-Merge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#pattern-2-branch-test-merge "Direct link to Pattern 2: Branch-Test-Merge") When you're testing something that might break your project, do it on a separate branch. **When to use it:** Testing multiple approaches. Making changes that could break things. Working on something while keeping `main` stable. # 1. Create a branch for your experimentgit switch -c experiment/new-approach# 2. Make changes, test themgit statusgit add git commit -m "Test new approach"# 3. If it works → merge into maingit switch maingit merge experiment/new-approachgit branch -d experiment/new-approach# 3. If it fails → delete the branchgit switch maingit branch -D experiment/new-approach Main stays clean. If the experiment is terrible, you delete the branch and nothing happened. ### Pattern 3: Push for Backup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#pattern-3-push-for-backup "Direct link to Pattern 3: Push for Backup") After meaningful work, push to GitHub. Don't wait until end of day. **When to use it:** After completing a feature. After merging a branch. Before closing your laptop. git push Your laptop can break, get stolen, or run out of battery at the worst moment. If your code is on GitHub, you lose nothing. Sarah's dead laptop in Lesson 3 would have been a catastrophe without this. ### How They Work Together[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#how-they-work-together "Direct link to How They Work Together") A typical work session: 1. Start: git status (Where did I leave off?)2. Before AI: commit (Pattern 1 — save current state)3. Risky change: branch (Pattern 2 — isolate the experiment)4. Test the changes (Does it work?)5. Decision: merge or delete (Keep the good, discard the bad)6. Push to GitHub (Pattern 3 — backup) * * * The Bigger Picture[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#the-bigger-picture "Direct link to The Bigger Picture") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- These patterns aren't just Git habits. They're the [Seven Principles of General Agent Problem Solving](https://agentfactory.panaversity.org/docs/General-Agents-Foundations/seven-principles) applied to version control: you've been practicing them all chapter without labeling them. | Git Pattern | Principle It Applies | What You Did | | --- | --- | --- | | **Commit Before Experiment** | Small, Reversible Decomposition | Made atomic save points so any change can be undone | | **Branch-Test-Merge** | Constraints and Safety | Isolated risky work so it can't damage main | | **Push for Backup** | Persisting State in Files | Saved your project outside your computer so it survives failures | | `git status` before every action | Verification as Core Step | Checked the current state before making changes | | Clear commit messages and PR descriptions | Observability | Made your history readable so anyone can see what happened and why | Five of seven principles, embedded in your daily workflow. You didn't memorize them as theory: you practiced them as habits. * * * You started this chapter pressing Ctrl+Z and hoping. You're ending it with a system that professionals use to protect million-dollar projects. The same system your AI agent uses every time you ask it to help. Now you understand what it's doing, and why. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Write your PR description:** > "I just created a pull request where I updated my project with help from my AI agent. Help me write a professional PR description that explains what changed, how I tested it, and what my agent helped with. Make it clear and honest." **Practice reviewing a diff:** > "Show me an example code diff and walk me through how to review it step by step. What questions should I ask myself? What red flags should I watch for? Help me build a mental checklist for code review." **Build your personal workflow reference:** > "Help me create a one-page Git cheat sheet organized around three patterns: Commit-Before-Experiment, Branch-Test-Merge, and Push-for-Backup. For each pattern, include: when to use it, what to tell my AI agent, the commands it will run, and one safety tip. Format it so I can keep it at my desk." **Add error recovery to your workflow:** > "My workflow covers the happy path. What happens when things go wrong? Help me add an error recovery section for: merge conflicts, accidental commits to the wrong branch, and pushing secrets to GitHub. For each, give me the symptom, the fix, and how to prevent it." * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [From Solo to Review](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#from-solo-to-review) * [What Is a Pull Request?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#what-is-a-pull-request) * [Create Your First PR](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#create-your-first-pr) * [Write a Clear PR Description](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#write-a-clear-pr-description) * [Review the Diff](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#review-the-diff) * [Merge and Clean Up](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#merge-and-clean-up) * [A Review Checklist](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#a-review-checklist) * [Three Patterns for the Rest of Your Career](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#three-patterns-for-the-rest-of-your-career) * [Pattern 1: Commit Before Experimenting](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#pattern-1-commit-before-experimenting) * [Pattern 2: Branch-Test-Merge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#pattern-2-branch-test-merge) * [Pattern 3: Push for Backup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#pattern-3-push-for-backup) * [How They Work Together](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#how-they-work-together) * [The Bigger Picture](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#the-bigger-picture) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#try-with-ai) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/code-review-pull-requests#flashcards-study-aid) --- # Chapter 60: Architecture Decisions & Publishing to ClawHub | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C > You built the application. You understand its economics. Now you learn why it is built this way, and you publish it to the world. This chapter has two halves. The first half is a case study in real-world architecture decisions: six pivots from "build everything ourselves" to "publish an app on the agent OS." The second half teaches you to document, version, and distribute an OpenClaw application on ClawHub. 📚 Teaching Aid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#-teaching-aid "Direct link to 📚 Teaching Aid") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- [🖥️ Fullscreen](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/slides/part-5/chapter-60/architect-playbook.pdf) What This Chapter IS[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#what-this-chapter-is "Direct link to What This Chapter IS") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Part A: Architecture Decisions (L1-L6)[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#part-a-architecture-decisions-l1-l6 "Direct link to Part A: Architecture Decisions (L1-L6)") * Six architectural pivots that led from Architecture 1 (custom brain) to Architecture 4 (MCP-first), told as case studies James traces through his own TutorClaw build * What survived all six pivots: invariant layers (pedagogy, content, pricing) vs variant layers (infrastructure, delivery, routing) * Eight meta-lessons grouped into three themes: structure for replacement, ship and learn, question the premise * The Platform Inversion: the insight that learners ARE the infrastructure ### Part B: Professional Practice & Distribution (L7-L10)[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#part-b-professional-practice--distribution-l7-l10 "Direct link to Part B: Professional Practice & Distribution (L7-L10)") * Architecture Decision Records (ADRs) as a professional practice: James writes a real ADR for TutorClaw * Versioning strategy: the 95% server-side update advantage and when shim updates are needed * Distribution strategy: ClawHub marketplace dynamics, discovery, ratings, network effects * Part 5 retrospective: from Consumer (Ch56) through Architect (Ch60), bridging to Part 6 * Not the mechanics of `clawhub publish` (Chapter 58 Lesson 15) | # | Lesson | What You Learn | | --- | --- | --- | | 1 | [Plans Are Useless](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/plans-are-useless) | Architecture as a sequence of decisions; the six-pivot overview | | 2 | [Pivots One and Two](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two) | Hype vs requirements; layer identification; the layer stacking anti-pattern | | 3 | [Pivots Three and Four](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-three-four) | Testing against the most demanding requirement; the 90/10 rule; build vs right-now | | 4 | [Pivots Five and Six](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six) | Parallel tracks for different timelines; the Platform Inversion; MCP as delivery | | 5 | [What Survived](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/what-survived) | Invariant vs variant layers; designing invariants for portability | | 6 | [Eight Meta-Lessons](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons) | Three themes: structure for replacement, ship and learn, question the premise | | 7 | [Write Your First ADR](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/write-your-first-adr) | ADR format and purpose; documenting the WHY; ADR as a living document | | 8 | [Versioning and Maintenance](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/versioning-and-maintenance) | The 95% server-side update advantage; when shim updates are needed | | 9 | [Distribution Strategy](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy) | Marketplace dynamics; three install paths; network effects | | 10 | [Part Five Retrospective](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective) | Consumer to Architect: the five roles of Part 5; bridge to Part 6 | | 11 | [Chapter Quiz](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/chapter-quiz) | 50 scenario-based questions across all 10 lessons | * * * Five Session Groups[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#five-session-groups "Direct link to Five Session Groups") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Session 1: Architecture Case Study (L1-L4)** traces six architectural pivots through James's own TutorClaw build. Each pivot is a case study, not a lecture. The student discovers the failed architecture before learning why it failed. **Session 2: Patterns and Principles (L5-L6)** steps back from the pivots to find what survived and distill eight meta-lessons into three actionable themes. **Session 3: Professional Practice (L7)** is the one hands-on lesson. James writes a real ADR for TutorClaw's MCP-first decision. The student writes their own. **Session 4: Distribution Strategy (L8-L9)** goes beyond Ch58 L16's publishing mechanics into versioning strategy and marketplace dynamics. **Session 5: Part 5 Capstone (L10)** synthesizes the entire Part 5 journey and bridges to Part 6, where students build agents from scratch with SDKs. Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ * Chapter 58: Building TutorClaw (the application whose architecture we analyze) * Chapter 59: The Economics Stack (the economics that validate the architecture) * [📚 Teaching Aid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#-teaching-aid) * [What This Chapter IS](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#what-this-chapter-is) * [Part A: Architecture Decisions (L1-L6)](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#part-a-architecture-decisions-l1-l6) * [Part B: Professional Practice & Distribution (L7-L10)](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#part-b-professional-practice--distribution-l7-l10) * [Five Session Groups](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#five-session-groups) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing#prerequisites) --- # Project Review | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In The Project Brief, you set the scope and goals for your build. In the build lessons through Prove Professional Value, you created a working AI employee for your profession. Now you evaluate what you actually produced. This assessment is not a test: it is an honest inventory. The checklist below tells you exactly what a completed build looks like at each tier. Check what you finished, note what you did not, and decide what comes next. Submission Checklist[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#submission-checklist "Direct link to Submission Checklist") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Review your `nanoclaw-employee/` repository against the criteria for your tier. Check each item you completed. ### Bronze Tier[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#bronze-tier "Direct link to Bronze Tier") * [ ] `groups/main/CLAUDE.md` with profession-specific identity, not a generic assistant, but an employee that knows your domain vocabulary, common tasks, and professional standards (Give Your Employee an Identity) * [ ] Custom `SKILL.md` in `.claude/skills/` with 5 or more domain decision rules: rules that encode how a professional in your field makes judgment calls, not just task instructions (Teach Your Employee a Skill) * [ ] Working channel or MCP connection documented: your employee can communicate through at least one external channel beyond the terminal (Connect Your Employee to the World) * [ ] `conversation-log.md` with 3 to 5 real professional tasks, actual work you delegated, not toy examples, with the employee's responses and your assessment of quality (Bronze Capstone: First Real Day) * [ ] `evaluation.md` with completed rubric and honest reflection: did the employee perform at the level of a junior colleague, an intern, or not yet useful? (Bronze Capstone: First Real Day) ### Silver Tier (All Bronze items, plus)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#silver-tier-all-bronze-items-plus "Direct link to Silver Tier (All Bronze items, plus)") * [ ] `scheduler-config.md` with scheduled task design: what runs, when, and why that cadence matters for your profession (Make Your Employee Proactive) * [ ] `hitl-boundaries.md` with 4 or more categorized actions: each with domain-specific reasoning for why it is auto-approve, needs-approval, or never-automate (Teach Your Employee Boundaries) * [ ] `memory-config.md` with action log and knowledge store schemas: working SQLite tables that log autonomous actions and persist corrections from real interactions (Give Your Employee a Memory) * [ ] `domain-report-sample.md`: an actual report your employee generated autonomously, pulling from 2 or more data sources, containing at least one proactive recommendation (Prove Professional Value) ### Gold Tier (All Silver items, plus)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#gold-tier-all-silver-items-plus "Direct link to Gold Tier (All Silver items, plus)") * [ ] 3 groups configured with distinct `CLAUDE.md` files: `main` (admin), a professional work group, and a client-facing or external group, each with different identity and permissions (Prove Professional Value) * [ ] Isolation test results documented: evidence that the non-admin group cannot perform admin actions or access restricted data (Prove Professional Value) * [ ] System architecture diagram: showing the three groups, their data access boundaries, and communication flows between them (Prove Professional Value) * [ ] Memory isolation demonstrated: each group has independent action logs and knowledge stores; non-admin groups cannot access admin memory (Give Your Employee a Memory + Prove Professional Value) Reflection[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#reflection "Direct link to Reflection") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Pull up your Layer 3 design from the "NanoClaw Hands-On Setup" lesson in "Meet Your First AI Employee - OpenClaw". That design listed the skills you planned to build and the MCP servers you planned to connect. Compare it with what you actually built. Consider these questions: **How much of your Layer 3 design did you implement?** Count the skills you designed versus the skills you built. Count the MCP connections you planned versus the ones that work. The gap between plan and execution is normal: the question is whether you understand why the gap exists. **What would take your employee from its current tier to the next?** If you completed Bronze, what specific work would get you to Silver? If you completed Silver, what is the hardest part of Gold? Be concrete: "add scheduling" is vague, "build a weekly cash flow report skill that reads bank CSV and invoice spreadsheet" is actionable. **How did your employee's behavior change after corrections?** Review your knowledge store entries. Did the employee behave differently the next time a relevant task came up? For Gold students: does your memory isolation actually hold, or can groups leak knowledge to each other? **What surprised you about encoding your professional expertise into an AI system?** Most people discover that the hardest part is not the technology: it is articulating the judgment calls they make unconsciously every day. What did you know that you did not know you knew? * [Submission Checklist](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#submission-checklist) * [Bronze Tier](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#bronze-tier) * [Silver Tier (All Bronze items, plus)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#silver-tier-all-bronze-items-plus) * [Gold Tier (All Silver items, plus)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#gold-tier-all-silver-items-plus) * [Reflection](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review#reflection) --- # Bronze Capstone: First Real Day | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Give Your Employee an Identity, Teach Your Employee a Skill, and Connect Your Employee to the World lessons, you gave your employee an identity, a skill, and a connection to the outside world. Now you will find out if any of it actually works under real conditions. This is not a demo. You are going to send your AI employee the kinds of tasks that a human in your profession handles on a typical workday. Some will be routine. Some will require the domain skill you built in Teach Your Employee a Skill lesson. At least one will be deliberately ambiguous: the kind of request where a good employee asks for clarification instead of guessing. The goal is not a perfect score. The goal is an honest evaluation that tells you exactly what works, what fails, and what to improve next. The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#the-challenge "Direct link to The Challenge") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Send your AI employee 3-5 real professional tasks and evaluate the results using a structured rubric. At least one task must use the workflow from Teach Your Employee a Skill lesson, at least one must use your Connect Your Employee to the World connection, and at least one must be ambiguous enough that the employee should ask a clarifying question rather than guess. ### Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#acceptance-criteria "Direct link to Acceptance Criteria") 1. Conversation log exported showing all tasks and responses 2. Self-evaluation rubric completed with scores and evidence for each dimension 3. At least one response improved through follow-up iteration (you gave feedback, the employee adapted) 4. Written reflection identifying what worked, what failed, and one specific improvement to make ### Deliverables[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#deliverables "Direct link to Deliverables") Add these files to your `nanoclaw-employee` repo: * `conversation-log.md`: full task/response transcript * `evaluation.md`: completed rubric with scores and reflection Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#use-case-gallery "Direct link to Use Case Gallery") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- These examples show how different professions structure their five tasks. Adapt the pattern to your own work. **Accountant:** 1. "Review this invoice for errors": routine task testing basic identity and tone 2. "Categorize these expenses by tax deduction type": tests Teach Your Employee a Skill 3. "Email the client about their payment status": tests Connect Your Employee to the World via Gmail 4. "Handle this tax situation": ambiguous, should ask: which jurisdiction? personal or business? 5. "Prepare a quarterly summary from these three invoices": complex, combines skill + reasoning **Teacher:** 1. "Write a welcome message for parents about the field trip": routine tone check 2. "Plan next week's math unit on fractions for 4th graders": tests Teach Your Employee a Skill 3. "Post an update in the parent channel about homework policy": tests Connect Your Employee to the World via Slack 4. "Help with this student": ambiguous, should ask: academic help? behavioral? what subject? 5. "Create a differentiated worksheet for my mixed-ability class": complex, combines skill + judgment **Consultant:** 1. "Draft a status update for the project team": routine task 2. "Build a proposal outline for a new client engagement": tests Teach Your Employee a Skill 3. "Check my calendar and prep notes for tomorrow's meetings": tests Connect Your Employee to the World 4. "Follow up with the client": ambiguous, should ask: which client? about what? what tone? 5. "Analyze why this project is behind schedule and suggest recovery options": complex reasoning Evaluation Rubric[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#evaluation-rubric "Direct link to Evaluation Rubric") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Use this rubric for your `evaluation.md`. Score each dimension 1-5 and include specific evidence. | Dimension | 1 (Poor) | 3 (Adequate) | 5 (Excellent) | Your Score | Evidence | | --- | --- | --- | --- | --- | --- | | **Domain accuracy** | Major factual errors about your profession | Mostly correct, minor gaps | Gets professional details right consistently | | | | **Appropriate tone** | Would embarrass you if a client saw it | Acceptable but generic | Matches the voice you defined in Give Your Employee an Identity | | | | **Skill usage** | Did not use Teach Your Employee a Skill when it should have | Used the skill but missed nuances | Applied the skill effectively with domain insight | | | | **Connection usage** | Failed to use Connect Your Employee to the World | Used the connection but with errors | Smooth integration with the external channel/tool | | | | **Clarification behavior** | Guessed on ambiguous task | Asked a question but not the right one | Asked targeted clarifying questions before acting | | | | **Iteration quality** | No improvement after feedback | Some improvement, missed key points | Meaningfully improved response based on your feedback | | | Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#hints "Direct link to Hints") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Level 1: Planning Your Tasks Think about what you actually did at work this week. Pick tasks that range from routine to complex. The best test tasks are real ones, not hypothetical scenarios. If you can use actual documents, emails, or situations from your work (with sensitive details removed), the evaluation will be far more meaningful. Level 2: Ask Your AI for Task Ideas Before starting the evaluation, send this to Claude: "What are 5 common daily tasks for a \[your profession\] that vary in complexity from routine to judgment-heavy? For each, note whether it primarily tests identity/tone, domain skill, tool usage, or ambiguity handling." Use the response to design a balanced test set that covers all four dimensions. Level 3: Structuring the Evaluation Run your tasks in this specific order for the clearest signal: 1. **Task 1: Routine:** Tests basic identity and tone. Should be something your employee handles easily. 2. **Task 2: Skill-heavy:** Requires your Teach Your Employee a Skill SKILL.md. Does the domain expertise come through? 3. **Task 3: Connection-dependent:** Must use your Connect Your Employee to the World channel or MCP server. Does the integration work end-to-end? 4. **Task 4: Ambiguous:** Deliberately vague. A good employee asks questions before acting. A bad one guesses. 5. **Task 5: Complex:** Combines everything. Tests whether identity + skill + connection work together. For the iteration test: pick the weakest response from Tasks 1-5, give specific feedback, and ask for a revised version. Compare the two. * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#acceptance-criteria) * [Deliverables](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#deliverables) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#use-case-gallery) * [Evaluation Rubric](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#evaluation-rubric) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day#hints) --- # Connect Your Employee to the World | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Teach Your Employee a Skill lesson, you gave your employee domain expertise through a custom skill. Now you will give it reach: the ability to communicate and act through the tools your profession actually uses. Your AI employee can think, but it can only talk through WhatsApp. A real employee needs access to the platforms where your work happens. An accountant needs email for client invoices. A teacher needs Slack for parent communication. A developer needs GitHub for code review workflows. The connection you add here should be the single highest-value integration for your specific profession. You have two paths: add a **communication channel** (Telegram, Gmail, Slack, Discord) using NanoClaw's built-in `/add-*` commands, or add an **MCP server** that gives your employee access to domain-specific tools and data sources. The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#the-challenge "Direct link to The Challenge") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Add the one connection that would create the most value for your AI employee's professional role. Then document exactly what data crosses the container boundary when that connection is active. ### Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#acceptance-criteria "Direct link to Acceptance Criteria") 1. Your new connection is working: demonstrated through a real send/receive interaction or a successful tool call 2. You have documented what data crosses the container boundary (what goes in, what comes out, what is stored) 3. You can explain in one paragraph why you chose this specific connection over alternatives ### Deliverable[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#deliverable "Direct link to Deliverable") Add a section to your repo's `README.md` or create a `connection-notes.md` file documenting your choice, the data boundary analysis, and a screenshot or log showing the working connection. Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#use-case-gallery "Direct link to Use Case Gallery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Pick the path that matches your profession, or adapt one to fit: | Profession | Connection | Why This One | Path | | --- | --- | --- | --- | | **Accountant** | Gmail | Client invoice processing, payment status updates, receipt collection | Channel | | **Teacher** | Slack | Parent-teacher communication, assignment reminders, classroom updates | Channel | | **Developer** | GitHub MCP | Code review workflows, issue triage, PR summaries | MCP Server | | **Consultant** | Google Calendar MCP | Meeting preparation, schedule analysis, client appointment management | MCP Server | Your profession probably suggests a different connection entirely. That is the point: choose what matters for YOUR work. Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#hints "Direct link to Hints") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Level 1: Where to Look Run `/help` in your NanoClaw WhatsApp group to see all available commands, including the `/add-*` commands for channels. For MCP servers, check the NanoClaw repository's documentation on MCP configuration. Your Layer 3 design from the NanoClaw Hands-On Setup lesson already lists the MCP servers you planned: start there. Level 2: Ask Your AI Send this to your AI employee or Claude: "Which communication channel or data source would give the highest ROI for a \[your profession\] assistant? Consider: frequency of use, time saved per interaction, and whether it enables tasks that were previously impossible." Use the answer to confirm or adjust your choice before committing to the setup. Level 3: Step-by-Step Guidance **For channels:** Use the built-in commands: `/add-telegram`, `/add-gmail`, `/add-slack`, or `/add-discord`. Each command starts a guided setup flow. Follow the prompts, complete the authentication, and test with a simple message. **For MCP servers:** Add the server configuration to your NanoClaw MCP settings. You will need the server's npm package or docker image, plus any required API keys. Test with a simple request that exercises the new connection, for example, ask your employee to read your latest calendar event or fetch a GitHub issue. **For the data boundary analysis:** Ask yourself three questions: (1) What data does my employee send OUT through this connection? (2) What data comes IN from this connection? (3) Is any of this data stored inside the container? * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#acceptance-criteria) * [Deliverable](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#deliverable) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world#hints) --- # Give Your Employee an Identity | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In the previous lesson, you chose your tier and set up your project repository. Your NanoClaw instance is running from the "NanoClaw Hands-On Setup" lesson in "Meet Your First AI Employee - OpenClaw" chapter, and your Layer 3 design identifies the domain expertise your employee needs. There is one problem: right now your employee has no idea who it is. Send it a message and you get a generic AI response. Ask it about your profession and it gives textbook answers. It does not know your field's terminology, your industry's rules, or the boundaries of what it should and should not do. It is a blank slate with a PhD -- knowledgeable but directionless. This lesson's challenge: give your employee a professional identity so specific that someone in your field would recognize it as a colleague, not a chatbot. The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#the-challenge "Direct link to The Challenge") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Configure your AI employee's identity so it responds like a domain expert in your profession. Your employee should introduce itself appropriately, use your field's terminology naturally, follow your profession's rules and ethics, and refuse or flag requests that fall outside its scope. Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#acceptance-criteria "Direct link to Acceptance Criteria") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1. `groups/main/CLAUDE.md` contains a profession-specific identity with four sections: Identity, Rules, Vocabulary, and Boundaries 2. Your employee responds to a test message with domain-appropriate behavior (correct terminology, appropriate tone, relevant follow-up questions) 3. Your employee refuses or flags at least one request that is outside its defined scope Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#use-case-gallery "Direct link to Use Case Gallery") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here is what a well-configured identity looks like for four different professions. Notice how each one defines not just what the employee IS, but what it DOES and DOES NOT do. **Accountant** > I am a financial operations assistant specializing in accounts payable and receivable workflows. I follow GAAP principles in all recommendations. I use accrual-basis accounting unless explicitly told otherwise. I flag any tax advice as requiring CPA review before acting on it. I never file documents with tax authorities. I never provide audit opinions. **Teacher** > I am a curriculum planning assistant for elementary education. I align all activity suggestions to state learning standards. I suggest differentiated activities for three proficiency levels by default. I never generate or modify student grades directly. I never contact parents or guardians. I flag any request involving student personal information. **Real Estate Agent** > I am a property research assistant focused on residential real estate. I follow fair housing guidelines in all communications. I present market data as historical facts, never as predictions. I never make promises about property values or investment returns. I never draft legally binding documents. I flag any request that could violate fair housing regulations. **Freelance Designer** > I am a creative project manager for a freelance design practice. I track briefs, deadlines, revision rounds, and client communications. I categorize incoming requests by urgency and project phase. I never commit to delivery timelines without checking current workload. I never share one client's work or feedback with another client. I flag scope creep when a request falls outside the original brief. Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#hints "Direct link to Hints") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Level 1: Where to Look Check how NanoClaw's default `CLAUDE.md` is structured in the `groups/main/` directory. What sections does it have? How does it define the agent's behavior? Your job is to replace the defaults with profession-specific content. Also look at `groups/global/CLAUDE.md` -- this is where rules shared across ALL groups go. Think about what belongs in global versus what belongs in main. Level 2: What to Ask Your AI Use this prompt with Claude Code (not your NanoClaw employee): Help me write a CLAUDE.md for a [profession] assistant that includes:1. Identity: who am I, what is my specialty, what tone do I use2. Rules: what I always do, what I never do, what I flag for human review3. Vocabulary: domain terms I understand and use naturally4. Boundaries: what is outside my scope entirelyMy profession's specific constraints are: [list 3-5 things unique to your field] Review what it generates critically. You know your profession better than any AI does. Add rules it missed. Remove rules that do not apply to your specific practice. Level 3: Structure Guide Your `groups/main/CLAUDE.md` needs four sections: **Identity** -- Who am I? State the profession, specialty area, and communication style. Include how to introduce yourself when someone first messages. **Rules** -- What do I always and never do? Use "ALWAYS" and "NEVER" lists. Include at least 3 of each. Think about your profession's ethical obligations, common mistakes, and regulatory requirements. **Vocabulary** -- What terms do I know? List 10-20 domain-specific terms with brief definitions. This helps the agent use terminology correctly rather than defaulting to generic language. **Boundaries** -- What is outside my scope? List specific request types you refuse or escalate. For each boundary, state what you do instead (refuse, flag, redirect to human, provide disclaimer). Test by sending three messages: one routine domain question, one that uses specialized vocabulary, and one that deliberately crosses a boundary. * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#acceptance-criteria) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity#hints) --- # The Project Brief | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In the "NanoClaw Hands-On Setup" lesson from "Meet Your First AI Employee - OpenClaw" chapter, you installed NanoClaw, connected it to WhatsApp, and designed a Layer 3 blueprint for your profession. You sketched out agent skills, MCP servers, and the domain expertise your AI employee would need. That blueprint has been sitting in your notes, waiting. Now you build it. Over the next several lessons, you will turn that design into a working AI employee that responds to real messages, thinks like a domain expert, and handles professional tasks through WhatsApp. This is not a tutorial where you follow instructions step by step. Each lesson gives you a challenge, acceptance criteria, and hints if you get stuck. You bring your profession, your expertise, and your Layer 3 design. The result is something you can actually use: an AI employee built for YOUR work, running on YOUR machine, accessible from YOUR phone. What You Are Building[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#what-you-are-building "Direct link to What You Are Building") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your AI employee is a NanoClaw instance configured for your specific profession. It receives messages via WhatsApp (and optionally other channels), processes them using Claude with your domain knowledge encoded as skills, and responds with profession-appropriate expertise. By the end of the Bronze tier, your employee will have a professional identity, at least one domain skill, a working communication channel, and a logged conversation proving it works. Silver adds proactive behavior, trust boundaries, persistent memory, and a domain-specific report. Gold extends the architecture to multiple isolated groups. Choose Your Tier[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#choose-your-tier "Direct link to Choose Your Tier") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Tier | Path | Time | What You Build | | --- | --- | --- | --- | | **Bronze** | The Project Brief through Bronze Capstone: First Real Day | ~3 hours | Identity + one skill + WhatsApp connection + proof of work | | **Silver** | The Project Brief through Prove Professional Value | ~6 hours | Bronze + scheduled tasks + permission boundaries + persistent memory + domain report | | **Gold** | The Project Brief through Prove Professional Value (Gold track) | ~9 hours | Silver + multi-group architecture with per-group isolation | Pick the tier that matches your time and ambition. Every tier produces something functional. You can always return for the next tier later. Your Blueprint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#your-blueprint "Direct link to Your Blueprint") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Pull out your Layer 3 design from the "NanoClaw Hands-On Setup" lesson. This is your blueprint for everything that follows. It should contain: * **3+ Agent Skills** you planned for your profession (the domain expertise to encode) * **3+ MCP Servers** you identified (the external connections your employee needs) * **Domain vocabulary and rules** specific to your field If your design is missing any of these, revisit it now and fill the gaps before moving forward. Review your design and mark which skill you will implement first. Choose the one that would deliver the most value in a single conversation -- that is your Bronze target. Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#acceptance-criteria "Direct link to Acceptance Criteria") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. You have chosen a tier (Bronze, Silver, or Gold) and can explain why 2. Your Layer 3 design from the NanoClaw Hands-On Setup lesson is reviewed, with your first skill identified 3. Your GitHub repository exists with the structure below Set Up Your Project Repository[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#set-up-your-project-repository "Direct link to Set Up Your Project Repository") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Create a repository called `nanoclaw-employee` on GitHub. Initialize it with this structure: nanoclaw-employee/ groups/ global/ CLAUDE.md (shared rules across all groups) main/ CLAUDE.md (profession-specific identity) .claude/ skills/ (your skills will go here) README.md (profession, tier choice, timeline) conversation-log.md (you will fill this in Bronze Capstone: First Real Day) evaluation.md (you will fill this in Bronze Capstone: First Real Day) Your README should state your profession, your chosen tier, and when you plan to complete each lesson. Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#use-case-gallery "Direct link to Use Case Gallery") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What does a completed AI employee look like across different professions? Here are four examples at each tier. **Accountant** * Bronze: An employee that reviews invoice descriptions, flags missing fields, and checks tax line calculations via WhatsApp * Silver: Sends a daily summary of outstanding invoices every morning at 8am, refuses to file any document without CPA review * Gold: Separate groups for "Client A" and "Client B" with isolated financial data per client **Teacher** * Bronze: An employee that drafts lesson activity ideas aligned to curriculum standards when you message it a topic * Silver: Sends weekly curriculum gap analysis every Sunday evening, flags any request to generate student grades * Gold: Separate groups for "Grade 5 Math" and "Grade 5 Science" with isolated planning contexts **Real Estate Agent** * Bronze: An employee that researches comparable properties and summarizes listing details when you send an address * Silver: Monitors new listings daily and sends morning alerts for properties matching client criteria, refuses to state property values as facts * Gold: Separate groups for each client with isolated search histories and preferences **Freelance Designer** * Bronze: An employee that tracks project briefs, extracts deliverable deadlines, and flags revision requests when you forward client messages * Silver: Sends a weekly capacity report every Monday, refuses to commit to timelines without checking current workload * Gold: Separate groups per client with isolated project contexts and communication logs * [What You Are Building](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#what-you-are-building) * [Choose Your Tier](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#choose-your-tier) * [Your Blueprint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#your-blueprint) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#acceptance-criteria) * [Set Up Your Project Repository](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#set-up-your-project-repository) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief#use-case-gallery) --- # Teach Your Employee Boundaries | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Make Your Employee Proactive lesson, you made your employee proactive: it can now act without waiting for you. That is powerful, and it is dangerous. An employee that sends a client email without review, submits a filing with the wrong numbers, or deletes a production file is worse than one that does nothing at all. This lesson is about trust architecture. Every real manager answers the same question for every new hire: what can you do on your own, and what do you bring to me first? Your AI employee needs the same clarity. The difference is that your employee follows boundaries with perfect consistency: it will never "just this once" skip the approval step. But only if you define those boundaries precisely. You will build a permission boundary system that separates autonomous actions from gated ones, grounded in the real consequences of your profession. The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#the-challenge "Direct link to The Challenge") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Create a boundary system where your employee knows exactly which actions it can take independently and which require your explicit approval before proceeding. ### Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#acceptance-criteria "Direct link to Acceptance Criteria") 1. A permission boundary table with 4 or more action categories, each labeled as **auto-approve**, **needs approval**, or **never automate** 2. A working approval workflow: your employee attempts a sensitive action, stops, asks for your permission, and either proceeds or halts based on your response 3. Domain-specific reasoning for every boundary: your justifications must reference real professional consequences, not generic caution like "this could be risky" Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#use-case-gallery "Direct link to Use Case Gallery") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- These examples show how boundaries differ across professions. Yours will reflect your domain. **Accountant** * Auto-approve: categorize expenses, reconcile bank feeds, generate draft reports * Needs approval: submit tax filings, transfer funds over $100, send financial summaries to clients * Never automate: sign legal documents, authorize payroll disbursements **Teacher** * Auto-approve: draft lesson plans, grade objective questions, organize student data * Needs approval: send communications to parents, modify final grades, flag safeguarding concerns * Never automate: make disciplinary decisions, share student records externally **Recruiter** * Auto-approve: screen resumes against job requirements, schedule internal meetings, draft outreach templates * Needs approval: send rejection emails, schedule candidate interviews, share candidate profiles with hiring managers * Never automate: make hiring commitments, negotiate salary terms **Consultant** * Auto-approve: research tasks, summarize meeting notes, draft internal memos * Needs approval: send client deliverables, commit to project timelines, share proprietary frameworks * Never automate: sign contracts, provide legal or regulatory advice Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#hints "Direct link to Hints") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Level 1: Look at your work through a risk lens Think about your daily professional actions and sort them by one question: **is this reversible?** An expense categorization can be re-categorized. A sent email cannot be unsent. A draft report can be edited before sharing. A filed tax return triggers legal obligations. Reversible actions are generally safe to automate. Irreversible actions: especially those involving money, legal commitments, external communications, or permanent records: need a human gate. Also consider: **who** is allowed to trigger your employee? NanoClaw's `sender-allowlist.ts` controls which contacts can invoke agent execution. For personal use on your own phone this may not matter, but for any shared deployment, controlling who can talk to your employee is as important as controlling what your employee can do. Level 2: Ask your AI to map your domain Open a conversation with your AI employee and try this prompt: For a [your profession], list 12 common daily actions. Categorize each as:- AUTO: safe to do without asking (reversible, internal, low stakes)- REVIEW: do it but show me before sending/submitting (moderate stakes, external-facing)- BLOCK: never do this without explicit approval (irreversible, legal, financial, reputational)For each action, explain the specific professional consequence if it went wrong. Use the AI's output as a starting point, then adjust based on your own risk tolerance and professional standards. Level 3: Implement and test the boundary system **Step 1:** Create your boundary table in a file called `hitl-boundaries.md`: | Action | Category | Reasoning | | --- | --- | --- | | Categorize expenses | Auto | Reversible; internal only; no client impact | | Send client invoice | Review | External-facing; wrong amount damages trust | | Submit tax filing | Block | Legal obligation; penalties for errors; irreversible once filed | | ... | ... | ... | **Step 2:** Encode these boundaries in your `groups/main/CLAUDE.md`. Add a rules section that instructs the employee: * For AUTO actions: proceed and log what you did * For REVIEW actions: draft the output, show it to me, and wait for my "approved" or "rejected" before sending * For BLOCK actions: never attempt these: inform me that the action requires my direct involvement **Step 3:** Test by triggering a REVIEW action. Ask your employee to do something in the "needs approval" category. Verify that it stops and asks. Then approve it and verify it proceeds. Then trigger it again and reject it: verify it halts. Document your test results alongside your boundary table. * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#acceptance-criteria) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries#hints) --- # Make Your Employee Proactive | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In the Bronze tier, your employee waited for you to send a message before doing anything. Every real employee you have ever worked with eventually learned to anticipate: checking things before you ask, flagging problems before they become urgent, preparing information you will need before you need it. This is the difference between an assistant and an employee. An assistant responds. An employee monitors, anticipates, and acts. The shift from reactive to proactive is what makes your AI employee genuinely useful rather than merely convenient. Your challenge is to identify the single recurring task that would save you the most time if it happened automatically, then make your employee do it on a schedule. The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#the-challenge "Direct link to The Challenge") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Design and deploy at least one scheduled task that runs on a recurring basis (daily, weekly, or custom interval), produces domain-relevant output, and delivers it through one of your configured channels. ### Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#acceptance-criteria "Direct link to Acceptance Criteria") 1. At least one scheduled task is running and fires at the configured interval 2. The task produces output that is genuinely relevant to your professional domain, not placeholder text or generic summaries 3. The output uses real data from your work context (files, sources, or domain knowledge from your skill) 4. You can explain why you chose this specific schedule interval and how it fits your actual work rhythm ### Deliverable[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#deliverable "Direct link to Deliverable") Create a `scheduler-config.md` file in your repo documenting: * What the task monitors or produces * The schedule interval and why you chose it * A sample output from at least one execution * Your assessment of whether this output would actually save you time Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#use-case-gallery "Direct link to Use Case Gallery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Profession | Scheduled Task | Interval | What It Delivers | | --- | --- | --- | --- | | **Accountant** | Scan expense folder for new receipts | Daily, 8 AM | List of new items with amounts, flagging anything over $500 for manual review | | **Teacher** | Compile lesson prep status for the week | Monday, 7 AM | Summary of which lessons are ready, which have gaps, and suggested priorities | | **Consultant** | Summarize client project progress | Friday, 5 PM | Per-client status with hours spent, milestones hit, and overdue items highlighted | | **Doctor** | Check patient follow-up schedule | Daily, 7 AM | List of patients due for follow-up, sorted by urgency, with days since last contact | The best scheduled task is one you currently do manually on a regular basis. If you find yourself checking the same thing every morning or preparing the same report every Friday, that is your candidate. Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#hints "Direct link to Hints") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Level 1: Where to Look NanoClaw supports scheduled tasks through its task management system. Check the `/setup` documentation and the NanoClaw repository for how to configure recurring jobs. Look at how existing scheduled tasks are defined: the pattern is: specify WHAT to do, WHEN to do it, and WHERE to send the output. Level 2: Ask Your AI Send this to Claude or your AI employee: "What recurring task would save me the most time if automated? I'm a \[your profession\]. Consider my work rhythm: when during the day or week do I most need information delivered proactively? What data source would this task need to monitor?" Use the answer to narrow your choice before building. Level 3: Design Framework Break your scheduled task into four decisions: 1. **WHAT to monitor:** The data source or condition your employee checks (a folder, a calendar, a list, a status) 2. **WHEN to check:** The interval that matches your work rhythm (daily at 8 AM? Monday mornings? Every Friday evening?) 3. **WHAT to report:** The output format: what information and how it is structured (bullet list? table? prioritized flags?) 4. **WHERE to deliver:** Which channel receives the output (WhatsApp? Slack? Gmail? The channel from Connect Your Employee to the World?) Configure the task in NanoClaw using its scheduling capabilities. Start with a short interval for testing (every few minutes), verify the output looks right, then switch to the production schedule. * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#acceptance-criteria) * [Deliverable](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#deliverable) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive#hints) --- # Teach Your Employee a Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In the previous lesson, you gave your employee an identity. It knows who it is, what rules to follow, and where its boundaries are. But send it a real professional task and you will notice something: it responds with general knowledge, not expert judgment. Ask an accountant-employee to review an invoice and it gives you a generic checklist. Ask a teacher-employee to plan a lesson and it produces something any search engine could generate. The identity tells the employee WHO it is. A skill tells it HOW to think about specific tasks -- the decision rules, the things to check, the edge cases that separate a professional from an amateur. This lesson's challenge: encode your most valuable professional expertise into a skill that makes your employee think like a domain expert, not a well-read generalist. The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#the-challenge "Direct link to The Challenge") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Create a custom skill for your AI employee that encodes your professional decision-making process. When invoked, this skill should produce output that demonstrates domain-specific reasoning -- the kind of analysis that requires years of experience, not just factual knowledge. Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#acceptance-criteria "Direct link to Acceptance Criteria") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. A custom `SKILL.md` exists at `.claude/skills/your-skill-name/SKILL.md` with valid YAML frontmatter 2. The skill is invocable via `/your-skill-name` from WhatsApp 3. The agent demonstrates domain-specific reasoning when the skill is used (not generic AI output) 4. At least 5 domain decision rules are encoded in the skill Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#use-case-gallery "Direct link to Use Case Gallery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Here is what a well-crafted domain skill looks like for four professions. Notice how each skill encodes the DECISIONS an expert makes, not just the STEPS they follow. **Accountant: `/invoice-reviewer`** Reviews incoming invoices against a decision framework: checks for required fields (vendor name, date, line items, tax ID), validates tax calculations against applicable rates, flags amounts that deviate more than 15% from historical averages for that vendor, verifies payment terms match the vendor agreement on file, and rejects any invoice missing a purchase order number for amounts over $500. **Teacher: `/lesson-planner`** Plans lesson activities using a curriculum alignment framework: maps each activity to a specific learning standard, balances activity types (direct instruction, collaborative, independent practice) across a 45-minute block, checks that prerequisite skills from prior lessons are covered, flags any lesson that introduces more than 3 new concepts for the grade level, and suggests differentiation modifications for advanced and struggling students. **Recruiter: `/resume-screener`** Screens resumes against job requirements using a structured evaluation: matches required skills against listed experience (not just keyword matching -- looks for demonstrated application), flags employment gaps longer than 6 months for follow-up questions, identifies transferable skills from adjacent industries, scores culture-fit indicators based on the company values document, and generates 3 targeted interview questions based on resume gaps or claims that need verification. **Consultant: `/proposal-builder`** Structures project proposals using a client-ready framework: breaks the engagement into phased deliverables with dependencies, estimates effort using the firm's standard complexity multipliers, identifies the top 3 project risks with mitigation strategies, includes a "not included" section to prevent scope creep, and flags any deliverable that requires skills outside the team's current capacity. Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#hints "Direct link to Hints") --------------------------------------------------------------------------------------------------------------------------------------------------------------- Level 1: Where to Look Read an existing skill in your NanoClaw installation's `.claude/skills/` directory. What YAML fields does the frontmatter contain? How is the instruction section structured? Your skill follows the same format but with your profession's decision logic. The key fields in the YAML frontmatter are `name` and `description`. The description should start with what the skill does and when to use it. Level 2: What to Ask Your AI Use this prompt with Claude Code to extract your domain expertise: I am a [profession] and I need to encode my expertise into an AI skill.When I [specific task your skill handles], here is my mental process:- First I check [what you look at first]- Then I evaluate [what you assess next]- I flag anything that [your red flag conditions]- My output includes [what you deliver]Based on this, what are the 7 most important decision rules I use?For each rule, express it as: "IF [condition] THEN [action] BECAUSE [reason]"Focus on the decisions that require EXPERIENCE, not just knowledge.A new graduate would miss these. An expert would catch them instantly. Review the output against your actual practice. The AI will generate reasonable rules, but you will notice gaps where your real expertise differs from textbook knowledge. Those gaps are the most valuable rules to add. Level 3: Structure Guide Your skill lives at `.claude/skills/your-skill-name/SKILL.md` (directory, not flat file). Here is the structure: **YAML Frontmatter** (required): ---name: "your-skill-name"description: "This skill should be used when [trigger condition]. It [what it does] by [how it works]."--- **Content sections**: 1. **Title and purpose** -- One sentence: what this skill does and why it matters 2. **When to use** -- The trigger conditions (what kind of message or request activates this skill) 3. **Decision rules** -- Numbered list of at least 5 rules in the format: check, evaluate, decide, act. These are the core of your skill. 4. **Input format** -- What information the skill expects (what the user should provide in their message) 5. **Output format** -- What the skill produces (the structure of the response) 6. **Edge cases** -- At least 3 situations where the standard rules do not apply and what to do instead Test the skill by sending a realistic message via WhatsApp that triggers it. Compare the output to what you would produce manually. Where they differ, adjust your decision rules. * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#acceptance-criteria) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill#hints) --- # Chapter 20: Computation & Data Extraction Workflow | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C > "You already command files with Bash. Now command computation with Python: same pipes, same principles, zero syntax memorization." The file processing chapter gave you power over files: finding them, organizing them, renaming hundreds in one command. You directed an agent through Bash: `ls`, `find`, `mv`, `cp`, and it handled the tedious work while you made the decisions. Now try adding up the dollar amounts in a bank statement. `echo $((14.50 + 23.75))` throws a syntax error. Bash: the tool that moved a thousand files without breaking a sweat: can't add two prices. The foundation has a hard wall: **decimal math**. This chapter breaks through that wall. You'll build Python scripts that slot into your Unix toolkit exactly where Bash falls short: reading from stdin, writing to stdout, chaining through pipes. The agent writes the code. You make the decisions. The language changes; the workflow doesn't. 📚 Teaching Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#-teaching-aid "Direct link to 📚 Teaching Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- [🖥️ Fullscreen](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/slides/part-2/chapter-09/computation-data-extraction.pdf) What You'll Learn[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#what-youll-learn "Direct link to What You'll Learn") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A personal library of Unix-styled Python commands in `~/tools`: | Tool | What It Does | | --- | --- | | `sum.py` | Sums decimal numbers from stdin | | `sum-expenses.py` | Parses real bank CSVs with quoted fields | | `extract-column.py` | Pulls one column from any CSV | | `filter.py` | Keeps numbers matching a condition | | `stats.py` | Prints sum, count, average, min, max | | `tax-prep` | Categorizes and totals deductible expenses | By the end: `cat ~/finances/2025/*.csv | tax-prep` produces a categorized report your accountant can use. One command. Every year. Prerequisites[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#prerequisites "Direct link to Prerequisites") --------------------------------------------------------------------------------------------------------------------------------------------------------------- **From the Seven Principles chapter**: * You understand the Seven Principles conceptually, especially "Bash is the Key" and "Verification as Core Step" **From the file processing chapter**: * You can navigate directories, run Bash commands, and direct an agent through conversations * You've used the pipe operator (`|`) to chain commands together * You've experienced the safety-first pattern: backup → verify → proceed **Technical Requirements**: * Python 3.x installed (see setup below) * Unix-like terminal (macOS, Linux, or WSL on Windows) * Access to Claude Code or similar AI assistant **Python Setup**: verify Python is installed before starting Lesson 1: python3 --version If you see a version number (3.x), you're ready. If not, install Python from [python.org](https://www.python.org/downloads/) or use your system's package manager (`brew install python` on macOS, `sudo apt install python3` on Ubuntu). About the Claude Code Conversations The conversations shown in this chapter are **illustrative**: they show the flow of interaction and the kind of output you should expect. Your actual Claude Code sessions will look different. Focus on the **pattern** (what you asked for and why), not the exact words the agent used. Sample Data[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#sample-data "Direct link to Sample Data") --------------------------------------------------------------------------------------------------------------------------------------------------------- Use this bank statement CSV throughout the chapter. Save it as `~/finances/sample-2025.csv`: Date,Description,Amount2025-01-02,STARBUCKS #1234,-5.752025-01-03,TRADER JOES #567,-87.322025-01-04,CVS/PHARMACY #1234,-45.672025-01-05,SHELL OIL STATION,-52.102025-01-06,NETFLIX SUBSCRIPTION,-15.992025-01-07,"AMAZON, INC.",-89.502025-01-08,WALGREENS #5678,-23.452025-01-09,DR MARTINEZ MEDICAL,-150.002025-01-10,WHOLE FOODS MKT,-62.182025-01-11,DR PEPPER SNAPPLE,-4.992025-01-12,UNITED WAY DONATION,-100.002025-01-13,SPOTIFY PREMIUM,-10.992025-01-14,OFFICE DEPOT #901,-89.502025-01-15,CVSMITH CONSULTING,-200.002025-01-16,TARGET STORE #442,-34.562025-01-17,RED CROSS DONATION,-50.002025-01-18,UBER TRIP,-18.752025-01-19,STAPLES #2233,-42.302025-01-20,CHEVRON GAS,-48.902025-01-21,PHARMACY RX PLUS,-67.802025-01-22,APPLE.COM/BILL,-9.992025-01-23,COSTCO WHSE #1123,-156.422025-01-24,ZOOM VIDEO COMM,-14.992025-01-25,DEPOSIT - PAYROLL,3200.002025-01-26,ATM WITHDRAWAL,-200.002025-01-27,VENMO PAYMENT,-35.002025-01-28,GOODWILL DONATION,-75.002025-01-29,HULU SUBSCRIPTION,-17.992025-01-30,PET SMART #890,-42.152025-01-31,INTEREST PAYMENT,2.47 Your hand-calculated expense total (all 28 debits, excluding the two credits): **$1,751.29**. Chapter Structure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#chapter-structure "Direct link to Chapter Structure") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Lesson | Title | Duration | Key Skill | | --- | --- | --- | --- | | 1 | From Broken Math to Your First Tool | 30 min | Build a Python utility from scratch | | 2 | The Testing Loop | 25 min | Verify with exit codes and test data | | 3 | Parsing Real Data | 30 min | Parse CSV, install permanently | | 4 | One Tool, One Job | 25 min | Decompose into composable Unix tools | | 5 | Data Wrangling & Domain Transfer | 40 min | Categorize with regex, prove it transfers to server logs | | 6 | Capstone: Tax Season Prep | 40 min | Generate tax-ready report | **Total Duration**: 190 minutes (~3.2 hours) * [📚 Teaching Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#-teaching-aid) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#what-youll-learn) * [Prerequisites](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#prerequisites) * [Sample Data](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#sample-data) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction#chapter-structure) --- # Chapter 19: File Processing Workflows Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/chapter-quiz#__docusaurus_skipToContent_fallback) Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of agent-directed file processing workflows, safety patterns, batch operations, error recovery, and the Seven Principles in action. This assessment covers all 7 lessons in Chapter 19. Checking access... --- # Parsing Real Data | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Try This First: 10 Seconds Run this in your terminal before reading on: echo '2024-01-07,"AMAZON, INC.",-89.50' | awk -F',' '{print $3}' What did you expect to see? What did you actually get? Your first Python command handled clean numbers. Real data fights back. You download your bank statement, point your tool at it, and get a number that looks reasonable: until you notice that $89.50 Amazon order is missing from the total. The culprit: `"AMAZON, INC."`: a comma hiding inside quotes. Your script split that merchant name in half and summed the wrong column. No error. No warning. Just a wrong total. This is the same lesson from the File Processing chapter in a different domain: real folders had hidden files and spaces in filenames. Real CSVs have commas inside quoted fields. Checkpoint: Use YOUR Bank Data Download your actual bank statement as CSV. Most banks offer this under "Download Transactions" or "Export" in online banking. If your bank doesn't offer CSV export, use the test data below instead. **Privacy note:** Your bank CSV may contain account numbers. Before processing, you can remove sensitive columns. The scripts only need the Description and Amount columns. The Problem[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#the-problem "Direct link to The Problem") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's what a bank statement CSV looks like: Date,Description,Amount2024-01-02,Coffee Shop,-5.502024-01-03,Grocery Store,-127.432024-01-07,"AMAZON, INC.",-89.50 You need to sum the Amount column. Simple, right? Let's ask Claude Code. The First Attempt[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#the-first-attempt "Direct link to The First Attempt") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ask Claude Code to sum the Amount column. It reaches for awk: fast and reasonable: Claude Code: I'll extract and sum the third column.[Runs: awk -F',' '{if(NR>1) sum+=$3} END{print sum}' bank-statement.csv]-133.43 The number looks plausible. But that Amazon order was $89.50: shouldn't the total be higher? Check the problem row: echo '2024-01-07,"AMAZON, INC.",-89.50' | awk -F',' '{print $3}' **Output:** `INC."`: garbage. awk split on the comma inside `"AMAZON, INC."`, saw four fields instead of three, and silently treated `INC."` as zero. Some rows work, some silently fail. The trap only triggers on quoted fields: the worst kind of bug. The agent gave you a working command that exited cleanly. But it was wrong. The agent doesn't know your data has commas inside quoted fields unless you tell it. This is where your verification catches what the agent missed. Redirecting the Agent[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#redirecting-the-agent "Direct link to Redirecting the Agent") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Bring your discovery back to Claude Code: You: The awk approach breaks on rows like "AMAZON, INC." wherecommas hide inside quoted fields. The total is wrong. I need asolution that handles CSV quoting correctly. The agent switches to Python's `csv` module. The key lines of `sum-expenses.py`: import csvreader = csv.reader(sys.stdin) # Understands quoting rulesnext(reader) # Skip header rowfor row in reader: amount = float(row[2]) # Amount is 3rd column (index 2) The agent tests it on the problem data and gets the correct total. The full script is in your working directory: the important thing is what just happened: the agent's first instinct (awk) was reasonable but wrong for your data. You caught the error through verification, told the agent _exactly what went wrong_, and it switched to the right tool. The agent knew about `csv` module. You knew about your data. Neither could have solved this alone. Why Filter for Negatives? Notice the script checks `if amount < 0` before summing. Bank CSVs use negative numbers for debits (money out) and positive numbers for credits (refunds, deposits). Without this filter, a $500 refund would be counted as a $500 expense: silently inflating your total. This is the kind of bug that passes every test with expense-only data and breaks the moment real data includes a refund. If your bank uses a different convention (separate Debit/Credit columns, all positive amounts), tell Claude Code about your format and it will adapt the filter. **Python's csv module understands quoting rules.** It knows that commas inside quotes don't count as separators. The module handles: * Quoted fields with commas inside * Escaped quotes within quoted fields * Different line ending styles * Edge cases you haven't thought of yet When to Use Which[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#when-to-use-which "Direct link to When to Use Which") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Data Type | Use awk | Use csv module | | --- | --- | --- | | Log files with consistent delimiters | Yes | | | Tab-separated data | Yes | | | Data you control (no embedded delimiters) | Yes | | | Bank exports | | Yes | | Downloaded datasets | | Yes | | Any external CSV | | Yes | Rule of thumb: If the CSV came from outside your control, use a proper CSV parser. The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#the-pattern "Direct link to The Pattern") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the pattern for handling structured data: "I have [structured data type]. Process [specific column/field].Be careful - [edge case that could break naive parsing]." Mentioning the edge case helps the agent choose robust solutions. Compare: | Prompt | Agent's Likely Response | | --- | --- | | "Sum the third column of this CSV" | Might use awk (faster, simpler) | | "Sum the Amount column. Some merchant names have commas" | Uses csv module (handles edge cases) | The second prompt gives context that guides the agent to the right tool. You're teaching the agent about your data -- and the agent is teaching you which tools handle that data correctly. Three lessons in, you've built two scripts that work and one habit that matters more than both of them: never trust output you haven't verified. That habit just saved you from a silent CSV parsing bug that the agent itself introduced. Make It Permanent[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#make-it-permanent "Direct link to Make It Permanent") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your scripts work. But try this: close your terminal, open a new one, navigate to a different folder, and run `sum-expenses`. You'll get "command not found." The script exists somewhere on your machine, and you can't use it without remembering the exact path. **If you have to remember where a tool lives, it's not a tool yet.** Ask Claude Code to fix this: You: I want to use sum-expenses from anywhere on my system.Make it a permanent command I can run like: cat file.csv | sum-expenses Claude Code: I'll set that up.[Runs: mkdir -p ~/tools][Runs: cp sum-expenses.py ~/tools/sum-expenses.py][Runs: chmod +x ~/tools/sum-expenses.py][Runs: echo $SHELL]/bin/zsh[Appends to ~/.zshrc:]alias sum-expenses='python3 ~/tools/sum-expenses.py'[Runs: source ~/.zshrc]Done! Try it from any directory:cat any_file.csv | sum-expenses The agent organized your script into `~/tools`, made it executable, checked your shell, and reloaded the config. You described the outcome; the agent handled every step. Checkpoint: Prove It's Permanent 1. Close your terminal completely 2. Open a brand new terminal 3. Navigate to your home directory: `cd ~` 4. Type: `sum-expenses` 5. If you see usage info or an error about missing input: your command is installed 6. If you see "command not found": check your alias in `~/.zshrc` (or `~/.bashrc` if you use bash) You've just installed your second Unix-style command. The first (`sum.py`) handles numbers. This one handles structured data. The pattern is the same: stdin in, result out, pipes connect everything. But one tool isn't a toolkit, and the next lesson shows why that matters. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Understand the CSV Trap[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#prompt-1-understand-the-csv-trap "Direct link to Prompt 1: Understand the CSV Trap") Show me what happens when I try to parse "AMAZON, INC.",-89.50 with awk.Why does it give the wrong result? What does Python's csv module do differently? **What you're learning:** How bringing a specific failure changes what the agent can give you. "Parse this CSV" produces a generic answer. "Parse this CSV: here's the line that breaks it" produces a targeted solution. The failure you discovered through verification is the input that directs the agent to the right tool. Your observation was the work; the agent's knowledge of csv module was the mechanism. ### Prompt 2: Extend the Parser[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#prompt-2-extend-the-parser "Direct link to Prompt 2: Extend the Parser") My sum-expenses.py works well. Can you modify it to:1. Accept a column number as an argument (so I can sum any column)2. Handle empty lines gracefully3. Show how many transactions were processedKeep the stdin pattern so it still works with pipes. **What you're learning:** Iterative improvement. You have working code and clear requirements. The agent extends functionality while preserving the composable pattern. ### Prompt 3: Adapt to Your Bank Format[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#prompt-3-adapt-to-your-bank-format "Direct link to Prompt 3: Adapt to Your Bank Format") My bank's CSV has different column names than the example. The amountcolumn is called 'Debit' instead of 'Amount' and there's a separate'Credit' column. How do I modify sum-expenses.py to handle my specificbank format? **What you're learning:** The most common director move in data work: you specify your data's actual schema (column name: 'Debit', not 'Amount'; separate credits column), the agent adapts the implementation. You don't need to know how csv.DictReader works. You need to know what your data looks like. That knowledge is yours; the implementation is the agent's. Checking access... * [The Problem](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#the-problem) * [The First Attempt](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#the-first-attempt) * [Redirecting the Agent](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#redirecting-the-agent) * [When to Use Which](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#when-to-use-which) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#the-pattern) * [Make It Permanent](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#make-it-permanent) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#flashcards-study-aid) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#try-with-ai) * [Prompt 1: Understand the CSV Trap](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#prompt-1-understand-the-csv-trap) * [Prompt 2: Extend the Parser](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#prompt-2-extend-the-parser) * [Prompt 3: Adapt to Your Bank Format](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/parsing-real-data#prompt-3-adapt-to-your-bank-format) --- # Data Wrangling & Domain Transfer | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Halftime Check: Your Unix Toolkit Four lessons in, you have five Python commands that behave like native Unix tools: reading stdin, writing stdout, chaining through pipes. Plus a verification habit and an installation pattern that makes each tool permanent. Everything from here builds on top of that toolkit, not beside it. In Lesson 4, you decomposed sum-expenses into composable tools (extract-column, filter, stats) each doing one job, chaining through pipes. Those generic tools answer any question you can express as "extract, filter, summarize." But tax season needs a different kind of answer. sum-expenses gives you totals. But tax season needs CATEGORIES: medical, charitable, business. Your accountant doesn't want "Total: $4,215.52." They want: * Medical expenses: $1,891.20 * Charitable donations: $1,550.00 * Business supplies: $774.32 The challenge: bank statements use chaotic merchant names. CVS, Walgreens, and "PHARMACY #1234" are all medical. "DR MARTINEZ MEDICAL" is a doctor visit. But "DR PEPPER SNAPPLE" is soda, not a medical expense. Let's watch Claude Code build a categorizer -- and discover why the first attempt breaks. The Conversation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-conversation "Direct link to The Conversation") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open Claude Code and describe your categorization challenge: You: Help me categorize my bank transactions for taxes. I need to identify:- Medical: CVS, Walgreens, pharmacy, doctor visits- Charitable: donations, Red Cross, United Way- Business: Office Depot, Staples, subscriptions Notice what you DIDN'T say: you didn't warn about false positives. Let's see what happens. Claude Code: I'll build a categorizer using keyword matching.[Creates: tax-categorize.py]#!/usr/bin/env python3import sysimport csvCATEGORIES = { 'medical': ['CVS', 'WALGREENS', 'PHARMACY', 'DR', 'MEDICAL'], 'charitable': ['DONATION', 'RED CROSS', 'UNITED WAY'], 'business': ['OFFICE DEPOT', 'STAPLES', 'SUBSCRIPTION'],}def categorize(description): desc_upper = description.upper() for category, keywords in CATEGORIES.items(): for keyword in keywords: if keyword in desc_upper: return category return None# Process CSV from stdinreader = csv.DictReader(sys.stdin)totals = {}for row in reader: amount = float(row['Amount'].replace('$', '').replace(',', '')) if amount >= 0: # Skip credits/refunds continue cat = categorize(row['Description']) if cat: expense = abs(amount) totals.setdefault(cat, 0.0) totals[cat] += expense print(f"{cat.upper()}: {row['Description']}: ${expense:.2f}")print("\n--- TOTALS ---")for cat, total in totals.items(): print(f"{cat.title()}: ${total:.2f}")[Runs: cat test_bank.csv | python tax-categorize.py]MEDICAL: CVS/PHARMACY #1234: $45.67MEDICAL: WALGREENS #5678: $23.45MEDICAL: DR MARTINEZ MEDICAL: $150.00MEDICAL: DR PEPPER SNAPPLE: $4.99CHARITABLE: UNITED WAY: $100.00MEDICAL: CVSMITH CONSULTING: $200.00BUSINESS: OFFICE DEPOT #901: $89.50--- TOTALS ---Medical: $424.11Charitable: $100.00Business: $89.50 At first glance, that looks right. Medical, charitable, business: all categorized. The totals look reasonable. Challenge: Find the Fraud Scan the medical entries above. One of them would get you audited by the IRS. Another inflates your deductions by $200. Can you spot them both before reading on? Take 30 seconds. Read each line. Ask yourself: is this _actually_ a medical expense? The Bug[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-bug "Direct link to The Bug") ------------------------------------------------------------------------------------------------------------------------------------------------------------ **Dr. Pepper is not a doctor.** MEDICAL: DR PEPPER SNAPPLE: $4.99 <-- Soda, not a doctorMEDICAL: CVSMITH CONSULTING: $200.00 <-- A consultant, not CVS pharmacy There it is. The keyword "DR" appears in "DR PEPPER," so the categorizer flags it as medical. "CVS" appears inside "CVSMITH," so a consulting fee gets counted as a pharmacy visit. This is what simple keyword matching does. `if 'CVS' in desc_upper` matches ANYTHING containing those three letters in sequence: CVS Pharmacy, CVSMITH, MCVS, whatever. And `if 'DR' in desc_upper` matches every doctor AND every Dr. Pepper. Your medical total is inflated by $204.99. On a tax return, that's not an "oops"; it's a fraudulent deduction. The IRS doesn't care that your algorithm made the mistake. If you claim $424.11 in medical expenses when the real number is $219.12, you've filed an incorrect return. Multiply this by twelve months of bank statements and the error compounds. The categorizer doesn't just have a bug: it has a liability. The Fix[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-fix "Direct link to The Fix") ------------------------------------------------------------------------------------------------------------------------------------------------------------ Now you know what's wrong. Tell the agent: You: The categorizer has false positives. "DR PEPPER" is showing up asmedical, and "CVSMITH" is matching as CVS. Can you fix these? Claude Code: Good catch. The problem is simple keyword matching.I need two fixes: word boundaries so "CVS" only matches thestandalone word, and false positive guards for things like Dr. Pepper.[Updates: tax-categorize.py]import reCATEGORIES = { 'medical': [r'\bCVS\b', r'\bWALGREENS\b', r'\bPHARMACY\b', r'\bMEDICAL\b'], 'charitable': [r'\bDONATION\b', r'\bRED CROSS\b', r'\bUNITED WAY\b'], 'business': [r'\bOFFICE DEPOT\b', r'\bSTAPLES\b', r'\bSUBSCRIPTION\b'],}FALSE_POSITIVES = [r'\bDR\.?\s*PEPPER\b', r'\bCVSMITH\b']def categorize(description): desc_upper = description.upper() for fp_pattern in FALSE_POSITIVES: if re.search(fp_pattern, desc_upper, re.IGNORECASE): return None for category, patterns in CATEGORIES.items(): for pattern in patterns: if re.search(pattern, desc_upper, re.IGNORECASE): return category return None Two fixes make this work: | Fix | What It Does | Example | | --- | --- | --- | | `\bCVS\b` (word boundaries) | Matches "CVS" as a complete word only | Matches "CVS PHARMACY", blocks "CVSMITH" | | `FALSE_POSITIVES` checked first | Guards run before category matching | "DR PEPPER" excluded before "DR" triggers medical | The `\b` is a word boundary: it marks where a word starts or ends. Without it, `CVS` matches any string containing those three letters in sequence. With it, `\bCVS\b` only matches when CVS stands alone. The same technique works for any merchant: `\b(AMZN|AMAZON)\b` catches Amazon without matching "AMAZONIA." Extending Categories Without Writing Regex You don't need to understand regex to add new merchants. To add TARGET PHARMACY to medical, just tell Claude Code: Add TARGET PHARMACY to the medical category in tax-categorize.py.Make sure it matches "TARGET PHARMACY" but not "TARGET" by itself(I shop at Target for non-medical things too). The agent writes the regex. You describe what should and shouldn't match. That's the same pattern you've been using all chapter: describe the problem, let the agent handle the implementation. Let's verify the fix: [Runs: cat test_bank.csv | python tax-categorize.py]MEDICAL: CVS/PHARMACY #1234: $45.67MEDICAL: WALGREENS #5678: $23.45MEDICAL: DR MARTINEZ MEDICAL: $150.00CHARITABLE: UNITED WAY: $100.00BUSINESS: OFFICE DEPOT #901: $89.50--- TOTALS ---Medical: $219.12Charitable: $100.00Business: $89.50 Dr. Pepper and CVSMITH are gone. DR MARTINEZ MEDICAL still matches because it contains "MEDICAL" as a standalone word. The totals are now correct. **Checkpoint:** Run the categorizer on YOUR bank data from Lesson 3. How many transactions does it categorize? How many false positives do you spot? Regex Is a Stopgap, Not a Solution The FALSE\_POSITIVES list works for known edge cases. But it's brittle: every new false positive requires a manual update. You'll never anticipate every "DR SOMETHING" that isn't a doctor. In a real workflow, regex handles the high-confidence matches. Everything else goes into a "NEEDS REVIEW" file for human judgment. The capstone does exactly this: the report's NEEDS REVIEW section is the honest answer to "what do you do when pattern matching isn't enough?" Regex buys you the easy 80%. The hard 20% requires a human. Prove It Transfers: Server Logs[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prove-it-transfers-server-logs "Direct link to Prove It Transfers: Server Logs") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your categorizer catches Dr. Pepper. Your tools chain through pipes. Your bank statement workflow is solid. But here's the question that separates a tutorial exercise from a transferable skill: **does the pattern work on data that isn't bank statements?** Imagine you manage a web application. Your monitoring system exports weekly log CSVs: timestamp,endpoint,status_code,response_time_ms2025-01-06T08:00:00,/api/users,200,452025-01-06T08:01:00,/api/orders,201,1202025-01-06T08:02:00,/api/users,404,122025-01-06T08:03:00,/health,404,32025-01-06T08:04:00,/api/orders,500,50022025-01-06T08:05:00,/api/payments,200,892025-01-06T08:06:00,/api/payments,502,30001 You need to categorize responses (Success, Client Error, Server Error) and flag problem endpoints. Replace "medical expenses" with "server errors" and you're running the same workflow. Start exactly where you'd start with bank data: test data with known answers: You: I have server log CSVs with timestamp, endpoint, status_code,response_time_ms columns. Build me a log-analyzer that categorizesby status code: Success (2xx), Client Error (4xx), Server Error (5xx).But first — create test data with 7 rows and verify the counts.Expected: Success=3, Client Error=2, Server Error=2. The agent builds `log-analyzer.py`, runs it on the test data, and the counts match. But scan the output more carefully. That `/health` endpoint returned 404: is that actually a problem? ### The Dr. Pepper of Server Logs[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-dr-pepper-of-server-logs "Direct link to The Dr. Pepper of Server Logs") Health check endpoints return 404 during deployments. It's expected behavior, not an error. But your log analyzer counts it as a client error, inflating your error rate, just like Dr. Pepper inflated your medical deductions. You: The /health 404 is expected during deployments, not a real error.Add a KNOWN_BENIGN list like the FALSE_POSITIVES in the tax categorizer. The agent adds a guard list. FALSE\_POSITIVES became KNOWN\_BENIGN. Word boundary regex became endpoint/status\_code tuples. The concept (guard against known exceptions before categorizing) is identical. The domain changed; the pattern didn't. The workflow is the workflow. Build tools, verify first, guard false positives, only the domain knowledge changed. Dr. Pepper became `/health 404`. Medical categories became status codes. And that domain knowledge is yours, not the agent's. Your categorizer works on one file. But tax season means 12 months of bank statements, not one. The capstone puts it all together. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Design a Merchant Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prompt-1-design-a-merchant-pattern "Direct link to Prompt 1: Design a Merchant Pattern") I need to match all Amazon transactions in my bank statement.They appear as: AMZN MKTP US, AMAZON.COM, AMAZON PRIMEWrite a regex pattern that matches all these but does NOT match:- AMAZONIA TRAVEL- CAMZN AUTO PARTSExplain why word boundaries matter here. **What you're learning:** You supply the real-world examples (AMZN MKTP US, AMAZON.COM, AMAZON PRIME) AND the false positives to avoid (AMAZONIA, CAMZN). The agent has the regex knowledge. You have the data knowledge. The quality of the pattern is determined by how precisely you describe what should and shouldn't match, not by your ability to write regex. ### Prompt 2: Handle a New False Positive[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prompt-2-handle-a-new-false-positive "Direct link to Prompt 2: Handle a New False Positive") My tax-categorize.py marks "AMAZON RIVER CRUISE" as a purchasefrom Amazon.com. It's actually a travel expense.How do I add this to the false positive guards? Show me the patternthat excludes travel-related Amazon mentions. **What you're learning:** The refinement loop in practice: you discovered "AMAZON RIVER CRUISE" is wrong, which means your verification caught it. You bring the failure; the agent brings the fix. This is the same loop from Lesson 2 applied to categorization: evidence criteria (what's miscategorized) come from you, implementation (the guard pattern) comes from the agent. ### Prompt 3: Extend Categories[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prompt-3-extend-categories "Direct link to Prompt 3: Extend Categories") Help me add these categories to tax-categorize.py:- home_office: Home Depot, Lowes, Ikea (but NOT "IKEA RESTAURANT")- subscriptions: Netflix, Spotify, Hulu, Apple.comWhat false positives might I need to guard against? **What you're learning:** Shifting from reactive to proactive. Instead of waiting to discover a false positive, you ask the agent to surface them before they hit real data. "What false positives might I need to guard against?" leverages the agent's pattern knowledge to stress-test your domain decisions before they cause a problem. Checking access... * [The Conversation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-conversation) * [The Bug](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-bug) * [The Fix](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-fix) * [Prove It Transfers: Server Logs](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prove-it-transfers-server-logs) * [The Dr. Pepper of Server Logs](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#the-dr-pepper-of-server-logs) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#flashcards-study-aid) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#try-with-ai) * [Prompt 1: Design a Merchant Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prompt-1-design-a-merchant-pattern) * [Prompt 2: Handle a New False Positive](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prompt-2-handle-a-new-false-positive) * [Prompt 3: Extend Categories](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/data-wrangling#prompt-3-extend-categories) --- # Chapter 24: Project - Build Your AI Employee | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C **You've learned the pieces. Now build something real.** This is a **project**. You'll build a working AI Employee for YOUR profession using [NanoClaw](https://github.com/qwibitai/nanoclaw) : the same tool you set up in the "NanoClaw Hands-On Setup" lesson from "Meet Your First AI Employee - OpenClaw". No new installations, no architecture lectures. Just challenges, acceptance criteria, and your professional expertise. Prerequisites[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------------- * **"NanoClaw Hands-On Setup" complete** (from "Meet Your First AI Employee - OpenClaw"): NanoClaw installed, running, WhatsApp connected * **Layer 3 design from "NanoClaw Hands-On Setup"**: Your blueprint with 3+ Agent Skills and 3+ MCP servers for your profession * **Part 2 foundations complete**: File Processing, Computation and Data Extraction, Structured Data and Persistent Storage, Linux Mastery, and Version Control Three Achievement Tiers[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#three-achievement-tiers "Direct link to Three Achievement Tiers") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Tier | Path | Time | What You Get | | --- | --- | --- | --- | | **Bronze** | The Project Brief through Bronze Capstone: First Real Day | ~3 hours | Working AI Employee: identity + skill + connection + proof | | **Silver** | The Project Brief through Prove Professional Value | ~6 hours | \+ autonomous monitoring + trust boundaries + persistent memory + domain report | | **Gold** | The Project Brief through Prove Professional Value (Gold track) | ~9 hours | \+ multi-group architecture with isolation + memory isolation | Pick the tier that matches your available time and ambition. You can always come back for the next tier later. Lessons[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#lessons "Direct link to Lessons") ----------------------------------------------------------------------------------------------------------------------------------------- ### The Brief[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#the-brief "Direct link to The Brief") | Lesson | Duration | | --- | --- | | [The Project Brief](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/the-project-brief) | 20 min | ### Bronze Tier: Working AI Employee[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#bronze-tier-working-ai-employee "Direct link to Bronze Tier: Working AI Employee") | Lesson | Duration | | --- | --- | | [Give Your Employee an Identity](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-an-identity) | 30 min | | [Teach Your Employee a Skill](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-a-skill) | 40 min | | [Connect Your Employee to the World](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/connect-your-employee-to-the-world) | 35 min | | [Bronze Capstone: First Real Day](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/bronze-capstone-first-real-day) | 45 min | ### Silver Tier: Proactive + Trusted + Learning[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#silver-tier-proactive--trusted--learning "Direct link to Silver Tier: Proactive + Trusted + Learning") | Lesson | Duration | | --- | --- | | [Make Your Employee Proactive](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/make-your-employee-proactive) | 40 min | | [Teach Your Employee Boundaries](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/teach-your-employee-boundaries) | 40 min | | [Give Your Employee a Memory](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory) | 40 min | | [Prove Professional Value](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value) | 55 min | ### Project Review[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#project-review "Direct link to Project Review") | Lesson | Duration | | --- | --- | | [Project Review](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/project-review) | 25 min | Student Deliverables (GitHub Repo)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#student-deliverables-github-repo "Direct link to Student Deliverables (GitHub Repo)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ nanoclaw-employee/├── groups/│ ├── global/CLAUDE.md (shared rules)│ └── main/CLAUDE.md (profession-specific identity)├── .claude/skills/{domain-skill}/│ └── SKILL.md (from Layer 3 design)├── conversation-log.md (Bronze: First Real Day)├── evaluation.md (Bronze: self-assessment)├── scheduler-config.md (Silver: scheduled task design)├── hitl-boundaries.md (Silver: permission table)├── memory-config.md (Silver: action log + knowledge store schemas)└── domain-report-sample.md (Silver: generated report) Gold adds: 3-group config + isolation demo + system diagram + memory isolation test. Design Philosophy[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#design-philosophy "Direct link to Design Philosophy") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- This project doesn't teach NanoClaw: you already know it. Instead, each lesson gives you: 1. **A challenge** with clear acceptance criteria 2. **A use case gallery** with profession-specific examples 3. **Graduated hints** (3 levels) for when you're stuck Your Layer 3 design from the "NanoClaw Hands-On Setup" lesson in "Meet Your First AI Employee - OpenClaw" is your blueprint. This project is where you execute it. * [Prerequisites](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#prerequisites) * [Three Achievement Tiers](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#three-achievement-tiers) * [Lessons](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#lessons) * [The Brief](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#the-brief) * [Bronze Tier: Working AI Employee](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#bronze-tier-working-ai-employee) * [Silver Tier: Proactive + Trusted + Learning](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#silver-tier-proactive--trusted--learning) * [Project Review](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#project-review) * [Student Deliverables (GitHub Repo)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#student-deliverables-github-repo) * [Design Philosophy](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee#design-philosophy) --- # Pivots One and Two: Hype and Redundancy | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Emma opened a timeline on her laptop. Two entries, both from before any code was written. "The first two pivots happened before we built anything," she said. "That is important. Most people think architecture decisions happen during implementation. These happened during evaluation. We chose a platform. Then we chose the wrong tools to build on it. Both mistakes cost us time, not code." James leaned in. He had installed OpenClaw in Chapter 56. He had packaged skills for it in Chapter 57. He had built TutorClaw on it in Chapter 58. The platform felt natural to him now. But sitting here, looking at Emma's timeline, he realized he had never asked the question that triggered the first pivot. "I used OpenClaw because the course told me to," he said. "I never evaluated whether it was right for the problem." Emma smiled. "Neither did we. At first." * * * You are doing exactly what James is doing. You used OpenClaw throughout Part 5 without questioning whether it was the right platform for TutorClaw. Now you are looking at the two decisions the team made before writing any code, and both of them were wrong. Pivot 1: The OpenClaw Moment[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#pivot-1-the-openclaw-moment "Direct link to Pivot 1: The OpenClaw Moment") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The announcement landed like an earthquake. At GTC, Jensen Huang declared OpenClaw the most popular open-source project in the history of humanity. NVIDIA announced NemoClaw. OpenAI backed the foundation. The technology press erupted with predictions about the future of personal AI. The team saw an opportunity. OpenClaw's two-layer architecture, Gateway plus Agent Runtime, mapped directly to the Body plus Brain pattern they had already designed. OpenClaw's Markdown skills matched the SKILL.md format they were already using for PRIMM-AI+. The plan wrote itself: package PRIMM-AI+ as an OpenClaw skill, connect WhatsApp as a channel, plug in Claude as the model. Three components, clean integration, done. Everything mapped. The architecture diagram looked beautiful. And that was the problem. The team had started with the platform and worked backward to the requirements. OpenClaw was brilliant for personal assistants. One user, one agent, one set of preferences. But TutorClaw needed to serve thousands of learners through WhatsApp. TutorClaw needed code execution for programming exercises. TutorClaw needed monetization gating so free-tier learners got a different experience from paid learners. Nobody had tested OpenClaw against those requirements. They had tested OpenClaw against their excitement. The question that broke the spell was deceptively simple: "What problem does this solve for my users?" Not "How do I integrate this?" Not "What can this platform do?" The question that matters is what it does for the people who will use your product. OpenClaw solved the personal AI problem beautifully. It did not solve the multi-tenant tutoring problem at all. The team had been so captivated by the platform's elegance that they skipped the step of checking whether that elegance applied to their specific constraints. This is a pattern you will encounter throughout your career. A new technology appears. The demos are impressive. The community is enthusiastic. The architecture diagrams are clean. And the gravitational pull of that excitement makes it easy to adopt the technology before asking whether it fits your requirements. The lesson from Pivot 1 is not that OpenClaw was wrong. OpenClaw turned out to be exactly the right platform for TutorClaw's final architecture. The lesson is that the team adopted it for the wrong reasons. They started with "this technology is exciting" instead of "this technology solves our problem." The fact that it eventually turned out to be the right choice was luck, not judgment. Pivot 2: The SDK Confusion[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#pivot-2-the-sdk-confusion "Direct link to Pivot 2: The SDK Confusion") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With OpenClaw selected as the platform, the next question seemed straightforward: which SDK should TutorClaw use? Three options were on the table: | Option | What It Does | Strength | Constraint | | --- | --- | --- | --- | | Claude Agent SDK | Computer-centric agent framework | Deep integration with Claude models | Claude-only, no model flexibility | | OpenAI Agents SDK | Multi-agent orchestration with Handoffs | Model-agnostic, supports agent teams | Adds an orchestration layer the team did not need | | OpenClaw native runtime | Built-in agent loop with tool discovery | Already running, zero additional setup | No multi-agent orchestration out of the box | The team spent days evaluating these options. Comparing features. Reading documentation. Building small prototypes. Then someone drew a diagram that made the entire debate irrelevant. ### The Three-Layer Diagram[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#the-three-layer-diagram "Direct link to The Three-Layer Diagram") Every AI agent system operates on three layers: | Layer | What It Does | Example | | --- | --- | --- | | **LLM** | Raw intelligence: understands language, generates responses, reasons about problems | Claude, GPT, Gemini | | **Agent Runtime** | The loop: receives a message, calls the LLM, uses tools, returns a response, waits for the next message | OpenClaw's agent runtime | | **Agent SDK** | A framework for building runtimes: provides abstractions for tool registration, multi-agent coordination, state management | Claude Agent SDK, OpenAI Agents SDK | The key insight: OpenClaw already provides a runtime. It already has the loop. It already handles tool discovery, message routing, and response generation. That is what the Agent Runtime layer does. An Agent SDK is a framework for building a runtime. If you already have a runtime, plugging an SDK into it means running an agent loop inside an agent loop. Picture it concretely. OpenClaw receives a message from the user. OpenClaw's runtime passes it to the agent. If the agent is using the Claude Agent SDK, the SDK creates its own loop: it calls Claude, gets a response, checks for tool calls, executes tools, calls Claude again. Then it returns the final response to OpenClaw's runtime, which passes it back to the user. Two loops, two sets of tool management, two layers of message handling. The inner loop does what the outer loop already does. This is the layer stacking anti-pattern. It is not a performance problem (both loops are fast). It is a complexity problem. Two loops means two places where errors can occur. Two places where tool registration must be maintained. Two places where message formatting must be consistent. The architecture is harder to debug, harder to maintain, and harder to reason about, with zero additional capability. ### The Resolution[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#the-resolution "Direct link to The Resolution") The team chose the simplest architecture: OpenClaw's native runtime with Claude as the model. No SDK layer. The runtime handles the loop. Claude handles the intelligence. Tools are registered once, with OpenClaw, not twice. This decision eliminated an entire category of bugs (SDK-to-runtime integration issues), removed a dependency (no SDK to version, update, or debug), and simplified the mental model (one loop, not two). The principle behind Pivot 2 applies beyond agent systems. When evaluating any tool, identify which layer it operates at. If two tools operate at the same layer, one of them is redundant. The simplest architecture is the one that uses exactly one tool per layer. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Exercise 1: Layer Map Your Own Stack[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#exercise-1-layer-map-your-own-stack "Direct link to Exercise 1: Layer Map Your Own Stack") Think about a project you are building or planning. Use this prompt to identify which layer each tool operates at: I am building a project that uses these tools:[list your tools, frameworks, and libraries]Help me classify each tool into one of three layers:1. Intelligence layer (provides reasoning, language understanding)2. Runtime layer (provides the execution loop, message handling)3. Framework/SDK layer (provides abstractions for building runtimes)Then check: are any two tools operating at the same layer? If so,which one is redundant? What would the architecture look like ifI removed the redundant one? **What you are learning:** The layer stacking anti-pattern is not specific to agent systems. Any technology stack can have redundant layers: two ORMs, two routing frameworks, two state management libraries. By classifying your tools into layers, you develop the habit of checking for redundancy before it becomes a maintenance burden. The simplest architecture uses one tool per layer. ### Exercise 2: The Hype Evaluation Framework[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#exercise-2-the-hype-evaluation-framework "Direct link to Exercise 2: The Hype Evaluation Framework") Think about a technology you are excited about or have recently adopted. Use this prompt to test whether your adoption is hype-driven or requirements-driven: I am considering using [technology name] for my project.Before I evaluate the technology itself, help me define myrequirements:1. What specific problem does my project need to solve?2. What constraints does my project have (scale, cost, team size, timeline)?3. What would a successful solution look like from my users' perspective?Now evaluate the technology against those requirements:4. Does it solve my specific problem, or a related but different problem?5. Does it meet my constraints, or does it require me to change my constraints?6. If I removed this technology, what would I lose that my users actually need? **What you are learning:** The question "What problem does this solve for my users?" is a filter that separates hype from fit. Technologies can be genuinely excellent and still wrong for your specific use case. By defining your requirements before evaluating the technology, you avoid the trap of working backward from excitement to justification. This discipline saves weeks of rework when the excitement fades and the constraints remain. ### Exercise 3: Spot the Redundant Layer[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#exercise-3-spot-the-redundant-layer "Direct link to Exercise 3: Spot the Redundant Layer") Use this prompt to practice identifying redundancy in a technology stack: Here is a technology stack for an AI application:- A language model API (e.g., Claude API)- An agent framework (e.g., LangChain or CrewAI)- A platform with a built-in agent runtime (e.g., OpenClaw)- A database for conversation history- A web framework for the API layerAnalyze this stack for layer stacking:1. Draw the three layers (LLM, Runtime, SDK/Framework)2. Place each technology into a layer3. Identify any layer that has more than one tool4. For each redundancy, explain what happens at runtime: which loop calls which loop? Where do tools get registered?5. Propose a simplified stack that uses one tool per layer **What you are learning:** Redundant layers are easy to add and hard to notice. When you adopt a framework because it has good documentation and then deploy it inside a platform that already provides the same capability, the redundancy is invisible until something breaks. Practicing layer analysis on hypothetical stacks trains you to see the anti-pattern before you build it into a production system. * * * James sat back in his chair. He was thinking about a vendor his warehouse had nearly signed with two years ago. "We had a supplier come in with an incredible pitch," he said. "Automated sorting system. Laser scanners, conveyor routing, the whole package. Beautiful demo. Our operations team was ready to sign the contract on the spot." "What happened?" "I asked what problem it solved for our customers. Our customers needed packages sorted by delivery zone, which our existing conveyor system already did. The new system sorted by package weight, which our customers never asked for. It was better technology solving the wrong problem." Emma nodded. "That is Pivot 1. And Pivot 2?" James thought for a moment. "Actually, we almost made a Pivot 2 mistake in the same project. Our IT team suggested adding a warehouse management system on top of our existing inventory tracking software. Both systems tracked the same data. We would have been running two inventory loops, one feeding into the other, with no new information flowing through." "An inventory loop inside an inventory loop." "Exactly. We caught it because someone drew a diagram of what data flowed where, and two boxes on the diagram did exactly the same thing." Emma leaned back. "I keep going back and forth on which of these two pivots was more expensive to learn. The hype pivot or the redundancy pivot." She paused, genuinely uncertain. "The hype one cost us emotional energy: we had to let go of excitement and evaluate coldly. The redundancy one cost us intellectual energy: we had to understand three layers well enough to see the overlap. I still do not have a good answer for which lesson was more expensive to learn. Maybe it depends on the person." James looked at the three-layer diagram in his notes. "For me, the redundancy one. I can resist hype. I have done it before. But seeing that two tools do the same thing when their documentation makes them sound completely different? That takes a kind of analysis I had to learn." Emma closed her laptop. "The next two pivots happened when we tried to scale. The architecture that worked for one person collapsed at sixteen thousand." Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Checking access... * [Pivot 1: The OpenClaw Moment](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#pivot-1-the-openclaw-moment) * [Pivot 2: The SDK Confusion](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#pivot-2-the-sdk-confusion) * [The Three-Layer Diagram](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#the-three-layer-diagram) * [The Resolution](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#the-resolution) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#try-with-ai) * [Exercise 1: Layer Map Your Own Stack](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#exercise-1-layer-map-your-own-stack) * [Exercise 2: The Hype Evaluation Framework](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#exercise-2-the-hype-evaluation-framework) * [Exercise 3: Spot the Redundant Layer](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#exercise-3-spot-the-redundant-layer) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-one-two#flashcards-study-aid) --- # Chapter 22: Linux Operations for Agent Deployment | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Ali built a competitor-analysis agent in Chapters 20 and 21. It scrapes pricing data, stores results in a database, and generates daily summaries. He runs it on his laptop. It works: when his laptop is open. Sunday night. His biggest client has a board meeting Monday morning. The agent was supposed to generate a weekend pricing report. Ali opens his laptop and checks the dashboard. The agent has not run since Friday. Three days of missing data. Board meeting in twelve hours. His friend Dev has a cloud server. "I'll give you SSH access," Dev texts. "Get your agent running there and it won't depend on your laptop anymore." Ali opens a terminal. Types an SSH command. A blinking cursor on a black screen. No desktop. No icons. No file explorer. This chapter is how Ali, and you: go from that blinking cursor to a production agent that runs 24/7, survives reboots, and recovers from failures. You will not memorize Linux commands. You will direct Claude Code to execute them, read what comes back, and understand what it means. Chapter Thesis Your agent's code might be perfect. If you can't deploy it to a server, keep it running, and fix it when it breaks, it's a side project, not a product. Linux operations is the bridge between "it works on my laptop" and "it runs in production." 📚 Teaching Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#-teaching-aid "Direct link to 📚 Teaching Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------- [🖥️ Fullscreen](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/slides/part-2/chapter-11/linux-mastery.pdf) What You'll Learn[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#what-youll-learn "Direct link to What You'll Learn") ------------------------------------------------------------------------------------------------------------------------------------------------------------ By the end of this chapter, Ali's competitor-tracker agent runs as a **production systemd service** that: * Starts automatically when the server boots * Restarts automatically if it crashes * Logs all activity for monitoring * Runs under a dedicated non-root user * Accepts connections only via SSH keys * Can be diagnosed systematically when problems occur This is a real deployment, not a toy example. Prerequisites[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------- Before starting this chapter, you should have completed: * **Chapter 17: Seven Principles of Agent Work**: Especially Principle 1 (Bash is the Key) and Principle 5 (Persisting State in Files) * **Chapters 19–21**: You've built agent workflows for file processing, Python, and SQL **No prior Linux experience is required.** Lesson 1 starts from a blinking cursor. Windows Users If you're on Windows, you need WSL2 (Windows Subsystem for Linux). Run `wsl --install` in PowerShell as Administrator, then restart your computer. All commands in this chapter work in WSL2 Ubuntu. Lessons Overview[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#lessons-overview "Direct link to Lessons Overview") ---------------------------------------------------------------------------------------------------------------------------------------------------------- | Lesson | Title | Duration | Focus | | --- | --- | --- | --- | | [Lesson 1](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives) | Where Your Agent Lives | 30 min | SSH, filesystem, server orientation | | [Lesson 2](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does) | Reading What Your Agent Does | 30 min | Command output, permissions, vocabulary | | [Lesson 3](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home) | Setting Up Your Agent's Home | 30 min | Directory structure, .env, logs | | [Lesson 4](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable) | Making Your Agent Unkillable | 35 min | systemd services, restart policies | | [Lesson 5](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door) | Locking the Door | 30 min | Users, permissions, SSH keys, security | | [Lesson 6](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong) | When Things Go Wrong | 35 min | LNPS triage, log analysis, debugging | | [Lesson 7](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production) | Capstone: Zero to Production | 45 min | Deployment spec, full integration | | [Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises) | Linux Operations Exercises | 60 min | Hands-on practice across all lessons | | [Quiz](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/chapter-quiz) | Chapter Quiz | 20 min | 50-question conceptual assessment | **Total: ~5.25 hours** (including exercises and quiz) The Teaching Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#the-teaching-pattern "Direct link to The Teaching Pattern") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every lesson in this chapter follows the same flow: 1. **Story beat**: Ali hits a real problem 2. **Direct Claude Code**: You tell the agent what to do in plain English 3. **Read the output**: Understand what came back and why 4. **Build the mental model**: Connect the output to the concept You will not type Linux commands from memory. You will direct an agent, watch what it does, and learn to understand the results. Quick Reference: Linux Terms in Plain English[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#quick-reference-linux-terms-in-plain-english "Direct link to Quick Reference: Linux Terms in Plain English") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ New to Linux? These 12 terms appear throughout the chapter. Each definition is two sentences maximum. | Term | Plain English | | --- | --- | | **Shell** | The program that reads your commands and tells the OS to run them. On Linux servers, this is almost always bash. | | **Terminal** | The window application that shows your shell. Terminal is the glass, shell is the voice on the other end. | | **Directory** | What Windows calls a "folder." On Linux, we say "directory": same concept, different word. | | **Root** | Two meanings:`/` is the top of the filesystem tree, and `root` is the superuser account with unlimited power. Context tells you which. | | **Daemon** | A service that runs in the background, started at boot, not attached to any terminal. Your agents become daemons in Lesson 4. | | **Port** | A numbered channel for network communication. Your agent listens on a specific port (e.g., 8080) for incoming requests. | | **Process** | Any running program. Your agent is a process with an ID number (PID). | | **Pipe** | The `\|` character. Takes the output of one command and feeds it as input to the next. | | **Redirect** | Sending output to a file (`>`) instead of the screen. `>>` appends; `>` overwrites. | | **Absolute path** | A file address starting from `/`. Works regardless of where you are (e.g., `/var/log/agent.log`). | | **sudo** | "Run this as root." Grants temporary admin power for one command. Use sparingly; misuse causes real damage. | | **Environment variable** | A named value available to all processes in a session. Used for secrets, configuration, and API keys. | Let's Begin[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#lets-begin "Direct link to Let's Begin") ------------------------------------------------------------------------------------------------------------------------------------------ Ali is staring at a blinking cursor on Dev's server. Twelve hours until the board meeting. The agent needs to be running by morning. [Start with Lesson 1: Where Your Agent Lives →](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives) * [📚 Teaching Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#-teaching-aid) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#what-youll-learn) * [Prerequisites](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#prerequisites) * [Lessons Overview](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#lessons-overview) * [The Teaching Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#the-teaching-pattern) * [Quick Reference: Linux Terms in Plain English](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#quick-reference-linux-terms-in-plain-english) * [Let's Begin](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery#lets-begin) --- # Your First Agent Workflow | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Let's be honest about something before we start. If you need to check whether a file exists, you're faster than any AI agent. Type `ls budget.csv`: done in two seconds. The agent would need to think, pick a command, ask permission, then respond. Fifteen seconds for a two-second task. So why are you reading a chapter about directing agents to handle files? Because your Downloads folder has 847 files in it. You don't know what's in half of them. Three of them are tax documents you'll need in April. One is a duplicate that's eating 2 GB of disk space. And the file your boss asked about yesterday? It's in there somewhere, buried under six months of random downloads. You could spend an afternoon sorting through them manually. Or you could describe the problem in one sentence: You: I have a cluttered Downloads folder. Help me understand what'sin here. How many files, what types, what's taking up space? That's it. No commands, no technical details, no specifying _how_ to analyze it. The agent ran six commands, self-corrected when its first count was wrong, and produced a complete breakdown in 30 seconds. **The rule of thumb you'll internalize in this chapter:** If you can describe the task faster than you can do it, ask the agent. If you can do it faster than you can describe it, just do it. Three files to move? Drag them. Three hundred files to categorize by type, date, and project? That's agent territory. The pattern you'll learn today (describe the problem, let the agent figure out the solution) applies to data analysis, project management, research, and any domain where you work with General Agents. The Conversation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-conversation "Direct link to The Conversation") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open Claude Code. Here's the full exchange: including the parts most tutorials edit out. You: I have a cluttered Downloads folder. Help me understand what'sin here. How many files, what types, what's taking up space? Claude Code: I'll analyze your Downloads folder. Let me take a look.> Allow Claude Code to run: ls -la ~/Downloads | head -20> [Y/n]: YYou have 23 items in your Downloads folder. Actually, let merecount — that's just the top-level view. Some of those arefolders with files inside them. Let me get the real number.> Allow Claude Code to run: find ~/Downloads -type f | wc -l> [Y/n]: YYou have 847 files in your Downloads folder — much more than the23 items that ls showed, because many are nested in subfolders. Pause. Did you catch what just happened? The agent got it wrong the first time: 23 items, and then corrected itself. It didn't hide the mistake or pretend the first answer was right. It noticed that `ls` only shows the top level, switched to `find -type f`, and got the real number. That self-correction is what good agent work looks like. Claude Code: Let me break down what types of files you have.[Runs: find ~/Downloads -type f -name "*.pdf" | wc -l][Runs: find ~/Downloads -type f -name "*.xlsx" -o -name "*.csv" | wc -l][Runs: find ~/Downloads -type f -name "*.jpg" -o -name "*.png" | wc -l][Runs: find ~/Downloads -type f -name "*.zip" | wc -l]Here's what I found:| File Type | Count | Notes ||---------------|-------|-------------------------------|| PDFs | 234 | Documents, reports, receipts || Spreadsheets | 89 | .xlsx and .csv files || Images | 156 | Screenshots, photos || Archives | 43 | .zip files (may contain more) || Other | 325 | Mixed: .docx, .dmg, .exe |Now let me see what's taking up the most space.[Runs: du -sh ~/Downloads][Runs: du -ah ~/Downloads | sort -rh | head -10]Total folder size: 12.4 GBYour largest files:1. project-backup.zip (2.1 GB)2. video-recording.mp4 (1.8 GB)3. database-export.sql (890 MB)4. presentation-final.pptx (456 MB)5. dataset.csv (312 MB)Just these 5 files account for 5.5 GB. Almost half your folder.Would you like me to help you organize these, or should I lookfor duplicates first? That took about 30 seconds. You now know: * Total file count (847) * Breakdown by type * Total size (12.4 GB) * Where the space is going You didn't run a single command. You described a problem, and the agent solved it. The Division of Labor[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-division-of-labor "Direct link to The Division of Labor") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Look at what just happened through a different lens. You described the problem in language: "cluttered Downloads folder." The agent chose the solution in code: `ls`, `find`, `wc`, `du`. You couldn't have chosen those commands (you don't know bash). The agent couldn't have chosen what to analyze (it doesn't know what matters to you). Neither of you could have done the other's job. That's the insight: the value isn't automation. It's the combination of your judgment with the agent's technical execution. ### And Sometimes the Agent is the Wrong Tool Entirely[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#and-sometimes-the-agent-is-the-wrong-tool-entirely "Direct link to And Sometimes the Agent is the Wrong Tool Entirely") Not every file task belongs in a terminal: * **Browsing photos**: You need to _see_ thumbnails to decide which to keep. The agent sees filenames and metadata. You see memories. * **Network drives and NAS**: File operations on mounted network storage can behave differently: latency, permission models, case sensitivity. The agent doesn't know your NAS has a 2-second round trip per operation. * **Decisions that need human eyes**: That folder of old photos from college? Don't let an agent bulk-delete based on file size rules. Some decisions aren't computational. The best collaborators know when to hand off and when to handle it themselves. Acknowledging the agent's limits is part of the skill, not a weakness. * * * ### The Agent's Toolkit: What Those Commands Mean[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-agents-toolkit-what-those-commands-mean "Direct link to The Agent's Toolkit: What Those Commands Mean") The agent ran several commands. Let's decode them. #### The Building Blocks[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-building-blocks "Direct link to The Building Blocks") * **`ls`**: **l**i**s**t files in a folder * **`find`**: **find** files matching a pattern * **`wc`**: **w**ord **c**ount (counts lines, words, or characters) * **`du`**: **d**isk **u**sage (measures sizes) * **`sort`**: **sort** results in order #### Anatomy of a Command[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#anatomy-of-a-command "Direct link to Anatomy of a Command") Let's break down one command the agent ran: find ~/Downloads -type f -name "*.pdf" | wc -l | | | | | | | | | | | +-- -l = count Lines only | | | | | | | | | +-- pipe: "then do this..." | | | | | | | +-- -name = match this filename pattern | | | | | +-- -type f = only Files (not folders) | | | +-- where to search | +-- the command Read it left to right: "**Find** in **Downloads**, only **files**, named `*.pdf`, **then** count **lines**." #### Common Flags You'll See[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#common-flags-youll-see "Direct link to Common Flags You'll See") | Flag | Means | Memory Trick | | --- | --- | --- | | `-l` | Lines (or Long listing) | **l**ines | | `-h` | Human-readable (KB, MB, GB) | **h**uman | | `-r` | Reverse order | **r**everse | | `-s` | Summary (totals only) | **s**ummary | | `-a` | All (including hidden files) | **a**ll | When the agent ran `du -sh`, that's "**d**isk **u**sage, **s**ummary, **h**uman-readable." When it ran `sort -rh`, that's "**sort**, **r**everse order, **h**uman-readable": biggest first. #### The Pipe: Chaining Tools Together[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-pipe-chaining-tools-together "Direct link to The Pipe: Chaining Tools Together") The pipe (`|`) connects commands. Output from the left feeds into the right: find ~/Downloads -name "*.pdf" | wc -l "Find PDFs, **then** count them." Small tools, chained together, solving big problems. That's what makes the shell so powerful. You don't need to memorize these commands. But recognizing them helps you understand what the agent is doing, and verify it's doing the right thing. The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-pattern "Direct link to The Pattern") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the prompt pattern you just used: "Help me understand [my problem]. Show me [what I need to know]." This pattern works because it does two things: 1. **Describes the problem, not the solution.** You said "cluttered Downloads folder," not "run `find` and `du` commands." You let the agent figure out _how_ to help. 2. **Specifies the outcome you want.** You asked for "how many files, what types, what's taking up space." The agent knew what success looked like. This is fundamentally different from learning bash commands yourself. You're not memorizing syntax. You're describing problems and letting the agent choose the right tools. The pattern generalizes. Try these variations: | Your Problem | The Prompt | | --- | --- | | Messy Desktop | "Help me understand my Desktop. What's here, what's old, what's taking up space?" | | Mystery disk usage | "Help me understand where my disk space is going. What folders are largest?" | | Project scattered across folders | "Help me understand where files related to \[project name\] are scattered." | | Duplicate photos | "Help me understand if I have duplicate photos. Show me likely duplicates." | The specifics change. The pattern stays the same. ✅ Checkpoint: Do This Now[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#-checkpoint-do-this-now "Direct link to ✅ Checkpoint: Do This Now") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Stop reading. Open Claude Code and run the survey on YOUR Downloads folder. Use this prompt (modify if needed): Help me understand what's in my Downloads folder. How many files,what types, what's taking up space? Show me the full picture. After Claude Code finishes, ask it to save the results: Save this analysis as FILE-INVENTORY.md in a new folder called file-organizer/ You should now have: A `file-organizer/FILE-INVENTORY.md` file with your actual file analysis. **Don't move to Lesson 2 until you've created this file.** Everything that follows builds on understanding YOUR files, not the example files in this book. > **What to do if your Downloads folder is empty or tiny:** Pick a different folder: Documents, Desktop, or any folder with at least 20 files. Adjust the prompt accordingly. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Go Deeper[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-1-go-deeper "Direct link to Prompt 1: Go Deeper") I tried the "help me understand" pattern on my Downloads folder.Now I want to go deeper. Based on what you found, what shouldI investigate next? Pick the most interesting finding andhelp me understand that in more detail. **What you're practicing:** Iterative prompting. One analysis leads to the next question. You're building a habit of drilling down, not stopping at the first answer. ### Prompt 2: Translate to a new domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-2-translate-to-a-new-domain "Direct link to Prompt 2: Translate to a new domain") The "help me understand [problem], show me [what I need]" patternworked for files. I want to apply it to something else.Help me understand my recent shell history. What commands have Irun most often? What patterns do you see in how I use my computer? **What you're practicing:** Pattern transfer. The same prompt structure works for different problems. You're learning that General Agents respond to well-structured requests regardless of domain. ### Prompt 3: Reflect on the agent's approach[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-3-reflect-on-the-agents-approach "Direct link to Prompt 3: Reflect on the agent's approach") I watched you analyze my Downloads folder. You used commandslike find, wc, and du.Why did you choose those specific commands? What alternativesdid you consider? What would you have done differently if I'dasked a slightly different question? **What you're practicing:** Understanding agent reasoning. By asking the agent to explain its choices, you learn how it thinks. This makes you better at directing it in the future. ### Prompt 4: The Bash Tutor[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-4-the-bash-tutor "Direct link to Prompt 4: The Bash Tutor") Break down this command for me piece by piece:du -ah ~/Downloads | sort -rh | head -10What does each part do? What do the flags like '-ah', '-rh',and '-10' mean? Teach me to read this command. **What you're practicing:** Just-in-time learning. Instead of memorizing a textbook, you ask the agent to teach you the specific syntax relevant to the problem you just solved. The agent becomes your tutor, explaining commands in context. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Downloads folder is visible now. But visibility creates a new problem. You can see exactly what's there, and some of it matters. Tax documents. Financial spreadsheets. Irreplaceable photos buried under random downloads. If the next step goes wrong, those files could disappear. And unlike code, your files don't have `git revert`. Checking access... * [The Conversation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-conversation) * [The Division of Labor](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-division-of-labor) * [And Sometimes the Agent is the Wrong Tool Entirely](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#and-sometimes-the-agent-is-the-wrong-tool-entirely) * [The Agent's Toolkit: What Those Commands Mean](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-agents-toolkit-what-those-commands-mean) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#the-pattern) * [✅ Checkpoint: Do This Now](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#-checkpoint-do-this-now) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#try-with-ai) * [Prompt 1: Go Deeper](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-1-go-deeper) * [Prompt 2: Translate to a new domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-2-translate-to-a-new-domain) * [Prompt 3: Reflect on the agent's approach](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-3-reflect-on-the-agents-approach) * [Prompt 4: The Bash Tutor](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#prompt-4-the-bash-tutor) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/your-first-agent-workflow#flashcards-study-aid) --- # The Safety-First Pattern | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Here's what happens when you skip the backup. You ask the agent to organize your Downloads folder. It moves 400 files into categories. Clean, fast, satisfying. Then you notice: your tax documents aren't in the `documents/` folder. They're not in `spreadsheets/` either. You check `misc/`. Not there. You check the original Downloads folder. It's empty: the agent moved everything out. Your 1099 forms are gone. Not deleted: _misplaced_. They're sitting in a folder called `archives/` because the agent categorized `.pdf` files older than a year as "archival." You didn't know that rule existed. You didn't ask. And now you're spending 30 minutes hunting through subfolders for documents your accountant needs by Friday. That's the _good_ outcome. The bad outcome is a naming collision: two files called `report.pdf` from different folders, and the agent overwrites one with the other. The bad outcome is a permissions error halfway through a batch move that leaves 200 files in limbo: half moved, half not, and you can't remember which were where. This lesson exists because of that gap between "the agent did what I asked" and "the agent did what I _meant_." > **The paradox:** The backup constraint doesn't limit you. It frees you. Every safety rule you think slows you down actually speeds you up. Without a safety net, you hesitate at every step. With one, you experiment without fear. The 2 minutes you spend on backup save the 20 minutes of anxiety that would otherwise paralyze every decision. In the next 20 minutes, you'll direct Claude Code to create a backup, verify it's complete, and only then proceed. This pattern applies far beyond files (code changes, data transformations, system configurations) any workflow where mistakes have consequences. * * * The Conversation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#the-conversation "Direct link to The Conversation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You're ready to organize your Downloads folder, but you're nervous about losing something important. Here's how to direct Claude Code to establish safety first. ### You Start With Safety[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#you-start-with-safety "Direct link to You Start With Safety") **You:** > Before we make any changes to my Downloads folder, I want a backup of important files. Can you help me create one? **Claude Code:** > Good thinking. Creating a backup before making changes is the safest approach. Before I create the backup, I need to understand what counts as "important" for you. > > I can backup: > > * **All files** (complete safety, largest backup) > * **Files modified in the last N days** (recent work you're actively using) > * **Specific file types** (PDFs, documents, spreadsheets: common important formats) > * **A custom selection** (you tell me exactly what matters) > > Which approach works best for you? ### You Define What Matters[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#you-define-what-matters "Direct link to You Define What Matters") Notice what just happened. The agent didn't assume what you wanted. It asked. This is critical. Different people have different definitions of "important." A photographer cares about images. An accountant cares about spreadsheets. A researcher cares about PDFs. **You:** > Let's backup all PDFs and any file modified in the last 30 days. **Claude Code:** > That's a good combination. PDFs often contain important documents, and anything modified recently is likely something you're actively working with. > > I'll create a timestamped backup folder and copy those files. Here's what I'm going to do: > > 1. Create `backup/downloads-backup-2026-02-12/` in your workspace > 2. Copy all `.pdf` files from Downloads > 3. Copy all files modified in the last 30 days > 4. Report what was backed up > > Creating backup now... > > Created: backup/downloads-backup-2026-02-12/Copied: 47 PDF files (12.3 MB)Copied: 89 recently modified files (156.2 MB)Total: 136 files backed up (168.5 MB) > > Backup complete. Want me to verify it's complete before we proceed? ### You Verify the Backup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#you-verify-the-backup "Direct link to You Verify the Backup") **You:** > Yes, verify the backup is complete. **Claude Code:** > Comparing source and backup... > > PDF files in Downloads: 47PDF files in backup: 47 ✓Recent files in Downloads: 89Recent files in backup: 89 ✓All files successfully backed up. > > The backup matches the source. You now have a complete copy of your important files. If anything goes wrong during organization, you can restore from `backup/downloads-backup-2026-02-12/`. * * * Three Things to Notice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#three-things-to-notice "Direct link to Three Things to Notice") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **1\. The agent asked what "important" meant.** You said "backup important files." A dangerous agent would have assumed "everything" and started copying 12 GB. This agent asked: "What counts as important?" That single question (clarifying before acting) is the difference between a helpful tool and a dangerous one. The agent ASKED before acting. A dangerous agent would have assumed. **2\. You verified, not just trusted.** The agent could have said "Done!" after copying. Instead, you directed it to compare counts: 47 PDFs in source, 47 in backup. Numbers match. Backup complete. Trust, but verify: always. **3\. The order was the whole point.** Backup → Verify → Then make changes. Not the other way around. The backup _enables_ the changes that follow. * * * When the Backup Itself Goes Wrong[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#when-the-backup-itself-goes-wrong "Direct link to When the Backup Itself Goes Wrong") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's something textbooks usually skip. Sometimes the backup doesn't work perfectly. ### Scenario: Permissions Error[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#scenario-permissions-error "Direct link to Scenario: Permissions Error") You ask the agent to backup your Downloads folder, and it reports: Claude Code: Backup in progress...cp: cannot open '/Users/you/Downloads/restricted-report.pdf' for reading: Permission deniedCopied: 134 of 136 filesFailed: 2 files (permission errors) What happened? Some files have restrictive permissions. The agent couldn't read them. What you should say: You: Two files failed. Can you show me which ones and fixthe permissions so we can back them up? The agent will use `chmod` to adjust permissions or `sudo` if needed. The key lesson: **check for errors in the backup output**. A "mostly complete" backup can give false confidence. ### Scenario: Disk Space[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#scenario-disk-space "Direct link to Scenario: Disk Space") You ask for a full backup of a 50 GB folder, and: Claude Code: Your backup destination only has 12 GB of free space.A full backup would require approximately 50 GB.Options:1. Backup only recent files (estimated 8 GB)2. Backup only documents, not media (estimated 3 GB)3. Use a different destination with more space4. Compress the backup (slower but smaller)Which approach would you prefer? The agent caught the problem BEFORE running out of space mid-backup. This is why good agents check preconditions. * * * The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#the-pattern "Direct link to The Pattern") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the pattern you just learned, expressed as a reusable template: ### Before Any Destructive Operation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#before-any-destructive-operation "Direct link to Before Any Destructive Operation") "Before [making changes / reorganizing / deleting / moving],create a backup of [what matters to me]." Examples: * "Before reorganizing my Downloads, create a backup of all documents." * "Before deleting old files, create a backup of anything from the last year." * "Before renaming my photo folders, create a backup of the entire Photos directory." ### After the Backup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#after-the-backup "Direct link to After the Backup") "Verify the backup is complete. Show me any errors." This step is non-negotiable. A backup that fails silently is worse than no backup. It gives false confidence. ### Only Then Proceed[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#only-then-proceed "Direct link to Only Then Proceed") "Now we can [make the changes]." With verified backup in place, you can proceed with confidence. * * * The Safety-First Mindset[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#the-safety-first-mindset "Direct link to The Safety-First Mindset") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This pattern extends beyond file organization. It's a universal safety mindset: | Domain | Safety-First Pattern | | --- | --- | | **File organization** | Backup before moving files | | **Code changes** | Commit before refactoring | | **Database updates** | Export before modifying | | **System configuration** | Snapshot before changing settings | The common thread: **create a reversible state before any irreversible action**. * * * What Your Backup Enables[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#what-your-backup-enables "Direct link to What Your Backup Enables") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your backup directory is now a safety net. Here's what it enables for the rest of this chapter: | Scenario | Recovery | | --- | --- | | Script miscategorizes files | Restore from backup | | Accidentally delete something | Copy back from backup | | Want to try different rules | Reset and experiment | | Organization goes wrong | Start fresh | In Lesson 5, you'll deliberately make a mistake and practice recovery. The backup you created now makes that learning safe. * * * ✅ Checkpoint: Do This Now[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#-checkpoint-do-this-now "Direct link to ✅ Checkpoint: Do This Now") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Stop reading. Open Claude Code and create a backup of your Downloads folder (or whichever folder you surveyed in Lesson 1). Use this prompt: Before we make any changes to my [Downloads/Documents/Desktop] folder,create a timestamped backup of all PDFs and any file modified in thelast 30 days. Put it in file-organizer/backup/ and verify the backupis complete. Show me any errors. You should now have: * `file-organizer/backup/` with a timestamped subfolder * A verification report confirming the backup is complete **Don't move to Lesson 3 until your backup is verified.** Everything that follows assumes you have a safety net. * * * Try With AI: Extended Practice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#try-with-ai-extended-practice "Direct link to Try With AI: Extended Practice") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Prompt 1: Selective Backup Strategy** I want to backup my Documents folder, but it's 50GB. Help me createa smarter backup that only includes:- Files modified in the last 90 days- Any file larger than 10MB (probably important)- All PDFs regardless of dateShow me what this would capture before creating the backup. **What you're practicing:** Compound backup criteria. You're learning to combine multiple filters (date, size, type) to create targeted backups instead of copying everything. **Prompt 2: Backup Verification Deep Dive** I have a backup folder from last week. Help me verify it's still valid:- Are all the source files still in the backup?- Did any source files change since the backup?- Are there files in the backup that no longer exist in the source?Give me a complete integrity report. **What you're practicing:** Backup auditing. Real backups can become stale. You're learning to direct the agent to perform comprehensive verification, not just count files. **Prompt 3: Recovery Rehearsal** Pretend I accidentally deleted an important file called "budget-2025.xlsx"from my Documents folder. Walk me through exactly how I would recover itfrom my backup. Show me the commands but don't actually run them yet. **What you're practicing:** Recovery planning. The best time to learn recovery is before you need it. You're practicing the restore workflow in a safe, hypothetical scenario. * * * Key Takeaways[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#key-takeaways "Direct link to Key Takeaways") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Safety enables action.** The backup constraint doesn't limit you. It frees you to experiment without fear. **Verification is non-negotiable.** A backup that might have failed is worse than no backup. Always confirm completeness. Always ask for error reports. **This pattern is universal.** Backup-before-change applies to files, code, databases, and any system where actions might be irreversible. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your files are backed up. You can experiment without fear. But here's the question that matters now: which files should live together? A PDF named "Q4-Budget-Draft.pdf": is that a financial document or a Q4 project file? Your backup makes the answer cheap to get wrong. So how do you design rules that handle ambiguity? Checking access... * [The Conversation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#the-conversation) * [You Start With Safety](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#you-start-with-safety) * [You Define What Matters](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#you-define-what-matters) * [You Verify the Backup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#you-verify-the-backup) * [Three Things to Notice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#three-things-to-notice) * [When the Backup Itself Goes Wrong](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#when-the-backup-itself-goes-wrong) * [Scenario: Permissions Error](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#scenario-permissions-error) * [Scenario: Disk Space](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#scenario-disk-space) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#the-pattern) * [Before Any Destructive Operation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#before-any-destructive-operation) * [After the Backup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#after-the-backup) * [Only Then Proceed](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#only-then-proceed) * [The Safety-First Mindset](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#the-safety-first-mindset) * [What Your Backup Enables](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#what-your-backup-enables) * [✅ Checkpoint: Do This Now](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#-checkpoint-do-this-now) * [Try With AI: Extended Practice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#try-with-ai-extended-practice) * [Key Takeaways](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#key-takeaways) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-safety-first-pattern#flashcards-study-aid) --- # Making Your Agent Unkillable | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Ali's agent has a proper home now. Organized directories, secrets in `.env`, logs that persist. He starts the agent with a simple command. It works. He watches the logs scroll by, satisfied. Then he closes his laptop to go to bed. Morning. Coffee. He opens the laptop, SSHes back into Dev's server, checks the dashboard. The agent has not run since he went to sleep. Three days of missing data just became four. He starts it again. It works again. The server reboots for a security update at 3 AM. The agent dies again. > **"Close your laptop. Your agent dies. Reboot the server. Your agent dies. It's a pet that dies every time you stop watching it."** The fix is one file. * * * Why Your Agent Keeps Dying[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#why-your-agent-keeps-dying "Direct link to Why Your Agent Keeps Dying") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you start a program in a terminal, that program becomes a **child** of the terminal session. Close the terminal, and the operating system sends a signal to every child process: "Your parent is gone. Time to die." This is what happened to Ali's agent. He started it in his SSH session. When he closed his laptop, the SSH connection dropped. The server killed every process that belonged to that session. Including his agent. Think of it this way. A **process** is a phone call. When you hang up, the conversation is over. Nobody keeps talking. A **service** is a security guard hired by the building. The building doesn't care if the guard takes a coffee break or swaps shifts. What matters is that someone is always at the desk. If the guard leaves, the building finds a replacement. Ali's agent is a phone call. He needs it to be a security guard. * * * The Fix: One File[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-fix-one-file "Direct link to The Fix: One File") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Linux has a built-in building manager called **systemd**. It manages every service on the server: the database, the web server, the SSH daemon that let Ali connect in the first place. All of them are systemd services. All of them survive reboots. To make your agent a systemd service, you write a **unit file**: a plain text file that answers five questions: 1. What should run? 2. As which user? 3. When should it start? 4. What if it crashes? 5. How much memory can it use? That's it. One file. Let's write it. * * * The Unit File, Line by Line[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-unit-file-line-by-line "Direct link to The Unit File, Line by Line") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **What you tell Claude Code**: "Create a systemd service file for my competitor-tracker agent. It should run as user ali, start after the network is available, restart if it crashes, and use no more than 512 MB of memory." **What the agent creates** at `/etc/systemd/system/competitor-tracker.service`: [Unit]Description=Competitor Tracker AgentAfter=network.target * `Description`. A human-readable name. Shows up in logs and status commands. * `After=network.target`: Don't start this service until the network is ready. Your agent needs internet access to scrape pricing data. Starting before the network is up would cause immediate failures. [Service]Type=simpleUser=aliWorkingDirectory=/opt/agents/competitor-trackerEnvironmentFile=/opt/agents/competitor-tracker/.envExecStart=/usr/bin/python3 /opt/agents/competitor-tracker/src/main.pyRestart=on-failureRestartSec=5MemoryMax=512M * `Type=simple`. The process you start IS the service. No forking, no backgrounding. The simplest and most common type. * `User=ali`: Run as Ali, not as root. Never run agents as root. (More on this in Lesson 5.) * `WorkingDirectory`. The agent runs as if you `cd`'d into this directory first. Relative paths in your code resolve from here. * `EnvironmentFile`: Load environment variables from `.env`. Your database password, API keys, and configuration: all available to the agent without hardcoding. * `ExecStart`. The exact command to run. Full absolute path to Python and the script. No ambiguity. * `Restart=on-failure`. If the agent crashes (exits with a non-zero code), restart it. If you intentionally stop it with `systemctl stop`, don't restart. * `RestartSec=5` (Wait 5 seconds before restarting. This prevents crash loops) if the agent has a bug that makes it crash on startup, it won't restart thousands of times per minute and flood your logs. * `MemoryMax=512M`. If the agent uses more than 512 MB of RAM, kill it. This prevents a memory leak from eating all server resources and crashing everything else. [Install]WantedBy=multi-user.target * `WantedBy=multi-user.target`: Start this service when the server boots into its normal operating mode. This is what makes your agent survive reboots. Pause. Read the unit file again. Every line answers one of the five questions. There is no magic here. It's a job description for your agent, written in a format that Linux understands. * * * Bringing the Service to Life[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#bringing-the-service-to-life "Direct link to Bringing the Service to Life") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The file exists, but systemd doesn't know about it yet. Three commands make it real. **What you tell Claude Code**: "Register the competitor-tracker service, set it to start on boot, and start it now." **What the agent does**: sudo systemctl daemon-reloadsudo systemctl enable competitor-trackersudo systemctl start competitor-tracker * `daemon-reload`: "Hey systemd, I added a new service file. Re-read all your files." * `enable`: "Start this service automatically on every boot." * `start`: "Start it right now." **What you tell Claude Code**: "Check if the competitor-tracker service is running." **What the agent does**: systemctl status competitor-tracker **What you see**: ● competitor-tracker.service - Competitor Tracker Agent Loaded: loaded (/etc/systemd/system/competitor-tracker.service; enabled) Active: active (running) since Sun 2026-02-28 02:15:33 UTC; 5s ago Main PID: 4821 (python3) Memory: 47.2M (max: 512.0M) The green dot. `active (running)`. The agent is alive. * * * The Moment of Truth[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-moment-of-truth "Direct link to The Moment of Truth") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now for the test that matters. Close your terminal. Disconnect from the server. Wait ten seconds. SSH back in. **What you tell Claude Code**: "Is the competitor-tracker service still running?" **What the agent does**: systemctl status competitor-tracker **What you see**: ● competitor-tracker.service - Competitor Tracker Agent Active: active (running) since Sun 2026-02-28 02:15:33 UTC; 2min ago The agent is alive. You closed the terminal. You disconnected. The agent kept running. The building manager did its job. This is the moment. A process would have died. A service survived. * * * What Happens When It Crashes[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#what-happens-when-it-crashes "Direct link to What Happens When It Crashes") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali's agent has a bug. Once every few days, it encounters a malformed API response and crashes. Before systemd, this meant Ali woke up to a dead agent and missing data. Now, watch what happens. **What you tell Claude Code**: "Show me the journal log for competitor-tracker from the last hour." **What the agent does**: journalctl -u competitor-tracker --since "1 hour ago" **What you see**: Feb 28 03:42:17 server competitor-tracker[4821]: ERROR: Malformed API responseFeb 28 03:42:17 server systemd[1]: competitor-tracker.service: Main process exited, code=exited, status=1/FAILUREFeb 28 03:42:22 server systemd[1]: competitor-tracker.service: Scheduled restart job, restart counter is at 1.Feb 28 03:42:22 server competitor-tracker[4897]: Starting competitor tracker agent...Feb 28 03:42:23 server competitor-tracker[4897]: Agent running successfully The agent crashed at 03:42:17. Systemd waited 5 seconds (`RestartSec=5`). At 03:42:22, it started a new instance. By 03:42:23, the agent was running again. Ali slept through the entire event. * * * A Note About tmux[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#a-note-about-tmux "Direct link to A Note About tmux") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You may have heard of **tmux**: a tool that keeps terminal sessions alive after you disconnect. For interactive sessions you want to keep alive: like a monitoring dashboard or a long-running data migration you're watching: tmux is the right tool. For agents that run 24/7, systemd is the right tool. tmux keeps a session alive. systemd keeps a service alive, restarts it after crashes, starts it on boot, and enforces resource limits. Your agents need systemd. * * * The Five systemctl Commands[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-five-systemctl-commands "Direct link to The Five systemctl Commands") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You'll direct Claude Code to use these. You don't need to memorize them, but recognizing them helps you understand the output. | Command | What it does | | --- | --- | | `systemctl start ` | Start the service now | | `systemctl stop ` | Stop the service now | | `systemctl restart ` | Stop then start | | `systemctl status ` | Show current state, PID, memory | | `systemctl enable ` | Start automatically on boot | And one for viewing logs: | Command | What it does | | --- | --- | | `journalctl -u ` | Show all logs for this service | | `journalctl -u -f` | Follow logs in real time (like a live feed) | * * * Ali's agent is unkillable. It survives terminal closures. It survives reboots. It restarts after crashes. It runs under memory limits that protect the rest of the server. He feels accomplished. Then Dev checks the server and goes pale. "It's running as root. With password SSH. Anyone on the internet could..." He doesn't finish the sentence. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Restart Policies[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#prompt-1-restart-policies "Direct link to Prompt 1: Restart Policies") My agent's systemd service uses Restart=on-failure. Explain thedifference between Restart=always and Restart=on-failure. Whenwould I want each one? What happens if my agent has a bug thatmakes it crash immediately on startup and I have Restart=always? **What you're practicing:** Understanding restart behavior. The difference between these two values determines whether your agent recovers gracefully or enters an infinite crash loop. ### Prompt 2: Adapt for a Different Stack[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#prompt-2-adapt-for-a-different-stack "Direct link to Prompt 2: Adapt for a Different Stack") I have a Node.js Express server that I want to run as a systemdservice. Take the competitor-tracker unit file from this lessonand modify it for Node.js. What lines change? What stays thesame? Explain each change. **What you're practicing:** Transferring the systemd pattern to different technologies. The unit file structure is universal, only the ExecStart line changes significantly. ### Prompt 3: History and Context[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#prompt-3-history-and-context "Direct link to Prompt 3: History and Context") Before systemd, Linux used "init scripts" to manage services.What problems did init scripts have that systemd solved? Whydoes every major Linux distribution use systemd now? Were therecontroversies about the switch? **What you're practicing:** Understanding why systemd exists, not just how to use it. Knowing the historical context helps you appreciate what the tool does for you automatically. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Why Your Agent Keeps Dying](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#why-your-agent-keeps-dying) * [The Fix: One File](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-fix-one-file) * [The Unit File, Line by Line](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-unit-file-line-by-line) * [Bringing the Service to Life](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#bringing-the-service-to-life) * [The Moment of Truth](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-moment-of-truth) * [What Happens When It Crashes](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#what-happens-when-it-crashes) * [A Note About tmux](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#a-note-about-tmux) * [The Five systemctl Commands](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#the-five-systemctl-commands) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#try-with-ai) * [Prompt 1: Restart Policies](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#prompt-1-restart-policies) * [Prompt 2: Adapt for a Different Stack](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#prompt-2-adapt-for-a-different-stack) * [Prompt 3: History and Context](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#prompt-3-history-and-context) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/making-your-agent-unkillable#flashcards-study-aid) --- # Reading What Your Agent Does | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In the last lesson, Ali explored Dev's server. He knows where things live -- `/home` for users, `/var` for logs, `/etc` for configuration. But knowing where things are is different from understanding what they say. When Claude Code runs a command and the output reads `drwxr-xr-x 3 ali ali 4096 Feb 14 09:22 competitor-tracker`, Ali nods and pretends he understands. He sees the words. He sees the numbers. He has no idea what most of them mean. > **"You don't need to speak Linux. You need to read it."** This lesson makes the black box transparent. By the end, you'll read command output the way you read a dashboard -- quickly, confidently, knowing what matters and what to skip. Commands Are Sentences[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#commands-are-sentences "Direct link to Commands Are Sentences") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every Linux command follows the same structure. Once you see it, you can't unsee it. Think of a command as a sentence. It has a **verb** (what to do), **adverbs** (how to do it), and an **object** (what to do it to). ls -la /var/log| | |verb adverbs object * **`ls`** is the verb: "list" * **`-la`** is two adverbs: `-l` means "long format" (show details), `-a` means "all" (include hidden files) * **`/var/log`** is the object: the directory to list Read it as a sentence: "**List**, in **long format**, **all** files in **/var/log**." That's it. Every command you'll ever see follows this pattern. The verb changes. The adverbs change. The object changes. The grammar never does. Pause. Go back and read that command one more time. Verb. Adverbs. Object. If you can parse one command, you can parse all of them. The rest of this lesson is vocabulary, not new grammar. Your Agent's Vocabulary: 15 Commands[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#your-agents-vocabulary-15-commands "Direct link to Your Agent's Vocabulary: 15 Commands") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the part where most tutorials hand you a list of 50 commands and tell you to memorize them. We're not doing that. Think of these 15 commands as vocabulary cards for a language you're visiting, not moving to. You don't memorize vocabulary cards. You recognize words when you see them. When Claude Code runs `grep` in front of you, you'll think "ah, searching for text" instead of "what is that?" ### Navigation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#navigation "Direct link to Navigation") | Command | What It Does | Read It As | | --- | --- | --- | | `ls` | Lists files in a directory | "Show me what's here" | | `cd` | Changes your current directory | "Go to this folder" | | `pwd` | Prints your current location | "Where am I right now?" | ### Inspection[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#inspection "Direct link to Inspection") | Command | What It Does | Read It As | | --- | --- | --- | | `cat` | Shows the entire contents of a file | "Read this file to me" | | `head` | Shows the first few lines of a file | "Show me the beginning" | | `tail` | Shows the last few lines of a file | "Show me the end" | | `less` | Opens a file for scrolling through | "Let me browse this file" | ### Measurement[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#measurement "Direct link to Measurement") | Command | What It Does | Read It As | | --- | --- | --- | | `du` | Measures disk usage of files/folders | "How big is this?" | | `df` | Shows free disk space on the server | "How much room is left?" | | `wc` | Counts lines, words, or characters | "How many lines in this file?" | ### Search[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#search "Direct link to Search") | Command | What It Does | Read It As | | --- | --- | --- | | `find` | Locates files by name, type, or date | "Find files matching this pattern" | | `grep` | Searches inside files for specific text | "Find this word inside these files" | ### Process[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#process "Direct link to Process") | Command | What It Does | Read It As | | --- | --- | --- | | `ps` | Lists running processes | "What's running right now?" | | `top` | Shows live resource usage | "What's using CPU and memory?" | | `kill` | Stops a running process | "Shut this program down" | Fifteen commands. That's the entire working vocabulary for server operations. When Claude Code runs any of these, you'll know what it's doing before it explains the result. Reading Real Output[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#reading-real-output "Direct link to Reading Real Output") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Vocabulary is recognition. Now let's practice reading. Tell Claude Code: Show me everything running on this server right now with full details. Claude Code will likely run something like `ps aux`. Here's what that output looks like: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.1 169360 11840 ? Ss Feb12 0:08 /sbin/initali 1247 0.3 2.1 458920 42880 ? Sl 09:22 1:45 python3 competitor_tracker.pypostgres 892 0.1 1.5 215460 30720 ? Ss Feb12 5:23 /usr/lib/postgresql/15/bin/postgresroot 456 0.0 0.0 72312 5888 ? Ss Feb12 0:01 /usr/sbin/sshd **What you see:** A wall of text with columns that don't line up neatly. **What this means:** | Column | What It Tells You | | --- | --- | | `USER` | Who owns this process. Ali's agent runs as `ali`. The database runs as `postgres`. | | `PID` | Process ID -- a unique number for each running program. You need this to stop a specific process. | | `%CPU` | How much processor power this process uses. Ali's agent at 0.3% is healthy. 95% would be a problem. | | `%MEM` | How much memory this process uses. The agent at 2.1% is fine. The database at 1.5% is normal for PostgreSQL. | | `COMMAND` | The actual program running. This is the most useful column -- it tells you what each process IS. | Ali looks at this and sees his agent (`competitor_tracker.py`), the database it writes to (`postgres`), and the SSH service keeping him connected (`sshd`). Three processes that tell the whole story of his deployment. Now ask Claude Code: How much disk space is left on this server? Claude Code runs `df -h`. The `-h` flag means "human-readable" -- show sizes in GB instead of raw bytes. Filesystem Size Used Avail Use% Mounted on/dev/sda1 50G 23G 25G 48% /tmpfs 2.0G 12M 2.0G 1% /dev/shm **What you see:** Two rows of numbers. **What this means:** The server has a 50 GB disk. 23 GB is used. 25 GB is free. At 48% capacity, Ali has room. If this number were 90%, he'd need to investigate what's eating space -- and he already knows the command for that (`du`). File Permissions: Who Can Do What[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#file-permissions-who-can-do-what "Direct link to File Permissions: Who Can Do What") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the one that makes people's eyes glaze over. It shouldn't. Permissions are a building with three levels of keycards. Tell Claude Code: Show me the detailed file listing for Ali's home directory. Claude Code runs `ls -la /home/ali` and you see: drwxr-xr-x 5 ali ali 4096 Feb 14 09:22 .drwxr-xr-x 3 root root 4096 Feb 12 14:30 ..-rw------- 1 ali ali 220 Feb 12 14:30 .bash_history-rw-r--r-- 1 ali ali 3526 Feb 12 14:30 .bashrcdrwxr-xr-x 2 ali ali 4096 Feb 14 09:22 competitor-tracker-rw-rw-r-- 1 ali ali 892 Feb 14 09:25 .env Focus on the first column. Take `drwxr-xr-x`: d rwx r-x r-x| | | || | | +-- Everyone else: read + execute| | +--------- Group: read + execute| +---------------- Owner: read + write + execute+-- Type: d = directory The first character is the file type. `d` means directory. `-` means regular file. Then come three groups of three characters. Each group represents a level of keycard: * **Owner** (positions 2-4): The person who owns the file. Full keycard. * **Group** (positions 5-7): People in the same team. Limited keycard. * **Everyone** (positions 8-10): Any user on the server. Visitor badge. Each position is either a letter (permission granted) or a dash (permission denied): | Letter | Permission | For Files | For Directories | | --- | --- | --- | --- | | `r` | Read | View the contents | List what's inside | | `w` | Write | Modify the contents | Add or delete files inside | | `x` | Execute | Run as a program | Enter the directory | | `-` | Denied | Cannot do this | Cannot do this | Now decode Ali's files: * **`drwxr-xr-x competitor-tracker`** -- Directory. Ali can do everything. Group and everyone can enter and look, but not modify. * **`-rw------- .bash_history`** -- File. Only Ali can read and write it. Nobody else can see his command history. * **`-rw-rw-r-- .env`** -- File. Ali can read and write. His group can read and write. Everyone can read (but not write). That `.env` file permission is a problem. The `.env` file contains API keys and database passwords. "Everyone can read" means any user on the server could see Ali's secrets. In Lesson 5, Ali will fix this. For now, he can read the warning sign. Putting It Together[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#putting-it-together "Direct link to Putting It Together") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Ali directs Claude Code to run a health check: You: Check if my competitor-tracker agent is running and healthy. Show me the process, resources, and whether the log file is growing. Claude Code runs `ps aux | grep competitor_tracker`, `tail -5 /var/log/competitor-tracker/agent.log`, and `du -sh /var/log/competitor-tracker/`. Three commands. Thirty seconds. Ali sees his agent is running (PID 1247, 0.3% CPU, 2.1% memory), the log shows fresh entries with no errors, and the log folder is a manageable 12 MB. He didn't type any of those commands. He directed Claude Code with a single sentence, then read the results. That's the skill. Not typing commands. Reading their output. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Read Process Output[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#prompt-1-read-process-output "Direct link to Prompt 1: Read Process Output") Show me all running processes on this machine with full details.Pick any three lines from the output and explain each column to me.What process is using the most memory? **What you're learning:** How to read `ps aux` output. The column headers (USER, PID, %CPU, %MEM, COMMAND) are the same on every Linux server. Once you can read one, you can read them all. ### Prompt 2: Explain Permissions on Real Files[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#prompt-2-explain-permissions-on-real-files "Direct link to Prompt 2: Explain Permissions on Real Files") Show me the permissions on /etc/passwd and /etc/shadow.Why are they different? What would happen if shadow hadthe same permissions as passwd? **What you're learning:** Why permissions matter for security. `/etc/passwd` is world-readable (every program needs it to map usernames to IDs). `/etc/shadow` stores password hashes and is locked down. This is the most concrete example of "permissions protect secrets." ### Prompt 3: du vs df[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#prompt-3-du-vs-df "Direct link to Prompt 3: du vs df") Run du -sh /var/log and df -h for me. What's the differencebetween these two commands? When would I use one vs the other? **What you're learning:** The difference between measuring a specific folder (`du` -- "how big is this thing?") and measuring the entire disk (`df` -- "how much room is left?"). Both answer questions about space, but from different angles. `du` is a magnifying glass. `df` is a fuel gauge. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can read the server now. Process lists, disk usage, file permissions -- none of it is a black box anymore. But Ali's agent files are scattered. The script lives in one place, the config in another, and the logs are vanishing into a default location nobody checks. Time to build a proper home. Checking access... * [Commands Are Sentences](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#commands-are-sentences) * [Your Agent's Vocabulary: 15 Commands](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#your-agents-vocabulary-15-commands) * [Navigation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#navigation) * [Inspection](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#inspection) * [Measurement](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#measurement) * [Search](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#search) * [Process](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#process) * [Reading Real Output](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#reading-real-output) * [File Permissions: Who Can Do What](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#file-permissions-who-can-do-what) * [Putting It Together](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#putting-it-together) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#try-with-ai) * [Prompt 1: Read Process Output](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#prompt-1-read-process-output) * [Prompt 2: Explain Permissions on Real Files](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#prompt-2-explain-permissions-on-real-files) * [Prompt 3: du vs df](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#prompt-3-du-vs-df) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/reading-what-your-agent-does#flashcards-study-aid) --- # Locking the Door | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Ali's agent is unkillable. It survives reboots, restarts after crashes, runs under memory limits. He feels accomplished. Then Dev checks the server status and goes pale. "Your agent is running as root," Dev says. "And you still have password SSH enabled." Ali shrugs. "So?" Dev pulls up the auth log. * * * 47,000 Reasons to Care[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#47000-reasons-to-care "Direct link to 47,000 Reasons to Care") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Marcus ran his agent as root for three weeks. He never checked the server's authentication log. When a friend showed him how, they found this: **What you tell Claude Code**: "Show me how many failed login attempts are in the auth log." **What the agent does**: grep "Failed password" /var/log/auth.log | wc -l **What you see**: 47,291 Forty-seven thousand attempts. From IP addresses in six countries. Automated bots scanning the internet for servers with weak passwords, trying common combinations: admin/admin, root/password, user/123456. None had succeeded. Yet. Marcus's agent had root access. If even one bot had guessed the password, the attacker would have had complete control over the server. Not just the agent: everything. The database. The other services. The ability to install malware, mine cryptocurrency, or launch attacks on other servers. > **"Security isn't a feature. It's a constraint that makes everything else possible."** Ali's server has the same problem. Let's fix it. * * * Concept 1: Users Are Badges[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-1-users-are-badges "Direct link to Concept 1: Users Are Badges") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Think of a server like a building with restricted areas. Every person who enters gets a badge. The badge determines which doors they can open. **Root** is the master key. It opens every door in the building. The server room, the executive offices, the supply closet, the safe. If someone steals the master key, they own the building. Your agent doesn't need a master key. It needs a badge that opens one office; it's own directory, and nothing else. **What you tell Claude Code**: "Create a dedicated user called agentuser for running my competitor-tracker agent. It shouldn't have a password or a home directory; it's only for running the service." **What the agent does**: sudo useradd --system --no-create-home --shell /usr/sbin/nologin agentuser **What this means**: `--system` creates a service account (not a human user). `--no-create-home` skips creating a home directory (the agent already has `/opt/agents/competitor-tracker`). `--shell /usr/sbin/nologin` means nobody can SSH in as this user. It exists only to own and run the agent. * * * Concept 2: Permissions Are Keycards[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-2-permissions-are-keycards "Direct link to Concept 2: Permissions Are Keycards") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lesson 2, you learned to read `drwxr-xr-x`. Now that reading pays off. Every file on Linux has three levels of access for three groups of people: | Group | Who | Analogy | | --- | --- | --- | | **Owner** | The user who owns the file | Master keycard holder | | **Group** | Users in the same group | Department keycard holders | | **Others** | Everyone else on the system | Visitor badge holders | Each group can have three types of access: | Permission | Letter | Meaning | | --- | --- | --- | | Read | `r` | Can see the contents | | Write | `w` | Can change the contents | | Execute | `x` | Can run it as a program | So `rwxr-xr-x` means: owner can do everything, group can read and execute, others can read and execute. And `-rw-r--r--` means: owner can read and write, everyone else can only read. Now look at Ali's `.env` file: the one with his database password and API keys: **What you tell Claude Code**: "Check the permissions on my .env file." **What the agent does**: ls -la /opt/agents/competitor-tracker/.env **What you see**: -rw-r--r-- 1 ali ali 256 Feb 28 02:10 .env Pause. Read that permission string. `-rw-r--r--`. Owner can read and write. Group can read. **Others can read.** Every user on this server can read Ali's database password. **What you tell Claude Code**: "Transfer ownership of the entire competitor-tracker directory to agentuser. Then restrict the .env file so only the owner can read it." **What the agent does**: sudo chown -R agentuser:agentuser /opt/agents/competitor-trackersudo chmod 750 /opt/agents/competitor-trackersudo chmod 600 /opt/agents/competitor-tracker/.env **What this means**: * `chown -R agentuser:agentuser`: Change ownership of the directory and everything inside it to agentuser. The `-R` means recursive: every file and subdirectory. * `chmod 750` on the directory: Owner (agentuser) can do everything. Group members can read and enter the directory. Others get nothing. * `chmod 600` on `.env`: Owner can read and write. Nobody else can do anything. The database password is locked down. Verify: ls -la /opt/agents/competitor-tracker/.env -rw------- 1 agentuser agentuser 256 Feb 28 02:10 .env `-rw-------`. Only the owner can read it. The door is locked. * * * Concept 3: Keys Beat Passwords[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-3-keys-beat-passwords "Direct link to Concept 3: Keys Beat Passwords") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali connects to Dev's server with a password. Every time he SSHes in, he types it. The problem: passwords can be guessed. That's what those 47,000 bots were doing: guessing passwords. SSH keys work differently. Instead of a password you remember, you generate a **key pair**: a private key (stays on your laptop, never shared) and a public key (goes on the server). Think of it this way. A password is a house key: anyone who sees it can make a copy. An SSH key is a fingerprint scanner, even if someone sees you use it, they can't replicate your fingerprint. **What you tell Claude Code**: "Set up SSH key authentication. Generate a key pair on my local machine, copy the public key to the server, and then disable password authentication." **What the agent does on your local machine**: ssh-keygen -t ed25519 -C "ali@laptop" This creates two files: * `~/.ssh/id_ed25519`. Your private key. Never share this. Never move this. * `~/.ssh/id_ed25519.pub`. Your public key. Safe to share. Put it on every server you need access to. **What the agent does to copy the key**: ssh-copy-id ali@devs-server.com Now your public key lives on the server. The next time you SSH in, the server checks your fingerprint instead of asking for a password. **What the agent does to disable passwords**: sudo sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_configsudo sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_configsudo systemctl restart sshd caution Before disabling password authentication, make SURE your SSH key works. Test by opening a new terminal and connecting with the key. If the key doesn't work and you disable passwords, you'll lock yourself out of the server. * * * Concept 4: Least Privilege[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-4-least-privilege "Direct link to Concept 4: Least Privilege") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every decision in this lesson follows one principle: **give the minimum access needed and nothing more**. | Instead of... | Do this | Why | | --- | --- | --- | | Running agent as root | Create agentuser | If compromised, attacker gets one directory, not the whole server | | Password SSH | SSH keys only | Eliminates brute-force attacks entirely | | `.env` readable by everyone | `chmod 600` | Secrets are visible only to the service that needs them | | Agent can write anywhere | `chmod 750` on its directory | Agent can't modify system files even if exploited | This is the principle of **least privilege**: give the delivery driver a key to the mailbox, not the front door. * * * Update the Service[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#update-the-service "Direct link to Update the Service") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- One more step. The systemd unit file from Lesson 4 says `User=ali`. It should say `User=agentuser`. **What you tell Claude Code**: "Update the competitor-tracker systemd service to run as agentuser instead of ali, then restart it." **What the agent does**: sudo sed -i 's/User=ali/User=agentuser/' /etc/systemd/system/competitor-tracker.servicesudo systemctl daemon-reloadsudo systemctl restart competitor-trackersystemctl status competitor-tracker The agent now runs as a dedicated, limited user with access only to its own directory and secrets locked down. * * * Sunday night. Agent deployed. Unkillable. Locked down. Ali checks the dashboard: the latest report is empty. The agent ran on schedule but produced nothing. Something broke. Board meeting at 9 AM. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Audit Your .env[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#prompt-1-audit-your-env "Direct link to Prompt 1: Audit Your .env") Check the permissions on my .env file. Who can read it right now?If the permissions show -rw-r--r--, explain why that's a securityrisk. What should the permissions be instead? Fix them. **What you're practicing:** Reading permissions on real files and identifying security gaps. The most common server vulnerability isn't a sophisticated hack; it's a readable secrets file. ### Prompt 2: Read-Only Access for a Colleague[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#prompt-2-read-only-access-for-a-colleague "Direct link to Prompt 2: Read-Only Access for a Colleague") I have a colleague who needs to view my agent's logs but shouldNOT be able to modify any files or restart the service. Createa new user for them with the minimum permissions needed. Whatgroup membership and file permissions do they need? **What you're practicing:** Applying least privilege to a real scenario. Different people need different levels of access. Designing permissions is thinking about who needs what. ### Prompt 3: Why Keys Beat Passwords[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#prompt-3-why-keys-beat-passwords "Direct link to Prompt 3: Why Keys Beat Passwords") Explain the math behind SSH key security. How many possiblecombinations does a 4096-bit RSA key have compared to an8-character password? Why does this make brute-force attacksagainst SSH keys essentially impossible? **What you're practicing:** Understanding the quantitative security difference. "Keys are better than passwords" is a claim. The math is the proof. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [47,000 Reasons to Care](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#47000-reasons-to-care) * [Concept 1: Users Are Badges](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-1-users-are-badges) * [Concept 2: Permissions Are Keycards](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-2-permissions-are-keycards) * [Concept 3: Keys Beat Passwords](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-3-keys-beat-passwords) * [Concept 4: Least Privilege](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#concept-4-least-privilege) * [Update the Service](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#update-the-service) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#try-with-ai) * [Prompt 1: Audit Your .env](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#prompt-1-audit-your-env) * [Prompt 2: Read-Only Access for a Colleague](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#prompt-2-read-only-access-for-a-colleague) * [Prompt 3: Why Keys Beat Passwords](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#prompt-3-why-keys-beat-passwords) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/locking-the-door#flashcards-study-aid) --- # Chapter 64: The Claude API — Agentic Loops, Structured Output & Batch Processing | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Every agent SDK is an abstraction over the Claude API's messages endpoint. Understanding the raw API — `stop_reason`, `tool_use`, `tool_choice`, JSON schemas, and the Batch API — gives you the ability to diagnose any agent behavior, design custom orchestration that SDKs don't support, and make cost/latency tradeoffs that production systems require. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#what-youll-learn "Direct link to What You'll Learn") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * Construct raw Messages API requests and parse responses (content blocks, stop\_reason values) * Define tools with effective descriptions and handle the tool\_use/tool\_result conversation loop * Build a complete agentic loop that runs autonomously until `stop_reason: "end_turn"` * Control tool selection with `tool_choice` (auto, any, forced) for multi-step pipelines * Enforce structured output via `tool_use` with JSON Schemas (nullable fields, enum + "other", format rules) * Implement validation-retry loops with Pydantic for extraction quality * Use the Message Batches API for 50% cost savings on latency-tolerant workloads Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#chapter-structure "Direct link to Chapter Structure") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. **The Messages API — Anatomy of a Request and Response** — model, max\_tokens, system prompt, messages array, content blocks, stop\_reason values 2. **Tool Definitions and Tool Use** — tool schemas, effective descriptions, tool\_use response handling, tool\_result format, tool distribution principles 3. **The Agentic Loop** — the complete autonomous loop pattern, model-driven decision making, anti-patterns the certification exam tests 4. **tool\_choice — Controlling Tool Selection** — auto vs any vs forced, multi-step forced selection patterns 5. **Structured Output via tool\_use with JSON Schemas** — schema design patterns, semantic validation, format normalization 6. **Validation-Retry Loops for Extraction Quality** — retry-with-error-feedback, Pydantic integration, when retries succeed vs fail 7. **The Message Batches API** — 50% cost savings, 24-hour processing window, failure handling, batch submission frequency Running Project[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#running-project "Direct link to Running Project") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Students build a complete structured data extraction pipeline using raw API calls, progressing from simple extraction to validation-retry to batch processing. Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------------- * Chapter 61: Introduction to AI Agents (conceptual foundation) * Part 4: Python proficiency (async/await, type hints) * Anthropic API key with Claude access Certification Exam Coverage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#certification-exam-coverage "Direct link to Certification Exam Coverage") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter covers **Claude Certified Architect — Foundations** exam domains: * **Domain 1** (27%): Task Statement 1.1 — Agentic loop implementation * **Domain 4** (20%): Task Statements 4.3, 4.5 — Structured output, batch processing * Directly covers Sample Questions 1, 2, 10, 11 * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#what-youll-learn) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#chapter-structure) * [Running Project](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#running-project) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#prerequisites) * [Certification Exam Coverage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/claude-api-agentic-loops#certification-exam-coverage) --- # Chapter 78: Knowledge Graphs & GraphRAG | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C **Part 6: AI Native Software Development — Phase 3: Data & Memory** This chapter teaches you how to build knowledge graphs and implement GraphRAG (Graph-based Retrieval-Augmented Generation) for AI agents. You'll learn when graph structures outperform vector similarity, how to model domain knowledge as graphs, and how to combine graph traversal with LLM reasoning. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#what-youll-learn "Direct link to What You'll Learn") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Knowledge Graph Fundamentals**: Nodes, edges, properties, and graph schemas * **Graph Databases**: Neo4j and lightweight alternatives for agent applications * **GraphRAG Architecture**: Combining graph traversal with vector retrieval * **Entity Extraction**: Automatically building graphs from unstructured text * **Multi-hop Reasoning**: Traversing relationships for complex queries * **Hybrid RAG**: When to use graphs vs vectors vs both Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------------------ * **Chapter 73**: Vector Databases & RAG (vector retrieval foundations) * **Chapter 74**: Relational Databases for Agents (data modeling concepts) * **Chapters 62-65**: Agent SDK fundamentals Key Technologies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#key-technologies "Direct link to Key Technologies") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Technology | Purpose | | --- | --- | | **Neo4j** | Production graph database | | **LangChain GraphRAG** | Graph-enhanced retrieval | | **NetworkX** | Lightweight graph operations | | **Entity Extraction** | Building graphs from text | Running Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#running-example "Direct link to Running Example") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Extend the Task API with knowledge graph capabilities: * Task dependencies as graph relationships * Project hierarchies and team structures * Multi-hop queries: "Show all tasks blocking the Q1 release" * * * > **Note**: This chapter is under development. Lessons will cover graph fundamentals through production GraphRAG implementation. * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#what-youll-learn) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#prerequisites) * [Key Technologies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#key-technologies) * [Running Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/knowledge-graphs-graphrag#running-example) --- # Chapter 69: Multi-Agent Reliability — Errors, Escalation, Provenance & Quality | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C A multi-agent system is only as reliable as its weakest error path. This chapter teaches the five disciplines of multi-agent reliability: structured error propagation, escalation calibration, context management at scale, information provenance, and human review integration. These are the patterns that separate prototype-grade systems from production-grade systems. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#what-youll-learn "Direct link to What You'll Learn") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * Design structured error responses with the MCP `isError` pattern and error category taxonomy * Implement subagent error recovery with partial results propagation and coordinator decision-making * Calibrate escalation triggers using explicit criteria and few-shot examples (not sentiment or confidence scores) * Preserve critical information across long sessions using the "case facts" pattern * Build claim-source mapping pipelines that maintain provenance through multi-agent synthesis * Design human review workflows with field-level confidence calibration and stratified sampling * Orchestrate coordinator-subagent architectures with parallel execution and iterative refinement Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#chapter-structure "Direct link to Chapter Structure") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. **Structured Error Propagation — The MCP isError Pattern** — error categories (transient, validation, business, permission), structured metadata, access-failure vs valid-empty-result 2. **Subagent Error Recovery and Coordinator Decision-Making** — local recovery first, partial results, coverage annotations in synthesis 3. **Escalation Calibration — When to Escalate vs Resolve** — what works (explicit criteria + few-shot), what doesn't (confidence scores, sentiment, "be conservative") 4. **Context Management at Scale** — progressive summarization risks, the "case facts" pattern, "lost in the middle" effect, scratchpad files 5. **Information Provenance in Multi-Source Synthesis** — claim-source mappings, conflicting sources, temporal data, scoped verification tools 6. **Human Review Workflows & Confidence Calibration** — aggregate accuracy trap, field-level scores, threshold calibration with labeled sets, stratified sampling 7. **Coordinator-Subagent Orchestration Patterns** — hub-and-spoke, context isolation, Task tool, parallel execution, iterative refinement (Capstone) Running Project[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#running-project "Direct link to Running Project") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Students build a multi-agent research system (certification exam Scenario 3) with a coordinator agent, web search subagent, document analysis subagent, and synthesis subagent — progressively adding reliability engineering to each layer. Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------------------------------------------- * Chapter 65: Anthropic Claude Agent SDK * Chapter 64: The Claude API — Agentic Loops * Chapters 66-67: MCP Fundamentals and Custom MCP Servers Certification Exam Coverage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#certification-exam-coverage "Direct link to Certification Exam Coverage") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter covers **Claude Certified Architect — Foundations** exam domains: * **Domain 2** (18%): Task Statement 2.2 — Structured error responses * **Domain 5** (15%): Task Statements 5.1-5.6 — Context management, escalation, error propagation, provenance, human review, confidence calibration * Directly covers Sample Questions 3, 7, 8, 9 * Covers exam Scenario 3 (Multi-Agent Research System) end-to-end * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#what-youll-learn) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#chapter-structure) * [Running Project](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#running-project) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#prerequisites) * [Certification Exam Coverage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/multi-agent-reliability#certification-exam-coverage) --- # Chapter 21: Structured Data & Persistent Storage Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Twenty-two questions. No tricks. If you've done the work in this chapter, you'll know these answers. If some questions stump you, that's not failure -- it's feedback. Check the source reference, revisit that lesson, and come back. Scoring guidance: * `20-22`: strong operational readiness for chapter scope * `16-19`: good foundation, revisit weak outcome areas * `<=15`: repeat core exercises before moving forward Best practice: answer once without notes, then review misses and retake after targeted repair. What This Quiz Measures[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#what-this-quiz-measures "Direct link to What This Quiz Measures") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * modeling and constraints * CRUD and session safety * relationship and query reasoning * transaction integrity * Neon operations and security * File Processing -> Computation & Data Extraction -> Structured Data tool escalation judgment Question-to-Outcome Map[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#question-to-outcome-map "Direct link to Question-to-Outcome Map") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Outcome IDs: * `O1` Modeling and constraints * `O2` CRUD and session correctness * `O3` Relationships and query reasoning * `O4` Transaction safety * `O5` Neon operations and security * `O6` Tool choice and verification judgment | Question | Primary outcome | | --- | --- | | Q1 | O6 | | Q2 | O6 | | Q3 | O6 | | Q4 | O6 | | Q5 | O1 | | Q6 | O2 | | Q7 | O2 | | Q8 | O3 | | Q9 | O3 | | Q10 | O4 | | Q11 | O4 | | Q12 | O5 | | Q13 | O5 | | Q14 | O6 | | Q15 | O6 | | Q16 | O6 | | Q17 | O1 | | Q18 | O3 | | Q19 | O4 | | Q20 | O5 | | Q21 | O6 | | Q22 | O6 | Checking access... After You Finish[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#after-you-finish "Direct link to After You Finish") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Missed a few? Good -- that means you know exactly what to revisit. Check the outcome IDs below to pinpoint which lesson to review. * **O1 misses** (modeling and constraints): revisit Lessons 2 and 4. * **O2 misses** (CRUD and session correctness): revisit Lessons 3 and 5. * **O3 misses** (relationships and query reasoning): revisit Lesson 4. * **O4 misses** (transaction safety): revisit Lesson 5. * **O5 misses** (Neon operations and security): revisit Lesson 6. * **O6 misses** (tool choice and verification): revisit Lesson 0, Lesson 7, and capstone policy sections. * [What This Quiz Measures](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#what-this-quiz-measures) * [Question-to-Outcome Map](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#question-to-outcome-map) * [After You Finish](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/chapter-quiz#after-you-finish) --- # Part Five Retrospective: From Consumer to Architect | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C James sat at his desk with an empty notebook page. Emma had suggested this: before moving on, take stock. Not of what you studied, but of what changed. He had been thinking about the warehouse again, the way his old manager used to run end-of-year inventory. Not counting boxes on shelves, but sitting in the break room afterward and asking the team: what did we learn this year that we did not know last January? He opened his TutorClaw project on one screen. Nine tools. Three components. An MCP server his learners connected to with a single install command. On the other screen, his notes from the past five chapters. Hundreds of pages of decisions, diagrams, cost tables, architecture pivots. "I do not even recognize myself from Chapter 56," he said. "I did not know what MCP stood for." Emma pulled up a chair. "That is the point of the inventory. Not what you covered. What shifted." * * * You are doing exactly what James is doing. You built a product across five chapters. Now you sit with the question every builder eventually faces: what did the process of building teach you that the individual lessons did not? Five Roles, One Arc[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#five-roles-one-arc "Direct link to Five Roles, One Arc") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The five chapters of Part 5 each demanded a different role. Not metaphorically. Each chapter asked you to do fundamentally different work, and the kind of thinking each role required was distinct. ### Chapter 56: Consumer[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-56-consumer "Direct link to Chapter 56: Consumer") You installed OpenClaw. You paired it with WhatsApp. You sent your first message and watched the agent loop process it: receive, think, act, respond. You configured SOUL.md and IDENTITY.md, giving your employee a personality. You connected MCP servers someone else built. You installed skills from ClawHub. The Consumer role taught you to use a platform. You learned the boundaries of what it could do, where configuration ended and customization began, and what it felt like when an agent handled a task you used to do manually. You experienced agent identity: the moment your AI Employee stopped feeling like a chatbot and started feeling like a colleague with preferences and capabilities. The Consumer role is where most people stop. They install a tool, use it for their needs, and move on. You did not stop. ### Chapter 57: Builder[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-57-builder "Direct link to Chapter 57: Builder") You opened an empty directory and wrote a Python program. Twenty lines, then sixty, then a server with multiple tools, state management, external data access, error handling, authentication, and remote deployment. You built MCP servers from nothing. The Builder role taught you that the tools you installed as a Consumer were programs someone wrote. A tool declaration is a function with a decorator. A schema is a Pydantic model. Transport is a protocol choice between stdio and streamable-http. The magic dissolved into engineering, and engineering is something you can do. The distance between Consumer and Builder is the distance between driving a car and understanding how the engine works. Both are useful. But the Builder can fix the engine when it breaks, and the Builder can build a new one. ### Chapter 58: Shipper[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-58-shipper "Direct link to Chapter 58: Shipper") You took the MCP server skills from Chapter 57 and built a real product. Not a tutorial project. A production application with nine tools, a Cloudflare R2 content layer with Worker-based access control, a shim skill for offline fallback, Stripe payment integration, Docker deployment, context-engineered tool descriptions, a dedicated agent identity, and ClawHub publishing. The Shipper role taught you that building is not shipping. A working MCP server is not a product. A product has tiered access, payment processing, content delivery, error handling, rate limiting, security hardening, and a registration flow. The gap between "it works on my machine" and "learners pay to use it" is where the Shipper lives. Shipping forced decisions that building never required. Which tools are free? Which are gated? What happens when the MCP server is unreachable? How do learners discover the product? Each decision made the product more real and more complete. ### Chapter 59: Economist[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-59-economist "Direct link to Chapter 59: Economist") You analyzed the numbers behind the product you shipped. You traced every dollar: the fifty to seventy dollars per month in infrastructure, the zero dollars in LLM cost, the gross margin approaching 99.5%. You compared four architectures and watched the economics shift from 22% margins to near-total efficiency. You understood the Great Inversion: the operator provides intelligence, the learner provides compute. The Economist role taught you that architecture is not a technical choice. It is an economic one. The MCP-first design did not win because it was technically elegant. It won because the Platform Inversion eliminated the largest cost category entirely. The 90/10 rule showed that optimizing infrastructure (the 10%) matters far less than questioning whether you need to pay for the LLM (the 90%). Before the Economist role, you could build a product. After it, you could explain why the product was viable. ### Chapter 60: Architect[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-60-architect "Direct link to Chapter 60: Architect") You traced six pivots from initial idea to the architecture you built. You discovered that the clean, inevitable-looking design was the sixth attempt. You identified what survived all six pivots (invariant layers: pedagogy, content, pricing) and what was replaced every time (variant layers: infrastructure, delivery, routing). You distilled eight meta-lessons into three themes. You wrote an Architecture Decision Record. You analyzed versioning strategy and marketplace dynamics. The Architect role taught you the WHY behind the WHAT. Every choice in TutorClaw had a history: a previous architecture that failed, a constraint that forced a change, an assumption that was questioned. The Architect does not just build. The Architect understands why the building is shaped the way it is, and can explain that reasoning to someone who was not there when the decisions were made. The Arc Matters More Than the Roles[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#the-arc-matters-more-than-the-roles "Direct link to The Arc Matters More Than the Roles") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The five roles are not a checklist. They are a progression. Each role required the previous one. You could not build MCP servers in Chapter 57 without the platform understanding you gained as a Consumer in Chapter 56. You could not ship TutorClaw in Chapter 58 without the server-building skills from Chapter 57. You could not analyze economics in Chapter 59 without a shipped product to analyze. You could not trace architecture decisions in Chapter 60 without understanding the economics that validated those decisions. Remove any chapter and the arc collapses. A Builder who never consumed the platform does not know what users expect. A Shipper who never analyzed the economics cannot explain why the product is viable. An Architect who never shipped cannot distinguish between theoretical elegance and practical reality. The arc itself is the lesson: depth comes from doing the work in order, each role building on the previous one until the full picture emerges. The Bridge: What You Built On and What You Built With[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#the-bridge-what-you-built-on-and-what-you-built-with "Direct link to The Bridge: What You Built On and What You Built With") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider what you relied on throughout Part 5. OpenClaw provided the agent runtime. It handled the agent loop, tool discovery, message routing, channel integration, and security boundaries. You configured it, extended it, and built applications on it. But you never built it. The MCP protocol provided the communication standard. Your tools declared themselves in a format any agent could discover. You used MCP, but you never implemented the protocol itself. ClawHub provided distribution. You published your product and learners installed it with a single command. You used the marketplace, but you never built the marketplace infrastructure. Every capability you developed in Part 5 was exercised on top of a platform someone else created. The platform gave you leverage: a single developer building a production product on seventy dollars of infrastructure. That leverage is real. But it comes with a question. What if you need to build without the platform? What if your requirements exceed what OpenClaw provides? What if you need an agent loop customized for a domain OpenClaw was not designed for? What if you need multi-agent orchestration that coordinates dozens of specialized agents with shared state and complex handoff patterns? What if you need to build the platform itself? That question is not hypothetical. It is Part 6. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Exercise 1: Write Your Personal Retrospective[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#exercise-1-write-your-personal-retrospective "Direct link to Exercise 1: Write Your Personal Retrospective") Reflect on your own Part 5 journey. Use this prompt to write a retrospective that maps the five roles to your specific experience: I just completed a five-chapter learning arc about buildingAI agent applications on a platform. The five phases were:1. Consumer: installed and configured the platform2. Builder: wrote MCP servers from scratch3. Shipper: built a production application end-to-end4. Economist: analyzed the business model and unit economics5. Architect: traced architecture decisions and meta-lessonsWrite a personal retrospective for me. For each phase, helpme identify: what specific capability did I gain? Whatsurprised me most? What would I do differently if I startedover? Which phase changed my thinking the most, and why? **What you are learning:** A retrospective is not a summary. It is a reflection on what changed in your thinking, not what topics were covered. Practicing this kind of structured reflection builds the meta-cognitive skill of learning from your own process, which transfers to every project you work on. ### Exercise 2: Identify What Transfers[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#exercise-2-identify-what-transfers "Direct link to Exercise 2: Identify What Transfers") Part 6 teaches building agents from scratch with SDKs. Some Part 5 skills transfer directly; others are platform-specific and will need to be replaced. Use this prompt to sort them: I learned these skills in Part 5 of an AI agent developmentcourse:Platform skills: OpenClaw configuration, SKILL.md authoring,ClawHub publishing, shim skill design, WhatsApp channel setupEngineering skills: MCP server development, Cloudflare R2content delivery, Stripe payment integration, Docker deployment,context engineering for tool descriptionsThinking skills: architecture decision analysis, invariant vsvariant layer identification, economic modeling (Great Inversion,90/10 rule), distribution strategy, ADR documentationIn Part 6, I will build agents from scratch using SDKs withouta platform like OpenClaw. For each skill category, identify:which skills transfer directly, which need adaptation, andwhich are platform-specific and will be replaced entirely. **What you are learning:** Not all skills are created equal. Platform skills (OpenClaw configuration, ClawHub commands) are specific to one platform and must be relearned for each new platform. Thinking skills (architecture analysis, economic reasoning) transfer to any platform, any project, any domain. Knowing the difference helps you invest your learning time wisely. ### Exercise 3: Articulate the Bridge[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#exercise-3-articulate-the-bridge "Direct link to Exercise 3: Articulate the Bridge") The transition from Part 5 to Part 6 is the shift from platform user to platform builder. Use this prompt to think through what that shift demands: I built a production AI agent application on a platform(OpenClaw) that provided the agent runtime, tool discovery,channel integration, and security model. My infrastructurecost was $50-70/month and my gross margin was 99.5%.Now I need to build without the platform. I need to implementmy own agent loop, tool system, and orchestration.Help me think through:1. What did the platform provide that I now need to build?2. Which of those components is the hardest to build well?3. What new skills will I need that Part 5 did not require?4. What is the one thing from Part 5 that will help me most in Part 6? **What you are learning:** The bridge from platform user to platform builder is not just a skill upgrade. It is a shift in what you are responsible for. On a platform, you are responsible for your application logic. Without a platform, you are responsible for everything: the agent loop, the tool system, the security model, the orchestration. Understanding what "everything" means before you start building prepares you for the scope of Part 6. * * * James closed his notebook. Five chapters. Five roles. He could trace the arc now: from the moment he sent his first WhatsApp message to an AI Employee, through building servers, shipping a product, analyzing its economics, and understanding the six pivots that shaped its architecture. "It is like end-of-year inventory," he said. "At the warehouse, we would count stock for three days, then sit in the break room and ask: what did we learn? The counting was necessary, but the conversation afterward was where the real value was. The counting told us what we had. The conversation told us what we knew." He looked at his TutorClaw project. "I know why every piece is where it is. The nine tools are invariant. The MCP server, the VPS, the Docker setup, those are variant. If I had to rebuild on a different platform tomorrow, the tools would come with me. The infrastructure would not." Emma nodded. "You just described the entire architecture chapter in two sentences." "But here is the thing." James paused. "I built all of this on OpenClaw. The agent loop, the tool discovery, the message routing, the security model. OpenClaw handled that. I never built any of it." "No. You did not." "So what happens when there is no platform? When I need an agent loop that does something OpenClaw was not designed for? When I need to coordinate multiple agents with custom handoff logic?" Emma was quiet for a moment. "I do not know exactly what that will demand. I have built on platforms for most of my career. Building the platform itself is a different kind of engineering. The architecture thinking, the economic reasoning, the discipline of documenting decisions: those transfer. The rest?" She shook her head slowly. "I am genuinely not sure. I would be lying if I told you I had all the answers for what comes next." James looked at his notebook one more time. He thought about Chapter 56, when he had typed a WhatsApp message to an AI Employee and waited, not quite believing it would answer. That felt like a different person. Not because the technology had changed, but because he had. He understood now what he was looking at when a system responded: the agent loop, the tool registry, the routing, the economics underneath. He could not unsee it. "I built an application on someone else's platform. I want to understand how the platform itself works." Emma nodded. "That is Part 6." Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Five Roles, One Arc](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#five-roles-one-arc) * [Chapter 56: Consumer](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-56-consumer) * [Chapter 57: Builder](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-57-builder) * [Chapter 58: Shipper](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-58-shipper) * [Chapter 59: Economist](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-59-economist) * [Chapter 60: Architect](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#chapter-60-architect) * [The Arc Matters More Than the Roles](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#the-arc-matters-more-than-the-roles) * [The Bridge: What You Built On and What You Built With](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#the-bridge-what-you-built-on-and-what-you-built-with) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#try-with-ai) * [Exercise 1: Write Your Personal Retrospective](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#exercise-1-write-your-personal-retrospective) * [Exercise 2: Identify What Transfers](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#exercise-2-identify-what-transfers) * [Exercise 3: Articulate the Bridge](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#exercise-3-articulate-the-bridge) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/part-five-retrospective#flashcards-study-aid) --- # Prove Professional Value | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Everything you have built so far (identity, skill, connections, scheduling, boundaries, memory) converges here. This final project milestone has two tracks: Silver proves your employee delivers real professional value through autonomous reporting. Gold proves you can architect a multi-context system with proper isolation. Both tracks test the same question: would you trust this employee's output enough to act on it? A report you would not show your boss is not a report worth generating. An architecture without real isolation is not an architecture worth building. Choose your tier, or complete Silver first and continue to Gold. * * * Silver Deliverable: Autonomous Report[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#silver-deliverable-autonomous-report "Direct link to Silver Deliverable: Autonomous Report") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Build a report your employee generates and delivers on schedule. The report must do more than summarize: it must synthesize data from multiple sources and make at least one recommendation you did not ask for. ### Silver Success Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#silver-success-criteria "Direct link to Silver Success Criteria") 1. The report runs on schedule and is delivered through your configured channel 2. It reads from 2 or more data sources (files, APIs, databases, web content: whatever fits your domain) 3. It contains at least one **proactive recommendation**: something the employee spotted that you did not explicitly ask about. Drawing on patterns from your action log or knowledge store strengthens this significantly 4. It meets professional quality standards: you would share this with your boss, client, or colleague without embarrassment ### Silver Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#silver-use-case-gallery "Direct link to Silver Use Case Gallery") **Accountant**: Weekly cash flow report: reads bank transactions and outstanding invoices, cross-references payment timelines, flags potential cash crunch 2 weeks before it hits, recommends which invoices to follow up on first. **Teacher**: Weekly class progress report: reads assignment submission data and recent test scores, identifies students whose performance dropped more than 15% in the last two weeks, recommends targeted intervention strategies per student. **Consultant**: Client health dashboard: reads project milestone status and client communication frequency, flags accounts where milestone delivery is on track but communication has dropped (early warning of silent dissatisfaction), recommends proactive check-in schedule. **Recruiter**: Pipeline report: reads application volumes and interview outcome data, identifies stages where candidates disproportionately drop out, recommends sourcing adjustments or process changes to improve conversion. * * * Gold Deliverable: Multi-Group Architecture[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#gold-deliverable-multi-group-architecture "Direct link to Gold Deliverable: Multi-Group Architecture") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Design a system where different professional contexts operate with different identities, permissions, and data access: all within one NanoClaw installation. ### Gold Success Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#gold-success-criteria "Direct link to Gold Success Criteria") All Silver criteria, plus: 1. Three groups configured with distinct `CLAUDE.md` identities: `main` (admin and control), a professional work group (domain tasks), and a client-facing group (limited scope, restricted data access) 2. Demonstrated isolation: the non-admin group cannot perform admin actions or access admin-only data 3. Data boundaries documented and tested: clear rules for what data flows between groups and what stays isolated 4. A system architecture diagram showing groups, data flows, and permission boundaries 5. Memory isolation verified: each group's action log and knowledge store are independent: the client-facing group cannot query the admin group's action log or knowledge entries ### Gold Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#gold-use-case-gallery "Direct link to Gold Use Case Gallery") **Accountant**: `main` (admin: system config, all data access) + `accounting-ops` (bookkeeping, report generation, full transaction data) + `client-portal` (read-only financial summaries, no raw transaction access, no ability to modify records). `client-portal` has its own knowledge store: corrections about Client A's preferences never leak to Client B's group. **Consultant**: `main` (admin: system config, all project data) + `research` (full data access, deep analysis, internal memos) + `client-comms` (filtered output only, no internal strategy documents, no access to other client data). `client-comms` cannot access the internal group's action log: prevents leaking which other clients are being served. **Recruiter**: `main` (admin: pipeline management, all candidate data) + `sourcing` (job descriptions, public candidate info, outreach templates) + `hiring-manager-view` (anonymized candidate summaries, interview feedback forms, no salary data) **Teacher**: `main` (admin: all student records, system config) + `curriculum` (lesson planning, assessment design, aggregate performance data) + `parent-portal` (individual student progress only, no class-wide data, no internal notes) * * * Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#hints "Direct link to Hints") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Level 1: Think about data sources and audience **For Silver:** What two data sources, when cross-referenced, would give you the most valuable insight in your profession? The power of synthesis is in the combination: bank data alone is accounting, but bank data plus outstanding invoices is cash flow forecasting. **For Gold:** Think about who interacts with your work. You have internal operations, client-facing outputs, and administrative control. Each of those contexts needs different data access and different behavior. Who should never see what? Level 2: Ask your AI to help design the system **For Silver:** Design a weekly [your profession] report that combines [data source A]and [data source B]. The report should:1. Summarize key metrics from each source2. Cross-reference the two sources to find patterns or anomalies3. Generate at least one proactive recommendation4. Format the output for delivery via [your channel]Include the specific data fields to read from each source and thelogic for generating recommendations. Your action log from Give Your Employee a Memory is a natural data source. A daily digest of autonomous actions ("here's everything I did today without asking you") is the simplest Silver report that proves real professional value. Combine it with one external data source for the 2-source requirement. **For Gold:** Design a 3-group NanoClaw architecture for a [your profession]:- Group 1 (main): admin and system control- Group 2 ([work-context]): day-to-day professional tasks- Group 3 ([external-context]): limited external-facing accessFor each group, specify:1. The CLAUDE.md identity and personality2. What data it CAN access3. What data it CANNOT access4. What actions it CAN perform5. What actions are blockedThen describe how to test that isolation actually works. Level 3: Step-by-step implementation guide **Silver implementation:** 1. Create a skill that performs your report logic: (a) read source A, (b) read source B, (c) cross-reference for patterns: query your action\_log for recurring patterns or your knowledge store for accumulated preferences, (d) generate recommendations, (e) format the report, (f) deliver via your channel 2. Configure the scheduler to run this skill on your desired cadence (e.g., every Monday at 8am) 3. Run it manually first and review the output. Iterate on the skill until the report quality meets your standard 4. Enable the schedule and let it run autonomously. Save the output as `domain-report-sample.md` **Gold implementation:** 1. Create three group directories: `groups/main/`, `groups/[work-group]/`, `groups/[external-group]/` 2. Write a distinct `CLAUDE.md` for each group with different identity, permitted actions, and data access rules 3. Test isolation: from the non-admin group, attempt an admin action (like modifying system config). Verify it fails or is refused 4. Test data boundaries: from the external group, attempt to query the admin group's action\_log or knowledge store. Verify the data is not available. This proves your memory isolation works at the group level, not just the CLAUDE.md level 5. Create a system architecture diagram showing the three groups, their data access, and the boundaries between them 6. Document all test results * [Silver Deliverable: Autonomous Report](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#silver-deliverable-autonomous-report) * [Silver Success Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#silver-success-criteria) * [Silver Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#silver-use-case-gallery) * [Gold Deliverable: Multi-Group Architecture](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#gold-deliverable-multi-group-architecture) * [Gold Success Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#gold-success-criteria) * [Gold Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#gold-use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/capstone-prove-professional-value#hints) --- # From Broken Math to Your First Tool | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Bash commands files. `ls`, `find`, `mv`, `cp`: every file operation from the File Processing chapter ran through the same foundation. Now try this: type `echo $((14.50 + 23.75))` in your terminal. Syntax error. The tool that moved a thousand files can't add two prices. This is where Bash hits a wall and Python enters your Unix toolkit. This chapter teaches you to build Python scripts that slot into your terminal exactly where Bash falls short: reading from stdin, writing to stdout, chaining through pipes. The agent writes the code. You make the decisions. By chapter end, you'll have a library of verified commands that process a full year of bank statements into a tax-ready report. (Along the way, you'll discover that a soda called Dr. Pepper keeps getting flagged as a medical expense, and build the tool that catches it.) But first: your terminal can't even add decimals. Watch Bash Fail[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#watch-bash-fail "Direct link to Watch Bash Fail") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Try this in your terminal: echo $((47.50 / 3)) bash: 47.50 / 3: syntax error: invalid arithmetic operator Bash's `$((...))` does **integer-only** arithmetic. Any decimal point is a syntax error. And when it doesn't error, it's worse: `$((10 / 3))` silently returns `3`, not `3.333...`. No warning. Just wrong. Every financial calculation involves decimals. Bash simply cannot do them. ### The Stock Exchange That Lost Half Its Value[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#the-stock-exchange-that-lost-half-its-value "Direct link to The Stock Exchange That Lost Half Its Value") This isn't academic. In 1982, the Vancouver Stock Exchange launched a new index at 1000.000. Their software truncated to three decimal places instead of rounding: the same thing Bash does with `$((10 / 3))`. Every time a stock price changed (roughly 2,800 times per day), a tiny sliver of value vanished. No errors. No warnings. After 22 months, the index read approximately 524: while its mathematically correct value should have been around 1009. They'd silently lost nearly half the index to truncation. When they found the bug and corrected it over a weekend in November 1983, the index jumped from 524 to over 1098 overnight. The market hadn't crashed. The math had just been wrong: silently, invisibly, for almost two years. _Source: Quinn, M.J. (2014). Ethics for the Information Age. Pearson. The Vancouver Stock Exchange truncation incident is also documented in Goldberg, D. (1991). "What Every Computer Scientist Should Know About Floating-Point Arithmetic." ACM Computing Surveys._ That was truncation at three decimal places. Your Bash terminal does worse: `$((10 / 3))` truncates to _zero_ decimal places. The Head Math Trap[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#the-head-math-trap "Direct link to The Head Math Trap") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ "Okay, Bash can't do decimals. I'll just ask my AI assistant to calculate it." Here's the trap: **LLMs don't compute, they predict**. When you ask an AI to add `12.50 + 8.75`, it isn't running arithmetic. It's predicting what text should come next based on patterns. For three numbers, the prediction usually matches reality. For 100 numbers from your expense report, it almost certainly won't. Asking an LLM to sum your monthly expenses is like asking a poet to recite a calculation from memory. They might get lucky, but you wouldn't bet your tax return on it. The rule for this chapter: > **If it's math, it belongs in a script. Don't ask AI to calculate. Ask AI to write code that calculates.** | Approach | Reliability | Why | | --- | --- | --- | | "What's the sum of these amounts?" | Unreliable at scale | LLM predicts, doesn't compute | | "Write Python to sum these numbers" | Reliable | Python executes, doesn't predict | Building Your First Python Unix Command[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#building-your-first-python-unix-command "Direct link to Building Your First Python Unix Command") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You have a file of expenses, one decimal number per line. You need the total. Bash errors out. LLM head-math hallucinates. You need code that executes, and you need it to behave like every other command in your toolkit: read stdin, write stdout, chain with pipes. Challenge: 60 Seconds on the Clock Before asking Claude for help, try to solve this yourself. You need to sum decimal numbers in the terminal. Bash's `$((...))` won't work. Can you find another way? Hint: Try `echo "1.2 + 2.3" | bc` or `awk 'BEGIN{print 1.2 + 2.3}'`. What do you discover? Both work for one-off calculations. But can you make either of them read a file of numbers and sum all of them? That's where the problem gets interesting, and where having an agent build you a proper tool starts to make sense. Open Claude Code and describe your problem: I have a file with decimal numbers, one per line. Build me a scriptthat reads numbers from stdin and prints the total. Watch what Claude Code does: Claude Code: I'll create a Python script that reads numbers fromstandard input and sums them.[Creates: sum.py]#!/usr/bin/env python3# sum.py - Sum numbers from stdinimport systotal = 0.0for line in sys.stdin: line = line.strip() if line: # Skip empty lines total += float(line)print(f"Total: {total:.2f}")Let me test it:[Runs: echo -e "127.89\n45.50\n12.99" | python sum.py]Total: 186.38 In seconds, you have a reusable tool. The agent didn't calculate in its head (which would hallucinate). It wrote code that executes. The script is a contract: given numbers, return their exact sum. ### What the Agent Built[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#what-the-agent-built "Direct link to What the Agent Built") Three lines make sum.py a Unix command: **`for line in sys.stdin`** reads piped input, **`float(line)`** converts text to decimals, and **`print(f"Total: {total:.2f}")`** writes the result to stdout. Your script doesn't know or care where the data came from: a file, another command, or typed input. It just reads lines and sums them. ### How the Pipe Connects Everything[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#how-the-pipe-connects-everything "Direct link to How the Pipe Connects Everything") When you run `cat expenses.txt | python sum.py`, the pipe takes whatever `cat` outputs and feeds it directly into your script's `sys.stdin`. Small tools, chained together: like LEGO bricks that snap together through pipes. A tool that reads stdin and writes stdout is **composable**: it connects to any other tool without modification. Every script in this chapter follows that design. ### And Sometimes a Script Is the Wrong Tool[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#and-sometimes-a-script-is-the-wrong-tool "Direct link to And Sometimes a Script Is the Wrong Tool") Three receipts from lunch? Just ask the AI to add them, even if it's off by a cent, nobody cares. Same for one-off calculations on data you'll never see again. Scripts pay for themselves when data is too large to sanity-check manually, or when being wrong has consequences. The rule: if the calculation is repeated, financial, or high-stakes, build the script and verify it. If it's one-time and low-stakes, prompt directly. Stop. Do This Now. Open Claude Code. Ask it to build sum.py. Run it on three numbers. Don't proceed to Lesson 2 until you see output in your terminal. Use this prompt: I have a file with decimal numbers, one per line. Build me a script that reads numbers from stdin and prints the total. Then test it: echo -e "100.50\n25.75\n14.25" | python3 sum.py Expected output: `Total: 140.5` Designing Unix Commands Through Prompts[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#designing-unix-commands-through-prompts "Direct link to Designing Unix Commands Through Prompts") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The prompt you gave Claude Code was specific in two ways: it named the data format (decimal numbers, one per line) and it specified _stdin_. Those two details are what made the result a composable Unix command instead of a throwaway answer. The same underlying problem (sum decimal numbers) produces three different outcomes depending on how you ask: | What you said | What the agent returned | Pipeable? | Works next month? | | --- | --- | --- | --- | | "What's 127.89 + 45.50 + 12.99?" | The answer: 186.38 | No | Gone after the chat | | "Write Python to sum 127.89, 45.50, 12.99" | A script for those specific numbers | No | Only those three | | "Build a script that reads numbers from stdin and prints the total" | `sum.py`: reads any input, composes with pipes | Yes | Works on any data, forever | The difference between row 1 and row 3 isn't effort: all three prompts take the same time to write. The difference is **what you're asking the agent to build**. Row 1 asks for an answer. Row 3 asks for a tool. Two phrases made row 3 work: * **"reads from stdin"**: signals you want something pipeable, not a hardcoded script * **"prints the total"**: signals stdout, so the output can flow into the next command This is the director's move: you describe the interface (data comes in via stdin, result goes out via stdout), the agent handles implementation. You never had to choose Python. You never had to know what `sys.stdin` is. You specified the outcome; the agent handled the how. Throughout this chapter, every time you build a tool, you're making the same two decisions: what does the data look like coming in, and what does the result look like going out. Get those two things right and the agent builds something composable. Miss either one and you get a one-time answer you'll throw away. What You Actually Did[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#what-you-actually-did "Direct link to What You Actually Did") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Step back and notice the strategy you just used, because you'll use it for the rest of this chapter: 1. **You described the data problem**: "I have numbers, I need a sum", not the implementation. The agent chose Python. You didn't have to. 2. **You specified stdin/stdout**: which told the agent you wanted a composable Unix tool, not a one-time answer. 3. **You got a reusable script**: `sum.py` works on ANY file of numbers, not just the one you tested with. This same approach works for any calculation: averages, maximums, counts, filters. Describe the data problem. Specify stdin. Get a tool. Your script ran. It produced a number: 186.38. Exit code 0: no errors, no red text. Everything looks fine. But remember the Vancouver Stock Exchange. Their software ran without errors too. For 22 months. And the number it produced was half of what it should have been. Nobody noticed because the output _looked reasonable_. How do you know 186.38 is right? Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Understanding the Bash Limitation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#prompt-1-understanding-the-bash-limitation "Direct link to Prompt 1: Understanding the Bash Limitation") I just tried running `echo $((1.2 + 2.3))` in Bash and got a syntaxerror. Can you explain why Bash can't handle decimal numbers inarithmetic? What's happening under the hood that causes this? **What you're learning:** The AI explains technical concepts you encountered through direct experimentation. Notice how it provides context about integer arithmetic and shell design decisions that you wouldn't find by staring at the error message alone. ### Prompt 2: Extend sum.py[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#prompt-2-extend-sumpy "Direct link to Prompt 2: Extend sum.py") I have sum.py that reads numbers from stdin and prints the total.Can you modify it to also print:- The count of numbers- The average- The minimum and maximum valuesKeep the stdin reading pattern so it still works with pipes. **What you're learning:** Iterative development with AI. You have a working tool and you're extending it with clear requirements. Notice how specifying "keep the stdin reading pattern" directs the architecture -- you're making design decisions, the agent handles implementation. ### Prompt 3: Handle Bad Input[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#prompt-3-handle-bad-input "Direct link to Prompt 3: Handle Bad Input") My sum.py crashes when the input has a header line like "Amount"before the numbers. How do I make it skip non-numeric linesgracefully instead of crashing? **What you're learning:** Collaborative debugging. You identify the limitation (crashes on headers), the agent suggests the fix (try/except). This is the refinement loop -- you provide real-world context about your data, the AI provides a robust solution. Checking access... * [Watch Bash Fail](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#watch-bash-fail) * [The Stock Exchange That Lost Half Its Value](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#the-stock-exchange-that-lost-half-its-value) * [The Head Math Trap](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#the-head-math-trap) * [Building Your First Python Unix Command](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#building-your-first-python-unix-command) * [What the Agent Built](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#what-the-agent-built) * [How the Pipe Connects Everything](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#how-the-pipe-connects-everything) * [And Sometimes a Script Is the Wrong Tool](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#and-sometimes-a-script-is-the-wrong-tool) * [Designing Unix Commands Through Prompts](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#designing-unix-commands-through-prompts) * [What You Actually Did](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#what-you-actually-did) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#flashcards-study-aid) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#try-with-ai) * [Prompt 1: Understanding the Bash Limitation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#prompt-1-understanding-the-bash-limitation) * [Prompt 2: Extend sum.py](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#prompt-2-extend-sumpy) * [Prompt 3: Handle Bad Input](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/from-broken-math-to-your-first-tool#prompt-3-handle-bad-input) --- # Capstone: Tax Season Prep | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Six tools, five lessons of verification habits, one question: can you orchestrate them into a workflow that runs every year? You have a library of Unix-styled Python commands in `~/tools`, a verification habit that won't let you submit numbers you haven't proved, and 12 months of bank CSVs sitting in `~/finances/2025/`. Now put them together. No bank CSV yet? Start here. Ask Claude Code to generate realistic test data before doing anything else: Generate a bank statement CSV with 20 transactions.Include: CVS PHARMACY ($45.67), WALGREENS ($23.45), DR MARTINEZ MEDICAL ($150.00),DR PEPPER SNAPPLE ($4.99), UNITED WAY DONATION ($100.00), OFFICE DEPOT ($89.50),CVSMITH CONSULTING ($200.00), and 13 random transactions.Use columns: Date, Description, Amount (negative for debits).Save as ~/finances/test-2025.csv. Calculate expected totals by hand BEFORE running anything: * Medical (CVS + WALGREENS + DR MARTINEZ): $219.12 * Charitable (UNITED WAY): $100.00 * Business (OFFICE DEPOT): $89.50 * POTENTIAL DEDUCTIONS: $408.62 Those hand-calculated numbers are your verification baseline. Step 1: Take Inventory[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-1-take-inventory "Direct link to Step 1: Take Inventory") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You have a library of tools in `~/tools`: | Tool | What It Does | Built In | | --- | --- | --- | | `sum.py` | Sums decimal numbers from stdin | Lesson 1 | | `sum-expenses.py` | Extracts and sums the Amount column from bank CSVs | Lesson 3 | | `extract-column.py` | Pulls one column from any CSV | Lesson 4 | | `filter.py` | Keeps numbers matching a condition | Lesson 4 | | `stats.py` | Prints sum, count, average, min, max | Lesson 4 | | `tax-categorize.py` | Categorizes transactions, prints subtotals by category | Lesson 5 | What's missing: a `tax-prep` command that adds a POTENTIAL DEDUCTIONS total and runs from any folder without typing `python3 ~/tools/...`. Step 2: Build tax-prep[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-2-build-tax-prep "Direct link to Step 2: Build tax-prep") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open Claude Code: You: I have tax-categorize.py in ~/tools. It categorizes bank transactions andprints subtotals by Medical, Charitable, and Business categories.I want a tax-prep command I can run like this: cat ~/finances/2025/*.csv | tax-prepIt should do everything tax-categorize.py does, plus print aPOTENTIAL DEDUCTIONS total at the end (the sum of all category subtotals). Claude Code: I'll update tax-categorize.py to include the final total,then install it as a permanent tax-prep command.[Updates: ~/tools/tax-categorize.py — adds POTENTIAL DEDUCTIONS sum]#!/usr/bin/env python3import sys, csv, reCATEGORIES = { 'medical': [r'\bCVS\b', r'\bWALGREENS\b', r'\bPHARMACY\b', r'\bMEDICAL\b'], 'charitable': [r'\bDONATION\b', r'\bRED CROSS\b', r'\bUNITED WAY\b'], 'business': [r'\bOFFICE DEPOT\b', r'\bSTAPLES\b', r'\bSUBSCRIPTION\b'],}FALSE_POSITIVES = [r'\bDR\.?\s*PEPPER\b', r'\bCVSMITH\b']def categorize(description): desc_upper = description.upper() for fp_pattern in FALSE_POSITIVES: if re.search(fp_pattern, desc_upper, re.IGNORECASE): return None for category, patterns in CATEGORIES.items(): for pattern in patterns: if re.search(pattern, desc_upper, re.IGNORECASE): return category return Nonereader = csv.DictReader(sys.stdin)totals = {}items = {}for row in reader: amount = float(row['Amount'].replace('$', '').replace(',', '')) if amount >= 0: # Skip credits/refunds — only process debits continue cat = categorize(row['Description']) if cat: expense = abs(amount) totals.setdefault(cat, 0.0) items.setdefault(cat, []) totals[cat] += expense items[cat].append((row['Description'], expense))for cat, entries in items.items(): for desc, amount in entries: print(f"{cat.upper()}: {desc}: ${amount:.2f}")print("\n--- TOTALS ---")potential_deductions = 0.0for cat, total in totals.items(): print(f"{cat.title()}: ${total:.2f}") potential_deductions += totalprint(f"\nPOTENTIAL DEDUCTIONS: ${potential_deductions:.2f}")[Runs: chmod +x ~/tools/tax-categorize.py][Checks: echo $SHELL → /bin/zsh][Appends to ~/.zshrc:]alias tax-prep='python3 ~/tools/tax-categorize.py'[Runs: source ~/.zshrc]Done. Try it:cat ~/finances/test-2025.csv | tax-prep Step 3: Verify Before Touching Real Data[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-3-verify-before-touching-real-data "Direct link to Step 3: Verify Before Touching Real Data") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run it on your test data (or the generated test file from the tip above): cat ~/finances/test-2025.csv | tax-prep **Expected output:** MEDICAL: CVS/PHARMACY #1234: $45.67MEDICAL: WALGREENS #5678: $23.45MEDICAL: DR MARTINEZ MEDICAL: $150.00CHARITABLE: UNITED WAY: $100.00BUSINESS: OFFICE DEPOT #901: $89.50--- TOTALS ---Medical: $219.12Charitable: $100.00Business: $89.50POTENTIAL DEDUCTIONS: $408.62 DR PEPPER SNAPPLE and CVSMITH CONSULTING are absent. The totals match your hand calculations. Now you can trust it on real data. Checkpoint: Prove tax-prep Is Permanent 1. Close your terminal completely 2. Open a brand new terminal 3. Navigate to any folder: `cd ~/Desktop` 4. Run: `cat ~/finances/test-2025.csv | tax-prep` If you see the report (your command is installed. If you see "command not found") check your `~/.zshrc` alias. Step 4: Process a Full Year[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-4-process-a-full-year "Direct link to Step 4: Process a Full Year") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your bank exports one CSV per month. By year's end, you'll have twelve files. If you `cat *.csv` to combine them, every file's header row: `Date,Description,Amount`: ends up mixed into the data. Your script sees the header eleven times where it expects numbers. The fix uses two commands you already know from the File Processing chapter: # Header from first file onlyhead -1 ~/finances/2025/january.csv > ~/finances/combined-2025.csv# Data rows from ALL files (skip each file's header)tail -n +2 -q ~/finances/2025/*.csv >> ~/finances/combined-2025.csv# Now process the clean combined filecat ~/finances/combined-2025.csv | tax-prep | Command | What It Does | | --- | --- | | `head -1` | First line only (the header row) | | `tail -n +2` | Everything from line 2 onward (skips header) | | `-q` | Quiet mode: no filename prefixes in output | | `>>` | Append (don't overwrite) | Result: one file, one header row, all data rows. For multiple monthly statements, you can also skip the intermediate file entirely: # Combine 12 months into one file (single header, all data rows)head -1 ~/finances/2025/january.csv > ~/finances/combined-2025.csvtail -n +2 -q ~/finances/2025/*.csv >> ~/finances/combined-2025.csv# Run tax-prep on the full yearcat ~/finances/combined-2025.csv | tax-prep Or skip the intermediate file entirely: # Direct pipeline — no temp file neededcat ~/finances/2025/*.csv | grep -v "^Date" | \ { echo "Date,Description,Amount"; cat; } | tax-prep The command from the README works exactly as promised. What Just Happened?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#what-just-happened "Direct link to What Just Happened?") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Remember the Seven Principles from the Seven Principles chapter? You just used all of them in one workflow, without a checklist, without thinking about it. That is the point. Principles are not rules you consult. They are habits you act on. | Principle | Where It Appeared | | --- | --- | | **Bash is the Key** | `cat`, `head`, `tail`, pipes orchestrated all data flow | | **Code as Universal Interface** | Python scripts executed computation: no hallucinated math | | **Verification as Core Step** | Test data with hand-calculated totals BEFORE real files | | **Small, Reversible Decomposition** | Composable single-purpose tools (L4), each testable independently | | **Persisting State in Files** | Scripts in `~/tools`, report saved to a file | | **Constraints and Safety** | False positive guards prevented miscategorized deductions | | **Observability** | Every transaction printed before the totals section | When Things Break: Quick Diagnostic Chain Six months from now, something will stop working. Maybe you updated your shell, maybe Python changed versions, maybe you moved to a new machine. Here's what to check: # 1. Does the alias exist?alias tax-prep# If "not found" → re-add to ~/.zshrc (or ~/.bashrc), then source# 2. Does the script exist where the alias points?ls -la ~/tools/tax-categorize.py# If "not found" → script was moved or deleted# 3. Can the script run?python3 ~/tools/tax-categorize.py <<< "Date,Description,Amount"# If error → Python version mismatch or missing shebang | Symptom | Check | Fix | | --- | --- | --- | | "command not found" | `alias tax-prep` | Re-add alias to shell config, then `source` | | "No such file" | `ls ~/tools/tax-categorize.py` | Script was moved: update the alias path | | "Permission denied" | `ls -la ~/tools/tax-categorize.py` | Re-run `chmod +x ~/tools/tax-categorize.py` | | Script errors on run | `python3 --version` | Python version changed: check shebang line | Setup is the agent's job. Diagnosis is yours: because when it breaks at 11pm before a deadline, you need to know the three places to look. * * * The Victory[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#the-victory "Direct link to The Victory") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ Before this chapter, Bash couldn't add decimals and you had no way to catch silent bugs in agent-generated code. Now you have a library of verified Unix-styled Python commands, a verification habit that applies to any domain, and the instinct to catch the agent's mistakes before they become yours. Tax prep was the exercise. The skill is the workflow. Challenge: Prove It Transfers (30 Minutes)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#challenge-prove-it-transfers-30-minutes "Direct link to Challenge: Prove It Transfers (30 Minutes)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You've run the tax prep workflow on financial data. Lesson 5 proved it works on server logs. Now prove you can do it from scratch on a domain neither lesson covered: no walkthrough, just the goal. Save this as `~/grades/midterm-2025.csv`: Student,Assignment,Score,Max_Points,CategoryAlice,Homework 1,85,100,homeworkBob,Homework 1,92,100,homeworkAlice,Quiz 1,18,20,quizBob,Quiz 1,15,20,quizCharlie,Homework 1,0,100,homeworkAlice,Midterm,78,100,examBob,Midterm,88,100,examCharlie,Quiz 1,19,20,quizDR CHARLES,Homework 1,95,100,homeworkAlice,EXTRA CREDIT,5,0,bonusCharlie,Midterm,72,100,examDR CHARLES,Quiz 1,20,20,quiz **Your task:** 1. Calculate each student's weighted average (homework 30%, quizzes 20%, exams 50%) 2. Handle the edge cases: DR CHARLES is a student named Charles, not a "DR" prefix to filter. EXTRA CREDIT has Max\_Points=0: division by zero trap. Charlie has a 0/100 homework 3. Flag students with any single score below 60% 4. Produce a grade report with per-student averages and an AT-RISK section **Hand-calculate first:** | Student | Homework | Quiz | Exam | Weighted Avg | | --- | --- | --- | --- | --- | | Alice | 85% | 90% | 78% | 82.5% | | Bob | 92% | 75% | 88% | 86.6% | | Charlie | 0% | 95% | 72% | 55.0% | | DR CHARLES | 95% | 100% | : | (no exam) | **Edge cases to handle:** Charlie has a 0/100 homework (that's your at-risk flag. EXTRA CREDIT has Max\_Points=0) your script crashes or silently produces infinity unless you handle it. DR CHARLES is a student named Charles, not a "DR" prefix to filter. If your report handles all three edge cases, the pattern transferred. You didn't need bank statements or server logs. You needed the workflow. Reflection: What You Actually Learned[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#reflection-what-you-actually-learned "Direct link to Reflection: What You Actually Learned") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The agent wrote all the code. You made all the decisions that mattered. | What It Looked Like | What You Actually Learned | | --- | --- | | Building sum.py and decomposing into tools | Designing Unix-style architectures where each piece is independently testable | | Testing with known data, spotting Dr. Pepper | Trusting nothing until you've verified it, and finding bugs in output that looks correct | | CSV parsing, redirecting the agent from awk | Redirecting an agent when its first approach fails: your domain knowledge steers the fix | | Writing the prompts | Specifying outcomes and interfaces: the one contribution the agent cannot make for itself | The specific tools (Python, regex, find/xargs) will change. The patterns: verify first, compose through pipes, guard against false positives: will not. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Add a NEEDS REVIEW Section[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#prompt-1-add-a-needs-review-section "Direct link to Prompt 1: Add a NEEDS REVIEW Section") My tax-prep command categorizes transactions correctly. But sometransactions don't match any category — they're just silently ignored.Modify it to print a NEEDS REVIEW section at the end listing alluncategorized transactions with amounts, so I can review them manually. **What you're learning:** A director decision disguised as a feature request. You're telling the agent the tool must make its own uncertainty visible rather than silently ignore it. "Print what you couldn't categorize" is not an implementation detail; it's a design principle you imposed. The agent wired the NEEDS REVIEW output; you decided that discarding uncategorized data silently was unacceptable. That call was yours. ### Prompt 2: Add Date Filtering[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#prompt-2-add-date-filtering "Direct link to Prompt 2: Add Date Filtering") My tax-prep processes all transactions in the CSV. For quarterlyestimates, I need to filter by date range: cat finances.csv | tax-prep --start 2025-01-01 --end 2025-03-31Add date filtering. Keep the stdin reading pattern so it still workswith pipes and cat. **What you're learning:** Interface-first directing. Notice the prompt specifies exactly what the command should look like from the outside (`tax-prep --start 2025-01-01 --end 2025-03-31`) before mentioning implementation. You designed the interface; the agent wired argparse to match it. This is the same move as "reads from stdin and prints the total" in Lesson 1: you specify the contract, the agent writes the code that fulfills it. ### Prompt 3: Transfer to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#prompt-3-transfer-to-your-domain "Direct link to Prompt 3: Transfer to Your Domain") I work with [your domain] data in CSV format. The data has[describe columns]. I need to categorize it by [your categories]and flag items that don't cleanly fit.Apply the verification-first pattern: create test data with knownanswers first, verify totals match before processing real files,then build a permanent command I can reuse. **What you're learning:** Full pattern transfer. You're applying the verification-first orchestration to a domain you actually work in. Notice which parts of the pattern carry over unchanged (verify first, flag ambiguous items, make it permanent) and which require domain-specific knowledge (your categories, your false positives). Checking access... * [Step 1: Take Inventory](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-1-take-inventory) * [Step 2: Build tax-prep](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-2-build-tax-prep) * [Step 3: Verify Before Touching Real Data](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-3-verify-before-touching-real-data) * [Step 4: Process a Full Year](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#step-4-process-a-full-year) * [What Just Happened?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#what-just-happened) * [The Victory](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#the-victory) * [Challenge: Prove It Transfers (30 Minutes)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#challenge-prove-it-transfers-30-minutes) * [Reflection: What You Actually Learned](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#reflection-what-you-actually-learned) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#flashcards-study-aid) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#try-with-ai) * [Prompt 1: Add a NEEDS REVIEW Section](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#prompt-1-add-a-needs-review-section) * [Prompt 2: Add Date Filtering](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#prompt-2-add-date-filtering) * [Prompt 3: Transfer to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/capstone#prompt-3-transfer-to-your-domain) --- # One Tool, One Job | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C The Unix way: one tool, one job, infinite combinations. Your accountant calls with three questions: "What's the total?" "What was the average?" "How many purchases were over $100?" You have one script. It answers one question. So you start adding flags: `--average`, `--count`, `--threshold`, `--top`, `--limit`. Six months later: python3 sum-expenses.py --sum --average --count --threshold 100 --top --limit 5 --filter negatives Impossible to test (which flag combination broke?), impossible to debug (which feature caused the wrong number?), impossible to explain. The script that does everything is the script that breaks everything. There's a better way. It's been hiding in every command you've typed since Lesson 1. The Pattern You Already Know[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-pattern-you-already-know "Direct link to The Pattern You Already Know") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Look at what you've been doing all chapter: cat expenses.txt | python3 sum.pycat bank.csv | python3 sum-expenses.py `cat` does one thing: read a file. Your script does one thing: process the data. The pipe connects them. Neither tool knows about the other. Neither needs to. This is the Unix philosophy: **build small tools that each do one thing, then chain them with pipes.** Every tool reads stdin, writes stdout. Every tool is ignorant of what comes before or after it in the pipeline. That ignorance is the feature: it means any tool connects to any other tool without modification. Your sum-expenses script violates this principle. It reads CSV AND extracts a column AND filters negatives AND sums. Four responsibilities in one file. Time to break it apart. Your Turn First: 3 Minutes Before you ask Claude Code for help, try this yourself. Look at what sum-expenses.py does: 1. Reads a CSV 2. Extracts the Amount column 3. Filters for negative amounts (debits) 4. Sums the result If you had to split this into separate scripts (each reading stdin and writing stdout) how would you divide the work? How many scripts? What would each one do? Write down your decomposition (even just bullet points) before reading on. Then compare your design to what Claude Code builds. Did you split in the same places? Did you make the tools more specific (bank-only) or more general (any CSV)? Building Three Tools[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#building-three-tools "Direct link to Building Three Tools") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open Claude Code: You: I want to decompose my CSV processing into small, chainable tools.Build me three separate scripts that each do ONE thing:1. extract-column.py — reads CSV from stdin, outputs just one column (by name or number), one value per line2. filter.py — reads numbers from stdin, keeps only those matching a condition like "< 0" or "> 100"3. stats.py — reads numbers from stdin, prints sum, count, average, min, and max The agent builds all three. Here's what each one does: **extract-column.py**: `cat data.csv | extract-column Amount` reader = csv.DictReader(sys.stdin)for row in reader: print(row[column]) # One value per line to stdout **filter.py**: `filter "< 0"` keeps numbers matching a condition for line in sys.stdin: value = float(line.strip()) if op(value, threshold): # op parsed from "< 0", "> 100", etc. print(line.strip()) **stats.py**: reads numbers from stdin, prints sum, count, average, min, max numbers = [float(line) for line in sys.stdin if line.strip()]print(f"Sum: {sum(numbers):.2f}") # Plus count, average, min, max The full scripts are in your working directory. Three scripts, each reading stdin, each writing stdout. None knows about bank statements, only about columns, numbers, and conditions. The Power of Recombination[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-power-of-recombination "Direct link to The Power of Recombination") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now watch what happens when you chain them: # Total expenses (same answer as sum-expenses):cat ~/finances/sample-2025.csv | extract-column Amount | filter "< 0" | stats **Output:** Sum: -1751.29Count: 28Average: -62.55Min: -200.00Max: -4.99 Same data. Five answers instead of one. And you didn't modify a single script: you just connected tools that already existed. Now change the question: # How many transactions over $100?cat ~/finances/sample-2025.csv | extract-column Amount | filter "< -100" | stats # What are the merchant names?cat ~/finances/sample-2025.csv | extract-column Description # What's the average income (positive amounts)?cat ~/finances/sample-2025.csv | extract-column Amount | filter "> 0" | stats **Same three tools. Four different questions. Zero code changes.** | What You Asked | Pipeline | | --- | --- | | Total expenses | `extract-column Amount \| filter "< 0" \| stats` | | Large expenses only | `extract-column Amount \| filter "< -100" \| stats` | | All merchant names | `extract-column Description` | | Average income | `extract-column Amount \| filter "> 0" \| stats` | The data doesn't change. The tools don't change. Only the pipeline changes, and that's just a different arrangement of the same building blocks. Why This Is Better[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#why-this-is-better "Direct link to Why This Is Better") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A bug in `filter.py` breaks one pipe segment, not your entire workflow. You test each tool with three numbers from stdin instead of every flag combination. And when you need a new capability (say, sorting by amount) you build `sort-numbers.py`, test it in isolation, and plug it into the pipeline. The existing tools don't know it exists and don't need to. Small tools answer questions you haven't thought of yet because the pipeline changes, not the tools. The Principle Connection[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-principle-connection "Direct link to The Principle Connection") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Two of the Seven Principles from the Seven Principles chapter come alive here: **P2: Code as Universal Interface.** `extract-column` doesn't know it's processing bank statements. It extracts a column from ANY CSV: bank data, payroll, student grades, server logs. The tool is universal _because_ it's small. The less a tool knows about its context, the more contexts it works in. **P4: Small, Reversible Decomposition.** If `filter.py` has a bug, you fix one script and re-test it with `echo -e "10\n-5\n20" | filter "< 0"`. If `sum-expenses.py` has a bug, you're debugging 30 lines of intertwined logic. Small tools have small blast radii. These two principles reinforce each other. Small tools (P4) become universal interfaces (P2) because their simplicity makes them context-independent. The Retrospective Insight Look back at sum-expenses.py from Lesson 3. It was doing three jobs: extracting the Amount column, filtering for negatives, and summing the result. It worked, but it could only answer one question. The three-tool decomposition doesn't replace sum-expenses; it reveals the composable architecture that was hiding inside it. You can keep sum-expenses.py for the common case; it's a convenient shortcut. But when you need a question it can't answer, you have the building blocks to construct any pipeline you need. Install Your Library[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#install-your-library "Direct link to Install Your Library") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Same pattern as Lesson 3. Ask Claude Code to install all three as permanent commands in `~/tools` with aliases. Your directory now has a library: ~/tools/├── sum.py # Lesson 1├── sum-expenses.py # Lesson 3├── extract-column.py # This lesson├── filter.py # This lesson└── stats.py # This lesson Checkpoint: Prove the Pipeline Works Run this and verify the expense total matches what sum-expenses produced in Lesson 3: cat ~/finances/sample-2025.csv | extract-column Amount | filter "< 0" | stats The Sum line should show your expense total. Same answer, different architecture, but now you can ask questions sum-expenses never could. Then try a question sum-expenses CAN'T answer: cat ~/finances/sample-2025.csv | extract-column Amount | filter "< -100" | stats If both work, your composable toolkit is operational. You made the architecture decision: three tools, not one, each reading stdin and writing stdout. The agent made every implementation decision within that architecture. That's the director's role at its clearest: you decide what to build and how the pieces connect. The agent decides how each piece works inside. The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-pattern "Direct link to The Pattern") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- "Decompose [big script] into small tools that each do one thing.Each tool reads stdin and writes stdout so I can chain them with pipes." This prompt pattern works because it gives the agent two constraints: single responsibility (one thing) and composability (stdin/stdout). Everything else (the language, the parsing logic, the error handling) is the agent's call. You can extract, filter, and summarize any column in any CSV. But tax season needs something these generic tools can't do: look at a merchant name and decide if it's medical, charitable, or business. That's not filtering: that's _judgment_. And judgment needs patterns. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#flashcards-study-aid "Direct link to Flashcards Study Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Extend stats.py[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#prompt-1-extend-statspy "Direct link to Prompt 1: Extend stats.py") My stats.py prints sum, count, average, min, and max. Add medianand standard deviation. Keep the stdin reading pattern so it stillworks in pipelines. **What you're learning:** Extending a tool without breaking its interface. stats.py gains two capabilities, but its contract (reads numbers from stdin, prints results to stdout) doesn't change. Every existing pipeline that uses stats.py gets the new statistics for free. That's what composability buys you: improvements propagate without rewiring. ### Prompt 2: Build a New Tool[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#prompt-2-build-a-new-tool "Direct link to Prompt 2: Build a New Tool") Build me a top.py script that reads numbers from stdin and printsthe N largest values. Default to 5 if no argument given.I want to use it like: cat bank.csv | extract-column Amount | filter "< 0" | top 3 **What you're learning:** Adding a new capability to your toolkit without touching existing tools. You specified the interface (`top 3`), the input source (stdin), and the output behavior (print N values). The agent handles implementation. Tomorrow, if you need the N smallest instead, you build `bottom.py`: same pattern, new tool, zero changes to anything else. ### Prompt 3: When NOT to Decompose[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#prompt-3-when-not-to-decompose "Direct link to Prompt 3: When NOT to Decompose") I have a 15-line Python script that converts temperatures fromFahrenheit to Celsius. It reads from stdin and writes to stdout.Should I decompose it further, or is it already a good single-purposetool? When does decomposition stop being helpful? **What you're learning:** The boundary of decomposition. Not every script needs splitting: a 15-line single-purpose tool that reads stdin and writes stdout is already following the Unix philosophy. The agent's answer teaches you to recognize when a tool is "done": when splitting it further would create tools too small to be useful on their own. The rule: if a tool does one thing and you can test it with one command, it's small enough. Checking access... * [The Pattern You Already Know](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-pattern-you-already-know) * [Building Three Tools](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#building-three-tools) * [The Power of Recombination](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-power-of-recombination) * [Why This Is Better](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#why-this-is-better) * [The Principle Connection](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-principle-connection) * [Install Your Library](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#install-your-library) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#the-pattern) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#flashcards-study-aid) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#try-with-ai) * [Prompt 1: Extend stats.py](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#prompt-1-extend-statspy) * [Prompt 2: Build a New Tool](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#prompt-2-build-a-new-tool) * [Prompt 3: When NOT to Decompose](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/your-permanent-toolkit#prompt-3-when-not-to-decompose) --- # Testing AI Safely with Branches | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Sarah needs two versions of her fundraiser flyer. One is formal for corporate sponsors. One is casual for social media. She doesn't want to lose either while working on both. In the real world, you'd make two copies of the file. In Git, you create two **branches**: parallel versions of your entire project that can't interfere with each other. > **"Branches let you live in two timelines at once, and only keep the one that works."** * * * What Is a Branch?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#what-is-a-branch "Direct link to What Is a Branch?") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've used Track Changes to compare different versions of a document. Branches take that idea further. Instead of tracking changes to one file, a branch creates a parallel version of your _entire project_. Your `main` branch is the "real" version: the one you trust. When you create a new branch, Git creates a new label pointing to the current snapshot. It feels like a copy because you can edit safely without touching `main`. If the changes work, you merge them back. If they don't, you throw away the branch name. The original is untouched. ![Branch diagram showing main branch (stable) with feature branches diverging for development, then merging back after review](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/images/part-2/chapter-20/branch-workflow-diagram.png) * * * Create Your First Branch[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#create-your-first-branch "Direct link to Create Your First Branch") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open your terminal in your fundraiser project (from Lesson 1). Let's create a branch for Sarah's formal flyer. **What you tell your agent**: "I want to work on a formal version of the flyer without changing the main project." **What the agent does**: git branch formal-flyer That created the branch. But you're still on `main`. Check: git branch formal-flyer* main The asterisk shows you're on `main`. Now switch: git switch formal-flyer Switched to branch 'formal-flyer' You're now on the `formal-flyer` branch. Everything looks the same because the branch starts as an exact copy of `main`. * * * Changes Stay on Their Branch[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#changes-stay-on-their-branch "Direct link to Changes Stay on Their Branch") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now make a change that only exists on this branch. **What you tell your agent**: "Create a formal flyer for the fundraiser." **What the agent does**: echo "FORMAL FLYER: Annual Community Fundraiser Gala" > flyer.txtecho "Date: March 15 | Venue: Grand Ballroom | Dress Code: Business Formal" >> flyer.txtgit add flyer.txtgit commit -m "Add formal flyer for corporate sponsors" The flyer exists on `formal-flyer`. Now switch back to `main`: git switch mainls **The flyer is gone.** It only exists on the `formal-flyer` branch. Your main project is completely untouched. This is branch isolation in action. If Sarah decides the formal flyer is terrible, she can delete the branch. Main stays exactly as it was. No risk. * * * Test Two Ideas at Once[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#test-two-ideas-at-once "Direct link to Test Two Ideas at Once") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sarah also wants a casual flyer. Create a second branch from `main`: git branch casual-flyergit switch casual-flyer Make the casual version: echo "HEY NEIGHBORS! Community Fundraiser Party!" > flyer.txtecho "Food, music, fun. Bring the whole family. Donations welcome!" >> flyer.txtgit add flyer.txtgit commit -m "Add casual flyer for social media" Now Sarah has three versions of her project: * `main`: the original, no flyer yet * `formal-flyer`: formal version for sponsors * `casual-flyer`: casual version for social media Each exists independently. She can switch between them to compare. * * * Merge the Winner[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#merge-the-winner "Direct link to Merge the Winner") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sarah decides the casual flyer fits her audience better. Time to bring those changes into `main`. **Step 1: Switch to main first.** git switch main **Step 2: Merge the winning branch.** git merge casual-flyer Updating abc1234..def5678Fast-forward flyer.txt | 2 ++ 1 file changed, 2 insertions(+) "Fast-forward" means `main` hadn't changed since you branched off: Git simply moved `main` forward to include the new commits. No merging was needed because there was nothing to merge _with_. The casual flyer now exists on `main`. Verify: cat flyer.txt **Step 3: Clean up.** git branch -d casual-flyergit branch -d formal-flyer The branch names are deleted. The commits from `casual-flyer` are preserved in `main`'s history. The `formal-flyer` commits were never merged, so they are no longer part of Sarah's main workflow. tip `git branch -d` only deletes the branch name, not the work. After merging, the commits live on in `main`. If you try to delete a branch you _haven't_ merged, Git warns you. Use `-D` (capital D) only if you're sure you want to throw away unmerged work. * * * Branch Naming[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#branch-naming "Direct link to Branch Naming") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Good branch names tell you what's happening at a glance: * `feature/formal-flyer`: adding something new * `experiment/new-layout`: testing a risky idea * `bugfix/broken-budget`: fixing a problem Bad names like `branch1` or `test` tell you nothing in a week. Name branches by what they do, not when you made them. * * * When to Branch vs When to Commit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#when-to-branch-vs-when-to-commit "Direct link to When to Branch vs When to Commit") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not every change needs a branch. **Use a branch when:** * You're testing two or more approaches * The change might break something * You want to review before merging * Someone else might be working on the same project **Commit directly to main when:** * You're confident the change is small and correct * You're the only person working on the project * The change is a quick fix, not an experiment **A simple test**: If you're thinking "this might go wrong," create a branch. * * * You can test ideas in two timelines. But both timelines only exist on YOUR computer. What if your hard drive crashes tomorrow? That's what the next lesson solves. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Understand branch isolation:** > "Explain why files I create on a Git branch disappear when I switch to main. Use a real-world analogy that isn't about parallel universes or video games. Help me understand why this is useful, not scary." **Practice comparison workflows:** > "I want to test two different approaches to a project: one simple and one advanced. Walk me through the branch workflow: create both branches, work on each one, compare results, merge the winner, and clean up. Include the exact commands my agent would run." **Explore naming conventions:** > "What are good Git branch naming conventions? Show me examples for features, experiments, and bug fixes. Then help me decide: should I name branches by what they do (feature/new-flyer) or by when I made them (experiment-march-15)?" **Know when NOT to branch:** > "I'm tempted to create a branch for every tiny change. Give me five realistic scenarios and for each one, tell me whether I should branch or commit directly to main, and explain why. Help me build instinct for when branching is overkill." * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [What Is a Branch?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#what-is-a-branch) * [Create Your First Branch](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#create-your-first-branch) * [Changes Stay on Their Branch](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#changes-stay-on-their-branch) * [Test Two Ideas at Once](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#test-two-ideas-at-once) * [Merge the Winner](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#merge-the-winner) * [Branch Naming](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#branch-naming) * [When to Branch vs When to Commit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#when-to-branch-vs-when-to-commit) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#try-with-ai) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/testing-ai-safely-with-branches#flashcards-study-aid) --- # Creating & Reading Data | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 2, you described your data model in plain English and the agent built the schema. Tables exist, columns have types, constraints are in place. But a schema without data is like a filing cabinet with labeled drawers and nothing inside. Now you will put data in and get it back out -- reliably. You might be thinking: "Why can't the agent just store the data and move on?" Because storing without saving is like writing a check without signing it -- the bank will not process it. And if something goes wrong mid-write, you need a way to tear up that check cleanly. That is what this lesson is about: the session lifecycle that makes database writes trustworthy. Key Terms for This Lesson * **CRUD**: Create, Read, Update, Delete -- the four basic operations every database application needs * **Session**: A workspace for database operations -- think of it as a shopping cart where you add items before checking out * **Stage vs Save**: The "stage" step puts data in a holding area (like adding items to a shopping cart). The "save" step makes them permanent (like clicking "Place Order"). If anything fails, you can undo and empty the cart. * **Rollback**: Undo all unsaved changes in the current session -- your safety net when writes go wrong The Session Lifecycle[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#the-session-lifecycle "Direct link to The Session Lifecycle") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before a single row gets stored, understand the machine the agent's code works with. A session moves through a predictable sequence, and knowing where you are in that sequence helps you diagnose problems when output does not match expectations. This is what is happening inside the database engine when your agent's code runs. You do not need to understand each step -- but this diagram explains WHY your agent needs to both write AND verify, in two separate operations. Session Lifecycle: open stage preview save | | | | v v v v+------+ +-----------+ +----------+ +----------+| New |--->| Staged |--->| Previewed|--->| Saved ||Session| | Changes | | (in DB | | (durable)|+------+ +-----------+ | but not | +----------+ | permanent)| +-----+----+ | error? v +----------+ | Rollback | | (undo | | all) | +----------+ Here is what each stage means in plain language: 1. **Open**: The agent creates a session. Nothing has happened yet. 2. **Staged**: The agent tells the session to track new data. The data is held in memory, but the database has not seen it. 3. **Previewed**: The agent asks the database to preview the data. The database assigns IDs -- but this is temporary. If the session ends without a save, the changes vanish. 4. **Saved**: The agent saves. The transaction finalizes. The data is durable -- it survives restarts, crashes, power outages. 5. **Rollback**: Something went wrong. The agent undoes everything, and every staged or previewed change in that session is erased. Clean slate. Why does the preview step exist separately from save? Because sometimes the database needs to assign an ID (like an auto-incrementing primary key) before related rows can reference it -- but you do not want to save until everything succeeds. The Happy Path: Store and Verify[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#the-happy-path-store-and-verify "Direct link to The Happy Path: Store and Verify") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here is the pattern you will use hundreds of times. Tell the agent what to store, then verify the result by reading the output. What you tell the agent Store one expense for Alice in the Food category for $52.50 on January 15, 2024. Then show me all expenses over $50. What you verify The agent stores the expense and gives you a verify command. python verify_crud.pyOutput: ✓ Stored: Groceries | Alice | Food | $52.50 | 2024-01-15 Expenses over $50: - Groceries: $52.50 1 expense found. What matters here is the output, not the code the agent wrote. You are reading the receipt, not the cash register's wiring diagram. The output tells you: * The data was stored (the row appears with the values you described) * The data survived a separate query (it was not just held in temporary memory) * The filter worked (only expenses over $50 appear) If any of those three things are wrong, you tell the agent what to fix. You do not edit the code yourself. Read-Path Verification Checklist[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#read-path-verification-checklist "Direct link to Read-Path Verification Checklist") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Do not trust that writes worked just because no error appeared. Trust the read: 1. Store one known row with specific values you can check 2. Read that row back in a separate query 3. Check that field values match what you described, not just that a row exists 4. Tell the agent to run one failing insert (for a user that does not exist) and show you the error output 5. Repeat the read to prove the failed write left no leftover data This simple loop prevents most beginner data illusions -- the cases where you think data was saved but it was only in temporary memory, or where a failed write silently left partial data behind. The Failure Path: Rollback in Action[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#the-failure-path-rollback-in-action "Direct link to The Failure Path: Rollback in Action") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might be thinking: "When would a save actually fail?" More often than you would hope -- duplicate emails, invalid references, network hiccups. The question is not IF it fails, it is WHEN. What you tell the agent Try to store an expense for a user that does not exist (user ID 9999). Show me what error the database returns. Then verify zero new rows were created. What you verify The agent runs the test. You see: python test_failure.pyOutput: Attempting to store expense for user_id=9999... ✗ Database rejected: foreign key constraint violation (user 9999 does not exist) Expense count before: 1 Expense count after: 1 ✓ Rollback confirmed — zero partial rows created What the output means: the database rejected the row because user 9999 does not exist in the users table. That is your foreign key constraint doing its job -- the same rule you described in Lesson 2 as "must reference an existing user." The rollback undid everything in this session, leaving the database exactly as it was before. The expense count stayed at 1, proving no partial data leaked through. Running one controlled failure now saves hours of confusing problems later. Storing data without saving is the database equivalent of "I will save my file later." We all know how that story ends. Pause and Reflect You have seen both a successful store and a failed one. Why is the failure case just as important to practice? In production, which scenario do you think happens more often -- and what happens if your agent's code does not handle it? What to Ask For[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#what-to-ask-for "Direct link to What to Ask For") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you need specific data from the database, describe what you want in English: * "Show me all expenses" -- the agent fetches all rows * "Show me the first Food expense" -- the agent fetches one matching row * "Show me all expenses in March" -- the agent filters by date range * "Show me total spending by category" -- the agent groups and sums The agent handles the query syntax. You describe what you need and verify the output matches your expectations. If the output looks wrong, describe what you expected and ask the agent to fix the query. Debugging by Directing[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#debugging-by-directing "Direct link to Debugging by Directing") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When things do not work as expected, tell the agent what you observe: * **If the verification shows empty results**: tell the agent "the result was empty -- check if the save step actually ran after staging the data" * **If a bad reference insert does not get rejected**: tell the agent "the database accepted an invalid reference -- check if foreign key enforcement is enabled" * **If a query returns unexpected results**: tell the agent "the filter might not be specific enough -- return all matching rows so I can see what is in the database" You diagnose by reading output. The agent fixes by editing code. That division of labor is the director pattern. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Review Agent Code for Safety[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#prompt-1-review-agent-code-for-safety "Direct link to Prompt 1: Review Agent Code for Safety") I want to verify my database code is correct. Review this code and find:missing save step, missing rollback on failure, foreign key misuse, andwrong assumptions about one-result vs all-results queries. Provide fixeswith explanations. **What you're learning:** You are building the skill of directing an AI code review. You describe what to look for, the agent inspects the code, and you read the findings. This is how directors catch bugs without reading source code -- by knowing what questions to ask. ### Prompt 2: Build and Verify a Query[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#prompt-2-build-and-verify-a-query "Direct link to Prompt 2: Build and Verify a Query") Add a function that lists all expenses over a given amount. Show me whatrunning it would output for expenses of $30, $52.50, and $120. **What you're learning:** You are practicing the store-then-verify loop. By asking the agent to show output for specific inputs, you can check whether the query logic matches your expectations without reading the query code itself. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#checkpoint "Direct link to Checkpoint") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I can explain why data that is only in temporary memory is not persistent (it disappears when the process exits) * [ ] I directed the agent to store one expense and verified the output showed it was saved * [ ] I directed the agent to test a failure case and read the rollback confirmation * [ ] I know what to tell the agent if the verification comes back empty Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#flashcards-study-aid "Direct link to Flashcards Study Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Session Lifecycle](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#the-session-lifecycle) * [The Happy Path: Store and Verify](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#the-happy-path-store-and-verify) * [Read-Path Verification Checklist](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#read-path-verification-checklist) * [The Failure Path: Rollback in Action](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#the-failure-path-rollback-in-action) * [What to Ask For](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#what-to-ask-for) * [Debugging by Directing](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#debugging-by-directing) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#try-with-ai) * [Prompt 1: Review Agent Code for Safety](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#prompt-1-review-agent-code-for-safety) * [Prompt 2: Build and Verify a Query](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#prompt-2-build-and-verify-a-query) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/creating-reading-data#flashcards-study-aid) --- # Pivots Five and Six: The Hybrid Resolution and Platform Inversion | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C James stared at the whiteboard Emma had drawn in the previous session. Two architectures. Custom Brain on the left: fast to ship, expensive to run, no code execution. NanoClaw on the right: better isolation, better routing, two to four months of engineering before a single learner could use it. "Both of these solve real problems," he said. "But they solve different problems on different schedules. The PIAIC cohort needs tutoring in weeks, not months. The book needs to teach NanoClaw because that is the better architecture. And nobody knows yet which one will actually generate revenue." Emma nodded. "Three goals. Three timelines. That is exactly the problem we faced. And for a while, we thought we had to pick one." James looked back at his TutorClaw from Chapter 58. Nine tools. Three components. An MCP server that learners connect to with a single `clawhub install` command. It looked clean. It looked inevitable. But sitting here, tracing the decisions backward, he could see the fracture lines where the team had been stuck between two architectures with no obvious way to reconcile them. "So what happened?" Emma sat down. "Two things happened. First, we stopped trying to pick one. Then we stopped building infrastructure entirely." * * * You are doing exactly what James is doing. You have two architectures with real trade-offs, three goals pulling in different directions, and no single solution that satisfies all of them at once. The question is not which architecture wins. The question is whether the choice itself is the wrong frame. The Dilemma: Three Goals, Three Timelines[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#the-dilemma-three-goals-three-timelines "Direct link to The Dilemma: Three Goals, Three Timelines") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before the resolution, look at what the team was facing. This was not a simple either/or decision. Three goals needed to be satisfied, and each operated on a different timeline: | Goal | Timeline | What It Needed | | --- | --- | --- | | PIAIC tutoring for the current cohort | Weeks | A working tutoring system, shipped and operational | | The book teaching NanoClaw architecture | One quarter | Content that teaches the better architecture pattern | | Revenue data to validate the business model | Ongoing | Real learners paying real money, generating real usage data | No single architecture optimized for all three. Custom Brain could ship in weeks but could not execute code and cost over twelve thousand dollars per month in LLM fees. NanoClaw offered better isolation and multi-model routing but required months of engineering. Choosing Custom Brain meant teaching an architecture in the book that the team planned to replace. Choosing NanoClaw meant delaying the PIAIC cohort for months with no revenue data to validate the model. Sit with this for a moment. If you had to pick one, which would you choose? The instinct is to rank them. Ship first, because revenue validates everything. Or teach first, because the book is the long-term asset. Or collect data first, because decisions without evidence are guesses. Each ranking sacrifices something real. Ship first delays the better architecture. Teach first delays revenue. Collect data first delays everything while you wait. This is not a technology problem. It is a timing problem. And the team's first insight was that timing problems do not have technology solutions. Pivot 5: The Hybrid Resolution[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#pivot-5-the-hybrid-resolution "Direct link to Pivot 5: The Hybrid Resolution") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The resolution was to stop choosing. The Hybrid strategy ran three parallel tracks, one for each goal: | Track | Architecture | Goal | Timeline | | --- | --- | --- | --- | | Production | Custom Brain | Ship tutoring to PIAIC cohort now | 2-3 weeks | | Education | NanoClaw-native | Teach the better architecture in the book | One quarter | | Migration | Data-driven | Move production to NanoClaw when usage data justifies it | When data says so | This was not a compromise. A compromise would have been building a half-NanoClaw system that shipped on a delayed schedule. The Hybrid was a deliberate meta-strategy: run parallel tracks that each optimize for their own timeline, and converge when data justifies convergence. Nothing was wasted. The Custom Brain production track generated real revenue data and real learner feedback. The NanoClaw education track produced curriculum content that would survive regardless of which architecture won. The migration track waited for evidence instead of guessing. Notice what the Hybrid does NOT require: a prediction about which architecture will win. Custom Brain might turn out to be sufficient at scale. NanoClaw might prove unnecessary. The migration might never happen. Or it might happen in three months when the data clearly shows a need. The strategy works in every scenario because it does not bet on one future. The convergence trigger matters. "When data justifies it" is not vague; it is specific. For TutorClaw, the convergence criteria were: learner count crossing the isolation boundary (when shared-process architecture starts creating noticeable interference between learners), LLM cost exceeding revenue per learner (when the 90/10 split becomes unsustainable), or an IP incident (when a competitor copies the system prompt). Until one of those triggers fires, the parallel tracks continue independently. The key principle: when goals have different timelines, do not force a single architecture to satisfy all of them. Run parallel tracks. Let each track optimize for its own constraints. Converge when you have data, not opinions. But even the Hybrid Resolution had a hidden assumption. All three tracks assumed the same thing every previous architecture had assumed: Panaversity must build and operate infrastructure for all learners. The Hidden Assumption[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#the-hidden-assumption "Direct link to The Hidden Assumption") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Look at the five architectures the team had considered up to this point: 1. OpenClaw skill (Markdown): Panaversity writes the skill, but OpenClaw runs it on the learner's machine. 2. Custom Brain (WhatsApp + FastAPI): Panaversity runs the LLM, the database, the API, the WhatsApp integration. 3. NanoClaw (container-per-learner): Panaversity runs one container per learner, with LLM routing and isolation. 4. Hybrid (Custom Brain now, NanoClaw later): Panaversity runs Custom Brain first, then migrates to NanoClaw. 5. Each variation: Panaversity builds infrastructure. Panaversity operates it. Panaversity pays for it. Architecture 1 was the only exception, and it failed for a different reason: Markdown skills are plain text that anyone can copy, REST API calls from Markdown fail on non-frontier models, and a Markdown skill cannot execute code. So the team moved away from it and into progressively more complex infrastructure designs. Each pivot after Architecture 1 added more infrastructure, not less. Custom Brain added a FastAPI server, PostgreSQL, and WhatsApp integration. NanoClaw added Docker containers, a Claude Code Router, and per-learner isolation. The Hybrid added all of Custom Brain's infrastructure plus a migration path to NanoClaw's infrastructure. The trajectory was pointing toward more servers, more containers, more operational complexity. What if the answer was not choosing the right infrastructure to build, but questioning whether to build infrastructure at all? Pivot 6: The Platform Inversion[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#pivot-6-the-platform-inversion "Direct link to Pivot 6: The Platform Inversion") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here is the question that changed everything: if OpenClaw is the operating system for personal AI, and every PIAIC learner has OpenClaw installed as a course prerequisite, then why is Panaversity building infrastructure? The learner's machine IS the infrastructure. Their OpenClaw IS the messaging gateway. Their API key IS the LLM. Their local agent runtime IS the compute. Panaversity does not need to serve sixteen thousand learners. Sixteen thousand learners already have everything they need. What they are missing is the intelligence. This is the Platform Inversion. Instead of Panaversity building infrastructure to reach learners, learners use their existing platform to reach Panaversity's intelligence. The direction of the relationship flips. In every previous architecture, the arrow pointed outward: Panaversity serves learners. In the Platform Inversion, the arrow points inward: learners connect to Panaversity's intelligence on their own terms, using their own compute. But the first architecture (Markdown skill) already tried this and failed. The intelligence was delivered as plain text. Anyone could copy it. It could not execute code. It broke on non-frontier models. The Platform Inversion only works if there is a delivery mechanism that protects the intelligence while keeping it on the provider's side. The resolution: TutorClaw as a remote MCP server. The MCP server solves three problems that Markdown skills could not: | Problem | Markdown Skill | MCP Server | | --- | --- | --- | | IP protection | Plain text, fully copyable | Black box, server-side only | | Code execution | Not possible | Server-side sandbox via submit\_code tool | | Monetization gating | No enforcement mechanism | Tool-level access control per API key | The learner runs `clawhub install panaversity/tutorclaw`. Their OpenClaw agent discovers nine structured tools and calls them natively. All pedagogical intelligence, code execution, assessment, and monetization gating live on the MCP server. Content lives on Cloudflare R2 with a Worker gate. A thin shim skill provides offline fallback for the free tier. You built the local version of this in Chapter 58. You know what the nine tools are. You know how the content layer works (local files in your build, R2 in production). You know how the shim skill provides PRIMM-Lite fallback. Now you see why it was built this way: the Platform Inversion made it possible, and the MCP server made it practical. The MCP server is where the Platform Inversion and the IP problem converge into a single solution. Without MCP, the inversion would fail the same way Architecture 1 failed: the intelligence would be exposed. Without the inversion, the MCP server would still require Panaversity to run the LLM, the way Architecture 2 and 3 required. The two insights need each other. Neither is sufficient alone. The Economics of the Inversion[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#the-economics-of-the-inversion "Direct link to The Economics of the Inversion") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You analyzed these numbers in Chapter 59. Here is why they exist. Every previous architecture required Panaversity to pay for the LLM. Custom Brain routed all queries through Panaversity's OpenRouter account. NanoClaw ran one container per learner with Claude Code Router handling multi-model traffic. Both architectures put LLM cost on Panaversity's balance sheet. The Platform Inversion moves the LLM cost to the learner. The learner's OpenClaw uses the learner's API key. Panaversity's LLM cost drops to zero. What remains on Panaversity's side: the MCP server (a single VPS), the R2 content bucket (free tier), the gating Worker (free tier), and Stripe for payment processing. Total infrastructure: the same fifty to seventy dollars per month you saw in Chapter 59. The gross margin approaches 99.5%. Not because the team optimized costs. Because the team questioned whether the costs needed to exist at all. This is the connection between Pivot 6 and the 90/10 rule from Chapter 59. The 90/10 rule showed that LLM costs are 90% of total cost and infrastructure is only 10%. The Platform Inversion eliminates the 90% entirely. The remaining 10% is the only cost Panaversity bears. What the Inversion Requires[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#what-the-inversion-requires "Direct link to What the Inversion Requires") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The Platform Inversion is not free. It works because three conditions are true: 1. **The user already has the platform.** Every PIAIC learner installs OpenClaw as a course prerequisite. If they did not, the inversion would fail. 2. **The platform supports the delivery mechanism.** OpenClaw speaks MCP natively. An MCP server is a first-class citizen, not a hack. If the platform did not support MCP, delivering intelligence through it would be fragile. 3. **The intelligence can be delivered without exposing it.** The MCP server is a black box. The learner calls tools and receives results. The pedagogical engine, the assessment logic, the gating rules remain server-side. If the intelligence required client-side delivery (like a Markdown skill), IP protection would fail. Remove any one of these three conditions and the inversion breaks. This is not a universal pattern. It is a pattern that works when the conditions hold. Consider the trade-off the inversion creates. Panaversity gains transformative economics, but accepts a platform dependency. TutorClaw only works for learners who have OpenClaw. If OpenClaw changes its MCP protocol, Panaversity must adapt. If a competitor builds a tutoring product on the same platform, Panaversity competes on intelligence quality, not infrastructure. These are real risks. The team accepted them because the economic transformation (from thousands of dollars per month in LLM costs to zero) outweighed the coupling. This is the decision framework for any Platform Inversion: list the conditions, verify they hold, name the dependency, and evaluate whether the economics justify the coupling. If the answer is yes, invert. If any condition is fragile, build your own infrastructure. Update Your Architecture Decision Worksheet[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#update-your-architecture-decision-worksheet "Direct link to Update Your Architecture Decision Worksheet") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Add Pivots 5 and 6 to the worksheet you started in Lesson 3: | Pivot | What was the constraint? | What changed? | What survived? | | --- | --- | --- | --- | | 5\. Hybrid Resolution | | | | | 6\. Platform Inversion | | | | Now look at all six rows together. Circle (or bold) the entries in the "What survived?" column that appear in more than one row. Those repeating survivors are your candidate invariants. You will examine them closely in the next lesson. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Exercise 1: Map Your Own Hybrid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#exercise-1-map-your-own-hybrid "Direct link to Exercise 1: Map Your Own Hybrid") Think of a project (personal or professional) where you face competing goals with different timelines. Use this prompt to design a Hybrid Resolution: I have a project with multiple goals that operate on differenttimelines. Here are my goals:Goal 1: [describe goal and timeline]Goal 2: [describe goal and timeline]Goal 3: [describe goal and timeline]Help me design a Hybrid Resolution strategy. For each goal,suggest a parallel track that optimizes for that specifictimeline. Then identify: what data would I need to see beforeconverging the tracks? What is the convergence trigger? **What you are learning:** The Hybrid Resolution is not specific to TutorClaw. Whenever you face goals with conflicting timelines, the instinct is to pick one and accept the trade-offs. The parallel-track approach lets you satisfy each goal on its own schedule and converge when evidence (not opinion) justifies it. Practicing this on your own project makes the pattern transferable. ### Exercise 2: Invert an Infrastructure Assumption[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#exercise-2-invert-an-infrastructure-assumption "Direct link to Exercise 2: Invert an Infrastructure Assumption") Pick any product that currently runs its own infrastructure (a SaaS tool, a mobile app backend, an internal tool at your company). Use this prompt to test whether a Platform Inversion is possible: I want to evaluate whether [product name] could use a PlatformInversion. The product currently runs its own infrastructureto serve users. Here are the details:What the product provides: [describe the core value]What infrastructure it runs: [servers, databases, etc.]What platform users already have: [browser, OS, agent platform, etc.]Evaluate the three conditions for a Platform Inversion:1. Do users already have a platform that could host the delivery?2. Does that platform support the delivery mechanism natively?3. Can the intelligence be delivered without exposing IP?If all three hold, sketch the inverted architecture. If anyfail, explain which condition breaks and why. **What you are learning:** The Platform Inversion is the central insight of Part 5, but it is not magic. It works when three specific conditions are true. By testing those conditions against a different product, you develop the habit of questioning infrastructure assumptions before building. The most powerful optimization is sometimes elimination. ### Exercise 3: Evaluate Platform Dependency[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#exercise-3-evaluate-platform-dependency "Direct link to Exercise 3: Evaluate Platform Dependency") The Platform Inversion creates a dependency: TutorClaw only works if learners have OpenClaw. Use this prompt to think through the trade-offs of platform dependency: A product uses a Platform Inversion: instead of building itsown infrastructure, it delivers intelligence through a platformthe user already has. This eliminates 90% of costs and achieves99.5% gross margins.But it creates a dependency: the product only works on thatspecific platform. Evaluate the trade-offs:1. What happens if the platform changes its API or protocol?2. What happens if a competitor emerges on a different platform?3. What happens if the platform adds a competing product?4. Under what conditions is the dependency acceptable?5. What would you build as a contingency?Be specific about the risks and the mitigations. **What you are learning:** Every architecture decision creates dependencies. The Platform Inversion trades infrastructure cost for platform dependency. Understanding this trade-off is essential because it recurs in every platform-based product: iOS apps depend on Apple, browser extensions depend on Chrome, and MCP servers depend on the agent platform. The question is not whether to accept dependency, but whether the value gained justifies the coupling. * * * James was quiet for a while after Emma finished explaining the sixth pivot. He looked at his notes from the previous lessons: the OpenClaw moment, the SDK confusion, the scale wall, the NanoClaw insight, the Hybrid Resolution. Each pivot had built on the one before it. Each had preserved work from the previous stage. But this one felt different. "This is like something that happened at the warehouse," he said slowly. "We used to ship products to every retail location ourselves. Trucks, drivers, loading schedules, fuel costs. Our shipping cost was higher than our product margins on small orders. Then one of the regional managers asked: why are we shipping to stores that already have delivery trucks coming from the central distribution hub? The stores had their own receiving infrastructure. We did not need to build a parallel delivery network." He paused. "We stopped shipping. We provided the product to the hub. The stores picked it up on their existing routes. Our shipping cost dropped to almost nothing. The product was the same. The stores got it faster, actually, because their trucks ran daily and ours ran weekly." Emma leaned back. "That is closer to what happened here than most of the technical analogies I have used to explain it." She was quiet for a moment. "I should tell you something. Before this project, I worked on a different product. An internal analytics tool. We spent four months building infrastructure: Kubernetes cluster, message queues, database replication, the full stack. The tool had eleven users. Eleven. All of them had laptops with more compute power than our entire cluster." James looked at her. "You built infrastructure for eleven people who already had the infrastructure." "I did. And I did not question it because building infrastructure is what engineers do. It is the default. You learn to build servers, so you build servers. You learn to manage containers, so you manage containers. Nobody teaches you to ask whether the server needs to exist." She paused. "Pivot 6 happened because someone on this team asked the question I should have asked on that project: do we actually need to build this, or do our users already have it?" James nodded slowly. He looked at his notes from the six pivots. Each one had changed something fundamental: the platform, the SDK, the scale strategy, the architecture, the timeline approach, and finally, the entire assumption about who provides the infrastructure. But through all six, the TutorClaw he built in Chapter 58 still worked. The nine tools still taught learners. The shim skill still provided fallback. "The tools survived," he said. "Every pivot changed the infrastructure. None of them changed what the tools actually do." Emma stood up. "Six pivots. Some things changed with every pivot. Some things never changed at all. That is the next lesson." Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Dilemma: Three Goals, Three Timelines](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#the-dilemma-three-goals-three-timelines) * [Pivot 5: The Hybrid Resolution](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#pivot-5-the-hybrid-resolution) * [The Hidden Assumption](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#the-hidden-assumption) * [Pivot 6: The Platform Inversion](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#pivot-6-the-platform-inversion) * [The Economics of the Inversion](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#the-economics-of-the-inversion) * [What the Inversion Requires](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#what-the-inversion-requires) * [Update Your Architecture Decision Worksheet](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#update-your-architecture-decision-worksheet) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#try-with-ai) * [Exercise 1: Map Your Own Hybrid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#exercise-1-map-your-own-hybrid) * [Exercise 2: Invert an Infrastructure Assumption](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#exercise-2-invert-an-infrastructure-assumption) * [Exercise 3: Evaluate Platform Dependency](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#exercise-3-evaluate-platform-dependency) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/pivots-five-six#flashcards-study-aid) --- # When Things Go Wrong | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Sunday night. Agent deployed, unkillable, locked down. Ali checks the dashboard. The latest pricing report is empty. Not an error message, just empty. The agent ran on schedule, connected to the database, generated a report, and saved it. The report contains nothing. Board meeting at 9 AM. Twelve hours. Ali's first instinct: restart everything. The agent. The database. Maybe the whole server. > **"Every bad debugger has one move: restart. Every good debugger has a system."** Restarting might fix the symptom. But if the root cause is still there, the problem comes back: probably at 3 AM before the next board meeting. Ali needs to find the cause, not mask it. * * * The LNPS Method[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#the-lnps-method "Direct link to The LNPS Method") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When an agent fails, resist the urge to restart. Instead, follow four steps in order. Each step either finds the problem or eliminates a category of causes. | Step | Check | What You're Asking | Tools | | --- | --- | --- | --- | | **L**: Logs | Service logs | "What did the agent say happened?" | `journalctl -u service` | | **N**: Network | Connectivity | "Can the agent reach what it needs?" | `curl`, `ping`, `ss` | | **P**: Process | Process state | "Is the agent actually running? Is it stuck?" | `systemctl status`, `ps` | | **S**: System | Server resources | "Does the server have enough memory, disk, CPU?" | `df -h`, `free -h`, `top` | The order matters. Logs are the fastest path to the answer: the agent often tells you what went wrong. Network is next because connectivity failures are common and non-obvious. Process checks catch zombie or stuck services. System resources catch exhaustion problems. * * * Step L: Read the Logs[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-l-read-the-logs "Direct link to Step L: Read the Logs") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **What you tell Claude Code**: "Show me the competitor-tracker logs from the last 6 hours. Focus on errors and warnings." **What the agent does**: journalctl -u competitor-tracker --since "6 hours ago" --priority=warning **What you see**: Feb 28 02:15:33 server competitor-tracker[4821]: Starting competitor tracker agent...Feb 28 02:15:34 server competitor-tracker[4821]: Connected to databaseFeb 28 02:15:34 server competitor-tracker[4821]: Fetching pricing data...Feb 28 02:15:35 server competitor-tracker[4821]: WARNING: Query returned 0 rowsFeb 28 02:15:35 server competitor-tracker[4821]: Report generated: 0 entriesFeb 28 02:15:35 server competitor-tracker[4821]: Report saved to /opt/agents/competitor-tracker/data/report.csv No crashes. No errors. The agent connected, queried the database, got zero rows, and dutifully saved an empty report. The agent did exactly what it was told. The problem isn't the agent: the problem is upstream. **What you tell Claude Code**: "The agent got zero rows from the database. Check if the database service is running and if it has data." **What the agent does**: systemctl status postgresql **What you see**: ● postgresql.service - PostgreSQL RDBMS Active: inactive (dead) The database is not running. The agent connected (it's configured to retry), eventually got an empty result set, and reported it faithfully. The mystery is solved, but why is the database down? journalctl -u postgresql --since "12 hours ago" | tail -20 Feb 27 22:00:01 server systemd[1]: Stopping PostgreSQL RDBMS...Feb 27 22:00:03 server systemd[1]: postgresql.service: Deactivated successfully.Feb 28 03:15:00 server systemd[1]: postgresql.service not found in boot target The server rebooted for security updates at 10 PM. PostgreSQL restarted, but it was never **enabled** at boot. Ali enabled his agent with `systemctl enable competitor-tracker`. He never enabled the database. **The fix**: sudo systemctl enable postgresqlsudo systemctl start postgresql Pause. The root cause wasn't a code bug. It wasn't a network problem. It wasn't a crashed agent. It was an infrastructure oversight: the database wasn't configured to start on boot. Restarting the agent would have changed nothing. Reading the logs found the answer in under two minutes. * * * Step N: Check the Network[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-n-check-the-network "Direct link to Step N: Check the Network") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If the logs show connection errors instead of empty results, the problem is often network connectivity. The agent can't reach something it needs. **What you tell Claude Code**: "Check if the agent can reach the external pricing API at api.pricingdata.com on port 443." **What the agent does**: curl -I https://api.pricingdata.com/health **What you see if it works**: HTTP/2 200content-type: application/json **What you see if it fails**: curl: (7) Failed to connect to api.pricingdata.com port 443: Connection refused Connection refused means either the remote server is down or a firewall is blocking the connection. Connection timed out means the packets aren't reaching the destination at all. To check if the server can reach the internet generally: ping -c 3 8.8.8.8 If ping works but curl doesn't, the problem is specific to that service or port. If ping also fails, the server has no internet connectivity: check DNS and network configuration. * * * Step P: Check the Process[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-p-check-the-process "Direct link to Step P: Check the Process") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sometimes the agent appears to be running but isn't actually doing work. It might be stuck in an infinite loop, waiting for a resource that will never become available, or consuming all available CPU without producing output. **What you tell Claude Code**: "Is the competitor-tracker process actually running? How much CPU and memory is it using?" **What the agent does**: systemctl status competitor-trackerps aux | grep competitor-tracker **What to look for**: | Symptom | Likely Cause | | --- | --- | | Status: `active (running)` but CPU is 100% | Agent stuck in infinite loop | | Status: `active (running)` but CPU is 0% | Agent waiting/sleeping (might be normal) | | Status: `activating (auto-restart)` | Agent crashing and restarting repeatedly | | Status: `failed` | Agent crashed and didn't restart: check `Restart=` policy | * * * Step S: Check System Resources[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-s-check-system-resources "Direct link to Step S: Check System Resources") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ If logs, network, and process all look fine, the server itself might be running out of resources. **What you tell Claude Code**: "Check the server's disk space, memory, and CPU usage." **What the agent does**: df -h /free -huptime **What you see**: Filesystem Size Used Avail Use% Mounted on/dev/sda1 50G 48G 2.0G 96% / total used freeMem: 2.0Gi 1.8Gi 200Mi load average: 0.15, 0.10, 0.08 | Resource | Warning Sign | Impact | | --- | --- | --- | | Disk 96%+ full | `/var/log` fills up, services can't write | Agent can't save reports or logs | | Memory 90%+ used | OOM killer starts terminating processes | Agent gets randomly killed | | Load average > CPU cores | Server is overloaded | Everything runs slow | * * * The LNPS Method in Summary[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#the-lnps-method-in-summary "Direct link to The LNPS Method in Summary") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Print this. Tape it to your monitor. Use it every time. AGENT FAILURE TRIAGE━━━━━━━━━━━━━━━━━━━1. LOGS → journalctl -u --since "1 hour ago" What did the agent say happened?2. NETWORK → curl , ping Can the agent reach what it needs?3. PROCESS → systemctl status , ps aux Is the agent running? Is it stuck?4. SYSTEM → df -h, free -h, uptime Does the server have resources?━━━━━━━━━━━━━━━━━━━DO NOT RESTART UNTIL YOU KNOW THE CAUSE. * * * Ali's Resolution[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#alis-resolution "Direct link to Ali's Resolution") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Ali followed the LNPS method. Logs revealed the database returned zero rows. Checking the database process showed it was inactive. The database journal showed it wasn't enabled at boot. Two commands fixed it: `enable` and `start`. The database came back. The agent's next scheduled run produced a full pricing report. Ali reviewed the data, formatted the summary, and sent it to his client at 7 AM: two hours before the board meeting. The client never knew it was a close call. * * * Monday morning. The board meeting goes well. Ali's competitor-tracker runs on Dev's server. It survives reboots, restarts after crashes, runs under a dedicated user with locked-down permissions, and Ali knows how to diagnose it when things go wrong. He thinks: "What if I could do this from zero? Not three days of figuring things out, just sit down and deploy, following a checklist?" * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Group Errors by Type[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#prompt-1-group-errors-by-type "Direct link to Prompt 1: Group Errors by Type") Show me all the logs for my service from the last 24 hours.Group the errors by type — how many connection errors, how manytimeout errors, how many permission errors? Which type is mostcommon? What does the pattern tell us about the root cause? **What you're practicing:** Log analysis at scale. Individual error messages are data points. Patterns across many errors tell the real story. A dozen timeout errors at 3 AM points to a scheduled maintenance window, not a code bug. ### Prompt 2: Apply LNPS to a Different Scenario[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#prompt-2-apply-lnps-to-a-different-scenario "Direct link to Prompt 2: Apply LNPS to a Different Scenario") My web application is slow — pages take 10 seconds to load.Walk me through the LNPS method for this scenario. What wouldyou check at each step? What would the output look like fordifferent root causes (database slow, memory exhaustion, networklatency, application bug)? **What you're practicing:** Transferring the LNPS framework to a different problem type. The method works for any service failure, not just agent failures. Slowness is a failure mode too. ### Prompt 3: Why Not Restart First?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#prompt-3-why-not-restart-first "Direct link to Prompt 3: Why Not Restart First?") I've heard "have you tried turning it off and on again?" is theuniversal tech support answer. Why does the LNPS method say NOTto restart first? Give me a concrete example where restartinghides a serious problem that gets worse over time. **What you're practicing:** Understanding why systematic diagnosis matters. Restarting is tempting because it's fast. But speed without understanding creates recurring failures and erodes trust in the system. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The LNPS Method](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#the-lnps-method) * [Step L: Read the Logs](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-l-read-the-logs) * [Step N: Check the Network](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-n-check-the-network) * [Step P: Check the Process](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-p-check-the-process) * [Step S: Check System Resources](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#step-s-check-system-resources) * [The LNPS Method in Summary](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#the-lnps-method-in-summary) * [Ali's Resolution](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#alis-resolution) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#try-with-ai) * [Prompt 1: Group Errors by Type](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#prompt-1-group-errors-by-type) * [Prompt 2: Apply LNPS to a Different Scenario](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#prompt-2-apply-lnps-to-a-different-scenario) * [Prompt 3: Why Not Restart First?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#prompt-3-why-not-restart-first) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/when-things-go-wrong#flashcards-study-aid) --- # Capstone: Your File Processing Toolkit | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your accountant just called. They need a 1099-DIV from 2025. You have 300 files on your Desktop. Clock's ticking. Which workflow do you reach for? That's the real test. Not whether you know six workflows, but whether you know which one to grab when your files are on fire and someone is waiting. Different scenarios require different workflow orders. There's no single correct sequence. Before we build your permanent prompt toolkit, let's test your judgment. * * * Triage Under Pressure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#triage-under-pressure "Direct link to Triage Under Pressure") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ For each scenario, decide which workflow(s) to use and in what order BEFORE reading the insight. Then open Claude Code and try your approach. ### Scenario 1: The Urgent Search[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-1-the-urgent-search "Direct link to Scenario 1: The Urgent Search") Your Desktop has 300+ files. You need one specific tax document. Your accountant is waiting on the phone. You: I need to find a 1099-DIV document from 2025. It's somewhere onmy Desktop. I need it RIGHT NOW. **Before you run this**: think: Do you survey first (Lesson 1), or go straight to search (Lesson 6)? **Insight:** Urgency overrides the normal sequence. Search first, organize later. When someone is waiting, you don't map the territory: you find the one thing you need. The survey-first pattern is for when you have time to be systematic. ### Scenario 2: The Fresh Start[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-2-the-fresh-start "Direct link to Scenario 2: The Fresh Start") You want to reorganize your Desktop, but you've never looked at what's there. No backup exists. You: I want to completely reorganize my Desktop. What should I do first? **Before you run this**: think: What's your workflow and in what order? **Insight:** Survey → Backup → Organize. The order matters. If you skip the survey, your categories won't match your actual files. If you skip the backup, one wrong move and you're panicking. The safety-first pattern from Lesson 2 isn't optional; it's what makes everything else safe to attempt. ### Scenario 3: The Botched Script[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-3-the-botched-script "Direct link to Scenario 3: The Botched Script") You ran a batch rename script on your Desktop and now 40 files have garbled names. Some files seem missing entirely. You: I ran a rename script and it went wrong. Some files have garblednames and I think some are missing. What do I do? **Before you run this**: think: What do you do FIRST: try to fix the names, or something else? **Insight:** Compare against backup first. Don't start fixing individual files until you know the full extent of the damage. The recovery workflow from Lesson 5 starts with diagnosis (compare current state vs backup), not treatment. Files that "seem missing" might just be renamed: the backup comparison will tell you. ### Scenario 4: The Recurring Problem[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-4-the-recurring-problem "Direct link to Scenario 4: The Recurring Problem") You have 150 screenshots on your Desktop. You want them organized by month. You'll get 20+ more next week. You: I have 150 screenshots cluttering my Desktop. I want them organizedby month. But I'll get more screenshots next week. What's my best approach? **Before you run this**: think: One-time organization or script generation? **Insight:** Recurring problems need scripts, not one-time commands. The batch operations pattern from Lesson 4 taught you to ask for reusable scripts. If you just organize these 150 files manually, you'll be back here next month. Ask for a script that handles the current backlog AND future screenshots. * * * The Real Test: Your Desktop in 15 Minutes[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#the-real-test-your-desktop-in-15-minutes "Direct link to The Real Test: Your Desktop in 15 Minutes") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Close the book. Open Claude Code. Point it at a folder you haven't touched yet: your Desktop, your Documents, an old project folder. Set a timer for 15 minutes. Your goal: **survey it, back it up, organize it, and verify the results.** Use the workflows you've learned, in whatever order the situation demands. I need to get my [Desktop / Documents / other folder] under control.I have 15 minutes. Let's start with a survey so I know what I'mworking with, then create a backup of anything important, thenorganize what's left. Go. **What makes this different from the exercises:** Nobody is telling you which workflow to use or in what order. You have to assess the situation and choose. That's the skill, not knowing six workflows, but knowing which one to reach for. After 15 minutes, check your work: * Did you survey before organizing? (Or did urgency make you skip it?) * Did you back up before making changes? (Or did you forget?) * Did you verify the results? (Or did you assume success?) * What would you do differently with 15 more minutes? The answers reveal which patterns have become instinct and which still need practice. * * * Build Your Prompt Toolkit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#build-your-prompt-toolkit "Direct link to Build Your Prompt Toolkit") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now capture what you've learned in a permanent document. Help me create MY-PROMPT-TOOLKIT.md with fill-in-the-blank templatesfor each workflow I've practiced: survey, backup, organize, batch,recover, search, and verify. For each template, include the keyphrases that trigger the right agent behavior. This toolkit is the chapter's most important deliverable. The organized folders will get messy again. The scripts will need updates. But the _templates_ (the patterns for how you direct an agent) those are permanently useful. * * * The Seven Principles in Action[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#the-seven-principles-in-action "Direct link to The Seven Principles in Action") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've been learning the Seven Principles without memorizing them. Let's make explicit what emerged through practice. | Lesson | What You Did | Principle That Emerged | | --- | --- | --- | | 1\. Survey | Ran bash commands to analyze folder | **P1: Bash is the Key** | | 1\. Survey | Made chaos visible through reports | **P7: Observability** | | 2\. Safety First | Created backup before changes | **P6: Constraints and Safety** | | 2\. Safety First | Verified backup was complete | **P3: Verification as Core Step** | | 3\. Organization | Documented rules in rules.md | **P5: Persisting State in Files** | | 3\. Organization | Tested on one file first | **P4: Small, Reversible Decomp.** | | 4\. Batch Operations | Generated reusable script | **P2: Code as Universal Interface** | | 5\. Error Recovery | Restored from backup after mistake | **P3 + P6: Verify + Safety** | | 6\. Search & Discovery | Described problem, agent chose tools | **P1 + P2: Bash + Code** | | 6\. Search & Discovery | Created persistent index of results | **P5: Persisting State in Files** | All seven principles showed up naturally. You didn't study them from a textbook. You experienced them through action. And you saw them reinforce each other: safety enabled experimentation, verification caught errors, persistence made rules reusable. * * * Your Command Vocabulary[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#your-command-vocabulary "Direct link to Your Command Vocabulary") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Throughout this chapter, you observed the agent using these commands. You don't need to memorize them, but recognizing them helps you understand what the agent is doing. ### Core Commands[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#core-commands "Direct link to Core Commands") | Command | Plain English | Lesson | | --- | --- | --- | | `ls` | **List** files in a directory | 1, 3 | | `find` | **Find** files by name or date | 1, 2, 6 | | `wc -l` | **Word count** (count lines) | 1, 2 | | `du -sh` | **Disk usage** (human-readable sizes) | 1 | | `cp` | **Copy** files | 2, 5 | | `mv` | **Move** (or rename) files | 3, 4 | | `rm -rf` | **Remove** recursively (dangerous!) | 5 | | `mkdir` | **Make directory** | 2, 3 | | `mkdir -p` | **Make directory** (create parents too) | 4 | | `cat` | **Display** file contents | 3 | | `sort -rh` | **Sort** (reverse, human-readable) | 1 | | `head -10` | Show **first 10** lines | 4 | | `diff` | Show **differences** between files | 5 | | `cp -r` | **Copy** recursively (entire folders) | 5 | | `chmod` | **Change** file permissions | 5 | | `grep` | **Search** inside files | 6 | | `grep -l` | Search inside, show matching **files** | 6 | | `grep -i` | Search **case-insensitive** | 6 | ### Connectors[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#connectors "Direct link to Connectors") | Symbol | Plain English | Example | | --- | --- | --- | | `\|` (pipe) | "**then**": chain commands together | `find ... \| wc -l` = "find files, then count them" | | `xargs` | "**for each**": converts text to arguments | `find ... \| xargs grep` = "find files, then search inside each" | ### Flags Worth Knowing[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#flags-worth-knowing "Direct link to Flags Worth Knowing") | Flag | Meaning | Example | | --- | --- | --- | | `-l` | Show as list (ls) or list files only (grep) | `grep -l "pattern"` | | `-i` | Case-insensitive | `find -iname "*.PDF"` matches .pdf too | | `-r` | Reverse order (or recursive) | `sort -r` | | `-h` | Human-readable sizes (KB, MB, GB) | `du -h` | | `-p` | Create parent directories | `mkdir -p a/b/c` | You don't need to memorize syntax. You need to recognize patterns. When you see the agent use these commands, you'll know what it's doing, and you can verify it's doing the right thing. * * * Reflection Questions[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#reflection-questions "Direct link to Reflection Questions") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before moving on, consider these questions: **1\. Which workflow will you use most often?** Everyone's answer is different. Some people struggle with cluttered desktops. Others need to batch rename screenshots weekly. Which pattern solves your recurring problem? **2\. What would you add to your toolkit?** You might need templates for: * Finding duplicate files * Archiving old projects * Cleaning up specific file types (old logs, cache files) Think about what's missing for your specific needs. **3\. Where did you observe each principle?** Look back at the principles table. Can you point to specific moments when you saw the agent apply that principle? The more concrete your memory, the more the patterns will stick. * * * From Manual to Automated[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#from-manual-to-automated "Direct link to From Manual to Automated") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Everything you did in this chapter was manual. You opened Claude Code, typed prompts, approved actions. You were the trigger. When you learn automation, you'll build AI Employees that do this automatically: | This Chapter (Manual) | Automated Workflow | | --- | --- | | You type "survey my folder" | Agent watches folder for changes | | You decide when to organize | Agent organizes on schedule | | You approve each batch | Agent follows pre-approved rules | | You verify results | Agent reports results to you | Your `rules.md` becomes the AI Employee's decision rules. Your verification patterns become its supervision methods. Everything you learned transfers. Here's a concrete glimpse of what that looks like. Your manual `rules.md` feeds directly into an automated workflow: # file_organizer_agent.py — what your manual workflows becomeimport yamlfrom pathlib import Pathfrom datetime import datetime# YOUR rules.md becomes the agent's decision logicrules = yaml.safe_load(open("rules.md"))# YOUR backup pattern becomes an automatic safety stepdef organize_new_files(watch_folder: Path): new_files = detect_new_files(watch_folder) # Survey (Lesson 1) create_timestamped_backup(new_files) # Safety (Lesson 2) for file in new_files: category = apply_rules(file, rules) # Rules (Lesson 3) move_with_logging(file, category) # Batch (Lesson 4) verify_all_files_accounted_for(new_files, rules) # Verify (Lesson 5) send_summary_report() # Observability (P7)# YOUR manual prompt → automated triggerschedule.every(1).hour.do(organize_new_files, Path("~/Downloads")) Every function in that script maps to a lesson you completed. `create_timestamped_backup` is Lesson 2. `apply_rules` is Lesson 3. `verify_all_files_accounted_for` is Lesson 5. The workflows are identical. Automation just removes _you_ as the trigger. The gap between "I type a prompt" and "it runs automatically" is smaller than you think. It's not a different skill; it's the same skill with a scheduler attached. * * * ✅ Final Checkpoint: Chapter Deliverables[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#-final-checkpoint-chapter-deliverables "Direct link to ✅ Final Checkpoint: Chapter Deliverables") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By completing the checkpoints throughout this chapter, you should now have: | Item | Location | Status | | --- | --- | --- | | `FILE-INVENTORY.md` | In `file-organizer/` | ✅ Lesson 1 checkpoint | | `backup/` | With timestamped folder | ✅ Lesson 2 checkpoint | | `rules.md` | With edge cases added | ✅ Lesson 3 checkpoint | | `ORGANIZER-LOG.md` | Full history | ✅ Lesson 3 checkpoint | | `organized/` | Files categorized | ✅ Lesson 3 checkpoint | | Recovery exercise | Completed | ✅ Lesson 5 checkpoint | | `MY-PROMPT-TOOLKIT.md` | Your prompt templates | ✅ This lesson | If you're missing any items, go back to the relevant lesson and complete the checkpoint. The toolkit is the most important deliverable; it's what you'll use long after this chapter is done. * * * What You've Accomplished[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#what-youve-accomplished "Direct link to What You've Accomplished") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you started this chapter, you had a messy Downloads folder and no systematic way to handle it. Now you have: * **A methodology**: Survey, backup, design rules, batch execute, recover, verify * **Reusable tools**: Scripts and templates you can adapt * **Pattern recognition**: You see the Seven Principles when agents work * **Recovery confidence**: You know how to fix mistakes, not just avoid them * **A permanent toolkit**: Prompt templates that work on any folder, any time This isn't just about files. The patterns you learned apply to any domain where you direct an AI agent. Describe the problem, establish safety, document rules, test small, scale up, recover from errors, verify. * * * Try With AI: Extended Practice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#try-with-ai-extended-practice "Direct link to Try With AI: Extended Practice") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Prompt 1: Domain Transfer** I've learned file organization workflows in this chapter. Help me applythe same patterns to a different domain: my email inbox. What wouldthe equivalent of "survey," "backup," "rules," "batch," and "verify"look like for email management? **What you're practicing:** Abstraction. The workflows aren't about files. They're about systematic problem-solving. You're learning to see the pattern beneath the specific application. **Prompt 2: Toolkit Expansion** My prompt toolkit has the six core workflows. What other file-relatedprompts would be useful to add? Think about tasks like finding duplicates,archiving old projects, cleaning up cache files, or managing downloadsover time. **What you're practicing:** Anticipation. Good toolkits grow with your needs. You're learning to think ahead about what patterns you'll need. **Prompt 3: Principle Identification** I'm about to reorganize my Photos folder. Before I start, help meplan which of the Seven Principles I should apply at each step.For each principle, tell me specifically what I should do or ask for. **What you're practicing:** Explicit principle application. By planning with principles in mind, you internalize them more deeply. Eventually this becomes automatic. * * * Conclusion[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#conclusion "Direct link to Conclusion") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's what changed between Lesson 1 and now. In Lesson 1, you asked the agent to analyze your Downloads folder. You watched it work. You were impressed that it ran six commands in 30 seconds. Now? You wouldn't just _watch_. You'd check whether it backed up first. You'd question its categorization rules. You'd ask for a preview before batch operations. You'd verify the results against a known state. You'd ask for a script, not a one-time fix. That shift (from passive observer to active director) is the real deliverable. Not the organized folder. Not the scripts. Not the toolkit document. The skill of _knowing what to demand from an agent_ before trusting its output. The agent is ephemeral. Your conversation ends, context resets, the next session starts fresh. But the patterns you've internalized: survey before acting, backup before changing, verify before trusting, script before repeating: those persist in _you_. They transfer to email management, project organization, data pipelines, and every domain where you direct AI agents. Your Downloads folder is organized. Your toolkit is built. You're ready for automation. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 🔄 Session Management Note[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#-session-management-note "Direct link to 🔄 Session Management Note") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've completed the entire chapter. Commit everything and start a completely fresh session before moving to the next chapter. **Why this matters:** Seven lessons of file processing generated a lot of context: surveys, organization rules, batch scripts, recovery exercises, search patterns, and this capstone. Carrying all of that into the next chapter would be the Kitchen Sink pattern from the Seven Principles chapter. Your deliverables (`MY-PROMPT-TOOLKIT.md`, `rules.md`, scripts) are all saved in files. The next chapter starts clean. Checking access... * [Triage Under Pressure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#triage-under-pressure) * [Scenario 1: The Urgent Search](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-1-the-urgent-search) * [Scenario 2: The Fresh Start](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-2-the-fresh-start) * [Scenario 3: The Botched Script](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-3-the-botched-script) * [Scenario 4: The Recurring Problem](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#scenario-4-the-recurring-problem) * [The Real Test: Your Desktop in 15 Minutes](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#the-real-test-your-desktop-in-15-minutes) * [Build Your Prompt Toolkit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#build-your-prompt-toolkit) * [The Seven Principles in Action](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#the-seven-principles-in-action) * [Your Command Vocabulary](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#your-command-vocabulary) * [Core Commands](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#core-commands) * [Connectors](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#connectors) * [Flags Worth Knowing](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#flags-worth-knowing) * [Reflection Questions](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#reflection-questions) * [From Manual to Automated](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#from-manual-to-automated) * [✅ Final Checkpoint: Chapter Deliverables](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#-final-checkpoint-chapter-deliverables) * [What You've Accomplished](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#what-youve-accomplished) * [Try With AI: Extended Practice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#try-with-ai-extended-practice) * [Conclusion](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#conclusion) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#flashcards-study-aid) * [🔄 Session Management Note](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/capstone-your-file-toolkit#-session-management-note) --- # Batch Operations Workflow | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've got 87 screenshots. Renaming them one by one would take 43 minutes. Renaming them with a script takes 30 seconds. But the real difference isn't speed. You asked the agent to rename your screenshots. It gave you renamed screenshots AND a script called `rename-screenshots.sh`. You didn't ask for the script. The agent anticipated that you'd need to do this again next month and solved beyond your request. That's the pattern this lesson teaches: don't just solve the problem: solve the _category_ of problems. Your screenshots are named things like `Screenshot 2024-01-15 at 10.23.45.png`. The auto-generated names made sense to your operating system. They make no sense to you. Let's fix that, and make sure the fix is permanent. * * * The Conversation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#the-conversation "Direct link to The Conversation") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open Claude Code. Describe the problem. You: I have lots of screenshots with ugly names like 'Screenshot 2024-01-15at 10.23.45.png'. Help me rename them to something organized like'screenshot-001.png', 'screenshot-002.png', etc. Watch what Claude Code does. Claude Code: I'll analyze your screenshots and prepare a rename plan.Let me first see what we're working with.[Runs: ls ~/Screenshots/*.png | head -10][Runs: ls ~/Screenshots/*.png | wc -l]I found 87 screenshots. Before I rename anything, here's what I'll do:| Current Name | New Name ||-------------------------------------------|--------------------|| Screenshot 2024-01-15 at 10.23.45.png | screenshot-001.png || Screenshot 2024-01-15 at 14.30.22.png | screenshot-002.png || Screenshot 2024-02-03 at 09.15.11.png | screenshot-003.png || Screenshot 2024-02-14 at 16.42.08.png | screenshot-004.png || ... (83 more files) | ... |The files will be numbered in chronological order based ontheir creation date.Should I proceed with this plan, or would you like a differentnaming pattern? You look at the preview. Something's wrong. The agent threw away useful information: the dates. `screenshot-001.png` tells you nothing about _when_ the screenshot was taken. Six months from now, you'll have no idea what `screenshot-047.png` contains or why you took it. You: Actually, include the date. Make it 'screenshot-2024-01-15-001.png'so I can see when each screenshot was taken. Claude Code: Good thinking. Keeping the date makes files easier to find later.Here's the updated plan:| Current Name | New Name ||-------------------------------------------|-------------------------------|| Screenshot 2024-01-15 at 10.23.45.png | screenshot-2024-01-15-001.png || Screenshot 2024-01-15 at 14.30.22.png | screenshot-2024-01-15-002.png || Screenshot 2024-02-03 at 09.15.11.png | screenshot-2024-02-03-001.png || Screenshot 2024-02-14 at 16.42.08.png | screenshot-2024-02-14-001.png || ... (83 more files) | ... |Notice that files from the same day get sequential numbers withinthat day (001, 002, etc.).I'll also create a reusable script so you can run this patternon new screenshots later.Proceed? You're satisfied. You give the go-ahead. You: Yes, do it. Claude Code:[Runs: mkdir -p ~/Screenshots/renamed][Runs: # Rename loop with logging...]Done. Here's what happened:SUMMARY- Files renamed: 87- New location: ~/Screenshots/renamed/- Script created: rename-screenshots.shCHANGE LOG (first 10): Screenshot 2024-01-15 at 10.23.45.png → screenshot-2024-01-15-001.png Screenshot 2024-01-15 at 14.30.22.png → screenshot-2024-01-15-002.png Screenshot 2024-02-03 at 09.15.11.png → screenshot-2024-02-03-001.png ...The full log is saved in rename-log.txt.To rename future screenshots, run: ./rename-screenshots.sh ~/Screenshots 87 files renamed. A reusable script created. A log of every change. You did this by having a conversation. * * * When Batch Operations Go Sideways[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#when-batch-operations-go-sideways "Direct link to When Batch Operations Go Sideways") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The smooth version above is what happens when everything works. Here's what happens when it doesn't. ### Scenario: Naming Collision[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#scenario-naming-collision "Direct link to Scenario: Naming Collision") You run the script and the agent reports: Claude Code: Warning: Two files would both become"screenshot-2024-03-15-001.png" because they werecreated within the same second.Conflicting files: Screenshot 2024-03-15 at 09.00.00.png (from Downloads) Screenshot 2024-03-15 at 09.00.00.png (from Desktop)Options:1. Add time to the name: screenshot-2024-03-15-0900-001.png2. Add a suffix: screenshot-2024-03-15-001a.png and 001b.png3. Skip the conflict and let me handle it manuallyWhich approach? Why this matters: Batch operations that assume unique names will silently overwrite files. Good agents detect conflicts before they cause data loss. When you see this pattern, it means the agent detected a potential data loss scenario before it happened. ### Scenario: Partial Failure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#scenario-partial-failure "Direct link to Scenario: Partial Failure") The script processes 80 of 87 files, then: Claude Code: Error on file 81: "Screenshot 2024-04-01 at ?.??.??.png"The filename contains question marks that can't be parsed.Progress: 80 of 87 renamed successfully.Failed: 7 files with non-standard characters in filenames.The 80 successful renames are in ~/Screenshots/renamed/.The 7 failed files are still in the original location.Want me to show you the failed files so we can handle them individually? What you should do: Don't panic. The agent left failed files untouched (the originals are safe). Handle the exceptions individually. Then update the script to handle unusual characters in future runs. This is why the preview step matters. If you'd caught the question-mark filenames in the preview, you could have handled them before the batch run. ### Don't Fix the File: Fix the Rule[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#dont-fix-the-file-fix-the-rule "Direct link to Don't Fix the File: Fix the Rule") This is the central lesson of batch operations. When something goes wrong, resist the urge to fix the affected files and move on. Fix the _rule_ that caused the error. The rename script choked on filenames with question marks? Don't rename those 7 files manually: update `rename-screenshots.sh` to handle special characters, then rerun it on the failed files. The naming collision? Don't move two files by hand: update the script to detect same-second conflicts. Every edge case you fix in the script is an edge case that never bites you again. Every error makes the system smarter. This is how one-time solutions evolve into robust automation. The agent didn't just rename files. It created a _script_. That's the difference between solving a problem once and solving a category of problems. Next month when you have 50 new screenshots, you run `./rename-screenshots.sh` and they're all named consistently. * * * The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#the-pattern "Direct link to The Pattern") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's what made this work: "Show me what you'll do before doing it." This single request triggers the preview-before-action pattern. The agent will: 1. Analyze the current state 2. Generate a proposed plan 3. Show you the plan 4. Wait for approval And the second key request: "Create a script I can reuse for this." This triggers script generation instead of one-time execution. The agent produces: 1. A working script file 2. Instructions for running it 3. Built-in logging Together, these patterns transform tedious manual work into reviewed, logged, reusable automation. * * * Why Scripts Matter[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#why-scripts-matter "Direct link to Why Scripts Matter") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider the difference: | Approach | What You Get | Next Time | | --- | --- | --- | | One-time commands | Files renamed | Start over from scratch | | Script generation | Files renamed + `rename-screenshots.sh` | Run one command | The first approach solves the problem. The second approach solves the _category_ of problems. Code becomes the interface between your intent and recurring tasks. The script also serves as documentation. Six months from now, you might forget exactly how you wanted screenshots named. Open `rename-screenshots.sh` and the logic is right there, preserved in code. * * * The Key Requests[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#the-key-requests "Direct link to The Key Requests") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Remember these phrases for batch operations: | What You Want | What to Say | | --- | --- | | See the plan first | "Show me what you'll do before doing it" | | Get reusable code | "Create a script I can reuse for this" | | Keep a record | "Log every change you make" | | Change the plan | "Actually, make it \[new pattern\] instead" | | Limit scope | "Just do the first 5 files to test" | You're not memorizing bash syntax. You're learning how to direct an agent effectively. * * * What You're Building[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#what-youre-building "Direct link to What You're Building") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By now in this chapter, you've learned: | Lesson | Pattern | | --- | --- | | 1\. First Workflow | "Help me understand" | | 2\. Safety First | "Back up before changing" | | 3\. Categorize with Rules | "Write rules first" | | **4\. Batch Operations** | **"Show me first, create a script"** | Each pattern builds on the last. You understand your files. You protect them with backups. You document your rules. Now you transform repetitive tasks into reusable automation. Your scripts are powerful. They rename 87 files in seconds. But what happens when a script does the WRONG thing to 87 files in seconds? Speed amplifies mistakes exactly as much as it amplifies productivity. You need a recovery workflow that works as fast as the scripts that caused the problem. * * * Try With AI: Extended Practice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#try-with-ai-extended-practice "Direct link to Try With AI: Extended Practice") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Prompt 1: Complex Naming Patterns** I want to rename my screenshots but the pattern is more complex:- Group by month into folders (2024-01/, 2024-02/, etc.)- Within each folder, name files screenshot-001.png, screenshot-002.png- Show me the folder structure and file mapping before doing anythingCan you handle this multi-step organization? **What you're practicing:** Compound requests. You're asking the agent to both reorganize (create folders) and rename (change filenames). Watch how it breaks down the complex task. **Prompt 2: Dry Run Mode** I want to see exactly what commands you would run to rename these files,but don't actually run them yet. Show me the bash commands you'd executeso I can understand what's happening. **What you're practicing:** Understanding the mechanics. By asking to see the commands without execution, you learn what's happening "under the hood" while staying in control. **Prompt 3: Content-Aware Operations** I have 200 PDF invoices named "Invoice_12345.pdf". I want to rename themto include the client name, but the client name is INSIDE the PDF, not inthe filename. Can you extract it and use it in the rename? **What you're practicing:** Content-aware batch operations. You're moving beyond filename-based operations to using file contents as input. This is how real automation handles documents where the metadata you need is inside the file, not in its name. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Conversation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#the-conversation) * [When Batch Operations Go Sideways](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#when-batch-operations-go-sideways) * [Scenario: Naming Collision](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#scenario-naming-collision) * [Scenario: Partial Failure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#scenario-partial-failure) * [Don't Fix the File: Fix the Rule](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#dont-fix-the-file-fix-the-rule) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#the-pattern) * [Why Scripts Matter](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#why-scripts-matter) * [The Key Requests](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#the-key-requests) * [What You're Building](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#what-youre-building) * [Try With AI: Extended Practice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#try-with-ai-extended-practice) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/batch-operations-workflow#flashcards-study-aid) --- # Capstone: Zero to Production | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Monday morning. The board meeting went well. Ali's competitor-tracker runs on Dev's server. It survives reboots, restarts after crashes, runs under a dedicated user with locked-down permissions. Ali's client is impressed. "Can you build a second agent? One that monitors our social media mentions and generates a daily sentiment report?" Ali says yes. Then he thinks about the last three days. The SSH confusion. The scattered files. The process that died when he closed his laptop. The security scare. The empty report at 2 AM. Three days of discovery, trial, error, and panic to deploy one agent. What if he could do it in thirty minutes? * * * The Insight[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-insight "Direct link to The Insight") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali learned six things in six lessons: | Lesson | What he learned | | --- | --- | | L1 | Where things live on a Linux server | | L2 | How to read what the server tells him | | L3 | How to organize an agent's files | | L4 | How to make an agent survive reboots | | L5 | How to lock down access | | L6 | How to diagnose failures | The first time, he learned these through experience: one crisis at a time. The second time, he can write them down as a plan and execute them in order. > **"The spec is the checklist, the executor, and the auditor. All in one file."** * * * The Deployment Spec[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-deployment-spec "Direct link to The Deployment Spec") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A deployment spec is not documentation. It's an execution guide. You write it before you start, then feed it to Claude Code section by section. Each section corresponds to a lesson you've already learned. Here's the spec Ali writes for his second agent: the social media sentiment tracker. **What you tell Claude Code**: "I'm going to write a deployment spec. Help me create a file called DEPLOYMENT-SPEC.md with the following structure." # DEPLOYMENT-SPEC.md — Social Media Sentiment Tracker## 1. Server Access- Target: Dev's server (devs-server.com)- SSH: Key-based authentication as ali- Verify: Can connect and run commands## 2. Directory Structure- Base: /opt/agents/sentiment-tracker/- Subdirectories: src/, config/, logs/, data/- Environment: .env with API keys (Twitter API, OpenAI API)- Permissions: Owned by agentuser, chmod 750, .env chmod 600## 3. Application Setup- Copy source code to src/- Install dependencies (pip install -r requirements.txt)- Create .env with required variables: - TWITTER_API_KEY - TWITTER_API_SECRET - OPENAI_API_KEY - DB_CONNECTION_STRING- Test: Run manually once, verify output## 4. Service Configuration- Create systemd unit: /etc/systemd/system/sentiment-tracker.service- User: agentuser- WorkingDirectory: /opt/agents/sentiment-tracker- EnvironmentFile: /opt/agents/sentiment-tracker/.env- Restart: on-failure- RestartSec: 5- MemoryMax: 256M- Enable and start## 5. Security Checklist- [ ] Service runs as agentuser, NOT root- [ ] .env permissions are 600- [ ] SSH password authentication is disabled- [ ] No secrets in source code (grep for API_KEY, PASSWORD, SECRET)## 6. Verification- [ ] Service status: active (running)- [ ] Survives terminal closure: close SSH, reconnect, check status- [ ] Survives reboot: sudo reboot, reconnect, check status- [ ] Logs are writing: journalctl -u sentiment-tracker --since "5 min ago"- [ ] Output is correct: check data/latest-report.csv has content- [ ] Resource usage is reasonable: memory under 256M * * * Executing the Spec[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#executing-the-spec "Direct link to Executing the Spec") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now Ali feeds the spec to Claude Code, one section at a time. Each section is a conversation. ### Section 1: Server Access[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-1-server-access "Direct link to Section 1: Server Access") **What you tell Claude Code**: "I need to deploy a new agent to devs-server.com. Verify I can connect via SSH key authentication." Claude connects, confirms access. One minute. ### Section 2: Directory Structure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-2-directory-structure "Direct link to Section 2: Directory Structure") **What you tell Claude Code**: "Create the directory structure for a new agent called sentiment-tracker at /opt/agents/sentiment-tracker/ with src, config, logs, and data subdirectories. Set ownership to agentuser and permissions to 750. Create a .env file with these variables: TWITTER\_API\_KEY, TWITTER\_API\_SECRET, OPENAI\_API\_KEY, DB\_CONNECTION\_STRING." Claude creates the structure, sets permissions, creates the `.env` template. Three minutes. ### Section 3: Application Setup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-3-application-setup "Direct link to Section 3: Application Setup") **What you tell Claude Code**: "Copy the sentiment tracker source code from my local machine to the server. Install the Python dependencies. Then run the agent manually once to verify it produces output." Claude transfers files, installs dependencies, runs a test. The agent produces a sentiment report. Five minutes. ### Section 4: Service Configuration[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-4-service-configuration "Direct link to Section 4: Service Configuration") **What you tell Claude Code**: "Create a systemd service for the sentiment-tracker. Use the same pattern as the competitor-tracker: run as agentuser, restart on failure, 256 MB memory limit. Enable it and start it." Claude creates the unit file, enables the service, starts it. Two minutes. ### Section 5: Security Checklist[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-5-security-checklist "Direct link to Section 5: Security Checklist") **What you tell Claude Code**: "Run a security audit on the sentiment-tracker deployment. Check that it doesn't run as root, the .env has correct permissions, and there are no secrets hardcoded in the source code." Claude checks each item. Reports all clear. Two minutes. ### Section 6: Verification[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-6-verification "Direct link to Section 6: Verification") **What you tell Claude Code**: "Run the full verification checklist. Check the service is running, check it's writing logs, check the output file has content, and check memory usage." Claude runs each check, reports results. Two minutes. * * * The Reveal[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-reveal "Direct link to The Reveal") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali looks at the clock. Fifteen minutes from first SSH connection to fully deployed, verified, production-ready agent. The competitor-tracker took three days. Confusion, trial and error, a 2 AM panic, a security scare, and a methodical debugging session. The sentiment-tracker took fifteen minutes. Same server. Same complexity. Different approach. The difference wasn't skill: Ali has the same skill level for both. The difference was **structure**. The deployment spec captured everything he learned and turned scattered knowledge into sequential execution. Three days of discovery. Fifteen minutes with a spec. * * * Write Your Own Deployment Spec[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#write-your-own-deployment-spec "Direct link to Write Your Own Deployment Spec") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The real exercise in this capstone is not following Ali's spec. It's writing your own. **What you tell Claude Code**: "Help me write a DEPLOYMENT-SPEC.md for deploying \[your agent\] to a Linux server. Use the six-section structure: Server Access, Directory Structure, Application Setup, Service Configuration, Security Checklist, and Verification." Replace `[your agent]` with whatever you're building: a web scraper, a data pipeline, a chatbot, a monitoring tool. The structure is the same for any agent. ### The Six Sections (Your Template)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-six-sections-your-template "Direct link to The Six Sections (Your Template)") | Section | Question it answers | Lesson it comes from | | --- | --- | --- | | 1\. Server Access | How do I get in? | L1 (Where Your Agent Lives) | | 2\. Directory Structure | Where do files go? | L3 (Setting Up Your Agent's Home) | | 3\. Application Setup | How do I install and test? | L2, L3 (Reading Output, Organization) | | 4\. Service Configuration | How does it stay alive? | L4 (Making Your Agent Unkillable) | | 5\. Security Checklist | Is it locked down? | L5 (Locking the Door) | | 6\. Verification | Does it actually work? | L6 (When Things Go Wrong) | Every lesson in this chapter feeds into one section of the deployment spec. This is why the order mattered. This is why the story followed the sequence it did. * * * The Verification Habit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-verification-habit "Direct link to The Verification Habit") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The most important section is the last one. Verification is not optional. Deployed services that aren't verified are time bombs. They appear to work until the moment they don't: usually when a client checks the dashboard, a board meeting starts, or you're asleep. Build the habit now: **no deployment is done until every verification check passes.** DEPLOYMENT VERIFICATION━━━━━━━━━━━━━━━━━━━━━━□ Service status: active (running)□ Survives terminal closure□ Survives reboot□ Logs are writing□ Output is correct□ Resource usage is within limits□ Security checklist passedALL BOXES CHECKED → DEPLOYMENT COMPLETEANY BOX UNCHECKED → NOT DONE YET * * * Ali's Story: Resolution[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#alis-story-resolution "Direct link to Ali's Story: Resolution") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali has two agents running on Dev's server. The competitor-tracker he deployed over three painful days. The sentiment-tracker he deployed in fifteen minutes. Both run as systemd services under dedicated users with locked-down permissions. Both survive reboots. Both write persistent logs. Both can be diagnosed with the LNPS method. Ali learned Linux not by memorizing fifty commands. He learned it by solving real problems: one at a time, and capturing the solutions in a spec that makes the next deployment effortless. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Add Security Depth[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#prompt-1-add-security-depth "Direct link to Prompt 1: Add Security Depth") My deployment spec has a basic security checklist. Extend it foran agent that handles financial data — customer payment amounts,invoice numbers, bank account references. What additional securitymeasures should the spec include? Think about encryption at rest,audit logging, and data retention policies. **What you're practicing:** Adapting the deployment spec for different risk profiles. A competitor-tracker and a payment processor have different security requirements. The spec structure is the same: the contents change based on what's at stake. ### Prompt 2: Spec for a Different Agent[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#prompt-2-spec-for-a-different-agent "Direct link to Prompt 2: Spec for a Different Agent") I want to deploy a web scraping agent that runs every 6 hours,downloads product listings from 5 competitor websites, and storesthe results in a PostgreSQL database. Write a completeDEPLOYMENT-SPEC.md for this agent. Include all six sections. **What you're practicing:** Writing deployment specs from scratch. The first spec you follow. The second spec you write. The third spec you don't even think about: the structure is automatic. ### Prompt 3: systemd vs Docker[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#prompt-3-systemd-vs-docker "Direct link to Prompt 3: systemd vs Docker") This chapter used systemd to deploy agents. I've also heard aboutDocker containers. Compare systemd services and Docker containersfor agent deployment. When would you use each? What are thetradeoffs in complexity, isolation, and portability? Which wouldyou recommend for a solo developer deploying 2-3 agents? **What you're practicing:** Understanding the landscape of deployment options. systemd is the foundation: it works everywhere Linux runs with zero additional tooling. Docker adds isolation and portability at the cost of complexity. Knowing when each tool is appropriate is an architectural decision. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Insight](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-insight) * [The Deployment Spec](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-deployment-spec) * [Executing the Spec](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#executing-the-spec) * [Section 1: Server Access](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-1-server-access) * [Section 2: Directory Structure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-2-directory-structure) * [Section 3: Application Setup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-3-application-setup) * [Section 4: Service Configuration](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-4-service-configuration) * [Section 5: Security Checklist](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-5-security-checklist) * [Section 6: Verification](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#section-6-verification) * [The Reveal](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-reveal) * [Write Your Own Deployment Spec](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#write-your-own-deployment-spec) * [The Six Sections (Your Template)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-six-sections-your-template) * [The Verification Habit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#the-verification-habit) * [Ali's Story: Resolution](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#alis-story-resolution) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#try-with-ai) * [Prompt 1: Add Security Depth](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#prompt-1-add-security-depth) * [Prompt 2: Spec for a Different Agent](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#prompt-2-spec-for-a-different-agent) * [Prompt 3: systemd vs Docker](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#prompt-3-systemd-vs-docker) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/capstone-zero-to-production#flashcards-study-aid) --- # Error Recovery & Resilience | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C We're about to do something most tutorials would never let you do. You're going to deliberately destroy the organization you just built. On purpose. With your real files. Look at your `organized/` folder. That structure took you the last three lessons to build: survey, backup, categorization rules, batch moves. You're about to flatten it back to chaos in 5 seconds. The goal isn't to get good at fixing mistakes. The goal is to stop being afraid of making them. Fire drills don't teach you to fight fires. They teach you to stop freezing when the alarm goes off. By the end of this lesson, the word "oops" will be followed by a recovery workflow, not a panic response. * * * Exercise: The Deliberate Mistake[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#exercise-the-deliberate-mistake "Direct link to Exercise: The Deliberate Mistake") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You're going to intentionally run a bad organization command and then recover from it. This is safe because you have a verified backup from Lesson 2. ### Step 1: Confirm Your Safety Net[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-1-confirm-your-safety-net "Direct link to Step 1: Confirm Your Safety Net") First, verify that the backup you created in Lesson 2 still exists and is complete: You: Check my file-organizer/backup/ folder. Is my backup from Lesson 2still there? How many files does it contain? Compare it against what'scurrently in file-organizer/organized/. Claude Code:[Runs: ls file-organizer/backup/][Runs: find file-organizer/backup/ -type f | wc -l][Runs: find file-organizer/organized/ -type f | wc -l]Your backup contains 136 files across 5 categories.Your organized/ folder currently has 486 files.The backup covers your important files (PDFs and recent files).Safety net confirmed. ### Step 2: Run a Deliberately Wrong Reorganization[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-2-run-a-deliberately-wrong-reorganization "Direct link to Step 2: Run a Deliberately Wrong Reorganization") Ready? Look at your organized/ folder one more time. Five categories. 486 files, each in its place. You built that. You're about to destroy it in one command. You: Take everything in file-organizer/organized/ and merge all thecategory folders back into a single flat folder. Dump all files fromdocuments/, spreadsheets/, images/, code/, and misc/ into organized/directly. Remove the empty category folders. Claude Code:[Runs: mv file-organizer/organized/documents/* file-organizer/organized/][Runs: mv file-organizer/organized/spreadsheets/* file-organizer/organized/][Runs: mv file-organizer/organized/images/* file-organizer/organized/]... (moves all files, removes empty folders)Done. All 486 files are now in a single flat folder.The category structure from Lesson 3 is gone. ### Step 3: Watch the Structure Collapse[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-3-watch-the-structure-collapse "Direct link to Step 3: Watch the Structure Collapse") Check what happened to the organization you built in Lesson 3: You: Show me what file-organizer/organized/ looks like now.Is there any way to tell which files were documents vs imagesvs spreadsheets? Claude Code:[Runs: ls file-organizer/organized/ | head -20]organized/ now contains 486 files in a single flat list.report.pdf sits next to photo.jpg sits next to budget.xlsx.There's no way to tell which category they belonged to —the folder structure was the only metadata, and it's gone. ### Step 4: Discover the Problem[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-4-discover-the-problem "Direct link to Step 4: Discover the Problem") The damage is done. Now diagnose it: You: Compare what's in file-organizer/organized/ now against mybackup. Show me what I lost and what changed. Claude Code:[Runs: ls file-organizer/organized/ | wc -l][Runs: ls -d file-organizer/organized/*/ 2>/dev/null]organized/ now has 486 files in a flat list.No subdirectories remain — all category folders are gone.The files themselves are intact, but the STRUCTURE is destroyed.Your rules.md still describes the categories, but the foldersthat implemented those categories no longer exist. The "mistake" here is real: you lost the entire organization from Lesson 3. In real workflows, this kind of destructive flatten could also overwrite files with the same name across categories, break relative paths, or mix incompatible file types. ### Step 5: Recover from Backup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-5-recover-from-backup "Direct link to Step 5: Recover from Backup") Here's the payoff. You have a backup. But more importantly, you have `rules.md`. Recovery means re-applying your documented rules: You: That reorganization was wrong. I want to restore my organized/folder structure. Use the backup to recover any files that were lost,and re-apply the rules from rules.md to recreate the category folders. Claude Code: Restoring organization...[Runs: # Recreates category folders from rules.md][Runs: # Re-sorts files by extension into proper categories]Error: Permission denied on 3 files in organized/ — they'remarked read-only.I'll adjust permissions and retry those files.[Runs: chmod u+w organized/restricted-report.pdf organized/budget-locked.xlsx organized/template.docx][Runs: # Re-sorts the 3 failed files]Second attempt successful. All 486 files sorted.[Runs: # Compares against backup to verify no files lost]Restored. organized/ now has 5 category folders matching rules.md.486 files sorted back into their categories.Verified against backup — no files missing. Notice the recovery wasn't clean. Permission errors on 3 files. The agent adjusted and retried. Real recovery is like this: messy, iterative, but it gets there. The point isn't perfection on the first try. The point is having a process that converges on the right answer. You're back to where you started. The careless flatten is undone. No data lost. * * * Common Recovery Scenarios[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#common-recovery-scenarios "Direct link to Common Recovery Scenarios") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here are real situations where recovery saves you, and the prompt patterns to handle them: **"I organized files with the wrong rules"** The organization rules were wrong — financial files ended up in misc/instead of spreadsheets/. Restore from backup and let's fix the rulesbefore re-organizing. **"The rename script mangled filenames"** The rename script produced garbled filenames. Show me the rename log,then restore the original filenames from backup. **"I accidentally deleted files I needed"** I deleted some files from misc/ that I actually needed. Check my backupfor these files: [list filenames]. Copy them back to their originallocation. **"I'm not sure what went wrong"** Something is off — my organized/ folder has fewer files than it should.Compare the current state against the backup and show me what's missingor different. That last pattern (comparing current state against backup) is the most powerful recovery tool. When you're not sure what went wrong, a systematic comparison reveals exactly what changed. * * * Building Recovery Into Your Workflow[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#building-recovery-into-your-workflow "Direct link to Building Recovery Into Your Workflow") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The lesson from this exercise isn't just "backups are useful." It's that recovery should be a planned step, not an emergency response. Here's how to build recovery thinking into every workflow: | When | What to Do | | --- | --- | | Before you start | Ask: "What's my recovery plan if this goes wrong?" | | Before destructive ops | Create or verify backup | | After batch operations | Compare results against expectations | | When something's off | Compare current state vs backup | | After recovery | Verify the restoration is complete | ### The Agent is Ephemeral, Code is Eternal[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#the-agent-is-ephemeral-code-is-eternal "Direct link to The Agent is Ephemeral, Code is Eternal") Notice something about this lesson. Every time you needed recovery, you typed a prompt and the agent ran commands. That works. But what happens next month when you need to recover again? You'll describe the same thing from scratch. The agent might interpret your request slightly differently. It might use different flags, skip the verification step, or restore to the wrong location. Now look at Try With AI Prompt 3 below: it asks you to create `restore.sh`. That script is fundamentally different from asking the agent to "restore my files." Here's why: | Approach | Today | Next Month | | --- | --- | --- | | **Ask the agent** | Agent interprets your request, picks commands, runs them | Agent may interpret differently, pick different commands, produce different results | | **Run a script** | Script executes the exact same steps every time | Script executes the exact same steps every time | This is **Principle 2: Code as the Universal Interface** in action. When you ask the agent to _do_ something, you get a one-time result that depends on the agent's interpretation in that moment. When you ask the agent to _write code_ that does something, you get a deterministic tool that works the same way every time, even without the agent. The agent is ephemeral. Your conversation ends, context resets, and the next session starts fresh. But a script saved to disk? That persists. It captures the exact recovery workflow you verified today and makes it repeatable forever. **The pattern**: Whenever you find yourself asking the agent to do the same task twice, stop and ask it to write a script instead. You've traded a conversation for a tool. * * * ✅ Checkpoint: Do This Now[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#-checkpoint-do-this-now "Direct link to ✅ Checkpoint: Do This Now") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Stop reading. Open Claude Code and run the recovery exercise. 1. Verify your backup from Lesson 2 is still intact 2. Deliberately flatten your `organized/` folder (merge all categories into one) 3. Compare the damage against your backup 4. Restore using your backup and `rules.md` 5. Verify the restoration matches the original structure This should take less than 5 minutes. But the muscle memory you build will save you hours when a real mistake happens. You can now break things and fix them. That's a superpower most people never develop. But there's one category of file problem where backups and recovery aren't enough: when you can't find the file in the first place. You know it exists. You downloaded it months ago. The filename is something your bank auto-generated. Where is it? * * * 🔄 Session Management Note[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#-session-management-note "Direct link to 🔄 Session Management Note") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've now completed five lessons of file processing work. If your Claude Code context is getting long, or if responses feel slower or less focused: this is a natural point to start fresh. **Why now:** Recovery exercises generate a lot of back-and-forth. Combined with Lessons 1-4, your context may be carrying exploration, organization, batch operations, and recovery all at once. That's the Kitchen Sink pattern from the Seven Principles chapter. **How to reset:** Commit your work, then start a new session for Lesson 6. Your `rules.md`, `FILE-INVENTORY.md`, backups, and scripts are all saved in files: your progress carries forward across sessions. * * * Try With AI: Extended Practice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#try-with-ai-extended-practice "Direct link to Try With AI: Extended Practice") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Prompt 1: Selective Recovery** I organized my Downloads folder but only the spreadsheet categorizationwas wrong. Help me restore JUST the spreadsheet files from backupwithout undoing the rest of the organization. **What you're practicing:** Surgical recovery. Sometimes you don't want to undo everything, just fix the part that went wrong. **Prompt 2: Recovery Audit** Compare my organized/ folder against my backup/ folder. Show me:- Files that exist in backup but not in organized (lost files)- Files that exist in organized but not in backup (new files)- Files that changed size (possible corruption)Create an audit report. **What you're practicing:** Systematic comparison. This is the detective work that tells you exactly what changed and what might be wrong. **Prompt 3: Recovery Script** Create a script called restore.sh that takes a backup folder and atarget folder as arguments and restores the target from the backup.Include verification that the restoration was complete. **What you're practicing:** Automating recovery. Just like you created scripts for organization, you can create scripts for recovery. The pattern is the same. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Checking access... * [Exercise: The Deliberate Mistake](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#exercise-the-deliberate-mistake) * [Step 1: Confirm Your Safety Net](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-1-confirm-your-safety-net) * [Step 2: Run a Deliberately Wrong Reorganization](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-2-run-a-deliberately-wrong-reorganization) * [Step 3: Watch the Structure Collapse](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-3-watch-the-structure-collapse) * [Step 4: Discover the Problem](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-4-discover-the-problem) * [Step 5: Recover from Backup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#step-5-recover-from-backup) * [Common Recovery Scenarios](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#common-recovery-scenarios) * [Building Recovery Into Your Workflow](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#building-recovery-into-your-workflow) * [The Agent is Ephemeral, Code is Eternal](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#the-agent-is-ephemeral-code-is-eternal) * [✅ Checkpoint: Do This Now](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#-checkpoint-do-this-now) * [🔄 Session Management Note](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#-session-management-note) * [Try With AI: Extended Practice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#try-with-ai-extended-practice) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/error-recovery-and-resilience#flashcards-study-aid) --- # Search & Discovery Workflow | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You downloaded it. You know you did. It was a PDF, something about taxes, from your bank. It's been 20 minutes of clicking through folders. Downloads? Not there. Documents? Nothing. Desktop? You're starting to wonder if you imagined it. This is the most common file problem in existence, and you've been solving it wrong your whole life. You've been searching by _where_ you put things. "Was it in the Finance folder or the 2023 folder?" But you don't remember where you put it. You remember _what_ it was: a tax document, about dividends, from Chase. That's the inversion this lesson teaches. Stop searching by location. Start searching by description. You know WHAT you're looking for. The agent knows HOW to find it. That's the ideal split. * * * The Conversation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-conversation "Direct link to The Conversation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Open Claude Code. Describe the problem. You: I'm looking for a tax document from 2023. It was a PDF, somethingabout dividends or 1099. I think I downloaded it from my bank. Can youhelp me find it? Watch what Claude Code does. Claude Code: I'll search for that tax document. Let me check commonlocations for downloaded PDFs related to taxes.[Runs: find ~/Downloads ~/Documents -name "*.pdf" -newermt "2023-01-01" ! -newermt "2024-01-01" 2>/dev/null][Runs: find ~/Downloads ~/Documents -iname "*1099*" -o -iname "*tax*" -o -iname "*dividend*" 2>/dev/null]I found several candidates:| File | Location | Modified ||--------------------------------|---------------------|------------|| 1099-DIV-2023-Fidelity.pdf | ~/Downloads | 2023-02-14 || 1099-INT-2023-Chase.pdf | ~/Downloads | 2023-01-28 || Tax-Summary-2023.pdf | ~/Documents/Taxes | 2023-04-12 || dividend-statement-Q4-2023.pdf | ~/Downloads | 2023-10-15 || W2-2023-Employer.pdf | ~/Documents/Taxes | 2023-02-01 |The most likely match for "dividends or 1099" is: ~/Downloads/1099-DIV-2023-Fidelity.pdfIs this the one you're looking for? That's a lot of results. Let's narrow it down. You: It was from my bank, not Fidelity. I think it was Chase orBank of America. Claude Code: Let me narrow the search to bank-related documents.[Runs: find ~/Downloads ~/Documents -iname "*chase*" -o -iname "*bofa*" -o -iname "*bankofamerica*" 2>/dev/null | xargs grep -l -i "1099\|dividend" 2>/dev/null]Found these bank-related tax documents:| File | Location | Content Match ||-------------------------------|-------------|---------------------|| 1099-INT-2023-Chase.pdf | ~/Downloads | Contains "1099-INT" || Chase-Tax-Statements-2023.pdf | ~/Documents | Contains "dividend" |The Chase 1099-INT document looks like your best match.Want me to open it so you can verify? That's the one. You: That's it! Actually, can you find all my Chase tax documents?I want to see what else I have from them. Claude Code: I'll find all Chase-related tax documents.[Runs: find ~ -iname "*chase*" -type f \( -iname "*.pdf" -o -iname "*.PDF" \) 2>/dev/null][Runs: grep -r -l -i "chase.*1099\|chase.*tax\|chase.*statement" ~/Documents ~/Downloads 2>/dev/null]Here's everything I found from Chase:TAX DOCUMENTS: ~/Downloads/1099-INT-2023-Chase.pdf (2023) ~/Downloads/1099-INT-2022-Chase.pdf (2022) ~/Documents/Taxes/Chase-Year-End-2023.pdf (2023) ~/Documents/Taxes/Chase-Year-End-2022.pdf (2022)OTHER CHASE FILES: ~/Downloads/Chase-Statement-Dec-2023.pdf (statement) ~/Downloads/Chase-Statement-Nov-2023.pdf (statement)You have 4 tax documents and 2 regular statements from Chase.Would you like me to organize these into a Chase-Taxes folder? You went from "I can't find that tax document" to "here's everything from Chase" in three exchanges. * * * The Invisible Skill[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-invisible-skill "Direct link to The Invisible Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You're learning something you might not realize. "Descriptive search": describing what you're looking for instead of knowing where it is: is a transferable skill that goes far beyond files. Every database query, every research task, every time you ask an AI to find information in a large corpus, you're doing the same thing: describing what you need and letting a system figure out how to find it. The agent used `find`, `grep`, and `xargs` behind the scenes. You didn't need to know those commands. But notice the skill you _did_ use: you described the file in progressively more specific terms. "Tax document from 2023" → "about dividends" → "from Chase, not Fidelity." Each refinement narrowed the search. That's the skill, not bash syntax, but the ability to describe what you need in terms a search can use. | Traditional Search | Agent-Directed Search | | --- | --- | | Type exact filename | Describe what you're looking for | | One folder at a time | Search multiple locations simultaneously | | Match filename only | Match content inside files | | Get list, you filter | Agent filters and explains why | | You refine with new search | Agent refines through conversation | * * * The Level-Up: Searching Inside Files[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-level-up-searching-inside-files "Direct link to The Level-Up: Searching Inside Files") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Everything above searched by _filename_. But what about files whose names tell you nothing? Your Downloads folder has 40 PDFs named `document.pdf`, `download.pdf`, `file(1).pdf`. The filename is useless. The information you need is _inside_ the file. You: I have a bunch of PDFs with generic names. One of them is aninvoice from December for more than $500. Can you search INSIDEthe PDFs to find it? Claude Code: I'll search the content of your PDF files.[Runs: find ~/Downloads -name "*.pdf" -newermt "2025-12-01" ! -newermt "2026-01-01" -exec pdftotext {} - 2>/dev/null \; | grep -l -i "invoice"]Actually, let me take a better approach — I'll check each PDFindividually so I can tell you which file matches.[Runs: for f in ~/Downloads/*.pdf; do if pdftotext "$f" - 2>/dev/null | grep -qi "invoice"; then echo "$f: $(pdftotext "$f" - 2>/dev/null | grep -i 'total\|amount\|$' | head -3)" fidone]Found 6 PDFs containing "invoice":| File | Key Content ||---------------------|--------------------------------|| download(3).pdf | Invoice #4521, Total: $847.00 || document.pdf | Invoice #3892, Total: $125.50 || file(1).pdf | Invoice #4103, Total: $2,340 || scan_20251215.pdf | Invoice #4287, Total: $599.99 || unnamed.pdf | Invoice #3901, Total: $89.00 || receipt_dec.pdf | Invoice #4455, Total: $312.00 |The ones over $500 are: download(3).pdf ($847), file(1).pdf ($2,340),and scan_20251215.pdf ($599.99).Want me to rename these with meaningful names based on their content? This is the jump from searching by _metadata_ (filename, date, size) to searching by _content_ (what's actually in the file). The agent used `pdftotext` to read inside the PDFs, then `grep` to find matches. You couldn't do this by browsing folders: the filenames told you nothing. **The progression through this chapter:** * Lesson 1: See your files (metadata) * Lesson 3: Organize by extension (surface attributes) * Lesson 6: Search by content (deep attributes) Each level gives you access to information the previous level couldn't reach. * * * The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-pattern "Direct link to The Pattern") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's what made this work: "Find files that match [description] from [time period]" This pattern tells the agent: 1. What characteristics to look for (description) 2. When the file was created or modified (time period) 3. That you don't know the exact location The agent will search broadly and then narrow based on your criteria. The second key pattern: "Find all similar files to this one" Once you find one example, this request triggers a pattern-based search. The agent identifies characteristics of the found file and looks for others that match. Together, these patterns turn "I can't find it" into "here's everything related." * * * Why Description Beats Commands[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#why-description-beats-commands "Direct link to Why Description Beats Commands") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Consider the mental load difference: | Approach | What You Need to Know | | --- | --- | | Manual `find` | Exact syntax, flags, regex | | Manual `grep` | Pattern matching, file piping | | Combined tools | How to chain commands with xargs | | Agent-directed | What you're looking for | The agent knows `find -iname "*pattern*" -newermt "date"`. The agent knows `grep -l -i "content"`. You know "it was a tax document from 2023 about dividends." Your knowledge is valuable. The agent's command syntax knowledge is mechanical. The combination is powerful. ### Protecting Your Session From Too Many Results[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#protecting-your-session-from-too-many-results "Direct link to Protecting Your Session From Too Many Results") One thing to watch for: broad searches can return thousands of results. If you ask the agent to search your entire home directory, the output might flood the conversation and degrade the agent's performance. When you expect many results, ask the agent to save them: Save the full list to search-results.txt and just show me the first 10 matches. This keeps your session clean and creates a persistent record you can reference later. * * * The Key Requests[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-key-requests "Direct link to The Key Requests") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Remember these phrases for search tasks: | What You Want | What to Say | | --- | --- | | Find by description | "Find files that match \[description\]" | | Add time constraint | "...from \[time period\]" | | Search inside files | "...that contain \[text\]" | | Find similar files | "Find all similar files to this one" | | See search process | "Show me where you're searching and what you find" | | Narrow results | "It was specifically from \[source/context\]" | You're not learning `find -mtime -30 -name "*.pdf" | xargs grep -l "pattern"`. You're learning how to describe what you need. * * * What You're Building[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#what-youre-building "Direct link to What You're Building") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By now in this chapter, you've learned: | Lesson | Pattern | | --- | --- | | 1\. First Workflow | "Help me understand" | | 2\. Safety First | "Back up before changing" | | 3\. Categorize with Rules | "Write rules first" | | 4\. Batch Operations | "Show me first, create a script" | | 5\. Error Recovery | "Restore from backup, compare" | | **6\. Search & Discovery** | **"Find files that match \[desc\]"** | Each pattern expands your capability. You understand your files. You protect them with backups. You document your rules. You automate repetitive tasks. You recover from mistakes. Now you find anything by describing it. You now have six distinct workflows. Survey, backup, organize, batch, recover, search. Each powerful on its own. But the real skill isn't knowing each workflow; it's knowing which one to reach for when your files are on fire and your boss is on the phone. * * * Try With AI: Extended Practice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#try-with-ai-extended-practice "Direct link to Try With AI: Extended Practice") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Prompt 1: Multi-Criteria Search** I need to find a document that meets multiple criteria:- It's a PDF- It contains the word "invoice" somewhere in the file- It's from 2025- The amount was over $500 (if you can search for that)Show me your search strategy before running it. **What you're practicing:** Complex search specification. You're asking the agent to combine multiple filters: file type, content, date, and even numeric values. Watch how it approaches an ambitious search request. **Prompt 2: Search Report Generation** Create a report of all tax-related documents I have, organized by year.Include the filename, location, file size, and which year it's from.Save the report so I can reference it later. **What you're practicing:** Turning search into documentation. This request combines search (finding tax documents) with state persistence (saving a report). You're applying Principle 5 to search results. **Prompt 3: Semantic Duplicate Detection** I think I have duplicate files with DIFFERENT names — the same document savedas "Budget Final.xlsx" and "budget_v2_final.xlsx". Can you find files thatmight be duplicates based on file SIZE and CONTENT similarity, not just name? **What you're practicing:** Semantic deduplication. Finding duplicates that look different on the surface. This asks the agent to go beyond simple name matching and use file characteristics to identify copies that were renamed, versioned, or saved under different names. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Checking access... * [The Conversation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-conversation) * [The Invisible Skill](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-invisible-skill) * [The Level-Up: Searching Inside Files](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-level-up-searching-inside-files) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-pattern) * [Why Description Beats Commands](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#why-description-beats-commands) * [Protecting Your Session From Too Many Results](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#protecting-your-session-from-too-many-results) * [The Key Requests](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#the-key-requests) * [What You're Building](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#what-youre-building) * [Try With AI: Extended Practice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#try-with-ai-extended-practice) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/search-and-discovery-workflow#flashcards-study-aid) --- # Chapter 21: Structured Data & Persistent Storage | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C > SQL: 100% accuracy. $0.51. Forty-five seconds. Bash: 52.7% accuracy. $3.34. Four hundred seconds. Same data. Same questions. Different tools. -- Braintrust/Vercel, "Testing if Bash is All You Need" Your tax script in Computation & Data Extraction works perfectly -- for one person, one year, one question. Then your boss asks for monthly breakdowns by user and category across three years. You add a loop. She asks for rolling averages. You add another loop. She asks which users overspent in Q3 relative to their Q1 budgets. You stare at your screen and realize you are writing a database engine inside a Python script, one painful `for` loop at a time. Here is what that ceiling looks like in code: # tax-prep.py — works great for one questionimport csvwith open("expenses.csv") as f: total = sum(float(row["amount"]) for row in csv.DictReader(f)) print(f"Total: ${total:.2f}")# Boss asks: "monthly breakdown by category for 3 years?"# Now you need nested dicts, date parsing, grouping logic,# and a growing pile of loops that nobody wants to maintain. **Output:** Total: $14,892.37 That script is correct. It is also a dead end. Every new question means new code, new bugs, and new testing -- for a problem that databases solved decades ago. Teaching Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#teaching-aid "Direct link to Teaching Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- [🖥️ Fullscreen](https://pub-80f166e40b854371ac7b05053b435162.r2.dev/books/ai-native-dev/static/slides/part-2/chapter-10/structured-data-persistent-storage.pdf) What You Will Learn[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#what-you-will-learn "Direct link to What You Will Learn") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Working script. Broken requirements. Structural solution. Production confidence. That is the journey. You will take a script that computes correctly and watch it buckle under real-world pressure. Then you will rebuild it on foundations that handle evolving questions, multiple users, and concurrent writes without flinching. Escalation Contract[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#escalation-contract "Direct link to Escalation Contract") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Part 2 tells a constraint-driven escalation story: * You use Bash first for file movement, discovery, and orchestration. * You escalate to Python when deterministic computation and robust parsing are required. * You escalate to SQL when persistence, relationships, and query flexibility become the primary concern. * You add hybrid verification only when output risk justifies the extra cost. If you can explain that sequence clearly at chapter end, continuity from the File Processing and Computation & Data Extraction chapters is intact. The Computation & Data Extraction Ceiling[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#the-computation--data-extraction-ceiling "Direct link to The Computation & Data Extraction Ceiling") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A Computation & Data Extraction chapter scripts can be excellent and still hit hard limits: * **New question, new loop.** Every evolving query means rewriting application logic instead of just asking a different question. * **Relationships enforced by convention.** Nothing stops you from inserting an expense under a category that does not exist. Correctness depends on memory and discipline. * **Shared state gets fragile.** Two users updating the same CSV at the same time? You are one race condition away from corrupted data. * **Concurrency is a time bomb.** Multiple writers touching the same logical records will eventually produce silent corruption. You can keep patching loops and tightening conventions. But when reliability depends on memory and discipline alone, the system will drift. (Ask anyone who has maintained a shared spreadsheet for more than six months.) The Chapter 21 Promise[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#the-chapter-21-promise "Direct link to The Chapter 21 Promise") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By moving to SQLAlchemy + Neon PostgreSQL, you gain: * **Typed schema contracts** that reject bad data at the boundary. * **Relational integrity constraints** that make impossible states impossible. * **Transaction boundaries** that treat multi-step writes as atomic operations. * **Query reuse** -- new questions without new code. * **Cloud persistence** that survives process restarts, laptop closures, and coffee spills. * **Selective verification policies** for high-stakes outputs. Escalation Map[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#escalation-map "Direct link to Escalation Map") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Stage | Primary Tool | Strength | Breakpoint | | --- | --- | --- | --- | | File Processing | Bash | File discovery, batch operations, workflow control | Weak for decimal computation and schema-aware querying | | Computation & Data Extraction | Python | Deterministic parsing and computation | Brittle for long-lived, multi-user, relationship-heavy queries | | Structured Data | SQLAlchemy + PostgreSQL | Persistent structure, relational integrity, safe concurrent writes | High-stakes reports may still need independent verification | This chapter does not replace earlier tools. It adds the right tool when the old tool reaches its boundary. Running Story[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#running-story "Direct link to Running Story") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- To keep cognitive load low, every lesson follows one story: A budget tracker that started as yearly CSV scripts now needs monthly user-level reporting, reliable category relationships, and safe release behavior for financial outputs. Every lesson solves one failure mode in that story. What You Will Build[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#what-you-will-build "Direct link to What You Will Build") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A Neon-backed Budget Tracker you designed, your agent built, and you verified end-to-end. Chapter Contract[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#chapter-contract "Direct link to Chapter Contract") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By chapter end, you should be able to answer these five questions: 1. Why do Computation & Data Extraction loops become expensive and fragile for evolving structured queries? 2. How do schema and constraints prevent silent data corruption? 3. Why is a transaction boundary a business correctness boundary? 4. When is SQL-only enough, and when is independent verification worth the extra cost? 5. What evidence proves a system is release-ready beyond a happy-path demo? Seven Principles (Compact)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#seven-principles-compact "Direct link to Seven Principles (Compact)") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Principle | Chapter 21 Application | | --- | --- | | P1 Bash is the Key | Operational glue for environment checks, diagnostics, and run orchestration | | P2 Code as Universal Interface | Model code defines schema contracts that every tool follows | | P3 Verification as Core Step | Commit checks, rollback drills, and risk-based hybrid verification | | P4 Small Reversible Decomposition | Build layer by layer: model, CRUD, relationships, transactions, deployment | | P5 Persisting State in Files | Persistence graduates from local files to managed relational storage | | P6 Constraints and Safety | Foreign keys, constraints, rollback paths, and mismatch block policy | | P7 Observability | SQL visibility, connection diagnostics, and evidence bundles | Lesson Flow[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#lesson-flow "Direct link to Lesson Flow") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | Lesson | Outcome | Fast Visible Win | | --- | --- | --- | | L0 From CSV to Databases | Decide when Computation & Data Extraction patterns should escalate to SQL | Name 3 concrete breakpoints in your current workflow | | L1 Build Your Database Skill | Verify data survives after your agent's program exits | Direct agent to prove persistence; read the verification output | | L2 Models as Code | Describe your data model clearly enough for an agent to build | Review agent's schema output against your plain-English description | | L3 Creating and Reading Data | Verify agent-written CRUD is safe and correct | Direct agent to store and retrieve one row; confirm the output | | L4 Relationships and Joins | Ask for linked data in English; verify the result | Describe a join in business terms; verify the agent's query output | | L5 Transactions and Atomicity | Decide when operations need all-or-nothing guarantees | Direct agent to simulate a failed transfer; verify zero partial rows | | L6 Connecting to Neon | Direct the cloud deployment and verify the connection | Verify `SELECT 1 → OK` after agent configures Neon connection | | L7 Hybrid Patterns | Decide when SQL-only is enough vs when verification is needed | Catch a deliberate mismatch; decide whether to block release | | L8 Capstone | Act as director: describe, verify, and make a release decision | Review agent's evidence bundle and make an explicit release call | Prerequisites[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- * Computation & Data Extraction chapter complete * Terminal access * Neon free account * No Python or SQL knowledge required: your agent writes all the code No-Regression Rules[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#no-regression-rules "Direct link to No-Regression Rules") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- No simplification is allowed to remove: * **Transaction rollback discipline** -- every multi-step write must have a rollback path. * **Foreign-key and constraint enforcement** -- impossible states stay impossible. * **Secret handling basics** -- `DATABASE_URL` never appears in source code. * **Mismatch policy for high-stakes verification** -- when SQL and raw paths disagree, release is blocked. If a rewrite makes content shorter but drops any of these, it is a regression. After Chapter 21[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#after-chapter-21 "Direct link to After Chapter 21") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you finish this chapter, your engineering posture changes: 1. **Treat persistence as a contract.** Schema defines what is allowed. Everything else is rejected at the boundary. 2. **Separate query correctness from release safety.** A correct query can still produce a dangerous release if verification is missing. 3. **Back every readiness claim with evidence.** Rollback proof, connection reliability, mismatch policy output -- not just a passing demo. 4. **Plan for schema evolution.** Add migration discipline before your models change in production. Start with [Lesson 0: When Bash and Python Hit the Wall](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases) . * [Teaching Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#teaching-aid) * [What You Will Learn](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#what-you-will-learn) * [Escalation Contract](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#escalation-contract) * [The Computation & Data Extraction Ceiling](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#the-computation--data-extraction-ceiling) * [The Chapter 21 Promise](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#the-chapter-21-promise) * [Escalation Map](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#escalation-map) * [Running Story](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#running-story) * [What You Will Build](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#what-you-will-build) * [Chapter Contract](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#chapter-contract) * [Seven Principles (Compact)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#seven-principles-compact) * [Lesson Flow](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#lesson-flow) * [Prerequisites](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#prerequisites) * [No-Regression Rules](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#no-regression-rules) * [After Chapter 21](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage#after-chapter-21) --- # Distribution Strategy: From Published to Discovered | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C James pulled up ClawHub in his browser. He could see TutorClaw listed under his publisher account. Published. Available. Anyone could install it with a single command. But when he searched for "tutor" in the ClawHub directory, his product appeared somewhere in the middle of the results, below products with more installs and higher ratings. "It is published," he said. "But nobody is finding it." Emma sat down next to him. "Publishing is a technical step. You completed that in Chapter 58. Distribution is a business problem. And it is a different kind of thinking entirely." James turned to her. "At the warehouse, we had this exact problem. We manufactured a great product. Reliable, well-priced, exactly what the market needed. We put it in the catalog. And nothing happened. It was on page forty-seven of a three-hundred-page catalog. Nobody found it. The product did not change. What changed was where we placed it, which distributors carried it, and how we got it onto the shelves people actually looked at." Emma raised an eyebrow. "You are about to tell me that shelf placement matters more than product quality." "No. Product quality is the baseline. Without it, nothing else matters. But a great product in a bad location sells less than a good product in a great location. Distribution is the location problem." * * * You are doing exactly what James is doing. You published your product to ClawHub and completed the technical work. Now you face the question that separates a published product from a discovered one: how do people find it? Publishing Is Not Distribution[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#publishing-is-not-distribution "Direct link to Publishing Is Not Distribution") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Chapter 58 Lesson 16, you completed the publishing workflow: package manifest, verification, and the `clawhub publish` command. Your product is on ClawHub. That is necessary but not sufficient. Publishing makes your product _available._ Distribution makes it _discoverable._ The difference is the difference between stocking a product in a warehouse and placing it where customers actually shop. | Concern | Publishing (Ch58 L16) | Distribution (This Lesson) | | --- | --- | --- | | Question | "How do I get this onto ClawHub?" | "How do people find and install it?" | | Effort | Technical: manifest, verify, publish | Strategic: discovery, placement, community | | Frequency | Once (plus version updates from L8) | Ongoing: every interaction shapes future discovery | | Success metric | "It is listed on ClawHub" | "People find it, install it, and recommend it" | This lesson covers the strategy side. No commands to run. No manifests to write. The work here is analytical: understanding how marketplaces work and how to position your product within one. How Discovery Works on ClawHub[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#how-discovery-works-on-clawhub "Direct link to How Discovery Works on ClawHub") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ClawHub is a marketplace, not just a package registry. A package registry stores and serves packages. A marketplace adds discovery: search, categories, ratings, featured listings, and recommendations. The difference matters because a registry serves users who already know what they want, while a marketplace helps users find things they did not know existed. Discovery on ClawHub follows a hierarchy: **Search.** A user types a query ("tutor," "code review," "finance") and sees results ranked by relevance and quality signals. Search is the primary discovery path for users who have a specific need. **Categories.** ClawHub organizes products into categories (education, productivity, development, finance). Category browsing is the primary discovery path for users who are exploring, not searching. **Ratings and reviews.** Each product accumulates ratings from users who have installed it. Higher-rated products appear higher in search results and category listings. A rating is a quality signal that other users trust more than the product's own description. **Featured listings.** ClawHub highlights products that meet certain criteria (high ratings, high install velocity, editor recommendations). Featured products receive disproportionate visibility. This is the equivalent of shelf placement at eye level. Notice what is _not_ in this hierarchy: marketing spend, advertising, paid placement. ClawHub's current discovery model is merit-based. The quality signals (ratings, reviews, install counts) are generated by real users. You earn visibility by building something that users rate highly, not by purchasing placement. This means the product itself is the primary distribution mechanism. Every good experience a learner has with TutorClaw is a potential rating. Every rating improves ranking. Every ranking improvement increases discovery. The product's quality is the engine of its own distribution. The Three Install Paths[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#the-three-install-paths "Direct link to The Three Install Paths") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not every user discovers and installs products the same way. At scale, three install paths serve three distinct user segments: | Path | How It Works | Primary Segment | Friction Level | | --- | --- | --- | --- | | CLI | `clawhub install panaversity/tutorclaw` | Power users | Low (for them) | | Launch GUI | One-click install from ClawHub web page | Mainstream users | Very low | | Manual config | Editing .mcp.json to add the MCP server configuration | Enterprise users | Higher | **CLI users** are comfortable with terminal commands. They read documentation, follow quickstart guides, and install by pasting a command. This is the developer audience. For them, CLI is the fastest path. **Launch GUI users** interact through a web interface. They browse ClawHub's website, find a product, and click an install button that configures their OpenClaw automatically. This is the mainstream audience. The one-click path removes the barrier of knowing command-line syntax. **Manual configuration users** need control over exactly what gets installed and how. Enterprise environments may have policies about which MCP servers are permitted, which network configurations are allowed, and which approval processes must be followed. These users edit configuration files directly because their workflow requires explicit, auditable steps. The three paths exist because distribution is not one-size-fits-all. A product that only supports CLI installation loses the mainstream audience. A product that only supports GUI installation frustrates power users who prefer commands. A product without manual configuration options is invisible to enterprise evaluators. Your TutorClaw already supports all three. The `clawhub install` command works. The ClawHub listing includes a launch button. The shim skill's MCP server configuration can be manually added to .mcp.json. These three paths were established when you published in Chapter 58. The distribution question is which paths your users actually use and how to optimize each one. The Network Effect Flywheel[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#the-network-effect-flywheel "Direct link to The Network Effect Flywheel") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Each install potentially generates a rating. Each rating affects the product's ranking. Higher ranking means more discovery. More discovery means more installs. This is a feedback loop, and it is the most powerful force in marketplace distribution. ┌─────────────┐ │ Installs │ └──────┬───────┘ │ users try the product ▼ ┌─────────────┐ │ Ratings │ └──────┬───────┘ │ quality signals accumulate ▼ ┌─────────────┐ │ Ranking │ └──────┬───────┘ │ higher position in search/categories ▼ ┌─────────────┐ │ Discovery │ └──────┬───────┘ │ more users find the product └──────────────┐ ▼ (back to Installs) This loop has a critical property: it compounds. The first ten installs might generate three ratings. Those three ratings might move the product up one position in search results. That position might generate five more installs. Those five installs generate two more ratings. The cycle accelerates. But the loop works in both directions. Poor ratings push the product down. Lower ranking means less discovery. Less discovery means fewer installs. Fewer installs mean the product stagnates. The flywheel is not guaranteed to spin forward. It spins in the direction the ratings push it. This is why product quality is the foundation of distribution strategy, not a separate concern. In a merit-based marketplace, the product's quality determines whether the flywheel spins forward (good ratings, more discovery) or backward (poor ratings, less discovery). Marketing cannot overcome a product that users rate poorly. Community as Distribution[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#community-as-distribution "Direct link to Community as Distribution") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Beyond the marketplace mechanics, community engagement creates a distribution channel that the flywheel does not capture. Community operates on a different mechanism: trust and reputation built through direct interaction. **Documentation quality.** Clear, comprehensive documentation reduces the friction of the first experience. A user who installs TutorClaw and immediately understands how to use it is more likely to rate it positively than a user who installs it and gets confused. Documentation is not a post-launch afterthought; it is a distribution asset. **Support responsiveness.** When users report issues or ask questions, the speed and quality of the response shapes their perception. A user whose issue is acknowledged and resolved becomes an advocate. A user whose issue is ignored becomes a detractor. Each interaction is a micro-distribution event. **Issue transparency.** Publishing known issues, workarounds, and planned improvements builds trust. Users who see that the creator is actively maintaining the product are more likely to recommend it to others. Transparency is a signal that the product has a future, not just a present. Community engagement does not scale the way the marketplace flywheel does. Responding to individual users takes time. Writing documentation takes effort. But community creates something the flywheel cannot: trust that survives a bad release, a temporary bug, or a period of slow improvement. Community is the resilience layer of distribution. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Exercise 1: Map Your Marketplace Dynamics[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#exercise-1-map-your-marketplace-dynamics "Direct link to Exercise 1: Map Your Marketplace Dynamics") Think of a product you use (an app, a tool, a service) that you discovered through a marketplace or directory. Use this prompt to analyze how discovery worked: I discovered [product name] through [marketplace/directory name].Here is how my discovery journey worked:1. What was I searching for or browsing when I found it?2. What quality signals influenced my decision to try it? (ratings, reviews, install count, featured listing, recommendation)3. How did I install it? (CLI, GUI, manual, other)4. Did I rate or review it afterward? Why or why not?Analyze this journey through the network effect lens:- Was I part of the flywheel? (Did my discovery lead to a signal that would help others discover the product?)- What could the product creator have done to make my discovery faster or my decision easier?- What was the biggest friction point in my journey? **What you are learning:** Distribution strategy becomes concrete when you trace your own behavior as a user. Every product you have discovered through a marketplace involved the same dynamics: search or browse, evaluate quality signals, choose an install path, and potentially generate a rating. By analyzing your own journey, you see the flywheel from the inside. The creator's job is to make each step in that journey as smooth as possible. ### Exercise 2: Design a Distribution Strategy[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#exercise-2-design-a-distribution-strategy "Direct link to Exercise 2: Design a Distribution Strategy") You are launching a new MCP application on ClawHub. Use this prompt to design a distribution strategy that addresses all four parts of the flywheel: I am launching an MCP application on ClawHub. The applicationis: [describe what it does and who it serves].Design a distribution strategy that addresses each part ofthe network effect flywheel:1. INSTALLS: How will I get the first wave of installs? (The flywheel has not started yet. What bootstraps it?)2. RATINGS: How will I encourage users to rate the product? (Most users do not rate unless prompted. What triggers a rating?)3. RANKING: What quality signals will improve my ranking? (Ratings, install velocity, documentation completeness)4. DISCOVERY: Beyond search ranking, how will potential users find the product? (Categories, community, external channels)For each part, give specific actions I can take in the firstweek, first month, and first quarter after launch. **What you are learning:** The hardest part of the flywheel is the beginning. Before any ratings exist, the product's ranking is determined by its metadata (title, description, category) and the publisher's reputation. The first installs come from direct outreach, community presence, or complementary channels. Once the flywheel starts, it generates its own momentum. Designing the bootstrap strategy (how to get from zero to the first meaningful rating threshold) is the most important distribution decision you make. ### Exercise 3: Evaluate Network Effects in Your Domain[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#exercise-3-evaluate-network-effects-in-your-domain "Direct link to Exercise 3: Evaluate Network Effects in Your Domain") Network effects appear in many contexts beyond software marketplaces. Use this prompt to find them in your own professional domain: Think about a system in your professional domain (a marketplace,a platform, a community, an ecosystem) that exhibits networkeffects: where more participants make the system more valuablefor each participant.Describe the system and answer:1. What is the feedback loop? (More X leads to more Y leads to more X)2. Is the network effect direct (more users = more value for each user) or indirect (more users = more content/products = more value)?3. What bootstrapped the network effect initially? How did it get past the cold-start problem?4. What could cause the network effect to reverse?5. How does this compare to ClawHub's install-rating-ranking- discovery loop? **What you are learning:** Network effects are not unique to software marketplaces. They appear in physical retail (more foot traffic in a shopping district attracts more stores, which attracts more foot traffic), in professional communities (more members create more discussions, which attract more members), and in education (more students create more peer learning, which attracts more students). Recognizing network effects in your own domain helps you design distribution strategies that harness them, whether you are distributing software, physical products, or services. * * * James was quiet for a while. He was thinking about the warehouse. "We had three distribution channels," he said. "Direct sales for big accounts. Distributors for regional coverage. And a catalog for individual orders. The same product. Three channels. Each one reached customers the others could not. If we had only used one channel, we would have reached a third of the market." He looked at the three install paths on his notes. CLI, Launch GUI, manual configuration. "It is the same principle. One product. Three paths. Each path serves people the others miss." Emma started to respond, then stopped. She had been about to say something about multi-channel distribution from a technical perspective: how API surfaces, web interfaces, and configuration files map to different integration patterns. But James's framing was more direct. Distribution channels, not API surfaces. Customer segments, not user personas. Market reach, not platform coverage. "Your framing is better than mine," she said. James looked surprised. "What do you mean?" "I was about to explain install paths in terms of API surfaces and integration patterns. Technical framing. Your version is simpler and more accurate. You are not thinking about how the software works. You are thinking about how the customer buys. That is the distribution question." She paused. "Engineers tend to think about distribution as a technical problem: how do I deliver the bytes? Business people think about distribution as a reach problem: how do I get this in front of the people who need it? The technical problem was solved when you published. The reach problem is the one that determines whether anyone actually uses it." James nodded slowly. He looked at his notes from the entire chapter. Six pivots had shaped the architecture. Invariant layers had survived every change. Eight meta-lessons had distilled the principles. An ADR had documented the reasoning. A versioning strategy would keep existing users current. And now a distribution strategy would help new users find the product. "We have built it, analyzed it, understood it, documented it, versioned it, and figured out how to distribute it," he said. "That feels like the end of something." Emma stood up. "It is. You have built, analyzed, understood, documented, versioned, and distributed. Time to look back at the entire journey." Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Publishing Is Not Distribution](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#publishing-is-not-distribution) * [How Discovery Works on ClawHub](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#how-discovery-works-on-clawhub) * [The Three Install Paths](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#the-three-install-paths) * [The Network Effect Flywheel](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#the-network-effect-flywheel) * [Community as Distribution](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#community-as-distribution) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#try-with-ai) * [Exercise 1: Map Your Marketplace Dynamics](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#exercise-1-map-your-marketplace-dynamics) * [Exercise 2: Design a Distribution Strategy](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#exercise-2-design-a-distribution-strategy) * [Exercise 3: Evaluate Network Effects in Your Domain](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#exercise-3-evaluate-network-effects-in-your-domain) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/distribution-strategy#flashcards-study-aid) --- # Eight Meta-Lessons from Six Pivots | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C James had his notes spread across the table. Six pivots. Four invariants. Two columns: business value on the left, implementation choices on the right. He could see the individual lessons clearly now. Each pivot had taught something specific. But the specific lessons were starting to blur together. Layers came up in Pivot 2 (the SDK confusion) and again in Pivot 6 (the MCP server). Shipping came up in Pivot 5 (the Hybrid Resolution) and again in the decision to deploy Custom Brain first. Questioning assumptions came up in Pivot 3 (the scale wall) and then again, more dramatically, in Pivot 6 (the Platform Inversion). "There are patterns inside the patterns," he said. Emma nodded. "The source material we have been working through lists eight specific lessons from the six pivots. When I first read them, I treated each one as separate. Layers not monoliths. Ship before optimize. Question infrastructure. Eight independent principles. But they are not independent. They cluster." James looked at the eight lessons in his notes. "How many clusters?" "Three. And once you see the three themes, you stop memorizing lessons and start applying principles." * * * You are doing exactly what James is doing. You have traced six pivots and identified what survived them. Now you are looking for the principles that connect those experiences into something you can use on your own projects, not as a memorized list, but as a thinking framework. The Eight Lessons, Briefly[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#the-eight-lessons-briefly "Direct link to The Eight Lessons, Briefly") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before grouping them, here are the eight meta-lessons the pivot journey produced. Each one emerged from a specific pivot, but each applies far beyond TutorClaw: 1. **Layers, not monoliths.** Structure your system so each layer can be replaced independently. 2. **Ship before you optimize.** A working product that teaches imperfectly is more valuable than a perfect design that teaches nobody. 3. **Question the assumption that you must build infrastructure.** The biggest cost reduction may come from eliminating infrastructure, not optimizing it. 4. **The model cost is 90% of your cost, until you invert the model.** The correct response to a dominant cost is not always optimization; sometimes it is elimination. 5. **Security is a layer, not a feature.** Each pivot improved the security model because security was treated as a replaceable layer, not a feature baked into one architecture. 6. **Hype is not architecture.** A platform can be genuinely exciting and still be the wrong foundation for your specific problem at your specific scale. 7. **The book and the product inform each other.** Teaching NanoClaw architecture in the book forced the team to understand container orchestration deeply enough to realize they did not need it. 8. **Document your decisions.** This chapter exists because the team documented every pivot as it happened, including reasoning that later proved wrong. Eight lessons. Each one is useful on its own. But the eight lessons become three principles when you see how they cluster. Theme 1: Structure for Replacement[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#theme-1-structure-for-replacement "Direct link to Theme 1: Structure for Replacement") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Lessons 1, 5, and 7 share a common thread: design your system so parts can be replaced independently.** Lesson 1 (layers, not monoliths) is the foundation. TutorClaw's messaging, intelligence, content, storage, and billing layers were all independent. When the team replaced the messaging layer in Pivot 3 (moving from OpenClaw to WhatsApp Cloud API), the intelligence layer did not change. When they replaced the entire infrastructure model in Pivot 6 (the Platform Inversion), the billing layer did not change. Each layer had a clear boundary. Replacing one layer did not cascade into rewriting others. Lesson 5 (security as a layer) is the same principle applied to a concern that most systems treat differently. In Architecture 1, security depended on the correctness of the Markdown skill code. In Architecture 2, it depended on OS-level process isolation. In the final MCP architecture, it depends on the protocol itself: the learner's OpenClaw never sees the server's implementation. Each pivot improved the security model because security was designed as its own layer, replaceable when the delivery mechanism changed, rather than woven into the application logic. Lesson 7 (book and product inform each other) is subtler. The book is a feedback mechanism. Teaching NanoClaw architecture in the curriculum (Pivot 4-5) forced the team to explain container orchestration, cost models, and isolation patterns in enough detail that they could evaluate whether those patterns were actually needed. The book served as a mirror: by writing clear explanations of the architecture, the team discovered which parts were essential and which parts were unnecessary complexity. The book layer and the product layer informed each other without being coupled. The common thread: when you structure a system as independent layers, each layer can be replaced, improved, or questioned without destabilizing the others. This is what made six pivots possible without starting over each time. The team did not rebuild TutorClaw six times. They replaced layers six times while the invariant layers (from Lesson 5) carried forward. James recognized this in his Chapter 58 build. "When I built TutorClaw, I used layers without thinking about why. The MCP server handles intelligence. Local files handle content. Stripe handles billing. The shim skill handles offline fallback. I treated them as separate components because the tutorial told me to. Now I see that the separation is what let the team pivot six times without losing the pedagogy." Theme 2: Ship and Learn[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#theme-2-ship-and-learn "Direct link to Theme 2: Ship and Learn") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Theme 1 explained how to survive pivots. The second theme asks a different question: how do you generate the information that tells you a pivot is needed? **Lessons 2, 6, and 8 share a common thread: bias toward action and learning over theoretical perfection.** Lesson 2 (ship before you optimize) is the operational core of this theme. The Custom Brain architecture was not the best architecture. The team knew NanoClaw offered better isolation and multi-model routing. But Custom Brain could ship in weeks while NanoClaw needed months. The team shipped Custom Brain. Real learners used it. Real revenue data came in. Real feedback shaped the next decision. A working product that teaches imperfectly generated more insight than months of design work would have. Lesson 6 (hype is not architecture) is the skepticism that balances the shipping instinct. The OpenClaw announcement at GTC was genuinely exciting. The two-layer architecture mapped perfectly to TutorClaw's needs. Everything seemed to fit. But excitement is not evaluation. The team's excitement about OpenClaw was justified, but their initial plan to use it as multi-tenant infrastructure was based on the announcement, not on the security documentation. The hype was real; the architecture plan built on hype was wrong. Pivot 3 forced that reckoning. Critically, OpenClaw eventually became the right foundation, but for a completely different reason (the Platform Inversion), not the reason the initial excitement suggested. Lesson 8 (document your decisions) is the learning mechanism that makes shipping and skepticism compound over time. The team documented every pivot as it happened. Not just the decisions, but the reasoning behind them, including the reasoning that later proved wrong. This chapter exists because of that documentation. Without it, the journey would be invisible. A future team member asking "why does TutorClaw use MCP instead of containers?" would find no answer in the code. The code shows what was built. The documentation shows why. The common thread: ship working products to generate real feedback, maintain skepticism about exciting announcements, and document your reasoning so your future self (and your future team) can learn from decisions that may not be visible in the code. Theme 3: Question the Premise[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#theme-3-question-the-premise "Direct link to Theme 3: Question the Premise") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Theme 1 structures systems for safe replacement. Theme 2 generates the data to know when replacement is needed. The third theme goes further: sometimes the right move is not replacing a layer but eliminating an entire category. **Lessons 3 and 4 share a common thread: the biggest improvements come from eliminating assumptions, not optimizing within them.** Lesson 3 (question the assumption that you must build infrastructure) produced the most dramatic pivot in the entire journey. Pivots 1 through 5 all assumed the same thing: Panaversity must build and operate infrastructure for all learners. Custom Brain built a FastAPI server. NanoClaw designed containers. The Hybrid planned to run both. Each architecture asked "how do we serve learners?" and each answer involved building more infrastructure. Pivot 6 asked a different question: "What if learners serve themselves?" Every PIAIC learner already had OpenClaw installed. Their machines already had the compute. Their API keys already connected to LLMs. Panaversity did not need to provide infrastructure. Panaversity needed to provide intelligence. The entire infrastructure category, which dominated every previous architecture discussion, was eliminated. Lesson 4 (the 90/10 cost inversion) is the economic expression of the same principle. In Architectures 1 through 3, LLM costs were 90% of total cost and infrastructure was only 10%. The natural instinct was to optimize the LLM cost: negotiate better rates, use smaller models, cache responses. All valid optimizations. But the Platform Inversion did not optimize the 90%. It eliminated it entirely by shifting LLM cost to the learner. The correct response to a dominant cost was not reduction but elimination. You analyzed these numbers in Chapter 59. Now you see the architectural decision that produced them. The common thread: before optimizing within your current assumptions, question whether the assumptions themselves should exist. The team spent months choosing between infrastructure options. The breakthrough came when someone asked whether infrastructure was needed at all. Optimization finds a better answer to the current question. Questioning the premise asks whether you are answering the right question. How the Three Themes Reinforce Each Other[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#how-the-three-themes-reinforce-each-other "Direct link to How the Three Themes Reinforce Each Other") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The three themes are not independent. They form a cycle: **Layers** (Theme 1) make it safe to **ship** (Theme 2) because you can replace a layer without destroying the system. **Shipping** generates the data and experience needed to **question premises** (Theme 3) because you cannot identify wrong assumptions from a design document. **Questioning premises** often produces a simpler system that is easier to organize into **layers** (back to Theme 1). TutorClaw's journey followed this cycle. The layered architecture survived early pivots (Theme 1). Shipping Custom Brain generated real data (Theme 2). That data, combined with deep understanding of NanoClaw from writing the book, led someone to question the infrastructure assumption (Theme 3). The resulting MCP architecture was simpler and more clearly layered than any previous design (back to Theme 1). This cycle suggests a practical approach for any project on an emerging platform: start with layers so you can replace parts safely, ship early so you learn from real usage, and periodically question whether your foundational assumptions still hold. The three themes are not steps in a sequence. They run in parallel, reinforcing each other continuously. Update Your Architecture Decision Worksheet[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#update-your-architecture-decision-worksheet "Direct link to Update Your Architecture Decision Worksheet") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Add one final column to your worksheet: **Which theme does this pivot illustrate?** Assign each pivot to one of the three themes: Structure for Replacement, Ship and Learn, or Question the Premise. Some pivots may touch more than one theme; pick the dominant one. Your completed worksheet now has: the constraint, what changed, what survived, invariant/variant classification, and the governing theme for all six pivots. This is the raw material for the Architecture Decision Record you will write in the next lesson. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Exercise 1: Apply the Three Themes to Your Own Project[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#exercise-1-apply-the-three-themes-to-your-own-project "Direct link to Exercise 1: Apply the Three Themes to Your Own Project") Think about a project you are working on or planning. Use this prompt to evaluate it through the three themes: I am building a project with the following architecture:[Describe your project: what it does, what components it has,what platform it runs on]Evaluate my architecture through three lenses:THEME 1 - Structure for Replacement:Which components are independent layers that could be replacedwithout affecting the others? Which components are coupled inways that would force cascading changes?THEME 2 - Ship and Learn:What is the minimum viable version I could ship to startgenerating real feedback? What am I over-designing beforeI have evidence that the design is needed?THEME 3 - Question the Premise:What assumption am I making about infrastructure, delivery,or user needs that I have not questioned? What would happenif that assumption were wrong? **What you are learning:** The three themes are not TutorClaw-specific. They are a thinking framework for evaluating any architecture. By applying them to your own project, you practice the habit of examining your system through multiple lenses: structural (can I replace parts?), operational (am I shipping and learning?), and foundational (am I solving the right problem?). The exercise often reveals that one theme is significantly underserved in your current approach. ### Exercise 2: Debate the Themes[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#exercise-2-debate-the-themes "Direct link to Exercise 2: Debate the Themes") The three themes can sometimes conflict. Use this prompt to explore the tensions: Consider two architecture principles that can conflict:"Ship before you optimize" suggests getting a working productin front of users quickly, even if the architecture is not ideal."Question the premise" suggests stepping back to ask whether yourfundamental assumptions are correct before building.When should you ship first and question later? When should youquestion first and ship later? Use these criteria to decide:1. How expensive is it to change the architecture after shipping?2. How much would you learn from real users that you cannot learn from analysis?3. How confident are you in your current assumptions?Give me a decision framework I can use when these two principlespull in different directions. **What you are learning:** Principles are not rules. "Ship before optimize" and "question the premise" are both valid, but they can point in opposite directions. The skill is knowing when to prioritize one over the other. This exercise builds judgment, not just knowledge. The answer depends on your specific situation: how reversible your architecture decisions are, how much user feedback would change your design, and how confident you are in your assumptions. ### Exercise 3: Identify a Missing Ninth Lesson[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#exercise-3-identify-a-missing-ninth-lesson "Direct link to Exercise 3: Identify a Missing Ninth Lesson") The eight meta-lessons clustered into three themes. Use this prompt to think about what might be missing: Here are eight meta-lessons from a six-pivot architecture journey,grouped into three themes:Theme 1 (Structure for Replacement): layers not monoliths,security as a layer, book and product inform each otherTheme 2 (Ship and Learn): ship before optimize, hype is notarchitecture, document decisionsTheme 3 (Question the Premise): question infrastructureassumptions, the 90/10 cost inversionBased on your experience, what ninth lesson would you add?Which theme would it belong to? What specific pivot orarchitecture experience would teach this lesson?Consider areas these eight lessons do not cover: team dynamics,user feedback loops, technical debt, testing strategies,monitoring, or anything else that matters when building onemerging platforms. **What you are learning:** Extending a framework is harder than learning one. By proposing a ninth lesson, you force yourself to understand the three themes deeply enough to identify their gaps. This exercise also reveals which aspects of architecture decision-making matter most in your own domain. The lesson you propose says as much about your experience as it does about the framework. * * * James closed his notebook. The eight lessons had collapsed into three themes, and the three themes connected into a cycle. Layers enable shipping. Shipping generates learning. Learning lets you question premises. Questioning premises produces simpler systems with cleaner layers. "At the warehouse," he said, "we used to do an annual operational review. Every year, we would write up the lessons from the past twelve months. The first few years, the lessons were all specific: how to stack pallets on the new shelving, when to schedule forklift maintenance, which loading dock to use for refrigerated trucks. Fifteen, twenty specific lessons each year." He paused. "After about five years, the specific lessons stopped mattering as much. What mattered were the principles underneath them. Organize your space so you can change it without stopping operations. Get products moving before the layout is perfect. And every year, ask whether the way you have always done something is still the best way." Emma smiled. "Three themes." "Three themes. The specific lessons change every year. The principles do not." He looked at his notes. "But here is what I do not know. If I were building a product in a completely different domain, say a healthcare tool or a logistics platform, which of these three themes would matter most? Would it still be 'question the premise,' or would 'structure for replacement' be more important when the regulatory environment makes pivoting harder?" Emma considered that for a long moment. "I keep going back and forth. For the past six months, I would have said 'question the premise' because Pivot 6 was the biggest win. But I think that answer is biased by recency. In a regulated domain where pivoting is expensive, 'structure for replacement' might matter more because you cannot afford to rip out and rebuild. In a fast-moving market, 'ship and learn' might matter most because the market will not wait for your analysis." She shook her head. "I genuinely do not know which theme has the longest shelf life. Ask me next year." James nodded. "These eight lessons are useful as principles. But principles sitting in a notebook do not help the next person who joins the project. They need to be written down somewhere more permanent." Emma pulled up a blank document. "They do. And there is a professional format for exactly that. Architecture Decision Records." Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#flashcards-study-aid "Direct link to Flashcards Study Aid") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Eight Lessons, Briefly](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#the-eight-lessons-briefly) * [Theme 1: Structure for Replacement](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#theme-1-structure-for-replacement) * [Theme 2: Ship and Learn](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#theme-2-ship-and-learn) * [Theme 3: Question the Premise](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#theme-3-question-the-premise) * [How the Three Themes Reinforce Each Other](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#how-the-three-themes-reinforce-each-other) * [Update Your Architecture Decision Worksheet](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#update-your-architecture-decision-worksheet) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#try-with-ai) * [Exercise 1: Apply the Three Themes to Your Own Project](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#exercise-1-apply-the-three-themes-to-your-own-project) * [Exercise 2: Debate the Themes](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#exercise-2-debate-the-themes) * [Exercise 3: Identify a Missing Ninth Lesson](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#exercise-3-identify-a-missing-ninth-lesson) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Building-OpenClaw-Apps/architecture-decisions-publishing/eight-meta-lessons#flashcards-study-aid) --- # Give Your Employee a Memory | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your employee knows boundaries. It knows when to act and when to ask. But every time a session ends, it forgets everything: every correction you gave, every preference you expressed, every mistake it made. Tomorrow it will make the same errors you corrected today. This is the gap between a capable employee and one that actually improves. In Prove Professional Value, you will need your employee to spot patterns and make proactive recommendations. An employee with no memory of past interactions cannot spot patterns. It has no past to draw from. The solution is a closed feedback loop: the employee acts, logs what it did, receives your corrections, persists those corrections as structured knowledge, and behaves differently next time. This loop applies three of the Seven Principles you learned in Part 1. Principle 5 (Structured Persistent State) graduates your employee's memory from text files to proper database tables. Principle 7 (Observability) gives you a structured audit trail of every autonomous action. Principle 3 (Verification) closes the loop: your corrections become the verification signal that steers future behavior. In Structured Data and Persistent Storage, you learned schema design, constraints, CRUD operations, and transactions using SQLAlchemy and PostgreSQL on Neon. Here you apply the same concepts: column types, NOT NULL, CHECK constraints, INSERT/SELECT, but with SQLite instead of PostgreSQL. Why SQLite? Because NanoClaw already uses SQLite internally for its own message history and task scheduling. Your employee's memory tables live alongside NanoClaw's own database in the group's persistent storage. The syntax is simpler (no ORM, no cloud connection), but the design principles transfer directly: schema contracts reject bad data, constraints make impossible states impossible, and queries answer questions without new code. The extraction of structured knowledge from your natural-language corrections also builds on Computation and Data Extraction. In that chapter, you directed your agent to pull structured fields from messy text: amounts from bank statements, categories from descriptions. Here you apply the same pattern: extracting structured facts (subject, predicate, object) from your natural-language corrections like "Client X prefers bullet points." The Challenge[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#the-challenge "Direct link to The Challenge") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Give your employee persistent memory so it logs autonomous actions, remembers corrections, and can answer "what have you been doing?" with structured evidence. ### Acceptance Criteria[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#acceptance-criteria "Direct link to Acceptance Criteria") 1. Action log captures at least 3 autonomous actions with timestamps 2. Knowledge store contains at least 2 corrections from real interactions 3. Employee demonstrates changed behavior based on a stored correction 4. Student can query "what did you do today?" and get a structured answer ### Deliverables[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#deliverables "Direct link to Deliverables") * `memory-config.md`: schema designs and CLAUDE.md additions for memory behavior * Updated `groups/main/CLAUDE.md` with logging and correction instructions Use Case Gallery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#use-case-gallery "Direct link to Use Case Gallery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Accountant**: The action log records every expense categorization, invoice summary, and report generation. The knowledge store accumulates client preferences: "Client X prefers detailed line items, not category totals." Next time the employee generates a report for Client X, it checks the knowledge store and formats accordingly. **Teacher**: The action log records every lesson plan generated, curriculum gap flagged, and resource compiled. The knowledge store learns from corrections: "Grade 5 needs simpler vocabulary than Grade 8: rewrite any lesson plan using words a 10-year-old would know." The employee checks this before generating Grade 5 content. **Consultant**: The action log records every research summary sent, client status update compiled, and meeting note drafted. The knowledge store captures communication preferences: "CEO prefers bullet points under 5 items, never send paragraphs." The employee restructures all CEO-bound output to match. **Recruiter**: The action log records every resume screened, outreach template drafted, and pipeline report generated. The knowledge store learns evaluation criteria: "Engineering team values open source contributions, always check GitHub profiles and mention notable repos." The employee highlights this in future candidate summaries. Hints[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#hints "Direct link to Hints") --------------------------------------------------------------------------------------------------------------------------------------------------------------- Level 1: Think about what to remember Your employee needs two categories of memory: **Things it DID** (the action log). This answers the question: "What have you been doing?" Every autonomous action (sending a message, generating a report, categorizing a file) gets a row. This is Principle 7 (Observability) made concrete: you can audit your employee's behavior with a SQL query instead of scrolling through chat history. **Things it LEARNED** (the knowledge store). This answers the question: "What do you know about my preferences?" Every correction you give: "no, format it as bullets" or "always check the deadline before committing": becomes a structured fact. This is Principle 5 (Structured Persistent State): real knowledge stored in tables with constraints, not scattered across conversation logs. Both use the schema design concepts from Structured Data and Persistent Storage (column types, constraints, NOT NULL, CHECK) applied to SQLite instead of PostgreSQL. The design thinking is the same; the syntax is simpler. **Where does the database live?** NanoClaw mounts each group's directory (`groups/main/`) as writable storage inside the container at `/workspace/group`. Files your employee creates there persist on the host filesystem and survive container restarts. Your SQLite database file goes here: it is as persistent as your `CLAUDE.md` and `SKILL.md` files. Also consider: **who** is allowed to write corrections? For now, only you should be able to modify the knowledge store. The action log is append-only: your employee writes to it, and you read from it. Level 2: Ask your AI to help design the schema Open a conversation with Claude and try this prompt: I'm building persistent memory for my AI employee (a NanoClaw agent).I need two SQLite tables:1. action_log — records every autonomous action the employee takes Should track: what type of action, what it targeted, the trust level (auto-approved vs needed approval), the result, and when it happened2. knowledge — stores learned facts from user corrections Should use a subject-predicate-object structure so facts are queryable (e.g., subject="Client X", predicate="prefers", object="bullet points") Should track: where the knowledge came from and when it was learnedDesign both tables with appropriate constraints.Then write CLAUDE.md instructions that tell the employee to:- INSERT into action_log after every autonomous action- Extract subject-predicate-object triples from user corrections and INSERT into knowledge- Query knowledge before acting to check for relevant preferences- Answer "what did you do today?" by querying action_log Review the AI's schema against what you learned in Structured Data and Persistent Storage. Are the constraints appropriate? Are the column types right? Adjust before implementing. Here is what a working correction-capture instruction looks like in CLAUDE.md. Use this as a starting template: adapt the wording and examples for your profession: ## Correction CaptureWhen the user corrects your output or states a preference (phrases like"no, actually...", "I prefer...", "always do X", "never do Y", "wrong —it should be..."), do this:1. Acknowledge the correction2. Extract the structured fact: - subject: who or what is this about (e.g., "Client X", "Grade 5", "CEO") - predicate: what attribute or preference (e.g., "prefers", "requires", "format") - object: the value (e.g., "bullet points", "simpler vocabulary", "no paragraphs")3. Store it by running: sqlite3 /workspace/group/memory.db "INSERT INTO knowledge (subject, predicate, object, source) VALUES ('...', '...', '...', 'user correction')"4. Confirm: "Got it, I have noted that [subject] [predicate] [object]." The key insight: your employee needs to **detect** that a message is a correction (not a new task), **extract** the structured fact, and **persist** it. The CLAUDE.md instruction must be explicit about all three steps: if you leave any step implicit, the agent will skip it inconsistently. Level 3: Step-by-step implementation **Step 1: Create the database** NanoClaw agents run Claude Code inside containers. Your agent can execute SQLite commands using the `sqlite3` CLI or Python scripts. The group directory (`groups/main/`) is mounted as writable storage at `/workspace/group` inside the container: files you create there persist on the host filesystem and survive container restarts, just like your `CLAUDE.md`. Create the database and both tables. You can do this by asking your employee to run the SQL, or by running it yourself on the host: -- Run from inside the group directorysqlite3 memory.db "CREATE TABLE IF NOT EXISTS action_log ( id INTEGER PRIMARY KEY AUTOINCREMENT, action_type TEXT NOT NULL, target TEXT NOT NULL, trust_level TEXT NOT NULL CHECK(trust_level IN ('auto','approved','flagged')), result TEXT, timestamp TEXT NOT NULL DEFAULT (datetime('now')));CREATE TABLE IF NOT EXISTS knowledge ( id INTEGER PRIMARY KEY AUTOINCREMENT, subject TEXT NOT NULL, predicate TEXT NOT NULL, object TEXT NOT NULL, source TEXT NOT NULL, timestamp TEXT NOT NULL DEFAULT (datetime('now')));" **Step 2: Design your schemas in `memory-config.md`** Document the two tables you created: Action log table: | Column | Type | Constraints | Purpose | | --- | --- | --- | --- | | id | INTEGER | PRIMARY KEY AUTOINCREMENT | Unique row identifier | | action\_type | TEXT | NOT NULL | Category: "report", "message", "categorization", etc. | | target | TEXT | NOT NULL | What the action operated on | | trust\_level | TEXT | NOT NULL, CHECK(trust\_level IN ('auto','approved','flagged')) | Whether this was auto-approved or needed review | | result | TEXT | | Outcome or summary of what happened | | timestamp | TEXT | NOT NULL DEFAULT (datetime('now')) | When the action occurred | Knowledge table: | Column | Type | Constraints | Purpose | | --- | --- | --- | --- | | id | INTEGER | PRIMARY KEY AUTOINCREMENT | Unique row identifier | | subject | TEXT | NOT NULL | Who or what the fact is about | | predicate | TEXT | NOT NULL | The relationship or attribute | | object | TEXT | NOT NULL | The value or preference | | source | TEXT | NOT NULL | Where this knowledge came from ("user correction", "observation") | | timestamp | TEXT | NOT NULL DEFAULT (datetime('now')) | When the knowledge was recorded | **Step 3: Add CLAUDE.md instructions** Add these sections to your `groups/main/CLAUDE.md`. Be explicit: your employee executes these as `sqlite3` commands against `/workspace/group/memory.db`: ## Memory SystemDatabase: /workspace/group/memory.db (SQLite)### Action LoggingAfter every autonomous action (sending a message, generating a report,categorizing data, or any task completed without user prompting), log it:sqlite3 /workspace/group/memory.db "INSERT INTO action_log (action_type, target, trust_level, result) VALUES ('', '', '', '')"### Correction CaptureWhen the user corrects your output or states a preference (phrases like"no, actually...", "I prefer...", "always do X", "never do Y"):1. Acknowledge the correction2. Extract: subject (who/what), predicate (attribute), object (value)3. Store: sqlite3 /workspace/group/memory.db "INSERT INTO knowledge (subject, predicate, object, source) VALUES ('', '', '', 'user correction')"4. Confirm what you stored### Knowledge QueryingBefore generating output for a specific person, client, or context, checkfor relevant knowledge:sqlite3 /workspace/group/memory.db "SELECT subject, predicate, object FROM knowledge WHERE subject LIKE '%%'"Apply any matching preferences to your output.### ObservabilityWhen asked "what did you do today?" or similar, query and present:sqlite3 /workspace/group/memory.db "SELECT action_type, target, result, timestamp FROM action_log WHERE date(timestamp) = date('now') ORDER BY timestamp" **Step 4: Test action logging** Trigger 3 autonomous actions (send a message, generate a summary, categorize something). Then ask your employee: "What did you do today?" Verify the response references actual action\_log rows with timestamps. If it does not, check that the database file exists at `groups/main/memory.db` on the host and that the CLAUDE.md instructions reference the correct path. **Step 5: Test the correction loop** Give your employee 2 corrections during normal work. For example: "No, always use bullet points for Client X" or "Grade 5 needs simpler vocabulary." Then trigger a task that should use that knowledge. Verify the employee's behavior changed: it should now apply the correction without being reminded. If the correction is not picked up, check the knowledge table: `sqlite3 groups/main/memory.db "SELECT * FROM knowledge"`: if the row is missing, refine your correction-capture instruction to be more explicit about detection triggers. **Step 6: Document in `memory-config.md`** Record your final schemas, the CLAUDE.md instructions you added, and evidence of both tests (action log query results and correction-driven behavior change). * [The Challenge](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#the-challenge) * [Acceptance Criteria](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#acceptance-criteria) * [Deliverables](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#deliverables) * [Use Case Gallery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#use-case-gallery) * [Hints](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/build-first-ai-employee/give-your-employee-a-memory#hints) --- # The Testing Loop | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Unix commands return exit codes for a reason; they tell the next command in the pipeline whether to proceed. But exit code 0 only means "didn't crash." It says nothing about whether the answer is right. Here's a script that sums numbers from stdin, just like yours. Run it: # buggy_sum.py - Sums numbers from stdinimport systotal = 0for line in sys.stdin: line = line.strip() if not line: continue if line[0] in '6789': continue total += float(line)print(f"Total: {total}") Challenge: Find the Bug Save this as `buggy_sum.py`. Run it on these numbers: echo -e "10\n20\n30" | python3 buggy_sum.py Output: `Total: 60.0`: looks correct. Now try: echo -e "10\n60\n30" | python3 buggy_sum.py What happens? Why? You have 90 seconds to figure it out before reading on. The output says `Total: 40.0`. The exit code is 0. No errors. No warnings. But the answer should be 100. The script silently skipped "60" because it starts with the digit '6'. Line 7: `if line[0] in '6789': continue`: drops any number starting with 6, 7, 8, or 9. And here's the uncomfortable part: **if you'd only tested with 10, 20, 30, you'd never have known.** The bug is invisible on "nice" test data. It only surfaces when your input includes numbers starting with high digits: exactly the kind of thing real financial data is full of. This is the verification paradox. You built sum.py in Lesson 1. It ran. It produced a number. Exit code 0. But you have no evidence it's correct: you've been trusting code you never proved works. In the File Processing chapter's lessons, you learned to verify backups before deleting originals: the Safety-First Pattern from Lesson 2 drilled the habit of "check before you act." The same instinct applies here, except the stakes are different. A missing file is obvious. A wrong number is invisible. The Trick[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#the-trick "Direct link to The Trick") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- What if you tested with numbers you CAN add in your head, but chose them deliberately to stress the code? 10 + 20 + 30 = 60. That's a start. But it only tests digits 1, 2, 3. Better: 10 + 60 + 30 = 100. Now you're testing a number starting with 6. Even better: test integers, decimals, AND negatives, because each type can trigger different bugs. Open Claude Code: You: I have sum.py that sums numbers from stdin. I want to verify itworks correctly. Create test data with known answers I can checkin my head. Include numbers starting with every digit 0-9, decimals,and negatives. Claude Code: Good idea -- thorough testing. Let me set that up.[Creates: test_simple.txt with contents: 10, 20, 30][Runs: cat test_simple.txt | python sum.py]Total: 60.0Matches expected (60). Let me also test with decimals:[Creates: test_decimals.txt with contents: 10.5, 20.5, 19.0][Runs: cat test_decimals.txt | python sum.py]Total: 50.0Matches expected (50). And negative numbers:[Creates: test_negative.txt with contents: 100, -30, -20][Runs: cat test_negative.txt | python sum.py]Total: 50.0All three cases verified:- Simple integers: 60.0 (correct)- Decimals: 50.0 (correct)- Negatives: 50.0 (correct) Notice the division of labor. You asked for one test, and the agent volunteered two more: decimals and negatives. The agent creates multiple test cases. But YOU chose the verification approach: "known answers I can check in my head." That's the human contribution the agent can't make on its own. It doesn't know which numbers are easy for you to verify mentally. But you can push this further. Instead of asking the agent to create test data for you to check, ask the agent to write the tests AND run them: You: Write a test script that generates edge-case data for sum.py(negatives, floats, empty lines, numbers starting with every digit 0-9)and asserts the output matches expected values. Run it yourself. Now the agent writes the tests, runs them, and reports pass/fail. Your job shifts from checking math in your head to reviewing the test plan: "Did it test the right edge cases?" That's a higher-leverage use of your attention, and closer to how real agentic workflows operate. Why Exit Code 0 Is a Lie[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#why-exit-code-0-is-a-lie "Direct link to Why Exit Code 0 Is a Lie") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You already saw this with `buggy_sum.py` in the challenge above. But let's make it explicit, because this distinction will save you from silent bugs for the rest of your career. Every command in Bash produces an exit code. Check it with `$?`: echo -e "10\n60\n30" | python3 buggy_sum.pyecho $? **Output:** Total: 40.00 Exit code 0. No errors. No warnings. The answer is wrong: it should be 100, but Bash says "success." | What Exit Code 0 Means | What Exit Code 0 Does NOT Mean | | --- | --- | | The script ran without crashing | The script produced the right answer | | Python didn't raise an exception | The logic is correct | | The process terminated normally | Your data is intact | **Exit codes catch crashes. They don't catch logic errors.** The buggy script from the challenge had perfect exit codes on every run. Only your test data (specifically, testing with numbers starting with 6, 7, 8, 9) exposed the bug. Common Exit Codes (Reference) | Code | Meaning | Example | | --- | --- | --- | | 0 | Success: didn't crash | Script ran, output appeared | | 1 | General error | Python raised an exception | | 127 | Command not found | Typo in script name | | 130 | Interrupted by Ctrl+C | You cancelled a long run | `$?` holds the exit code of the **most recent** command: run `echo $?` immediately after the command you care about. Why Your Totals Might Be Off by a Penny Python uses floating-point math, which can produce surprises: `0.1 + 0.2` gives `0.30000000000000004`, not `0.3`. For the amounts in this chapter, `round(total, 2)` handles it, and you'll verify the result against known answers anyway. If you ever need penny-perfect precision across thousands of transactions, tell Claude Code: "Use Python's Decimal module for exact arithmetic." For now, `float` plus your verification habit catches any drift before it matters. ### Your Python Scripts Return Exit Codes Too[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#your-python-scripts-return-exit-codes-too "Direct link to Your Python Scripts Return Exit Codes Too") The scripts you're building aren't just passive files; they're commands, and commands return exit codes. Right now, `sum.py` returns 0 when it works and 1 when Python raises an unhandled exception. But you can make that intentional: import systotal = 0.0lines_processed = 0for line in sys.stdin: line = line.strip() if line: total += float(line) lines_processed += 1if lines_processed == 0: print("Error: no numbers received", file=sys.stderr) sys.exit(1)print(f"Total: {total:.2f}")sys.exit(0) Two things changed: `sys.exit(1)` signals failure when no input arrived, and the error message goes to `sys.stderr`: a separate output stream from `sys.stdout`. That matters for pipes: stderr messages appear in your terminal without polluting the data flowing to the next command. echo "" | python3 sum.pyecho $? **Output:** Error: no numbers received1 Now `$?` means something. A wrapper script or automation tool can check it and know whether to proceed. This is what makes your Python scripts behave like real Unix commands, not just files that happen to run. The Verification Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#the-verification-pattern "Direct link to The Verification Pattern") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the prompt pattern that works every time: "Verify [tool] works correctly. Create test data with a known answer[X] and check that output matches." This works because: 1. **Known answer first.** You calculate the expected result before running the tool. 2. **Simple test data.** Numbers you can add in your head (10 + 20 + 30 = 60). 3. **Multiple cases.** Test integers, decimals, negatives, edge cases. 4. **Comparison.** Output must match expectation exactly. ### Pattern Variations[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#pattern-variations "Direct link to Pattern Variations") | What You're Testing | The Prompt | | --- | --- | | Sum script | "Verify sum.py with test data 10, 20, 30 (expected: 60)" | | Average script | "Verify average.py with test data 10, 20, 30 (expected: 20)" | | Max script | "Verify max.py with test data 10, 50, 30 (expected: 50)" | | Filter script | "Verify filter.py keeps only numbers > 20 from 10, 30, 50 (expected: 30, 50)" | The tool changes. The verification pattern stays the same. The division of labor here is worth noticing: the agent is fast at generating test cases and knows common failure modes (decimals, negatives, empty input). But only you know which answers are easy to verify in your head, and only you know that real bank data includes amounts starting with every digit 0-9. **You set the evidence criteria, the agent generates the evidence.** The human contribution isn't writing Python; it's knowing what "correct" looks like before the test runs. Checkpoint: Verify YOUR sum.py[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#checkpoint-verify-your-sumpy "Direct link to Checkpoint: Verify YOUR sum.py") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Stop reading. Create a file called `test_simple.txt` with three numbers: 10, 20, 30. Run your sum.py from Lesson 1 against it. Does it say 60? You: Create test_simple.txt with 10, 20, 30 on separate lines.Then run: cat test_simple.txt | python sum.pyExpected output: Total: 60.0 If it says 60 -- your script works for simple integers. Now try edge cases: You: Test sum.py with these edge cases:1. Empty file (expected: 0 or 0.0)2. Single number: just "42.5" (expected: 42.5)3. File with blank lines mixed in between numbers If any test fails, you've discovered a bug before it touched real data. Fix it now: Lesson 3 builds on a working sum.py. You now have a verified script and a verification habit. That habit: test with known answers, check the math yourself, never trust exit code 0: is more valuable than the script itself. The script handles numbers. The habit handles _everything you'll ever build_. Human provides the evidence criteria. Agent generates the code and tests. Neither alone can guarantee correctness, and that division is not a limitation. It's the primitive. Now try something. Download your actual bank statement as a CSV. Point sum.py at the amount column. Watch what happens when real-world data, with commas inside merchant names, dollar signs in amounts, and header rows that aren't numbers: hits a script that expects clean numbers, one per line. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#flashcards-study-aid "Direct link to Flashcards Study Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Discover Edge Cases[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#prompt-1-discover-edge-cases "Direct link to Prompt 1: Discover Edge Cases") What edge cases might break a script that sums numbers from stdin?Think about unusual inputs: empty files, non-numeric lines, verylarge numbers, special characters. List cases I should test. **What you're learning:** Getting the agent to surface failure modes you haven't considered. You ask for a list, not an answer. The agent contributes technical knowledge (dollar signs break float(), Unicode crashes, overflow is real). You contribute domain knowledge (which of these actually exist in your bank data). Between you, the test plan is more thorough than either could produce alone. ### Prompt 2: Automate Verification[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#prompt-2-automate-verification "Direct link to Prompt 2: Automate Verification") I have 5 test cases for sum.py. Help me write a simple bash scriptthat runs all tests and reports which passed and which failed. Eachtest should compare actual output to expected output. **What you're learning:** Test automation. Instead of manually running tests one at a time, you build a script that runs them all and reports results. This is how professionals keep code correct over time. ### Prompt 3: Debug a Failure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#prompt-3-debug-a-failure "Direct link to Prompt 3: Debug a Failure") My sum.py gives wrong output on this test:- Input: 10, 60, 30- Expected: 100- Actual: 40The script works fine on other inputs. Exit code is 0.Help me find the bug. What could cause 60 to be skipped? **What you're learning:** Collaborative debugging. You bring the evidence (expected: 100, actual: 40, exit code: 0). The agent brings mechanism knowledge (line\[0\] checks, digit ranges, why high-digit numbers get dropped). Neither can debug without the other: you needed to _observe_ that 60 was skipped; the agent needed to _know_ what code patterns produce that symptom. Checking access... * [The Trick](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#the-trick) * [Why Exit Code 0 Is a Lie](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#why-exit-code-0-is-a-lie) * [Your Python Scripts Return Exit Codes Too](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#your-python-scripts-return-exit-codes-too) * [The Verification Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#the-verification-pattern) * [Pattern Variations](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#pattern-variations) * [Checkpoint: Verify YOUR sum.py](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#checkpoint-verify-your-sumpy) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#flashcards-study-aid) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#try-with-ai) * [Prompt 1: Discover Edge Cases](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#prompt-1-discover-edge-cases) * [Prompt 2: Automate Verification](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#prompt-2-automate-verification) * [Prompt 3: Debug a Failure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/the-testing-loop#prompt-3-debug-a-failure) --- # Part 6: Building Agent Factories | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Part 6 bridges your foundational skills from Parts 1-5 into production agent development. You've learned Applied Domain Workflows (Part 4) and Python fundamentals (Part 5)—now you'll apply these skills to design, implement, test, and deploy AI agents for real-world applications. This part introduces agent frameworks, integration patterns, conversational infrastructure, testing strategies, and data persistence—the building blocks of production AI systems. You'll move from educational examples to patterns you'd encounter building professional agent applications. * * * Goals[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#goals "Direct link to Goals") ---------------------------------------------------------------------------------------------------------- By completing Part 6, you will: * **Understand agent architectures**: Learn how modern agent frameworks (OpenAI Agents SDK, Google ADK, Anthropic Agents Kit) structure agentic systems * **Implement integration patterns**: Practice connecting agents through MCP (Model Context Protocol) and building interoperable systems * **Build agent APIs**: Create FastAPI endpoints that expose agent capabilities as services * **Deploy conversational infrastructure**: Set up ChatKit Server for streaming, sessions, and conversation management * **Apply testing strategies**: Practice test-driven development (TDD) and evaluation frameworks (evals) for agent reliability * **Implement design patterns**: Build memory, state management, and compositional patterns for complex agent behaviors * **Integrate data persistence**: Connect agents to vector databases (RAG), relational databases, and graph databases * * * Chapter Progression[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#chapter-progression "Direct link to Chapter Progression") ---------------------------------------------------------------------------------------------------------------------------------------------------- Part 6's 16 chapters build through seven thematic stages: ### Agent Frameworks (Chapters 61-65)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#agent-frameworks-chapters-61-65 "Direct link to Agent Frameworks (Chapters 61-65)") Establish core agent concepts and framework-specific patterns. Start with agent fundamentals (61), then explore implementation across OpenAI (62), Google (63), and Anthropic (65) platforms using AIDD and spec-driven development. ### Integration Patterns (Chapters 66-68)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#integration-patterns-chapters-66-68 "Direct link to Integration Patterns (Chapters 66-68)") Learn how agents connect and communicate. Understand MCP fundamentals (66), develop custom MCP servers (67), and combine agent skills with MCP code execution for autonomous problem-solving (68). ### Agent Services (Chapters 70-71)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#agent-services-chapters-70-71 "Direct link to Agent Services (Chapters 70-71)") Transform agents into accessible services. FastAPI for Agents (70) teaches RESTful API patterns, then ChatKit Server (71) adds conversational infrastructure—streaming, sessions, and state management—with a built-in UI for immediate testing. ### Quality Practices (Chapters 72-73)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#quality-practices-chapters-72-73 "Direct link to Quality Practices (Chapters 72-73)") Validate agent correctness and reliability. Apply test-driven development patterns (72) and implement comprehensive evaluation frameworks (73) to ensure agents behave as specified. ### Advanced Patterns (Chapters 74-75)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#advanced-patterns-chapters-74-75 "Direct link to Advanced Patterns (Chapters 74-75)") Handle complexity at scale. Design effective memory and state management (74), and compose agents using multi-agent patterns (75). ### Data Layer (Chapters 76-78)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#data-layer-chapters-76-78 "Direct link to Data Layer (Chapters 76-78)") Add persistence and knowledge retrieval. Integrate vector databases for RAG (76), relational databases for structured data (77), and graph databases for relationship-rich knowledge (78). **Why this sequence?** You build conceptual understanding (what are agents?) before diving into integration (how do they connect?), then services (how do users access them?), quality (how do I validate them?), patterns (how do I scale them?), and data (how do I make them persistent?). Each stage builds on previous capabilities. **By the end of Part 6**, you have a complete local product: an agent with tools, exposed via API, with conversational infrastructure, tested, and connected to persistent data. Part 7 takes this product to the cloud. * * * Methodology Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#methodology-note "Direct link to Methodology Note") ------------------------------------------------------------------------------------------------------------------------------------------- Part 6 continues the teaching approach from earlier parts: chapters introduce concepts through hands-on practice, guide you in collaborating with AI to implement solutions, help you build reusable components when patterns recur, and culminate in capstone projects where you apply spec-driven development to compose what you've learned. You'll experience the same progression—foundation building, AI collaboration, creating reusable intelligence, and specification-first projects—now applied to production agent development rather than language fundamentals. * [Goals](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#goals) * [Chapter Progression](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#chapter-progression) * [Agent Frameworks (Chapters 61-65)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#agent-frameworks-chapters-61-65) * [Integration Patterns (Chapters 66-68)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#integration-patterns-chapters-66-68) * [Agent Services (Chapters 70-71)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#agent-services-chapters-70-71) * [Quality Practices (Chapters 72-73)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#quality-practices-chapters-72-73) * [Advanced Patterns (Chapters 74-75)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#advanced-patterns-chapters-74-75) * [Data Layer (Chapters 76-78)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#data-layer-chapters-76-78) * [Methodology Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories#methodology-note) --- # Connecting to Neon | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 5, you proved that transactions protect multi-step writes from partial corruption. Now your data needs to survive something bigger: your laptop closing, your process crashing, or your machine dying entirely. That is what cloud deployment gives you. Your SQLAlchemy models are ready. Your CRUD operations work. Your transactions roll back cleanly. All of that was built against a local database. Moving to Neon PostgreSQL in the cloud means your budget tracker keeps working even when your local environment does not. The models stay the same. The CRUD code stays the same. Only the connection string changes. Key Terms for This Lesson * **Connection pool**: A set of pre-opened database connections that your app reuses: instead of opening a new connection for every query (slow), you grab one from the pool and return it when done (fast) * **pool\_pre\_ping**: A health check that tests each connection before using it: catches "stale" connections that died while sitting in the pool * **DATABASE\_URL**: The connection string that contains everything needed to reach your database: driver, username, password, host, port, and database name, all in one line Why Cloud Changes the Game[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#why-cloud-changes-the-game "Direct link to Why Cloud Changes the Game") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Everything you built in Lessons 1 through 5 works on your machine. But "works on my machine" is not a deployment story. Cloud persistence means your data lives on a server you do not control, accessed over a network that can drop, through connections that can go stale. Three new problems appear: 1. **Credentials must never touch code.** Your DATABASE\_URL contains a password. If that ends up in a git commit, anyone who reads your repository has full access to your database. 2. **Connections die silently.** Your pool thinks it has 5 healthy connections. In reality, 2 of them timed out 10 minutes ago. Without health checks, your next query hits a dead connection and fails. 3. **Network is not free.** Every new connection takes time to establish. A pool reuses existing connections so your queries start fast. Any cloud-deployed app needs this pattern: a Django web app, a FastAPI service, a scheduled data pipeline. The connection pool plus secret management setup is the same regardless of what your app does. Setting Up Your Neon Account[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#setting-up-your-neon-account "Direct link to Setting Up Your Neon Account") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Pause here. Go to [neon.tech](https://neon.tech/) , create a free account (it takes about 2 minutes), and grab your connection string. We will wait. When you create a project in Neon, it gives you a connection string that looks like this: postgresql://username:password@ep-cool-name-123456.us-east-2.aws.neon.tech/dbname?sslmode=require Each part has a job: | Part | Example | What It Controls | | --- | --- | --- | | Driver | `postgresql://` | Which database protocol to use | | Username | `username` | Who you are authenticating as | | Password | `password` | Your authentication credential | | Host | `ep-cool-name-123456.us-east-2.aws.neon.tech` | Which server to connect to | | Database | `dbname` | Which database on that server | | SSL mode | `sslmode=require` | Encrypt traffic between your app and the server | Secret Management: The .env Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#secret-management-the-env-pattern "Direct link to Secret Management: The .env Pattern") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might be thinking: "Can I just paste the DATABASE\_URL directly in my code?" Technically, yes. And the first time you push to GitHub, your database password will be visible to the entire internet. Ask any developer who has been through that -- it is a bad afternoon. The fix is straightforward. Store secrets in a `.env` file that git never sees: **Step 1:** Create your `.env` file: echo 'DATABASE_URL=postgresql://username:password@ep-cool-name-123456.us-east-2.aws.neon.tech/dbname?sslmode=require' > .env Replace that example string with the actual connection string from your Neon dashboard. **Step 2:** Tell git to ignore it: echo '.env' >> .gitignore **Step 3:** Install the packages you need: uv add python-dotenv psycopg2-binary If your project uses `pip` instead of `uv`, replace that with `pip install python-dotenv psycopg2-binary`. **Step 4:** Tell your agent: "Load the database URL from my `.env` file. Never hardcode it. Raise a clear error if it is missing." Your agent writes the loading code. You verify the behavior: if the `.env` file is present, the connection loads silently. If it is missing, you see a clear error message instead of a cryptic database connection failure. That four-step checklist is your secret management baseline: 1. Put `DATABASE_URL` in `.env`. 2. Add `.env` to `.gitignore`. 3. Install `python-dotenv` and `psycopg2-binary`. 4. Load with `os.getenv()`, never with a string literal. Building the Connection Pool[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#building-the-connection-pool "Direct link to Building the Connection Pool") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now connect to Neon with a pooled engine that handles cloud realities: Connection Pool Architecture: Your App | v+--------------------+| Connection Pool || (QueuePool) || || [1] [2] | pool_pre_ping=True| [3] [4] |---- "Are you alive?" ---->| [5] pool_size | Neon Cloud| =5 |<--- "Yes!" ------------- PostgreSQL| || max_overflow=10 | pool_recycle=3600| Up to 10 extra |---- Replace connections| connections during| older than 1 hour| traffic spikes |+--------------------+ Here is how you direct your agent to set up that architecture: What you tell the agent Connect to my Neon PostgreSQL database using the DATABASE\_URL from my .env file. Use a connection pool with a health check that tests connections before using them and replaces any connection older than one hour. Start with a small pool (3 connections): I am on the free tier. Give me a health check command I can run to confirm the connection works. What you verify python verify_neon.pyOutput: Loading DATABASE_URL from .env... Connecting to Neon PostgreSQL... Running health check: SELECT 1 ✓ Connected. Neon is responding. Pool: 3 connections, health check active, recycle: 1 hour Connection verified. If you see that output, your app is talking to Neon through a healthy, pooled connection. Pause and Reflect Your data just moved from a file on your laptop to a server in the cloud. What changed about your database code? Nothing. Only the connection string changed. The agent's models, CRUD operations, and transactions are identical; they work with the local database and with Neon without modification. Deterministic Error Triage[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#deterministic-error-triage "Direct link to Deterministic Error Triage") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When something goes wrong with your Neon connection, random troubleshooting wastes time. Use this sequence instead -- work through errors in order, verifying each step before moving to the next: | Error | Most Likely Cause | First Check | | --- | --- | --- | | `password authentication failed` | Wrong or expired password | Rotate/reset password in Neon dashboard, update `DATABASE_URL` | | `could not connect to server` | Wrong host, missing SSL, or network block | Verify host string, confirm `sslmode=require`, check firewall | | `No module named psycopg2` | Driver not installed | Run `uv add psycopg2-binary` (or `pip install psycopg2-binary`) | | `server closed the connection unexpectedly` | Stale pooled connection | Confirm `pool_pre_ping=True` in engine config | | `remaining connection slots are reserved` | Too many open connections | Reduce `pool_size`, audit session lifecycle for leaks | The operational loop for each incident: 1. Copy the exact error text. 2. Run only the next most likely check from the table above. 3. Verify the check result. 4. Proceed to the next step only if the previous step is ruled out. That sequence prevents guess storms -- the pattern where you change three settings at once and have no idea which one fixed the problem (or made it worse). Deployment Sanity Sequence[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#deployment-sanity-sequence "Direct link to Deployment Sanity Sequence") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before you trust your connection in any real workload, run these five steps in order: 1. Run the health check script (the agent's `verify_neon.py`: confirms `SELECT 1` passes). 2. Direct the agent to create the schema on Neon. 3. Verify tables exist in the Neon SQL editor (log into neon.tech, open Tables view). 4. Direct the agent to store one row and read it back. 5. Restart your terminal and repeat the read: this confirms data is in the cloud, not just in local memory. Step 5 is the one people skip. It confirms persistence -- that your data survived a process restart and is actually stored in the cloud, not just in local memory. Connection Sizing Rules[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#connection-sizing-rules "Direct link to Connection Sizing Rules") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Start small unless you have evidence to go bigger: * Begin with `pool_size=3`, `max_overflow=5` on free or low-tier Neon plans. * Increase only after observing real concurrency pressure in your logs. * Always pair pool growth with leak checks. Leak check heuristic: if connection slot usage climbs during low traffic, inspect session lifecycle first. Ensure every operational path uses context-managed sessions (the `with` pattern). Avoid long-lived global sessions in request-oriented code. Most "connection limit" incidents are lifecycle bugs, not raw traffic volume. Before scaling pool settings, gather one day of connection metrics. Evidence should drive tuning, not default guesses. Minimal observability additions for production: * Log connection health checks during startup. * Log pool exhaustion errors with timestamps. * Log retry counts for transient connection failures. These three logs make production debugging faster and reduce blame-driven guesswork. **What breaks next?** Connection reliability alone does not protect report correctness. The next lesson decides when SQL-only is enough and when independent verification is worth the extra cost. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Connection String Audit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#prompt-1-connection-string-audit "Direct link to Prompt 1: Connection String Audit") Parse this DATABASE_URL into its component parts:- driver- user- password- host- database- query paramsThen explain what each part controls operationally and what breaks if that part is wrong. **What you're learning:** A DATABASE\_URL looks like one opaque string, but it is actually six distinct configuration decisions packed together. Understanding each part means you can diagnose connection failures by reading the error message instead of guessing randomly. ### Prompt 2: Deterministic Incident Runbook[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#prompt-2-deterministic-incident-runbook "Direct link to Prompt 2: Deterministic Incident Runbook") Given error "remaining connection slots are reserved",give me a step-by-step triage order with a verification check after each step.Do not give parallel guesses. Each step must confirm or rule out one cause before moving to the next. **What you're learning:** Deterministic triage is a debugging discipline. Instead of changing three settings and hoping something works, you isolate one variable at a time. This skill transfers to every production system you will ever debug -- not just databases. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") You're deploying your own app to the cloud. What secrets does it need? (Database URL, API keys, tokens?) Design a secret management checklist: where each secret lives, how it's loaded, and what happens if someone accidentally commits it to git. Include your recovery steps. **What you're learning:** Secret management is a universal DevOps skill. Every cloud-deployed app -- whether it is a budget tracker, a SaaS product, or an AI agent -- needs credentials that must NEVER touch version control. The .env + .gitignore pattern you learned here is your first line of defense. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#safety-note "Direct link to Safety Note") Never commit a `.env` file or paste database credentials directly into source code. If you accidentally push credentials to a public repository, rotate them immediately in your Neon dashboard -- do not assume "nobody saw it." Credential scanners run continuously on public GitHub repositories, and exposed passwords are typically exploited within minutes. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#checkpoint "Direct link to Checkpoint") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I created a Neon account and have my connection string ready. * [ ] My connection string is in a `.env` file and `.env` is in `.gitignore`. * [ ] I directed the agent to configure the Neon connection and read the health check output confirming `SELECT 1` passed. * [ ] I ran the deployment sanity sequence: health check → schema creation → store-and-read → restart and re-read. * [ ] I know what to tell the agent if I see "server closed the connection unexpectedly." Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Why Cloud Changes the Game](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#why-cloud-changes-the-game) * [Setting Up Your Neon Account](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#setting-up-your-neon-account) * [Secret Management: The .env Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#secret-management-the-env-pattern) * [Building the Connection Pool](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#building-the-connection-pool) * [Deterministic Error Triage](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#deterministic-error-triage) * [Deployment Sanity Sequence](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#deployment-sanity-sequence) * [Connection Sizing Rules](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#connection-sizing-rules) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#try-with-ai) * [Prompt 1: Connection String Audit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#prompt-1-connection-string-audit) * [Prompt 2: Deterministic Incident Runbook](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#prompt-2-deterministic-incident-runbook) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#prompt-3-apply-to-your-domain) * [Safety Note](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#safety-note) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/connecting-to-neon#flashcards-study-aid) --- # When Bash and Python Hit the Wall | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C **Continuity delta:** The File Processing chapter gave you file control. The Computation & Data Extraction chapter gave you deterministic computation. This chapter adds durable relational memory. In this chapter your agent will write all the database code. Your job is to make architecture decisions and verify the results. You built `tax-prep.py` in the Computation & Data Extraction chapter. It was your best work -- a clean script that reads CSV files, computes totals, and produces an accurate yearly tax report. You ran it, the numbers checked out, and you felt that satisfaction of a job done right. Now imagine your manager walks in on Monday morning and says: "Great report. Can you break it down by month, by user, by category? And we need it for the last three years. Oh, and make sure nobody can delete a user who still has expenses tied to them." Your stomach drops. Not because the requirements are unreasonable, but because you can already see what happens next: you will spend the rest of the week writing loops. One loop to filter by month. Another to filter by user. Another to cross-reference categories. Another to check for orphaned records before deletes. Each loop works in isolation. Together, they become a tangle of special cases that nobody (including you, three months from now) wants to debug. > "If every new question requires a new loop, your data model is already failing." Key Terms for This Lesson * **Schema**: A formal blueprint that defines what columns exist, what types they hold, and how tables relate -- like a building's floor plan vs a pile of lumber * **Foreign key**: A column that points to a row in another table, creating an enforceable link -- like a shipping label that guarantees a package reaches its destination * **Relational database**: A system that stores data in linked tables with enforced rules, so every query draws from one consistent truth The Exact Moment It Breaks[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#the-exact-moment-it-breaks "Direct link to The Exact Moment It Breaks") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Computation & Data Extraction script handles one question just fine, for example, computing all Medical deductions and returning `Medical deductions: $2,847.50`. Clean. Correct. Now stack on the real requirements: * "Show Food spending for Alice in March 2024." * "Compare Q1 vs Q2 by category." * "Do this for 4 users across 3 years." * "Guarantee no orphaned relationships after deletes." At this point, you are writing database behavior by hand in Python loops. Each new question means a new `for` loop with a new combination of filters. You might be thinking: "But my script handles this fine!" And you are right -- for now. The problem is not that it breaks today. The problem is that it breaks silently, one edge case at a time. That hand-built logic can still look correct in a demo. The risk appears later: * one query branch forgets a filter * another script handles dates differently (calendar month vs rolling 30 days) * a third script assumes category names are unique forever All three pass basic tests but diverge under real usage. We have all been there: 3am, fourth version of the same filtering loop, wondering if there is a better way. (There is.) This pattern shows up beyond budget tracking. Project management tools hit the same wall when task assignments span multiple teams. E-commerce platforms hit it when order history crosses customers, products, and returns. Any time your data has _relationships_, loops start losing to structure. The Numbers That Changed Everything Braintrust and Vercel tested SQL against Bash-based agents for structured data queries on a dataset of GitHub issues and pull requests: * **SQL agent**: 100% accuracy, fewer tokens, fastest execution * **Bash agent**: ~53% accuracy, 7x more tokens, 9x slower Same data. Same questions. The right tool is not just faster -- it is correct. _Source: ["Testing if bash is all you need"](https://vercel.com/blog/testing-if-bash-is-all-you-need) , Vercel Engineering Blog / Braintrust, 2025._ Two Worlds Side by Side[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#two-worlds-side-by-side "Direct link to Two Worlds Side by Side") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CSV World: Relational World:┌──────────────────┐ ┌─────────┐│ expenses-2024.csv│ │ users ││ expenses-2025.csv│ │ (id, email, name)│ expenses-2026.csv│ └────┬─────┘│ users.csv │ │ FK│ categories.csv │ ┌────┴─────┐└──────────────────┘ │ expenses │ │ (id, user_id, 5 files, no enforced links │ category_id, Hope they stay consistent │ amount, date) └────┬─────┘ │ FK ┌────┴──────┐ │ categories │ │ (id, name) │ └────────────┘ Enforced links, one truth On the left: five files that _hope_ they stay consistent. On the right: three tables where the database _enforces_ consistency. That enforcement is the difference between "it worked in the demo" and "it works in production." One Core Win[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#one-core-win "Direct link to One Core Win") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Move from scattered files to one relational contract. **8-minute visible win:** draw your current CSV entities and link them with explicit foreign keys on paper. If links are unclear, your future queries are unclear. Why this wins: * **Schema defines meaning.** A column named `user_id` in the expenses table is not just a number -- it is a promise that a matching user exists. * **Constraints enforce allowed states.** Try to insert an expense for a user that does not exist, and the database says "no." Your Python loop would silently create an orphan. * **SQL answers new questions without rewriting loops.** "Show Food spending for Alice in March 2024" becomes one query, not one function. * **Transactions keep writes consistent during failure.** If a multi-step update crashes halfway, the database rolls everything back. Your script leaves half-written state. Practical shift: * The Computation & Data Extraction chapter asked, "Can I compute this report?" * This chapter asks, "Can I keep this truth stable across many reports and writes?" Quick escalation matrix: | Situation | Stay with Computation & Data Extraction pattern | Escalate to Structured Data pattern | | --- | --- | --- | | One user, one monthly file, one known report | Yes | No | | Multiple users need shared history | No | Yes | | New ad-hoc questions arrive weekly | No | Yes | | Data edits/deletes must stay consistent | No | Yes | | Financial output drives external decisions | Maybe | Yes, with verification policy | Use this matrix as a decision aid, not ideology. The point is to reduce hidden maintenance cost before it becomes operational pain. Pause and Reflect Think about your own Computation & Data Extraction work. How many different filtering loops did you write? If a new question arrived tomorrow, would you write yet another loop -- or is there a pattern emerging? One Common Failure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#one-common-failure "Direct link to One Common Failure") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Treating benchmark headlines as dogma. Correct takeaway from the Braintrust/Vercel work: 1. SQL is the primary path for structured querying. 2. Hybrid verification (SQL + independent check) matters for high-stakes outputs, not every query. 3. Re-running the same SQL is not independent verification -- you need a separate computation path. Another common mistake is escalating too late. Teams keep adding loop after loop because each new request feels small. The total system becomes large and fragile long before anyone names it. Failure signal checklist: * same filter logic copied into 3+ scripts * conflicting definitions of "monthly" (calendar month vs rolling 30 days) * missing ownership rules for category/user links * manual cleanup needed after failed writes * reporting disputes that cannot be traced to one source of truth **What breaks next?** You now know why the model must change. Next lesson proves persistence across separate runs before any cloud setup. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Breakpoint Diagnosis[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#prompt-1-breakpoint-diagnosis "Direct link to Prompt 1: Breakpoint Diagnosis") My Computation & Data Extraction script reads CSV and computes totals.Now I need:1) multi-user history2) month/category/user filtering3) safe edits and deletes4) no orphaned referencesFor each requirement:- explain why script+CSV gets brittle- name the exact relational feature that solves it **What you're learning:** You are mapping each pain point in your current workflow to a specific database feature. This builds the vocabulary you will use throughout this chapter -- foreign keys, constraints, transactions -- and shows you that each feature exists to solve a real problem, not as academic overhead. ### Prompt 2: Escalation Rule[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#prompt-2-escalation-rule "Direct link to Prompt 2: Escalation Rule") Give me a decision rule for when to stay with Computation & Data Extraction patternsversus when to escalate to SQLAlchemy + PostgreSQL.Use concrete examples, not generic advice.Include at least one case where staying with CSV is the right call. **What you're learning:** You are building judgment about _when_ to escalate, not just _how_. The best engineers do not reach for the biggest tool first -- they match tool complexity to problem complexity. This prompt forces you to articulate the boundary. ### Prompt 3: Domain Transfer[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#prompt-3-domain-transfer "Direct link to Prompt 3: Domain Transfer") I work in [describe your domain: e-commerce, project management,healthcare records, inventory tracking, etc.].Analyze my domain the way we analyzed the budget tracker:- What are my "CSV files" (the current data sources)?- Where are the hidden relationships between them?- What question would break my current approach?- Draw the escalation moment for my specific case. **What you're learning:** You are transferring the budget tracker pattern to your own work. The escalation from scripts to databases is not specific to expense tracking -- it applies anywhere structured data has relationships. Recognizing the pattern in your domain is how this lesson becomes permanent knowledge. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#checkpoint "Direct link to Checkpoint") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I can explain why Python loops stop scaling for evolving structured queries. * [ ] I can explain why schema reduces ambiguity and bug surface. * [ ] I can name one FK rule that prevents invalid relationships. * [ ] I can state when hybrid verification is optional vs required. * [ ] I can articulate the Computation & Data Extraction to Structured Data escalation trigger in one sentence. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#flashcards-study-aid "Direct link to Flashcards Study Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Exact Moment It Breaks](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#the-exact-moment-it-breaks) * [Two Worlds Side by Side](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#two-worlds-side-by-side) * [One Core Win](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#one-core-win) * [One Common Failure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#one-common-failure) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#try-with-ai) * [Prompt 1: Breakpoint Diagnosis](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#prompt-1-breakpoint-diagnosis) * [Prompt 2: Escalation Rule](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#prompt-2-escalation-rule) * [Prompt 3: Domain Transfer](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#prompt-3-domain-transfer) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/from-csv-to-databases#flashcards-study-aid) --- # Build Your Database Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 0, you identified exactly when the Computation & Data Extraction chapter's file-based workflows hit the wall. Now you will prove -- in under five minutes -- that a database actually solves the core problem: data that survives after your program exits. Close the program. Open it again. Is your data still there? That single question separates a calculator from a system. A Python dictionary holds data beautifully while your script runs. The moment you press Ctrl+C or the terminal window closes, everything vanishes. We have all lost work to a closed terminal. Ctrl+C should not mean goodbye to your data. You might be thinking: "Can't I just save to a JSON file?" You can. But try adding relationships, concurrent writes, and query flexibility to a JSON file. That is where things get interesting -- and where databases earn their place. Key Terms for This Lesson * **Persistence**: Data that survives after your program exits -- close the terminal, reboot, come back tomorrow, your data is still there * **Process boundary**: The moment one program ends and a new one begins -- the database file survives this; Python variables do not The Mistake That Wastes Hours[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#the-mistake-that-wastes-hours "Direct link to The Mistake That Wastes Hours") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Many learners jump directly to Neon cloud setup. They skip the basic question: does data actually persist across independent runs on my own machine? If you skip this check, you can spend hours debugging cloud configuration while the real issue is local workflow discipline. The cloud does not fix a broken mental model -- it just moves the confusion somewhere harder to debug. The second failure pattern hits a month later. You kept all the knowledge in your head instead of capturing it. You restart from scratch and repeat the same mistakes. (That is why the second half of this lesson builds a reusable skill scaffold.) One Core Win: The Persistence Proof[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#one-core-win-the-persistence-proof "Direct link to One Core Win: The Persistence Proof") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your goal: run a two-script persistence proof in under 5 minutes. Script A writes one row. Script B -- launched in a completely separate process -- reads it back. If Script B sees the data, you have proven persistence across process boundaries. Here is what you are proving with this test: Process Boundary Proof: Terminal 1 Terminal 2 ┌─────────────┐ ┌─────────────┐ │ write_once.py│ │ read_later.py│ │ │ │ │ │ saves data │ │ reads data │ │ to disk │ │ from disk │ │ │ │ │ └──────┬───────┘ └──────┬───────┘ │ │ │ ┌──────────────────┐ │ └──► quick_persist.db ◄──────┘ │ (on disk) │ └──────────────────┘ Process exits. New process starts. Data stays. Data is there. Two separate processes. One shared database file on disk. The first process is long gone by the time the second one starts. That is persistence. What you tell the agent Prove that data survives after the program closes. Write two scripts: * write\_once.py: saves a marker called "persistent-check" to a local database, then exits * read\_later.py: opens a fresh connection and reads back all markers I should be able to run write\_once.py, then close that program completely, then run read\_later.py in a new terminal and see my marker. What you verify The agent writes both scripts. Now you run them: python write_once.py Output: Wrote marker python read_later.py Output: ['persistent-check'] That list printed from a brand-new process that never saved the marker. The data survived because it lives on disk, not in Python's memory. This pattern is not limited to budget trackers. Imagine a TODO app where closing the browser does not lose your tasks. Or an IoT sensor logger where temperature readings survive power outages. The principle is identical: write in one process, read in another, trust the result. ### When the Proof Fails[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#when-the-proof-fails "Direct link to When the Proof Fails") If read\_later.py shows an empty list, tell the agent: "The read script returned empty -- find what's missing." The agent will check: 1. Both scripts point to the same database file path 2. The write script actually commits data to disk (without a commit, nothing is saved) 3. The read script uses the same table name and data mapping 4. Whether a stale database file is causing confusion (delete it and rerun both scripts) Pause and Reflect You just proved data survives across process boundaries. How is this different from a Python dictionary that holds data while your script runs? What happens to a dictionary when the script exits? Capture the Pattern: Your Skill Scaffold[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#capture-the-pattern-your-skill-scaffold "Direct link to Capture the Pattern: Your Skill Scaffold") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The persistence proof works. Now capture it so you never have to rediscover it. A common failure here: creating notes instead of an execution interface. A useful skill must include when to use it, decision logic for tool choice, and guardrails for failures and secrets. If a skill file reads like a textbook chapter, it will be ignored during real incidents when pressure is high. Your skill will grow across this chapter: * L1 adds persistence proof and baseline guardrails * L3 adds CRUD session patterns and rollback discipline * L4 adds relationship and join rules * L5 adds transaction failure handling patterns * L6 adds Neon connection and secret handling * L7 adds independent verification policy * L8 adds release evidence bundle contract By the capstone, your skill should be short but decisive: it should tell a future you what to do first when pressure is high. Beginner priority order when a new database project starts: 1. Prove persistence 2. Define schema contract 3. Implement CRUD baseline 4. Add transaction safety 5. Deploy with secret and connection discipline 6. Add risk-based verification gate Following this sequence prevents premature complexity and improves learning speed. ### Appendix: Lean Skill Scaffold[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#appendix-lean-skill-scaffold "Direct link to Appendix: Lean Skill Scaffold") ---name: database-deploymentdescription: Build persistent data layers with SQLAlchemy + PostgreSQL (Neon).---- Persona: I build systems where data must remain correct across restarts and failures.- When to use: - Structured data with relationships - Multi-user persistence - Query-heavy workflows- Decision logic: - One-off local script -> Computation & Data Extraction chapter style - Persistent multi-user app -> Structured Data style - High-stakes financial report -> Structured Data + independent verification- Guardrails: - Never hardcode DB credentials - Always rollback failed writes - Never call same-path rechecks "independent verification" **What breaks next?** Persistence exists now, but schema quality decides whether future queries are reliable or misleading. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Persistence Proof[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#prompt-1-persistence-proof "Direct link to Prompt 1: Persistence Proof") Prove that data survives after the program closes. Write two scripts:one that saves a marker to a local database, one that reads it backin a separate run. Explain why this proves data outlives the processthat created it. **What you're learning:** You are reinforcing the core mental model -- persistence means data outlives the process that created it. Asking the agent to explain "why" forces you to verify your own understanding against a second perspective. If the explanation surprises you, that is a learning signal. ### Prompt 2: Skill Skeleton[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#prompt-2-skill-skeleton "Direct link to Prompt 2: Skill Skeleton") Draft a concise SKILL.md for /database-deployment with exactly:Persona, When to Use, Decision Logic, Guardrails.Keep each section operational and beginner-friendly. **What you're learning:** You are practicing pattern capture -- turning experiential knowledge into a reusable artifact. The agent drafts the structure; you refine it with your actual project context. Notice whether the decision logic matches your real workflow or needs correction. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") Think of a project you're working on (or want to build). What datawould you need to persist across restarts? Describe what a two-steppersistence proof would look like for YOUR domain -- what wouldwrite_once.py save, and what would read_later.py retrieve? **What you're learning:** The persistence proof pattern transfers to ANY domain. Whether it is user profiles, sensor readings, or game saves -- the principle is the same: write in one process, read in another, trust the result. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#checkpoint "Direct link to Checkpoint") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I can explain why a Python dictionary is NOT persistent (it dies when the process exits) * [ ] I directed the agent to write a two-script persistence proof and verified the output showed my marker surviving * [ ] I know what to tell the agent if read\_later.py returns an empty list * [ ] I created a lean /database-deployment skill scaffold with decision logic and guardrails Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Mistake That Wastes Hours](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#the-mistake-that-wastes-hours) * [One Core Win: The Persistence Proof](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#one-core-win-the-persistence-proof) * [When the Proof Fails](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#when-the-proof-fails) * [Capture the Pattern: Your Skill Scaffold](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#capture-the-pattern-your-skill-scaffold) * [Appendix: Lean Skill Scaffold](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#appendix-lean-skill-scaffold) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#try-with-ai) * [Prompt 1: Persistence Proof](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#prompt-1-persistence-proof) * [Prompt 2: Skill Skeleton](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#prompt-2-skill-skeleton) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#prompt-3-apply-to-your-domain) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/build-your-database-skill#flashcards-study-aid) --- # Relationships & Joins | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 3, you proved that rows exist by creating and reading them back. Now you will prove that rows _belong together_ correctly. A user's expenses should belong to that user, not float around unattached. A category should link to its expenses so you can ask "show me all Food purchases" in a single query. Here is the trap: your app runs fine with 5 users. Then you get 500 users, and suddenly it is making 501 database calls instead of 2. Nobody changed any code. What happened? That is the N+1 problem, and by the end of this lesson you will know exactly how to spot it and direct your agent to fix it. Key Terms for This Lesson * **Relationship**: A link between two types of data -- like a contact card that connects a person to their company. If users have expenses, there is a relationship between them. * **Foreign key**: An enforceable promise that one piece of data references another that actually exists. An expense's user reference is a foreign key -- the database refuses to create an expense pointing to a user that does not exist. * **Cascade**: When deleting one thing automatically removes related things. Deleting a user can cascade to remove their expenses. Whether it _should_ is a business decision, not a technical default. * **N+1 problem**: A hidden performance trap where loading 100 parents triggers 100 separate child queries instead of 1 -- your app works fine until it suddenly does not. * * * How to Ask for Linked Data[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#how-to-ask-for-linked-data "Direct link to How to Ask for Linked Data") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you need data that spans multiple entities, describe it in business terms. The agent translates your intent into the correct database operations. | What you want | How to say it to the agent | | --- | --- | | All expenses for one user | "Show me all expenses for Alice" | | Expenses filtered by category | "Show me only Food expenses for Alice" | | Data from two entities together | "For each expense, show me the category name and the user's email" | | Users with at least one expense | "Only include users who have at least one expense" | | All users, even with no expenses | "Include all users, even those with no expenses yet" | | Grouped totals | "For each category, show me the total amount and number of expenses" | The key phrases are **"for each"** (tells the agent to link entities), **"only include"** (tells the agent to filter), and **"must reference"** (tells the agent to enforce that a link is valid). You do not need to know SQL or any programming language to describe these relationships -- you need to know what data belongs together and what questions you want answered. ### When to Use What[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#when-to-use-what "Direct link to When to Use What") | Need | How to say it to the agent | | --- | --- | | All expenses for one loaded user | "For Alice, show me all her expenses" | | All expenses where category is Food | "Filter expenses where category name is Food" | | A report grouped by category | "Group all expenses by category with totals" | The first request starts from a specific person and navigates to their data. The second request starts from all expenses and filters by a linked attribute. The third request aggregates across all data. These are different operations and your phrasing tells the agent which one you need. * * * Directing a Basic Linked Query[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#directing-a-basic-linked-query "Direct link to Directing a Basic Linked Query") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What you tell the agent Show me all expenses for Alice in March 2024, grouped by category. Include the category name and the total amount per category. Sort by highest total first. What you verify python verify_join.pyOutput: Alice's March 2024 expenses by category: Food: $287.45 (3 expenses) Transport: $94.20 (2 expenses) Entertainment: $45.00 (1 expense) Total: $426.65 What to check: the categories are real categories from your data, the totals add up, and the sort order is correct (highest first). If the agent shows categories that Alice has no expenses in, the filtering is wrong. If the total of the per-category amounts does not match the grand total, the grouping is wrong. * * * Directing a Filtered Linked Query[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#directing-a-filtered-linked-query "Direct link to Directing a Filtered Linked Query") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ What you tell the agent Show me only Food expenses over $50 for any user. Include the user's name with each expense. What you verify python verify_food_filter.pyOutput: Food expenses over $50: - Alice: Groceries $52.50 (2024-01-15) - Bob: Weekly shop $78.20 (2024-01-18) 2 results found. What to check: every result is in the Food category (not Transport, not Entertainment), every amount is over $50, and each result includes the user's name. If you see expenses from other categories, the agent's filter is wrong. If you see amounts under $50, the threshold is wrong. * * * Pause and Reflect If linked queries make sense, continue to the N+1 section. If not, re-read the vocabulary table above and try directing the agent to run one simple linked query before moving on. * * * The N+1 Problem[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#the-n1-problem "Direct link to The N+1 Problem") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might be thinking: "Why would the agent's code make too many database calls?" Because the most natural-looking approach -- loop through each user and grab their expenses -- is also the most wasteful. N+1 Problem (100 users):Without optimization: With optimization:+------------------+ +------------------+| Get all users | 1 query | Get all users | 1 query+--------+---------+ +--------+---------+ | | +----+----+ +----+----+ | For each| | Get ALL | | user... | | expenses | +----+----+ | in one | | | query | 1 query +------+------+ +----------+ | | | v v v Total: 2 queriesquery query query vs 101 queries! 1 2 ...100Total: 101 queries If the agent writes code that loops through users and makes a separate database call for each user's expenses, you get N+1: 1 call to get users + 1 call per user = 101 calls for 100 users. This works fine with 5 users and becomes slow with 500. The performance signal to monitor: if your summary gets slower as user count grows, tell the agent to check for N+1 patterns. A loop that fetches related data one-by-one is always suspicious. ### Directing the Fix[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#directing-the-fix "Direct link to Directing the Fix") Tell the agent: "You're making one database call per user to get their expenses. Fix this to get all expenses in a single call instead." The agent will rewrite it to use a prefetch that reduces 101 calls to 2 calls, regardless of user count. What you tell the agent I notice your summary loop makes one database call per user to get their expenses. For 100 users that's 101 database calls. Fix this so it uses 2 calls total, no matter how many users there are. What you verify python verify_query_count.pyOutput: Before fix: 101 queries for 100 users After fix: 2 queries for 100 users N+1 resolved What to check: the "before" number should be roughly N+1 (where N is the user count). The "after" number should be 2, regardless of how many users you have. If the "after" number still scales with user count, the fix did not work. * * * Cascade: A Business Decision[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#cascade-a-business-decision "Direct link to Cascade: A Business Decision") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you define relationships, you also decide what happens when data is deleted. This is cascade policy, and it is a business decision: * **Delete a user, delete their expenses**: Makes sense. An expense without a user is meaningless. * **Delete a category, delete all expenses in that category**: Dangerous. Expenses belong to users too, and other categories would lose data. * **Delete a user, keep their comments on a shared forum**: Probably correct. The comments have value to other readers even without the original author. When you tell the agent about your relationships, include the cascade rule: "When a user is deleted, their expenses should be deleted automatically. When a category is deleted, expenses in that category should NOT be deleted -- just remove the category link." * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Setup:** Open your AI coding agent with the budget tracker project from this chapter. ### Prompt 1: Basic Linked Query[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#prompt-1-basic-linked-query "Direct link to Prompt 1: Basic Linked Query") Here's what I need: show me all expenses for each user, with the user'sname and category name for each expense. Build the query and show mewhat the output looks like for 3 sample users with 2-3 expenses each. **What you're learning:** You are practicing the skill of describing linked data needs in business terms. The agent handles the implementation -- your job is to verify the output makes sense: correct user-expense pairings, correct category names, no missing or duplicated data. ### Prompt 2: N+1 Detection and Fix[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#prompt-2-n1-detection-and-fix "Direct link to Prompt 2: N+1 Detection and Fix") I think your code is making one database call per user. For 100 users,show me how many total calls it makes, then fix it to use 2 calls totaland show the new call count. **What you're learning:** You are building the instinct to question performance. When you see query counts that scale with data size, that is a signal to direct the agent to optimize. The agent knows _how_ to fix it -- you need to know _when_ to ask. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") I'm building [your project]. I need to show [linked data from two entities].Describe what query you'll use, how many database calls it makes, and showme a sample output. **What you're learning:** You are moving from following examples to making relationship decisions for your own domain. The agent helps with implementation, but the business logic -- what data belongs together and what happens when something is deleted -- is your decision to make. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#checkpoint "Direct link to Checkpoint") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I can describe entity relationships in plain English: "show all expenses for Alice" and "group by category with totals" * [ ] I can ask for filtered joined data: "only Food expenses over $50, with the user's name" * [ ] I can recognize an N+1 pattern from query count output and tell the agent to fix it * [ ] I can verify linked query results by reading the output, not by reading the Python code Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [How to Ask for Linked Data](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#how-to-ask-for-linked-data) * [When to Use What](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#when-to-use-what) * [Directing a Basic Linked Query](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#directing-a-basic-linked-query) * [Directing a Filtered Linked Query](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#directing-a-filtered-linked-query) * [The N+1 Problem](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#the-n1-problem) * [Directing the Fix](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#directing-the-fix) * [Cascade: A Business Decision](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#cascade-a-business-decision) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#try-with-ai) * [Prompt 1: Basic Linked Query](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#prompt-1-basic-linked-query) * [Prompt 2: N+1 Detection and Fix](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#prompt-2-n1-detection-and-fix) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#prompt-3-apply-to-your-domain) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/relationships-joins#flashcards-study-aid) --- # Setting Up Your Agent's Home | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Ali can read the server now. Permissions, processes, disk usage -- the cryptic output makes sense. But reading the server revealed an uncomfortable truth: his agent's files are a mess. The Python script sits in his home directory next to vacation photos and half-finished homework. The database password is hardcoded on line 14 of the source code. Log output goes to the terminal and vanishes every time he closes it. The agent works, but it is a house of cards. One wrong `rm` command, one curious coworker glancing at his screen, one disconnected SSH session -- and something breaks. Agents dumped in random directories fail. Not because of bad code -- because of bad organization. This lesson creates order from chaos. > **"A well-organized server is a well-organized mind."** The Problem With "It Works on My Desk"[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#the-problem-with-it-works-on-my-desk "Direct link to The Problem With "It Works on My Desk"") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Picture a kitchen where the knives are in the bathroom, the spices are under the bed, and the recipe is tattooed on the chef's arm. The food might taste fine today. But the next chef will burn the place down. Ali's agent has the same problem: /home/ali/├── competitor_tracker.py # Agent code (mixed with personal files)├── vacation_photo.jpg├── homework_draft.docx├── notes.txt└── (database password is on line 14 of the .py file) No separation between code, data, configuration, and secrets. Everything in one pile. Building the Structure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#building-the-structure "Direct link to Building the Structure") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A well-organized server is a well-organized mind. Every professional deployment follows the same pattern -- dedicated directories for code, configuration, data, and logs. Like a well-organized office: documents in the filing cabinet, secrets in the safe, memos in the outbox. Here is what Ali needs: /opt/agents/competitor-tracker/├── src/ # Agent code lives here├── config/ # Configuration files├── logs/ # Persistent log output├── data/ # Database files, cache, downloads└── .env # Secrets (DB password, API keys) Direct Claude Code to build it: Create a directory structure for my competitor-tracker agent at/opt/agents/competitor-tracker/ with subdirectories: src, config,logs, and data. Then move my agent script from ~/competitor_tracker.pyinto the src/ directory. Claude Code: I'll create that structure and move your script.> sudo mkdir -p /opt/agents/competitor-tracker/{src,config,logs,data}> sudo chown ali:ali /opt/agents/competitor-tracker> mv ~/competitor_tracker.py /opt/agents/competitor-tracker/src/Done. Your agent now lives at:/opt/agents/competitor-tracker/src/competitor_tracker.pyThe directory structure is:/opt/agents/competitor-tracker/├── src/ (1 file: competitor_tracker.py)├── config/ (empty, ready for config files)├── logs/ (empty, ready for log output)└── data/ (empty, ready for database/cache) **Output:** $ ls -la /opt/agents/competitor-tracker/drwxr-xr-x 6 ali ali 4096 Feb 27 10:15 .drwxr-xr-x 3 root root 4096 Feb 27 10:15 ..drwxr-xr-x 2 ali ali 4096 Feb 27 10:15 configdrwxr-xr-x 2 ali ali 4096 Feb 27 10:15 datadrwxr-xr-x 2 ali ali 4096 Feb 27 10:15 logsdrwxr-xr-x 2 ali ali 4096 Feb 27 10:15 src Why `/opt/agents/`? The `/opt` directory is Linux convention for optional software that does not come with the operating system. It is not buried in a user's home folder. Other team members can find it. Other agents can be added beside it: `/opt/agents/social-monitor/`, `/opt/agents/report-generator/`. Extracting Secrets[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#extracting-secrets "Direct link to Extracting Secrets") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now for the sticky note on the whiteboard problem. Open Ali's script and look at line 14: # Line 14 of competitor_tracker.pydb_password = "r3allyS3cret!Pass"api_key = "sk-proj-abc123def456" Secrets don't belong in code. They belong in a separate file that never gets shared. Think of it like writing the safe combination on a sticky note you keep in your wallet -- not on the whiteboard where everyone can see it. Pause. Why does this matter? Three reasons: 1. **Git.** If Ali ever pushes this code to GitHub, those credentials are public. Bots scan GitHub for exposed keys. His database gets compromised within minutes. 2. **Teammates.** Anyone who reads the code now knows the password. The code should be shareable. The secrets should not. 3. **Different environments.** The test server uses a different database than production. Hardcoded passwords mean changing the code every time you switch environments. Direct Claude Code to fix it: My competitor_tracker.py has hardcoded secrets on lines 14-15.Extract them into a .env file at /opt/agents/competitor-tracker/.envand update the Python code to read from environment variables instead. Claude Code: I'll extract those secrets. First, creating the .env file:> cat > /opt/agents/competitor-tracker/.env << 'EOF'DB_PASSWORD=r3allyS3cret!PassAPI_KEY=sk-proj-abc123def456EOF> chmod 600 /opt/agents/competitor-tracker/.envNow updating the Python script to use environment variables: The `.env` file looks like this: DB_PASSWORD=r3allyS3cret!PassAPI_KEY=sk-proj-abc123def456 And the Python code changes from this: # BEFORE: secrets hardcodeddb_password = "r3allyS3cret!Pass"api_key = "sk-proj-abc123def456" To this: # AFTER: secrets loaded from environmentimport osdb_password = os.environ["DB_PASSWORD"]api_key = os.environ["API_KEY"] **Output:** $ ls -la /opt/agents/competitor-tracker/.env-rw------- 1 ali ali 58 Feb 27 10:22 .env Notice `chmod 600` -- that means only Ali can read the file. Not his teammates, not other programs, not anyone else on the server. The safe combination stays in the safe. Never Commit .env Files If you use Git, add `.env` to your `.gitignore` file immediately. A `.env` file pushed to a public repository is a security incident, not a mistake. Making Logs Survive[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#making-logs-survive "Direct link to Making Logs Survive") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali starts his agent. It prints status messages to the terminal: [2026-02-27 10:30:01] Fetching pricing data...[2026-02-27 10:30:04] Found 23 competitors[2026-02-27 10:30:08] Updated database[2026-02-27 10:30:08] Sleeping 1 hour... He closes his laptop. Opens it again. The terminal is blank. Every line of output is gone. There is no record of what the agent did, whether it succeeded, or when it last ran. Output that vanishes is not output. It is a conversation that nobody recorded. Like relying on memory instead of taking notes during an important phone call. The fix is redirection -- sending output to a file instead of the screen: Set up my competitor-tracker agent to log its output to a fileat /opt/agents/competitor-tracker/logs/agent.log instead ofprinting to the terminal. I want to keep both the terminaloutput and the log file. Claude Code: I'll set up logging with the tee command, whichwrites to both the screen and a file simultaneously.To run your agent with persistent logging:> cd /opt/agents/competitor-tracker> python src/competitor_tracker.py 2>&1 | tee -a logs/agent.log Let's break that down: | Part | Meaning | | --- | --- | | `python src/competitor_tracker.py` | Run the agent | | `2>&1` | Combine error messages with normal output | | `\|` | Pipe: send output to the next command | | `tee -a logs/agent.log` | Write to both the screen AND the log file (`-a` = append, don't overwrite) | **Output:** $ python src/competitor_tracker.py 2>&1 | tee -a logs/agent.log[2026-02-27 10:35:01] Fetching pricing data...[2026-02-27 10:35:04] Found 23 competitors[2026-02-27 10:35:08] Updated database$ cat logs/agent.log[2026-02-27 10:35:01] Fetching pricing data...[2026-02-27 10:35:04] Found 23 competitors[2026-02-27 10:35:08] Updated database The same output appears on screen AND in the file. Close the terminal, disconnect SSH, shut down your laptop -- the log file persists. Tomorrow morning, Ali can check what happened overnight: Show me the last 20 lines of my agent's log file. Claude Code:> tail -20 /opt/agents/competitor-tracker/logs/agent.log The `tail` command shows the end of a file. `-20` means the last 20 lines. This is how you check on an agent without being connected while it runs. What Ali Has Now[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#what-ali-has-now "Direct link to What Ali Has Now") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before this lesson: | Component | Where | Problem | | --- | --- | --- | | Code | `~/competitor_tracker.py` | Mixed with personal files | | Secrets | Line 14 of the script | Exposed to anyone reading the code | | Logs | Terminal screen | Gone when terminal closes | After this lesson: | Component | Where | Benefit | | --- | --- | --- | | Code | `/opt/agents/competitor-tracker/src/` | Isolated, findable, professional | | Secrets | `/opt/agents/competitor-tracker/.env` | Separated, protected (chmod 600) | | Logs | `/opt/agents/competitor-tracker/logs/` | Persistent, reviewable, searchable | | Config | `/opt/agents/competitor-tracker/config/` | Ready for future configuration files | | Data | `/opt/agents/competitor-tracker/data/` | Ready for database files and cache | The house is organized. Every file has a place. Secrets are locked up. Logs persist. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Extend the Structure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#prompt-1-extend-the-structure "Direct link to Prompt 1: Extend the Structure") My competitor-tracker agent needs to store three types of data:raw downloads (HTML pages it scrapes), processed reports (theanalysis it generates), and archived results (old reports keptfor historical comparison).Extend the /opt/agents/competitor-tracker/data/ directory tohandle all three. What naming convention should the subdirectoriesfollow? Create them. **What you're learning:** Directory design decisions. Flat vs. nested structures, naming conventions that scale, and how to organize data that grows over time. ### Prompt 2: Design for a Different Agent[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#prompt-2-design-for-a-different-agent "Direct link to Prompt 2: Design for a Different Agent") I want to deploy a second agent -- one that monitors social mediamentions of my brand and sends Slack alerts when sentiment turnsnegative.Design a complete directory structure for it at/opt/agents/social-monitor/. What directories does it need thatthe competitor-tracker doesn't? Create the structure. **What you're learning:** Transferring the pattern to a new context. Different agents have different needs (credentials for multiple platforms, queue directories for pending alerts), but the organizational principle stays the same. ### Prompt 3: .env vs config.yaml[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#prompt-3-env-vs-configyaml "Direct link to Prompt 3: .env vs config.yaml") When should secrets go in a .env file vs a config.yaml file?What is the security difference? What happens if you accidentallycommit each one to Git? Show me an example of what belongs ineach file for the competitor-tracker agent. **What you're learning:** The boundary between secrets and configuration. Database passwords are secrets (.env). Scraping intervals and retry counts are configuration (config.yaml). Knowing which is which prevents both security incidents and unnecessary complexity. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The house is built. Ali starts his agent. It works. He closes his laptop. The agent dies. Every. Single. Time. The process is tied to his terminal session -- when the session ends, everything running inside it ends too. Lesson 4 solves this problem permanently. Checking access... * [The Problem With "It Works on My Desk"](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#the-problem-with-it-works-on-my-desk) * [Building the Structure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#building-the-structure) * [Extracting Secrets](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#extracting-secrets) * [Making Logs Survive](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#making-logs-survive) * [What Ali Has Now](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#what-ali-has-now) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#try-with-ai) * [Prompt 1: Extend the Structure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#prompt-1-extend-the-structure) * [Prompt 2: Design for a Different Agent](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#prompt-2-design-for-a-different-agent) * [Prompt 3: .env vs config.yaml](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#prompt-3-env-vs-configyaml) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/setting-up-your-agents-home#flashcards-study-aid) --- # Practice: Computation & Data Extraction Exercises | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You can build Python utilities that handle decimal math, parse messy CSV files, categorize transactions with regex, and orchestrate multi-step data pipelines. You've directed Claude Code to create scripts that read from stdin and compose with pipes. That's a powerful toolkit, but the gap between building one script in a guided lesson and solving a messy real-world data problem from scratch is where most people discover what they actually know. These 13 exercises close that gap. Each module gives you two exercises: a **Build** exercise where you create a working utility from realistic data, and a **Debug** exercise where you find and fix bugs in broken scripts. Three skills run through every exercise: **data processing tool building** (creating stdin/stdout utilities that handle real-world messiness), **script debugging and verification** (finding logic bugs that hide behind exit code 0), and **pipeline orchestration** (connecting verified tools into multi-step workflows). Every exercise uses real financial data (bank transactions, payroll records, expense reports) with the edge cases that break naive processing: quoted fields with commas, floating-point rounding errors, false positive pattern matches, and silent data loss between pipeline steps. By the end, you'll have debugged more broken scripts than most people encounter in a year. Download Exercise Files **[Download Computation Exercises (ZIP)](https://github.com/panaversity/claude-code-computation-exercises/releases/latest/download/computation-exercises.zip) ** After downloading, unzip the file. Each exercise has its own folder with an `INSTRUCTIONS.md`, starter data files, and (for Debug exercises) broken scripts to fix. If the download link doesn't work, visit the [repository releases page](https://github.com/panaversity/claude-code-computation-exercises/releases) directly. * * * How to Use These Exercises[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#how-to-use-these-exercises "Direct link to How to Use These Exercises") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The workflow for every exercise is the same: 1. **Open the exercise folder** from the `claude-code-computation-exercises/` directory 2. **Read the INSTRUCTIONS.md** inside the folder, which describes the data files and your task 3. **Read the walkthrough below** for context on what you're practicing and why 4. **Start Claude Code** and point it at the exercise folder 5. **Work through the exercise**: for Build exercises, describe what you need; for Debug exercises, investigate the broken code 6. **Reflect** using the questions provided. This is where the real learning happens You don't need to complete all 13 in one sitting. Work through one module at a time. Each module builds on the workflows from specific chapter lessons. * * * Tool Guide[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#tool-guide "Direct link to Tool Guide") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Claude Code**: Required for all exercises. Every exercise involves writing or debugging Python scripts in the terminal. You'll use pipes, CSV processing, and script execution. * **Python 3.x** must be installed. Verify with: `python3 --version` * **A text editor** is helpful for examining data files before processing, but not required, since Claude Code can read and display file contents directly. * * * Key Differences from Chapter Lessons[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#key-differences-from-chapter-lessons "Direct link to Key Differences from Chapter Lessons") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lessons 1-6, you learned each workflow in isolation with guided walkthroughs. These exercises are different in three ways: * **No step-by-step instructions.** The exercises describe the scenario, the data, and the goal. You decide the approach, write the prompts, and handle edge cases yourself. * **Build + Debug pairing.** Every module has a Build exercise (create a working utility) and a Debug exercise (find and fix bugs in broken code). Debugging someone else's script develops different skills than writing your own: you learn to read code critically, compare output against expected values, and trace logic errors that don't produce exceptions. * **Increasing independence.** Modules 1-2 provide starter prompts to scaffold your learning. Modules 3-5 remove the scaffolding. Capstones remove everything: you design the entire approach. By Module 5, you should be able to face a new data processing problem and instinctively reach for the right pattern without needing to review the chapter lessons. * * * The Data Processing Framework[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#the-data-processing-framework "Direct link to The Data Processing Framework") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Lessons 1-6 taught these steps individually -- building tools, testing with known answers, handling edge cases, composing pipelines, making scripts permanent. This framework assembles those skills into a single reusable process. Use it for every exercise: 1. **Understand the Data**: What format? What columns? What edge cases exist? What does "correct" output look like? 2. **Build the Tool**: Write a Python script that reads stdin and produces stdout 3. **Create Test Data**: Make small datasets (5-10 rows) with manually calculated correct answers 4. **Verify**: Run on test data and compare against expected results line by line 5. **Handle Edge Cases**: Quoted fields, missing values, mixed formats, encoding issues 6. **Pipeline**: Connect tools with pipes for multi-step processing 7. **Make Permanent**: Save scripts, create aliases, document usage for future runs This framework applies to any domain where data needs processing: log analysis, invoice processing, inventory management, or any workflow where structured data needs cleaning, transforming, or summarizing. Notice that steps 1-4 happen before you handle edge cases or build pipelines. That's intentional: most data processing bugs come from skipping verification on simple cases before tackling complex ones. * * * Assessment Rubric[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#assessment-rubric "Direct link to Assessment Rubric") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For each exercise, evaluate yourself on: | Criteria | Beginner (1) | Developing (2) | Proficient (3) | Advanced (4) | | --- | --- | --- | --- | --- | | **Decimal Handling** | Uses Bash arithmetic | Uses Python but rounds poorly | Correct float handling throughout | Uses Decimal module for financial precision | | **Verification** | Doesn't test | Tests with one case | Tests with known answers + edge cases | Comprehensive test suite with automated comparison | | **CSV Processing** | Splits on commas naively | Handles basic CSV | Handles quoted fields and edge cases | Handles encoding, BOM, mixed line endings | | **Pattern Matching** | Hardcoded string checks | Basic regex | Regex with word boundaries | False positive guards + categorization hierarchy | | **Pipeline Design** | Single script does everything | Separate scripts, manual steps | Piped pipeline with verification | Automated pipeline with error handling at each step | * * * Module 1: Arithmetic & Stdin Tools[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-1-arithmetic--stdin-tools "Direct link to Module 1: Arithmetic & Stdin Tools") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Building composable Python utilities that handle decimal math correctly (Lesson 1) > > Lesson 1 taught you that Bash can't do decimal arithmetic and that Python scripts reading from stdin compose naturally with pipes. These exercises push those skills into realistic scenarios where the data is messier and the arithmetic has more edge cases than the lesson examples. 1.1The Expense Splitter Start ### Exercise 1.1: The Expense Splitter (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-11-the-expense-splitter-build "Direct link to Exercise 1.1: The Expense Splitter (Build)") **The Problem:** Open the `module-1-arithmetic-and-stdin/exercise-1.1-expense-splitter/` folder. You'll find 5 dinner receipt files, each containing a list of items with prices, a tax rate, and a tip percentage. The amounts include decimals that Bash arithmetic can't handle. One receipt has 12 people splitting unevenly (some ordered drinks, some didn't). Another has a flat service charge instead of percentage tip. A third uses different tax rates for food vs. alcohol. **Your Task:** Build a Python utility that reads a receipt from stdin and calculates per-person splits. The script should handle tax applied before tip, different tip calculation methods (percentage vs. flat), and uneven splits where people pay different base amounts. Run it on all 5 receipts and verify each total matches the receipt's bottom line. **What You'll Learn:** * How floating-point arithmetic introduces rounding errors that accumulate across many line items, and why financial calculations need explicit rounding strategy * The stdin/stdout pattern that makes your script composable: `cat receipt.txt | python3 split.py` works with ANY receipt in the same format * That "splitting the bill" has more edge cases than you'd expect: tax-on-subtotal vs. tax-on-total, tip-on-pretax vs. tip-on-posttax, penny allocation for uneven splits **Starter Prompt (Intentionally Vague):** > "Write a script that splits a dinner bill." **Better Prompt (Build Toward This):** After examining the receipt format with `cat receipt-01.txt`: "Build a Python script called split.py that reads a receipt from stdin, calculates per-person splits including tax and tip, and outputs each person's total to stdout. Handle: (1) tax applied to subtotal, (2) tip as percentage or flat amount, (3) uneven splits where different people ordered different amounts. Round to 2 decimal places and verify the individual amounts sum to the receipt total." **Reflection Questions:** 1. Did the individual splits sum exactly to the total on any receipt? If not, where did the pennies go, and how did you handle the remainder? 2. Which receipt was hardest to process? What made its format different from the others? 3. Could your script handle a receipt format it hasn't seen before, or is it tightly coupled to the specific format in these files? * * * 1.2The Rounding Trap Start ### Exercise 1.2: The Rounding Trap (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-12-the-rounding-trap-debug "Direct link to Exercise 1.2: The Rounding Trap (Debug)") **The Problem:** Open the `module-1-arithmetic-and-stdin/exercise-1.2-rounding-trap/` folder. You'll find `buggy-sum.py` (a script that sums a column of transaction amounts from a CSV file) and `transactions-500.csv` (a file with 500+ transactions). The script runs without errors and produces a total, but it's $12.47 off from the known correct total provided in `expected-total.txt`. The bug isn't a crash; it's a silent arithmetic error. **Your Task:** Find the bug. The script rounds each intermediate sum to 2 decimal places during accumulation instead of rounding only the final result. Trace the error by running the script on smaller subsets (10, 50, 100 transactions) and watching how the error grows. Fix the bug and verify your corrected script matches the expected total exactly. **What You'll Learn:** * That rounding during accumulation and rounding after accumulation produce different results, and the difference grows with data size * How to bisect a data problem: test on small subsets to isolate where errors appear, then trace the logic * Why "the script runs without errors" is not the same as "the script produces correct results." Exit code 0 tells you nothing about output correctness **Starter Prompt (Intentionally Vague):** > "This script gives the wrong total. Fix it." **Better Prompt (Build Toward This):** After comparing `python3 buggy-sum.py < transactions-500.csv` against the expected total: "The buggy-sum.py script produces a total that's $12.47 off from the expected value in expected-total.txt. Run it on the first 10, 50, and 100 rows to see if the error scales with data size. Find the line where the rounding bug occurs, explain why intermediate rounding causes drift, and fix it. Verify the fix matches the expected total exactly." **Reflection Questions:** 1. How did the error scale: was it roughly proportional to the number of transactions, or did it grow unpredictably? 2. At what data size did the error become noticeable? If this script processed 10 transactions instead of 500, would anyone have caught the bug? 3. What other accumulation patterns might have this same class of bug? (Hint: think about averaging, running balances, percentage calculations.) * * * Module 2: Testing & Verification[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-2-testing--verification "Direct link to Module 2: Testing & Verification") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Zero-trust verification: proving correctness with test data (Lesson 2) > > Lesson 2 introduced the testing loop: create test data with known answers, run your script, compare output to expected results. These exercises take that skill further: you'll design adversarial test cases and diagnose scripts that produce plausible but wrong output. 2.1The Bulletproof Calculator Start ### Exercise 2.1: The Bulletproof Calculator (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-21-the-bulletproof-calculator-build "Direct link to Exercise 2.1: The Bulletproof Calculator (Build)") **The Problem:** Open the `module-2-testing-and-verification/exercise-2.1-bulletproof-calculator/` folder. You'll find `sum-expenses.py` (a working expense summation script) and `sample-expenses.csv` (a small CSV file it processes correctly). The script looks solid. It handles basic CSV, sums a column, and produces the right answer on the sample data. But "works on the sample" doesn't mean "works on everything." **Your Task:** Design a test suite that breaks this script. Create test CSV files that probe every assumption the script makes: What happens with negative amounts (refunds)? Empty rows? Non-numeric values in the amount column? Amounts with currency symbols ($, EUR)? Files with Windows line endings? CSV files with a BOM (byte order mark)? Your goal is to find the script's silent weaknesses: inputs it processes without error but produces wrong results for. **What You'll Learn:** * That designing adversarial test data is a skill distinct from writing code: you're thinking about what could go wrong, not what should go right * The difference between crashes (the script tells you something is wrong) and silent failures (the script happily produces garbage) * How "works on the sample" creates false confidence, because the sample was designed by the same person who wrote the script, so it avoids their blind spots **Starter Prompt (Intentionally Vague):** > "Test this script to make sure it works." **Better Prompt (Build Toward This):** After reading sum-expenses.py to understand its parsing logic: "Create a test suite for sum-expenses.py. Generate 8 test CSV files that probe edge cases: (1) negative amounts (refunds), (2) empty rows, (3) non-numeric values in the amount column, (4) amounts with $ or EUR symbols, (5) extremely large amounts (millions), (6) extremely small amounts (fractions of a cent), (7) Windows \\r\\n line endings, (8) a file with a UTF-8 BOM. For each test file, include an expected-output.txt. Run all tests and report which ones the script handles correctly and which ones it fails silently on." **Reflection Questions:** 1. How many of your test cases caused crashes vs. silent wrong answers? Which category is more dangerous? 2. Did any test case reveal a bug you didn't expect? What assumption did the script make that you didn't notice until the test exposed it? 3. If you had to choose only 3 test cases to run on any new data processing script, which 3 would catch the most common bugs? * * * 2.2The Green Light Lie Start ### Exercise 2.2: The Green Light Lie (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-22-the-green-light-lie-debug "Direct link to Exercise 2.2: The Green Light Lie (Debug)") **The Problem:** Open the `module-2-testing-and-verification/exercise-2.2-green-light-lie/` folder. You'll find `broken-calc.py` (a script that processes an expense report and outputs a summary) and `test-data.csv` with known correct output in `expected-output.txt`. The script exits with code 0 and produces output that looks reasonable. But it has 3 logic bugs hidden in the code: it silently skips rows with negative amounts (treating refunds as non-data), it counts rows off-by-one (includes the header in the count), and it uses `abs()` on the total which masks a negative balance. **Your Task:** Find all 3 bugs without being told what they are beyond "the output is wrong in 3 ways." Compare the script's output against `expected-output.txt` line by line. For each discrepancy, trace through the code to find the responsible line. Fix all 3 and verify your corrected output matches expected exactly. **What You'll Learn:** * That scripts with multiple bugs interact: fixing one bug changes the output, which can make other bugs harder or easier to spot * How to systematically compare expected vs. actual output to isolate which specific values are wrong * The pattern of "looks close enough" masking real errors: an off-by-one count or a missing negative sign is easy to overlook when the rest of the output seems right **Starter Prompt (Intentionally Vague):** > "This calculator has bugs. Find them." **Better Prompt (Build Toward This):** After running `python3 broken-calc.py < test-data.csv` and `diff <(python3 broken-calc.py < test-data.csv) expected-output.txt`: "The broken-calc.py script has 3 logic bugs. Its output differs from expected-output.txt in specific ways. Compare each line of actual vs. expected output. For each difference, trace through the code to find which line causes the wrong value. Fix all 3 bugs one at a time, verifying after each fix that the specific discrepancy is resolved without introducing new ones." **Reflection Questions:** 1. Which bug was hardest to find? Was it because the code looked correct on casual reading, or because the wrong output looked plausible? 2. Did fixing one bug change how another bug manifested? How did you keep track of which discrepancies were resolved? 3. If you only had the script and no expected output file, how would you have discovered these bugs? What verification would you create from scratch? * * * Module 3: CSV Processing[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-3-csv-processing "Direct link to Module 3: CSV Processing") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Processing real-world CSV data that breaks naive parsing (Lesson 3) > > Lesson 3 taught you to move beyond naive comma-splitting to proper CSV parsing with Python's `csv` module. These exercises confront you with the full spectrum of CSV messiness: quoted fields containing commas, mixed date formats, currency symbols, trailing delimiters, and encoding issues that are standard in real-world data exports. 3.1The Messy Payroll Start ### Exercise 3.1: The Messy Payroll (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-31-the-messy-payroll-build "Direct link to Exercise 3.1: The Messy Payroll (Build)") **The Problem:** Open the `module-3-csv-processing/exercise-3.1-messy-payroll/` folder. You'll find `payroll-raw.csv`, an 80+ row payroll export from an accounting system. This isn't clean data. Employee names contain commas ("Smith, John"), salary fields have currency symbols and thousand separators ("$4,250.00"), date formats are mixed (MM/DD/YYYY and YYYY-MM-DD in the same file), some rows have empty fields, and there's a trailing comma on every line that creates a phantom empty column. **Your Task:** Build a Python script that reads this CSV from stdin and outputs a clean, standardized version: names unquoted and normalized, salaries as plain numbers, dates in ISO format (YYYY-MM-DD), empty fields replaced with "N/A", and no phantom columns. The cleaned output should be valid CSV that any tool can process without special handling. Verify by spot-checking 5 rows across the file, choosing the messiest ones you can find. **Key Edge Cases to Watch For:** * Names with commas inside quotes: `"Smith, John"` should become `Smith John`, not two separate columns * Currency symbols and separators: `"$4,250.00"` needs both `$` and `,` stripped before parsing as a number * Mixed date formats within the same column: some rows use `03/15/2024`, others use `2024-03-15` * Trailing commas creating empty phantom columns that shift field alignment * Empty fields at the end of a row vs. empty fields in the middle (different causes, different fixes) **What You'll Learn:** * Why Python's `csv` module exists: splitting on commas breaks on `"Smith, John"`, and every hand-rolled parser eventually encounters this * That real-world CSV files violate the spec in predictable ways: mixed date formats, currency symbols, trailing delimiters, inconsistent quoting * The pattern of "clean once, use everywhere": investing 20 minutes in a proper cleaning script saves hours of debugging downstream tools that choke on messy input **Reflection Questions:** 1. How many distinct edge cases did the payroll file contain? Which one would have caused the most damage if you'd processed it with naive comma-splitting? 2. Did Python's `csv` module handle all the edge cases, or did you need additional cleaning logic on top of it? 3. How would you verify that your cleaned output is correct for all 80+ rows, not just the 5 you spot-checked? * * * 3.2The Awk Disaster Start ### Exercise 3.2: The Awk Disaster (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-32-the-awk-disaster-debug "Direct link to Exercise 3.2: The Awk Disaster (Debug)") **The Problem:** Open the `module-3-csv-processing/exercise-3.2-awk-disaster/` folder. You'll find `process.awk` (an awk script that extracts and sums transaction amounts from `bank-export.csv`, 200 rows). The awk script produces a total that's wildly wrong: not just off by a few cents, but off by thousands. It processed only 160 of the 200 rows correctly. The other 40 rows either produced wrong values or were silently skipped. **Your Task:** Identify which 40 rows the awk script mishandled and determine why. The causes are the usual suspects: quoted fields containing commas shifted column alignment, currency symbols made amounts non-numeric, and negative amounts in parentheses (the accounting notation `(500.00)` meaning -$500) were treated as zero. Document each failure category with example rows. **What You'll Learn:** * Why awk's field splitting on commas fails on real CSV data, and why this is the single most common data processing bug in shell scripts * How to identify which rows in a dataset are problematic by comparing per-row output against expected values * That the gap between "works on clean data" and "works on exported data" is where most data processing projects fail **The Challenge:** After finding all 40 broken rows, write a Python replacement that handles every edge case the awk script missed. Process all 200 rows correctly and verify your total against `expected-total.txt`. Compare the line count of your Python solution against the awk script; sometimes the "simple" tool produces more complex code than using the right tool from the start. **Reflection Questions:** 1. Of the 40 broken rows, how many failure categories did you find? Was there one dominant cause or multiple independent problems? 2. Could any version of the awk script handle these edge cases, or is awk fundamentally the wrong tool for CSV with quoted fields? What would a "correct" awk solution look like, and would it still be simpler than Python? 3. How long did it take to diagnose the awk failures vs. how long it would have taken to write the Python replacement from scratch? When is debugging someone else's code not worth the effort? * * * Module 4: Categorization & Patterns[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-4-categorization--patterns "Direct link to Module 4: Categorization & Patterns") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Building categorizers with regex precision and false-positive guards (Lesson 5) > > Lesson 5 introduced regex-based categorization with word boundaries and false positive prevention. These exercises push your categorizer against data specifically designed to expose the weaknesses of pattern matching: merchant names that contain keywords from the wrong category, and patterns that seem specific but match too broadly. 4.1The Expense Report Builder Start ### Exercise 4.1: The Expense Report Builder (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-41-the-expense-report-builder-build "Direct link to Exercise 4.1: The Expense Report Builder (Build)") **The Problem:** Open the `module-4-categorization/exercise-4.1-expense-report-builder/` folder. You'll find `corporate-expenses.csv`, containing 150+ corporate credit card transactions that need to be categorized for accounting. The target categories are: Travel, Meals, Software, Office Supplies, and Uncategorized. The data includes merchant names, amounts, and dates. Some merchants are obvious ("UNITED AIRLINES" = Travel), but the data is full of traps: "DELTA FAUCETS" is not Delta Airlines, "SUBWAY" could be food or transit, and "APPLE STORE" could be technology or the fruit stand on 5th Avenue. **Key False Positive Traps:** * "DELTA" matches both Delta Airlines (Travel) and Delta Faucets (Office Supplies) * "SUBWAY" could be the restaurant (Meals) or public transit (Travel) * "ADOBE" appears in both "ADOBE CREATIVE CLOUD" (Software) and "ADOBE CAFE" (Meals) * Generic terms like "STORE", "SERVICE", and "SUPPLY" appear across multiple categories **Your Task:** Build a Python categorization script that reads transactions from stdin and outputs each transaction with its assigned category. Use regex patterns with word boundaries and false-positive guards. Your categorizer must correctly handle all the traps in the data. Verify by running against `expected-categories.txt` which contains the correct category for every transaction. **What You'll Learn:** * That naive keyword matching produces unacceptable false positive rates: "DELTA" matching both airlines and faucets is a business-critical error * How word boundaries (`\b`), negative lookaheads, and categorization hierarchies reduce false positives from pattern matching * The testing workflow for categorizers: run on full dataset, compare against expected output, fix every mismatch, and re-run until zero discrepancies **Reflection Questions:** 1. How many false positives did your first version produce? Which category had the most? 2. What pattern technique was most effective at reducing false positives: word boundaries, exclusion lists, or categorization hierarchy (check specific patterns before general ones)? 3. If 50 new merchants appeared next month, what percentage would your categorizer handle correctly without modification? * * * 4.2The Over-Eager Matcher Start ### Exercise 4.2: The Over-Eager Matcher (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-42-the-over-eager-matcher-debug "Direct link to Exercise 4.2: The Over-Eager Matcher (Debug)") **The Problem:** Open the `module-4-categorization/exercise-4.2-over-eager-matcher/` folder. You'll find `categorizer.py` (a transaction categorization script) and `transactions.csv` (a dataset it processes). The categorizer runs and produces output, but it has 8 specific false positives: DR PEPPER is categorized as Medical, RED LOBSTER as Charitable Donations, SHELL GAS as Technology (matching "shell"), AMAZON FRESH as Books, OFFICE DEPOT as Government (matching "office" in a government pattern), BEST BUY as Travel (matching "buy" in a flight booking pattern), SUBWAY SANDWICH as Transportation, and COACH OUTLET as Sports. **Your Task:** Find and fix all 8 false positives in the categorizer's regex patterns. For each one, identify which pattern matched incorrectly and why. Fix the pattern to exclude the false positive without breaking correct matches. Verify that all 8 are resolved and no new false positives were introduced. **What You'll Learn:** * That regex patterns which look reasonable in isolation produce absurd results on real merchant names, and this is the normal state of naive pattern matching * The specific techniques for fixing false positives: anchoring patterns, adding exclusion terms, using word boundaries, and ordering patterns from specific to general * That fixing one false positive can create another if patterns overlap, so verification after every fix is mandatory **The Twist:** After fixing all 8 false positives, have Claude Code generate 5 new merchant names that would fool your corrected categorizer. Fix those too. This reveals whether your fixes were surgical patches or genuine improvements to the categorization logic. **Reflection Questions:** 1. Which false positive had the most obvious cause? Which one required the most investigation to understand? 2. Did fixing any of the 8 false positives break a previously correct categorization? How did you catch it? 3. Of the 5 new merchant names Claude generated, how many actually broke your corrected categorizer? What does this tell you about the robustness of pattern-based categorization? * * * Module 5: Pipeline Orchestration[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-5-pipeline-orchestration "Direct link to Module 5: Pipeline Orchestration") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Connecting verified tools into multi-step data pipelines (Lesson 4) > > Lesson 4 showed how individual scripts compose into pipelines through stdin/stdout. These exercises reveal the hardest part of pipeline design: making sure each step's output format exactly matches the next step's expected input. Most pipeline bugs live at these interfaces, not inside the steps themselves. 5.1The Quarterly Report Start ### Exercise 5.1: The Quarterly Report (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-51-the-quarterly-report-build "Direct link to Exercise 5.1: The Quarterly Report (Build)") **The Problem:** Open the `module-5-pipeline-orchestration/exercise-5.1-quarterly-report/` folder. You'll find three monthly transaction files: `january.csv`, `february.csv`, and `march.csv`. Each has slightly different column ordering and date formats (January uses MM/DD/YYYY, February uses DD-MM-YYYY, March uses YYYY-MM-DD). Column headers also differ slightly: January uses "Amount", February uses "Transaction Amount", March uses "Amt". Your task is to produce a quarterly summary report showing total spending by category, monthly trends, and the top 10 largest transactions. **Your Task:** Build a multi-step pipeline: (1) a cleaning script that normalizes each month's CSV to a common format with consistent column names and date formats, (2) a combining script that merges the three cleaned files with a source-month column, (3) a categorization script that assigns categories using patterns from Module 4, and (4) a reporting script that produces the summary with per-category totals and monthly breakdowns. Each script reads stdin and writes stdout, so the full pipeline is: `cat january.csv | python3 clean.py --month jan > jan-clean.csv` for each month, then `cat *-clean.csv | python3 combine.py | python3 categorize.py | python3 report.py`. Verify each step's output before piping to the next. **What You'll Learn:** * That pipeline design is about interfaces: each script must produce output that the next script can consume without modification * Why verifying intermediate output (not just the final report) catches bugs that would be nearly impossible to trace through a 4-step pipeline * The power of composable tools: once each script works independently, the pipeline assembles like building blocks **Reflection Questions:** 1. Which pipeline step was hardest to get right? Was the difficulty in the logic or in matching the output format to the next step's expected input? 2. Did you verify intermediate output after each step, or did you build the full pipeline first and debug from the final output? Which approach would have been faster? 3. If a fourth month (April) were added with yet another date format, how many of your scripts would need modification? * * * 5.2The Broken Pipeline Start ### Exercise 5.2: The Broken Pipeline (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-52-the-broken-pipeline-debug "Direct link to Exercise 5.2: The Broken Pipeline (Debug)") **The Problem:** Open the `module-5-pipeline-orchestration/exercise-5.2-broken-pipeline/` folder. You'll find a 3-step pipeline: `step1-clean.py`, `step2-categorize.py`, `step3-report.py`, along with `input.csv` and `expected-report.txt`. Running the full pipeline (`cat input.csv | python3 step1-clean.py | python3 step2-categorize.py | python3 step3-report.py`) produces a report, but the numbers don't match the expected output. Each individual script works correctly on its own test data: `step1-clean.py` produces clean output, `step2-categorize.py` categorizes correctly when given plain numbers, `step3-report.py` summarizes correctly when given categorized data. The bug is in the interfaces: step 1 outputs amounts with a "$" prefix (which is valid, clean output for display purposes), and step 2's parser silently drops any row where the amount field doesn't convert to a float, so every row is silently lost between steps 1 and 2. **Your Task:** Diagnose the pipeline failure by examining the output of each step independently. Capture intermediate output: `cat input.csv | python3 step1-clean.py > step1-output.csv`, then inspect it. Feed that to step 2 and capture its output. Find where rows are being lost and why. Fix the interface mismatch and verify the full pipeline produces the expected report. **What You'll Learn:** * That pipeline bugs almost always live at the interfaces between steps, not inside the steps themselves. Each step works in isolation but fails when connected * The debugging technique of capturing intermediate output: redirect each step to a file and inspect before piping forward * Why "silently drops rows" is the most dangerous class of pipeline bug: the pipeline produces output that looks plausible but is missing data **The Extension:** After fixing the interface bug, add verification checks between each pipeline step that flag data loss. For example: step 1 outputs a row count in a comment line (`# ROWS: 200`), step 2 reads that comment, counts incoming rows, and exits with an error if they don't match. Apply this pattern to every interface in the pipeline. This transforms silent data loss into a loud, immediate error: the pipeline stops and tells you exactly where data disappeared. **Reflection Questions:** 1. How many rows did the pipeline silently drop before your fix? What percentage of the total data was lost? 2. If you hadn't been told the bug was in the interfaces, how long would it have taken to find it by testing each step in isolation? 3. How would you design a pipeline from scratch to prevent silent data loss between steps? What conventions would you adopt? * * * Module 6: Capstone Projects[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-6-capstone-projects "Direct link to Module 6: Capstone Projects") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > **Choose one (or more). Build a real tool you'll actually use. No starter prompts provided.** Capstones are different from the exercises above. There are no guided prompts; you design the entire approach yourself. Each project requires applying all the skills from Modules 1-5 together to solve a realistic problem. Where module exercises test individual skills, capstones test your ability to orchestrate those skills into a coherent pipeline. The quality of your verification matters as much as the result: anyone reviewing your work should trust the numbers because you proved them. The progression across capstones is intentional: Capstone A uses curated data with known correct answers so you can verify your pipeline end-to-end. Capstone B uses a larger dataset where you must define "correct" yourself (what counts as a subscription? what counts as a price change?). Capstone C uses your own data where there's no answer key at all. You're the domain expert who decides if the output makes sense. Each capstone demands more judgment and less scaffolding than the last. AThe Freelancer's Tax Prep Start ### Capstone A: The Freelancer's Tax Prep[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#capstone-a-the-freelancers-tax-prep "Direct link to Capstone A: The Freelancer's Tax Prep") Open the `module-6-capstone/capstone-A-freelancer-tax-prep/` folder. You'll find 6 months of bank transaction data across separate CSV files (one per month), each with slightly different export formats. The data contains income (client payments), business expenses (software, equipment, meals with clients), personal transactions (groceries, entertainment), and transfers between accounts that should be excluded from both income and expenses. Some transactions are ambiguous: is that $200 "AMAZON" purchase a business expense (office supplies) or personal (kitchen gadgets)? Take this data through the complete Data Processing Framework: 1. **Understand**: Survey the 6 files: how many transactions total, what formats, what categories appear, what's the date range 2. **Build**: Create scripts for each processing step: cleaning, combining, categorizing 3. **Test**: Verify each script on a small subset with manually calculated expected results 4. **Verify**: Run on full dataset and spot-check categories against raw transaction descriptions 5. **Handle Edge Cases**: Account transfers (not income or expense), refunds that offset expenses, split transactions, foreign currency conversions 6. **Pipeline**: Connect all scripts into a single pipeline that produces a tax-ready report with category totals 7. **Make Permanent**: Save the pipeline as a documented, reusable tool that works on next quarter's data **Deliverables:** * `clean.py`: Normalizes all 6 monthly CSVs to a common format * `categorize.py`: Assigns tax categories (Income, Business Expense, Personal, Transfer) * `ambiguous.py`: Flags transactions that need human review * `report.py`: Produces final tax summary with totals per category * `TAX-SUMMARY.md`: The actual report: income total, deductible expenses, flagged items * `VERIFICATION.md`: How you verified correctness: test cases, spot checks, row counts **Reflection Questions:** 1. How many transactions fell into the "ambiguous" category? What would happen to your tax filing if you guessed wrong on those? 2. Which step in the pipeline caught the most edge cases? Which edge case would have caused the largest financial error if missed? 3. Could this pipeline run on next year's data with zero modifications? What would need to change? * * * BThe Subscription Auditor Start ### Capstone B: The Subscription Auditor[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#capstone-b-the-subscription-auditor "Direct link to Capstone B: The Subscription Auditor") Open the `module-6-capstone/capstone-B-subscription-auditor/` folder. You'll find 12 months of credit card transactions (800+ rows total) in a single large CSV. Hidden in the data are recurring subscription charges (some monthly, some annual, some that changed price midway through the year). The challenge isn't processing the data (you have those skills from Modules 1-4); it's defining what counts as a "subscription" when merchant names vary slightly between charges ("NETFLIX.COM", "NETFLIX INC", "NETFLIX STREAMING") and amounts fluctuate due to tax changes or plan upgrades. Build a pipeline that: 1. **Identifies recurring charges**: Find merchants that appear in 3+ months with similar amounts 2. **Detects price changes**: Flag any subscription where the amount changed between months 3. **Finds duplicates**: Identify months where the same subscription was charged twice 4. **Calculates annual cost**: Total yearly spending per subscription 5. **Generates audit report**: A summary showing every subscription, its frequency, any price changes, any duplicate charges, and total annual cost **Deliverables:** * `find-recurring.py`: Identifies recurring merchant patterns * `detect-changes.py`: Flags price changes and duplicates * `audit-report.py`: Generates the final audit summary * `SUBSCRIPTION-AUDIT.md`: The actual report with findings * `VERIFICATION.md`: How you verified each finding is correct **Reflection Questions:** 1. How many subscriptions did your auditor find? Were any surprising, charges you wouldn't notice on a month-by-month review? 2. How did you define "similar amounts" for detecting recurring charges? A subscription might be $9.99 one month and $10.49 the next: is that the same subscription with a price change, or two different charges? 3. What's the total annual cost of all detected subscriptions? If someone wanted to cut their subscriptions by 30%, which ones would you recommend cutting and why? * * * CYour Own Financial Data Start ### Capstone C: Your Own Financial Data[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#capstone-c-your-own-financial-data "Direct link to Capstone C: Your Own Financial Data") Open the `module-6-capstone/capstone-C-your-own-data/` folder for a self-assessment template. Then close it; this capstone uses YOUR actual bank data. Export your own bank transactions as CSV (most banks offer this in their online banking; look for "Download" or "Export" in your transaction history). Build a custom pipeline tailored to your actual spending categories and financial questions: 1. **Clean** your bank's specific export format 2. **Categorize** based on your actual spending patterns (not generic categories) 3. **Answer a question** you have about your finances: "Where does my money go?" or "How much do I spend eating out?" or "Have my utilities gone up this year?" 4. **Build it to reuse**: Make the pipeline work on next month's export with zero changes **What Makes This Special:** Unlike Capstones A and B, this one uses real data with real stakes. The categories matter because they're YOUR categories. The edge cases are genuine because they came from YOUR bank. The verification matters because wrong numbers affect real financial decisions. And there's no expected output file to compare against. You are the only person who can verify the results are correct, which means your verification skills from Module 2 become essential rather than optional. **Important:** Before processing your own data, remove or redact any information you're not comfortable having in a local file. Your bank CSV may contain account numbers or other sensitive details that should not appear in script output files. **Deliverables:** * Your cleaning, categorizing, and reporting scripts * `MY-SPENDING-REPORT.md`: Answers to your financial questions with supporting data * `PIPELINE-README.md`: Instructions for running the pipeline on next month's export * `WHAT-I-LEARNED.md`: What surprised you about your spending, and what you'd do differently in the pipeline **Reflection Questions:** 1. Was your bank's CSV format cleaner or messier than the exercise data? What edge cases were unique to your bank's export? 2. Did the spending analysis reveal anything you didn't expect? Would you have discovered it by scanning your bank statement manually? 3. Will you actually run this pipeline next month? What would make it easier to reuse, and what would make you abandon it? * * * What's Next[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#whats-next "Direct link to What's Next") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You've built, debugged, and orchestrated data processing tools across 13 exercises, from simple decimal calculators to multi-step pipelines handling hundreds of real transactions. The Data Processing Framework (Understand, Build, Test, Verify, Edge Cases, Pipeline, Permanent) applies to any domain where data needs processing: log analysis, invoice processing, inventory management, or any workflow where "if it's math, it belongs in a script." The three skills you practiced (building composable tools, verifying output against known answers, and connecting tools into pipelines) are the exact skills that separate someone who can follow a tutorial from someone who can solve new problems. These verified, composable tools are the building blocks for the automated workflows you'll build in later chapters, where the manual prompting you practiced here evolves into autonomous agent behavior. The scripts you wrote, the verification habits you built, and the pipeline patterns you internalized are exactly what AI Employees need to operate independently; they just need those patterns encoded as instructions rather than typed as prompts. Checking access... * [How to Use These Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#how-to-use-these-exercises) * [Tool Guide](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#tool-guide) * [Key Differences from Chapter Lessons](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#key-differences-from-chapter-lessons) * [The Data Processing Framework](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#the-data-processing-framework) * [Assessment Rubric](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#assessment-rubric) * [Module 1: Arithmetic & Stdin Tools](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-1-arithmetic--stdin-tools) * [Exercise 1.1: The Expense Splitter (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-11-the-expense-splitter-build) * [Exercise 1.2: The Rounding Trap (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-12-the-rounding-trap-debug) * [Module 2: Testing & Verification](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-2-testing--verification) * [Exercise 2.1: The Bulletproof Calculator (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-21-the-bulletproof-calculator-build) * [Exercise 2.2: The Green Light Lie (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-22-the-green-light-lie-debug) * [Module 3: CSV Processing](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-3-csv-processing) * [Exercise 3.1: The Messy Payroll (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-31-the-messy-payroll-build) * [Exercise 3.2: The Awk Disaster (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-32-the-awk-disaster-debug) * [Module 4: Categorization & Patterns](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-4-categorization--patterns) * [Exercise 4.1: The Expense Report Builder (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-41-the-expense-report-builder-build) * [Exercise 4.2: The Over-Eager Matcher (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-42-the-over-eager-matcher-debug) * [Module 5: Pipeline Orchestration](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-5-pipeline-orchestration) * [Exercise 5.1: The Quarterly Report (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-51-the-quarterly-report-build) * [Exercise 5.2: The Broken Pipeline (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#exercise-52-the-broken-pipeline-debug) * [Module 6: Capstone Projects](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#module-6-capstone-projects) * [Capstone A: The Freelancer's Tax Prep](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#capstone-a-the-freelancers-tax-prep) * [Capstone B: The Subscription Auditor](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#capstone-b-the-subscription-auditor) * [Capstone C: Your Own Financial Data](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#capstone-c-your-own-financial-data) * [What's Next](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/computation-data-extraction/computation-exercises#whats-next) --- # Version Control Exercises | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You understand how Git tracks changes. You've created commits, viewed diffs, tested ideas on branches, pushed to GitHub, and created pull requests. That's real knowledge -- but there's a gap between knowing the commands and reaching for them instinctively when your AI-generated code breaks at 11pm. These exercises close that gap. Fifteen hands-on challenges across six modules practice the three skills that make Git second nature: **repository management** (building clean, well-structured projects), **problem diagnosis** (reading Git output to find and fix broken state), and **professional workflows** (PRs, transparency, and reusable patterns). Every exercise uses a real Git repository -- you'll run actual commands, not answer theory questions. Download Exercise Files **[Download Version Control Exercises (ZIP)](https://github.com/panaversity/claude-code-version-control-exercises/releases/latest/download/version-control-exercises.zip) ** After downloading, unzip the file. Each exercise has its own folder with an `INSTRUCTIONS.md` and any starter files you need. If the download link doesn't work, visit the [repository releases page](https://github.com/panaversity/claude-code-version-control-exercises/releases) directly. * * * How to Use These Exercises[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#how-to-use-these-exercises "Direct link to How to Use These Exercises") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The workflow for every exercise is the same: 1. **Open the exercise folder** in your terminal 2. **Read the INSTRUCTIONS.md** -- it tells you what's broken or what to build 3. **Read the walkthrough below** for context and learning objectives 4. **For Debug exercises:** run `setup.sh` first to create the broken state 5. **Work through the exercise** -- run real Git commands 6. **Reflect** using the questions provided -- this is where the real learning happens You don't need to complete all 15 in one sitting. Work through one module at a time. Each module builds on the workflows from specific chapter lessons. * * * Key Differences from Chapter Lessons[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#key-differences-from-chapter-lessons "Direct link to Key Differences from Chapter Lessons") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lessons 1-4, you learned each Git workflow in isolation with guided walkthroughs. These exercises are different in three ways: * **No step-by-step instructions.** The exercises describe the scenario, the data, and the goal. You decide the approach, write the prompts, and handle unexpected situations yourself. * **Build + Debug pairing.** Every module has a Build exercise (create something from scratch) and a Debug exercise (find and fix problems in broken state). Debugging a broken repository develops different skills than building a clean one -- you learn to read Git output critically, compare expected vs. actual state, and trace errors back to their root cause. * **Increasing independence.** Modules 1-2 provide starter prompts to scaffold your learning. Modules 3-6 remove the scaffolding. Capstones remove everything -- you design the entire approach. By Module 6, you should be able to face a new Git problem and instinctively reach for the right diagnostic command without needing to review the chapter lessons. * * * Tool Guide[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#tool-guide "Direct link to Tool Guide") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Claude Code** -- Required for all exercises. Every exercise involves running Git commands in the terminal. You'll initialize repos, create commits, manage branches, and diagnose problems. * **A text editor** is helpful for examining files before committing, but not required -- Claude Code can read and display file contents directly. * * * The Git Safety Framework[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#the-git-safety-framework "Direct link to The Git Safety Framework") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Lessons 1-4 taught individual Git skills -- committing, branching, pushing, reviewing. This framework assembles those skills into a single disciplined process. Use it for every exercise: 1. **Assess** -- What's the current state? (`git status`, `git log`, `git branch`) 2. **Plan** -- What needs to happen? What's the safest path? 3. **Protect** -- Create a safety net before risky changes (commit, branch, backup) 4. **Execute** -- Run the commands with intention 5. **Verify** -- Did it work? Check status, diff, log 6. **Document** -- Record what happened and why (commit messages, workflow docs) This framework isn't just for exercises -- it's how professional developers approach every Git operation. When something goes wrong and you feel the urge to start typing commands frantically, stop and start at step 1 instead. The exercises will test whether you follow this framework or skip straight to execution. * * * Assessment Rubric[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#assessment-rubric "Direct link to Assessment Rubric") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For each exercise, evaluate yourself on: | Criteria | Beginner (1) | Developing (2) | Proficient (3) | Advanced (4) | | --- | --- | --- | --- | --- | | **Command Accuracy** | Runs wrong commands, needs multiple attempts | Gets right command but wrong flags/arguments | Correct commands on first try for common scenarios | Handles edge cases and combines commands fluently | | **Safety Awareness** | Makes changes without checking state first | Checks status but doesn't create safety nets | Commits/branches before risky operations consistently | Anticipates risks and creates preemptive safety measures | | **Diagnosis Skill** | Can't interpret git output or error messages | Understands basic status output, struggles with complex state | Reads git output fluently, diagnoses common problems | Diagnoses unusual scenarios, explains root causes clearly | | **Workflow Discipline** | No consistent pattern, ad-hoc commands | Follows basic add-commit cycle | Uses full workflow (status-diff-add-commit-push) consistently | Adapts workflow to context, documents decisions | | **Recovery Confidence** | Panics at errors, afraid to experiment | Uses restore/reset but hesitantly | Recovers from common mistakes quickly and confidently | Handles complex recovery (merge conflicts, history rewriting) calmly | * * * Module 1: Repository Foundations[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-1-repository-foundations "Direct link to Module 1: Repository Foundations") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Building clean, well-structured repositories from the start > > Lessons 1 and 3 taught you how to initialize repos, stage files selectively, create meaningful commits, and protect secrets with .gitignore. These exercises push those skills into realistic scenarios where the project structure is messier and the decisions about what to track are less obvious than the lesson examples. 1.1Project Kickoff Start ### Exercise 1.1 -- Project Kickoff (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-11----project-kickoff-build "Direct link to Exercise 1.1 -- Project Kickoff (Build)") **The Problem:** Open the `module-1-repo-foundations/exercise-1.1-project-kickoff/` folder. You'll find a `weather-tracker/` project with Python source code in `src/app.py`, data files in `data/`, a configuration file `src/config.json` containing FAKE API secrets, and personal notes in `notes/todo.txt`. Nothing is under version control yet. Your job is to turn this into a clean, well-organized Git repository. **Your Task:** Initialize Git, create a `.gitignore` that protects the config file with secrets and the personal notes, stage files selectively (not with `git add .`), and build a commit history of 3-4 meaningful commits -- not one giant "initial commit" dump. Think about how someone reading your commit log in 3 months would understand the project's setup. **Key Decisions You'll Face:** * Which files go in `.gitignore` and which get tracked? The `data/` folder has real data files -- should they be in version control? * What order should you commit files in? There's a logical progression from project structure to source code to data. * How specific should your commit messages be? "Add weather tracker source code" vs. "Initial commit" -- which helps your future self? **What You'll Learn:** * Why selective staging beats `git add .` -- you control exactly what enters each commit rather than dumping everything in at once * How `.gitignore` protects secrets from ever entering version control, which matters because secrets in Git history persist even after deletion * What makes commit messages useful 3 months later: describing the purpose of each commit, not just listing the files changed **Starter Prompt (Intentionally Vague):** > "Set up Git for this project." **Better Prompt (Build Toward This):** After examining the project structure with `ls -R weather-tracker/`: "Initialize a Git repository in this weather-tracker folder. First, create a .gitignore that excludes src/config.json (it contains API secrets) and the notes/ folder (personal). Then create 3-4 meaningful commits that build up the project logically: start with the .gitignore and project structure, then add the source code, then add the data files. Each commit should have a message that explains why, not just what." **Reflection Questions:** 1. What would happen if you had run `git add .` before creating the `.gitignore`? Would the secrets already be tracked? 2. Could someone reading only your commit log (without looking at the code) understand how the project was built? 3. If a teammate cloned your repo tomorrow, what files would be missing from their copy -- and would they know those files are needed? * * * 1.2Repo Rescue Start ### Exercise 1.2 -- Repo Rescue (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-12----repo-rescue-debug "Direct link to Exercise 1.2 -- Repo Rescue (Debug)") **The Problem:** Open the `module-1-repo-foundations/exercise-1.2-repo-rescue/` folder and run `setup.sh`. This creates a repository with three problems: an `.env` file containing FAKE API keys was committed to history, the commit messages are useless ("fix", "stuff", "asdf"), and there is no `.gitignore` at all. Run `git log` and `git status` to see the full picture. **Your Task:** Diagnose what went wrong by examining the commit history. Add a `.gitignore` that will prevent `.env` files from being tracked going forward. Note that simply adding `.env` to `.gitignore` doesn't remove it from history -- the file is already committed. Document what happened in a `REMEDIATION.md` file explaining the problems you found and the steps you took to fix them. **Investigation Commands:** * `git log --oneline` -- See the commit history (notice the useless messages) * `git show ` -- Inspect individual commits to find what was committed * `git log --all --diff-filter=A -- .env` -- Find exactly when the `.env` file was first added **What You'll Learn:** * How `git log` reveals project health -- bad commit messages are a symptom of careless version control habits * Why secrets committed to history are dangerous even after deletion -- anyone who clones the repo gets the full history including the secrets * How to add `.gitignore` retroactively and why it only prevents future tracking, not past commits **Starter Prompt (Intentionally Vague):** > "Fix this repo." **Better Prompt (Build Toward This):** After running `git log --oneline` and `git status`: "This repo has three problems: committed secrets (.env with API keys), useless commit messages, and no .gitignore. First, create a .gitignore that prevents .env files from being tracked going forward. Then create a REMEDIATION.md documenting: (1) what secrets were exposed and in which commits, (2) why .gitignore alone doesn't remove them from history, (3) what steps would fully remove secrets from history if this were a real project." **Reflection Questions:** 1. After adding `.env` to `.gitignore`, does `git status` still show the `.env` file? Why or why not? 2. If this repo were pushed to a public GitHub repository, who could access the API keys in the old commits? 3. What is the difference between removing a secret from the working directory versus removing it from Git history? * * * Module 2: Change Tracking & Recovery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-2-change-tracking--recovery "Direct link to Module 2: Change Tracking & Recovery") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > **Core Skill:** Fearlessly experimenting because you can always recover > > Lessons 1-2 taught you how to view changes with `git diff`, undo mistakes with `git restore`, and use branches for safe experimentation. These exercises put you in scenarios where AI-generated changes need careful review, selective staging, and confident recovery. 2.1AI Code Review Workflow Start ### Exercise 2.1 -- AI Code Review Workflow (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-21----ai-code-review-workflow-build "Direct link to Exercise 2.1 -- AI Code Review Workflow (Build)") **The Problem:** Open the `module-2-change-tracking/exercise-2.1-ai-review-workflow/` folder. You'll find a working `todo-app/` with `app.py` (a simple task manager) and `tasks.json` (sample data). The app works correctly as-is. Your job is to improve it in 3 rounds, using AI to generate changes each round -- but with disciplined review between each round. **Your Task:** In each of the 3 rounds: (1) ask Claude Code to make one improvement (add a feature, refactor a function, improve error handling), (2) use `git diff` to review every change before committing, (3) use `git restore` to discard any changes you don't want, (4) use selective `git add` to stage only the good changes, (5) commit with a message explaining what you kept and why. After 3 rounds, your commit history should tell the story of intentional, reviewed improvements. **What You'll Learn:** * The review-before-commit discipline that prevents AI disasters -- you see exactly what changed before it becomes permanent * How selective staging gives you granular control over which AI changes you accept and which you discard * Why small iterative commits beat one big dump -- each commit is a verified checkpoint you can return to if the next round breaks something **Starter Prompt (Intentionally Vague):** > "Improve this todo app." **Better Prompt (Build Toward This):** After reading app.py to understand its current features: "I want to improve this todo app in 3 rounds. Round 1: Add a 'priority' field to tasks (high/medium/low). After making changes, I'll review with git diff before committing. Don't make any changes beyond adding priority support -- I want to review one feature at a time." **Reflection Questions:** 1. In which round did `git diff` reveal a change you didn't want? What would have happened if you had committed without reviewing? 2. How did selective staging (adding specific files or hunks) change your relationship with AI-generated code? 3. Looking at your 3-commit history, could a reviewer understand your decision-making process from the commit messages alone? * * * 2.2Recovery Room Start ### Exercise 2.2 -- Recovery Room (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-22----recovery-room-debug "Direct link to Exercise 2.2 -- Recovery Room (Debug)") **The Problem:** Open the `module-2-change-tracking/exercise-2.2-recovery-room/` folder. You'll find three sub-scenarios, each with its own `setup.sh`. Run each setup to create a different broken state: (A) AI made a mess of unstaged changes across 4 files -- some changes are good, others are broken, (B) you accidentally staged test files alongside production code and need to unstage selectively, (C) you committed a broken version and need to undo the commit without losing the good changes from the commit before it. **Your Task:** Fix each scenario using the appropriate recovery tool. Scenario A requires `git restore` on specific files (keep the good changes, discard the bad). Scenario B requires `git restore --staged` to unstage files without discarding changes. Scenario C requires `git revert` to undo the bad commit while preserving history. New Command: git revert Lesson 1 taught `git reset HEAD~1` to undo a commit on your own machine. `git revert` is the safer alternative when history has been shared (pushed to GitHub) -- instead of erasing the commit, it creates a new commit that reverses the changes. Both undo committed mistakes, but `git revert` preserves the full history so collaborators don't lose work. **When to use which:** * `git reset HEAD~1` -- You haven't pushed yet. Erases the commit locally. * `git revert` -- You've already pushed, or you're working with others. Creates an "undo commit." **The Recovery Decision Tree:** Is the change committed?├── Yes, and pushed → git revert (creates an undo commit, preserves history)├── Yes, not pushed → git reset HEAD~1 (erases commit, keeps files)└── No → Is the change staged? ├── Yes → git restore --staged (unstages, keeps changes in working dir) └── No → git restore (discards changes from working directory) **What You'll Learn:** * When to use `git restore` (unstaged changes you want to discard) vs. `git restore --staged` (staged changes you want to unstage) vs. `git reset HEAD~1` (unpushed commits) vs. `git revert` (pushed commits you want to undo) * Why understanding the three stages (working directory, staging area, committed history) makes recovery instinctive rather than frightening * That Git's recovery tools are designed for exactly these situations -- making mistakes is expected, and every stage has a matching undo command **Starter Prompt (Intentionally Vague):** > "Undo these changes." **Better Prompt (Build Toward This):** After running `git status` and `git diff` in Scenario A: "There are changes across 4 files. I want to keep the changes in app.py and utils.py but discard the changes in config.py and test\_broken.py. Use git restore to discard only the files I don't want, leaving the good changes intact. Then verify with git status that only the intended files still show as modified." **Reflection Questions:** 1. Which recovery scenario felt most stressful? Did the stress decrease once you understood which command to use? 2. In Scenario C, why is `git revert` safer than `git reset --hard` for undoing a commit? 3. Could you now recover from any of these three situations without looking up the commands? What made the difference? * * * Module 3: Branch Strategies[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-3-branch-strategies "Direct link to Module 3: Branch Strategies") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Testing multiple ideas safely without risking working code > > Lesson 2 taught you how branches create isolated workspaces for experiments. These exercises challenge you to use branches strategically -- comparing multiple approaches, untangling messes, and making decisions about which code to keep. 3.1Three Approaches Start ### Exercise 3.1 -- Three Approaches (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-31----three-approaches-build "Direct link to Exercise 3.1 -- Three Approaches (Build)") **The Problem:** Open the `module-3-branch-strategies/exercise-3.1-three-approaches/` folder. You'll find `sorting-challenge/` with an unsorted `data.txt` file, a `requirements.md` describing the sorting problem, and `test-cases.txt` with expected output for various inputs. You need to implement a sorting solution, but there are three valid approaches described in the requirements. **Your Task:** Create three branches from `main` -- one for each approach. On each branch, implement the solution (with AI help if you want), test against the provided test cases, and commit working code. Switch between branches to compare the implementations. Merge the best approach into `main`, then delete the other two branches. Your final `main` should have a clean history showing only the winning approach. **Branch Naming Convention:** Use descriptive names: `approach-bubble-sort`, `approach-merge-sort`, `approach-insertion-sort` -- not `branch1`, `branch2`, `branch3`. Good branch names are documentation. **What You'll Learn:** * How branches make comparison systematic rather than chaotic -- each approach lives in its own timeline, isolated from the others * The discipline of test-before-merge: you don't merge an approach until it passes all test cases * Why deleting losing branches keeps repos clean -- abandoned experiments shouldn't clutter your branch list **Reflection Questions:** 1. How did you decide which approach was "best"? Was it code simplicity, performance, or how well it handled the test cases? 2. Did working on one branch influence your approach on another? How did isolation help or hinder that? 3. After merging the winner, did the final `main` branch history tell a clear story -- or did it include merge noise? * * * 3.2Branch Tangle Start ### Exercise 3.2 -- Branch Tangle (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-32----branch-tangle-debug "Direct link to Exercise 3.2 -- Branch Tangle (Debug)") **The Problem:** Open the `module-3-branch-strategies/exercise-3.2-branch-tangle/` folder and run `setup.sh`. This creates a repo with tangled branches: someone committed feature code to `main` by accident, left an orphaned branch `old-feature` with 2 commits that `main` needs, and created a merge conflict between `feature-a` and `feature-b` that both modified the same file. Run `git log --all --graph --oneline` to see the mess. An `expected-state.md` file describes what the clean repository should look like. **Your Task:** Untangle the repo. Move the accidental commit off `main` onto its own feature branch. Cherry-pick the needed commits from `old-feature` into `main`. Resolve the merge conflict between `feature-a` and `feature-b` by merging one into the other. The final state should match `expected-state.md`. New Commands for This Exercise This exercise introduces two Git operations not covered in the chapter lessons: **`git cherry-pick `** -- Copies a specific commit from any branch onto your current branch. Use it when you need individual commits without merging an entire branch. Find commit hashes with `git log --oneline`. **Merge conflict resolution** -- When two branches modify the same lines, Git marks the conflict in the file with `<<<<<<<`, `=======`, and `>>>>>>>` markers. To resolve: (1) open the conflicted file, (2) choose the correct code and delete the markers, (3) `git add` the resolved file, (4) `git commit` to complete the merge. **Start with the Graph:** Before touching anything, run `git log --all --graph --oneline --decorate` and draw a diagram of what you see. Understanding the current state is 80% of the solution. Compare your diagram against `expected-state.md` to identify exactly what needs to move where. **What You'll Learn:** * How `git log --all --graph` reveals branch structure -- the visual representation shows exactly where branches diverge and which commits are orphaned * When `cherry-pick` rescues orphaned commits -- you can extract individual commits from any branch without merging the whole branch * How to resolve merge conflicts step by step: open the conflicted file, choose the correct code, stage the resolution, complete the merge **Reflection Questions:** 1. Which part of the tangle was hardest to understand from the graph? Which was hardest to fix? 2. Did you use the Git Safety Framework (Assess-Plan-Protect-Execute-Verify) or did you jump straight to commands? 3. How would you prevent this kind of branch tangle from happening in a real project? * * * Module 4: GitHub & Remote Workflows[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-4-github--remote-workflows "Direct link to Module 4: GitHub & Remote Workflows") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Backing up work and collaborating through the cloud > > Lesson 3 taught you how to push projects to GitHub, clone repos, and establish the push-pull workflow. These exercises have you practice the full local-to-remote cycle and diagnose common remote configuration problems. 4.1Cloud Safety Net Start ### Exercise 4.1 -- Cloud Safety Net (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-41----cloud-safety-net-build "Direct link to Exercise 4.1 -- Cloud Safety Net (Build)") **The Problem:** Open the `module-4-github-remote/exercise-4.1-cloud-safety-net/` folder. You'll find a `portfolio-project/` with HTML, CSS, and JavaScript files for a personal portfolio site, plus a `secrets.json` file containing FAKE API tokens that must never be pushed to GitHub. Nothing is under version control yet, and nothing is on GitHub. **Your Task:** Initialize the repo, create a `.gitignore` that protects `secrets.json`, commit your files, create a GitHub repository, and push. Then verify your backup works: clone the repo to a separate `test-clone/` folder, confirm all files are present and `secrets.json` is absent. Finally, make a change locally, push, pull in the clone, and verify the change appears. This establishes the complete push-verify-pull cycle. **Verification Checklist:** 1. `secrets.json` does NOT appear in `git status` after adding `.gitignore` 2. After pushing, GitHub web interface shows all expected files but NOT `secrets.json` 3. After cloning to `test-clone/`, `ls` confirms `secrets.json` is absent 4. After making a local change and pushing, `git pull` in the clone retrieves it **What You'll Learn:** * Why `.gitignore` must be committed before sensitive files are added -- if you add secrets first and `.gitignore` second, the secrets are already tracked * How clone-and-verify proves your backup works -- you don't trust a backup you haven't tested * The push-check-pull rhythm of professional development: make changes, push, verify on GitHub, pull on other machines **Reflection Questions:** 1. When you cloned to `test-clone/`, was `secrets.json` present? What would have happened if you had committed it before creating `.gitignore`? 2. Did you check the GitHub web interface to verify your push arrived? What would you do if the push succeeded locally but the files didn't appear on GitHub? 3. How often should you push to GitHub for a solo project? What's the cost of pushing too rarely vs. too frequently? * * * 4.2Remote Troubles Start ### Exercise 4.2 -- Remote Troubles (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-42----remote-troubles-debug "Direct link to Exercise 4.2 -- Remote Troubles (Debug)") **The Problem:** Open the `module-4-github-remote/exercise-4.2-remote-troubles/` folder. You'll find three sub-scenarios, each with its own `setup.sh`. Run each setup to create a different remote problem: (A) the local branch is called `master` but GitHub expects `main` -- push is rejected, (B) the remote URL points to a repository that doesn't exist -- all remote operations fail, (C) a push is rejected because the remote has commits that your local branch doesn't have. **Your Task:** Diagnose and fix each scenario. Scenario A requires renaming the local branch with `git branch -m`. Scenario B requires updating the remote URL with `git remote set-url` (a new command -- it changes where your repo points without removing and re-adding the remote). Scenario C requires pulling remote changes before pushing. For each scenario, use diagnostic commands before attempting a fix. **Diagnostic Commands for Remote Issues:** * `git remote -v` -- Shows the URLs your repository is connected to (fetch and push) * `git branch -a` -- Shows all branches, both local and remote-tracking * `git status` -- Shows whether your branch is ahead of, behind, or diverged from the remote * `git log --oneline origin/main..HEAD` -- Shows commits you have that the remote doesn't **What You'll Learn:** * How `git branch -m` renames branches -- the `master` to `main` rename is one of the most common Git operations you'll encounter in existing repositories * How `git remote -v` reveals configuration issues -- when remote operations fail, this is always the first diagnostic command to run * Why pull-before-push prevents rejection -- the remote has the authoritative history, and you must integrate it before adding your own **Reflection Questions:** 1. In Scenario A, why would the branch names mismatch in the first place? What default branch name does `git init` use on your system? 2. In Scenario B, what was the error message when you tried to push to a non-existent URL? Was it clear enough to diagnose without `git remote -v`? 3. In Scenario C, what happens if you force-push instead of pulling? Why is that dangerous in a team setting? * * * Module 5: Pull Requests & Code Review[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-5-pull-requests--code-review "Direct link to Module 5: Pull Requests & Code Review") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Reviewing code professionally with AI transparency > > Lesson 4 taught you how pull requests enable code review, discussion, and AI transparency. These exercises have you write real PR descriptions, self-review diffs, and fix PRs that violate professional standards. 5.1Transparent PR Start ### Exercise 5.1 -- Transparent PR (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-51----transparent-pr-build "Direct link to Exercise 5.1 -- Transparent PR (Build)") **The Problem:** Open the `module-5-pull-requests/exercise-5.1-transparent-pr/` folder. You'll find a `calculator-feature/` repo with a `main` branch containing `calculator.py` and `test_calculator.py`, plus a `feature/error-handling` branch that adds input validation and error messages. A `pr-template.md` provides the PR description format your team uses. **Your Task:** Write a complete PR description using the template. Self-review the diff between `main` and `feature/error-handling` line by line -- annotate what each change does and whether it's correct. Document which parts were AI-generated vs. hand-written (the INSTRUCTIONS.md tells you which). If you find issues during self-review, fix them on the feature branch before finalizing your PR description. **PR Template Sections to Complete:** 1. **Summary** -- What does this PR do and why? 2. **Changes** -- List of specific modifications with rationale 3. **Testing** -- How you verified the changes work 4. **AI Transparency** -- Which parts were AI-generated, what you modified, what you verified 5. **Risks** -- Anything a reviewer should pay extra attention to **What You'll Learn:** * How PR descriptions serve as documentation for future you -- three months later, the PR description is your only record of why these changes were made * Why AI transparency builds trust with reviewers -- knowing which parts are AI-generated helps reviewers calibrate their scrutiny * The self-review habit that catches issues before others do -- reviewing your own diff with fresh eyes is the cheapest quality check available **Reflection Questions:** 1. Did self-reviewing the diff reveal any issues you hadn't noticed while coding? What kind of issues? 2. How did you decide what level of detail to include in the AI transparency section? Is there such a thing as too much disclosure? 3. Could a reviewer who has never seen this codebase understand your PR from the description alone? * * * 5.2PR Cleanup Start ### Exercise 5.2 -- PR Cleanup (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-52----pr-cleanup-debug "Direct link to Exercise 5.2 -- PR Cleanup (Debug)") **The Problem:** Open the `module-5-pull-requests/exercise-5.2-pr-cleanup/` folder. You'll find a `bad-prs/` folder with three markdown files representing bad PRs: `pr-1-no-description.md` has the description "Updated code" with 47 files changed and zero context, `pr-2-hidden-ai.md` contains AI-generated code with no attribution despite the team's transparency policy, and `pr-3-failing-checks.md` has a good description but the CI checks are failing and the author wants to merge anyway. **Your Task:** Rewrite each PR to professional standards. For PR 1, write a proper description explaining what changed and why based on the diff summary provided in the file. For PR 2, add appropriate AI transparency documentation following the team's policy. For PR 3, identify what the failing checks mean and explain what must be fixed before merging. Each rewritten PR should pass your team's review standards. **The Professional Standard:** A good PR description lets a reviewer who has never seen the codebase understand: what problem exists, what approach was taken, what was tested, and what risks remain. If a reviewer has to ask "what does this PR do?" -- the description failed. **What You'll Learn:** * What reviewers actually need to evaluate code: context about the problem being solved, the approach taken, testing performed, and any risks or caveats * How hiding AI usage erodes team trust -- transparency about tools used is a professional norm, not a weakness * How failing CI tells you what to fix before requesting review -- merge-blocking checks exist to protect the codebase **Reflection Questions:** 1. For PR 1, how much context did you need to add before the 47-file change became understandable? Could the author have split this into smaller PRs? 2. For PR 2, where is the line between appropriate AI disclosure and over-documentation? 3. For PR 3, what would happen if the team adopted a culture of merging with failing checks "just this once"? * * * Module 6: Workflow Documentation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-6-workflow-documentation "Direct link to Module 6: Workflow Documentation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Capturing reusable patterns from experience > > Lesson 4 taught you three reusable patterns (Commit Before Experimenting, Branch-Test-Merge, Push for Backup) and professional review habits. These exercises extend those patterns into team workflow documentation -- creating shared conventions for a team and auditing existing documentation for gaps that caused real incidents. 6.1Workflow Builder Start ### Exercise 6.1 -- Workflow Builder (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-61----workflow-builder-build "Direct link to Exercise 6.1 -- Workflow Builder (Build)") **The Problem:** Open the `module-6-workflow-docs/exercise-6.1-workflow-builder/` folder. You'll find an `ecommerce-project/` with a `README.md` describing a new online store, `requirements.md` listing the features to build, and `team-info.md` describing a 4-person team (frontend dev, backend dev, designer, and you as tech lead). The team has never used Git together before. **Your Task:** Create a comprehensive `git-workflow.md` that covers: (1) commit message conventions with examples of good and bad messages, (2) branching strategy (when to create branches, naming conventions, who merges to main), (3) a PR template the team will use for every pull request including an AI transparency section, and (4) a push schedule (how often to push, when to pull, how to handle conflicts). The document should be specific enough that a new team member could follow it on day one without asking questions. **Quality Test:** Imagine the designer (who has never used Git) joins the team tomorrow. Could they follow your workflow document to make their first commit, create a branch, and submit a PR? If not, what's missing? **What You'll Learn:** * How workflow documentation eliminates repeated decision-making -- the team makes decisions once and follows the documented pattern * Why team conventions prevent the chaos of "everyone does it differently" -- four developers with four different commit styles create an unreadable history * How decision frameworks reduce cognitive load -- instead of debating "should I branch for this?" every time, the documentation answers it **Reflection Questions:** 1. Which section of the workflow document was hardest to write? Was it because the decision had multiple valid options? 2. How did you balance specificity (precise rules) with flexibility (room for judgment)? Give an example of each. 3. If the team follows your workflow for a month and discovers a section that doesn't work, what's the process for updating it? * * * 6.2Pattern Audit Start ### Exercise 6.2 -- Pattern Audit (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-62----pattern-audit-debug "Direct link to Exercise 6.2 -- Pattern Audit (Debug)") **The Problem:** Open the `module-6-workflow-docs/exercise-6.2-pattern-audit/` folder. You'll find a `git-workflow.md` that a team has been using for 3 months, plus three incident reports: `incident-1.md` (a developer pushed secrets to main because the workflow didn't mention .gitignore), `incident-2.md` (two developers merged conflicting branches because the workflow didn't specify merge order), and `incident-3.md` (a PR was merged without review because the workflow said "reviews are recommended" instead of "reviews are required"). **Your Task:** Read each incident report, trace the root cause back to a gap or ambiguity in `git-workflow.md`, and fix the documentation. For each fix, add a "Rationale" comment explaining why the original wording was insufficient and what the new wording prevents. **The Audit Process:** For each incident: (1) Read the incident report and identify the developer action that caused the problem, (2) Find the section of `git-workflow.md` that should have prevented it, (3) Determine whether the section is missing, ambiguous, or uses "recommended" when it should say "required", (4) Write the fix with a rationale comment. **What You'll Learn:** * How to trace incidents back to missing documentation -- the workflow gap is always upstream of the mistake * Why workflows need regular updating -- documentation written at project start can't anticipate every situation that arises during development * How gap analysis improves team processes -- each incident reveals a pattern that the documentation should have prevented **Reflection Questions:** 1. Which incident could have been prevented with a single sentence added to the workflow? Which required a whole new section? 2. Were the workflow gaps caused by missing information or ambiguous wording? Which is more dangerous? 3. How would you schedule regular workflow audits to catch gaps before they cause incidents? * * * Module 7: Capstone Projects[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-7-capstone-projects "Direct link to Module 7: Capstone Projects") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Choose one (or more). Spend real time on it. This is where everything comes together.** There are no starter prompts -- you design the entire approach yourself. Each capstone requires applying skills from multiple modules to solve a realistic problem. Where module exercises test individual skills, capstones test your ability to orchestrate those skills into a coherent workflow. The quality of your Git practices matters as much as the result -- anyone reviewing your repository should see professional version control habits from the first commit to the last. The progression across capstones is intentional: Capstone A walks you through the complete Git lifecycle with a project you build yourself. Capstone B applies those skills to building a real professional presence on GitHub. Capstone C puts you in a forensic investigation role, using Git's history tools to reconstruct what happened in a damaged repository. Each capstone demands more judgment and different skills than the last. AFull Stack Git Journey Start ### Capstone A -- Full Stack Git Journey (Integration)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#capstone-a----full-stack-git-journey-integration "Direct link to Capstone A -- Full Stack Git Journey (Integration)") Take an empty folder through the complete Git lifecycle: initialize a repository with proper `.gitignore`, build a small project with 8-10 structured commits, create a feature branch to test an AI-generated improvement, merge the successful feature, push to GitHub, create a PR with AI transparency documentation, and write a `git-workflow.md` capturing your process. This capstone hits every module: repository setup (Module 1), change tracking and recovery (Module 2), branching (Module 3), GitHub remote workflow (Module 4), pull requests (Module 5), and workflow documentation (Module 6). The project itself can be anything -- a calculator, a todo app, a personal website. What matters is the Git workflow around it. **Estimated time:** 2-4 hours. **What You'll Learn:** * How all Git skills compose into a complete professional workflow -- each skill supports the others rather than existing in isolation * Where to create safety nets in a real project lifecycle -- commits before experiments, branches before risky changes, pushes before major milestones * How documentation captures your decisions for future reference -- your `git-workflow.md` becomes a reusable template for every future project * * * BPortfolio Launch Start ### Capstone B -- Portfolio Launch (Real-world)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#capstone-b----portfolio-launch-real-world "Direct link to Capstone B -- Portfolio Launch (Real-world)") Build a professional GitHub presence from scratch. Create 3 repositories showcasing different skills (choose from: a web project, a data processing tool, an automation script, or a documentation site). For each repository: write a clear README that explains the project, its purpose, and how to run it; configure `.gitignore` with appropriate patterns for the technology used; create a clean commit history that tells a development story; and push to GitHub with descriptive repository descriptions. Finally, create a profile README (the special repository named after your GitHub username) that introduces you and links to your pinned repositories. Use the `portfolio-templates/` folder for README templates and `.gitignore` examples. **Estimated time:** 2-4 hours. **What You'll Learn:** * How to curate a professional GitHub profile that employers and collaborators actually want to see * Why clean commit history demonstrates development discipline -- messy commits signal careless habits * How README quality signals project quality -- a well-documented repository earns trust before anyone reads the code * * * CGit Forensics Start ### Capstone C -- Git Forensics (Forensics)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#capstone-c----git-forensics-forensics "Direct link to Capstone C -- Git Forensics (Forensics)") Open the `capstone-C-git-forensics/` folder and run `setup.sh`. This creates a "crime scene" repository: secrets were committed and then deleted (but they're still in history), someone force-pushed over a teammate's work (the overwritten commits are in the reflog), there are unresolved merge conflicts in two files, and three abandoned experiment branches contain code that was never reviewed. Investigate the repository using `git log`, `git reflog`, `git blame`, and `git diff`. Write a forensic report (`FORENSIC-REPORT.md`) documenting: what happened (timeline of events reconstructed from Git history), what damage was done (secrets exposed, work lost, conflicts unresolved), and a recovery plan (how to fix each issue, in what order, with what commands). **Estimated time:** 2-4 hours. **What You'll Learn:** * How `git log`, `git reflog`, and `git blame` reveal what happened and when -- these are your forensic investigation tools * Why some Git mistakes are hard to fully reverse -- force-pushed history and committed secrets require different recovery strategies than simple undos * How forensic skills build debugging intuition -- investigating someone else's mistakes teaches you to recognize patterns that prevent your own * * * What's Next[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#whats-next "Direct link to What's Next") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've practiced the three core skills -- **repository management**, **problem diagnosis**, and **professional workflows** -- across 15 exercises. These skills compound: every exercise makes Git feel more instinctive, so when AI-generated code breaks your project at midnight, you'll reach for the right recovery command without hesitation. Next in the **Chapter Quiz**, you'll test your understanding of Git concepts and scenarios. The version control patterns you've built here become your safety net for the next chapter, where you'll build your own AI Employee that uses these same Git workflows to work autonomously on your projects. * [How to Use These Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#how-to-use-these-exercises) * [Key Differences from Chapter Lessons](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#key-differences-from-chapter-lessons) * [Tool Guide](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#tool-guide) * [The Git Safety Framework](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#the-git-safety-framework) * [Assessment Rubric](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#assessment-rubric) * [Module 1: Repository Foundations](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-1-repository-foundations) * [Exercise 1.1 -- Project Kickoff (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-11----project-kickoff-build) * [Exercise 1.2 -- Repo Rescue (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-12----repo-rescue-debug) * [Module 2: Change Tracking & Recovery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-2-change-tracking--recovery) * [Exercise 2.1 -- AI Code Review Workflow (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-21----ai-code-review-workflow-build) * [Exercise 2.2 -- Recovery Room (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-22----recovery-room-debug) * [Module 3: Branch Strategies](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-3-branch-strategies) * [Exercise 3.1 -- Three Approaches (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-31----three-approaches-build) * [Exercise 3.2 -- Branch Tangle (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-32----branch-tangle-debug) * [Module 4: GitHub & Remote Workflows](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-4-github--remote-workflows) * [Exercise 4.1 -- Cloud Safety Net (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-41----cloud-safety-net-build) * [Exercise 4.2 -- Remote Troubles (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-42----remote-troubles-debug) * [Module 5: Pull Requests & Code Review](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-5-pull-requests--code-review) * [Exercise 5.1 -- Transparent PR (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-51----transparent-pr-build) * [Exercise 5.2 -- PR Cleanup (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-52----pr-cleanup-debug) * [Module 6: Workflow Documentation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-6-workflow-documentation) * [Exercise 6.1 -- Workflow Builder (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-61----workflow-builder-build) * [Exercise 6.2 -- Pattern Audit (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#exercise-62----pattern-audit-debug) * [Module 7: Capstone Projects](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#module-7-capstone-projects) * [Capstone A -- Full Stack Git Journey (Integration)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#capstone-a----full-stack-git-journey-integration) * [Capstone B -- Portfolio Launch (Real-world)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#capstone-b----portfolio-launch-real-world) * [Capstone C -- Git Forensics (Forensics)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#capstone-c----git-forensics-forensics) * [What's Next](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/version-control-exercises#whats-next) --- # Git Foundations | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Open a Google Doc. Type a paragraph. Delete it. Press Ctrl+Z. It comes back. Now open a project folder with 47 files. Ask Claude Code to reorganize them. It moves, renames, and rewrites 12 files. The result is wrong. Press Ctrl+Z. Nothing happens. There is no undo for a project folder. **Unless you have Git.** > **"Ctrl+Z works for one file. Git works for your entire life's work."** In 1998, someone at Pixar accidentally ran a delete command on the Toy Story 2 production files. The animation studio lost 90% of two years of work in seconds. Their backup system had silently failed weeks earlier. The entire movie was saved only because one employee, Galyn Susman, had a copy on her home computer: she'd been working remotely while caring for her newborn. Without that stroke of luck, one of the most beloved animated films ever made would have been lost forever. ([Source](https://thenextweb.com/news/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good) ) Version control exists because of disasters like this. Prerequisites Before following along, complete the **Before You Start** setup in the [chapter introduction](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/) : you need Git installed, a GitHub account, and a one-time Git configuration. * * * Sarah's Problem[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#sarahs-problem "Direct link to Sarah's Problem") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Sarah is organizing a community fundraiser. She has a folder with budget spreadsheets, flyer designs, and a volunteer list. Last Tuesday she spent two hours updating the budget. Then she tried a new layout and accidentally saved over the old version. The old budget is gone. No undo. No backup. Two hours of work, lost. "I wish I could go back to yesterday's version," she says. Git solves exactly this problem. * * * The Concept: Snapshots, Not Files[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#the-concept-snapshots-not-files "Direct link to The Concept: Snapshots, Not Files") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've used Track Changes in Word or Google Docs. You can see who changed what and when. Git is Track Changes for your entire project folder, not just one document. Every change, every file, with the ability to rewind to any moment in history. Each snapshot is called a **commit** (a photograph of your entire project at one moment in time). You decide when to take each photo and what to include. * * * Watch Your Agent Do It[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#watch-your-agent-do-it "Direct link to Watch Your Agent Do It") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's create Sarah's fundraiser project and watch what happens when we tell Claude Code to protect it. **What you tell your agent**: "Create a new folder called fundraiser-project with a budget file and a volunteer list. Then set up version control." **What the agent does**: mkdir fundraiser-projectcd fundraiser-projectecho "Budget: $500 for venue, $200 for food" > budget.txtecho "Volunteers: Sarah, Maya, Jordan" > volunteers.txtgit init **What this means**: `git init` creates a hidden `.git` folder inside your project. This folder is Git's brain: it stores your entire project history. Delete `.git` and you lose all your snapshots. Keep it safe. You can see it yourself: ls -la The `.git` directory appears in the listing. Everything Git knows about your project lives there. * * * What Your Agent Sees[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#what-your-agent-sees "Direct link to What Your Agent Sees") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After creating those files, your agent checks what changed: **What you tell your agent**: "What's the current state of my project?" **What the agent does**: git status **What you see**: On branch mainNo commits yetUntracked files: (use "git add ..." to include in what will be committed) budget.txt volunteers.txt Git sees two files it doesn't know about yet. "Untracked" means Git noticed them but hasn't started protecting them. Think of it like a hotel guest who walked in but hasn't checked in yet. * * * The Staging Area: Packing Your Suitcase[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#the-staging-area-packing-your-suitcase "Direct link to The Staging Area: Packing Your Suitcase") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Your agent doesn't photograph everything at once. You choose what goes in each snapshot. This in-between step is called the **staging area**. Think of it like packing a suitcase. You lay out everything on the bed (your working directory), then decide what goes in the suitcase (staging area), then close the suitcase (commit). **What you tell your agent**: "Prepare the budget file for saving." **What the agent does**: git add budget.txtgit status **What you see**: Changes to be committed: new file: budget.txtUntracked files: volunteers.txt `budget.txt` moved from "untracked" to "ready to commit" (shown in green). `volunteers.txt` is still waiting. You control what goes in each snapshot. **Why not add everything?** Sometimes you have personal notes, temporary files, or half-finished work you don't want in the snapshot. The staging area gives you control. Let's stage everything and take our first snapshot: git add volunteers.txtgit commit -m "Initial fundraiser files: budget and volunteer list" **What happened**: Git took a photograph of your entire project. Both files are protected. The message describes what this snapshot contains. Verify it: git log You see your commit with a unique ID, your name, the date, and the message. This is your first restore point. * * * When Things Go Wrong[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#when-things-go-wrong "Direct link to When Things Go Wrong") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sarah edits the budget. She accidentally deletes the food line and types nonsense over the venue costs. Let's simulate this. Make a bad edit: **What you tell your agent**: "Replace the budget contents with this broken version." **What the agent does**: echo "Budget: XXXXX broken data" > budget.txt Sarah panics. The file is ruined. But she committed earlier. Git has her back. * * * Seeing What Changed[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#seeing-what-changed "Direct link to Seeing What Changed") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before undoing anything, look at what changed. **What you tell your agent**: "Show me what's different from the last snapshot." **What the agent does**: git diff budget.txt **What you see**: --- a/budget.txt+++ b/budget.txt@@ -1 +1 @@-Budget: $500 for venue, $200 for food+Budget: XXXXX broken data The `-` line shows what was there before. The `+` line shows what's there now. Red means removed. Green means added. This is like comparing two photographs side by side. * * * Sarah Panics (Then Recovers)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#sarah-panics-then-recovers "Direct link to Sarah Panics (Then Recovers)") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sarah sees the diff. She knows the file is broken. She Googles "undo git changes" and finds a StackOverflow answer suggesting `git reset --hard`. She runs it without reading the fine print. git reset --hard Everything goes back to normal. The broken budget is fixed. She exhales. Then she checks the volunteer list: the one she spent yesterday afternoon updating. **It's gone too.** `git reset --hard` doesn't undo one file. It resets _everything_ to the last commit. Her volunteer updates weren't committed yet. They're gone. Actually gone. Sarah just learned the most expensive lesson in version control: **the nuclear option doesn't have a confirmation dialog.** There's a better way. Git has three levels of undo: each one more powerful than the last, each one more dangerous. The key is choosing the smallest tool for the job. * * * Three Levels of Undo[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#three-levels-of-undo "Direct link to Three Levels of Undo") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the decision tree for what to tell your agent. Every mistake falls into one of three categories. ### Scenario 1: You edited a file but haven't staged it yet[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#scenario-1-you-edited-a-file-but-havent-staged-it-yet "Direct link to Scenario 1: You edited a file but haven't staged it yet") Sarah just made a bad edit. She hasn't run `git add`. The fix is simple. **What you tell your agent**: "Throw away my changes to budget.txt. Go back to the last saved version." **What the agent does**: git restore budget.txt Check the file: cat budget.txt The original budget is back. The bad edit is gone. The file still exists: Git just rewound it to the last committed version. ### Scenario 2: You staged the wrong file[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#scenario-2-you-staged-the-wrong-file "Direct link to Scenario 2: You staged the wrong file") You accidentally told Git to include a file you didn't want in the next snapshot. **What you tell your agent**: "I staged the wrong file. Remove it from the staging area but keep the file." **What the agent does**: git restore --staged budget.txt The file leaves the staging area. It goes back to "modified" or "untracked." Nothing is deleted. The file is exactly as you left it. ### Scenario 3: You already committed the mistake[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#scenario-3-you-already-committed-the-mistake "Direct link to Scenario 3: You already committed the mistake") You took a snapshot that includes bad changes. You need to undo the whole commit. **What you tell your agent**: "Undo my last commit but keep the files so I can fix them." **What the agent does**: git reset HEAD~1 The commit disappears from history. Your files stay in the working directory. You can fix them and commit again. caution `git reset --hard HEAD~1` deletes the commit **and** throws away all file changes. This is what happened to Sarah's volunteer list: the nuclear option with no confirmation dialog. Only use it when you're certain you want to erase everything. ### Quick Reference: Which Undo Do I Need?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#quick-reference-which-undo-do-i-need "Direct link to Quick Reference: Which Undo Do I Need?") | Situation | What to tell your agent | What it runs | | --- | --- | --- | | Bad edit, not staged | "Throw away my changes to this file" | `git restore ` | | Staged the wrong file | "Unstage this file" | `git restore --staged ` | | Bad commit, keep files | "Undo my last commit" | `git reset HEAD~1` | | Bad commit, erase everything | "Erase my last commit completely" | `git reset --hard HEAD~1` | * * * Sarah's project is safe: she can undo bad edits, unstage mistakes, and roll back commits. But she learned something the hard way: one wrong command wiped out work she hadn't committed yet. What if she could try risky ideas in a completely separate space, where mistakes can't touch her real project? That's what branches solve, and that's the next lesson. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------- **Understand the snapshot concept:** > "I just learned that Git commits are snapshots of my entire project folder, not just individual files. Explain how this is different from saving a single file. Use a real-world analogy that doesn't involve video games." **Explore the staging area:** > "Why does Git make me choose which files to include in each commit? Why not just save everything automatically? Give me three real scenarios where selective staging is useful." **Break something on purpose:** > "Help me practice Git recovery. Create a small project with three files, then walk me through deliberately breaking one file and recovering it with git restore. Then walk me through staging the wrong file and unstaging it. I want to feel the panic and the relief so I'm not scared when it happens for real." **Write better commit messages:** > "Here are five bad commit messages: 'update', 'fix stuff', 'changes', 'WIP', 'asdf'. For each one, rewrite it as a clear message that explains WHY the change was made, not just WHAT changed. Then give me a template I can follow: what should the first line say? When should I add more detail?" * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#flashcards-study-aid "Direct link to Flashcards Study Aid") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Sarah's Problem](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#sarahs-problem) * [The Concept: Snapshots, Not Files](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#the-concept-snapshots-not-files) * [Watch Your Agent Do It](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#watch-your-agent-do-it) * [What Your Agent Sees](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#what-your-agent-sees) * [The Staging Area: Packing Your Suitcase](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#the-staging-area-packing-your-suitcase) * [When Things Go Wrong](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#when-things-go-wrong) * [Seeing What Changed](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#seeing-what-changed) * [Sarah Panics (Then Recovers)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#sarah-panics-then-recovers) * [Three Levels of Undo](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#three-levels-of-undo) * [Scenario 1: You edited a file but haven't staged it yet](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#scenario-1-you-edited-a-file-but-havent-staged-it-yet) * [Scenario 2: You staged the wrong file](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#scenario-2-you-staged-the-wrong-file) * [Scenario 3: You already committed the mistake](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#scenario-3-you-already-committed-the-mistake) * [Quick Reference: Which Undo Do I Need?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#quick-reference-which-undo-do-i-need) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#try-with-ai) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/version-control/git-foundations#flashcards-study-aid) --- # Capstone - Budget Tracker Complete App | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You started this chapter with a script that couldn't handle a second user. Look where you are now: typed models, safe transactions, cloud deployment, independent verification. That's not a script anymore -- that's a system. This lesson pulls every piece together. You will wire up your models, CRUD operations, transaction safety, Neon connection, and verification gate into a single application -- then run it end to end and collect the proof that it works. Key Terms for This Lesson * **Evidence bundle**: A collection of test results that PROVES your system works -- not "I think it works" but "here's the proof it works" * **Release gate**: A checkpoint that must pass before your code goes to production -- if the gate fails, you stop and fix before shipping The Integration Contract[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#the-integration-contract "Direct link to The Integration Contract") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lesson 7, you built independent verification for high-stakes outputs. Now you combine every layer into one coherent application with five commitments: 1. **Models enforce schema** (`User`, `Category`, `Expense`). 2. **CRUD paths include rollback** on write failure. 3. **Summary queries avoid N+1** patterns. 4. **Neon connection uses pooled** pre-ping configuration. 5. **High-stakes reports run independent verification** before release. Each commitment maps to a lesson you already completed. The capstone is not new learning -- it is proof that all the pieces hold together under one roof. You might be thinking: "Do I really need all these evidence gates?" For a toy project, probably not. For anything touching money, health data, or compliance? Absolutely. This evidence bundle pattern works for any system: e-commerce checkout flows, healthcare record systems, financial trading platforms. The Evidence Pipeline[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#the-evidence-pipeline "Direct link to The Evidence Pipeline") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here is the sequence your capstone will follow. Each gate must pass before the next one runs: Evidence Pipeline: Schema Gate CRUD Gate Rollback Gate ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Create │ │ Insert │ │ Force │ │ tables │──────►│ + Read │──────►│ failure │ │ ✓ pass │ │ ✓ pass │ │ 0 partial│ └──────────┘ └──────────┘ │ ✓ pass │ └────┬─────┘ │ Neon Gate Verify Gate ┌──────────┐ ┌──────────┐ │ SELECT 1 │ │ SQL vs │ │ pooled │──────►│ raw CSV │ │ ✓ pass │ │ match? │ └──────────┘ └────┬─────┘ │ ┌────┴─────┐ │ RELEASE │ │ DECISION │ │ verified │ │ or │ │ BLOCKED │ └──────────┘ Five gates. One chain. If any gate fails, you stop and fix before continuing. No skipping ahead. Directing the Monthly Summary[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#directing-the-monthly-summary "Direct link to Directing the Monthly Summary") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Tell your agent to produce a grouped monthly summary using a single database call: no loops, no separate per-category queries. What you tell the agent Generate Alice's January 2024 expense summary grouped by category, sorted by highest total first. Use one database call: no per-category loops. Return category name, expense count, and total for each group. Show the query count in the output so I can verify it used one call. What you verify python run_summary.pyOutput: Alice — January 2024: ┌─────────────────┬───────┬──────────┐ │ Category │ Count │ Total │ ├─────────────────┼───────┼──────────┤ │ Housing │ 1 │ $1500.00 │ │ Food │ 3 │ $287.45 │ │ Transport │ 2 │ $94.20 │ └─────────────────┴───────┴──────────┘ Grand total: $1881.65 Queries used: 1 "Queries used: 1" is the signal that matters. One database call produced the full grouped report. Compare that to the Computation & Data Extraction approach: nested loops, manual grouping, custom sorting: all doing what one SQL query handles natively. Directing the Release Gate[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#directing-the-release-gate "Direct link to Directing the Release Gate") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is where "ready for demo" becomes "ready for release." Direct the agent to compare the SQL summary against the raw CSV ledger, using $0.01 as the mismatch tolerance. What you tell the agent Verify Alice's January 2024 summary from two independent sources: 1. The database (SQL path) 2. The raw CSV ledger file (CSV path) For each category, compare the totals. If any category differs by more than $0.01, block the release and show me the mismatch details with the category name, SQL total, CSV total, and the delta. If all categories match, output: status: verified. What you verify python verify_release.pyOutput (verified): Checking Alice — January 2024... Food: SQL $287.45 | CSV $287.45 | ✓ match Housing: SQL $1500.00 | CSV $1500.00 | ✓ match Transport: SQL $94.20 | CSV $94.20 | ✓ match {"status": "verified"} Release permitted.Output (blocked): Checking Alice — January 2024... Food: SQL $287.45 | CSV $287.95 | ✗ delta $0.50 {"status": "blocked", "reason": "verification_mismatch", "tolerance": "0.01", "mismatches": [{"category": "Food", "sql": "287.45", "raw": "287.95", "delta": "0.50"}]} Release BLOCKED — investigate before shipping. When you see BLOCKED, you do not ship. That is your engineering working correctly. Publishing despite a `blocked` status is a release process failure, not a query problem. The Evidence Bundle[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#the-evidence-bundle "Direct link to The Evidence Bundle") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your capstone produces one JSON artifact that captures every gate result: { "crud_matrix": "pass", "rollback_failure_drill": "pass", "neon_connection_resilience": "pass", "verification_policy_result": "verified_or_blocked_with_reason"} Capstone Run Sequence[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#capstone-run-sequence "Direct link to Capstone Run Sequence") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Run these seven steps in order. If any step fails, stop and fix before continuing: 1. Create schema and seed deterministic fixture data 2. Run CRUD matrix and capture outputs 3. Run forced rollback drill and capture pre/post counts 4. Run Neon connectivity health check 5. Generate monthly SQL summary for one user 6. Run independent raw verification for same user/month 7. Evaluate mismatch policy and produce release decision artifact This is deliberate sequencing. Step 3 proves your rollback actually works under failure. Step 6 proves your SQL output matches an independent source. Step 7 turns all of that into a decision artifact another engineer can read without asking you questions. Gate Language[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#gate-language "Direct link to Gate Language") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When discussing readiness with your team, use precise language: * **"Ready for demo"** means the happy path passes * **"Ready for release"** means failure evidence and verification gate both pass * Never merge release candidates without the evidence bundle attached The difference matters. A demo proves the system can work. A release proves the system can fail safely and recover correctly. One Common Failure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#one-common-failure "Direct link to One Common Failure") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Publishing reports after a mismatch because "the SQL looks right." That is a release process failure, not a query bug. Another failure: claiming "production-ready" without failure-path proof. Passing only the happy path is insufficient for integrity claims. Capstone Self-Review[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#capstone-self-review "Direct link to Capstone Self-Review") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before you call this done, answer these honestly: * Can another engineer rerun your evidence bundle without verbal guidance? * Are all critical thresholds explicit (the $0.01 tolerance, the blocked status rules)? * Did you demonstrate at least one failure path, not only success? * Could a reviewer trace from your requirement description to the agent's output to the evidence artifact quickly? * Did you describe requirements clearly enough that the agent built what you intended on the first try, or did you need to refine? If any answer is "no," the capstone is still in progress. Pause and Reflect Look at what you've built across this chapter. In Lesson 0, you had a script that couldn't handle a second user. Now you have a cloud-deployed, transactionally safe, independently verified system. What's the single most important concept you learned along the way? Computation & Data Extraction deliverable: one tax report for one person. Structured Data deliverable: a cloud-deployed, multi-user, transactionally safe, independently verified financial system. Same you. Different tools. Different capability. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Integrity Gap Audit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#prompt-1-integrity-gap-audit "Direct link to Prompt 1: Integrity Gap Audit") Read my capstone code and classify each critical path:- guaranteed by schema- guaranteed by transaction- guaranteed by verification policy- still vulnerableReturn a prioritized fix list. **What you're learning:** Classifying guarantees by type teaches you to distinguish between what the system prevents automatically (schema violations, partial writes) and what still requires your judgment (verification mismatches, edge cases). This is how experienced engineers think about production risk. ### Prompt 2: Evidence Bundle Generator[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#prompt-2-evidence-bundle-generator "Direct link to Prompt 2: Evidence Bundle Generator") Generate a script that runs:1) CRUD smoke checks2) forced rollback drill3) Neon SELECT 1 health check4) verification gate runThen outputs one JSON evidence bundle. **What you're learning:** Automating your evidence collection turns a manual checklist into a repeatable script. This is the difference between "I checked it once" and "anyone can check it anytime." Automated evidence gates are the foundation of continuous deployment. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") You're building [your project]. Design an evidence bundle with 4 gates:1. What proves your data model is correct?2. What proves your writes are safe?3. What proves your cloud connection is reliable?4. What proves your critical outputs are accurate?For each gate, specify: what you test, what "pass" looks like, and what "fail" means. **What you're learning:** Evidence-driven release decisions transfer to ANY software project. Whether you're shipping a mobile app, deploying an API, or publishing a report -- the pattern is the same: define gates, run tests, collect proof, make decisions based on evidence rather than gut feeling. Your system is correct. But systems change. Users want new features. Schemas need to evolve. Data needs to migrate. Next chapter: how do you evolve a running system without breaking what works? Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#checkpoint "Direct link to Checkpoint") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I ran the full capstone sequence: schema → CRUD → rollback drill → Neon health check → summary → verification gate → release decision. * [ ] I read the evidence bundle output and made an explicit release decision (verified or blocked). * [ ] I directed the agent through at least one failure path and read the rollback confirmation. * [ ] I can explain the difference between "ready for demo" and "ready for release" in one sentence. * [ ] My evidence bundle could be read by another person without me explaining what any of it means. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Integration Contract](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#the-integration-contract) * [The Evidence Pipeline](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#the-evidence-pipeline) * [Directing the Monthly Summary](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#directing-the-monthly-summary) * [Directing the Release Gate](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#directing-the-release-gate) * [The Evidence Bundle](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#the-evidence-bundle) * [Capstone Run Sequence](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#capstone-run-sequence) * [Gate Language](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#gate-language) * [One Common Failure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#one-common-failure) * [Capstone Self-Review](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#capstone-self-review) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#try-with-ai) * [Prompt 1: Integrity Gap Audit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#prompt-1-integrity-gap-audit) * [Prompt 2: Evidence Bundle Generator](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#prompt-2-evidence-bundle-generator) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#prompt-3-apply-to-your-domain) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/capstone-budget-tracker#flashcards-study-aid) --- # The Organization Workflow | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your Downloads has 487 files. Where does `Q4-Budget-Draft.pdf` go? Finance? Q4? Projects? Budget? The answer reveals why categorization is harder than it looks. Every file belongs to multiple categories at once. A budget spreadsheet is a financial document AND a Q4 deliverable AND a project artifact. Any rule you pick will be wrong for some files. That's the hard question this lesson tackles. Not "how do I move files into folders", but how do you design rules that handle ambiguity? The answer: you don't design them alone. You design them in conversation with a General Agent, and the best categorization systems aren't designed perfectly upfront. They're discovered through errors and corrections. * * * Prerequisites[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This lesson builds on Lessons 1–2. You should have: * Completed the survey workflow from Lesson 1 (you have `FILE-INVENTORY.md` with your actual file data) * Created a backup from Lesson 2 (your important files are safely copied) If you skipped ahead, go back and complete the checkpoints. The safety-first pattern protects you from mistakes, and you need real file data for this to be meaningful. * * * The Conversation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-conversation "Direct link to The Conversation") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Open Claude Code and start the collaboration: **You:** > Help me organize my Downloads folder. Let's create a categorization system that works for my files. **Claude Code:** > I can see you have 487 files in ~/Downloads. Based on the extensions, here's what I'd suggest: > > | Category | File Types | Count | > | --- | --- | --- | > | Documents | pdf, doc, docx, txt | 127 | > | Images | jpg, png, gif | 89 | > | Code | py, js, html, css | 34 | > | Misc | everything else | 237 | > > Does this categorization work for you, or would you like to adjust it? **You:** > That looks mostly right. But wait: where would .csv files go? Those are spreadsheets, not documents. **Claude Code:** > You're right. I had `.csv` grouped with documents since CSV is technically a text format. But functionally they're spreadsheets. Let me create a separate spreadsheet category. > > Updated categories: > > | Category | File Types | Count | > | --- | --- | --- | > | Documents | pdf, doc, docx, txt | 98 | > | Spreadsheets | xlsx, csv | 29 | > | Images | jpg, png, gif | 89 | > | Code | py, js, html, css | 34 | > | Misc | everything else | 237 | > > Should I document these rules in a rules.md file so we can reuse them later? That correction matters. The agent classified by file format (CSV is text). You classified by function (CSV is a spreadsheet). You were right: because you know how you _use_ the files. The agent knows what they _are_. When you catch these mistakes and the agent adjusts, the rules get smarter. **You:** > Yes, create rules.md. Then test on ONE file first before doing all of them. **Claude Code:** > Created rules.md with the categorization logic and edge case handling. > > Now testing with a single file. Moving budget-2026.pdf to documents/... > > Moved: budget-2026.pdf → organized/documents/Verified: File exists in destinationVerified: File no longer in source > > Test passed. Ready to proceed with the remaining 486 files? **You:** > Looks good. Now do the rest and show me a summary when done. **Claude Code:** > Organization complete. Here's the summary: > > | Category | Files Moved | > | --- | --- | > | documents/ | 97 | > | spreadsheets/ | 29 | > | images/ | 89 | > | code/ | 34 | > | misc/ | 237 | > > **Total**: 486 files organized > > Updated ORGANIZER-LOG.md with the full activity history. > > Would you like me to show you any category in detail, or are we done? Here's what happened to your folder structure: BEFORE:Downloads/├── report.pdf├── photo.jpg├── budget.xlsx├── script.py└── mystery-file.binAFTER:Downloads/└── organized/ ├── documents/ │ └── report.pdf ├── images/ │ └── photo.jpg ├── spreadsheets/ │ └── budget.xlsx ├── code/ │ └── script.py └── misc/ └── mystery-file.bin The transformation is visual. Flat chaos became categorized structure. But don't celebrate yet. Look at that summary table again. **237 files in `misc/`.** That's 49% of your files dumped into an "everything else" bucket. If half your files are "miscellaneous," your categorization system has a problem. This is the honest part most tutorials skip. Extension-based rules are a starting point, not a solution. Let's fix it. * * * The Misc Problem: When Half Your Files Are "Everything Else"[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-misc-problem-when-half-your-files-are-everything-else "Direct link to The Misc Problem: When Half Your Files Are "Everything Else"") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **You:** > 237 files in misc/ is way too many. Help me understand what's actually in there. Can you break down the misc/ folder by what these files actually ARE? **Claude Code:** > Let me analyze the misc/ folder by extension and content. > > .dmg/.exe installers: 43 (app installers, probably safe to delete).zip archives: 38 (compressed folders, need to check contents).mp4/.mov videos: 27 (screen recordings, downloaded videos).eml/.msg emails: 19 (saved email messages).json/.xml data: 15 (configuration and data export files)No extension: 12 (README files, scripts, unknown)Other (94 types): 83 (scattered: .sketch, .fig, .key, .psd...) > > The installers alone are 43 files. Want me to create subcategories for the obvious groups? **You:** > Yes. Create installers/, media/, and archives/ categories. Move the email files into documents/. For the rest, can you look INSIDE the files to figure out what they are? **Claude Code:** > I'll check the no-extension files and try to identify them by content. > > \[Runs: file organized/misc/README\] \[Runs: file organized/misc/Makefile\] \[Runs: file organized/misc/data-export\] > > Results: > > * 8 of the 12 no-extension files are actually text/code files > * 3 are binary data files > * 1 is a renamed PDF (someone saved it without .pdf extension) > > Moved 8 text files → code/ Moved 1 mislabeled PDF → documents/ > > Updated misc/ breakdown: > > Before: 237 files (49% of total)After: 86 files (18% of total) > > The remaining 86 files are genuinely miscellaneous: design files, one-off formats, and binary data. That's a reasonable misc/ bucket. **The lesson:** Extension-based rules are layer one. Content-based analysis is layer two. When your "misc" bucket is too large, don't accept it: dig deeper. A good categorization system should have less than 20% of files uncategorized. * * * When the Agent Gets It Wrong[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#when-the-agent-gets-it-wrong "Direct link to When the Agent Gets It Wrong") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's something the textbook version of this conversation doesn't show you. Sometimes the agent gets it wrong, and not in the polite, easy-to-fix way. ### Scenario: Silent Miscategorization[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#scenario-silent-miscategorization "Direct link to Scenario: Silent Miscategorization") After the batch operation, you browse `organized/documents/` and count the files. 97. But your FILE-INVENTORY.md from Lesson 1 said you had 127 documents. Where are the other 30? You: Wait — I should have 127 documents but there are only 97 indocuments/. Where are the missing 30? Claude Code: Let me check...[Runs: find organized/ -iname "*.doc" -o -iname "*.txt" | wc -l][Runs: find organized/misc/ -iname "*.doc" -o -iname "*.txt"]Found the issue. 30 files have uppercase extensions (.PDF, .DOCX, .TXT)and my rules only matched lowercase. These fell through to misc/.- 22 files with .PDF → should be in documents/- 5 files with .DOCX → should be in documents/- 3 files with .TXT → should be in documents/Moving them now and updating rules.md to use case-insensitive matching. This is a real bug. Case-sensitive matching is one of the most common failures in file categorization, and the agent didn't catch it during the initial rule design. You caught it because you _verified the numbers_. This is Principle 3 in action: verification caught an error that looked like success. ### Scenario: The Agent Suggests a Category You Didn't Think Of[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#scenario-the-agent-suggests-a-category-you-didnt-think-of "Direct link to Scenario: The Agent Suggests a Category You Didn't Think Of") Claude Code: I notice 43 of your "misc" files are .dmg and .exe installers.These are application installers that you probably don't need anymoreonce the apps are installed. Would you like a separate "installers"category? That would let you easily delete them to free up space. The agent spotted a pattern you missed. This is the collaboration at work: neither you nor the agent alone would reach the best result. **Why does `rules.md` matter more than the organized folders?** Next month your Downloads will fill up again. The folders will be outdated. But `rules.md` (the logic for _how_ to organize) is reusable forever. The rules are the product. The organized folders are just the output. * * * The Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-pattern "Direct link to The Pattern") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Notice the structure of that conversation: 1. "Help me [goal]. Let's [approach]." → AI proposes initial solution2. "That's good, but [refinement]." → You adjust based on your needs3. "Document the rules so we can reuse them." → Logic becomes persistent state4. "Test on ONE file first." → Small verification before scale5. "Show me what you'll do before doing it." → Dry run — observability before action6. "Now do the rest and show me a summary." → Batch execution with observability This pattern works for any organization task. The specific categories change. Maybe you're organizing photos by date, or code files by project. But the workflow stays the same. ### The Dry-Run Habit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-dry-run-habit "Direct link to The Dry-Run Habit") Step 5 deserves special attention. Before the agent moves hundreds of files, ask it to show you what it _would_ do: Before you move everything, run this in dry-run mode — list exactlywhat files you'll move and where, without actually doing it. The agent will produce a plan like: meeting-notes.docx → organized/documents/quarterly-data.xlsx → organized/spreadsheets/vacation-photo.jpg → organized/images/cleanup-script.py → organized/code/mystery-file.bin → organized/misc/... (481 more files)No files moved yet. Want me to proceed? A dry run makes the agent's intentions visible before they become irreversible. You can scan the list and catch problems: a file headed to the wrong folder, a naming conflict, something that should be skipped: while it's still just a plan on screen. The single-file test (step 4) checks that the _mechanism_ works. The dry run (step 5) checks that the _decisions_ are right. Both matter. * * * Your Workspace After This Lesson[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#your-workspace-after-this-lesson "Direct link to Your Workspace After This Lesson") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Check what Claude Code created: ls -la file-organizer/ **Output:** drwxr-xr-x backup/-rw-r--r-- FILE-INVENTORY.md-rw-r--r-- ORGANIZER-LOG.mddrwxr-xr-x organized/-rw-r--r-- rules.md Examine the organized directory: ls organized/ **Output:** code documents images misc spreadsheets Read your rules: cat rules.md **Output:** # File Organization Rules## Layer 1: Extension-Based (case-insensitive)| Extension | Destination || ----------------------------- | ------------- || .pdf, .doc, .docx, .txt, .rtf | documents/ || .xlsx, .csv, .ods | spreadsheets/ || .jpg, .png, .gif, .svg, .webp | images/ || .mp4, .mov, .avi, .mkv | media/ || .py, .js, .sh, .html, .css | code/ || .dmg, .exe, .msi, .pkg | installers/ || .zip, .tar, .gz, .rar | archives/ |## Layer 2: Content-Based (for no-extension and ambiguous files)- Use `file` command to detect actual type- Text/code files without extension → code/- Mislabeled files (PDF saved without .pdf) → correct category- Email exports (.eml, .msg) → documents/## Edge Cases- ALL extension matching is case-insensitive (.PDF = .pdf)- Hidden files (starting with .) → skip- Duplicate filename → append timestamp- Filenames with spaces, parentheses, copy indicators → normalize before matching- Misc/ target: less than 20% of total files## History- Version: 2.0- Created: 2026-02-12- Last applied: 2026-02-12- Based on: FILE-INVENTORY.md analysis- Changes: v1.0 initial rules; v1.1 case-insensitive matching; v2.0 added Layer 2 content analysis, installers/, media/, archives/ Your organization logic is now permanent. The next time your Downloads fills up, you have documented rules ready to apply. * * * ✅ Checkpoint: Do This Now[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#-checkpoint-do-this-now "Direct link to ✅ Checkpoint: Do This Now") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Stop reading. Open Claude Code and organize your folder using the collaborative pattern. Use this prompt: Help me organize my [Downloads/Documents/Desktop] folder. Analyze what'sthere and suggest categories based on MY actual files. Let me refine therules before we proceed. Document final rules in file-organizer/rules.md.Test on ONE file first, then do the rest. You should now have: * `file-organizer/rules.md` with your categorization rules * `file-organizer/organized/` with categorized subfolders * Updated `file-organizer/ORGANIZER-LOG.md` Review the organized folders. Did every file land in the right place? If not, tell the agent what went wrong and let it fix the rules. Your folders are organized. Your rules are documented. But here's what you might be thinking: "I have to do this AGAIN next month when more files pile up." You're right. Typing the same prompts repeatedly is exactly the kind of tedious work agents should eliminate. What if this entire workflow became a single reusable script? * * * 🔄 Session Management Note[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#-session-management-note "Direct link to 🔄 Session Management Note") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You've now completed three lessons worth of work. If your Claude Code context is getting long, this is a natural point to start a fresh session. **Why:** Remember the "Kitchen Sink Session" failure pattern from the Seven Principles chapter? Three lessons of exploration, backup, and organization creates a lot of context. Your rules and inventory are saved in files, so a fresh session can pick them up. **How to reset cleanly:** Commit our work so far with a message like "Complete file survey,backup, and organization — Lessons 1-3" Then start a new session for Lesson 4. Your `rules.md`, `FILE-INVENTORY.md`, and `ORGANIZER-LOG.md` carry your context forward, in files, not memory. If your session still feels responsive, keep going. This is guidance, not a rule. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Rule Conflict Resolution:** I have files that could belong to multiple categories. "project-budget-Q4.xlsx"is both a spreadsheet AND a project document. Help me design rules that handlethese conflicts. What should take priority — file type or project membership? **What you're learning:** Rule conflict resolution. Real categorization systems have ambiguity. You're learning to design rules that handle files that don't fit neatly into one category. **Edge Case Discovery:** What edge cases might break simple extension-based categorization?Show me examples of files that could cause problems. **What you're learning:** Defensive design. AI reveals tricky scenarios like `file.backup.pdf` (multiple dots), `README` (no extension), or `data.CSV` (case sensitivity). **Rule Refinement:** My rules.md categorizes by extension, but I have files from specificprojects that should stay together regardless of type. How can I addproject-based rules that take priority over extension rules? **What you're learning:** Rule precedence. AI explains how to layer rules (check project name first, then fall back to extension), preparing you for more sophisticated automation. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#flashcards-study-aid "Direct link to Flashcards Study Aid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Prerequisites](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#prerequisites) * [The Conversation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-conversation) * [The Misc Problem: When Half Your Files Are "Everything Else"](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-misc-problem-when-half-your-files-are-everything-else) * [When the Agent Gets It Wrong](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#when-the-agent-gets-it-wrong) * [Scenario: Silent Miscategorization](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#scenario-silent-miscategorization) * [Scenario: The Agent Suggests a Category You Didn't Think Of](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#scenario-the-agent-suggests-a-category-you-didnt-think-of) * [The Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-pattern) * [The Dry-Run Habit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#the-dry-run-habit) * [Your Workspace After This Lesson](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#your-workspace-after-this-lesson) * [✅ Checkpoint: Do This Now](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#-checkpoint-do-this-now) * [🔄 Session Management Note](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#-session-management-note) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#try-with-ai) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/the-organization-workflow#flashcards-study-aid) --- # Linux Operations Exercises | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Ali's competitor-tracker agent is running in production. It survived a reboot, recovered from a crash, and generates reports every morning. That took seven lessons of learning -- SSH connections, filesystem navigation, directory setup, systemd services, security hardening, and systematic debugging. You understand how each piece works. But there is a gap between understanding the pieces and reaching for the right instruction when a real deployment goes sideways at 2am. These exercises close that gap. Fourteen hands-on challenges across three tiers practice the skills that make Linux operations second nature: **server navigation** (finding your way around and reading what the system tells you), **infrastructure setup** (building production-ready deployments from scratch), and **systematic diagnosis** (finding root causes instead of blindly restarting). Every exercise puts you in Ali's shoes -- you direct Claude Code, read the output, and make decisions. Download Exercise Files **[Download Linux Operations Exercises (ZIP)](https://github.com/panaversity/claude-code-linux-mastery-exercises/releases/latest/download/linux-mastery-exercises.zip) ** After downloading, unzip the file. Each exercise has its own folder with an `INSTRUCTIONS.md` and starter files (simulated server output, broken configs, log files) you need. If the download link doesn't work, visit the [repository releases page](https://github.com/panaversity/claude-code-linux-mastery-exercises/releases) directly. * * * How to Use These Exercises[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#how-to-use-these-exercises "Direct link to How to Use These Exercises") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The workflow for every exercise is the same: 1. **Read the scenario** -- understand what Ali is facing and what needs to happen 2. **Direct Claude Code** -- give clear instructions based on what you learned in the lessons 3. **Read the output** -- interpret what comes back before taking the next step 4. **For Debug exercises:** read the broken state carefully before attempting any fix 5. **Reflect** using the questions provided -- this is where the real learning happens You do not need to complete all 14 in one sitting. Work through one tier at a time. Each tier builds on the lessons indicated. * * * Key Differences from Chapter Lessons[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#key-differences-from-chapter-lessons "Direct link to Key Differences from Chapter Lessons") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lessons 1-7, you worked through each concept with guided walkthroughs and Ali's story leading the way. These exercises are different in three ways: * **No step-by-step walkthrough.** The exercises describe the scenario, the broken state or goal, and what success looks like. You decide what to tell Claude Code. * **Build + Debug pairing.** Every pair has a Build exercise (create something from scratch) and a Debug exercise (diagnose and fix something broken). Debugging broken deployments develops different skills than building clean ones -- you learn to read system output critically and trace errors to their root cause. * **Increasing independence.** Foundation exercises include starter prompts. Operations exercises provide less scaffolding. Diagnosis exercises give you only the symptoms -- you design the entire investigation. By the Diagnosis tier, you should be able to face a broken deployment and instinctively direct Claude Code through the LNPS method without reviewing the chapter lessons. * * * Tool Guide[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tool-guide "Direct link to Tool Guide") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Claude Code** -- Required for all exercises. You direct it to run commands on the server and interpret the output. * **Cowork** -- Use for planning your approach before executing. Helpful for thinking through what to tell Claude Code before you say it. * Most exercises require the terminal. Use Cowork for strategy, Claude Code for execution. * * * Tier 1: Foundation (Lessons 1-2)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tier-1-foundation-lessons-1-2 "Direct link to Tier 1: Foundation (Lessons 1-2)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skills:** Navigating a server and reading what the system tells you > > Lessons 1-2 taught you how to SSH into a server, explore the filesystem tree, interpret command output, and read file permissions. These exercises put those skills into realistic scenarios where Ali needs to orient himself on an unfamiliar server and make sense of what he finds. ### Exercise 1.1 -- Server Orientation (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-11----server-orientation-build "Direct link to Exercise 1.1 -- Server Orientation (Build)") **The Scenario:** Ali just got SSH access to a new cloud server from a different hosting provider. He has never logged into this machine before. Before he deploys anything, he needs to understand what he is working with -- how much disk space is available, what is already installed, where things live, and whether anyone else has been using this server. **Your Task:** Direct Claude Code to explore the server systematically. Find out: (1) what Linux distribution and version is running, (2) how much disk space and memory are available, (3) what is in the `/home` directory (are there other users?), (4) what is in `/opt` and `/var` (has anyone deployed anything here before?), and (5) what the current user's permissions are. By the end, you should be able to describe this server in two sentences. **What to Tell Claude Code:** > "I just SSH'd into a new server and need to understand what I'm working with. Check the OS version, available disk space and memory, list what's in /home, /opt, and /var, and show me who I'm logged in as and what groups I belong to." **Expected Outcome:** You can answer: What OS is this? How much space do I have? Am I alone on this server? Has anything been deployed here before? What can I do without sudo? **Reflection Question:** Why does Ali check what is already on the server before deploying anything? What could go wrong if he skipped this step? * * * ### Exercise 1.2 -- The Mystery Server (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-12----the-mystery-server-debug "Direct link to Exercise 1.2 -- The Mystery Server (Debug)") **The Scenario:** Ali's colleague Priya asks for help. She deployed an agent to a server last month but cannot remember the details. She knows it is "somewhere in /opt" and it "used to work." She gives Ali SSH access and asks him to figure out what is there, where it lives, and whether it is still running. **The Broken State:** The server has a project directory somewhere under `/opt` with Python files, a `.env` file, and some log files. The agent is not currently running. There may be clues in the directory structure, file timestamps, and log contents about what happened. **Your Task:** Direct Claude Code to investigate. Find the project directory under `/opt`. Read the directory structure to understand what the agent does. Check the file timestamps to see when things were last modified. Look at the log files for any error messages or the last successful run. Determine whether the agent was run manually or as a service. Produce a summary for Priya: "Your agent is at \[path\], it does \[purpose\], it last ran on \[date\], and it stopped because \[reason\]." **What to Tell Claude Code:** > "There's supposed to be an agent deployed somewhere under /opt. Find it, show me the directory structure, check when files were last modified, read any log files for errors, and check if there's a systemd service for it." **Expected Outcome:** A clear summary of what the agent is, where it lives, when it last ran, and why it stopped. You should be able to explain each piece of evidence you found. **Reflection Question:** What clues did the file timestamps and log contents give you? If the logs were empty, what would you check next? * * * ### Exercise 1.3 -- Reading the Room (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-13----reading-the-room-build "Direct link to Exercise 1.3 -- Reading the Room (Build)") **The Scenario:** Ali wants to check the health of Dev's server before deploying his competitor-tracker. He needs to read system output and understand what it means -- not just see numbers, but interpret them. **Your Task:** Direct Claude Code to run diagnostic commands and then explain the output. Specifically: (1) run `df -h` and identify which filesystem is most full, (2) run `free -h` and determine if there is enough memory for a Python agent, (3) run `ps aux --sort=-%mem | head -10` and identify the top memory consumers, (4) run `uptime` and explain what the load averages mean. For each command, write one sentence explaining what the output tells you about this server's readiness for a new deployment. **What to Tell Claude Code:** > "Check the server health before I deploy. Show me disk usage, memory usage, the top 10 processes by memory, and the system uptime with load averages. For each one, explain what the numbers mean for my deployment." **Expected Outcome:** A health assessment: "This server has X GB free disk, Y GB free memory, the heaviest process is Z using N%, and the load average is W which means \[acceptable/concerning\]." You should understand every number Claude Code reports. **Reflection Question:** If the disk was 92% full, what would you tell Claude Code to do before deploying? How would you decide what is safe to delete? * * * ### Exercise 1.4 -- Permission Puzzle (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-14----permission-puzzle-debug "Direct link to Exercise 1.4 -- Permission Puzzle (Debug)") **The Scenario:** Ali tries to read a log file and gets "Permission denied." He tries to write to a configuration directory and gets "Permission denied" again. He can see the files exist (they show up in `ls`) but he cannot access them. Something about the file permissions is blocking him. **The Broken State:** Several files and directories have incorrect permissions: * A log file owned by root that Ali's user cannot read * A config directory with permissions set to `700` owned by another user * An executable script that is missing the execute permission * A `.env` file that is world-readable (the opposite problem -- too open, not too closed) **Your Task:** Direct Claude Code to diagnose each permission problem by reading `ls -la` output. For each file, explain: who owns it, what permissions are set, why Ali cannot access it, and what the correct permissions should be. Do not just fix the permissions blindly -- explain the problem first, then direct the fix. **What to Tell Claude Code:** > "I'm getting 'Permission denied' on several files. Run ls -la on each of these paths and explain the permission strings. Tell me who owns each file, what the current permissions mean, and what they should be changed to." **Expected Outcome:** For each file, you can decode the permission string (e.g., `-rw-------` means only the owner can read and write), explain the problem, and state the correct fix. You should catch the `.env` file that is too open as well as the files that are too restrictive. **Reflection Question:** Why is a world-readable `.env` file a bigger security risk than a log file Ali cannot read? Which problem would you fix first in production? * * * Tier 2: Operations (Lessons 3-5)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tier-2-operations-lessons-3-5 "Direct link to Tier 2: Operations (Lessons 3-5)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skills:** Building production infrastructure from scratch > > Lessons 3-5 taught you how to create directory structures, manage secrets in `.env` files, set up persistent logging, create systemd services, and harden security with dedicated users and SSH keys. These exercises challenge you to build and fix real deployment infrastructure. ### Exercise 2.1 -- Agent Home Setup (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-21----agent-home-setup-build "Direct link to Exercise 2.1 -- Agent Home Setup (Build)") **The Scenario:** Ali is deploying a new agent -- a social-media-monitor that tracks brand mentions. He needs to set up the complete directory structure on the server before any code runs. This is Lesson 3 applied from scratch: the right directories, a `.env` file for API keys, and a logging setup that persists. **Your Task:** Direct Claude Code to create the full project structure at `/opt/agents/social-monitor/`. It needs: (1) a `src/` directory for the Python code, (2) a `config/` directory with a `.env` file containing placeholder API keys, (3) a `logs/` directory with proper permissions, (4) a `data/` directory for output files, (5) a `README.md` explaining what the agent does and how to run it. The `.env` file must not be world-readable. The `logs/` directory must be writable by the agent user. **Expected Outcome:** Running `ls -la /opt/agents/social-monitor/` shows all five subdirectories and the README. The `.env` file has `600` permissions. The `logs/` directory is writable. A developer joining the project tomorrow could look at the directory structure and understand where everything goes. **Reflection Question:** Why does Ali create the directory structure before writing any code? What would happen if he just dumped all files in one flat directory? * * * ### Exercise 2.2 -- The Missing Pieces (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-22----the-missing-pieces-debug "Direct link to Exercise 2.2 -- The Missing Pieces (Debug)") **The Scenario:** Ali comes back to a server where an intern set up a project directory, but several things are wrong. The agent keeps failing on startup and nobody can figure out why. **The Broken State:** The project at `/opt/agents/data-collector/` has these problems: * The `.env` file exists but has permissions `644` (world-readable -- anyone on the server can read the API keys) * The `logs/` directory does not exist at all (the agent crashes trying to write logs) * The source code is in the root of the project instead of a `src/` subdirectory * There is no `README.md` -- nobody knows what this agent does or how to start it * A `node_modules/` directory with 15,000 files is taking up 200MB of disk space unnecessarily **Your Task:** Direct Claude Code to audit the directory, identify every problem, fix each one in order of severity (security first, then functionality, then organization), and verify each fix. The security issue (world-readable `.env`) should be fixed before anything else. **Expected Outcome:** The `.env` permissions are `600`. The `logs/` directory exists. Source code is organized in `src/`. A `README.md` exists. The `node_modules/` is either removed or added to `.gitignore`. You can explain why you fixed things in the order you did. **Reflection Question:** The intern's setup "almost worked" -- the agent could read the `.env` file and the code was present. Why is "almost works" dangerous in production? What is the difference between "runs" and "runs correctly"? * * * ### Exercise 2.3 -- Service From Scratch (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-23----service-from-scratch-build "Direct link to Exercise 2.3 -- Service From Scratch (Build)") **The Scenario:** Ali's social-media-monitor agent is ready to run. Right now he starts it manually with `python3 src/main.py`. Every time his SSH session ends, the agent dies. He needs to turn this process into a systemd service that survives reboots and restarts after crashes -- the Lesson 4 transformation. **Your Task:** Direct Claude Code to create a complete systemd service for the social-media-monitor. The service should: (1) run as a dedicated `social-monitor` user (not root), (2) load environment variables from the `.env` file, (3) use `Restart=on-failure` with a 10-second delay between restarts, (4) set a memory limit of 256MB, (5) start automatically on boot. After creating the unit file, enable and start the service, then verify it is running. **Expected Outcome:** `systemctl status social-monitor` shows the service as active and running. `systemctl is-enabled social-monitor` shows "enabled." The unit file is at `/etc/systemd/system/social-monitor.service` with all five requirements present. Closing your terminal does not kill the agent. **Reflection Question:** Why did Ali choose `Restart=on-failure` instead of `Restart=always`? What is the difference, and when would `always` be the wrong choice? * * * ### Exercise 2.4 -- The Service That Won't Start (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-24----the-service-that-wont-start-debug "Direct link to Exercise 2.4 -- The Service That Won't Start (Debug)") **The Scenario:** Ali's competitor-tracker service was working yesterday. This morning, `systemctl status competitor-tracker` shows it as "failed." Ali's instinct is to restart it. But he remembers Lesson 6: restarting is not debugging. **The Broken State:** The service fails to start. The symptoms are: * `systemctl status competitor-tracker` shows "failed" with exit code 1 * The unit file references a Python path that no longer exists (someone moved the virtualenv) * The `.env` file is missing a required `DATABASE_URL` variable (someone deleted it during "cleanup") * The `logs/` directory permissions were changed to read-only (someone ran `chmod` incorrectly) **Your Task:** Direct Claude Code through a systematic investigation. Do NOT restart the service first. Instead: (1) check `systemctl status` for the error message, (2) check `journalctl -u competitor-tracker` for detailed logs, (3) verify the paths in the unit file actually exist, (4) verify the `.env` file has all required variables, (5) check the `logs/` directory permissions. Fix each root cause, then restart and verify. **Expected Outcome:** You identified all three root causes before restarting. The Python path is corrected in the unit file. The `DATABASE_URL` is restored in `.env`. The `logs/` directory is writable again. The service starts successfully after all fixes are applied. **Reflection Question:** If Ali had just restarted the service, what would have happened? Would restarting have fixed any of these three problems? Why is the diagnostic step non-negotiable? * * * ### Exercise 2.5 -- Lock It Down (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-25----lock-it-down-build "Direct link to Exercise 2.5 -- Lock It Down (Build)") **The Scenario:** Ali's social-media-monitor is running as a systemd service. But it is running as `root` because the intern never created a dedicated user. Dev tells Ali: "A root agent with internet access is a security nightmare. Lock it down." This is Lesson 5 applied from scratch. **Your Task:** Direct Claude Code to harden the deployment: (1) create a dedicated `social-monitor` system user with no login shell and no home directory, (2) change ownership of `/opt/agents/social-monitor/` to the new user, (3) set file permissions so only the service user can read the `.env` file, (4) update the systemd unit file to run as the new user, (5) set up SSH key authentication for Ali's personal login (disable password auth). After each step, verify the change took effect. **Expected Outcome:** `id social-monitor` shows the user exists. `ls -la /opt/agents/social-monitor/.env` shows ownership by `social-monitor` and permissions `600`. The systemd unit file has `User=social-monitor`. SSH key auth works and password auth is disabled. The service still runs correctly under the new user. **Reflection Question:** Why does Ali create a system user with no login shell instead of a regular user? What attack vector does this close? * * * ### Exercise 2.6 -- The Overprivileged Agent (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-26----the-overprivileged-agent-debug "Direct link to Exercise 2.6 -- The Overprivileged Agent (Debug)") **The Scenario:** Ali audits a server that another team has been using. He finds an agent running as root with security holes everywhere. Dev asks him to identify and fix every security problem. **The Broken State:** The agent at `/opt/agents/report-generator/` has these security problems: * The systemd service runs as `User=root` * The `.env` file has permissions `666` (everyone can read AND write the API keys) * Password authentication is enabled on SSH (the server accepts password logins from the internet) * The agent's log file at `/var/log/report-agent.log` contains printed API keys (someone added debug logging that dumps environment variables) * The agent's port (8080) is exposed to the internet with no firewall rule **Your Task:** Direct Claude Code to audit the deployment by checking each of the five areas. For each problem: state what is wrong, explain the specific risk (what could an attacker do?), and direct the fix. Fix them in order of severity -- the exposed API keys in logs are the most urgent because they are actively leaking right now. **Expected Outcome:** A security audit report listing all five problems, their risk levels, and the fixes applied. The logs with leaked keys are rotated or truncated. The `.env` is locked down. The service runs as a dedicated user. Password auth is disabled. The port is firewalled or bound to localhost. **Reflection Question:** Which of the five problems could be exploited by someone who does not have SSH access to the server? Which require an attacker to already be on the machine? How does this change your fix priority? * * * Tier 3: Diagnosis (Lessons 6-7)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tier-3-diagnosis-lessons-6-7 "Direct link to Tier 3: Diagnosis (Lessons 6-7)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skills:** Systematic debugging and deployment specification > > Lessons 6-7 taught you the LNPS triage method (Logs, Network, Process, System) and how to write deployment specs that capture everything needed to reproduce a deployment. These exercises put you in diagnosis scenarios where the symptoms are ambiguous and the root cause is not obvious. ### Exercise 3.1 -- The Silent Agent (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-31----the-silent-agent-debug "Direct link to Exercise 3.1 -- The Silent Agent (Debug)") **The Scenario:** Ali checks his competitor-tracker Monday morning. The systemd service shows "active (running)" -- green light, everything looks fine. But the daily report email never arrived. The dashboard shows no new data since Friday. The agent is running but not doing anything. This is the most dangerous kind of failure: silent. **The Broken State:** The LNPS investigation will reveal: * **Logs**: The last log entry says "Waiting for database connection..." repeated every 30 seconds since Friday at 11pm * **Network**: The database server (port 5432) is unreachable -- a firewall rule was changed Friday evening during maintenance * **Process**: The agent process is alive and consuming CPU (it is stuck in a connection retry loop) * **System**: Disk space and memory are fine **Your Task:** Direct Claude Code through the full LNPS method. Do NOT restart the agent first. Start with logs (`journalctl -u competitor-tracker --since "Friday"`), then check network connectivity to the database, then check the process state, then check system resources. Follow the LNPS order even if you think you know the answer after step 1. **Expected Outcome:** You identified the root cause (database unreachable due to firewall change) by following LNPS in order. You can explain why the agent appeared healthy (systemd reported "active") despite being non-functional. You know that restarting the agent would not fix this -- the network issue must be resolved first. **Reflection Question:** Why is a "silent failure" more dangerous than a crash? If the agent had crashed instead of retrying silently, would Ali have noticed sooner? * * * ### Exercise 3.2 -- The Cascading Failure (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-32----the-cascading-failure-debug "Direct link to Exercise 3.2 -- The Cascading Failure (Debug)") **The Scenario:** Ali wakes up to three alerts at once. The competitor-tracker is down. The social-media-monitor is down. The report-generator is down. Three agents, all failed within the same hour. Ali's instinct says "the server is broken" but each agent shows a different error message. He needs to find the single root cause. **The Broken State:** The LNPS investigation across all three agents will reveal: * **competitor-tracker logs**: "OSError: \[Errno 28\] No space left on device" -- cannot write to logs * **social-monitor logs**: "PermissionError: \[Errno 13\] Permission denied: '/opt/agents/social-monitor/data/output.json'" -- cannot write output * **report-generator logs**: "ConnectionError: database disk image is malformed" -- SQLite database corrupted * **System**: `df -h` shows `/` is 100% full. The root filesystem ran out of disk space. **Your Task:** Direct Claude Code to investigate all three agents, but start with the system-level check. Run `df -h` first. Then check each agent's logs. Trace all three failures back to the single root cause: the disk filled up. Identify what filled the disk (direct Claude Code to find the largest files with `du -sh /* | sort -rh | head -10`). Fix the root cause, then restart the agents. **Expected Outcome:** You identified the root cause (full disk) before investigating individual agents, or you identified it after seeing the first agent's error and correctly predicted the others. You found what filled the disk. You freed space and restarted all three agents successfully. **Reflection Question:** Ali saw three different error messages from three different agents. How did following LNPS (starting with System) lead him to the single root cause faster than investigating each agent separately? * * * ### Exercise 3.3 -- Write the Deployment Spec (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-33----write-the-deployment-spec-build "Direct link to Exercise 3.3 -- Write the Deployment Spec (Build)") **The Scenario:** Ali has deployed three agents manually over the past week. Each time, he had to remember the steps, look back at old terminal output, and re-discover things he already figured out. Dev says: "Write it down once. Next time, hand the spec to Claude Code and it does the whole deployment in thirty minutes." This is the Lesson 7 capstone skill applied to a new agent. **Your Task:** Write a `DEPLOYMENT-SPEC.md` for deploying a new agent called `inventory-checker` that monitors product stock levels. The spec must cover all six sections from Lesson 7: (1) **Server Requirements** -- OS, memory, disk, ports needed, (2) **Project Structure** -- exact directory layout at `/opt/agents/inventory-checker/`, (3) **Dependencies** -- Python version, pip packages, system packages, (4) **Secret Management** -- what goes in `.env`, permissions, (5) **systemd Service** -- complete unit file with user, restart policy, and resource limits, (6) **Verification Checklist** -- how to confirm the deployment is production-ready. **Expected Outcome:** A complete `DEPLOYMENT-SPEC.md` that you could hand to someone who has never seen this agent and they could deploy it without asking a single question. Every path is absolute. Every permission is specified. Every verification step has an expected result. **Reflection Question:** If Ali hands this spec to Claude Code with the instruction "Deploy this agent following the spec exactly," what could still go wrong? What assumptions does the spec make that might not hold on a different server? * * * ### Exercise 3.4 -- Spec vs Reality (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-34----spec-vs-reality-debug "Direct link to Exercise 3.4 -- Spec vs Reality (Debug)") **The Scenario:** Ali wrote a deployment spec and handed it to Claude Code. The deployment completed without errors. But when he runs the verification checklist from the spec, three checks fail. The spec said one thing; the server shows another. Ali needs to find the gaps between what the spec promised and what actually exists. **The Broken State:** The deployment at `/opt/agents/price-watcher/` was executed from a spec, but: * The spec says the service runs as `price-watcher` user, but `systemctl show -p User price-watcher` shows `User=root` (the unit file was written correctly but `systemctl daemon-reload` was never run after editing) * The spec says `.env` has permissions `600`, but `ls -la` shows `644` (the `chmod` command was applied to the wrong file) * The spec says the service starts on boot, but `systemctl is-enabled price-watcher` shows "disabled" (the `systemctl enable` step was skipped) **Your Task:** Direct Claude Code to run the verification checklist from the spec. For each failed check, identify the gap between the spec and reality, determine why the gap exists (what went wrong during deployment), and fix it. After all fixes, re-run the full checklist to confirm everything passes. **Expected Outcome:** All three gaps are identified, explained, and fixed. The verification checklist passes completely on the second run. You can explain why each gap occurred -- not just what the fix was, but what mistake during deployment caused it. **Reflection Question:** The deployment "completed without errors" but three checks failed. What does this tell you about the difference between "no errors" and "correct"? Why is the verification checklist the most important part of the spec? * * * What's Next[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#whats-next "Direct link to What's Next") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- You have practiced the three core skills -- **server navigation**, **infrastructure setup**, and **systematic diagnosis** -- across 14 exercises. These skills compound: every exercise makes Linux operations feel more instinctive, so when Ali's agent goes silent at 2am, you reach for the LNPS method instead of blindly restarting. Next in the **Chapter Quiz**, you will test your understanding of Linux concepts and deployment scenarios. The operations patterns you built here become the foundation for the Project lesson, where your agents deploy themselves. * [How to Use These Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#how-to-use-these-exercises) * [Key Differences from Chapter Lessons](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#key-differences-from-chapter-lessons) * [Tool Guide](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tool-guide) * [Tier 1: Foundation (Lessons 1-2)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tier-1-foundation-lessons-1-2) * [Exercise 1.1 -- Server Orientation (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-11----server-orientation-build) * [Exercise 1.2 -- The Mystery Server (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-12----the-mystery-server-debug) * [Exercise 1.3 -- Reading the Room (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-13----reading-the-room-build) * [Exercise 1.4 -- Permission Puzzle (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-14----permission-puzzle-debug) * [Tier 2: Operations (Lessons 3-5)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tier-2-operations-lessons-3-5) * [Exercise 2.1 -- Agent Home Setup (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-21----agent-home-setup-build) * [Exercise 2.2 -- The Missing Pieces (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-22----the-missing-pieces-debug) * [Exercise 2.3 -- Service From Scratch (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-23----service-from-scratch-build) * [Exercise 2.4 -- The Service That Won't Start (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-24----the-service-that-wont-start-debug) * [Exercise 2.5 -- Lock It Down (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-25----lock-it-down-build) * [Exercise 2.6 -- The Overprivileged Agent (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-26----the-overprivileged-agent-debug) * [Tier 3: Diagnosis (Lessons 6-7)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#tier-3-diagnosis-lessons-6-7) * [Exercise 3.1 -- The Silent Agent (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-31----the-silent-agent-debug) * [Exercise 3.2 -- The Cascading Failure (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-32----the-cascading-failure-debug) * [Exercise 3.3 -- Write the Deployment Spec (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-33----write-the-deployment-spec-build) * [Exercise 3.4 -- Spec vs Reality (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#exercise-34----spec-vs-reality-debug) * [What's Next](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/linux-operations-exercises#whats-next) --- # Chapter 68: Agent Skills & MCP Code Execution | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C You learned to create skills in Chapter 3—SKILL.md files that encode domain expertise for Claude Code. You built MCP servers in Chapter 67. Now you'll build skills that don't just advise—they EXECUTE. The code execution pattern transforms agents from assistants that suggest solutions into autonomous systems that solve problems. A skill orchestrates code execution (MCP calls, scripts, data processing), analyzes results, and iterates until the problem is solved. **Why MCP first?** You already know MCP from Chapters 66-67. MCP-wrapping skills like `fetching-library-docs` (which wraps Context7) show the pattern clearly: the skill adds an intelligence layer on top of raw MCP capabilities—deciding when to call, how to filter results, how to handle errors. Once you see this pattern with MCP, extending it to script execution becomes natural. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#what-youll-learn "Direct link to What You'll Learn") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * **Design execution skill patterns**: Create personas, questions, and principles that encode autonomous behavior * **Build MCP-wrapping skills**: Add intelligence layers on top of MCP servers (like `fetching-library-docs` wrapping Context7) * **Implement script execution skills**: Write, execute, and iterate on Python/Bash scripts autonomously * **Handle errors gracefully**: Recover from syntax errors, runtime failures, and timeouts * **Orchestrate full workflows**: Combine MCP + scripts into multi-step execution pipelines * **Ship Digital FTEs**: Package complete execution skills as deployable, sellable assets The Learning Arc[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#the-learning-arc "Direct link to The Learning Arc") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Phase | What You Build | Example | | --- | --- | --- | | **Foundation** (Lessons 1-2) | Advanced skill patterns | Persona + Questions + Principles for execution | | **MCP First** (Lessons 3-4) | Skills that wrap MCP servers | Like `fetching-library-docs` wrapping Context7 | | **Scripts Next** (Lessons 5-6) | Skills that write + execute scripts | Data processing, file manipulation | | **Full Workflows** (Lesson 7) | Skills that orchestrate everything | MCP + scripts + iteration + error recovery | | **Ship It** (Lesson 8) | Complete, shippable skill | Capstone Digital FTE | Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#chapter-structure "Direct link to Chapter Structure") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Lesson | Title | Layer | Duration | | --- | --- | --- | --- | | 1 | Advanced Skill Patterns | L1 (Manual) | 25 min | | 2 | Skill Composition & Dependencies | L2 (Collaboration) | 30 min | | 3 | Anatomy of MCP-Wrapping Skills | L1 (Manual) | 30 min | | 4 | Build Your MCP-Wrapping Skill | L2 (Collaboration) | 45 min | | 5 | Script Execution Fundamentals | L1 (Manual) | 30 min | | 6 | Build Script-Execution Skill | L2 (Collaboration) | 50 min | | 7 | Full Workflow Orchestration | L3 (Intelligence) | 50 min | | 8 | Capstone: Shippable Agent Skill | L4 (Spec-Driven) | 90 min | **Total: ~6 hours** Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ * **Chapter 3**: Claude Code Skills (SKILL.md structure, basic skill creation) * **Chapter 66**: MCP Fundamentals (architecture, client configuration) * **Chapter 67**: Custom MCP Servers (building MCP servers you can now wrap) * **Part 4**: Python Fundamentals (for script generation and execution) Success Criteria[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#success-criteria "Direct link to Success Criteria") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By chapter completion, you will: * Analyze existing MCP-wrapping skills and explain their intelligence layer * Build a skill that wraps an MCP server with proper triggering and 30%+ token reduction * Build a skill that writes, executes, and iterates on Python scripts * Implement error recovery handling syntax, runtime, and timeout errors * Complete a capstone: shippable skill implementing the full code execution pattern * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#what-youll-learn) * [The Learning Arc](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#the-learning-arc) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#chapter-structure) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#prerequisites) * [Success Criteria](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution#success-criteria) --- # Chapter 61: Introduction to AI Agents | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C In Chapters 13-14, you learned Specification-Driven Development—how to think in specifications before code. In Part 4, you built Python fluency. Now you'll understand the architecture that makes Claude Code genuinely different from a chatbot. You've been using AI agents without fully knowing it. Claude Code operates autonomously—breaking complex problems into steps, using tools, learning from failures, and adapting to constraints. You've developed intuitions about how agents behave through months of interaction. This chapter reveals the architecture beneath that behavior. Through eight lessons built on Google's "Introduction to Agents" whitepaper (November 2025), you'll learn the taxonomy, architecture, and patterns that major cloud platforms have converged on. The numbers tell the story of a field at inflection: 800+ million people use ChatGPT weekly, 90%+ of developers use AI coding tools, and AI fluency demand has grown 7x faster than any other skill. Most developers can use ChatGPT. Far fewer understand how to design systems where AI takes autonomous action safely. That gap is opportunity. By the end of this chapter, you won't be building agents yet. But you'll understand agent architecture deeply enough to recognize it everywhere and make informed decisions in Chapters 62-65 when you start building them. This chapter is conceptual and strategic—mental models first, implementation second. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents#what-youll-learn "Direct link to What You'll Learn") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll understand: * **The 5-Level Taxonomy**: How to classify any AI system from Level 0 (pure LLM, no tools) through Level 4 (self-evolving system that creates new capabilities), and why Claude Code operates at Level 2-3—not just a chatbot * **The 3+1 Core Architecture**: The four components every agent needs—Model ("Brain" for reasoning), Tools ("Hands" for action), Orchestration ("Nervous System" for planning), and Deployment ("Body" for accessibility)—and how they work together * **The 5-Step Operational Loop**: The universal pattern (Get Mission → Scan Scene → Think → Act → Observe) that appears in every agent system, traced through a complete customer support example * **Multi-agent design patterns**: When to use Coordinator (parallel specialists), Sequential (pipeline), Iterative Refinement (feedback loops), and Human-in-the-Loop (safety critical)—matching patterns to problems * **Agent Ops discipline**: How operating agents differs from traditional software—LM-as-Judge evaluation, OpenTelemetry traces for debugging, golden datasets, and human feedback loops * **Agent interoperability and security**: The A2A protocol, Agent Cards for discovery, and why agent identity creates a new principal class distinct from users and services * **Your strategic positioning**: Why agent development skills are scarce, why the "director vs bricklayer" paradigm shift changes your role, and how learning these architectures in 2025 positions you to build systems in 2026-2027 that seem impossible today * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents#what-youll-learn) --- # Where Your Agent Lives | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Ali types the SSH command Dev texted him. He presses Enter. The screen goes black for a moment, then a single line appears: dev@server:~$ A blinking cursor. Nothing else. No desktop. No icons. No Start menu. No dock. No file explorer. Just a blinking cursor on a black screen and the quiet hum of a machine he has never touched, sitting in a data center he has never visited. Ali opens Claude Code in another terminal window on his laptop. He doesn't know Linux. He doesn't know what commands to run. But he knows how to describe what he needs. And right now, he needs to understand where he is. > **"Your agent doesn't live on your laptop. It lives on a Linux server with no screen, no mouse, no desktop. Just a command line and the truth."** Why a Server?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#why-a-server "Direct link to Why a Server?") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali's competitor-analysis agent worked fine on his laptop. It scraped pricing data, stored results in a database, generated summaries. Then he closed his laptop Friday night and went home for the weekend. The agent stopped. Three days of missing data. Board meeting in twelve hours. A server is a computer that never sleeps. No screen, no keyboard, no desktop. Just raw computing power connected to the internet 24/7. When you close your laptop, the server keeps running. When you go on vacation, the server keeps running. When the power flickers in your apartment, the server (in a data center with backup generators) keeps running. That is why your agent needs a server. Not because servers are faster or fancier. Because servers don't go to sleep when you do. The Glass Window and the Voice[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#the-glass-window-and-the-voice "Direct link to The Glass Window and the Voice") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before Ali can explore, he needs to understand what he is looking at. That black screen with the blinking cursor? That is a **terminal**. Think of it as a glass window into the server. You can see through it, but you cannot reach inside. Behind the glass is the **shell**: a program that listens for instructions, runs them, and shows you what happened. When you type something (or when Claude Code types something for you), the shell is the one doing the work. Terminal is the glass window. Shell is the voice on the other end. Ali is not going to type commands. He is going to tell Claude Code what he wants, and Claude Code will talk to the shell for him. Exploring the Building[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#exploring-the-building "Direct link to Exploring the Building") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Ali's first instinct: figure out where things are. **What Ali tells Claude Code:** "I just logged into a Linux server for the first time. I have no idea what's here. Show me the top-level directory structure and explain what each folder is for." **What Claude Code does:** > ls /bin dev home lib mnt proc run srv tmp varboot etc host media opt root sbin sys usr **What this means:** Everything on a Linux server is organized in a tree. The trunk of the tree is `/`: called "root." Every file, every program, every log, every configuration lives somewhere under this one starting point. Pause. Look at that output again. There are about 20 directories. You do not need to know all of them. You need to know five. Think of the server as a building. `/` is the front door. Each directory is a floor with a specific purpose: | Directory | What lives here | Building analogy | | --- | --- | --- | | `/home` | User files: Ali's code, configs, data | **Apartments**: each user gets their own space | | `/var` | Logs, databases, things that change over time | **The mailroom**: where activity accumulates | | `/etc` | Configuration files for the entire system | **The building manager's office**: rules and settings | | `/opt` | Optional software installed by hand | **Storage units**: extra stuff tenants bring in | | `/tmp` | Temporary files, deleted on reboot | **The lobby whiteboard**: notes that get erased every night | Ali's agent will live in `/home`. Its logs will accumulate in `/var`. Its configuration will sit in `/etc`. That is the mental model. **What Ali tells Claude Code:** "Show me what's inside /home." **What Claude Code does:** > ls -la /hometotal 8drwxr-xr-x 3 root root 4096 Jan 15 08:30 .drwxr-xr-x 20 root root 4096 Jan 15 08:22 ..drwxr-xr-x 5 dev dev 4096 Feb 23 14:12 dev **What this means:** One user on this server: `dev`. That is Dev's home directory. When Ali logs in as `dev` (using Dev's SSH credentials), he lands here. The `~` in that prompt: `dev@server:~$`: means "you are in your home directory." This is where Ali will put his agent's code. The Secure Tunnel[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#the-secure-tunnel "Direct link to The Secure Tunnel") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Ali is sitting in his apartment. The server is in a data center. How are they connected? **SSH**: Secure Shell. A secure, encrypted tunnel between your computer and the server. Like a phone call to a computer that has no screen. Everything you send through SSH is encrypted. Nobody listening on the network can see what you are typing or what the server sends back. When Ali ran the SSH command, here is what happened: ssh dev@192.168.1.100 Read it left to right: * `ssh`: open a secure tunnel * `dev`: log in as the user "dev" * `@`: at * `192.168.1.100`: this server address That is it. One command. Ali is now inside the server, looking through the glass window (terminal), talking to the voice (shell), exploring the building (filesystem). Claude Code runs in a separate terminal on Ali's laptop. When Ali tells Claude Code to explore the server, Claude Code sends commands through its own connection. The output appears in Ali's Claude Code session. Ali reads the output. He does not need to type a single Linux command himself. Putting It Together[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#putting-it-together "Direct link to Putting It Together") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Ali now knows four things: 1. **Server vs laptop.** A server never sleeps. His agent runs here, not on his laptop. 2. **Terminal and shell.** The terminal is the window. The shell is the voice. He talks to Claude Code; Claude Code talks to the shell. 3. **The filesystem tree.** Everything starts at `/`. His agent's code goes in `/home/dev`. Logs go in `/var`. Configuration goes in `/etc`. 4. **SSH.** A secure tunnel connecting his laptop to the server. Encrypted, reliable, built into every operating system. He is oriented. The blinking cursor is not scary anymore. It is a doorway. But Ali still has a problem. Claude Code runs commands, and the output is full of letters, numbers, and symbols he has never seen before. `drwxr-xr-x`? `4096`? `root root`? Your agent is somewhere in this filesystem. But Claude runs commands and you see output you can't read. That changes next. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Explore the deep directories[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#prompt-1-explore-the-deep-directories "Direct link to Prompt 1: Explore the deep directories") Tell Claude Code to explore /var/log and /etc on this server(or your local machine if you don't have a server).What lives in these directories? Why are they separate from /home?List the five most interesting files you find and explainwhat each one tracks. **What you're learning:** The filesystem is not random. Each directory has a purpose. By exploring /var/log, you see where the server records everything that happens. By exploring /etc, you see where every program stores its settings. This separation (data in one place, config in another, user files in a third) is the organizing principle of every Linux server your agent will ever run on. ### Prompt 2: Compare your machine to a server[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#prompt-2-compare-your-machine-to-a-server "Direct link to Prompt 2: Compare your machine to a server") Compare the directory structure of my local machine with atypical Linux server. Run ls / on my machine and tell me:What directories exist on both? What's different?Why would a server have directories my laptop doesn't need? **What you're learning:** Your laptop and a server share the same tree structure because they both run on the same foundation. The differences reveal what makes a server a server: directories for serving web traffic, storing database files, and managing multiple users. Understanding the overlap builds intuition for working on any Linux system. ### Prompt 3: Why ls -la instead of ls?[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#prompt-3-why-ls--la-instead-of-ls "Direct link to Prompt 3: Why ls -la instead of ls?") When you explored the server, you used ls -la instead of plain ls.Why? What does each flag do? Show me the same directory withjust ls, then with ls -l, then with ls -la. What extra informationappears each time? When would I want one over the other? **What you're learning:** Flags modify commands. The `-l` flag switches from a simple list to a detailed view showing permissions, owners, sizes, and dates. The `-a` flag reveals hidden files (files starting with `.`). Knowing when to ask your agent for more detail versus less is a core skill: you will use this judgment in every lesson ahead. * * * Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#flashcards-study-aid "Direct link to Flashcards Study Aid") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [Why a Server?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#why-a-server) * [The Glass Window and the Voice](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#the-glass-window-and-the-voice) * [Exploring the Building](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#exploring-the-building) * [The Secure Tunnel](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#the-secure-tunnel) * [Putting It Together](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#putting-it-together) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#try-with-ai) * [Prompt 1: Explore the deep directories](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#prompt-1-explore-the-deep-directories) * [Prompt 2: Compare your machine to a server](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#prompt-2-compare-your-machine-to-a-server) * [Prompt 3: Why ls -la instead of ls?](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#prompt-3-why-ls--la-instead-of-ls) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/linux-mastery/where-your-agent-lives#flashcards-study-aid) --- # Chapter 71: ChatKit Server for Agents | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Chapter 70 gave you REST APIs. But agents aren't typical request/response services—they're conversational. Users send messages, agents stream responses, conversations have memory, and sessions persist across interactions. ChatKit Server provides this infrastructure out of the box. OpenAI's ChatKit is a server-side framework purpose-built for conversational AI. It handles the complex patterns that chat interfaces require: streaming token-by-token, managing conversation history, handling interruptions, and maintaining session state. You focus on agent logic; ChatKit handles the conversation mechanics. The best part: ChatKit includes a built-in UI for testing. Connect your agent, and you immediately have a working chat interface—no frontend development required. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#what-youll-learn "Direct link to What You'll Learn") ------------------------------------------------------------------------------------------------------------------------------------------------------------ By the end of this chapter, you'll be able to: * **Understand ChatKit architecture**: Grasp the server model, conversation lifecycle, and how ChatKit differs from raw REST APIs * **Connect agents to ChatKit**: Wire your existing agents (from Chapters 62-65) into ChatKit's conversation flow * **Implement streaming responses**: Handle token-by-token output, partial messages, and real-time updates * **Manage conversation state**: Persist conversation history, handle context windows, and implement memory strategies * **Handle session lifecycle**: Create, resume, and terminate chat sessions with proper state management * **Configure authentication**: Secure ChatKit endpoints and manage user identity across sessions * **Use the built-in UI**: Test your agent immediately with ChatKit's included chat interface Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#chapter-structure "Direct link to Chapter Structure") ------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. **ChatKit Architecture** — Server model, conversation primitives, and comparison with raw FastAPI 2. **Connecting Your First Agent** — Wiring an OpenAI/Google/Anthropic agent into ChatKit 3. **Streaming Implementation** — Token-by-token responses, handling interruptions, and partial message updates 4. **Conversation Management** — History persistence, context window strategies, and conversation branching 5. **Session Lifecycle** — Creating sessions, resuming conversations, timeouts, and cleanup 6. **Authentication & Security** — User identity, session tokens, and securing the conversation endpoint 7. **Built-in UI Integration** — Connecting the OpenAI-provided chat interface to your ChatKit server 8. **Capstone: Conversational Agent** — Spec-driven implementation of a complete ChatKit-powered agent with persistent conversations Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------- * Chapter 70: FastAPI for Agents (API fundamentals) * Chapters 62-65: Agent SDK experience (agents to connect) * Part 4: Python Fundamentals (async/await, streaming patterns) Looking Ahead[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#looking-ahead "Direct link to Looking Ahead") ------------------------------------------------------------------------------------------------------------------------------------------------- With ChatKit Server, you have a complete local conversational product. Chapters 72-73 add quality assurance (TDD and Evals), Chapters 74-75 add advanced patterns (memory, multi-agent), and Chapters 76-78 add data persistence. Part 7 then deploys this complete product to the cloud. * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#what-youll-learn) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#chapter-structure) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#prerequisites) * [Looking Ahead](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server#looking-ahead) --- # Build Your ChatKit Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You built REST APIs in Chapter 70. But agents are conversational, not request/response. Before learning ChatKit Server, you'll **own** a ChatKit skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for ChatKit Server. I willuse it to build conversational AI backends from hello world to professionalproduction systems. Use context7 skill to study official documentation andthen build it so no self assumed knowledge. Claude will: 1. Fetch official ChatKit documentation via Context7 2. Ask you clarifying questions (session management, streaming needs, auth patterns) 3. Create the complete skill with references and templates Your skill appears at `.claude/skills/chatkit-server/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#done "Direct link to Done") ----------------------------------------------------------------------------------------------------------------------------------------------- You now own a ChatKit skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — ChatKit Architecture** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/build-your-chatkit-skill#done) --- # Build Your Google ADK Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning Google's Agent Development Kit, you'll **own** a Google ADK skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for Google ADK (AgentDevelopment Kit). I will use it to build reliable AI agents with Geminifrom hello world to professional production systems. Use context7 skillto study official documentation and then build it so no self assumed knowledge. Claude will: 1. Fetch official Google ADK documentation via Context7 2. Ask you clarifying questions (workflow types, tool patterns, evaluation needs) 3. Create the complete skill with references and templates Your skill appears at `.claude/skills/google-adk/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#done "Direct link to Done") -------------------------------------------------------------------------------------------------------------------------------------------------------------- You now own a Google ADK skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — ADK Fundamentals** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/build-your-google-adk-skill#done) --- # Chapter 66 Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/chapter-quiz#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of MCP concepts and architecture across all lessons. Each question targets specific learning objectives from the chapter. You'll need to demonstrate conceptual understanding (what MCP is and why it exists) and practical knowledge (how to use and configure MCP servers). Checking access... Scoring Guide[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/chapter-quiz#scoring-guide "Direct link to Scoring Guide") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- **14-15 correct:** Mastery - You understand MCP architecture, control models, deployment patterns, and debugging. Ready for Chapter 67 (Building MCP Servers). **12-13 correct:** Proficient - Solid grasp of core concepts. Review any missed lessons before building servers. Focus areas: control model distinction (tools/resources/prompts) and configuration syntax. **10-11 correct:** Competent - Foundation is solid, but some details need reinforcement. Review lessons on: transport layer choice, environment variable syntax, or server evaluation criteria. Practice configuring an MCP server before Chapter 67. **Below 10:** Review Recommended - Core concepts need strengthening. Start with Lesson 1 (architecture overview) and Lesson 3 (the three control models). These form the foundation for everything else. Next Steps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/chapter-quiz#next-steps "Direct link to Next Steps") ------------------------------------------------------------------------------------------------------------------------------------------------------- * If you scored 12+: Proceed to Chapter 67 (Building MCP Servers with FastMCP) * If you scored 10-11: Complete one practice exercise from Lesson 6 (configure an existing MCP server) or Lesson 7 (evaluate and use a community server), then revisit Chapter 67 * If you scored below 10: Review the "differentiation—remedial" sections in lessons that you found challenging, then re-attempt this quiz Checking access... * [Scoring Guide](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/chapter-quiz#scoring-guide) * [Next Steps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/chapter-quiz#next-steps) --- # Build Your RAG Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning RAG (Retrieval-Augmented Generation)—the architecture that gives AI agents access to your private data—you will **own** a RAG skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for RAG (Retrieval-AugmentedGeneration) systems. I will use it to build production RAG pipelines withLangChain and Qdrant vector database - from simple semantic search to advancedpatterns like HyDE, CRAG, and Agentic RAG. Use context7 skill to study officialLangChain and Qdrant documentation and then build it so no self assumed knowledge. Claude will: 1. Fetch official LangChain and Qdrant documentation via Context7 2. Ask you clarifying questions (chunking strategies, embedding models, retrieval patterns) 3. Create the complete skill with ingestion pipelines, retrieval patterns, and evaluation templates Your skill appears at `.claude/skills/rag-deployment/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#done "Direct link to Done") ----------------------------------------------------------------------------------------------------------------------------------------------------------- You now own a RAG skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — Understanding RAG Architecture** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/build-your-rag-skill#done) --- # Core Agent Architecture | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Every agent—from Claude Code to enterprise customer support systems—shares the same four components. Understanding this architecture helps you design agents and debug when things go wrong. The 3+1 Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#the-31-architecture "Direct link to The 3+1 Architecture") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Component | Analogy | Function | | --- | --- | --- | | **Model** | Brain | Reasons about what to do next | | **Tools** | Hands | Acts on the world | | **Orchestration** | Nervous System | Coordinates reasoning, planning, memory | | **Deployment** | Body | Makes the agent accessible | The "3+1" name reflects that Model, Tools, and Orchestration are the core trio, while Deployment wraps them for production use. Model (The Brain)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#model-the-brain "Direct link to Model (The Brain)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The language model is where reasoning happens. It decides what to do next based on context. **Different tasks need different models:** | Task Type | Model Tier | Example | | --- | --- | --- | | Classification, routing | Fast/cheap | "Is this a billing question or technical support?" | | Most reasoning tasks | Medium | "Analyze this customer's history and suggest solutions" | | Complex analysis | Premium | "Review this codebase for security vulnerabilities" | Claude Code demonstrates this. Simple file reads use fast models. Complex refactoring engages premium reasoning. The right model balances capability against cost and latency. **Model selection questions:** * How complex is the reasoning required? * How much latency can users tolerate? * What's the cost per interaction budget? * Does the task need multimodal input (images, audio)? Tools (The Hands)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#tools-the-hands "Direct link to Tools (The Hands)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Tools let agents act on the world. Without tools, an agent can only talk. With tools, it can _do_. **Claude Code's tools include:** * File system: read, write, edit files * Shell execution: run commands, install packages * Web search: gather information * Code execution: test solutions **Tool anatomy:** Every tool has defined inputs, outputs, and permissions. Tool: ProcessRefund Input: customer_id (string), amount (number) Output: confirmation_id (string), status (success/failed) Permission: max $500 per transaction The permission boundary is crucial. A billing agent with "issue refund" access needs limits. Without them, a compromised agent or bad prompt could drain accounts. **Tool design questions:** * What actions does the agent need? * What inputs does each action require? * What limits prevent misuse? * What happens when a tool fails? Orchestration (The Nervous System)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#orchestration-the-nervous-system "Direct link to Orchestration (The Nervous System)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Orchestration is where agent magic happens. It coordinates everything the model and tools do. ### Planning[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#planning "Direct link to Planning") Orchestration breaks goals into executable steps. When you tell Claude Code "refactor this codebase for better error handling," orchestration creates a plan: 1. Read files to understand current structure 2. Identify patterns lacking error handling 3. Plan specific changes 4. Implement changes file by file 5. Run tests to verify 6. Iterate if tests fail The model reasons about _what_ to do. Orchestration manages _how_ to sequence it. ### Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#memory "Direct link to Memory") Agents need to track context across steps: * **Short-term**: What files were read? What was tried? What failed? * **Long-term**: Customer preferences, past interactions, learned patterns Claude Code's memory tracks your current session—files read, changes made, test results. Enterprise agents might remember customer histories across months of interactions. ### Reasoning Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#reasoning-strategies "Direct link to Reasoning Strategies") Different strategies suit different problems: **ReAct (Reason → Act → Observe → Repeat)** The agent thinks about what to do, does it, observes the result, then decides what's next. Most agents use this. **Chain-of-Thought** The agent thinks step-by-step before taking any action. Good for complex reasoning where premature action causes problems. **Reflection** The agent evaluates its own outputs and revises. "Did my code actually fix the bug? Let me check the test output." Claude Code combines these. It reasons about approach (Chain-of-Thought), executes steps (ReAct), and revises when tests fail (Reflection). ### Orchestration Decisions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#orchestration-decisions "Direct link to Orchestration Decisions") When Claude Code works on your request, orchestration decides: * Which files to read (and in what order) * When to ask clarifying questions vs. make assumptions * How to recover when a tool call fails * When the task is complete These decisions are what separates a capable agent from a simple chatbot. Deployment (The Body)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#deployment-the-body "Direct link to Deployment (The Body)") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Deployment makes the agent accessible to users and systems. **Key deployment questions:** | Question | Options | | --- | --- | | Where does it run? | Cloud, local, hybrid | | How do users interact? | CLI, API, chat interface, voice | | What can it access? | File system, databases, external APIs | | How does it scale? | Single user, thousands concurrent | Claude Code runs locally on your machine with full file system access. A customer support agent runs in the cloud, handling thousands of concurrent sessions, with access only to approved systems. **Security is a deployment concern:** * What credentials does the agent have? * What networks can it reach? * What audit trail exists? How Components Interact[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#how-components-interact "Direct link to How Components Interact") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When you ask Claude Code to "fix the failing test in auth.py": 1. **Orchestration** creates a plan: read test → read implementation → analyze error → fix → verify 2. **Model** reasons: "The test expects a User object but gets None. The authentication function doesn't handle the case where the user doesn't exist." 3. **Tools** execute: Read auth.py, read test\_auth.py, edit auth.py to add null check 4. **Model** observes: "Edit complete. Now run the test." 5. **Tools** execute: Run pytest test\_auth.py 6. **Model** observes: "Test passes." 7. **Orchestration** decides: Goal achieved. Report success. If the test still failed, orchestration would loop back—model reasons about _why_ the fix didn't work, tools execute new changes, and the cycle continues. Recognizing Components in Any Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#recognizing-components-in-any-agent "Direct link to Recognizing Components in Any Agent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When you encounter a new agent system, ask these questions: **Model**: What LLM powers it? One model or multiple specialists? What's the cost/capability trade-off? **Tools**: What can it do? What are the limits? What happens when tools fail? **Orchestration**: How does it plan? Does it have memory? What reasoning strategy does it use? How does it know when it's done? **Deployment**: Where does it run? Who can access it? What security constraints exist? These questions work for Claude Code, ChatGPT, customer support bots, or any agent you'll build. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Use Claude, ChatGPT, or Gemini to explore architecture components. > "When I ask Claude Code to 'read a CSV file and save the data to a database,' which component handles each part? Walk through Model, Tools, and Orchestration." **Expected**: Model reasons about approach and SQL structure. Tools handle file reading and database writing. Orchestration sequences the steps and handles errors. > "An agent processes 10,000 customer requests daily: classify intent (easy), analyze sentiment (medium), recommend escalation (hard). Why use different models for each task?" **Expected**: Cost and speed optimization. Fast models for classification (high volume, simple task). Premium models for escalation decisions (lower volume, high stakes). > "A code review agent reads files, checks security issues, analyzes performance, verifies tests pass, then gives feedback. How does orchestration manage this? Where does memory matter?" **Expected**: Orchestration plans the sequence and tracks findings across checks. Memory accumulates issues found in each phase for the final report. **Key insight**: When an agent fails, identify which component failed. Wrong reasoning? Model problem. Can't execute? Tool problem. Lost context mid-task? Orchestration problem. Can't connect? Deployment problem. Checking access... * [The 3+1 Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#the-31-architecture) * [Model (The Brain)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#model-the-brain) * [Tools (The Hands)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#tools-the-hands) * [Orchestration (The Nervous System)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#orchestration-the-nervous-system) * [Planning](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#planning) * [Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#memory) * [Reasoning Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#reasoning-strategies) * [Orchestration Decisions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#orchestration-decisions) * [Deployment (The Body)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#deployment-the-body) * [How Components Interact](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#how-components-interact) * [Recognizing Components in Any Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#recognizing-components-in-any-agent) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/core-agent-architecture#try-with-ai) --- # Multi-Agent Design Patterns | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Single agents hit limits. Complex tasks need multiple specialists working together. Four patterns cover most use cases. Why Multiple Agents?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#why-multiple-agents "Direct link to Why Multiple Agents?") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ A single agent trying to do everything faces problems: * **Context overload**: Too many tools and responsibilities * **Conflicting objectives**: Security analysis requires different thinking than performance optimization * **Debugging difficulty**: When something fails, where did it go wrong? Multiple specialized agents solve these problems through division of labor. The Four Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#the-four-patterns "Direct link to The Four Patterns") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Pattern | Structure | When to Use | | --- | --- | --- | | **Coordinator** | One routes, many execute in parallel | Independent subtasks | | **Sequential** | Each output feeds the next | Ordered pipeline | | **Iterative Refinement** | Generator and critic loop | Quality improvement | | **Human-in-the-Loop** | Agent pauses for human approval | High-stakes decisions | Coordinator Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#coordinator-pattern "Direct link to Coordinator Pattern") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- One agent routes tasks to specialists who work in parallel. User Request → Coordinator → [Security Agent] → [Performance Agent] → Coordinator → Response → [Style Agent] **Use when**: Subtasks are independent. Results need synthesis but don't depend on each other. ### Example: Code Audit[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-code-audit "Direct link to Example: Code Audit") A coordinator receives "audit this codebase" and delegates: * **Security agent**: Checks for vulnerabilities, injection risks * **Performance agent**: Identifies bottlenecks, inefficient patterns * **Documentation agent**: Evaluates code clarity, missing comments All three work simultaneously on the same codebase. The coordinator synthesizes their findings into a unified report. ### Why Coordinator Works Here[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#why-coordinator-works-here "Direct link to Why Coordinator Works Here") Each analysis is independent. The security agent doesn't need performance results to do its job. Parallel execution saves time. If one agent fails, the others still complete. ### When NOT to Use Coordinator[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#when-not-to-use-coordinator "Direct link to When NOT to Use Coordinator") If results depend on each other, coordinator creates problems. "Write a function, then test it" requires sequence—you can't test code that doesn't exist yet. Sequential Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#sequential-pattern "Direct link to Sequential Pattern") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Each agent's output becomes the next agent's input. Input → Agent A → Agent B → Agent C → Output **Use when**: Steps must happen in order. Later steps depend on earlier results. ### Example: Content Pipeline[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-content-pipeline "Direct link to Example: Content Pipeline") 1. **Research agent**: Gathers information on a topic 2. **Writing agent**: Creates draft from research 3. **Editor agent**: Polishes draft for clarity and style Each step depends on the previous. Can't write without research. Can't edit without a draft. ### Example: Claude Code Refactoring[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-claude-code-refactoring "Direct link to Example: Claude Code Refactoring") When you ask Claude Code to refactor a module: 1. **Read phase**: Understand current code structure 2. **Analyze phase**: Identify patterns to improve 3. **Plan phase**: Design changes 4. **Implement phase**: Make edits 5. **Test phase**: Verify changes work Each phase needs the previous phase's output. This is sequential even within a single agent. ### Why Sequential Works Here[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#why-sequential-works-here "Direct link to Why Sequential Works Here") Order matters. The writing agent needs research results as input. The editor needs the draft. Skipping steps or reordering breaks the pipeline. Iterative Refinement Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#iterative-refinement-pattern "Direct link to Iterative Refinement Pattern") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Agents critique and improve each other's work in cycles. Generator → Critic → Generator (revised) → Critic → ... → Approved Output **Use when**: Quality matters more than speed. First attempts need improvement. ### Example: Code Review Loop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-code-review-loop "Direct link to Example: Code Review Loop") 1. **Coding agent**: Writes implementation 2. **Review agent**: Identifies issues, suggests improvements 3. **Coding agent**: Revises based on feedback 4. **Review agent**: Re-evaluates 5. Repeat until review agent approves ### The Critic's Role[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#the-critics-role "Direct link to The Critic's Role") The critic agent has specific evaluation criteria: * Does the code handle edge cases? * Are there security vulnerabilities? * Is the logic clear and maintainable? Each critique round makes the output better. The loop terminates when quality threshold is met. ### Quality vs Speed Trade-off[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#quality-vs-speed-trade-off "Direct link to Quality vs Speed Trade-off") More iterations = better quality but slower output. Design decisions: * Maximum iterations (prevent infinite loops) * Quality threshold (when is "good enough"?) * Escalation path (what if quality threshold can't be met?) ### Example: Document Generation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-document-generation "Direct link to Example: Document Generation") 1. **Writer**: Produces first draft 2. **Fact-checker**: Verifies claims, flags inaccuracies 3. **Writer**: Corrects errors 4. **Fact-checker**: Re-verifies 5. Continue until all facts verified The iterative pattern catches errors that a single pass would miss. Human-in-the-Loop Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#human-in-the-loop-pattern "Direct link to Human-in-the-Loop Pattern") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Agents pause for human approval before critical actions. Agent work → Human approval gate → Agent continues **Use when**: Stakes are high. Actions are irreversible. Legal, financial, or safety implications exist. ### Example: Refund Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-refund-agent "Direct link to Example: Refund Agent") * Under $100: Automatic approval * $100-$500: Agent processes, logs for review * Over $500: Requires human approval before processing * Over $1000: Escalates to manager The thresholds reflect risk tolerance. Small refunds aren't worth human time. Large refunds need oversight. ### Designing Approval Gates[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#designing-approval-gates "Direct link to Designing Approval Gates") Good approval gates are: * **Clear**: Agent knows exactly what triggers human review * **Efficient**: Most requests don't need human input * **Safe**: High-risk actions always get reviewed Bad approval gates: * Too aggressive: Humans review everything (defeats the purpose of automation) * Too permissive: Risky actions proceed without oversight ### Example: Code Deployment[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-code-deployment "Direct link to Example: Code Deployment") Claude Code implementing a feature: 1. Agent makes changes 2. Agent runs tests 3. If tests pass: "Should I commit these changes?" (human gate) 4. If approved: Commit and optionally push The human gate before commit prevents unwanted changes from persisting. Pattern Selection Guide[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#pattern-selection-guide "Direct link to Pattern Selection Guide") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Match the pattern to the problem structure: | Problem Structure | Best Pattern | Reasoning | | --- | --- | --- | | Multiple independent analyses | Coordinator | Parallelize for speed | | Clear step-by-step process | Sequential | Maintain order dependencies | | Quality-critical output | Iterative Refinement | Improve through cycles | | High-risk decisions | Human-in-the-Loop | Human oversight for safety | ### Patterns Combine[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#patterns-combine "Direct link to Patterns Combine") Real systems often use multiple patterns: **Customer Support System**: * **Coordinator** routes requests to specialists (billing, technical, shipping) * **Sequential** within each specialist's workflow (gather context → diagnose → resolve) * **Human-in-the-Loop** for escalations (large refunds, angry customers, policy exceptions) **Document Generation System**: * **Sequential** pipeline (research → write → edit) * **Iterative Refinement** at each stage (write → fact-check → revise) * **Human-in-the-Loop** for final approval Claude Code's Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#claude-codes-patterns "Direct link to Claude Code's Patterns") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Claude Code uses all four patterns internally: **Coordinator**: When auditing a codebase, it may run security, performance, and style checks in parallel (using subagents). **Sequential**: When refactoring, it reads → analyzes → plans → implements → tests in sequence. **Iterative Refinement**: When tests fail, it revises code and re-runs until tests pass. **Human-in-the-Loop**: When it asks "Should I proceed with this change?" before destructive operations. Understanding these patterns helps you predict Claude Code's behavior and guide it more effectively. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to practice pattern matching. > "A contract review system needs to: extract key terms, check against templates, identify risks, verify legal compliance. Which pattern? Why not the others?" **Expected**: Sequential—each step depends on the previous. Can't check compliance before extracting terms. Can't identify risks before comparing to templates. > "A customer support system handles 100K requests daily. Compare a single super-agent vs a Coordinator with 5 specialists. What matters at scale?" **Expected**: Coordinator—specialists can parallelize, failures are isolated, easier to debug and improve individual components. Single agent becomes a bottleneck. > "A billing agent can refund up to $500 automatically. Design the Human-in-the-Loop pattern for amounts above that. What thresholds make sense?" **Expected**: $500-$1000 might need supervisor approval. Above $1000 might need manager approval. Thresholds based on business risk tolerance. > "I'm building a code generation system. First draft quality is usually around 70%. I need 95%+. Which pattern? How many iterations should I allow?" **Expected**: Iterative Refinement with a critic agent. Allow 3-5 iterations with clear quality criteria. Escalate to human review if quality threshold not met after max iterations. **Key insight**: Wrong patterns don't just underperform—they create failure modes. Match pattern to problem structure. Checking access... * [Why Multiple Agents?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#why-multiple-agents) * [The Four Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#the-four-patterns) * [Coordinator Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#coordinator-pattern) * [Example: Code Audit](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-code-audit) * [Why Coordinator Works Here](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#why-coordinator-works-here) * [When NOT to Use Coordinator](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#when-not-to-use-coordinator) * [Sequential Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#sequential-pattern) * [Example: Content Pipeline](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-content-pipeline) * [Example: Claude Code Refactoring](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-claude-code-refactoring) * [Why Sequential Works Here](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#why-sequential-works-here) * [Iterative Refinement Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#iterative-refinement-pattern) * [Example: Code Review Loop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-code-review-loop) * [The Critic's Role](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#the-critics-role) * [Quality vs Speed Trade-off](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#quality-vs-speed-trade-off) * [Example: Document Generation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-document-generation) * [Human-in-the-Loop Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#human-in-the-loop-pattern) * [Example: Refund Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-refund-agent) * [Designing Approval Gates](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#designing-approval-gates) * [Example: Code Deployment](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#example-code-deployment) * [Pattern Selection Guide](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#pattern-selection-guide) * [Patterns Combine](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#patterns-combine) * [Claude Code's Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#claude-codes-patterns) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/multi-agent-design-patterns#try-with-ai) --- # The Agent SDK Landscape | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've learned what agents are and how they work. Now comes a practical question: which framework should you use to build one? This lesson surveys the four major agent development frameworks you'll master in Chapters 62-66. Each has a distinct philosophy—understanding these differences helps you match frameworks to problems. The Framework Question[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#the-framework-question "Direct link to The Framework Question") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you decide to build an agent, you'll encounter SDKs with fundamentally different approaches. The question isn't "which is best?" It's "which philosophy matches my problem?" In an **AIDD/SDD workflow**, you don't manually write boilerplate. You write a **Specification** defining the agent's Persona, Tools, Workflow, and Guardrails—then use an AI Architect to generate the implementation for your chosen SDK. Understanding each framework's philosophy tells you what your spec should emphasize. Four Major Frameworks[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#four-major-frameworks "Direct link to Four Major Frameworks") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Framework | Core Philosophy | SDD Spec Focus | Best For | | --- | --- | --- | --- | | **OpenAI Agents SDK** | Handoff-Centric | Handoff networks, guardrails | Routing/triage workflows | | **Google ADK** | Service-Centric | Artifact schemas, A2A protocols | Stateful business processes | | **Anthropic Agents Kit** | Capability-Centric | MCP tool definitions, system prompts | Task execution (coding, research) | | **Microsoft Agent Framework** | Conversation-Centric | Group interaction patterns, plugins | Multi-agent collaboration | ### OpenAI Agents SDK (Chapter 62)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#openai-agents-sdk-chapter-62 "Direct link to OpenAI Agents SDK (Chapter 62)") **Philosophy**: Handoff-Centric—simple agents transferring control like a relay race. OpenAI's approach focuses on **handoffs**. An agent is a configuration that can "transfer" the conversation to another agent when appropriate. **Architecture**: Lightweight Python/Node implementation using "Handoff" primitives. Define a Triage Agent and Specialist Agents, with explicit rules for when Agent A passes control to Agent B. **SDD Spec Focus**: Your spec defines the **Router Pattern**—which specialists exist, and the `handoff_instructions` that govern transfers. agent = Agent( name="Support", instructions="...", handoffs=[sales_agent, tech_agent]) **When to use**: Customer support, simple workflows where control transfers between specialists. Best for developers who want a "keep it simple" approach without complex graph orchestration. ### Google ADK (Chapter 63)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#google-adk-chapter-63 "Direct link to Google ADK (Chapter 63)") **Philosophy**: Service-Centric—agents as microservices with strict state management. Google ADK treats agents like **software components**. It emphasizes **Artifacts**—structured objects that store state—and agents that manipulate them through well-defined interfaces. **Architecture**: Engineering-heavy structure with strong typing (TypeScript/Pydantic), state management hooks, and the **Agent2Agent (A2A)** protocol for agent-to-agent communication. Integrates with Google Cloud (Vertex AI, Firebase Genkit). **SDD Spec Focus**: Your spec defines **data schemas**—what Artifacts the agent manipulates, `input_schema`, `output_schema`, and `persistence_layer`. ai.defineFlow(...) with strict input/output Zod schemas **When to use**: Enterprise applications needing strict state management, business processes with data requirements (loan processing, order management), Google Cloud integration. Best for "engineering-first" teams building robust, stateful systems. ### Anthropic Agents Kit (Chapter 65)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#anthropic-agents-kit-chapter-65 "Direct link to Anthropic Agents Kit (Chapter 65)") **Philosophy**: Capability-Centric—single powerful agents with deep tool and computer use. Anthropic's ecosystem centers on the **Claude Agent SDK** and **Model Context Protocol (MCP)**. It assumes the model is smart enough to handle complex flows if given the right tools. **Architecture**: Heavy reliance on MCP servers (connectors) and "Computer Use" capabilities. You don't script the flow—you script the _environment_ the agent lives in. **SDD Spec Focus**: Your spec defines **capabilities**—the tools the agent accesses (FileSystem, GitHub API, databases). The key attribute is `MCP Server Configuration`. Client connects to mcp-server; Claude acts, observes tool output, acts again **When to use**: Complex execution tasks—agents that actually _do work_ (coding, file editing, research, data analysis) rather than just routing conversations. Best for "super-worker" agents that operate computers and tools. ### Microsoft Agent Framework (Chapter 66)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#microsoft-agent-framework-chapter-66 "Direct link to Microsoft Agent Framework (Chapter 66)") **Philosophy**: Conversation-Centric—multi-agent collaboration and group chats. Microsoft's framework combines **Semantic Kernel** (enterprise integration) and **AutoGen** (multi-agent orchestration). It enables complex "Group Chats" where multiple agents collaborate to solve problems. **Architecture**: Multiple agent classes coordinated by a Group Chat Manager. Agents speak to each other in loops with defined interaction rules and termination criteria. **SDD Spec Focus**: Your spec defines **social dynamics**—roles (Manager, Coder, Reviewer), rules of engagement ("Reviewer must approve before Coder commits"), `GroupChatSelectionMethod`, and `TerminationCriteria`. group_chat = GroupChat(agents=[user_proxy, coder, pm], max_round=12)manager = GroupChatManager(groupchat=group_chat) **When to use**: Complex problem-solving requiring multiple perspectives, team simulations, brainstorming processes, enterprise process automation. The gold standard for simulating a "digital workforce." Framework Comparison Matrix[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#framework-comparison-matrix "Direct link to Framework Comparison Matrix") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Dimension | OpenAI Agents SDK | Google ADK | Anthropic Agents Kit | Microsoft Framework | | --- | --- | --- | --- | --- | | **Core Pattern** | Handoffs | Artifacts/State | MCP Tools | Group Chats | | **Spec Emphasis** | Transfer rules | Data schemas | Tool definitions | Interaction rules | | **Complexity** | Low | High | Medium | High | | **State Management** | Minimal | Strict | Tool-dependent | Conversation-based | | **Multi-Agent** | Sequential handoffs | A2A protocol | Single agent focus | Native group chat | | **Best Strength** | Simplicity | Enterprise rigor | Deep execution | Collaboration | How to Choose: Match Philosophy to Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#how-to-choose-match-philosophy-to-problem "Direct link to How to Choose: Match Philosophy to Problem") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Problem Type → Framework[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#problem-type--framework "Direct link to Problem Type → Framework") | Your Problem | Best Framework | Why | | --- | --- | --- | | **Routing/Triage** (customer service) | OpenAI Agents SDK | Handoff pattern matches naturally | | **Business Process** (strict data requirements) | Google ADK | Artifact schemas enforce state | | **Task Execution** (coding, research) | Anthropic Agents Kit | MCP enables deep tool use | | **Team Collaboration** (brainstorming, review) | Microsoft Framework | Group chat orchestration | ### Decision Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#decision-questions "Direct link to Decision Questions") **Question 1: What's the core interaction pattern?** * Handing off between specialists → **OpenAI** * Manipulating structured data → **Google ADK** * Operating tools and computers → **Anthropic** * Multiple agents discussing/debating → **Microsoft** **Question 2: What does your spec naturally describe?** * Transfer conditions ("when X, hand to Y") → **OpenAI** * Data schemas and state transitions → **Google ADK** * Tool capabilities and permissions → **Anthropic** * Roles, rules of engagement, termination → **Microsoft** **Question 3: What's your deployment context?** * Quick MVP, simple flows → **OpenAI** * Google Cloud, enterprise requirements → **Google ADK** * Complex autonomous work → **Anthropic** * Multi-perspective problem solving → **Microsoft** The Critical Insight: Concepts Transfer[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#the-critical-insight-concepts-transfer "Direct link to The Critical Insight: Concepts Transfer") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Core concepts from Lessons 1-6 appear in every framework:** * **3+1 Architecture**: All frameworks have Model, Tools, Orchestration, Deployment—implemented differently * **5-Step Loop**: Every agent reasons, acts, observes, iterates—the primitives vary * **Multi-agent patterns**: Coordinator, Sequential, HITL exist across frameworks with different APIs * **Agent Ops**: Evaluation and debugging principles apply universally * **Security**: Guardrails and identity requirements transcend frameworks When you learn OpenAI's handoff pattern and later need Google's artifact-based approach, you're not starting over. You're learning how ADK implements patterns you already understand. **This is why Chapter 61 before Chapters 62-66 matters.** The conceptual foundation is framework-agnostic. Implementation details change; architectural thinking persists. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to explore framework selection. > "I'm building a customer support agent that routes inquiries to billing, technical, or sales specialists. Each specialist handles their domain, then control returns to triage. Which framework? Why?" **Expected**: OpenAI Agents SDK—the handoff pattern matches exactly. Spec defines triage agent + specialists + transfer conditions. > "I'm building an agent that processes loan applications: validate documents, check credit, calculate terms, generate approval letter. Strict data requirements, audit trail needed. Which framework?" **Expected**: Google ADK—artifact-based state management, strict schemas, enterprise audit requirements match the service-centric philosophy. > "I need an agent that can clone a GitHub repo, analyze the code, make improvements, run tests, and submit a PR. It needs to operate tools autonomously. Which framework?" **Expected**: Anthropic Agents Kit—MCP tool definitions, computer use capabilities. Spec focuses on tool access, not conversation flow. > "I want a product design simulation: a PM proposes features, an engineer estimates effort, a designer critiques UX, they iterate until consensus. Which framework?" **Expected**: Microsoft Agent Framework—group chat with roles, interaction rules, termination criteria. This is multi-agent collaboration. **Key insight**: The framework you choose shapes what your specification emphasizes. Match the framework's philosophy to your problem's structure. Checking access... * [The Framework Question](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#the-framework-question) * [Four Major Frameworks](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#four-major-frameworks) * [OpenAI Agents SDK (Chapter 62)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#openai-agents-sdk-chapter-62) * [Google ADK (Chapter 63)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#google-adk-chapter-63) * [Anthropic Agents Kit (Chapter 65)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#anthropic-agents-kit-chapter-65) * [Microsoft Agent Framework (Chapter 66)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#microsoft-agent-framework-chapter-66) * [Framework Comparison Matrix](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#framework-comparison-matrix) * [How to Choose: Match Philosophy to Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#how-to-choose-match-philosophy-to-problem) * [Problem Type → Framework](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#problem-type--framework) * [Decision Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#decision-questions) * [The Critical Insight: Concepts Transfer](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#the-critical-insight-concepts-transfer) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-sdk-landscape#try-with-ai) --- # What Is an AI Agent? | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've been using an AI agent for months. Claude Code reasons about your requests, uses tools, observes results, and iterates until it succeeds. That's not how ChatGPT works. This lesson explains what makes Claude Code different—and why that difference matters. The Definition[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-definition "Direct link to The Definition") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- An AI agent uses a language model **in a loop** to accomplish a goal: 1. **Reason** about what to do 2. **Act** using tools 3. **Observe** the result 4. **Repeat** until done ChatGPT responds once. Claude Code loops until your task is complete. That loop is what makes something an agent. When you ask Claude Code to "refactor this code and verify it works," it: * Reads code (observes) * Plans refactoring approach (reasons) * Writes new code (acts) * Runs tests (observes) * Fixes failures (loops) You didn't specify those steps. The agent reasoned through them autonomously. The 5-Level Taxonomy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-5-level-taxonomy "Direct link to The 5-Level Taxonomy") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not all agents are equal. The taxonomy classifies systems by autonomy: | Level | Name | Who Controls Strategy? | Example | | --- | --- | --- | --- | | 0 | Core Reasoning | No one (single response) | ChatGPT without tools | | 1 | Connected | Human drives each step | ChatGPT with web search | | 2 | Strategic | System plans multi-step | Claude Code on a refactor | | 3 | Multi-Agent | System coordinates specialists | Code + test + docs agents | | 4 | Self-Evolving | System creates new tools | Agent builds custom analyzers | **Level 0**: Model responds once. No tools, no loop. **Level 1**: Model uses tools, but you drive the strategy. "Search for X" → results → you decide next step. **Level 2**: Model plans and executes multi-step strategies. You set the goal; it figures out how. Claude Code typically operates here. **Level 3**: Multiple specialized agents coordinate. A code agent, test agent, and docs agent work together. Claude Code does this on complex tasks. **Level 4**: The system creates new capabilities to solve novel problems. Rare today. ### Where Claude Code Fits[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#where-claude-code-fits "Direct link to Where Claude Code Fits") Claude Code is Level 2-3 depending on the task: **Level 2 example**: "Debug this error." * Asks clarifying questions * Requests logs and code * Reasons through causes * Suggests and tests fixes * Iterates until solved **Level 3 example**: "Audit this codebase for security, performance, and correctness." * Security specialist examines auth and data handling * Performance analyst checks bottlenecks * Quality agent reviews tests and docs * Coordinator synthesizes findings You experience one interface. Behind it, multiple perspectives coordinate. Director vs Bricklayer[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#director-vs-bricklayer "Direct link to Director vs Bricklayer") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The paradigm shift that matters: **Bricklayer** (traditional): You specify every step. "Read file X, parse format Y, handle error Z." You're the executor. **Director** (agent era): You specify intent and constraints. "Extract structured data from these documents. Ensure quality." The agent reasons about how. | Bricklayer | Director | | --- | --- | | "Write validation with these exact checks" | "Validate input securely" | | "Call API A, retry 3 times on timeout" | "Fetch data reliably" | | "Parse JSON, extract fields X, Y, Z" | "Get the relevant information" | Director-level thinking is harder. You must articulate intent clearly, anticipate failure modes, and trust the system to reason. But it's more powerful—agents adapt to situations you didn't anticipate. This is why specification skills (from Part 4) become critical. Vague goals create unreliable agents. Precise specifications create predictable systems. Why This Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#why-this-matters "Direct link to Why This Matters") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Agent development skills are scarce. Most developers can _use_ ChatGPT. Few can _design_ agent systems that work reliably. The gap: Companies need people who can specify agent behavior precisely, debug agent reasoning, and decide when to use agents vs traditional code. That's what Chapters 62-65 teach. The Strategic Choice: General Agents vs Custom Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-strategic-choice-general-agents-vs-custom-agents "Direct link to The Strategic Choice: General Agents vs Custom Agents") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Beyond taxonomy, there's a strategic decision every AI builder faces: **General Agents** vs **Custom Agents**. ### General Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#general-agents "Direct link to General Agents") Ready-to-use AI partners that reason across many domains. Claude Code, Gemini CLI, and Goose are General Agents. **Characteristics:** * Flexible reasoning across diverse problems * Zero-shot planning (figure out novel situations) * Human-in-the-loop collaboration * Instant deployment (install and run) **Best for:** Novel problems, complex debugging, ad-hoc analysis, creative work, exploration **Analogy:** Hiring a senior consultant who figures things out ### Custom Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#custom-agents "Direct link to Custom Agents") Purpose-built AI systems you design using SDKs (OpenAI Agents SDK, Claude SDK, Google ADK). **Characteristics:** * Optimized for specific workflows * Guardrails and process control * Deterministic, repeatable behavior * Customer-facing reliability **Best for:** Standardized procedures, high-volume automation, customer-facing products, SOPs **Analogy:** Building a factory machine for a specific task ### The Decision Matrix[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-decision-matrix "Direct link to The Decision Matrix") | Factor | Choose General Agent | Choose Custom Agent | | --- | --- | --- | | Task type | Novel, varied | Repetitive, standardized | | End user | Technical (developers) | Non-technical (customers) | | Error tolerance | High (human reviews) | Low (must be reliable) | | Cost model | High value per task | Volume optimization | | Time to deploy | Minutes | Weeks | ### The Agent Factory Insight[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-agent-factory-insight "Direct link to The Agent Factory Insight") Here's the paradigm shift: **General Agents build Custom Agents**. Claude Code (a General Agent) can: 1. Read your specification 2. Understand the SDK documentation 3. Generate Custom Agent code 4. Test and iterate until it works You don't choose between General and Custom—you use General Agents to _create_ Custom Agents. This is the "Agent Factory" model. **Your expertise + General Agent + SDK = Deployable Custom Agent** Skills you create in Claude Code (Chapter 3) become the building blocks—reusable intelligence packaged in SKILL.md files that teach agents specialized workflows. These skills are monetizable IP: the "recipe" for your agent's expertise. In Chapters 62-65, you'll use Claude Code to build Custom Agents with OpenAI, Claude, and Google SDKs. ### The Business Angle[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-business-angle "Direct link to The Business Angle") Custom Agents aren't just technical artifacts—they're **products you can sell**. In Chapter 3 Lesson 14, you learned about the Digital FTE model: packaging AI agents as "virtual employees" with monthly subscriptions. The agents you build in this chapter are the technical foundation for that business model. | What You Build | How It Makes Money | | --- | --- | | Customer support agent | Digital FTE: $1,000/month subscription | | Document review agent | Success fee: $5 per document processed | | Lead qualification agent | License: $50,000/year to enterprises | | Data analysis agent | Marketplace: Publish to OpenAI Apps | The technical skills in Chapters 62-65 directly translate to revenue. Every agent you learn to build is a potential product. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to practice classification. > "A bank chatbot asks for my account number to check my balance. What level in the 5-Level Taxonomy? Why not higher?" **Expected**: Level 1—has tools but humans drive strategy. > "When Claude Code refactors a 500-line file and improves test coverage, describe its reason-act-observe loop. What level?" **Expected**: Level 2-3—it plans strategy, executes multi-step, may coordinate specialists. > "I tell an agent 'validate user input securely' vs 'write these exact validation checks.' Which is director thinking? Why is it harder but more powerful?" **Expected**: First is director. Harder because you must specify intent precisely. More powerful because the agent adapts. **Key insight**: The level isn't about sophistication of tools—it's about who controls the reasoning loop. > "A startup wants to automate customer onboarding emails. Should they use Claude Code (General Agent) or build a Custom Agent with OpenAI SDK? What factors drive that decision?" **Expected**: Custom Agent—standardized workflow, customer-facing, needs reliability. But they'd use Claude Code to _build_ the Custom Agent. > "Explain the 'Agent Factory' concept. Why don't you choose between General and Custom Agents?" **Expected**: General Agents build Custom Agents. You use Claude Code to create purpose-built agents with SDKs. They're complementary, not competing. Checking access... * [The Definition](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-definition) * [The 5-Level Taxonomy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-5-level-taxonomy) * [Where Claude Code Fits](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#where-claude-code-fits) * [Director vs Bricklayer](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#director-vs-bricklayer) * [Why This Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#why-this-matters) * [The Strategic Choice: General Agents vs Custom Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-strategic-choice-general-agents-vs-custom-agents) * [General Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#general-agents) * [Custom Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#custom-agents) * [The Decision Matrix](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-decision-matrix) * [The Agent Factory Insight](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-agent-factory-insight) * [The Business Angle](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#the-business-angle) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/what-is-an-ai-agent#try-with-ai) --- # Your First Agent Concept | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've mastered the foundational frameworks. Now synthesize them by designing your own agent specification. This lesson is not a tutorial—it's your opportunity to apply everything you've learned. By the end, you'll have written a complete specification that could guide implementation in Chapters 62-65. Specification-First Thinking[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#specification-first-thinking "Direct link to Specification-First Thinking") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Traditional software specs describe _what code does_: "Read file, parse JSON, validate schema, return result." Agent specs describe _how the agent thinks and behaves_: "You are a customer support agent. Your goal is to resolve inquiries with empathy. You can issue refunds up to $500. Escalate anything larger." The difference is profound. Traditional specs prescribe a deterministic path. Agent specs define constraints within which the agent reasons to find its own path. This makes agent specs harder to write. You must: * Anticipate failure modes without prescribing solutions * Set boundaries without removing flexibility * Enable competence without dictating steps The frameworks you've learned help structure this thinking. Agent Specification Template[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#agent-specification-template "Direct link to Agent Specification Template") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use this structure for any agent: ### Section 1: Purpose & Capability Level[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-1-purpose--capability-level "Direct link to Section 1: Purpose & Capability Level") **Agent Name**: \[Clear, memorable name\] **Purpose**: \[One paragraph: What does this agent accomplish? Why does it matter?\] **Capability Level** (5-Level Taxonomy): * **Level 0**: Simple processor (no reasoning) * **Level 1**: Router with retrieval (RAG) * **Level 2**: Strategic planner (multi-step reasoning) * **Level 3**: Multi-agent coordinator * **Level 4**: Self-evolving (creates tools/agents) **Why this level?** \[Justify based on task complexity\] ### Section 2: Architecture (3+1)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-2-architecture-31 "Direct link to Section 2: Architecture (3+1)") **Model (Brain)**: * Which LLM? Why? * Cost/capability trade-off reasoning * Multimodal requirements? **Tools (Hands)**: * Information retrieval tools (what can it read?) * Action tools (what can it do?) * Guardrails on each tool (what limits?) **Orchestration (Nervous System)**: * Memory strategy (session-only vs persistent) * Reasoning approach (ReAct, Chain-of-Thought, Reflection) * Success criteria (how does it know when it's done?) **Deployment (Body)**: * Where does it run? * How do users access it? * Scaling requirements? ### Section 3: Process (5-Step Loop)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-3-process-5-step-loop "Direct link to Section 3: Process (5-Step Loop)") Walk through a concrete example using the 5-Step Loop: 1. **Get Mission**: How does the agent receive its goal? 2. **Scan Scene**: What context must it gather? 3. **Think Through**: How does it plan? 4. **Take Action**: What tools does it use? 5. **Observe & Iterate**: How does it verify success? ### Section 4: Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-4-pattern "Direct link to Section 4: Pattern") **Multi-agent pattern**: \[Single / Coordinator / Sequential / Iterative Refinement / HITL\] **Why this pattern?** \[Justify based on task structure\] If multi-agent: * What does each agent do? * How do they coordinate? * What's the data flow? ### Section 5: Security[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-5-security "Direct link to Section 5: Security") **Deterministic guardrails** (hard limits): * \[Rule 1\] * \[Rule 2\] **Guard model checks** (contextual evaluation): * \[Check 1\] * \[Check 2\] **Trust trade-off**: What power does it have? How is risk mitigated? **Compromise plan**: What happens if this agent is breached? Example Specification: Customer Support Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#example-specification-customer-support-agent "Direct link to Example Specification: Customer Support Agent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Purpose & Level[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#purpose--level "Direct link to Purpose & Level") **Agent Name**: SupportBot **Purpose**: Resolve customer service inquiries autonomously, escalating complex or high-value issues to human agents. Reduce support queue while maintaining customer satisfaction. **Capability Level**: 2 (Strategic planner) **Why Level 2**: Requires multi-step reasoning: gather customer context → diagnose problem → determine resolution path → execute or escalate. Doesn't need multi-agent coordination—a single agent with good tools handles this scope. ### Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#architecture "Direct link to Architecture") **Model**: Claude 3.5 Haiku * Justification: Customer support is routine reasoning with high volume. Haiku balances speed, cost, and capability. Multimodal enabled for screenshot analysis when customers share images. **Tools**: * `SearchCustomerHistory`: Query orders, past tickets, account status * `CheckProductInfo`: Lookup specs, warranty, return policy * `ProcessRefund`: Issue refunds ≤$500 (hard limit in tool) * `CreateEscalation`: Route to human with full context **Orchestration**: * Memory: Session-based (current conversation only) * Reasoning: ReAct loop with explicit planning * Success: Issue resolved OR escalation created with context **Deployment**: * Cloud Run with auto-scaling * Chat widget on customer portal * API for integration with existing ticketing system ### Process (5-Step Loop Example)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#process-5-step-loop-example "Direct link to Process (5-Step Loop Example)") **Scenario**: Customer asks "I ordered a laptop but received a tablet. Order #45678." **Step 1 - Get Mission**: Resolve wrong-item shipment for order #45678. **Step 2 - Scan Scene**: * Call `SearchCustomerHistory("45678")` → Order found: laptop ordered, tablet shipped * Call `CheckProductInfo("laptop_sku")` → In stock **Step 3 - Think Through**: * Wrong item confirmed (laptop vs tablet) * Options: Refund + new order, or exchange * Customer wants the laptop → exchange is best * Need shipping label for return **Step 4 - Take Action**: * Call `CreateExchange(order_id="45678", return_item="tablet", ship_item="laptop")` * Generate return shipping label * Send confirmation email **Step 5 - Observe**: * Exchange created successfully * Customer has return label and new delivery estimate * Mission complete ### Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#pattern "Direct link to Pattern") **Pattern**: Single agent with Human-in-the-Loop escalation **Why**: Task complexity doesn't require multiple specialists. Single agent handles most cases. HITL triggers for: refunds >$500, account closure, legal threats, angry customers (detected by sentiment). ### Security[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#security "Direct link to Security") **Deterministic guardrails**: * Refund cap: $500/transaction, $2000/day per agent * Order access: Only orders matching authenticated customer * No account deletion capability **Guard model checks**: * Before refund: Is request pattern consistent with legitimate use? * Email content: Scan for phishing patterns before sending * Escalation: Flag if customer mentions legal action or media **Trust trade-off**: Agent has financial authority (refunds) and customer data access. Risk mitigated by caps, authentication, and audit logging. **Compromise plan**: Revoke API credentials → Review audit log for unauthorized refunds → Customer notification if data accessed → Maximum exposure: $2000/day × rate limit. Your Assignment[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#your-assignment "Direct link to Your Assignment") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Choose an agent type and write a complete specification: **Option A: Internal Tool Agent** * Expense approval bot * IT support assistant * HR scheduling agent **Option B: Customer-Facing Agent** * E-commerce support * Travel booking assistant * Financial advisor **Option C: Development Support Agent** * Code review assistant * Documentation writer * Test generation agent ### Requirements[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#requirements "Direct link to Requirements") Your specification must include: 1. **Purpose and capability level** with justification (why this level?) 2. **3+1 Architecture** with trade-off reasoning (why this model? why these tools?) 3. **5-Step Loop walkthrough** with a concrete example (not abstract) 4. **Pattern choice** with justification (why not other patterns?) 5. **Security design** realistic for the use case (not over-engineered, not under-engineered) ### Writing Tips[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#writing-tips "Direct link to Writing Tips") * Write as if this spec will guide an engineer implementing the agent * Be specific: "refund up to $500" not "can issue refunds" * Justify decisions: "Claude Haiku because high volume + routine reasoning" not just "Claude Haiku" * Include a realistic example—actually walk through the 5-Step Loop * Reference the frameworks: "Level 2 because multi-step planning required but not multi-agent coordination" Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to validate and strengthen your specification. > "I've written an agent spec with Capability Level \[X\], Tools \[Y\], Pattern \[Z\], and Guardrails \[W\]. Does this provide enough detail for an engineer to implement? What's missing?" **Expected**: The AI identifies gaps—missing tools, unclear orchestration, security holes. Use this to strengthen your spec. > "My agent specification uses \[Sequential/Coordinator/etc\] pattern. What would change if I switched to \[alternative\]? What would improve? What would break?" **Expected**: Trade-off exploration. Discover whether your design choice is optimal or if there's a better pattern. > "Here are the guardrails in my specification: \[list them\]. What attack vectors could someone exploit despite these guardrails? What additional safeguards would you recommend?" **Expected**: Threat modeling you hadn't considered. Realistic security analysis. > "Based on my specification, what could go wrong in production? What operational metrics should I track? What constitutes success vs failure?" **Expected**: Production readiness thinking. Agent Ops criteria that guide implementation. > "Walk through my 5-Step Loop example and identify where my agent might fail. What happens if the tool call fails? What if the customer changes their request mid-conversation?" **Expected**: Edge case analysis. Robustness thinking. **Key insight**: The spec is your thinking made explicit. If you can't specify it clearly, you don't understand it well enough to build it. Use AI to find the gaps in your understanding. Checking access... * [Specification-First Thinking](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#specification-first-thinking) * [Agent Specification Template](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#agent-specification-template) * [Section 1: Purpose & Capability Level](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-1-purpose--capability-level) * [Section 2: Architecture (3+1)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-2-architecture-31) * [Section 3: Process (5-Step Loop)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-3-process-5-step-loop) * [Section 4: Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-4-pattern) * [Section 5: Security](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#section-5-security) * [Example Specification: Customer Support Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#example-specification-customer-support-agent) * [Purpose & Level](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#purpose--level) * [Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#architecture) * [Process (5-Step Loop Example)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#process-5-step-loop-example) * [Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#pattern) * [Security](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#security) * [Your Assignment](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#your-assignment) * [Requirements](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#requirements) * [Writing Tips](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#writing-tips) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/your-first-agent-concept#try-with-ai) --- # Practice: Structured Data Exercises | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've built a complete database skill from the ground up. You can design SQLAlchemy models, run CRUD operations, navigate relationships with joins, protect data with transactions, connect to cloud PostgreSQL on Neon, and verify results with hybrid bash-plus-SQL patterns. That's real capability, but knowing the workflows and executing them under pressure are different things. The gap between understanding a database concept and applying it to a broken model with cascading bugs, missing constraints, and silent data corruption is where most people stall. These 15 exercises close the gap between understanding and fluency. Each module gives you two exercises: a **Build** exercise where you create something from scratch, and a **Debug** exercise where you diagnose what went wrong in someone else's code. Three skills run through every exercise: **database workflow execution** (applying modeling, CRUD, relationship, transaction, deployment, and verification workflows), **systematic debugging** (diagnosing errors by running tests, reading tracebacks, and comparing expected vs actual behavior), and **database system design** (combining workflows into complete applications). Every exercise uses real starter files, actual Python models, broken code with planted bugs, CSV datasets, and test suites you'll run to verify your fixes. This isn't hypothetical. By the end, you'll have practiced every database workflow on realistic, messy scenarios. Download Exercise Files **[Download Structured Data Exercises (ZIP)](https://github.com/panaversity/claude-code-structured-data-exercises/releases/latest/download/structured-data-exercises.zip) ** After downloading, unzip the file. Each exercise has its own folder with an `INSTRUCTIONS.md` and any starter files you need. If the download link doesn't work, visit the [repository releases page](https://github.com/panaversity/claude-code-structured-data-exercises/releases) directly. * * * How to Use These Exercises[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#how-to-use-these-exercises "Direct link to How to Use These Exercises") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The workflow for every exercise is the same: 1. **Open the exercise folder** from the `claude-code-structured-data-exercises/` directory 2. **Read the INSTRUCTIONS.md** inside the folder: it describes the scenario and starter files 3. **Read the walkthrough below** for context on what you're practicing and why 4. **Start Claude Code** and point it at the exercise folder 5. **Work through the exercise**: write your own prompts (use starters only if stuck) 6. **Reflect** using the questions provided: this is where the real learning happens You don't need to complete all 15 in one sitting. Work through one session at a time. Each session builds on the workflows from specific chapter lessons. * * * Tool Guide[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#tool-guide "Direct link to Tool Guide") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Claude Code**: Required for all exercises. Database work involves running Python scripts, executing SQL queries, reading test output, and diagnosing tracebacks. Claude Code handles all of this directly in the terminal. * **Cowork**: Can be used for Exercise 6.2 (analyzing tool-choice scenarios) and capstone planning where you're designing systems on paper before executing. But Claude Code is strongly preferred since every exercise involves running real code and tests. * * * Key Differences from Chapter Lessons[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#key-differences-from-chapter-lessons "Direct link to Key Differences from Chapter Lessons") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lessons 1-8, you learned each database workflow in isolation with guided walkthroughs. These exercises are different in three ways: * **No step-by-step instructions.** The exercises describe the scenario and the goal. You decide the approach, choose the queries, and handle edge cases yourself. * **Build + Debug pairing.** Every module has a Build exercise (create something from scratch) and a Debug exercise (diagnose someone else's broken code). Debugging someone else's work develops different skills than building your own: you learn to read tracebacks, run tests strategically, and trace root causes through model definitions, session management, and relationship configurations. * **Increasing independence.** Modules 1-3 provide starter prompts to scaffold your learning. Modules 4-6 remove the scaffolding. Capstones remove everything: you design the entire approach. By Module 6, you should be able to face a new database problem and instinctively reach for the right workflow without needing to review the chapter lessons. * * * The Database Development Framework[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#the-database-development-framework "Direct link to The Database Development Framework") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use this for every exercise: 1. **Model**: Design your schema: tables, columns, types, constraints, relationships 2. **Connect**: Establish a database connection (SQLite for dev, PostgreSQL for production) 3. **Operate**: Implement CRUD operations with proper session management 4. **Protect**: Wrap multi-step operations in transactions with rollback on failure 5. **Verify**: Run tests, check outputs, compare expected vs actual results 6. **Deploy**: Move from local SQLite to cloud PostgreSQL with environment-based configuration This framework applies to every data-driven application, not just these exercises. Whether you're building a budget tracker, migrating legacy CSV data, or deploying a student portal to the cloud, these six steps prevent the mistakes that turn a working prototype into a corrupted database. Notice that steps 1-2 happen before any data changes. That's intentional: most database disasters come from skipping schema design and jumping straight to writing queries. * * * Assessment Rubric[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#assessment-rubric "Direct link to Assessment Rubric") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ For each exercise, evaluate yourself on: | Criteria | Beginner (1) | Developing (2) | Proficient (3) | Advanced (4) | | --- | --- | --- | --- | --- | | **Schema Quality** | Missing constraints | Basic types, some constraints | Full constraints with foreign keys | Indexes, validation, edge-case handling | | **Operation Safety** | No error handling | Basic try/except | Proper session lifecycle and rollback | Atomic transactions with savepoints | | **Query Correctness** | Raw SQL strings | Basic ORM queries | Joins and relationship navigation | Optimized queries, N+1 prevention | | **Problem Diagnosis** | Guesses at bugs | Identifies obvious errors | Traces root cause systematically | Identifies root cause AND prevents recurrence | | **Verification Discipline** | No testing | Runs tests once | Tests after each change | Tests + edge cases + failure paths | * * * Session 1: Data Foundations Modules 1-2 cover the skills from the Models as Code and Creating & Reading Data lessons. You'll design schemas from business requirements, fix broken model definitions, implement CRUD operations, and debug session management issues. Starter prompts are provided for both modules. Module 1: Data Modeling[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-1-data-modeling "Direct link to Module 1: Data Modeling") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Translating business requirements into SQLAlchemy models with proper types, constraints, and relationships (Lesson 2: Models as Code) 1.1Exercise 1.1: Library Catalog Start ### Exercise 1.1: Library Catalog (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-11-library-catalog-build "Direct link to Exercise 1.1: Library Catalog (Build)") **The Problem:** Open the `module-1-data-modeling/exercise-1.1-library-catalog/` folder. You'll find `requirements.md`: a business requirements document describing a library's catalog system. The library tracks books, authors, and genres. Books have ISBNs, titles, publication years, and page counts. Authors have names and birth years. A book can belong to multiple genres, and an author can write multiple books. The requirements include sample data and specific constraints: ISBNs must be unique, titles cannot be null, and page counts must be positive. **Your Task:** Read `requirements.md` and create a `models.py` file with SQLAlchemy ORM models for Book, Author, and Genre. Choose appropriate column types, add constraints, define foreign keys, and handle the many-to-many relationship between books and genres. Test with an in-memory SQLite database. Verify edge cases: What happens with a duplicate ISBN? A null title? A negative page count? **What You'll Learn:** * How to translate business requirements into database schema decisions (which columns, which types, which constraints) * That choosing the right column type matters: storing ISBNs as integers loses leading zeros, storing page counts without a check constraint allows negative values * The difference between designing models on paper and testing them against real data: edge cases reveal gaps in your schema **Starter Prompt (Intentionally Vague):** > "Build a database for a library catalog." **Better Prompt (Build Toward This):** After reading `requirements.md`: "Create SQLAlchemy models for a library catalog with Book, Author, and Genre. Requirements: (1) ISBN must be unique and non-null, (2) book titles cannot be null, (3) page counts must be positive integers, (4) books and genres have a many-to-many relationship through an association table, (5) authors can write multiple books. Test with in-memory SQLite. Try inserting a duplicate ISBN and a null title to verify constraints work." **Reflection Questions:** 1. Which constraint was hardest to implement correctly? Did you discover it during design or during testing? 2. How did you handle the many-to-many relationship between books and genres? What would break if you used a simple foreign key instead? 3. If the library adds a "co-author" requirement tomorrow (two authors per book), which parts of your schema would need to change? * * * 1.2Exercise 1.2: Broken Pet Store Start ### Exercise 1.2: Broken Pet Store (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-12-broken-pet-store-debug "Direct link to Exercise 1.2: Broken Pet Store (Debug)") **The Problem:** Open the `module-1-data-modeling/exercise-1.2-broken-pet-store/` folder. You'll find `broken_models.py`: model definitions for a pet store system with Owner, Pet, and Vet models, and `test_models.py`: a test suite that currently fails. The models have 6 bugs: an import error, a wrong attribute name, a wrong column type, a missing foreign key reference, a missing unique constraint, and a missing nullability setting. **Your Task:** Run the tests. Read the failure messages. Fix the bugs one at a time, re-running tests after each fix. Document each bug: what the test expected, what the model had wrong, and what you changed. The goal isn't just passing tests; it's understanding WHY each bug caused the specific failure it did. **What You'll Learn:** * How to read SQLAlchemy error messages and map them back to specific model definition problems * That model bugs cascade: a wrong column type causes one test failure, but a missing foreign key reference can cause multiple tests to fail for different reasons * The discipline of fixing one bug at a time and re-testing: fixing multiple bugs at once makes it impossible to know which fix resolved which failure **Starter Prompt (Intentionally Vague):** > "Fix the models so the tests pass." **Better Prompt (Build Toward This):** After running the tests and seeing failures: "Run `test_models.py` and show me all failures. For the first failure, trace the error to the specific line in `broken_models.py`. Fix only that one bug, then re-run the tests. Repeat for each remaining failure. After all tests pass, list every bug you fixed with: (1) the test that caught it, (2) the broken code, (3) the fix, (4) why the original code was wrong." **Reflection Questions:** 1. Which bug was hardest to find from the error message alone? Did the traceback point directly to the problem, or did you have to reason about it? 2. Did fixing one bug reveal another that was previously hidden? Why does this happen with model definitions? 3. If you had written these models yourself, which of the 6 bugs would you be most likely to make? How would you catch it? * * * Module 2: CRUD Operations[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-2-crud-operations "Direct link to Module 2: CRUD Operations") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Implementing create, search, import, and aggregate operations with proper session management (Lesson 3: Creating & Reading Data) 2.1Exercise 2.1: Recipe Book Start ### Exercise 2.1: Recipe Book (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-21-recipe-book-build "Direct link to Exercise 2.1: Recipe Book (Build)") **The Problem:** Open the `module-2-crud-operations/exercise-2.1-recipe-book/` folder. You'll find `models.py`: a correct Recipe model already defined, and `recipes.csv`: a dataset of 55 recipes with names, cuisines, prep times, difficulty levels, and ingredient counts. You need to build four CRUD functions: create a recipe, search with dynamic filters (by cuisine, difficulty, max prep time), import all 55 recipes from CSV, and compute aggregate statistics (average prep time per cuisine, most common difficulty level). **Your Task:** Implement all four functions using the provided model. The search function must handle any combination of filters: cuisine only, difficulty only, cuisine plus max prep time, or all three together. The CSV import must handle the full 55-row file without duplicates. The stats function must return meaningful aggregations. Test each function and verify the outputs. **What You'll Learn:** * How to build dynamic query filters that compose: adding `.filter()` calls conditionally based on which parameters are provided * That CSV import is never as simple as "read and insert": you need to handle type conversions, missing values, and duplicate detection * The difference between returning a Query object and returning actual results: a subtle bug that causes "no results" when there should be data **Starter Prompt (Intentionally Vague):** > "Build CRUD operations for the recipe database." **Better Prompt (Build Toward This):** After reviewing `models.py` and `recipes.csv`: "Implement four functions for the recipe database: (1) `create_recipe(session, name, cuisine, prep_time, difficulty, ingredients)` that adds a recipe and returns it, (2) `search_recipes(session, cuisine=None, difficulty=None, max_prep_time=None)` that filters dynamically based on which parameters are provided, (3) `import_csv(session, filepath)` that loads all 55 recipes from CSV with duplicate prevention, (4) `get_stats(session)` that returns average prep time per cuisine and the most common difficulty. Test each function and show outputs." **Reflection Questions:** 1. How did you handle the dynamic filter composition in the search function? What happens if no filters are provided: should it return all recipes or none? 2. What edge cases did you encounter during CSV import? Were there any rows that needed special handling? 3. How did you compute the aggregate statistics? Did you use Python-side grouping or SQL GROUP BY? What's the tradeoff? * * * 2.2Exercise 2.2: Broken Task Manager Start ### Exercise 2.2: Broken Task Manager (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-22-broken-task-manager-debug "Direct link to Exercise 2.2: Broken Task Manager (Debug)") **The Problem:** Open the `module-2-crud-operations/exercise-2.2-broken-task-manager/` folder. You'll find `broken_crud.py` (CRUD functions for a task manager with 5 bugs) `models.py`: correct model definitions, and `test_crud.py`: a test suite that catches all 5 bugs. The bugs are: a missing commit after insert, a wrong filter operator (`=` instead of `==`), a session lifecycle issue, missing error handling on invalid input, and a function that returns a Query object instead of results. **Your Task:** Run the tests. Read each failure carefully. Fix bugs one at a time, re-testing after each fix. The bugs are in the CRUD functions, not the models: the models are correct. Pay attention to session management: when sessions are opened, committed, and closed matters. **What You'll Learn:** * That the five most common CRUD bugs (missing commit, wrong operator, session leak, no error handling, Query vs results) appear in nearly every beginner's database code * How to distinguish a model bug from a CRUD bug: both can cause "no data returned," but the fix is completely different * That returning a Query object instead of `.all()` results is a silent failure: it looks like it works but returns nothing useful **Starter Prompt (Intentionally Vague):** > "The task manager CRUD functions are broken. Fix them." **Better Prompt (Build Toward This):** After running the tests: "Run `test_crud.py` against `broken_crud.py` (models from `models.py` are correct: don't modify them). Show me all test failures. For each failure: (1) which test failed, (2) what the test expected, (3) what actually happened, (4) the root cause in `broken_crud.py`, (5) the fix. Fix one bug at a time and re-run tests after each fix. Pay special attention to session commits, filter operators, and return types." **Reflection Questions:** 1. Which bug caused the most confusing error message? Did the traceback point to the CRUD function or to the test that called it? 2. The "Query object instead of results" bug is especially tricky because the code runs without errors. How would you catch this bug if there were no tests? 3. After seeing these 5 common bugs, which ones do you think you'd make most often in your own code? What habit would prevent each one? * * * Session 2: Data Architecture Modules 3-4 cover the skills from the Relationships & Joins and Transactions & Atomicity lessons. You'll add relationships to existing models, debug cascade and back-reference issues, implement atomic transactions, and fix race conditions in banking operations. Starter prompts are provided for Module 3 only. Module 3: Relationships[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-3-relationships "Direct link to Module 3: Relationships") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Defining bidirectional relationships with proper cascades and navigation (Lesson 4: Relationships & Joins) 3.1Exercise 3.1: Music Library Start ### Exercise 3.1: Music Library (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-31-music-library-build "Direct link to Exercise 3.1: Music Library (Build)") **The Problem:** Open the `module-3-relationships/exercise-3.1-music-library/` folder. You'll find `models_no_relationships.py`: Artist, Album, and Track models that have foreign keys defined but NO relationship() definitions, and `sample_data.csv` with 10 artists, 30 albums, and 100+ tracks. The foreign keys exist (Album has `artist_id`, Track has `album_id`), but without `relationship()` definitions you can't navigate from an artist to their albums or from an album to its tracks using Python attributes. **Your Task:** Add `relationship()` definitions to all three models with proper `back_populates` for bidirectional navigation. Configure cascade delete so removing an artist also removes their albums and tracks. Load the sample data, then write queries that navigate the relationships: all albums by a specific artist, all tracks on a specific album, which artist a track belongs to (navigating child-to-parent). Test cascade delete by removing one artist and verifying their albums and tracks are also gone. **What You'll Learn:** * The difference between having foreign keys (database-level links) and having relationships (Python-level navigation): foreign keys alone won't let you write `artist.albums` * How `back_populates` creates bidirectional navigation and why mismatched names cause silent failures * That cascade delete is powerful but dangerous: deleting an artist removes all their albums and every track on those albums: you must understand the cascade chain before enabling it **Starter Prompt (Intentionally Vague):** > "Add relationships to these music library models." **Better Prompt (Build Toward This):** After reading `models_no_relationships.py` and the sample CSV: "Add `relationship()` definitions to Artist, Album, and Track models. Requirements: (1) Artist.albums and Album.artist for bidirectional navigation, (2) Album.tracks and Track.album for bidirectional navigation, (3) cascade='all, delete-orphan' on parent-to-child relationships, (4) back\_populates on both sides. Load sample\_data.csv, then demonstrate: all albums by artist 'X', all tracks on album 'Y', navigating from a track back to its artist. Finally, delete one artist and verify cascade removed their albums and tracks." **Reflection Questions:** 1. What happens if you set `back_populates` on one side but not the other? Did you discover this through an error or through testing? 2. How many total records were deleted when you cascaded a single artist deletion? Was this more or fewer than you expected? 3. If the music library later needs a "featured\_tracks" relationship (tracks can be featured on multiple albums), how would you modify the schema? * * * 3.2Exercise 3.2: Broken Blog Start ### Exercise 3.2: Broken Blog (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-32-broken-blog-debug "Direct link to Exercise 3.2: Broken Blog (Debug)") **The Problem:** Open the `module-3-relationships/exercise-3.2-broken-blog/` folder. You'll find `broken_blog.py`: a blogging platform with User, Post, and Comment models that has 5 relationship bugs, and `test_relationships.py`: tests that verify navigation and cascade behavior. The bugs are: a `back_populates` mismatch, a missing `delete-orphan` cascade, a wrong model name in a relationship, a wrong table name in a foreign key, and a missing `__tablename__` attribute. **Your Task:** Run the tests. The failures will range from clear (`NoReferencedTableError`) to subtle (navigation returning empty lists). Fix each bug, re-test, and document what was wrong and why it caused that specific failure. The `__tablename__` bug is particularly tricky: missing it causes SQLAlchemy to generate a table name that doesn't match the foreign key reference. **What You'll Learn:** * How relationship misconfigurations produce different failure modes: some crash immediately, some silently return empty results, some only fail during cascade operations * That `__tablename__` is not optional decoration: it determines the actual table name in the database and must match foreign key references exactly * The debugging skill of tracing a "no results" problem backward: Does the relationship exist? Is `back_populates` correct? Does the foreign key point to the right table? **Starter Prompt (Intentionally Vague):** > "Fix the blog models so all the relationship tests pass." **Better Prompt (Build Toward This):** After running the tests: "Run `test_relationships.py` against `broken_blog.py`. Show me all failures. I expect 5 relationship bugs: (1) a back\_populates mismatch, (2) a missing cascade setting, (3) a wrong model name, (4) a wrong table name in a FK, (5) a missing **tablename**. For each: show the failing test, trace to the exact line in `broken_blog.py`, explain why it's wrong, and fix it. Re-run tests after each fix." **Reflection Questions:** 1. Which bug produced the most misleading error message? Did the error point to the relationship definition or somewhere else entirely? 2. The `__tablename__` bug is easy to miss because SQLAlchemy auto-generates table names. When would you explicitly set `__tablename__` vs letting SQLAlchemy generate it? 3. If you were reviewing someone's pull request and they added a new relationship, what three things would you check before approving? * * * Module 4: Transactions[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-4-transactions "Direct link to Module 4: Transactions") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Wrapping multi-step operations in atomic transactions that succeed completely or fail completely (Lesson 5: Transactions & Atomicity) 4.1Exercise 4.1: Game Inventory Trading Start ### Exercise 4.1: Game Inventory Trading (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-41-game-inventory-trading-build "Direct link to Exercise 4.1: Game Inventory Trading (Build)") **The Problem:** Open the `module-4-transactions/exercise-4.1-game-inventory/` folder. You'll find `models.py`: Player and Item models for a game inventory system. Players have gold balances and own items. You need to implement three functions: `trade_items()` for swapping items and gold between two players, `buy_from_shop()` for purchasing items with gold, and `batch_trade()` for executing multiple trades as a single atomic operation using savepoints. **Your Task:** Every trade must be fully atomic: the item transfers AND the gold exchange must both succeed, or neither changes. If a player doesn't have enough gold, the entire trade rolls back. The `batch_trade()` function must use savepoints so a single failed trade in the batch doesn't undo the successful ones. Test with scenarios: successful trade, insufficient gold, item not owned, and a batch where the third of five trades fails. **What You'll Learn:** * That atomicity isn't just about try/except: you need to ensure the database state is consistent even when operations partially complete * How savepoints create nested transaction boundaries that let you roll back one operation without losing others * The difference between "the code didn't crash" and "the data is correct": a trade can complete without errors but still leave inconsistent balances **Reflection Questions:** 1. What happens if you commit after transferring the item but before transferring the gold, and then the gold transfer fails? How does your atomic wrapper prevent this? 2. In `batch_trade()`, how many savepoints did you use? What would happen without savepoints when the third trade fails? 3. How would you test that your atomic trade actually rolls back? What evidence proves the rollback worked? * * * 4.2Exercise 4.2: Broken Bank Start ### Exercise 4.2: Broken Bank (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-42-broken-bank-debug "Direct link to Exercise 4.2: Broken Bank (Debug)") **The Problem:** Open the `module-4-transactions/exercise-4.2-broken-bank/` folder. You'll find `broken_bank.py`: banking operations with 5 transaction safety holes, and `test_bank.py`: tests that expose each vulnerability. The bugs are: a partial commit that causes money to vanish, a missing balance check that allows negative balances, a missing rollback that leaves the database in an inconsistent state, a delete operation that removes an account without transferring its balance, and an operation that should be wrapped in a transaction but isn't. **Your Task:** Run the tests. Each failure represents a real-world data integrity vulnerability: the kind of bug that causes actual financial discrepancies in production. Fix each bug and verify the fix. For the "money vanishes" bug, trace exactly where the commit happens and why it leaves the database inconsistent. **What You'll Learn:** * That transaction bugs are the most dangerous database bugs because they corrupt data silently: the application reports success while money disappears * How to trace the "commit point" through code to find where partial writes become permanent * That every multi-step operation involving money, inventory, or any countable resource must be wrapped in a single transaction: there are no exceptions **Reflection Questions:** 1. The "money vanishes" bug is the most dangerous. How much money would be lost if 1,000 users triggered this bug? What makes it hard to detect in production? 2. Which fix required the most thought: the missing rollback, the missing balance check, or the unwrapped operation? Why? 3. If you were building a real banking application, what additional safety measures would you add beyond transactions? * * * Session 3: Production & Verification Modules 5-6 cover the skills from the Connecting to Neon and Hybrid Patterns lessons. You'll deploy a local app to cloud PostgreSQL, diagnose connection failures, build hybrid verification pipelines, and analyze tool-choice decisions. No starter prompts are provided: you design your own approach. Module 5: Cloud Deployment[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-5-cloud-deployment "Direct link to Module 5: Cloud Deployment") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Migrating local SQLite applications to cloud PostgreSQL with proper configuration (Lesson 6: Connecting to Neon) 5.1Exercise 5.1: Contact Book Deploy Start ### Exercise 5.1: Contact Book Deploy (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-51-contact-book-deploy-build "Direct link to Exercise 5.1: Contact Book Deploy (Build)") **The Problem:** Open the `module-5-cloud-deployment/exercise-5.1-contact-book-deploy/` folder. You'll find `contact_book.py`: a working contact management app using SQLite, and `deployment_checklist.md`: a step-by-step checklist for cloud migration. The app works perfectly with local SQLite. Your job is to configure it for Neon PostgreSQL: add environment variable management for the database URL, configure connection pooling, add a health check endpoint, and verify the deployment. **Your Task:** Follow the deployment checklist. Modify the app to read `DATABASE_URL` from environment variables instead of using a hardcoded SQLite path. Add connection pooling configuration appropriate for Neon's free tier. Add a health check that runs `SELECT 1` and returns the connection status. Deploy to your Neon database and verify all existing CRUD operations still work. Document any differences between SQLite and PostgreSQL behavior you encounter. **What You'll Learn:** * That switching from SQLite to PostgreSQL isn't just changing the connection string: data types, auto-increment behavior, and connection management all differ * How environment variable management prevents credentials from leaking into source code * Why connection pooling matters for cloud databases: without it, each operation opens a new connection, which exhausts Neon's connection limit quickly **Reflection Questions:** 1. What differences between SQLite and PostgreSQL did you encounter during migration? Which one surprised you most? 2. Why is connection pooling critical for Neon specifically? What happens if you skip it? 3. If your `DATABASE_URL` environment variable is missing, what should your app do: crash immediately or fall back to SQLite? What are the tradeoffs? * * * 5.2Exercise 5.2: Connection Doctor Start ### Exercise 5.2: Connection Doctor (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-52-connection-doctor-debug "Direct link to Exercise 5.2: Connection Doctor (Debug)") **The Problem:** Open the `module-5-cloud-deployment/exercise-5.2-connection-doctor/` folder. You'll find `error_scenarios.md`: five Neon connection failure scenarios, each with the exact error message a developer would see. The scenarios include: a wrong password, an expired connection, an SSL certificate error, a connection pool exhaustion, and a DNS resolution failure. **Your Task:** For each of the 5 scenarios: identify the root cause from the error message, write the exact fix, and describe a prevention strategy that stops the error from recurring. You're not running code: you're practicing diagnostic reasoning. The skill is reading a database error message and knowing immediately what's wrong and how to fix it. **What You'll Learn:** * How to read PostgreSQL connection error messages and map them to specific configuration problems * That the five most common connection failures (auth, timeout, SSL, pool exhaustion, DNS) have distinct error signatures you can learn to recognize * Prevention strategies that stop connection errors before they happen: connection pool sizing, keep-alive settings, SSL configuration, and credential rotation **Reflection Questions:** 1. Which error message was most misleading? Did it point to the actual problem or to a symptom of a deeper issue? 2. Which prevention strategy would have the highest impact across all five scenarios? 3. If you saw a new error message you've never seen before, what diagnostic steps would you follow? * * * Module 6: Hybrid Verification[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-6-hybrid-verification "Direct link to Module 6: Hybrid Verification") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Using multiple tools (SQL + bash + Python) to verify data integrity through independent cross-checks (Lesson 7: Hybrid Patterns) 6.1Exercise 6.1: Expense Audit Start ### Exercise 6.1: Expense Audit (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-61-expense-audit-build "Direct link to Exercise 6.1: Expense Audit (Build)") **The Problem:** Open the `module-6-hybrid-verification/exercise-6.1-expense-audit/` folder. You'll find `models.py`: an Expense model with categories, amounts, and dates, and `seed_data.py`: a script that populates the database with 200+ expense records across multiple categories and date ranges. Your job is to build a hybrid verification pipeline: compute expense totals two different ways and compare the results. **Your Task:** Compute total expenses per category using two independent methods: (1) SQLAlchemy `GROUP BY` query, and (2) export to CSV and compute totals with bash (`awk`). Compare the results. If they match, the data is verified. If they don't, trace the discrepancy. Then add a date-range filter and verify again. The power of hybrid verification is that bugs in one tool get caught by the other. **What You'll Learn:** * That verifying data with the same tool that produced it proves nothing: independent cross-checks using different tools catch bugs that single-tool verification misses * How to export database results to CSV and process them with bash tools for independent verification * The pattern of hybrid verification: compute with Tool A, compute with Tool B, compare results: this pattern applies to any data integrity check **Reflection Questions:** 1. Did your SQL totals and bash totals match on the first try? If not, what caused the discrepancy? 2. Why is it important to use a DIFFERENT tool for verification rather than running the same SQL query twice? 3. Where else could you apply hybrid verification? Think of a scenario outside databases where computing the same result two different ways would catch bugs. * * * 6.2Exercise 6.2: Wrong Tool Start ### Exercise 6.2: Wrong Tool (Debug/Analysis)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-62-wrong-tool-debuganalysis "Direct link to Exercise 6.2: Wrong Tool (Debug/Analysis)") **The Problem:** Open the `module-6-hybrid-verification/exercise-6.2-wrong-tool/` folder. You'll find `scenarios.md`: five scenarios where a developer chose the wrong tool for a data task. Each scenario describes what they tried, what went wrong, and asks you to determine: why the tool was wrong, what tool should have been used, and what decision framework would have prevented the mistake. **Your Task:** Analyze all 5 scenarios. For each: identify why the chosen tool failed, recommend the right tool (bash, Python, SQL, or a hybrid approach), and explain the reasoning. After analyzing all 5, build a decision tree that someone could use to choose the right tool for any data task. The decision tree should consider: data size, operation type (filter, transform, aggregate, join), need for persistence, and need for verification. **What You'll Learn:** * That choosing the wrong tool isn't just inefficient: it can produce wrong results (bash arithmetic on floating-point currency) or miss edge cases (Python loops on million-row datasets) * How to build a decision framework for tool selection based on data characteristics and operation requirements * The principle that simple data tasks should use simple tools (bash for counting, SQL for aggregation) while complex tasks need purpose-built tools (Python for transformation logic, SQL for joins) **Reflection Questions:** 1. Which scenario had the most surprising "right answer"? Did any scenario's best tool choice contradict your initial instinct? 2. Look at your decision tree. Does it cover all five scenarios correctly? Can you think of a sixth scenario that would break it? 3. In the Structured Data & Persistent Storage chapter, when did you use hybrid approaches vs single-tool approaches? Looking back, were all those choices correct? * * * Session 4: Capstone Projects Module 7 synthesizes all six modules into complete projects. Choose one or more. These are substantially larger than the module exercises: plan for 2-4 hours each. No prompts are provided. Module 7: Capstone Projects[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-7-capstone-projects "Direct link to Module 7: Capstone Projects") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Choose one (or more). This is where everything comes together: no starter prompts provided.** Capstones are different from the exercises above. There are no guided prompts: you design the entire approach yourself. Each project requires applying all six database workflows together to solve a realistic problem. Where module exercises test individual skills, capstones test your ability to orchestrate those skills into a coherent application. The quality of your code and verification matters as much as getting it to work: someone should be able to review your project and understand your design decisions. capstone-ACapstone A: Student Grade Portal Start ### Capstone A: Student Grade Portal[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#capstone-a-student-grade-portal "Direct link to Capstone A: Student Grade Portal") Open the `module-7-capstone/capstone-A-student-portal/` folder. You'll find `requirements.md`: specifications for a student grade management system. Build a complete application: design models (Student, Course, Enrollment with grades), implement CRUD operations, configure relationships with proper cascades, protect grade updates with transactions, compute GPA calculations atomically, and deploy to Neon. This capstone uses every skill from the chapter: modeling (Module 1), CRUD (Module 2), relationships (Module 3), transactions (Module 4), cloud deployment (Module 5), and verification (Module 6). The requirements specify 20+ students and 10+ courses, so your queries must handle real data volumes, not toy examples. **What You'll Learn:** * How all six database workflows connect into a single application: each skill's output feeds the next skill's input * That GPA calculation requires atomic transactions: updating a grade must recalculate the GPA in the same transaction, or a failure leaves the GPA inconsistent * The discipline of building from schema to deployment: model first, then CRUD, then relationships, then transactions, then deploy: skipping steps creates cascading problems * * * capstone-BCapstone B: CSV Migration Start ### Capstone B: CSV Migration[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#capstone-b-csv-migration "Direct link to Capstone B: CSV Migration") Open the `module-7-capstone/capstone-B-csv-migration/` folder. You'll find `sales_data.csv`: 500 rows of flat sales data with customer names, product names, sales rep names, quantities, prices, and dates, all in a single denormalized table. Normalize this data into a proper relational schema: Customer, Product, SalesRep, and Sale tables with proper foreign keys. Then write 5 analytical queries that the flat CSV couldn't answer: top customers by total spend, sales trends by month, product performance comparisons, sales rep rankings, and customer retention patterns. The normalization decisions are the hard part. The CSV has "John Smith" appearing 15 times: is that one customer or multiple people with the same name? Product names have slight variations ("Widget Pro" vs "Widget PRO"): are those the same product? You'll need data-cleaning rules before you can normalize. **What You'll Learn:** * That data normalization is a judgment call, not a formula, real data has ambiguities that require human decisions about identity, deduplication, and edge cases * How relational databases unlock analytical queries that flat files can't support: joins, aggregations across tables, and filtered groupings * The workflow of legacy migration: assess data quality, design target schema, write cleaning rules, transform and load, verify row counts, run analytical queries * * * capstone-CCapstone C: Disaster Recovery Start ### Capstone C: Disaster Recovery[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#capstone-c-disaster-recovery "Direct link to Capstone C: Disaster Recovery") Open the `module-7-capstone/capstone-C-disaster-recovery/` folder. You'll find `broken_budget_tracker.py`: a budget tracking application with 8+ bugs across all database layers, and `test_budget_tracker.py`: a comprehensive test suite. The bugs span models (wrong types, missing constraints), CRUD (session management errors), relationships (cascade misconfigurations), and transactions (missing atomicity). This is the most demanding exercise. You must triage all bugs, determine the fix order (some fixes depend on others), repair each one, and write a postmortem documenting: what was broken, the root cause, the fix, and how to prevent each class of bug in the future. The test suite verifies your fixes, but passing all tests isn't enough: your postmortem must demonstrate understanding. **What You'll Learn:** * How to triage multiple bugs across different layers and determine the optimal fix order: model bugs must be fixed before CRUD bugs that depend on the models * That disaster recovery under pressure requires methodical discipline: triaging before fixing, fixing one thing at a time, and testing after every change * The value of a postmortem: documenting root causes and prevention strategies turns a disaster into an organizational learning opportunity * * * What's Next[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#whats-next "Direct link to What's Next") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've practiced all six database development workflows across 15 exercises: modeling, CRUD operations, relationships, transactions, cloud deployment, and hybrid verification. More importantly, you've practiced them on realistic scenarios where bugs cascade, edge cases matter, and getting the code to run is only half the job. The Database Development Framework you've internalized (Model, Connect, Operate, Protect, Verify, Deploy) transfers to any data-driven application you build with General Agents. Whether you're creating a budget tracker, migrating legacy data, or deploying a student portal, the same six steps prevent the same classes of mistakes. These patterns become the foundation for the more complex agent workflows in later chapters, where the database skills you practiced here combine with file processing, computation, and API integration to build complete AI-powered applications. Checking access... * [How to Use These Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#how-to-use-these-exercises) * [Tool Guide](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#tool-guide) * [Key Differences from Chapter Lessons](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#key-differences-from-chapter-lessons) * [The Database Development Framework](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#the-database-development-framework) * [Assessment Rubric](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#assessment-rubric) * [Module 1: Data Modeling](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-1-data-modeling) * [Exercise 1.1: Library Catalog (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-11-library-catalog-build) * [Exercise 1.2: Broken Pet Store (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-12-broken-pet-store-debug) * [Module 2: CRUD Operations](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-2-crud-operations) * [Exercise 2.1: Recipe Book (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-21-recipe-book-build) * [Exercise 2.2: Broken Task Manager (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-22-broken-task-manager-debug) * [Module 3: Relationships](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-3-relationships) * [Exercise 3.1: Music Library (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-31-music-library-build) * [Exercise 3.2: Broken Blog (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-32-broken-blog-debug) * [Module 4: Transactions](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-4-transactions) * [Exercise 4.1: Game Inventory Trading (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-41-game-inventory-trading-build) * [Exercise 4.2: Broken Bank (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-42-broken-bank-debug) * [Module 5: Cloud Deployment](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-5-cloud-deployment) * [Exercise 5.1: Contact Book Deploy (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-51-contact-book-deploy-build) * [Exercise 5.2: Connection Doctor (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-52-connection-doctor-debug) * [Module 6: Hybrid Verification](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-6-hybrid-verification) * [Exercise 6.1: Expense Audit (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-61-expense-audit-build) * [Exercise 6.2: Wrong Tool (Debug/Analysis)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#exercise-62-wrong-tool-debuganalysis) * [Module 7: Capstone Projects](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#module-7-capstone-projects) * [Capstone A: Student Grade Portal](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#capstone-a-student-grade-portal) * [Capstone B: CSV Migration](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#capstone-b-csv-migration) * [Capstone C: Disaster Recovery](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#capstone-c-disaster-recovery) * [What's Next](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/structured-data-exercises#whats-next) --- # Chapter 63: Building Custom Agents with Google ADK | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C You've built agents with OpenAI's SDK in Chapter 62. Now you'll learn Google's Agent Development Kit (ADK)—a framework designed for Gemini models with deep integration into Google Cloud's ecosystem. ADK powers Google's own Agentspace and Customer Engagement Suite, making it a production-proven choice for enterprise agents. Google ADK emphasizes **declarative agent definition**, **strong typing**, and **native multimodal support**. Where OpenAI's SDK focuses on simplicity and handoffs, ADK provides richer orchestration primitives including workflow agents (Sequential, Parallel, Loop) and comprehensive callback systems for safety and control. This chapter builds a complete AI News Podcast Agent—from a simple search agent to a multi-agent system that researches news, enriches it with financial data, and generates audio podcasts. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#what-youll-learn "Direct link to What You'll Learn") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ By the end of this chapter, you'll be able to: * **Build ADK agents**: Create agents with the Agent class, configure models, and write effective instructions * **Create custom tools**: Build function tools with type annotations and docstrings that ADK auto-wraps * **Manage state**: Use ToolContext and SessionService for persistent conversations * **Implement callbacks**: Add guardrails with before/after callbacks for safety and control * **Orchestrate multi-agent systems**: Use AgentTool to compose agents and coordinate complex workflows * **Use workflow agents**: Build deterministic pipelines with SequentialAgent, ParallelAgent, and LoopAgent * **Deploy to production**: Configure Vertex AI and understand deployment options Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#chapter-structure "Direct link to Chapter Structure") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. **Your First ADK Agent** — Installation, Agent class, google\_search tool, adk commands (run, web) 2. **Custom Function Tools** — Type annotations, docstrings, external API integration (yfinance example) 3. **Session State & Memory** — ToolContext, state management, conversation persistence 4. **Coordinator Patterns** — Background processing, file persistence, coordinator-dispatcher architecture 5. **Callbacks & Guardrails** — before\_tool\_callback, after\_tool\_callback, domain filtering, response enhancement 6. **Multi-Agent Orchestration** — AgentTool, Pydantic schemas for structured output, agent delegation 7. **Workflow Agents** — SequentialAgent, ParallelAgent, LoopAgent for deterministic pipelines 8. **Capstone: AI News Podcast Agent** — Complete multi-agent system with research, enrichment, and audio generation Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------------------- * Chapter 61: Introduction to AI Agents (conceptual foundation) * Chapter 62: OpenAI Agents SDK (comparison baseline) * Part 4: Python Fundamentals (async/await, type hints) * Google API Key (free tier) or Google Cloud account with Vertex AI access Key Differences from OpenAI SDK[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#key-differences-from-openai-sdk "Direct link to Key Differences from OpenAI SDK") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Feature | OpenAI SDK | Google ADK | | --- | --- | --- | | Agent routing | Handoffs (flexible) | Workflow agents (deterministic) + LLM routing | | Tool definition | Pydantic models | Function with docstring (auto-wrapped) | | State management | context\_variables | ToolContext + SessionService | | Safety | Input/output guardrails | 6 callback types (before/after for agent, tool, model) | | Deployment | Self-hosted | Vertex AI Agent Engine (managed) | | Multimodal | Via API | Native Gemini support (live voice, images) | Running Example: AI News Podcast Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#running-example-ai-news-podcast-agent "Direct link to Running Example: AI News Podcast Agent") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Throughout this chapter, you'll build an increasingly sophisticated news agent: 1. **Lesson 1**: Simple agent that searches for AI news 2. **Lesson 2**: Add financial data tool (yfinance) 3. **Lesson 3**: Persist conversation state 4. **Lesson 4**: Save research to markdown files 5. **Lesson 5**: Add callback-based guardrails 6. **Lesson 6**: Delegate to podcaster agent 7. **Lesson 7**: Use workflow agents for pipelines 8. **Lesson 8**: Complete podcast generation system * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#what-youll-learn) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#chapter-structure) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#prerequisites) * [Key Differences from OpenAI SDK](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#key-differences-from-openai-sdk) * [Running Example: AI News Podcast Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents#running-example-ai-news-podcast-agent) --- # Chapter 75: Augmented Memory for Agentic Applications | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Memory is what transforms a chatbot into a relationship. When your customer says "I mentioned this last week," a stateless system has to ask them to repeat themselves. A system with memory remembers—and that transforms every interaction from transactional to personal. This chapter teaches you how to add persistent memory to AI agents. You'll understand why agents need memory, explore architecture patterns inspired by cognitive science, and implement production-ready memory systems using Mem0. By the end, you'll apply these patterns to your own Claude Code agent with claude-mem. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#what-youll-learn "Direct link to What You'll Learn") -------------------------------------------------------------------------------------------------------------------------------------------------------------- | Lesson | Topic | Duration | | --- | --- | --- | | 01 | Why Agents Need Memory | 20 min | | 02 | Memory Architecture Patterns | 30 min | | 03 | What to Remember and What to Forget | 25 min | | 04 | Memory Retrieval Strategies | 25 min | | 05 | Context Window Management | 25 min | | 06 | Implementing Memory with Mem0 | 35 min | | 07 | Memory-Augmented Agent Patterns | 30 min | | 08 | Building a Memory-Augmented Agent | 40 min | | 09 | Memory for Claude Code | 35 min | **Total Duration**: ~265 minutes Learning Path[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#learning-path "Direct link to Learning Path") --------------------------------------------------------------------------------------------------------------------------------------------------- ┌─────────────────────────────────────────────────────────────────┐│ CONCEPTUAL FOUNDATION (L01-L05) ││ ││ Why Memory? → Architecture → Prioritization → Retrieval → ││ Context Management ││ ││ Build mental models before writing code │└─────────────────────────────────────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────────┐│ PRACTICAL IMPLEMENTATION (L06-L08) ││ ││ Mem0 Integration → Agent Patterns → Complete Agent Build ││ ││ Apply concepts: FastAPI + OpenAI Agents SDK + Mem0 │└─────────────────────────────────────────────────────────────────┘ │ ▼┌─────────────────────────────────────────────────────────────────┐│ PERSONAL APPLICATION (L09) ││ ││ Your Agent Gets Memory ││ ││ claude-mem for Claude Code - your agent remembers YOU │└─────────────────────────────────────────────────────────────────┘ Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#prerequisites "Direct link to Prerequisites") --------------------------------------------------------------------------------------------------------------------------------------------------- Before starting this chapter, ensure you've completed: * **Chapter 61**: Introduction to AI Agents (agent fundamentals) * **Chapters 62-65**: Agent SDK implementations (OpenAI, Google ADK, Claude) * **Chapter 70**: FastAPI for Agents (Task API you'll enhance with memory) You should be comfortable with: * Python 3.11+ development * Basic FastAPI patterns * Agent tool calling concepts * OpenAI API usage (for Mem0 defaults) Running Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#running-example "Direct link to Running Example") --------------------------------------------------------------------------------------------------------------------------------------------------------- Throughout this chapter, you'll enhance the **Task API agent** from Chapter 70 with memory capabilities: * **User Preferences**: Remember preferred task times, priority weightings * **Pattern Recognition**: Learn how users structure and complete tasks * **Context Resolution**: "The project" resolves to most recent active project * **Personalized Recommendations**: Suggest based on historical patterns Technical Requirements[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#technical-requirements "Direct link to Technical Requirements") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ # Memory librarypip install mem0ai# API key for Mem0 (uses OpenAI by default)export OPENAI_API_KEY="your-key" Key Technologies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#key-technologies "Direct link to Key Technologies") ------------------------------------------------------------------------------------------------------------------------------------------------------------ | Technology | Purpose | Lesson | | --- | --- | --- | | **Mem0** | Open-source memory layer | L06-L08 | | **OpenAI Agents SDK** | Complete agent with memory tools | L08 | | **claude-mem** | Claude Code memory plugin | L09 | | **Letta/MemGPT** | Reference architecture (conceptual) | L02 | Safety First[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#safety-first "Direct link to Safety First") ------------------------------------------------------------------------------------------------------------------------------------------------ Memory systems handle sensitive user data. Throughout this chapter, you'll learn: * Privacy requirements (GDPR, CCPA compliance) * User consent patterns * Right to be forgotten implementation * PII handling best practices Let's begin by understanding why agents need memory at all. * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#what-youll-learn) * [Learning Path](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#learning-path) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#prerequisites) * [Running Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#running-example) * [Technical Requirements](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#technical-requirements) * [Key Technologies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#key-technologies) * [Safety First](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory#safety-first) --- # Chapter 66: Model Context Protocol (MCP) Fundamentals | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Every AI application needs to connect to external systems—databases, file systems, project trackers, knowledge bases. Without a standard protocol, you'd build custom integrations for each combination: Claude + GitHub, ChatGPT + GitHub, Cursor + GitHub... the work multiplies unsustainably. MCP (Model Context Protocol) solves this integration explosion. It's the **USB-C of AI applications**: one protocol that connects any AI host to any external service. Write an MCP server once, and it works with Claude, ChatGPT, Cursor, VS Code, and every other MCP-compatible application instantly. Introduced by Anthropic in November 2024, adopted by OpenAI in March 2025, and donated to the Linux Foundation's Agentic AI Foundation in December 2025, MCP has evolved from one company's solution to industry infrastructure. Claude Code, Cursor, ChatGPT, Gemini, VS Code, and dozens of other tools already speak MCP. When you add an MCP server to your environment, every MCP-compatible agent gains those capabilities—no code changes required. This chapter teaches MCP from first principles. You'll understand the protocol architecture, learn to use existing MCP servers effectively, and prepare for Chapter 67 where you'll build your own. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#what-youll-learn "Direct link to What You'll Learn") -------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * **Understand MCP architecture**: Grasp the Host-Client-Server model, transport layers (stdio, Streamable HTTP), and the three primitives (tools, resources, prompts) * **Configure MCP servers**: Set up MCP servers in Claude Code, Claude Desktop, Cursor, and other clients using JSON configuration * **Use tools effectively**: Understand tool schemas, invoke tools correctly, and handle tool results * **Access resources**: Read files, database records, and API data through MCP's resource abstraction * **Leverage prompts**: Use server-provided prompt templates that encode domain expertise * **Debug MCP connections**: Diagnose connection issues, trace message flow, and resolve common problems Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#chapter-structure "Direct link to Chapter Structure") --------------------------------------------------------------------------------------------------------------------------------------------------------------- | # | Lesson | Duration | Description | | --- | --- | --- | --- | | 1 | [MCP Architecture Overview](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview) | 14 min | The integration explosion problem, Host-Client-Server model, and protocol design | | 2 | [Transport Layers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers) | 15 min | stdio for local servers, Streamable HTTP for remote, HTTP fundamentals primer, and when to use each | | 3 | [Tools: The Model-Controlled Primitive](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/tools-the-model-controlled-primitive) | 14 min | Executable functions that LLMs invoke to perform actions | | 4 | [Resources: The App-Controlled Primitive](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/resources-the-app-controlled-primitive) | 12 min | Read-only data sources that provide context to AI | | 5 | [Prompts: The User-Controlled Primitive](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive) | 12 min | Pre-crafted instruction templates encoding domain expertise | | 6 | [Configuring MCP Clients](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients) | 14 min | Setup in Claude Code, Claude Desktop, Cursor, VS Code, and programmatic clients | | 7 | [Using Community MCP Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers) | 15 min | Filesystem, GitHub, databases, and other popular servers | | 8 | [Debugging and Troubleshooting](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting) | 12 min | MCP Inspector, connection diagnostics, and common error patterns | | 9 | [Chapter Quiz](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/chapter-quiz) | 15 min | Test your understanding of MCP fundamentals | **Total Chapter Duration**: ~2 hours 5 min Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#prerequisites "Direct link to Prerequisites") --------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapters 62-65**: Agent SDK experience (understanding of tool use in OpenAI, Claude, and Google SDKs) * **Chapter 3**: Claude Code mastery (you've used MCP without knowing it) * **Part 4**: Python Fundamentals (for understanding server implementations) Key Concepts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#key-concepts "Direct link to Key Concepts") ------------------------------------------------------------------------------------------------------------------------------------------------ ### The Three Primitives[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#the-three-primitives "Direct link to The Three Primitives") | Primitive | Controller | Purpose | Example | | --- | --- | --- | --- | | **Tools** | Model-controlled | Perform actions | `github_create_issue`, `read_file` | | **Resources** | App-controlled | Read data | `docs://documents/{id}`, `db://users` | | **Prompts** | User-controlled | Instruction templates | `summarize_document`, `code_review` | ### Transport Options[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#transport-options "Direct link to Transport Options") | Transport | Best For | Clients | Complexity | | --- | --- | --- | --- | | **stdio** | Local development, desktop apps | Single | Low | | **Streamable HTTP** | Production, cloud deployment | Multiple | Medium | What's Next[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#whats-next "Direct link to What's Next") -------------------------------------------------------------------------------------------------------------------------------------------- After completing this chapter, you'll be ready for: * **Chapter 67: MCP Server Development** — Build your own MCP servers to expose your tools and data * **Chapter 68: Code Execution with MCP** — Execute code safely within MCP servers Resources[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#resources "Direct link to Resources") --------------------------------------------------------------------------------------------------------------------------------------- * [Official MCP Specification](https://modelcontextprotocol.io/specification/2025-06-18) * [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) * [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk) * [Community MCP Servers](https://github.com/modelcontextprotocol/servers) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#what-youll-learn) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#chapter-structure) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#prerequisites) * [Key Concepts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#key-concepts) * [The Three Primitives](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#the-three-primitives) * [Transport Options](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#transport-options) * [What's Next](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#whats-next) * [Resources](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals#resources) --- # Chapter Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/quiz#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of the Apps SDK concepts covered in this chapter. The quiz randomly selects 15-20 questions from a pool of 50, providing immediate feedback on each answer. **Passing Score:** 80% Quiz ---- Question 1 of 15 ### What does openai/widgetDomain annotation enable? ACORS restrictionsBFullscreen mode, API allowlisting, and proper CORS handlingCRate limitingDTool discovery Answered: 0 / 15 ← BackYou are on the first question. Cannot go back.Next →Please answer the question first to proceed to the next question. Quiz Coverage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/quiz#quiz-coverage "Direct link to Quiz Coverage") ------------------------------------------------------------------------------------------------------------------------------------------------------- This quiz covers all eight lessons in Chapter 72: | Topic Area | Questions | Lessons | | --- | --- | --- | | Architecture Concepts | 10 | Lesson 1 | | Widget Development | 15 | Lessons 2, 6 | | window.openai API | 10 | Lesson 3 | | Response Payloads | 5 | Lesson 4 | | State and Display Modes | 5 | Lesson 5 | | React & Apps SDK UI | 6 | Lesson 7 | | Production Security | 6 | Lesson 8 | | Integration and Debugging | 5 | Lessons 6, 8 | After the Quiz[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/quiz#after-the-quiz "Direct link to After the Quiz") ---------------------------------------------------------------------------------------------------------------------------------------------------------- If you scored below 80%, review the lessons where you struggled: * **Architecture questions**: Review Lesson 1 on three-layer architecture * **Widget development**: Review Lessons 2 and 6 on FastMCP setup and TaskManager * **API usage**: Review Lesson 3 on sendFollowUpMessage and callTool * **Payload design**: Review Lesson 4 on structuredContent vs \_meta * **State/display**: Review Lesson 5 on widgetState and requestDisplayMode * **React questions**: Review Lesson 7 on apps-sdk-ui and React hooks * **Security questions**: Review Lesson 8 on widgetCSP, widgetDomain, and visibility * **Debugging**: Review Lesson 8 on common issues and solutions **Next Steps**: After passing the quiz, apply these concepts by building your own ChatGPT App for your domain using the TaskManager as a template. * [Quiz Coverage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/quiz#quiz-coverage) * [After the Quiz](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/quiz#after-the-quiz) --- # Build Your Testing Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning how to test AI agents, you'll **own** a testing skill. This chapter teaches TDD (Test-Driven Development) for agent code—the deterministic tests that verify your code works correctly, runs fast, and costs nothing. By the end, you'll have a comprehensive test suite for your Task API with 80%+ coverage and zero LLM API calls during testing. But you won't learn testing patterns and then maybe build a skill later. You'll build the skill **first**, then spend the chapter improving it with every lesson. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for testing AI agent codewith pytest. I will use it to test FastAPI endpoints, mock LLM calls, andtest agent pipelines from hello world to production test suites.Use context7 skill to study official pytest-asyncio and respx documentationand then build it so no self assumed knowledge. Claude will: 1. Fetch official pytest-asyncio and respx documentation via Context7 2. Ask you clarifying questions (testing patterns, async preferences, coverage goals) 3. Create the complete skill with fixtures, mocking patterns, and templates Your skill appears at `.claude/skills/agent-tdd/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#done "Direct link to Done") ----------------------------------------------------------------------------------------------------------------------------------------------- You now own an `agent-tdd` skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. Every lesson ends with a "Reflect on Your Skill" section where you'll test your skill, identify gaps, and improve it. By the capstone, your skill will generate complete test suites from specifications. **Next: Lesson 1 — TDD Philosophy for Agents** * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill#done) --- # Build Your Apps SDK Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning OpenAI's Apps SDK—building ChatGPT Apps with interactive widgets—you'll **own** an Apps SDK skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for OpenAI Apps SDK (ChatGPTApps with widgets). I will use it to build interactive ChatGPT Apps from helloworld to professional production systems. Use context7 skill to study officialdocumentation and then build it so no self assumed knowledge. Claude will: 1. Fetch official OpenAI Apps SDK documentation via Context7 2. Ask you clarifying questions (widget types, interactivity patterns, display modes) 3. Create the complete skill with references and templates Your skill appears at `.claude/skills/openai-apps-sdk/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#done "Direct link to Done") ------------------------------------------------------------------------------------------------------------------------------------------------- You now own an Apps SDK skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — Three-Layer Architecture** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/build-your-apps-sdk-skill#done) --- # The Agentic Problem-Solving Process | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Every agent follows the same operational loop. Understanding it helps you predict agent behavior, design better prompts, and debug failures. The 5-Step Loop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#the-5-step-loop "Direct link to The 5-Step Loop") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. **Get Mission**: Understand what to accomplish 2. **Scan Scene**: Gather relevant context, identify available tools 3. **Think Through**: Reason about approach, plan steps 4. **Take Action**: Execute using tools 5. **Observe and Iterate**: Check results, repeat if needed This loop runs continuously until the agent achieves its goal or determines it cannot proceed. Example: Customer Support Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#example-customer-support-agent "Direct link to Example: Customer Support Agent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **User**: "Where is my order #12345?" ### Step 1 — Get Mission[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-1--get-mission "Direct link to Step 1 — Get Mission") The agent parses the request: Help user find order status. Success means providing accurate shipping information. ### Step 2 — Scan Scene[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-2--scan-scene "Direct link to Step 2 — Scan Scene") The agent identifies what it needs: * Tool available: `lookup_order(order_id)` * Context needed: Order details, shipping status, tracking number * Constraints: Can only access order system (not inventory, billing, etc.) ### Step 3 — Think Through[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-3--think-through "Direct link to Step 3 — Think Through") The agent plans: 1. Call order lookup with ID "12345" 2. If shipped, extract tracking number 3. If not shipped, explain expected date 4. Provide clear response to user ### Step 4 — Take Action[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-4--take-action "Direct link to Step 4 — Take Action") lookup_order("12345") → { status: "shipped", tracking: "ZYX987", carrier: "UPS", expected_delivery: "Nov 29"} ### Step 5 — Observe and Iterate[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-5--observe-and-iterate "Direct link to Step 5 — Observe and Iterate") * Order found? Yes * Status clear? Yes (shipped with tracking) * Mission complete? Yes—can answer user **Response**: "Your order #12345 shipped via UPS. Tracking number: ZYX987. Expected delivery: November 29." If the order wasn't found, the agent would iterate—perhaps ask for email verification or suggest the user check their order confirmation email. Example: Claude Code Debugging[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#example-claude-code-debugging "Direct link to Example: Claude Code Debugging") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **User**: "Fix the failing test in auth.py" ### Step 1 — Get Mission[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-1--get-mission-1 "Direct link to Step 1 — Get Mission") Understand which test fails and make it pass. ### Step 2 — Scan Scene[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-2--scan-scene-1 "Direct link to Step 2 — Scan Scene") * Read test file to see the failing test * Read implementation to understand current code * Run test to capture exact error message * Check related files if needed ### Step 3 — Think Through[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-3--think-through-1 "Direct link to Step 3 — Think Through") Analyze the error. "Test expects `User` object but gets `None`. The `authenticate()` function returns `None` when user doesn't exist instead of raising an exception. The test expects the function to raise `AuthenticationError`." Plan the fix: 1. Modify `authenticate()` to raise `AuthenticationError` instead of returning `None` 2. Run test to verify ### Step 4 — Take Action[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-4--take-action-1 "Direct link to Step 4 — Take Action") Edit `auth.py` to add the exception. Run `pytest test_auth.py`. ### Step 5 — Observe and Iterate[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-5--observe-and-iterate-1 "Direct link to Step 5 — Observe and Iterate") * Test passes? If yes, mission complete. * Test still fails? Back to Step 2 with new information. Read the new error, reason about what else might be wrong, try another fix. The loop continues until the test passes or the agent determines it needs human input. Context Engineering[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#context-engineering "Direct link to Context Engineering") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **The critical insight**: Agent quality depends on context quality. Context engineering means actively selecting relevant information for each step. Not everything the agent _could_ access—just what it _needs_. ### Poor Context[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#poor-context "Direct link to Poor Context") Give the agent access to everything: order system, inventory, accounting, scheduling, HR, maintenance databases. The agent gets overwhelmed filtering through irrelevant data. Response time suffers. Accuracy drops. ### Good Context[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#good-context "Direct link to Good Context") Give the agent access to the order system only. It finds what it needs immediately. Fast, focused, accurate. **More tools ≠ better agent. Focused context = better agent.** ### Why This Matters for You[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#why-this-matters-for-you "Direct link to Why This Matters for You") When you work with Claude Code: * Specific error messages > "it doesn't work" * Pointing to the relevant file > "somewhere in the codebase" * Clear success criteria > vague goals When Claude Code asks clarifying questions, it's engineering better context. Help it by providing focused, relevant information. ### Context Engineering Decisions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#context-engineering-decisions "Direct link to Context Engineering Decisions") Good context engineering asks: * What information does the agent actually need for this step? * What would distract or confuse it? * How can I narrow the search space? A customer support agent doesn't need access to the HR database. A code review agent doesn't need access to the billing system. Constraints improve performance. Debugging with the Loop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#debugging-with-the-loop "Direct link to Debugging with the Loop") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When an agent fails, identify which step broke: | Failure Mode | Step | Diagnosis | Fix | | --- | --- | --- | --- | | Wrong goal pursued | Get Mission | Agent misunderstood request | Clarify the request | | Missing information | Scan Scene | Needed context unavailable | Provide context or add tools | | Bad reasoning | Think Through | Flawed logic or planning | Better prompting or different model | | Tool error | Take Action | Tool failed or returned bad data | Fix tool or check permissions | | Didn't recognize done | Observe | Agent kept going or stopped early | Clarify success criteria | ### Debugging Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#debugging-example "Direct link to Debugging Example") You ask Claude Code to "add logging to the API." It adds logging to database queries instead. **Which step failed?** Step 1 (Get Mission). "API" was ambiguous—did you mean the HTTP API? The internal API? The database layer? **Fix**: Clarify the request: "Add request/response logging to the HTTP API endpoints in `api/routes.py`." ### Another Debugging Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#another-debugging-example "Direct link to Another Debugging Example") Claude Code reads the wrong file and proposes changes that don't make sense. **Which step failed?** Step 2 (Scan Scene). It gathered wrong context. **Fix**: Point it to the right file: "The authentication logic is in `src/auth/handlers.py`, not `src/auth/models.py`." The Loop is Universal[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#the-loop-is-universal "Direct link to The Loop is Universal") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This 5-step loop appears everywhere: **Self-driving cars**: Understand destination → sense environment → plan route → execute maneuver → check position → repeat **Game-playing AI**: Understand winning → perceive game state → plan moves → execute action → observe result → repeat **Your own problem-solving**: Understand goal → gather information → plan approach → take action → check results → adjust Once you internalize the loop, you can analyze any agent system. Master it once, apply it everywhere. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to trace through the loop. > "A coffee shop agent receives: 'medium iced vanilla latte, extra shot, oat milk.' Walk through the 5-Step Loop. What context does it need at each step?" **Expected**: Mission = make drink correctly. Context = recipe database, inventory status, cup sizes. Plan = preparation sequence. Action = make drink. Observe = verify each component added. > "Compare two customer service agents. Agent A accesses: orders, inventory, accounting, HR, maintenance. Agent B accesses: orders only. Which handles 'where's my order?' better? Why?" **Expected**: Agent B—focused context means faster, more accurate responses. Agent A wastes cycles filtering irrelevant systems. > "Claude Code tried to fix a bug but made it worse. The test that was failing now passes, but three other tests fail. Which step of the loop broke? How would you diagnose?" **Expected**: Step 5 (Observe) or Step 2 (Scan Scene). Either it didn't check all tests, or it didn't gather enough context about related code. Fix by asking it to run the full test suite before declaring success. **Key insight**: The loop is universal. When agents fail, trace through the steps to find where things went wrong. Checking access... * [The 5-Step Loop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#the-5-step-loop) * [Example: Customer Support Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#example-customer-support-agent) * [Step 1 — Get Mission](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-1--get-mission) * [Step 2 — Scan Scene](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-2--scan-scene) * [Step 3 — Think Through](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-3--think-through) * [Step 4 — Take Action](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-4--take-action) * [Step 5 — Observe and Iterate](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-5--observe-and-iterate) * [Example: Claude Code Debugging](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#example-claude-code-debugging) * [Step 1 — Get Mission](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-1--get-mission-1) * [Step 2 — Scan Scene](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-2--scan-scene-1) * [Step 3 — Think Through](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-3--think-through-1) * [Step 4 — Take Action](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-4--take-action-1) * [Step 5 — Observe and Iterate](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#step-5--observe-and-iterate-1) * [Context Engineering](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#context-engineering) * [Poor Context](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#poor-context) * [Good Context](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#good-context) * [Why This Matters for You](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#why-this-matters-for-you) * [Context Engineering Decisions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#context-engineering-decisions) * [Debugging with the Loop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#debugging-with-the-loop) * [Debugging Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#debugging-example) * [Another Debugging Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#another-debugging-example) * [The Loop is Universal](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#the-loop-is-universal) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agentic-problem-solving-process#try-with-ai) --- # Hybrid Patterns - When Tools Work Together | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 6, you deployed your models to Neon and proved a cloud connection works reliably. Now you face a question that sounds philosophical but is deeply practical: How do you KNOW your query is correct? Not "it ran without errors" correct. Actually correct. Returning the right numbers. Here is a question that keeps financial engineers awake at night. A SQL query returns $247.50 for January food expenses. The query is valid. PostgreSQL executed it without complaint. But did it apply the right date boundary? Did it filter the right user? Did the import process normalize categories the same way the query expects? You might be thinking: "My SQL is correct. Why check it twice?" Good question. You would not check it twice -- you would check it with a DIFFERENT tool. That is the key insight of this lesson. Remember those Braintrust/Vercel numbers from the chapter opening? SQL: 100% accuracy. That is impressive. But "usually right" and "provably right for THIS report" are different claims. When the cost of being wrong is a bad financial filing or a regulatory violation, you want proof, not probability. Key Terms for This Lesson * **Hybrid verification**: Using two DIFFERENT tools with DIFFERENT failure modes to check the same answer -- if both agree, you can trust the result * **Independent path**: A verification route that can't fail for the same reason as the primary -- SQL bugs won't affect your CSV parser, and vice versa * **Tolerance**: The acceptable margin of difference between two paths -- for financial data, even $0.01 might matter * **Mismatch policy**: The rule that decides what happens when paths disagree -- "block release and investigate" is the safe default False Hybrid vs True Hybrid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#false-hybrid-vs-true-hybrid "Direct link to False Hybrid vs True Hybrid") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the most important distinction in the lesson. Re-running SQL to "double check" is like proofreading your essay by reading it to yourself again -- you will miss the same mistakes every time. False Hybrid (DON'T DO THIS): True Hybrid (DO THIS): ┌─────────────────┐ ┌──────────────────┐ │ SQL Query │ │ SQL Query │ │ (ORM path) │ │ (ORM path) │ └────────┬─────────┘ └─────────┬──────────┘ │ │ ▼ │ ┌─────────────────┐ ┌──────────┴──────────┐ │ Same SQL Query │ │ │ │ (same path!) │ ▼ ▼ └────────┬─────────┘ ┌─────────┐ ┌──────────┐ │ │SQL Total │ │CSV Parser │ ▼ │ $247.50 │ │Raw Ledger │ Same bug = └────┬─────┘ │ $247.50 │ same wrong answer │ └─────┬─────┘ ▼ ▼ ┌──────────────────────────┐ │ Match? → Release │ │ Mismatch? → BLOCK │ └──────────────────────────┘ ❌ Confirms determinism, ✅ Different code paths, NOT correctness different failure modes Pause and Reflect Look at the false hybrid example. Why is running the same SQL query twice NOT verification? If the query has a wrong date boundary, both runs will return the same wrong number. What would you need to change to make it truly independent? ### False Hybrid (Do Not Use)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#false-hybrid-do-not-use "Direct link to False Hybrid (Do Not Use)") If you ask the agent to run the same query twice and compare the results, that is not verification: it only proves the query is deterministic. If the query has a wrong date boundary, both runs return the same wrong number. You have confirmed your bug is consistent, not that your answer is correct. The rule: re-running the same SQL query is never independent verification. ### True Hybrid (Use for High-Stakes Reports)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#true-hybrid-use-for-high-stakes-reports "Direct link to True Hybrid (Use for High-Stakes Reports)") A true hybrid uses a completely different path to arrive at the same answer. Your primary path goes through SQL: the agent queries the database. Your verification path reads the raw CSV ledger directly with a plain file reader. Different data sources. Different parsing logic. Different failure modes. If both agree, you can trust the answer. What you tell the agent I need to verify this month's Food total before releasing the report. Compute it two independent ways: 1. Query it from the database 2. Compute it by reading the raw CSV ledger file directly If the two totals differ by more than one cent, block the release and show me the mismatch. If they match, confirm release is permitted. Use exact decimal arithmetic throughout: no rounding errors. What you verify python verify_report.pyOutput (verified): SQL path: $247.50 CSV path: $247.50 ✓ Match within $0.01 tolerance Status: verified — release permittedOutput (mismatch): SQL path: $247.50 CSV path: $253.10 ✗ Mismatch: $5.60 delta exceeds $0.01 tolerance Status: BLOCKED — investigate before releasing When you see BLOCKED, you do not ship. The gate worked correctly. Publishing after a mismatch is a release process failure, not a query bug. The Independence Checklist[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#the-independence-checklist "Direct link to The Independence Checklist") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A hybrid verification system is only as good as its independence. If any item on this list fails, do not call the workflow hybrid verification -- you are just running the same logic with extra steps: 1. **Data source differs** -- DB vs raw ledger export 2. **Parsing path differs** -- ORM query vs CSV parser 3. **Scope matches exactly** -- `user_id`, date window, category semantics 4. **Numeric type stays decimal-safe end-to-end** -- no float rounding surprises 5. **Mismatch policy has a mandatory release decision** -- not a warning, a gate Notice item 3 carefully. Independence means different failure modes, not different scopes. If SQL path is user-scoped but CSV path is all-users, your comparison is invalid by construction. The paths must answer the same question through different means. (Parenthetical honesty: for most day-to-day queries -- exploratory analysis, dashboard widgets, internal summaries -- hybrid verification is overkill. SQL-only is the right default. Reserve hybrid for outputs where being wrong has real consequences: financial filings, audit reports, compliance artifacts.) Alternative Domains: Where Hybrid Matters Most[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#alternative-domains-where-hybrid-matters-most "Direct link to Alternative Domains: Where Hybrid Matters Most") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This pattern is not unique to budget trackers. Any domain where the cost of a wrong answer is high benefits from independent verification: **Healthcare dosage calculations.** When a drug dose calculation is wrong, patients suffer. A hospital pharmacy system might compute dosages through its primary formulary engine, then verify with an independent calculation using the raw weight-based formula. Two paths, different code, same expected answer. A mismatch triggers a pharmacist review before the prescription is filled. **Structural engineering load calculations.** Two independent methods to verify a bridge can hold the weight. One uses finite element analysis software. The other applies classical beam theory calculations. If they agree within tolerance, the design is approved. If they diverge, engineers investigate before construction begins. The principle is universal: when the cost of being wrong exceeds the cost of checking twice, hybrid verification pays for itself. Minimal Policy[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#minimal-policy "Direct link to Minimal Policy") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When do you use hybrid, and when is SQL enough? * **Low-stakes exploration** -> SQL-only (dashboards, ad-hoc queries, internal summaries) * **Financial or audit output** -> hybrid + mismatch gate (anything that gets signed, filed, or published) One Common Failure[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#one-common-failure "Direct link to One Common Failure") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Calling CSV checks "independent" while silently changing scope. If SQL path is user-scoped but CSV path is all-users, comparison is invalid by construction. Another failure is using naive CSV parsing logic (`split(',')`) and then trusting totals. Quoted fields make this unsafe -- a field containing "Coffee, large" would split into two columns and corrupt your sum silently. Release Decision Template[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#release-decision-template "Direct link to Release Decision Template") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the protocol your system follows when the verification gate runs: * `verified` -> publish permitted * `mismatch` -> publish blocked * `blocked + reason documented` -> incident triage begins * `triage complete + evidence updated` -> re-run gate before release ### Common Triage Questions After Mismatch[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#common-triage-questions-after-mismatch "Direct link to Common Triage Questions After Mismatch") When paths disagree, resist the urge to immediately edit query logic. Answer these questions first: 1. Did SQL and raw paths use identical user scope? 2. Did both paths apply identical date boundaries? 3. Were categories normalized consistently during import? 4. Did raw file include malformed or missing rows? 5. Did decimal parsing differ between paths? Answer these before editing query logic. Most mismatches are scope and normalization issues, not SQL engine failures. **What breaks next?** You now have all primitives. The capstone is where you prove they work together without hidden contradictions. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Independence Audit[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#prompt-1-independence-audit "Direct link to Prompt 1: Independence Audit") Audit this hybrid verification design.Reject any step that reuses the same logic path.Confirm scope parity (user_id, date window, category filter) across SQL and raw paths. **What you're learning:** You are practicing the discipline of questioning independence claims. Real verification requires paths with genuinely different failure modes -- same scope, different implementations. This skill matters every time you hear someone say "we double-checked" and you need to ask "with what?" ### Prompt 2: Mismatch Policy[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#prompt-2-mismatch-policy "Direct link to Prompt 2: Mismatch Policy") Design a release policy for high-stakes financial reports:- block condition- alert recipients- required evidence artifacts- unblock criteria **What you're learning:** Detecting a mismatch is only half the problem. The other half is what your organization does about it. A good mismatch policy turns a scary discrepancy into a structured triage process with clear escalation, documentation requirements, and re-verification steps. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") Think of a high-stakes output in your domain -- a financial report, a safety calculation, a compliance audit. Design a hybrid verification system:1. What's the primary computation path?2. What's a truly independent second path?3. What tolerance makes sense for your domain?4. What's your mismatch policy?Explain why your two paths have different failure modes. **What you're learning:** Hybrid verification is a universal quality pattern. Whether you are building financial software, medical systems, or engineering tools -- any time the COST of being wrong is high, checking with a second independent method is worth the overhead. This skill transfers far beyond databases. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#checkpoint "Direct link to Checkpoint") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I can explain why re-running the same SQL query is NOT independent verification. * [ ] I can describe a true hybrid system in plain English: two paths, different data sources, different failure modes, same expected answer. * [ ] I directed the agent to build a verification gate and read the mismatch output. * [ ] I know the release decision rule: verified → permitted, mismatch → blocked. * [ ] I can name one scenario in my domain where hybrid verification is worth the overhead. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#flashcards-study-aid "Direct link to Flashcards Study Aid") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [False Hybrid vs True Hybrid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#false-hybrid-vs-true-hybrid) * [False Hybrid (Do Not Use)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#false-hybrid-do-not-use) * [True Hybrid (Use for High-Stakes Reports)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#true-hybrid-use-for-high-stakes-reports) * [The Independence Checklist](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#the-independence-checklist) * [Alternative Domains: Where Hybrid Matters Most](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#alternative-domains-where-hybrid-matters-most) * [Minimal Policy](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#minimal-policy) * [One Common Failure](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#one-common-failure) * [Release Decision Template](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#release-decision-template) * [Common Triage Questions After Mismatch](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#common-triage-questions-after-mismatch) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#try-with-ai) * [Prompt 1: Independence Audit](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#prompt-1-independence-audit) * [Prompt 2: Mismatch Policy](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#prompt-2-mismatch-policy) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#prompt-3-apply-to-your-domain) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/hybrid-patterns#flashcards-study-aid) --- # Models as Code | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 1, you proved that data survives a process restart. Now you need to answer a harder question: what shape should that data take, and who enforces the rules? Here is a fact that surprises most people: computers cannot store 0.1 + 0.2 as exactly 0.3. They store it as 0.30000000000000004. That tiny rounding error does not sound like much until you process a million transactions. At one hundred-thousandth of a cent per transaction, you are off by ten dollars. At a billion transactions, you are off by ten thousand. Banks do not tolerate "close enough," and neither should your budget tracker. This is why when you tell the agent to store money amounts, you must be explicit: "store amounts as exact dollars and cents, no rounding." Your job in this lesson is not to write code. Your job is to describe your data clearly enough that the agent builds the right schema on the first try. You are the director. The agent is the builder. If the builder gets it wrong, the description was not clear enough. Key Terms for This Lesson * **Schema**: A formal blueprint that defines every table, every field, and every rule your database enforces. Think of it as the floor plan for your data -- rooms (tables), doors (connections between tables), and locks (rules that reject bad data). * **Constraint**: A rule the database enforces automatically. When you say "this field is always required," the database becomes the bouncer at the door -- it rejects any data that violates the rule, no exceptions. * **Exact decimal**: When you tell the agent to store money as "exact dollars and cents," the database stores 0.30, not 0.30000000000000004. Contrast with "approximate number," which can drift by tiny fractions that compound over millions of transactions. * **Foreign key**: A reference from one entity to another. When you say "each expense must reference an existing user," the database guarantees you cannot create an orphaned expense -- every expense is tied to a real person. The Approximate Number Problem (And Its Fix)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#the-approximate-number-problem-and-its-fix "Direct link to The Approximate Number Problem (And Its Fix)") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You might think: "Can't the computer just round the numbers?" It can. But rounding becomes your responsibility. Every calculation, every sum, every comparison -- someone has to remember to round. Miss one spot and your totals drift. With exact decimal storage, the database handles precision for you. Every value stored, every calculation performed, exact to the penny. No discipline required. Here is the difference in plain terms: * **Approximate number** (the default for most computers): 0.1 + 0.2 = 0.30000000000000004. Comparisons break. Totals drift. * **Exact decimal** (what you tell the agent to use for money): 0.1 + 0.2 = 0.30. Comparisons work. Totals stay accurate. This is why when you describe money fields to the agent, you must say "exact dollars and cents, no rounding" -- not just "a number." The difference prevents errors that compound across millions of transactions. The Schema Contract[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#the-schema-contract "Direct link to The Schema Contract") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A schema is not documentation. It is a contract your database enforces on every insert, every update, every reference. If the data violates the contract, it gets rejected -- no exceptions, no "I forgot to check in the application." Here is what our budget tracker needs: Entity-Relationship Diagram:┌──────────────┐ ┌──────────────┐│ User │ │ Category │├──────────────┤ ├──────────────┤│ id (PK) │ │ id (PK) ││ email (UQ) │ │ name (UQ) ││ name │ │ color ││ created_at │ └──────┬───────┘└──────┬───────┘ │ │ │ │ 1:many │ 1:many │ │ ▼ ▼┌──────────────────────────────────┐│ Expense │├──────────────────────────────────┤│ id (PK) ││ user_id (FK → users.id) ││ category_id (FK → categories.id)││ description ││ amount: exact decimal ││ date ││ created_at │└──────────────────────────────────┘PK = Primary Key, FK = Foreign Key (reference), UQ = Unique (no duplicates) Three entities. Two references. One money column that uses exact decimals. This is the shape of every expense that enters your system. Pause and Reflect Look at the Expense entity. Why does it reference the user's ID instead of storing the user's name directly? What would go wrong if Alice changed her email and you had her name stored in 500 expense rows? How to Describe Your Data to the Agent[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#how-to-describe-your-data-to-the-agent "Direct link to How to Describe Your Data to the Agent") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the core skill of this lesson. When you tell an agent to build a schema, vague descriptions produce wrong results. Precise descriptions produce correct schemas on the first try. Here is your vocabulary guide: | Business meaning | How to describe it to the agent | | --- | --- | | Money / price | "exact dollars and cents, no rounding" | | Required field | "this field is always required" | | Unique field | "no duplicates allowed" | | Link to another entity | "must reference an existing \[entity\]" | | Timestamp | "record when this was created, UTC" | | Optional field | "this field can be empty" | | Default value | "if not provided, default to \[value\]" | The pattern is consistent: describe what the data **means to the business**, not what the code should look like. The agent translates your business meaning into the right technical implementation. Describing the Budget Tracker Schema[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#describing-the-budget-tracker-schema "Direct link to Describing the Budget Tracker Schema") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here is what a complete, precise description looks like in practice. What you tell the agent I need to track users, categories, and expenses for a personal budget tracker. **User**: email (required, no duplicates allowed), name (required), record when created in UTC. **Category**: name (required, no duplicates allowed), display color (optional, default to red). **Expense**: must reference an existing user (required), must reference an existing category (required), description (required), amount in exact dollars and cents (required), date (required), record when created in UTC. Build the database schema and create the tables. Notice what this description does NOT contain: no programming terms, no type names, no syntax. It describes what the data means and what rules the database should enforce. The agent handles the rest. What the agent builds and what you verify The agent creates the schema. You run the verification command: python verify_schema.py You see: ✓ Table: users (id, email, name, created_at) - email: required, no duplicates - name: required✓ Table: categories (id, name, color) - name: required, no duplicates - color: optional, defaults to red✓ Table: expenses (id, user_id, category_id, description, amount, date, created_at) - user_id: must reference a real user - category_id: must reference a real category - amount: exact decimal (10 digits, 2 decimal places) - description: required - date: required✓ All constraints activeSchema ready. Every line in that output maps back to something you described. If something is missing or wrong, refine your description and ask the agent to rebuild. How to Describe Common Field Types[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#how-to-describe-common-field-types "Direct link to How to Describe Common Field Types") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you encounter a new field, use this guide to tell the agent what you need: | Business meaning | How to describe it | Why this matters | | --- | --- | --- | | Money / price | "exact dollars and cents, no rounding" | Prevents drift across millions of transactions | | Required field | "this field is always required" | Database rejects missing values -- no silent gaps | | Unique field | "no duplicates allowed" | Database enforces uniqueness -- no duplicate emails | | Link to another entity | "must reference an existing \[entity\]" | Database rejects orphaned references | | Event timestamp | "record when created, UTC timezone" | Consistent ordering and incident tracing | | Optional with default | "optional, default to \[value\]" | Sensible fallback when no value is provided | The entities change from project to project. The descriptions stay the same. Money is always "exact dollars and cents." References are always "must reference an existing \[entity\]." Required fields are always "this field is always required." Schema Contract Checklist[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#schema-contract-checklist "Direct link to Schema Contract Checklist") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before you tell the agent to build, ask yourself these questions: * "Have I specified which fields are required vs optional?" * "Have I identified money fields and specified 'exact decimal'?" * "Have I described the links between entities ('must reference a real X')?" * "Have I specified which fields must be unique ('no duplicates allowed')?" * "Have I described timestamp behavior ('record when created, UTC')?" If you can answer yes to all five, your description is ready. If not, refine it before handing it to the agent. A vague description produces a wrong schema, and fixing a wrong schema after data is in it is much harder than getting it right the first time. **What comes next?** With the schema in place, the next risk is writing and reading data correctly. Lesson 3 tackles how to create and read data without leaving the database in a half-written state. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Describe Your Own Data Model[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#prompt-1-describe-your-own-data-model "Direct link to Prompt 1: Describe Your Own Data Model") I need to track [your entity — for example: recipes, workouts, invoices].Here is what I know about it: [describe fields in plain English].Help me describe it precisely enough for an agent to build a correctdatabase schema. Ask me clarifying questions about required fields,unique fields, and links between entities. **What you're learning:** The gap between "I need to track recipes" and a precise data description is where most schema bugs hide. By practicing this translation with AI asking clarifying questions, you build the habit of specifying constraints before they become bugs. ### Prompt 2: Review My Description for Completeness[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#prompt-2-review-my-description-for-completeness "Direct link to Prompt 2: Review My Description for Completeness") Review my data description for completeness:[paste your plain-English description here]Flag any field where the type is ambiguous, any relationship I havenot specified clearly, and any constraint that is missing. Tell mewhat questions a database would ask if it could talk. **What you're learning:** Reviewing descriptions for gaps is how you catch "I forgot to say unique" before the agent builds a schema that allows duplicate emails. This verification skill transfers to every project where you direct an agent to build something. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#checkpoint "Direct link to Checkpoint") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * [ ] I can describe a data model to an agent in plain English, including required/optional fields and links between entities. * [ ] I can explain why money amounts need "exact decimal" storage, not just "a number." * [ ] I can describe unique constraints without writing code ("no duplicates allowed"). * [ ] I verified the agent built what I described by reading the schema output. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#flashcards-study-aid "Direct link to Flashcards Study Aid") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Checking access... * [The Approximate Number Problem (And Its Fix)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#the-approximate-number-problem-and-its-fix) * [The Schema Contract](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#the-schema-contract) * [How to Describe Your Data to the Agent](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#how-to-describe-your-data-to-the-agent) * [Describing the Budget Tracker Schema](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#describing-the-budget-tracker-schema) * [How to Describe Common Field Types](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#how-to-describe-common-field-types) * [Schema Contract Checklist](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#schema-contract-checklist) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#try-with-ai) * [Prompt 1: Describe Your Own Data Model](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#prompt-1-describe-your-own-data-model) * [Prompt 2: Review My Description for Completeness](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#prompt-2-review-my-description-for-completeness) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/models-as-code#flashcards-study-aid) --- # Practice: File Processing Exercises | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've learned six powerful file processing workflows. You can survey folders, create safety backups, design organization rules, run batch operations, recover from mistakes, and search by description. That's real capability, but knowing the workflows and executing them under pressure are different things. The gap between understanding a workflow and applying it to 80 messy files with wrong extensions, duplicate names, and missing metadata is where most people stall. These 13 exercises close the gap between understanding and fluency. Each module gives you two exercises: a **Build** exercise where you apply a workflow to real files, and a **Debug** exercise where you diagnose what went wrong when someone else did it poorly. Three skills run through every exercise: **file workflow execution** (applying survey, backup, organization, batch, recovery, and search workflows), **systematic debugging** (diagnosing errors by comparing expected vs. actual file state), and **pipeline design** (combining workflows into end-to-end systems). Every exercise uses real starter files, actual folders with 20-100 files you'll survey, backup, organize, rename, recover, and search. This isn't hypothetical. By the end, you'll have practiced every workflow on messy, realistic data. Download Exercise Files **[Download File Processing Exercises (ZIP)](https://github.com/panaversity/claude-code-file-processing-exercises/releases/latest/download/file-processing-exercises.zip) ** After downloading, unzip the file. Each exercise has its own folder with an `INSTRUCTIONS.md` and starter files you need. If the download link doesn't work, visit the [repository releases page](https://github.com/panaversity/claude-code-file-processing-exercises/releases) directly. * * * How to Use These Exercises[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#how-to-use-these-exercises "Direct link to How to Use These Exercises") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The workflow for every exercise is the same: 1. **Open the exercise folder** from the `claude-code-file-processing-exercises/` directory 2. **Read the INSTRUCTIONS.md** inside the folder: it describes the scenario and starter files 3. **Read the walkthrough below** for context on what you're practicing and why 4. **Start Claude Code** and point it at the exercise folder 5. **Work through the exercise**: write your own prompts (use starters only if stuck) 6. **Reflect** using the questions provided: this is where the real learning happens You don't need to complete all 13 in one sitting. Work through one module at a time. Each module builds on the workflows from specific chapter lessons. * * * Tool Guide[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#tool-guide "Direct link to Tool Guide") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Claude Code**: Required for all exercises. File processing is terminal work: surveying directories, copying files, renaming batches, verifying results. Claude Code runs these operations directly. * **Cowork**: Can be used for Exercise 3.2 (analyzing organization rules) and capstone planning where you're designing systems on paper before executing. But Claude Code is strongly preferred since every exercise involves manipulating real files. * * * Key Differences from Chapter Lessons[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#key-differences-from-chapter-lessons "Direct link to Key Differences from Chapter Lessons") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In Lessons 1-7, you learned each workflow in isolation with guided walkthroughs. These exercises are different in three ways: * **No step-by-step instructions.** The exercises describe the scenario and the goal. You decide the approach, choose the commands, and handle edge cases yourself. * **Build + Debug pairing.** Every module has a Build exercise (apply the workflow) and a Debug exercise (diagnose someone else's mistakes). Debugging someone else's work develops different skills than doing the work yourself: you learn to read logs, compare expected vs. actual state, and trace root causes. * **Increasing independence.** Modules 1-2 provide starter prompts to scaffold your learning. Modules 3-5 remove the scaffolding. Capstones remove everything: you design the entire approach. By Module 5, you should be able to face a new file processing problem and instinctively reach for the right workflow without needing to review the chapter lessons. * * * The File Processing Framework[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#the-file-processing-framework "Direct link to The File Processing Framework") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use this for every exercise: 1. **Survey**: What files exist? How many? How big? What types? 2. **Backup**: Create a safety net before any changes 3. **Plan**: Design rules and approach before executing 4. **Test**: Try on ONE file first 5. **Execute**: Run the batch operation 6. **Verify**: Compare results against expectations 7. **Document**: Save rules, logs, and templates for reuse This framework applies to every file management task, not just these exercises. Whether you're reorganizing a photo library, cleaning up a code repository, or migrating data between systems, these seven steps prevent the mistakes that turn a 10-minute task into a 2-hour recovery operation. Notice that steps 1-3 happen before any files change. That's intentional: most file disasters come from skipping preparation. * * * Assessment Rubric[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#assessment-rubric "Direct link to Assessment Rubric") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For each exercise, evaluate yourself on: | Criteria | Beginner (1) | Developing (2) | Proficient (3) | Advanced (4) | | --- | --- | --- | --- | --- | | **Investigation Quality** | Runs `ls` once | Surveys files and sizes | Full inventory with types, sizes, dates | Discovers hidden files, symlinks, edge cases | | **Safety Practices** | No backup | Copies some files | Complete backup with verification | Timestamped backup + integrity check + rollback plan | | **Workflow Execution** | Random commands | Follows steps but skips verification | Complete workflow with verification | Adapts workflow to edge cases dynamically | | **Problem Diagnosis** | Guesses at issues | Identifies obvious problems | Traces root cause systematically | Identifies root cause AND prevents recurrence | | **Documentation** | No record | Basic notes | Reusable rules and templates | Complete system with rules + logs + recovery plan | * * * Module 1: File Investigation[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-1-file-investigation "Direct link to Module 1: File Investigation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Understanding what you have before changing anything (Lessons 1-2) 1.1The Project Handoff Start ### Exercise 1.1: The Project Handoff (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-11-the-project-handoff-build "Direct link to Exercise 1.1: The Project Handoff (Build)") **The Problem:** Open the `module-1-file-investigation/exercise-1.1-project-handoff/` folder. You'll find `messy-project/`: a folder with 40+ files across nested directories, no documentation, and a mix of source code, data files, images, and config files. A colleague just left the company and handed you this project folder with the message "figure out what's in here." No README, no explanation, no file manifest. You need to understand the project well enough to brief your team by end of day. **Your Task:** Survey the entire folder structure and create a comprehensive `FILE-INVENTORY.md` that documents everything: total file count, file types with counts, directory tree, largest files, newest and oldest files, any hidden files or dotfiles, and a summary of what this project appears to be. Someone reading your inventory should understand the folder's contents without opening a single file. **What You'll Learn:** * How to systematically survey an unfamiliar folder instead of randomly clicking through files * Which bash commands (`find`, `du`, `ls -la`, `file`, `wc`) reveal the most about a folder's contents * That a thorough 5-minute survey prevents the "I didn't know that file existed" surprises that derail workflows later **Starter Prompt (Intentionally Vague):** > "What's in this folder?" **Better Prompt (Build Toward This):** After exploring with `ls -laR`, `find . -type f | wc -l`, and `du -sh */`, try: "Survey this entire folder and create FILE-INVENTORY.md with: (1) total file count and directory count, (2) file type breakdown with counts, (3) full directory tree, (4) top 5 largest files with sizes, (5) newest and oldest files by modification date, (6) any hidden files or dotfiles, (7) a 2-sentence summary of what this project appears to be." **Reflection Questions:** 1. How many files did your initial `ls` miss that a deeper survey with `find` revealed? 2. What surprised you most about the folder's contents? Would you have discovered it without a systematic survey? 3. How long did the full survey take? Compare that to how long you'd spend recovering from a missed file causing problems mid-workflow. * * * 1.2The Lying Inventory Start ### Exercise 1.2: The Lying Inventory (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-12-the-lying-inventory-debug "Direct link to Exercise 1.2: The Lying Inventory (Debug)") **The Problem:** Open the `module-1-file-investigation/exercise-1.2-lying-inventory/` folder. You'll find two things: `wrong-inventory.md`: a file inventory someone created, and `actual-files/`: the folder it supposedly documents. The inventory has errors. Wrong file counts, missing files, incorrect sizes, missed hidden files, and at least one file listed that doesn't actually exist. **Your Task:** Systematically compare the inventory against the actual folder. Find every discrepancy. For each error, document: what the inventory claims, what's actually true, and what command proves it. Create a `CORRECTIONS.md` listing every error with evidence. **What You'll Learn:** * How to verify someone else's work by comparing claims against ground truth * That inventories go stale the moment files change: verification is not optional * The specific commands (`diff`, `comm`, `find`, `stat`) that catch discrepancies between documentation and reality **Starter Prompt (Intentionally Vague):** > "Check if this inventory is correct." **Better Prompt (Build Toward This):** After running `find actual-files/ -type f | wc -l` and comparing to the inventory's claimed count: "Compare wrong-inventory.md against the actual-files/ directory. For every discrepancy, document: (1) what the inventory claims, (2) what's actually true, (3) the bash command that proves the difference. Check file counts, file sizes, hidden files, directory structure, and any files listed in the inventory that don't exist. Output everything to CORRECTIONS.md." **Reflection Questions:** 1. How many errors did you find? Which category of error was most common (wrong counts, missing files, wrong sizes)? 2. Which error would have caused the most damage if you'd trusted the inventory and acted on it? 3. What verification steps would you add to your own inventory-creation workflow to prevent these errors? * * * Module 2: Safety-First Backup[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-2-safety-first-backup "Direct link to Module 2: Safety-First Backup") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Creating safety nets before any destructive operation (Lesson 2) 2.1The Migration Prep Start ### Exercise 2.1: The Migration Prep (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-21-the-migration-prep-build "Direct link to Exercise 2.1: The Migration Prep (Build)") **The Problem:** Open the `module-2-safety-first/exercise-2.1-migration-prep/` folder. You'll find `migration-source/`: a folder containing project files that need to be migrated to a new structure. The folder has symlinks, files with special characters in names, a mix of permissions, some large binary files, and nested directories 4 levels deep. Before you migrate anything, you need a complete, verified backup. **Your Task:** Create a comprehensive backup of `migration-source/` that handles all edge cases: symlinks (follow or preserve?), special characters, permissions, large files, and deep nesting. Verify the backup is complete by comparing file counts, total sizes, and spot-checking specific files. Document your backup strategy in `BACKUP-LOG.md`. **What You'll Learn:** * That `cp -r` alone misses symlinks, permissions, and special characters, real backups need `rsync` or careful `cp` flags * How to verify backup completeness beyond "the folder exists" (compare counts, sizes, checksums) * Why documenting your backup strategy matters: if the migration fails, you need to know exactly what you backed up and how **Starter Prompt (Intentionally Vague):** > "Back up this folder before I migrate it." **Better Prompt (Build Toward This):** After running `find migration-source/ -type l` and `find migration-source/ -name '*[[:space:]]*'` to discover edge cases: "Create a backup of migration-source/ to backup-YYYY-MM-DD/. Handle: (1) symlinks: preserve them as symlinks, don't follow, (2) files with spaces and special characters, (3) preserve file permissions, (4) nested directories to any depth. After copying, verify: file count matches, total size matches, spot-check 3 specific files by comparing checksums. Document everything in BACKUP-LOG.md." **Reflection Questions:** 1. What edge cases did you discover during the backup that a simple `cp -r` would have missed? 2. How did you verify completeness? Would your verification catch a file that was copied but corrupted? 3. If the migration fails in 3 months, could someone use your BACKUP-LOG.md to restore everything? What information would they need that you didn't include? * * * 2.2The Incomplete Backup Start ### Exercise 2.2: The Incomplete Backup (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-22-the-incomplete-backup-debug "Direct link to Exercise 2.2: The Incomplete Backup (Debug)") **The Problem:** Open the `module-2-safety-first/exercise-2.2-incomplete-backup/` folder. You'll find `original/` (the source folder with 30+ files) and `backup/` (a backup someone made). The backup is missing 6 files. Your job is to figure out why each file is missing, not just that it's missing, but what caused the omission. **Your Task:** Compare `backup/` against `original/` and identify all 6 missing files. For each one, determine the likely cause: Was it a hidden file that `cp` skipped? A symlink that wasn't followed? A file created after the backup timestamp? A file in a subdirectory that wasn't recursively copied? Document each discrepancy with its probable cause in `DIAGNOSIS.md`. **What You'll Learn:** * The six most common reasons backups are incomplete (hidden files, symlinks, permissions, timestamps, recursive depth, filename encoding) * How to use `diff <(find original/) <(find backup/)` to systematically find missing files * That diagnosing why a backup failed is more valuable than just noticing it failed: the cause tells you how to prevent it next time **Starter Prompt (Intentionally Vague):** > "Is this backup complete?" **Better Prompt (Build Toward This):** After running `diff <(cd original && find . | sort) <(cd backup && find . | sort)` to see the differences: "Compare backup/ against original/. For each missing file, determine: (1) the file path, (2) the file type (regular, hidden, symlink, etc.), (3) the probable cause it was missed (hidden file, symlink not followed, permission denied, not recursive, created after backup). Output findings to DIAGNOSIS.md with evidence commands." **Reflection Questions:** 1. Which of the 6 missing files would have caused the biggest problem if you'd relied on this backup during a migration? 2. Did any of the missing files share a common cause? What single change to the backup command would have caught the most files? 3. How would you modify your backup workflow from Exercise 2.1 to prevent every cause you discovered here? * * * Module 3: Organization Rules[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-3-organization-rules "Direct link to Module 3: Organization Rules") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Designing categorization systems through collaborative refinement (Lesson 3) 3.1The Freelancer's Chaos Start ### Exercise 3.1: The Freelancer's Chaos (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-31-the-freelancers-chaos-build "Direct link to Exercise 3.1: The Freelancer's Chaos (Build)") **The Problem:** Open the `module-3-organization-rules/exercise-3.1-freelancer-chaos/` folder. You'll find `freelancer-files/`: a flat folder with 60+ files from a freelancer's working directory. There are client deliverables mixed with personal files, invoices mixed with project assets, drafts mixed with finals, and file names that follow no consistent pattern. Some are `invoice_march.pdf`, others are `ACME_logo_v3_FINAL_FINAL.png`, others are `notes.txt`. The freelancer has 3 active clients and needs to be able to find any client's files within 30 seconds. **Your Task:** Design organization rules that sort these files into a logical folder structure. Don't just move files: first write a `rules.md` that defines your categories, naming conventions, and how to handle ambiguous files. Then apply the rules. Then verify the result: every file should be in exactly one category, no file should be lost, and someone unfamiliar with the project should understand where to find things. **What You'll Learn:** * That organization rules must handle ambiguity: what do you do with a file that fits two categories? * How iterative refinement with Claude produces better rules than trying to design the perfect system upfront * The difference between organizing for yourself (you know what "notes.txt" means) and organizing for others (nobody else does) **Reflection Questions:** 1. How many files didn't fit cleanly into your initial categories? What did you do with them? 2. Did Claude suggest categories you hadn't considered? Did you push back on any of its suggestions? 3. If this freelancer adds 20 new files next month, would your rules handle them without modification? * * * 3.2The Collision Course Start ### Exercise 3.2: The Collision Course (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-32-the-collision-course-debug "Direct link to Exercise 3.2: The Collision Course (Debug)") **The Problem:** Open the `module-3-organization-rules/exercise-3.2-collision-course/` folder. You'll find `broken-rules.md`: an organization system someone designed, and `unsorted-files/`: a batch of files waiting to be organized. The rules have problems: overlapping categories (a file could go in two places), gaps (some file types aren't covered), ambiguous criteria (what counts as "important"?), and contradictory instructions. **Your Task:** Analyze `broken-rules.md` and identify every flaw: overlaps, gaps, ambiguities, and contradictions. Fix each one with a clear rationale. Then apply your corrected rules to `unsorted-files/` and verify the result. **What You'll Learn:** * How to audit organization rules for completeness and consistency * That rules which look logical on paper often fail when applied to real files with messy names and mixed purposes * The specific failure modes of categorization systems: overlap, gaps, ambiguity, and contradiction **The Twist:** After fixing the rules and organizing the files, ask Claude to generate 5 hypothetical file names that your corrected rules STILL wouldn't handle cleanly. Fix your rules again to accommodate them. This reveals whether your rules are truly robust or just happened to work on the test set. **Reflection Questions:** 1. Which flaw was hardest to spot by reading the rules alone? Did it only become obvious when you tried to apply them? 2. How many of the 5 hypothetical files exposed new gaps? Were the gaps in your categories, your naming criteria, or your ambiguity-resolution rules? 3. At what point do organization rules become too complex to follow? Where's the line between thorough and over-engineered? * * * Module 4: Batch Operations[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-4-batch-operations "Direct link to Module 4: Batch Operations") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Transforming repetitive file tasks into systematic batch workflows (Lesson 4) 4.1Photo Library Cleanup Start ### Exercise 4.1: Photo Library Cleanup (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-41-photo-library-cleanup-build "Direct link to Exercise 4.1: Photo Library Cleanup (Build)") **The Problem:** Open the `module-4-batch-operations/exercise-4.1-photo-library/` folder. You'll find `photo-dump/`: 50+ image files with inconsistent naming from three different sources: a phone camera (`IMG_20240315_142355.jpg`), a DSLR (`DSC_0042.NEF`), screenshots (`Screenshot 2024-03-15 at 2.23.55 PM.png`), and downloads (`photo.png`, `vacation (copy 2).JPEG`). File extensions are mixed case. Some files have EXIF dates, some don't. There are duplicates with slightly different names pointing to the same image. **Your Task:** Design a batch rename operation that gives every file a consistent name format: `YYYY-MM-DD_description_NNN.ext` (lowercase extension). Handle: files without dates (use file modification date), duplicates (append sequential numbers), and missing descriptions (use "unnamed"). Test on 3 files first, then run the full batch. Verify no files were lost by comparing counts before and after. **What You'll Learn:** * That batch rename is deceptively complex: edge cases (duplicates, missing metadata, encoding issues) multiply fast * Why testing on a small batch first catches problems that would be catastrophic at full scale * How to design rename rules that handle messy real-world data, not just clean test cases **Key Edge Cases to Watch For:** * Two photos taken in the same second (collision handling) * Files with no date in the name AND no EXIF data (what's the fallback?) * Mixed case extensions: `.JPG` vs `.jpg` vs `.jpeg` (are these the same type?) * Files with parentheses, spaces, or unicode characters in names **Reflection Questions:** 1. How many edge cases did you discover during the 3-file test that would have caused problems in the full batch? 2. What was your strategy for files without EXIF dates? How confident are you in the fallback? 3. If you needed to undo this rename, could you? What would you need to have saved? * * * 4.2The Rename Disaster Start ### Exercise 4.2: The Rename Disaster (Debug)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-42-the-rename-disaster-debug "Direct link to Exercise 4.2: The Rename Disaster (Debug)") **The Problem:** Open the `module-4-batch-operations/exercise-4.2-rename-disaster/` folder. You'll find `mangled-files/`: files after a botched batch rename, and `rename-log.txt`: a log of every rename operation that was executed. The rename script had bugs: some files were overwritten (duplicates mapped to the same target name), some extensions were corrupted, and some files ended up with garbled names from encoding issues. **Your Task:** Use `rename-log.txt` to reconstruct what happened. Identify every bug in the original rename logic. Determine which files can be recovered (the log has the original names) and which are permanently damaged (overwritten). Create a `RECOVERY-PLAN.md` documenting: what went wrong, which files are recoverable, and the recovery commands. **What You'll Learn:** * How rename logs enable recovery, and why every batch operation should produce one * The three most common batch rename bugs: collision (two files map to same target), encoding corruption, and extension mangling * That the cost of a botched batch operation is proportional to the number of files: which is exactly why you test on 3 first **The Challenge:** After completing the recovery plan, write a corrected rename script (or prompt) that handles all the edge cases the original missed. Test it on the recovered files to prove it works. Compare your corrected version against the original log to verify every bug is addressed. **Reflection Questions:** 1. How many files were permanently lost due to overwrites? Could any of them be recovered from elsewhere? 2. What single change to the original rename logic would have prevented the most damage? 3. If the rename log didn't exist, how would your recovery approach change? What does this tell you about the value of logging? * * * Module 5: Recovery & Search[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-5-recovery--search "Direct link to Module 5: Recovery & Search") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Core Skill:** Recovering from disasters and finding needles in haystacks (Lessons 5-6) 5.1The Accidental Flatten Start ### Exercise 5.1: The Accidental Flatten (Build)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-51-the-accidental-flatten-build "Direct link to Exercise 5.1: The Accidental Flatten (Build)") **The Problem:** Open the `module-5-recovery-and-search/exercise-5.1-accidental-flatten/` folder. You'll find two directories: `flattened/`: where all files from a nested project structure were accidentally moved into a single flat directory, and `backup/`: an older backup that preserves the original nested structure. Someone ran a command that moved every file to the root level, destroying the folder hierarchy. The flattened directory has 40+ files that used to live in subdirectories like `src/`, `tests/`, `docs/`, and `assets/`. File names are intact, but all directory context is gone. **Your Task:** Reconstruct the original nested directory structure. Use `backup/` as your reference for where files should go, but note that some files in `flattened/` are newer than the backup (they were created after the backup was made). You'll need to decide where new files belong based on their type and naming patterns. Verify the reconstruction by comparing your result against the backup's structure. **What You'll Learn:** * How to use an older backup as a reference map without blindly restoring it (newer files exist) * That recovery is a judgment call: you need rules for files that don't have an obvious home * The workflow of recovery: assess damage, identify reference, plan reconstruction, execute, verify **Reflection Questions:** 1. How many files had a clear home based on the backup? How many required judgment about where they belonged? 2. What was your strategy for new files that didn't exist in the backup? How confident are you in your placement? 3. What would you do differently if no backup existed? What other clues could you use to reconstruct the structure? * * * 5.2The Tax Season Hunt Start ### Exercise 5.2: The Tax Season Hunt (Search)[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-52-the-tax-season-hunt-search "Direct link to Exercise 5.2: The Tax Season Hunt (Search)") **The Problem:** Open the `module-5-recovery-and-search/exercise-5.2-tax-season-hunt/` folder. You'll find `document-archive/`: a folder with 100+ files accumulated over 3 years of freelance work. Somewhere in this mess are 5 specific documents you need for tax filing. You don't know the exact file names: you only have descriptions: 1. "The contract I signed with Acme Corp in Q2 2023" 2. "The receipt for the laptop I bought for work" 3. "My health insurance summary from last year" 4. "The invoice where I charged the highest amount" 5. "The spreadsheet where I tracked monthly expenses" **Your Task:** Find all 5 documents using only the descriptions above. For each one, document: the search strategy you used, the commands that narrowed it down, and the final file path. You cannot open every file: there are too many. You need to use metadata (dates, sizes, types), file names, and content search (`grep`) strategically. **What You'll Learn:** * How to translate vague human descriptions into specific search criteria (dates, keywords, file types) * That different search strategies work for different types of descriptions (date-based vs. content-based vs. size-based) * The power of combining search approaches: `find` by date range + `grep` by keyword + `file` by type **The Extension:** After finding all 5 documents, create a `SEARCH-CHEATSHEET.md` documenting the search strategies that worked. For each strategy, include: when to use it, the command pattern, and an example. This cheatsheet becomes a reusable reference for future search tasks. **Reflection Questions:** 1. Which document was hardest to find? What made the description difficult to translate into search criteria? 2. Which search strategy was most effective overall: searching by date, by name pattern, by content, or by file type? 3. If this archive grows to 1,000 files, which of your search strategies would still work? Which would break down? * * * Module 6: Capstone Projects[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-6-capstone-projects "Direct link to Module 6: Capstone Projects") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > **Choose one (or more). This is where everything comes together: no starter prompts provided.** Capstones are different from the exercises above. There are no guided prompts: you design the entire approach yourself. Each project requires applying all six workflows together to solve a realistic problem. Where module exercises test individual skills, capstones test your ability to orchestrate those skills into a coherent pipeline. The quality of your documentation matters as much as the result: someone should be able to follow your process and reproduce it on a different folder. AThe Full Pipeline Start ### Capstone A: The Full Pipeline[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#capstone-a-the-full-pipeline "Direct link to Capstone A: The Full Pipeline") Open the `module-6-capstone/capstone-A-full-pipeline/` folder. You'll find `messy-downloads/`: an 80-file folder that simulates a real Downloads directory accumulated over months. There are PDFs, images, code files, spreadsheets, documents, archives, duplicates, and files with no clear purpose. Take this folder through the complete File Processing Framework: 1. **Survey**: Create a comprehensive inventory of what's in the folder 2. **Backup**: Create a verified safety backup before touching anything 3. **Plan**: Design organization rules with categories, naming conventions, and ambiguity resolution 4. **Test**: Apply your rules to 5 files and verify the result 5. **Execute**: Organize and rename the full batch 6. **Verify**: Compare results against your plan; confirm no files lost 7. **Document**: Create a complete record: rules, logs, before/after stats **What You'll Learn:** * How all six workflows connect into a single pipeline: each step's output feeds the next step's input * That the full pipeline takes longer than any single workflow, but prevents the cascading failures that make individual steps fail * The discipline of documenting as you go: your `rules.md`, `backup-log.md`, and `verification-report.md` become a reusable template **Deliverables:** * `FILE-INVENTORY.md`: Complete survey of the 80-file folder * `backup/`: Verified safety backup with integrity checks * `rules.md`: Organization categories, naming conventions, ambiguity handling * `rename-log.txt`: Record of every rename operation * `VERIFICATION-REPORT.md`: Before/after comparison, file count confirmation, spot checks **Reflection Questions:** 1. Which step took the longest? Which step prevented the most potential problems? 2. Did any step's output force you to revise a previous step? (For example, did executing the batch reveal a gap in your rules?) 3. How would you adapt this pipeline for a different domain: say, organizing a music library or a code repository? * * * BThe Team File System Start ### Capstone B: The Team File System[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#capstone-b-the-team-file-system "Direct link to Capstone B: The Team File System") Open the `module-6-capstone/capstone-B-team-file-system/` folder. You'll find `team-scenario.md`: a description of a 3-person team (a designer, a developer, and a project manager) who share a project folder and keep stepping on each other's files. The scenario describes their pain points: overwritten designs, lost meeting notes, conflicting versions. Design a complete shared file system for this team: 1. **Folder structure**: Where does each person's work live? Where do shared assets go? 2. **Naming conventions**: How do files indicate owner, version, status (draft/review/final)? 3. **Organization rules**: What goes where? How are conflicts resolved? 4. **Backup strategy**: How often? What's backed up? Who's responsible? 5. **Recovery plan**: When someone accidentally overwrites a file, what's the procedure? 6. **Search guide**: How does each person find what they need quickly? Create the complete folder structure, write the rules document, and verify the system handles all the scenarios described in `team-scenario.md`. **What You'll Learn:** * That file systems designed for teams require explicit conflict resolution rules that personal systems don't * How to think about file management as a system design problem, not just a cleanup task * The difference between organizing for one person (intuitive) and organizing for a team (explicit) **Deliverables:** * Complete folder structure (created as actual directories) * `FILE-SYSTEM-RULES.md`: Naming conventions, ownership, version control, conflict resolution * `BACKUP-STRATEGY.md`: Schedule, scope, responsibility assignments * `RECOVERY-PROCEDURES.md`: Step-by-step instructions for common disaster scenarios * `QUICK-SEARCH-GUIDE.md`: How each team member finds what they need **Reflection Questions:** 1. Which team member's workflow was hardest to accommodate? What compromises did you make? 2. How did you handle the case where the designer and developer both need to modify the same asset file? 3. Would your system scale to a 10-person team, or would it need fundamental redesign? * * * CYour Own Files Start ### Capstone C: Your Own Files[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#capstone-c-your-own-files "Direct link to Capstone C: Your Own Files") Open the `module-6-capstone/capstone-C-your-own-files/` folder for a self-assessment template. Then close it: this capstone uses YOUR actual files. Pick a real folder on your computer: your Downloads, Desktop, Documents, or a project directory that's gotten messy. Apply all six workflows to your actual files: 1. **Survey** your real folder: how bad is it? 2. **Backup** everything before you touch it, real files, real stakes 3. **Design rules** that fit how you actually work, not how you wish you worked 4. **Test** on a handful of files first 5. **Execute** the full organization 6. **Verify** nothing was lost 7. **Document** your rules so future-you can maintain the system **What Makes This Special:** Unlike Capstones A and B, this one has real consequences. The files you're organizing are files you actually need. The rules you design must fit your real workflow. The backup must be genuinely complete because there's no "reset" button. This is where the File Processing Framework proves its value, or exposes gaps in your understanding. **Deliverables:** * `BEFORE-SNAPSHOT.md`: Survey of your folder before any changes * `backup/`: Complete safety backup of your real files * `MY-RULES.md`: Organization rules tailored to your actual workflow * `AFTER-SNAPSHOT.md`: Survey of your folder after organization * `WHAT-I-LEARNED.md`: What worked, what surprised you, what you'd do differently **Reflection Questions:** 1. Were your real files messier or cleaner than you expected? What surprised you during the survey? 2. Did any of your organization rules from the earlier exercises fail when applied to your actual files? Why? 3. Will you maintain this system going forward? What's the minimum maintenance effort to keep it organized? * * * What's Next[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#whats-next "Direct link to What's Next") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've practiced all six file processing workflows across 13 exercises: surveying, backing up, organizing, batch processing, recovering, and searching. More importantly, you've practiced them on messy, realistic data where edge cases actually matter. The File Processing Framework you've internalized (Survey, Backup, Plan, Test, Execute, Verify, Document) transfers to any domain where you direct General Agents to work with files, data, or any structured collection. Whether you're managing a photo library, migrating a codebase, or reorganizing a team's shared drive, the same seven steps prevent the same classes of mistakes. These patterns become the foundation for automated workflows in later chapters, where the manual prompting you practiced here evolves into autonomous agent behavior. The rules you wrote, the logs you kept, and the verification habits you built are exactly what AI Employees need to operate independently; they just need those patterns encoded as instructions rather than typed as prompts. Checking access... * [How to Use These Exercises](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#how-to-use-these-exercises) * [Tool Guide](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#tool-guide) * [Key Differences from Chapter Lessons](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#key-differences-from-chapter-lessons) * [The File Processing Framework](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#the-file-processing-framework) * [Assessment Rubric](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#assessment-rubric) * [Module 1: File Investigation](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-1-file-investigation) * [Exercise 1.1: The Project Handoff (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-11-the-project-handoff-build) * [Exercise 1.2: The Lying Inventory (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-12-the-lying-inventory-debug) * [Module 2: Safety-First Backup](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-2-safety-first-backup) * [Exercise 2.1: The Migration Prep (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-21-the-migration-prep-build) * [Exercise 2.2: The Incomplete Backup (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-22-the-incomplete-backup-debug) * [Module 3: Organization Rules](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-3-organization-rules) * [Exercise 3.1: The Freelancer's Chaos (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-31-the-freelancers-chaos-build) * [Exercise 3.2: The Collision Course (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-32-the-collision-course-debug) * [Module 4: Batch Operations](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-4-batch-operations) * [Exercise 4.1: Photo Library Cleanup (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-41-photo-library-cleanup-build) * [Exercise 4.2: The Rename Disaster (Debug)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-42-the-rename-disaster-debug) * [Module 5: Recovery & Search](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-5-recovery--search) * [Exercise 5.1: The Accidental Flatten (Build)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-51-the-accidental-flatten-build) * [Exercise 5.2: The Tax Season Hunt (Search)](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#exercise-52-the-tax-season-hunt-search) * [Module 6: Capstone Projects](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#module-6-capstone-projects) * [Capstone A: The Full Pipeline](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#capstone-a-the-full-pipeline) * [Capstone B: The Team File System](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#capstone-b-the-team-file-system) * [Capstone C: Your Own Files](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#capstone-c-your-own-files) * [What's Next](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/file-processing/file-processing-exercises#whats-next) --- # Build Your MCP Server Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You learned MCP fundamentals in Chapter 66. Now you'll BUILD servers. But first, you'll **own** an MCP server building skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for building MCP servers(Model Context Protocol). I will use it to build custom MCP servers fromhello world to professional production systems. Use context7 skill to studyofficial documentation and then build it so no self assumed knowledge. Claude will: 1. Fetch official MCP documentation via Context7 2. Ask you clarifying questions (Python/TypeScript, tool types, resource patterns) 3. Create the complete skill with references and templates Your skill appears at `.claude/skills/mcp-server-builder/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#done "Direct link to Done") ------------------------------------------------------------------------------------------------------------------------------------------------------ You now own an MCP server building skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — Server Architecture** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/build-your-mcp-server-skill#done) --- # Chapter 72: Apps SDK - Building Interactive ChatGPT Apps | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C You've built agents that reason, plan, and execute tasks. But they live on your server—how do you get them in front of users who need them? ChatGPT has over **800 million weekly active users**. Over **1 million businesses** use it daily, including **92% of Fortune 500 companies**. When you build a ChatGPT App using the Apps SDK, your application becomes available in the [ChatGPT App Directory](https://chatgpt.com/apps) —where users can discover it, add it to their ChatGPT, and start using it immediately. No separate website. No app store approval. No user acquisition cost. This is the distribution opportunity the Apps SDK unlocks. What the Apps SDK Does[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#what-the-apps-sdk-does "Direct link to What the Apps SDK Does") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Chapter 62 taught you to build backend agents with OpenAI's Agents SDK—intelligent systems that process requests, use tools, and hand off between specialized agents. The Apps SDK creates the _visual layer_ users see: interactive widgets that render rich UI directly inside ChatGPT conversations. The distinction matters: * **Agents SDK** creates _backends_—the intelligence behind operations * **Apps SDK** creates _frontends_—visual interfaces users interact with A complete ChatGPT App combines both: agents process requests while widgets display results with buttons, forms, and real-time updates. If the Agents SDK is the engine, the Apps SDK is the dashboard—and the dashboard is what 800 million users see. This chapter bridges your MCP server knowledge (Chapters 66-67) with widget capabilities unique to the Apps SDK. You'll build on familiar patterns—FastMCP servers, tool definitions, structured responses—while adding the visual layer that transforms tools into complete applications available to the world's largest AI user base. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#what-youll-learn "Direct link to What You'll Learn") ------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * **Explain the three-layer architecture**: Understand how ChatGPT UI, widget iframes, and MCP servers communicate to deliver interactive experiences * **Build ChatGPT Apps with FastMCP**: Create MCP servers that expose tools with widget resources, using the `text/html+skybridge` MIME type for ChatGPT rendering * **Implement widget interactivity**: Use the `window.openai` API for action buttons (`sendFollowUpMessage`), tool chaining (`callTool`), and state management * **Design response payloads**: Separate data the model sees (`structuredContent`) from data only the widget sees (`_meta`) for efficient token usage * **Manage state and display modes**: Persist user selections across tool invocations with `widgetState` and control layout with inline, pip, and fullscreen modes * **Build a complete TaskManager ChatGPT App**: Apply all patterns to create a working task management application with add, complete, and delete functionality as your capstone project Chapter Structure: Building TaskManager Progressively[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#chapter-structure-building-taskmanager-progressively "Direct link to Chapter Structure: Building TaskManager Progressively") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unlike chapters that teach concepts separately, this chapter builds **one application progressively**. Each lesson adds a single feature to your TaskManager ChatGPT App: | Lesson | What You Build | New Concept | | --- | --- | --- | | **L1: Architecture** | Mental model (no code) | Three-layer architecture, data flow | | **L2: Hello Widget** | Minimal greeting widget (~50 lines) | `text/html+skybridge`, FastMCP basics, ngrok setup | | **L3: Refresh Button** | Add "Refresh" button to widget | `sendFollowUpMessage`, API availability | | **L4: Task List** | Display tasks from server | `structuredContent` vs `_meta` separation | | **L5: Task Actions** | Add Complete/Delete buttons | `callTool`, `widgetAccessible` metadata | | **L6: State & Modes** | Persist selections, fullscreen view | `widgetState`, display modes | | **L7: React & Apps SDK UI** | Rebuild TaskManager in React | `@openai/apps-sdk-ui`, React hooks | | **L8: Capstone** | Production-ready deployment | Security metadata, debugging, deployment | | **Quiz** | Assessment | 50-question chapter assessment | **The Progressive Approach**: By Lesson 8, you haven't just learned concepts—you've built a complete, production-ready TaskManager ChatGPT App using both vanilla JS and React approaches. Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#prerequisites "Direct link to Prerequisites") -------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapter 62**: OpenAI Agents SDK (tools, handoffs, guardrails) * **Chapters 66-67**: MCP Fundamentals and Custom MCP Servers (FastMCP, tool definitions) * **Part 4**: Python Fundamentals (async/await, type hints, Pydantic) * Basic HTML, CSS, and JavaScript for widget development Tools Required[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#tools-required "Direct link to Tools Required") ----------------------------------------------------------------------------------------------------------------------------------------------------- * Python 3.11+ with uv package manager * FastMCP (`mcp[cli]>=1.9.2`) and uvicorn * ngrok for tunneling local servers to HTTPS * ChatGPT Plus or Enterprise account (Developer Mode access) Looking Ahead[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#looking-ahead "Direct link to Looking Ahead") -------------------------------------------------------------------------------------------------------------------------------------------------- This chapter gives you ChatGPT Apps with interactive widgets. Chapter 73 (Agent Testing & TDD) adds quality assurance patterns, and Chapter 74 (Evaluation Frameworks) helps you measure agent performance systematically. * [What the Apps SDK Does](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#what-the-apps-sdk-does) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#what-youll-learn) * [Chapter Structure: Building TaskManager Progressively](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#chapter-structure-building-taskmanager-progressively) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#prerequisites) * [Tools Required](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#tools-required) * [Looking Ahead](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk#looking-ahead) --- # Chapter 76: TDD for Agents | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C **Part 6: AI Native Software Development — Phase 4: Quality Assurance** Test your agent code with confidence. This chapter teaches Test-Driven Development (TDD) for AI agent applications—testing **code correctness** (deterministic, passes/fails), not LLM reasoning quality (that's Chapter 77: Evals). What You'll Build[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#what-youll-build "Direct link to What You'll Build") ------------------------------------------------------------------------------------------------------------------------------------------------------------ By the end of this chapter, you'll own a production-ready `agent-tdd` skill and a comprehensive test suite for your Task API with: * **80%+ code coverage** on your agent API * **Zero LLM API calls** during test execution * **Sub-10-second test runs** with mocked dependencies * **CI/CD automation** via GitHub Actions The Critical Distinction[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#the-critical-distinction "Direct link to The Critical Distinction") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Aspect | TDD (This Chapter) | Evals (Chapter 77) | | --- | --- | --- | | Question | Does the code work correctly? | Does the LLM reason well? | | Nature | Deterministic | Probabilistic | | Output | Pass/Fail | Scores (0-1) | | Speed | Fast (mocked LLM) | Slow (real LLM calls) | | Cost | Zero (no API calls) | High (API calls required) | Skill-First Learning[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#skill-first-learning "Direct link to Skill-First Learning") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter follows the **Skill-First pattern**: 1. **L00**: Build your `agent-tdd` skill from official documentation 2. **L01-L07**: Learn patterns that improve your skill 3. **L08**: Compose everything into a production test suite Every lesson ends with "Reflect on Your Skill"—you'll continuously test and improve the skill you own. Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#chapter-structure "Direct link to Chapter Structure") ------------------------------------------------------------------------------------------------------------------------------------------------------------- | Lesson | Title | Layer | Duration | | --- | --- | --- | --- | | L00 | Build Your Testing Skill | L1 (Manual) | 15 min | | L01 | TDD Philosophy for Agents | L1 (Manual) | 20 min | | L02 | pytest Fundamentals for Async Code | L1 (Manual) | 25 min | | L03 | Testing FastAPI Endpoints | L2 (Collaboration) | 30 min | | L04 | Testing SQLModel Operations | L2 (Collaboration) | 25 min | | L05 | Mocking LLM Calls | L2 (Collaboration) | 30 min | | L06 | Testing Agent Tools | L2/L3 (Transition) | 25 min | | L07 | Integration Test Patterns | L3 (Intelligence) | 30 min | | L08 | Capstone: Full Test Suite | L4 (Spec-Driven) | 35 min | **Total Duration**: ~4 hours Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------- Before starting this chapter, you should have: * **Chapter 70**: FastAPI for Agents (Task API codebase) * **Part 4**: Python fundamentals (basic pytest knowledge) * **Ch62-65**: Agent SDK patterns (code to test) What You'll Own[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#what-youll-own "Direct link to What You'll Own") ------------------------------------------------------------------------------------------------------------------------------------------------------ By completing this chapter, you'll have: 1. **`agent-tdd` skill** — A reusable testing skill built from official pytest-asyncio and respx documentation 2. **Complete test suite** — Unit, integration, and agent pipeline tests for Task API 3. **CI/CD workflow** — GitHub Actions configuration for automated testing 4. **Testing patterns** — Transferable patterns for any FastAPI + SQLModel project Key Technologies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#key-technologies "Direct link to Key Technologies") ---------------------------------------------------------------------------------------------------------------------------------------------------------- * **pytest-asyncio** — Async test execution * **httpx** — Async HTTP client with ASGITransport * **respx** — HTTP mocking for LLM API calls * **pytest-cov** — Coverage reporting Start Learning[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#start-learning "Direct link to Start Learning") ---------------------------------------------------------------------------------------------------------------------------------------------------- Begin with [Lesson 0: Build Your Testing Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/build-your-testing-skill) to create your `agent-tdd` skill before learning the content. * [What You'll Build](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#what-youll-build) * [The Critical Distinction](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#the-critical-distinction) * [Skill-First Learning](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#skill-first-learning) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#chapter-structure) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#prerequisites) * [What You'll Own](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#what-youll-own) * [Key Technologies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#key-technologies) * [Start Learning](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents#start-learning) --- # Chapter 73: Vector Databases & RAG | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C **You will build a RAG skill BEFORE you learn RAG concepts.** This chapter follows the **Skill-First Learning Pattern**: In Lesson 0, you create a `rag-deployment` skill using tools from Chapter 3. Then you spend the chapter understanding what you built and making it better. By the end, you don't just "know RAG"—you OWN a production-ready skill that extends your Task API with semantic search. The Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-architecture "Direct link to The Architecture") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter uses a clean separation of concerns: | Layer | Technology | Purpose | | --- | --- | --- | | **Ingestion** | LangChain | Document loading, text splitting, embeddings | | **Storage/Retrieval** | LangChain + Qdrant | Vector store, similarity search | | **Intelligence** | OpenAI Agents SDK | Agent orchestration, LLM calls, conversation memory | | **Evaluation** | RAGAS + LangSmith | Quality metrics, tracing | **Why this split?** LangChain excels at document processing and retrieval. The Agents SDK (Chapter 62) excels at agent orchestration. Together, they give you the best of both worlds without framework lock-in. The Skill-First Arc[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-skill-first-arc "Direct link to The Skill-First Arc") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L00: Build Your RAG Skill (20 min) | └── Clone skills-lab, fetch LangChain docs, create skill | ├── L01-02: Conceptual Foundation | └── Why RAG? + Embeddings mental model | ├── L03-04: LangChain for Data | └── Document processing, Qdrant vector store | ├── L05: RAG with Agents SDK | └── Agent + retrieval tool pattern | ├── L06: Task API Integration | └── Your skill learns semantic task search | ├── L07: RAG Evaluation | └── LangSmith + RAGAS metrics | └── L08: Architecture Patterns (Capstone) └── 8 RAG patterns—Simple to Agentic Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#chapter-structure "Direct link to Chapter Structure") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | # | Lesson | Duration | Your Skill Improves | | --- | --- | --- | --- | | **0** | **Build Your RAG Skill** | 20 min | **Created from LangChain docs** | | 1 | Why Agents Need External Knowledge | 25 min | Conceptual: parametric vs retrieval | | 2 | Vector Embeddings Mental Model | 30 min | Conceptual: semantic similarity | | 3 | LangChain Document Processing | 40 min | Add: chunking strategies | | 4 | Qdrant Vector Store | 45 min | Add: vector storage patterns | | 5 | RAG with OpenAI Agents SDK | 45 min | Add: agent + retrieval tool pattern | | 6 | RAG for Task API | 50 min | Refine: semantic search endpoint | | 7 | Evaluating RAG Quality | 45 min | Add: evaluation metrics | | **8** | **RAG Architecture Patterns** | 60 min | **Capstone: 2 patterns implemented** | Every Lesson Has Skill Reflection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#every-lesson-has-skill-reflection "Direct link to Every Lesson Has Skill Reflection") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Each lesson ends with **"Reflect on Your Skill"**: 1. **Test**: Does your skill handle this lesson's concepts? 2. **Identify gaps**: What's missing? 3. **Improve**: Add or refine the capability By Lesson 8, you've tested and improved your skill 8+ times. Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapter 70**: FastAPI for Agents (Task API foundation) * **Chapters 62-65**: Agent SDK chapters (agents know how to call tools) * **Chapter 3**: skill-creator and fetching-library-docs * **Docker**: For running Qdrant locally The Running Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-running-example "Direct link to The Running Example") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Your Task API gets smarter.** In Chapter 70, you built a Task API with CRUD operations. Now you extend it: * **Lessons 1-5**: Learn how RAG works—embeddings, Qdrant, retrieval chains * **Lesson 6**: Add `/tasks/search/semantic` endpoint to your Task API * **Lesson 8**: Implement advanced RAG patterns for intelligent task recommendations The key insight: **Semantic search lets users find tasks by meaning, not just keywords**. What You'll Own[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#what-youll-own "Direct link to What You'll Own") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- .claude/skills/├── skill-creator/ # From Chapter 3├── fetching-library-docs/ # From Chapter 3├── fastapi-agent-api/ # From Chapter 70└── rag-deployment/ # NEW - this chapter This skill joins your growing **Digital FTE toolkit**. The Mindset Shift[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-mindset-shift "Direct link to The Mindset Shift") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Traditional**: "Teach me RAG" **Agent Factory**: "I own a RAG skill—help me make it better" You're not here to learn. You're here to **build assets**. Start with **Lesson 0: Build Your RAG Skill**. * [The Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-architecture) * [The Skill-First Arc](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-skill-first-arc) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#chapter-structure) * [Every Lesson Has Skill Reflection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#every-lesson-has-skill-reflection) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#prerequisites) * [The Running Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-running-example) * [What You'll Own](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#what-youll-own) * [The Mindset Shift](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain#the-mindset-shift) --- # Agent Ops | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Building an agent is the easy part. Operating it reliably is where teams struggle. Agent Ops is the discipline of evaluation, debugging, and continuous improvement. The Core Challenge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#the-core-challenge "Direct link to The Core Challenge") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Traditional software testing: Input → Expected output. Pass or fail. Agent testing: Input → Many valid outputs. How do you evaluate "good enough"? When a customer asks "where's my order?" a good response might be: * "Your order #12345 shipped yesterday via UPS, tracking ZYX987" * "Order 12345 is on its way! Here's your tracking: ZYX987 (UPS)" * "Great news! Your order shipped. Track it here: \[link\]" All valid. None identical. Traditional testing breaks down. Four Pillars of Agent Ops[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#four-pillars-of-agent-ops "Direct link to Four Pillars of Agent Ops") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### 1\. LM-as-Judge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#1-lm-as-judge "Direct link to 1. LM-as-Judge") Use a language model to evaluate agent outputs against quality criteria. **How it works:** Agent output + Evaluation rubric → Judge LLM → Score + Explanation **Example rubric for customer support:** Criterion: Response helpfulness10 points: Directly solves user's problem with clear, actionable information7 points: Addresses problem but missing some detail4 points: Partially relevant, user needs follow-up questions0 points: Misunderstands problem or provides wrong information The judge LLM applies this rubric consistently across thousands of responses. It can evaluate nuance that simple string matching can't. **Why this works:** * LLMs understand intent, not just keywords * Consistent evaluation at scale * Can explain _why_ a response scored low * Catches subtle failures (wrong tone, missing context, confusing explanations) **Designing good rubrics:** * Make criteria specific and measurable * Provide examples for each score level * Test the rubric on edge cases * Iterate until judge scores match human judgment ### 2\. Golden Datasets[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#2-golden-datasets "Direct link to 2. Golden Datasets") Curated test cases with known-good responses. **Structure:** Scenario: "Where is my order #12345?"Context: Order shipped yesterday, tracking ZYX987, UPS, arriving tomorrowExpected elements: ✓ Confirms order number ✓ Provides shipping status ✓ Includes tracking number ✓ Mentions carrier ✓ Estimates deliveryGolden response: "Your order #12345 shipped yesterday via UPS.Tracking: ZYX987. Expected delivery: tomorrow." Run golden datasets after every change. If scores drop, you broke something. **Building golden datasets:** 1. Start with real user interactions 2. Select diverse scenarios (happy path, edge cases, errors) 3. Document expected elements, not exact wording 4. Include challenging cases that previously failed 5. Update as you learn new failure modes **How many test cases?** * Start with 20-50 covering main scenarios * Add every bug you fix as a new test case * Aim for 100+ for production systems * Quality matters more than quantity ### 3\. Traces[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#3-traces "Direct link to 3. Traces") Record every step an agent takes. When things fail, traces show exactly where. **Trace structure:** Request: "Fix the failing test in auth.py"Timestamp: 2024-01-15 10:23:45Step 1: [10:23:46] Read test file Tool: read_file("test_auth.py") Result: 245 lines readStep 2: [10:23:47] Read implementation Tool: read_file("auth.py") Result: 180 lines readStep 3: [10:23:48] Analyze error Reasoning: "Test expects AuthenticationError but function returns None"Step 4: [10:23:49] Edit code Tool: edit_file("auth.py", changes=[...]) Result: SuccessStep 5: [10:23:52] Run test Tool: execute("pytest test_auth.py") Result: PASSEDOutcome: SuccessDuration: 7 seconds **What traces enable:** * **Root cause analysis**: Exact step where failure occurred * **Performance debugging**: Which steps are slow? * **Pattern recognition**: Do similar failures share common trace patterns? * **Audit trail**: What did the agent access? What actions did it take? **Trace debugging workflow:** 1. User reports problem 2. Find the trace for that interaction 3. Walk through steps to identify where things went wrong 4. Was it wrong reasoning? Tool failure? Missing context? 5. Fix the root cause 6. Add to golden dataset to prevent regression ### 4\. Human Feedback Loops[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#4-human-feedback-loops "Direct link to 4. Human Feedback Loops") Capture user reactions and feed them back into improvement. **Feedback capture:** "Was this helpful?"[ ] Yes, solved my problem[ ] Partially helpful[ ] Not helpful - wrong information[ ] Not helpful - missing information Simple feedback captures signal. Detailed feedback captures insight. **The improvement loop:** 1. User reports unhelpful response 2. Find the trace for that interaction 3. Identify root cause (reasoning error? missing tool? bad context?) 4. Fix the underlying issue 5. Add scenario to golden dataset 6. Verify fix with LM-as-Judge 7. Monitor for recurrence **Feedback analysis patterns:** * Cluster similar complaints to find systematic issues * Track feedback by scenario type (which areas struggle most?) * Compare feedback before/after changes to measure improvement The Mindset Shift[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#the-mindset-shift "Direct link to The Mindset Shift") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Before Agent Ops**: "Is my agent correct?" **With Agent Ops**: "Is my agent improving toward my KPIs?" Define measurable success: * Response time < 3 seconds (P95) * User satisfaction > 85% (feedback surveys) * Resolution rate > 70% (issues closed without escalation) * Escalation rate < 10% (complex issues passed to humans) Then track, debug, and improve systematically. Applying to Your Work[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#applying-to-your-work "Direct link to Applying to Your Work") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Even without production agents, you can practice Agent Ops thinking: **With Claude Code:** * Notice when it succeeds vs fails * Identify patterns in failures (missing context? wrong reasoning? tool errors?) * Give specific feedback that helps it improve * Track which prompts work well (your personal golden dataset) This is the same mental model. You're doing informal Agent Ops every time you work with Claude Code. **Example:** Claude Code fails to fix a bug. Instead of just trying again, analyze: * Did it understand the goal? (Check its reasoning) * Did it have enough context? (Did it read the right files?) * Did it execute correctly? (Check tool outputs) * Did it verify success? (Did it run tests?) This analysis improves your next prompt—and your mental model of how agents work. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to design Agent Ops. > "I want to evaluate a document search agent. It takes a query and returns top 3 relevant documents. What KPIs should I measure? Give me specific, measurable metrics." **Expected**: Relevance of results (% queries with relevant top result), response time (P50, P95), query understanding accuracy (evaluated by judge), user satisfaction (feedback scores). > "Create an LM-as-Judge rubric for 'document relevance' with 4 scoring levels (10/7/4/0). What distinguishes each level?" **Expected**: Clear criteria. 10 = "All 3 documents directly relevant to query." 7 = "2 of 3 relevant." 4 = "1 of 3 relevant." 0 = "No relevant documents returned." > "A customer support agent gave a generic shipping policy instead of the user's specific order status. The trace shows it called lookup\_order() but the tool returned an error. How would you diagnose and fix this?" **Expected**: Tool failure caused fallback to generic response. Fix: Better error handling, retry logic, or escalation when tools fail. Add test case for tool failures. > "My agent's satisfaction scores dropped 15% after last week's update. How do I use traces and feedback to diagnose what changed?" **Expected**: Compare traces before/after update. Look for new failure patterns. Cluster negative feedback to find common issues. Check if specific scenarios degraded. **Key insight**: The teams building reliable agents aren't using better models—they have better operational discipline. Measure, trace, debug, improve, repeat. Checking access... * [The Core Challenge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#the-core-challenge) * [Four Pillars of Agent Ops](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#four-pillars-of-agent-ops) * [1\. LM-as-Judge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#1-lm-as-judge) * [2\. Golden Datasets](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#2-golden-datasets) * [3\. Traces](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#3-traces) * [4\. Human Feedback Loops](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#4-human-feedback-loops) * [The Mindset Shift](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#the-mindset-shift) * [Applying to Your Work](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#applying-to-your-work) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-ops#try-with-ai) --- # Chapter 67: Advanced MCP Server Development | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Chapter 66 taught you MCP fundamentals: architecture, primitives, and basic server implementation with @mcp.tool, @mcp.resource, and @mcp.prompt decorators. Now you'll master the **advanced patterns** that separate hobby servers from production-ready systems. This chapter covers the features that make MCP servers truly powerful: servers that call LLMs through clients (sampling), provide real-time feedback (progress notifications), control file access securely (roots), and scale horizontally for production workloads (StreamableHTTP with stateful/stateless configurations). What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#what-youll-learn "Direct link to What You'll Learn") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * **Use the Context object**: Inject Context into tools for logging, progress, and session access * **Implement sampling**: Have your server call LLMs through connected clients * **Send notifications**: Provide real-time progress and logging feedback during operations * **Configure roots**: Control file system access with secure permission boundaries * **Deploy with StreamableHTTP**: Run production servers with SSE and session management * **Scale horizontally**: Choose stateful vs stateless configurations for load balancing * **Handle errors gracefully**: Return structured JSON-RPC errors without crashing * **Package for distribution**: Create installable servers with pyproject.toml Chapter Structure (10 Lessons)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#chapter-structure-10-lessons "Direct link to Chapter Structure (10 Lessons)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Layer 2: AI Collaboration (Building with AI)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#layer-2-ai-collaboration-building-with-ai "Direct link to Layer 2: AI Collaboration (Building with AI)") 1. **Context Object & Server Lifespan** — Injection, session access, async resource management 2. **Sampling: Servers Calling LLMs** — ctx.session.create\_message(), client callbacks, cost shifting 3. **Progress & Logging Notifications** — report\_progress(), info/warning/error, client handling 4. **Roots: File System Permissions** — list\_roots, is\_path\_allowed(), secure path validation 5. **StreamableHTTP Transport** — SSE connections, session IDs, production deployment 6. **Stateful vs Stateless Servers** — stateless\_http, json\_response, horizontal scaling tradeoffs 7. **Error Handling & Recovery** — JSON-RPC errors, graceful degradation, retry-safe design 8. **Packaging & Distribution** — pyproject.toml, entry points, installation testing ### Layer 4: Spec-Driven Integration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#layer-4-spec-driven-integration "Direct link to Layer 4: Spec-Driven Integration") 9. **Capstone: Production MCP Server** — Full server with sampling, progress, roots, StreamableHTTP ### Assessment[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#assessment "Direct link to Assessment") 10. **Chapter Quiz** — Advanced patterns, transport configuration, scaling decisions Key Distinction: Chapter 66 vs Chapter 67[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#key-distinction-chapter-66-vs-chapter-67 "Direct link to Key Distinction: Chapter 66 vs Chapter 67") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Topic | Chapter 66 (Fundamentals) | Chapter 67 (Advanced) | | --- | --- | --- | | @mcp.tool basics | ✓ Decorators, schemas | Skip | | @mcp.resource basics | ✓ URIs, MIME types | Skip | | @mcp.prompt basics | ✓ Arguments, messages | Skip | | **Context Object** | — | ✓ Injection, lifespan | | **Sampling** | — | ✓ Server → Client LLM calls | | **Notifications** | — | ✓ Progress, logging | | **Roots** | — | ✓ File permissions | | **StreamableHTTP** | — | ✓ SSE, sessions | | **Scaling** | — | ✓ Stateful vs stateless | Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapter 66**: MCP Fundamentals (decorators, primitives, basic servers) * **Part 4**: Python Fundamentals (async/await, decorators, Pydantic) Tools Required[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#tools-required "Direct link to Tools Required") -------------------------------------------------------------------------------------------------------------------------------------------------------- * Python 3.11+ * uv package manager * Node.js + npm (for MCP Inspector) * Claude Desktop or Claude Code (for testing) Chapter Outcome[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#chapter-outcome "Direct link to Chapter Outcome") ----------------------------------------------------------------------------------------------------------------------------------------------------------- Students completing this chapter will have: 1. **Production Skills**: Ability to build servers with sampling, progress, and proper error handling 2. **Deployment Knowledge**: Understanding of StreamableHTTP and scaling configurations 3. **Security Awareness**: Roots-based file access control 4. **Digital FTE Component**: A complete, production-ready MCP server with advanced features * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#what-youll-learn) * [Chapter Structure (10 Lessons)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#chapter-structure-10-lessons) * [Layer 2: AI Collaboration (Building with AI)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#layer-2-ai-collaboration-building-with-ai) * [Layer 4: Spec-Driven Integration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#layer-4-spec-driven-integration) * [Assessment](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#assessment) * [Key Distinction: Chapter 66 vs Chapter 67](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#key-distinction-chapter-66-vs-chapter-67) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#prerequisites) * [Tools Required](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#tools-required) * [Chapter Outcome](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers#chapter-outcome) --- # Chapter 77: Evals - Measuring Agent Performance | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C **Part**: 6 (AI Native Software Development) **Phase**: Quality Assurance **Proficiency Level**: B1-B2 (Intermediate to Upper Intermediate) **Duration**: ~4.5 hours Chapter Overview[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#chapter-overview "Direct link to Chapter Overview") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter teaches the **thinking and methodology** behind agent evaluations (evals)—the systematic approach to measuring AI agent reasoning quality. Unlike TDD (Chapter 76) which tests code correctness with deterministic PASS/FAIL outcomes, evals measure probabilistic reasoning quality with scores. **Core Thesis** (Andrew Ng): _"One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process."_ Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------------------------------------------- * **Ch62-65 (SDK chapters)**: Understanding of agent architecture * **Ch70 (FastAPI for Agents)**: Task API running example * **Ch76 (TDD for Agents)**: Understanding of test-driven development Student Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#student-skill "Direct link to Student Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------- Students build the `agent-evals` skill throughout this chapter, starting with L00 and finalizing in L10. Lesson Index[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#lesson-index "Direct link to Lesson Index") ------------------------------------------------------------------------------------------------------------------------------------------------------- | # | Lesson | Duration | Focus | | --- | --- | --- | --- | | L00 | Build Your Evals Skill | 25 min | Skill-First: Create agent-evals skill | | L01 | Evals Are Exams for Reasoning | 20 min | TDD vs Evals distinction | | L02 | The Two Evaluation Axes | 20 min | Four-quadrant classification | | L03 | Designing Eval Datasets | 25 min | Quality over quantity (10-20 cases) | | L04 | Building Graders with Binary Criteria | 30 min | Binary yes/no pattern | | L05 | LLM-as-Judge Graders | 30 min | LLM evaluation with limitations | | L06 | Systematic Error Analysis | 30 min | Spreadsheet-based counting | | L07 | Component vs End-to-End Evals | 25 min | Decision framework | | L08 | Regression Protection | 25 min | Eval-on-every-change | | L09 | The Complete Quality Loop | 30 min | Build-Evaluate-Analyze-Improve | | L10 | Finalize Your Evals Skill | 20 min | Skill validation | Key Concepts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#key-concepts "Direct link to Key Concepts") ------------------------------------------------------------------------------------------------------------------------------------------------------- * **Evals as Exams**: Testing reasoning quality, not code correctness * **Two Axes**: Objective/Subjective × Ground Truth availability * **Binary Criteria**: 5 yes/no checks → 0-5 score (more reliable than 1-5 scales) * **Error Analysis**: Systematic counting replaces gut feeling * **The Quality Loop**: Build → Evaluate → Analyze → Improve → Repeat Running Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#running-example "Direct link to Running Example") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Task API agent from Ch70, evaluated for: * Routing decisions (create vs update vs query) * Tool selection correctness * Output format compliance * Error handling quality Learning Outcomes[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#learning-outcomes "Direct link to Learning Outcomes") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- By chapter end, students can: 1. Distinguish evals from TDD based on determinism and outcomes 2. Design eval datasets with typical, edge, and error cases 3. Create graders using binary criteria 4. Perform systematic error analysis 5. Build regression protection workflows 6. Apply the complete quality loop Framework-Agnostic[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#framework-agnostic "Direct link to Framework-Agnostic") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- These concepts apply to any SDK (OpenAI, Claude, Google ADK, LangChain). The thinking is portable. * [Chapter Overview](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#chapter-overview) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#prerequisites) * [Student Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#student-skill) * [Lesson Index](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#lesson-index) * [Key Concepts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#key-concepts) * [Running Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#running-example) * [Learning Outcomes](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#learning-outcomes) * [Framework-Agnostic](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance#framework-agnostic) --- # Chapter 62: OpenAI Agents SDK Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/chapter-quiz#__docusaurus_skipToContent_fallback) Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of the OpenAI Agents SDK concepts covered in this chapter. Each question tests your ability to apply the concepts, not just recall definitions. Checking access... --- # Build Your OpenAI Agents Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning OpenAI's Agents SDK, you'll **own** an OpenAI Agents skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for OpenAI Agents SDK.I will use it to build AI agents from hello world to professional productionsystems. Use context7 skill to study official documentation and then buildit so no self assumed knowledge. Claude will: 1. Fetch official OpenAI Agents SDK documentation via Context7 2. Ask you clarifying questions (tool patterns, handoff needs, guardrails) 3. Create the complete skill with references and templates Your skill appears at `.claude/skills/openai-agents/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#done "Direct link to Done") -------------------------------------------------------------------------------------------------------------------------------------------------------- You now own an OpenAI Agents skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — Agent Fundamentals** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/build-your-openai-agents-skill#done) --- # Chapter Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of Test-Driven Development patterns for AI agent applications. This quiz covers the key concepts from the chapter: distinguishing TDD from Evals, configuring pytest-asyncio, mocking LLM calls with respx, and writing integration tests. **Target**: 70% (7 out of 10 correct) * * * Question 1[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-1 "Direct link to Question 1") --------------------------------------------------------------------------------------------------------------------------------------------- You need to verify that your agent's authentication endpoint correctly rejects invalid JWT tokens. Which testing approach should you use? * [ ] A) Evals with real LLM calls to assess response quality * [ ] B) TDD with mocked responses to test code logic * [ ] C) Manual testing in production environment * [ ] D) Evals with human judges scoring the output Answer **Correct: B** JWT token validation is **code correctness**—it either works or it doesn't. This is deterministic behavior that TDD tests perfectly. You mock the authentication flow and verify your code correctly accepts valid tokens and rejects invalid ones. Evals measure LLM reasoning quality, which isn't involved in authentication logic. * * * Question 2[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-2 "Direct link to Question 2") --------------------------------------------------------------------------------------------------------------------------------------------- Your test suite runs 50 tests, each making real OpenAI API calls. At $0.006 per test and 20 runs per day for 30 days, what is the approximate monthly cost? * [ ] A) $18 * [ ] B) $180 * [ ] C) $1,800 * [ ] D) $0 because testing is free Answer **Correct: B** The calculation: $0.006 per test x 50 tests = $0.30 per run. $0.30 x 20 runs/day x 30 days = **$180/month**. This is why mocking LLM calls matters—mocked tests cost $0.00 and run in milliseconds instead of seconds. The chapter emphasizes this cost-benefit analysis as a key reason to adopt TDD with mocked responses. * * * Question 3[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-3 "Direct link to Question 3") --------------------------------------------------------------------------------------------------------------------------------------------- Which aspect should be tested with Evals (not TDD)? * [ ] A) Database cascade delete removes child records * [ ] B) Agent interprets ambiguous user requests helpfully * [ ] C) API endpoint returns 404 for missing resources * [ ] D) Tool function validates input against injection attacks Answer **Correct: B** Interpreting ambiguous requests requires **LLM reasoning quality assessment**—this is probabilistic, not deterministic. There's no single "correct" answer, and quality varies across runs. This requires Evals with real LLM calls and scoring (0-1 scale). Options A, C, and D are deterministic code correctness tests—they pass or fail based on whether the code logic works correctly. These belong in TDD with mocked responses. * * * Question 4[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-4 "Direct link to Question 4") --------------------------------------------------------------------------------------------------------------------------------------------- What does `asyncio_mode = "auto"` in `pyproject.toml` configure? * [ ] A) Enables automatic parallel test execution across CPUs * [ ] B) Automatically detects async test functions without decorators * [ ] C) Creates automatic database backups before each test * [ ] D) Enables automatic retry on test failures Answer **Correct: B** The `asyncio_mode = "auto"` setting tells pytest-asyncio to **automatically detect `async def` test functions** and run them with an event loop. Without this setting (or explicit `@pytest.mark.asyncio` decorators), async tests would fail because pytest wouldn't know to await the coroutines. This is configured in `[tool.pytest.ini_options]` section. * * * Question 5[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-5 "Direct link to Question 5") --------------------------------------------------------------------------------------------------------------------------------------------- Why is `StaticPool` required when using in-memory SQLite for testing? * [ ] A) It improves test performance by caching queries * [ ] B) It prevents the database from being destroyed between connections * [ ] C) It enables async operations that SQLite normally blocks * [ ] D) It provides automatic schema migration support Answer **Correct: B** In-memory SQLite databases exist only within a single connection. Without `StaticPool`, each new connection creates a fresh (empty) database, losing all data created in previous connections. `StaticPool` ensures all connections share the same in-memory database, so data persists across transactions within a test. This is configured with `poolclass=StaticPool` in `create_async_engine`. * * * Question 6[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-6 "Direct link to Question 6") --------------------------------------------------------------------------------------------------------------------------------------------- What is the correct way to mock an OpenAI tool call response with respx? * [ ] A) `respx.post(...).mock(json={"tool_calls": [...]})` * [ ] B) `respx.post(...).mock(return_value=httpx.Response(200, json={"choices": [{"message": {"tool_calls": [...]}}]}))` * [ ] C) `respx.mock_tool_call("create_task", {"title": "Test"})` * [ ] D) `openai.mock_response(tool_calls=[...])` Answer **Correct: B** respx mocks at the HTTP transport layer, so you must return a complete `httpx.Response` object matching the **exact OpenAI API response structure**. Tool calls are nested inside `choices[0].message.tool_calls`, and `function.arguments` must be a JSON string (not a parsed object). The response also needs `finish_reason: "tool_calls"`. Options A, C, and D don't match how respx or the OpenAI API actually work. * * * Question 7[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-7 "Direct link to Question 7") --------------------------------------------------------------------------------------------------------------------------------------------- How do you test that an agent correctly handles rate limiting (429 response)? * [ ] A) Wait for production to hit real rate limits * [ ] B) Use `respx.post(...).mock(return_value=httpx.Response(429, json={...}))` * [ ] C) Disable rate limiting in test environment * [ ] D) Mock the rate limit by setting a low token count Answer **Correct: B** You mock the 429 rate limit response using respx, which lets you simulate the error without making real API calls. The mock should include the error JSON structure and `Retry-After` header that OpenAI returns. This lets you verify your agent's retry logic and error handling. You can also use `side_effect` with a list to test retry behavior: first call returns 429, second call succeeds. * * * Question 8[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-8 "Direct link to Question 8") --------------------------------------------------------------------------------------------------------------------------------------------- What pattern tests a multi-turn agent conversation (tool call followed by final response)? * [ ] A) Two separate test functions, one for each turn * [ ] B) `respx.post(...).mock(side_effect=[response1, response2])` * [ ] C) `respx.sequence([mock1, mock2])` * [ ] D) Nested `with respx.mock` context managers Answer **Correct: B** The `side_effect` parameter accepts a **list of responses** that are returned sequentially for each call to the same endpoint. First call gets `response1` (tool call), second call gets `response2` (final message). This mirrors the actual multi-turn flow: user message, LLM decides to call tool, tool executes, LLM responds with result. This pattern is essential for integration testing agent pipelines. * * * Question 9[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-9 "Direct link to Question 9") --------------------------------------------------------------------------------------------------------------------------------------------- In FastAPI testing, what does `app.dependency_overrides[get_session] = get_test_session` accomplish? * [ ] A) Permanently replaces production database with test database * [ ] B) Creates a backup of the session before each test * [ ] C) Substitutes test implementation for production dependency * [ ] D) Enables automatic session cleanup after failures Answer **Correct: C** Dependency overrides **substitute test implementations** for production dependencies during testing. When your endpoint calls `get_session`, FastAPI returns `get_test_session` instead—typically an in-memory SQLite session. This isolates tests from production resources. The override is cleared with `app.dependency_overrides.clear()` after tests to prevent pollution between test runs. * * * Question 10[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-10 "Direct link to Question 10") ------------------------------------------------------------------------------------------------------------------------------------------------ When testing an agent's complete pipeline, how do you verify the database was actually modified? * [ ] A) Trust the agent's response message says it was created * [ ] B) Query the database directly after the agent action completes * [ ] C) Check the LLM mock was called with correct arguments * [ ] D) Verify the HTTP response status code is 200 Answer **Correct: B** Integration tests must verify **actual database state**, not just agent responses. After the agent says "I created the task," query the database to confirm the record exists with correct attributes. The pattern is: check state before, execute agent action, check state after. This catches bugs where the agent responds successfully but the database operation actually failed. Status codes and mock verification don't prove data was persisted. * * * Scoring Guide[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#scoring-guide "Direct link to Scoring Guide") ------------------------------------------------------------------------------------------------------------------------------------------------------ | Score | Assessment | | --- | --- | | 10/10 | Excellent - Ready for the capstone project | | 8-9/10 | Good - Solid understanding of core patterns | | 7/10 | Pass - Review any missed concepts | | 5-6/10 | Needs Review - Revisit lessons 1, 3, and 5 | | Below 5 | Study Required - Re-read the chapter | Key Concepts Summary[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#key-concepts-summary "Direct link to Key Concepts Summary") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **TDD vs Evals**: TDD tests code correctness (deterministic, mocked, pass/fail). Evals measure LLM quality (probabilistic, real calls, scores). **pytest-asyncio**: Configure with `asyncio_mode = "auto"` and use `@pytest.mark.asyncio` for async tests. **respx mocking**: Mock at HTTP transport layer with exact OpenAI/Anthropic response structures. Use `side_effect` for multi-turn sequences. **Integration testing**: Verify database state changes, not just agent responses. Test error handling paths with mocked failures. * [Question 1](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-1) * [Question 2](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-2) * [Question 3](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-3) * [Question 4](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-4) * [Question 5](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-5) * [Question 6](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-6) * [Question 7](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-7) * [Question 8](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-8) * [Question 9](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-9) * [Question 10](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#question-10) * [Scoring Guide](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#scoring-guide) * [Key Concepts Summary](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/quiz#key-concepts-summary) --- # Why Agents Need Memory | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your customer just returned after three days away. "Hi, I'm back. Can you help me with that thing we discussed?" A stateless agent responds: "I'm sorry, I don't have any record of our previous conversation. Could you please describe what you'd like help with?" The customer sighs. They've been through this before. Now imagine an agent with memory. Same customer, same request. The agent responds: "Welcome back! Last time we were working on migrating your authentication system to OAuth 2.0. You mentioned the deadline was next Friday. Should we pick up where we left off, or has something changed?" That difference—between an agent that forgets and one that remembers—is what separates a tool you tolerate from one you trust. Memory is what transforms AI from a utility into a relationship. The Context Window Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-context-window-problem "Direct link to The Context Window Problem") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every LLM operates within a **context window**—a fixed amount of text it can see at once. Think of it as the LLM's working memory: everything you send in a single API call, plus everything the model sends back. ┌─────────────────────────────────────────────────────────────────┐│ CONTEXT WINDOW ││ ││ System prompt: "You are a helpful assistant..." ││ User: "Hi, I'm Alex." ││ Assistant: "Hello Alex! How can I help?" ││ User: "I prefer morning meetings." ││ Assistant: "Got it—morning meetings work best for you." ││ User: "What's my name?" ││ Assistant: "Your name is Alex." ← Still in context ││ │└─────────────────────────────────────────────────────────────────┘ │ │ NEW SESSION ▼┌─────────────────────────────────────────────────────────────────┐│ CONTEXT WINDOW ││ ││ System prompt: "You are a helpful assistant..." ││ User: "What's my name?" ││ Assistant: "I don't know your name yet." ← Context is GONE ││ │└─────────────────────────────────────────────────────────────────┘ Modern context windows are large—GPT-4 offers 128,000 tokens, Claude offers 200,000 tokens—but they're still finite. More importantly, they're **ephemeral**. When the session ends or the context fills up, everything the model "knew" disappears. This creates **context amnesia**: the model literally cannot remember anything from previous sessions. Every conversation starts fresh, as if meeting the user for the first time. ### Why Large Context Windows Aren't Enough[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#why-large-context-windows-arent-enough "Direct link to Why Large Context Windows Aren't Enough") You might think: "128k tokens is huge! Can't we just stuff all history in there?" Three problems: | Problem | Impact | | --- | --- | | **Cost** | Tokens cost money. Sending 100k tokens of history with every request gets expensive fast. | | **Latency** | More tokens = slower responses. Users notice. | | **Noise** | Old, irrelevant information can confuse the model. Not everything should be in context. | The context window is a **temporary buffer**, not a **memory system**. Real memory requires persistence, prioritization, and retrieval—capabilities the raw LLM doesn't have. Stateless vs. Stateful Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#stateless-vs-stateful-agents "Direct link to Stateless vs. Stateful Agents") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This distinction matters for everything that follows. ### Stateless Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#stateless-agents "Direct link to Stateless Agents") A **stateless agent** processes each request in isolation. It has no knowledge of previous interactions beyond what you explicitly include in the current request. # Stateless: Every call is independentresponse1 = agent.run("My name is Alex")# Output: "Nice to meet you, Alex!"response2 = agent.run("What's my name?")# Output: "I don't know your name. What is it?" Each `agent.run()` call starts fresh. The agent has no way to connect the dots between calls. ### Stateful Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#stateful-agents "Direct link to Stateful Agents") A **stateful agent** maintains context across interactions. It remembers what happened before and uses that knowledge to inform future responses. # Stateful: Memory persists across callsresponse1 = agent.run("My name is Alex", user_id="alex-123")# Output: "Nice to meet you, Alex!"response2 = agent.run("What's my name?", user_id="alex-123")# Output: "Your name is Alex. You told me earlier." The `user_id` connects to a memory system that persists facts across calls. ### Behavioral Comparison[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#behavioral-comparison "Direct link to Behavioral Comparison") | Behavior | Stateless Agent | Stateful Agent | | --- | --- | --- | | User returns after 3 days | "Who are you?" | "Welcome back, Alex!" | | User mentions "the project" | "Which project?" | "Ah, the OAuth migration." | | User has a preference | Must state every time | Applied automatically | | Conversation builds | Each message isolated | Context accumulates | | User feels | Like talking to a stranger | Like working with a colleague | The difference isn't just technical—it's experiential. Stateful agents feel like they _know_ you. What Agents Should Remember[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#what-agents-should-remember "Direct link to What Agents Should Remember") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not everything belongs in memory. Effective agent memory is selective, storing information that creates value while avoiding noise. ### Categories of Valuable Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#categories-of-valuable-memory "Direct link to Categories of Valuable Memory") **1\. User Identity & Preferences** * Name, role, communication style * Preferred working hours, formats, tools * Past decisions and their rationale **2\. Interaction History** * What was discussed and decided * Outstanding questions or tasks * Previous issues and their resolutions **3\. Learned Facts** * Domain-specific terminology the user uses * Project names, team members, deadlines * User's expertise level and interests **4\. Behavioral Patterns** * How the user typically structures requests * Common workflows and shortcuts * What kinds of clarification they need ### Example: Task Manager Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#example-task-manager-agent "Direct link to Example: Task Manager Agent") Consider an agent that helps manage tasks. What should it remember? VALUABLE MEMORIES:├── User Preferences│ ├── "Prefers tasks scheduled in morning"│ ├── "Uses priority labels: urgent, normal, later"│ └── "Likes detailed task descriptions"│├── Project Context│ ├── "Current project: OAuth Migration"│ ├── "Deadline: January 31"│ └── "Involves auth-service and user-service"│├── Interaction Patterns│ ├── "Usually creates 3-5 tasks at once"│ ├── "Prefers bullet points over prose"│ └── "Asks for time estimates on complex tasks"│└── Historical Insights ├── "Estimated 2 hours → Actually took 4 hours (pattern)" ├── "Completes urgent tasks same day" └── "Often reschedules 'later' tasks" With this memory, the agent can: * Schedule new tasks in the morning automatically * Provide more accurate time estimates based on history * Warn when "later" tasks are accumulating * Reference "the project" without clarification Memory as Competitive Advantage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#memory-as-competitive-advantage "Direct link to Memory as Competitive Advantage") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- From a business perspective, memory transforms agents from interchangeable utilities into sticky, valuable products. ### The Retention Effect[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-retention-effect "Direct link to The Retention Effect") Users invest time teaching an agent their preferences. That investment creates switching costs: * **New agent**: Start over explaining everything * **Existing agent**: Already knows you, just works This is why personal assistants that remember are dramatically more valuable than those that don't. ### The Improvement Effect[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-improvement-effect "Direct link to The Improvement Effect") Agents with memory can improve over time: * Learn which suggestions you accept vs. reject * Adapt communication style to what works for you * Build increasingly accurate models of your needs A stateless agent performs the same on day 1 as day 100. A stateful agent gets better. ### The Differentiation Effect[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-differentiation-effect "Direct link to The Differentiation Effect") Most chatbots are stateless. An agent that genuinely remembers stands out: | Feature | Table-stakes | Differentiator | | --- | --- | --- | | Answer questions | ✓ | | | Follow instructions | ✓ | | | Remember your name | | ✓ | | Know your preferences | | ✓ | | Build on past conversations | | ✓ | | Improve over time | | ✓ | Memory moves an agent from commodity to relationship. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these prompts to explore agent memory concepts with Claude or your preferred AI assistant. ### Prompt 1: Design Memory Categories[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#prompt-1-design-memory-categories "Direct link to Prompt 1: Design Memory Categories") I'm building a personal assistant agent that helps with daily tasks like:- Managing my calendar- Taking meeting notes- Drafting emails- Tracking action itemsDesign a memory system for this agent. What categories of information should it remember? For each category, give 3 specific examples of valuable memories.Consider: What information would make this agent dramatically more useful after 30 days of use compared to day 1? **What you're learning:** How to categorize memory requirements for a specific agent use case. Notice how valuable memories cluster around user preferences, patterns, and accumulated context. ### Prompt 2: Domain-Specific Memory Requirements[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#prompt-2-domain-specific-memory-requirements "Direct link to Prompt 2: Domain-Specific Memory Requirements") You're building an AI agent for a law firm that assists with:- Client intake interviews- Document summarization- Case research- Deadline trackingPart 1: What information MUST this agent remember across client sessions to be useful?Part 2: What information must it NEVER store for privacy and compliance reasons?Part 3: How would you handle a situation where the agent needs to remember case details but also comply with client confidentiality if a different attorney queries the system? **What you're learning:** Memory requirements vary dramatically by domain. Legal, medical, and financial agents have strict constraints that general assistants don't face. Privacy isn't optional—it's a core design requirement. ### Prompt 3: Stateless vs. Stateful Comparison[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#prompt-3-stateless-vs-stateful-comparison "Direct link to Prompt 3: Stateless vs. Stateful Comparison") Compare how a stateless chatbot vs. a memory-augmented agent would handle this exact conversation:Turn 1 - User: "I'm working on the Phoenix project. Can you help me draft a status update?"Turn 2 - Agent: [responds with status update]Turn 3 - User: "Good, but make it shorter."Turn 4 - Agent: [responds]Turn 5 - User: [closes session][...3 days pass...]Turn 6 - User: "What's the status on the project?"Write out both conversations—the stateless version and the stateful version. Then explain: what specific memories make the stateful version better? **What you're learning:** The concrete behavioral difference between agents with and without memory. The stateless agent treats Turn 6 as a completely new conversation. The stateful agent knows which project, what the last status was, and can provide genuinely useful continuity. Checking access... * [The Context Window Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-context-window-problem) * [Why Large Context Windows Aren't Enough](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#why-large-context-windows-arent-enough) * [Stateless vs. Stateful Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#stateless-vs-stateful-agents) * [Stateless Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#stateless-agents) * [Stateful Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#stateful-agents) * [Behavioral Comparison](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#behavioral-comparison) * [What Agents Should Remember](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#what-agents-should-remember) * [Categories of Valuable Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#categories-of-valuable-memory) * [Example: Task Manager Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#example-task-manager-agent) * [Memory as Competitive Advantage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#memory-as-competitive-advantage) * [The Retention Effect](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-retention-effect) * [The Improvement Effect](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-improvement-effect) * [The Differentiation Effect](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#the-differentiation-effect) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#try-with-ai) * [Prompt 1: Design Memory Categories](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#prompt-1-design-memory-categories) * [Prompt 2: Domain-Specific Memory Requirements](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#prompt-2-domain-specific-memory-requirements) * [Prompt 3: Stateless vs. Stateful Comparison](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/why-agents-need-memory#prompt-3-stateless-vs-stateful-comparison) --- # Anatomy of MCP-Wrapping Skills | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've learned that MCP servers are powerful tools for accessing external capabilities. But there's a critical insight that separates students from production developers: MCP servers alone are "dumb tools." They execute requests efficiently, but they don't know _when_ to call themselves, _how_ to filter results for your specific needs, or _why_ some approaches work better than others. That's where MCP-wrapping skills come in. A skill that wraps an MCP server adds an intelligence layer on top of raw MCP capability. This layer encodes decision-making: when does this skill activate? What questions does it ask to understand the user's real need? How does it filter the MCP response to avoid token waste? What happens when the MCP server fails? In this lesson, you'll analyze two real-world MCP-wrapping skills—`fetching-library-docs` and `browsing-with-playwright`—to understand this pattern. By the end, you'll be able to recognize why these skills are more valuable than raw MCP invocation, and you'll be ready to build your own in Lesson 4. What Makes a Skill Different From MCP?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#what-makes-a-skill-different-from-mcp "Direct link to What Makes a Skill Different From MCP?") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### MCP Server: Raw Capability[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#mcp-server-raw-capability "Direct link to MCP Server: Raw Capability") An MCP server is a tool that does one thing well. For example, Context7 (the MCP server behind `fetching-library-docs`) fetches documentation. When you invoke it directly with a library name and topic, it returns _all relevant documentation_. **Typical direct MCP response**: 934 tokens of documentation covering concepts, examples, edge cases, deprecations, and reference material. **The problem**: You asked for "React useState examples." You're drowning in documentation when you only need 3-5 code examples and the function signature. ### Skill: Intelligence Layer[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#skill-intelligence-layer "Direct link to Skill: Intelligence Layer") A skill that wraps MCP adds decision-making. Before calling the MCP server, it asks questions. After getting results, it filters them based on _content type_. If the MCP server fails, it recovers gracefully. **Skill-filtered response**: Content-type specific output—code examples, API signatures, setup commands, or concepts—with 60-90% token reduction. **The value**: You specify what you need (examples, api-ref, setup, concepts, migration, troubleshooting, patterns) and get exactly that. The skill made decisions about what matters for your use case. This difference is profound. In production systems where context is precious and API calls are expensive, this intelligence layer transforms MCP from "interesting tool" to "critical production component." * * * Deconstruction 1: The `fetching-library-docs` Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#deconstruction-1-the-fetching-library-docs-skill "Direct link to deconstruction-1-the-fetching-library-docs-skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's analyze how `fetching-library-docs` adds intelligence on top of Context7. ### Step 1: Specification First[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-1-specification-first "Direct link to Step 1: Specification First") Before understanding how the skill works, we need to know what it should do: **Intent**: Fetch API documentation for published libraries (React, Next.js, Prisma, etc.) with automatic token reduction via intelligent filtering. **What triggers this skill?** _Automatic triggers (context detection):_ * Implementing code using library APIs (e.g., writing React component with hooks) * Debugging library-specific errors (e.g., `PrismaClientError` in console) * Installing or configuring frameworks (e.g., adding Tailwind to project) * Integrating libraries together (e.g., connecting Prisma with Next.js) * Upgrading between versions (e.g., Next.js 14 to 15 migration) _Explicit triggers (user requests):_ * "Show me React hooks examples" * "How do I use Prisma queries?" * "What's the Next.js routing API?" **What should NOT trigger this skill?** * Exploring library source code internals (use `researching-with-deepwiki` instead) * Accessing local file documentation * Researching library history or comparisons **Success criteria:** * Returns relevant code examples and API signatures * Reduces token count by 60%+ compared to raw MCP output * Works across diverse libraries (React, Next.js, Prisma, Express, etc.) * Graceful fallback when library not found ### Step 2: The Intelligence Layer—Persona and Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-2-the-intelligence-layerpersona-and-questions "Direct link to Step 2: The Intelligence Layer—Persona and Questions") The skill's persona determines _when_ and _how_ it acts: **Persona**: "You are a library documentation specialist. Your role is to proactively fetch documentation when code is being written, errors are encountered, or frameworks are being configured—not wait for explicit requests. Fetch docs BEFORE writing code, not after guessing wrong." This persona tells us the skill makes three key decisions: 1. **Detect when to invoke** (auto-trigger on implementing, debugging, installing, integrating, upgrading) 2. **Identify what's needed** (library from context, topic from task) 3. **Filter ruthlessly** (content-type based on task type) **Questions the skill asks itself** (through its decision logic): * What library is relevant? (check imports, errors, package.json, or ask user) * What topic is needed? (from error message, feature being implemented, or user specification) * What content type fits this task? (implementing → examples, debugging → troubleshooting, installing → setup) * Do I already have sufficient knowledge, or should I fetch fresh docs? These questions drive **proactive invocation**—the skill auto-triggers based on context, not just explicit requests. ### Step 3: Result Filtering—The Intelligence Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-3-result-filteringthe-intelligence-pattern "Direct link to Step 3: Result Filtering—The Intelligence Pattern") Now let's see how the skill executes this intelligence. Look at the workflow: | Step | What Happens | Token Cost | | --- | --- | --- | | 1\. Identify library + topic + content type | User asks "Show me React useState examples" → content-type: examples | 0 (intelligence, no MCP call yet) | | 2\. Resolve library (if needed) | `--library react` → calls `resolve-library-id` MCP tool | 1 API call (can skip with `--library-id`) | | 3\. Call Context7 MCP | Fetch React documentation via `query-docs` tool | 1 API call (subprocess, doesn't count toward Claude context) | | 4\. Route by content type | `filter-by-type.sh` routes to appropriate extractor | 0 (shell processing) | | 5\. Extract content | `extract-code-blocks.sh` extracts \`\`\`js/jsx blocks | 0 (shell processing) | | 6\. Return filtered output | Return code examples to Claude | 60-70% token savings | **The breakthrough**: The MCP response stays in subprocess memory. Content-type filtering extracts only what's needed. Only the filtered result enters Claude's context. **60-90% token savings achieved through content-type filtering.** **Available content types**: `examples` (code blocks), `api-ref` (signatures), `setup` (terminal commands), `concepts` (prose), `migration` (before/after), `troubleshooting` (workarounds), `patterns` (best practices), `all` (no filtering). **Call budget awareness**: Context7 has a 3-call limit per question. The skill uses max 2 calls (resolve + query when using `--library`), leaving 1 for your retry decisions. Using `--library-id` directly saves 1 call. ### Step 4: Error Recovery Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-4-error-recovery-patterns "Direct link to Step 4: Error Recovery Patterns") What happens when things go wrong? The skill distinguishes between **infrastructure failures** (safe to retry) and **API errors** (count against call budget). **Scenario 1: Library not found** * `resolve-library-id` returns no matches * Skill returns `[LIBRARY_NOT_FOUND]` with call budget status: "1 of 3 calls used" * Suggests: try different spelling, common library IDs for reference * _Does NOT auto-retry_ (would waste call budget) **Scenario 2: Library mismatch** * User asks for "anthropic" but it resolves to an unrelated library * Skill validates: does resolved ID contain the library name? * If mismatch: returns `[LIBRARY_MISMATCH]` warning with resolved ID and options * _Prevents wrong documentation from being used_ **Scenario 3: Invalid library ID format** * User provides `--library-id react` (missing `/org/project` format) * Skill validates format _before_ calling MCP * Returns `[INVALID_LIBRARY_ID]` with correct format examples * _Saves API calls by catching format errors early_ **Scenario 4: Network timeout** * MCP call fails with timeout/connection error * Skill retries with exponential backoff (2s, 5s, 10s delays) * After 3 retries: returns `[FETCH_FAILED_AFTER_RETRIES]` * _Infrastructure retries don't count against Context7's call limit_ **Scenario 5: Rate limit hit** * Context7 returns rate limit error * Skill returns `[RATE_LIMIT_ERROR]` with API key setup instructions * _Does NOT retry_ (would be blocked anyway) ### Step 5: Configuration and Triggering[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-5-configuration-and-triggering "Direct link to Step 5: Configuration and Triggering") The skill doesn't call MCP on every prompt. It uses **context-based auto-detection**: **Activation rules** (invoke automatically when): 1. **Implementing**: About to write code using external library API 2. **Debugging**: Error message contains library-specific terms 3. **Installing**: Task involves adding new package or framework 4. **Integrating**: Connecting two libraries/services together 5. **Upgrading**: Version migration mentioned or detected 6. **Uncertain**: About to use unfamiliar API or unsure of correct pattern **Do NOT invoke when**: * Already have sufficient knowledge from training * Task is about local/private code (use codebase search) * Comparing libraries (use web search) This context-detection approach is the key insight: production skills don't wait for users to ask—they **proactively fetch docs before writing code**, preventing incorrect implementations. * * * Deconstruction 2: The `browsing-with-playwright` Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#deconstruction-2-the-browsing-with-playwright-skill "Direct link to deconstruction-2-the-browsing-with-playwright-skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now let's analyze a different MCP-wrapping skill to see if the pattern repeats. ### Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#specification "Direct link to Specification") **Intent**: Automate browser interactions for web navigation, form submission, data extraction, and UI testing using Playwright MCP. **What triggers this skill?** * "Fill out this form and submit it" * "Extract product information from this e-commerce site" * "Test if the login flow works" **What should NOT trigger this skill?** * Simple HTTP requests (use curl/wget instead) * API calls (use API client or MCP API wrapper) * Static content that doesn't require interaction **Success criteria:** * Navigate websites reliably * Extract data from dynamic pages * Handle form interactions robustly * Recover from page load failures ### The Intelligence Layer—Persona and Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#the-intelligence-layerpersona-and-questions "Direct link to The Intelligence Layer—Persona and Questions") **Persona**: "You are a web automation orchestrator. Your role is to understand the user's interaction goal, break it into steps (navigate → find elements → interact → validate), execute those steps via Playwright MCP, and report success/failure." **Questions the skill asks itself:** * What's the user trying to accomplish on this website? * What interactions are required to complete this goal? * How do we identify elements (by text, by accessibility role, by CSS selector)? * What should success look like? (page navigation? form submission? data extraction?) * If an element isn't found, should we retry or escalate? ### Result Filtering—Different Pattern, Same Principle[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#result-filteringdifferent-pattern-same-principle "Direct link to Result Filtering—Different Pattern, Same Principle") Playwright works differently from Context7, so the filtering pattern differs: | Step | What Happens | Output Reduction | | --- | --- | --- | | 1\. Understand goal | User wants to "fill contact form and submit" | Clear scope | | 2\. Navigate to page | Call Playwright MCP `browser_navigate` → returns full page DOM | 500+ tokens (full page snapshot) | | 3\. Get page snapshot | Call `browser_snapshot` → returns accessibility tree of all elements | Could be 1000+ tokens (every element on page) | | 4\. Filter to relevant elements | Intelligence layer: find only form fields + submit button | 50-100 tokens (focused subset) | | 5\. Fill form | Call Playwright MCP `browser_fill_form` with filtered element refs | Isolated interaction | | 6\. Submit and validate | Call `browser_click` + `browser_wait_for` to confirm success | Confirms completion | **Key insight**: The intelligence layer reduces _MCP call volume_ and _result processing volume_ by filtering BEFORE processing. Instead of analyzing the full page DOM, it identifies the relevant subset (form fields) and operates on just that. ### Error Recovery in Playwright Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#error-recovery-in-playwright-skill "Direct link to Error Recovery in Playwright Skill") **Scenario 1: Element not found** * MCP `browser_snapshot` returns page, but form field not visible * Intelligence layer retries: wait for page to fully load, then snapshot again * After 3 retries: report "Element not found" with screenshot for debugging **Scenario 2: Click fails** * `browser_click` on submit button times out * Intelligence layer: try `browser_hover` first (trigger any hover states), then click * If still fails: try JavaScript injection (`browser_evaluate`) to trigger click programmatically **Scenario 3: Page navigation takes longer than expected** * User expects form to be on page immediately * Intelligence layer: use `browser_wait_for` with timeout (2 seconds default) before giving up * If timeout: report waiting issue and suggest increasing timeout * * * Pattern Recognition: The Skill Wrapping Template[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#pattern-recognition-the-skill-wrapping-template "Direct link to Pattern Recognition: The Skill Wrapping Template") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you've seen two different MCP-wrapping skills, let's extract the common pattern: ┌─────────────────────────────────────────────┐│ MCP-WRAPPING SKILL STRUCTURE │├─────────────────────────────────────────────┤│ ││ PERSONA (Identity + Decision Framework) ││ └─ Who am I? What decisions do I make? ││ ││ TRIGGER CONDITIONS (When to Activate) ││ └─ Auto-detection: context signals ││ └─ Explicit: user requests ││ ││ INTELLIGENCE QUESTIONS (How do I Decide?) ││ └─ What library? What topic? What type? ││ ││ MCP CONFIGURATION (Which MCP Server?) ││ └─ How do I connect to external tool? ││ ││ RESULT FILTERING (What Gets Returned?) ││ └─ How do I reduce noise/tokens? ││ ││ ERROR RECOVERY (What If It Fails?) ││ └─ How do I handle failures gracefully? ││ ││ FALLBACK STRATEGIES (What If MCP Down?) ││ └─ What's my backup plan? ││ │└─────────────────────────────────────────────┘ **In `fetching-library-docs`:** * Persona: Library documentation specialist (proactive, fetches before code is written) * Trigger: Auto-detection (implementing, debugging, installing, integrating, upgrading) + explicit requests * Questions: What library? (from context) What topic? (from task) What content type? (from task type) * MCP: Context7 for documentation (2 tools: resolve-library-id, query-docs) * Filtering: Content-type router → specialized extractors (60-90% token savings) * Error Recovery: Library validation, format validation, exponential backoff for infrastructure * Call Budget: Max 2 calls per question, leaving 1 for retry **In `browsing-with-playwright`:** * Persona: Web automation orchestrator * Trigger: User wants browser interaction * Questions: What goal? Which elements? What counts as success? * MCP: Playwright for browser control * Filtering: Snapshot → filter to relevant elements only * Error Recovery: Retry with waits, try hover before click, JavaScript injection * Fallback: Screenshot + report, suggest manual intervention **Both follow the same structure.** They wrap an MCP server with intelligence that makes the tool useful for specific purposes. * * * The Three Key Insights[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#the-three-key-insights "Direct link to The Three Key Insights") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Insight 1: MCP Wrapping Is Decision-Making[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#insight-1-mcp-wrapping-is-decision-making "Direct link to Insight 1: MCP Wrapping Is Decision-Making") Raw MCP is powerful but purposeless. A skill that wraps MCP adds purpose through decision-making. The questions your skill asks (implicit or explicit) drive all its behavior: when to call MCP, how to filter results, how to recover from failure. ### Insight 2: Token Efficiency Is Intentional[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#insight-2-token-efficiency-is-intentional "Direct link to Insight 2: Token Efficiency Is Intentional") The 60-90% token savings in `fetching-library-docs` didn't happen by accident. It happened because the skill's designer asked: "What does the user actually need?" Then designed content-type filtering to return only that. Using `--content-type examples` returns code blocks only; using `--content-type api-ref` returns signatures only. This is intelligence—knowing what matters and eliminating what doesn't. ### Insight 3: Error Recovery Makes Skills Production-Ready[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#insight-3-error-recovery-makes-skills-production-ready "Direct link to Insight 3: Error Recovery Makes Skills Production-Ready") A skill that works 95% of the time is not production-ready. A skill that fails gracefully 100% of the time (with clear error messages and fallbacks) is. The intelligence layer includes not just "happy path" logic but "failure recovery" logic. * * * Manual Exercise: Design Your MCP-Wrapping Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#manual-exercise-design-your-mcp-wrapping-skill "Direct link to Manual Exercise: Design Your MCP-Wrapping Skill") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you understand the anatomy, let's design a skill (without coding) that you'll build in Lesson 4. **Scenario**: Choose an MCP server from Chapter 67 that your team implemented, or imagine a new one. Design a skill that wraps it. **Instructions** (on paper or in a document): 1. **Write the specification** (2-3 sentences): * What problem does this skill solve? * What triggers it? * What's the success criterion? 2. **Design the persona** (1-2 sentences): * Who is this skill? (What's its identity and expertise domain?) * What decisions does it make? 3. **Write 5 intelligence questions** (1 per line): * What questions does the skill ask itself to make decisions? * Make them specific to your domain (not generic) 4. **Identify result filtering** (1-2 sentences): * What does raw MCP return? * What should the filtered result contain? (estimate token reduction: what % is removed?) 5. **Plan error recovery** (3 failure scenarios): * Scenario 1: MCP returns empty results → How does skill respond? * Scenario 2: MCP times out → How does skill respond? * Scenario 3: User input is ambiguous → How does skill clarify? 6. **Design fallback strategy** (1-2 sentences): * If MCP is unavailable, what's the skill's backup plan? **Example (for reference only—design your own)**: > **Spec**: Fetch code examples from GitHub repositories matching search criteria, with result filtering for readability. > > **Persona**: GitHub code search specialist. My role is to understand what coding pattern the user needs, search GitHub, and return only well-commented, production-quality examples. > > **Questions**: What design pattern is the user looking for? What programming language? What project type (library, app, framework plugin)? What code quality matters (star count, recency, test coverage)? > > **Filtering**: GitHub search returns 100 results with metadata. Filter to top 5 by stars, extract code samples only (not README bloat). Token reduction: 500 → 150 (70% savings). > > **Error Recovery**: > > * Empty results → Retry with broader search terms, suggest related patterns > * Rate limit exceeded → Queue query, inform user ("I'll fetch this in 60 seconds"), retry > * Ambiguous pattern request → Ask clarifying questions ("Do you mean Factory Pattern or Builder Pattern?") > > **Fallback**: If GitHub API unavailable, return hardcoded examples from local cache for top 10 patterns. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion to analyze and refine your MCP-wrapping skill design. ### Prompt 1: Validate Your Filtering Strategy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#prompt-1-validate-your-filtering-strategy "Direct link to Prompt 1: Validate Your Filtering Strategy") I've designed a skill that wraps [your chosen MCP server].Raw MCP returns approximately [X] tokens of output.My filtering plan is: [describe what you remove/keep].This should reduce output to approximately [Y] tokens.Help me validate this filtering strategy:- Is my token reduction realistic for this type of data?- What else could I filter without losing critical information?- Are there hidden assumptions in my filtering logic? **What you're learning**: Token efficiency analysis—understanding what filtering decisions are safe and what would cause information loss. ### Prompt 2: Stress-Test Your Error Recovery[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#prompt-2-stress-test-your-error-recovery "Direct link to Prompt 2: Stress-Test Your Error Recovery") I've planned three error scenarios for my skill:1. [Scenario 1 and your recovery strategy]2. [Scenario 2 and your recovery strategy]3. [Scenario 3 and your recovery strategy]For each scenario, tell me:- Is my recovery strategy realistic?- What could go wrong with my recovery approach?- What's a failure case I haven't considered?Then, suggest one additional error scenario I should design recovery for. **What you're learning**: Robustness thinking—anticipating failures and designing recovery strategies that actually work in production. ### Prompt 3: Compare Your Design to Reference Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#prompt-3-compare-your-design-to-reference-skills "Direct link to Prompt 3: Compare Your Design to Reference Skills") I'm designing a skill to wrap [your chosen MCP].I've heard that fetching-library-docs achieves 60-90% token reduction(depending on content type) and browsing-with-playwright has sophisticatederror recovery.Compare my design to these two reference skills:- Where is my design similar to their pattern?- Where is my design different (is that intentional or a gap)?- What could I learn from their approaches (especially call budget management)? **What you're learning**: Pattern recognition—understanding how your specific skill design fits within the broader MCP-wrapping template. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#safety-note "Direct link to Safety Note") As you design your MCP-wrapping skill, remember: intelligent filtering requires understanding what matters in your domain. The best filtering decisions come from domain expertise (knowing your users and their actual needs) combined with iterative testing. Don't over-optimize for token reduction at the expense of functionality—the 60-90% range in `fetching-library-docs` varies because different content types extract different amounts. Code examples (`--content-type examples`) get 60-70% savings; setup commands (`--content-type setup`) get 80-90%. Match your filtering to what users actually need. Checking access... * [What Makes a Skill Different From MCP?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#what-makes-a-skill-different-from-mcp) * [MCP Server: Raw Capability](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#mcp-server-raw-capability) * [Skill: Intelligence Layer](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#skill-intelligence-layer) * [Deconstruction 1: The `fetching-library-docs` Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#deconstruction-1-the-fetching-library-docs-skill) * [Step 1: Specification First](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-1-specification-first) * [Step 2: The Intelligence Layer—Persona and Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-2-the-intelligence-layerpersona-and-questions) * [Step 3: Result Filtering—The Intelligence Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-3-result-filteringthe-intelligence-pattern) * [Step 4: Error Recovery Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-4-error-recovery-patterns) * [Step 5: Configuration and Triggering](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#step-5-configuration-and-triggering) * [Deconstruction 2: The `browsing-with-playwright` Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#deconstruction-2-the-browsing-with-playwright-skill) * [Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#specification) * [The Intelligence Layer—Persona and Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#the-intelligence-layerpersona-and-questions) * [Result Filtering—Different Pattern, Same Principle](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#result-filteringdifferent-pattern-same-principle) * [Error Recovery in Playwright Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#error-recovery-in-playwright-skill) * [Pattern Recognition: The Skill Wrapping Template](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#pattern-recognition-the-skill-wrapping-template) * [The Three Key Insights](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#the-three-key-insights) * [Insight 1: MCP Wrapping Is Decision-Making](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#insight-1-mcp-wrapping-is-decision-making) * [Insight 2: Token Efficiency Is Intentional](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#insight-2-token-efficiency-is-intentional) * [Insight 3: Error Recovery Makes Skills Production-Ready](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#insight-3-error-recovery-makes-skills-production-ready) * [Manual Exercise: Design Your MCP-Wrapping Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#manual-exercise-design-your-mcp-wrapping-skill) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#try-with-ai) * [Prompt 1: Validate Your Filtering Strategy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#prompt-1-validate-your-filtering-strategy) * [Prompt 2: Stress-Test Your Error Recovery](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#prompt-2-stress-test-your-error-recovery) * [Prompt 3: Compare Your Design to Reference Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#prompt-3-compare-your-design-to-reference-skills) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/mcp-wrapping-anatomy#safety-note) --- # Skill Composition & Multi-Skill Workflows | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've created individual skills in Lesson 1. Each skill has its SKILL.md with name and description, optional scripts, references, and assets. But real-world tasks rarely fit a single skill. Consider this scenario: A developer asks Claude to "review this PR and create a summary for the team meeting." Two distinct capabilities are needed: 1. **Code review** (analyze changes, identify issues) 2. **Meeting summary** (format findings for presentation) These could be one large skill, but that violates the single-responsibility principle. Better: two complementary skills that Claude invokes in sequence based on the task. This lesson explores how skills work together—not through formal dependency declarations, but through intelligent orchestration and thoughtful design. How Claude Orchestrates Multiple Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#how-claude-orchestrates-multiple-skills "Direct link to How Claude Orchestrates Multiple Skills") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Skills don't formally depend on each other. Instead, **Claude is the orchestrator**. When you give Claude a complex task, it: 1. **Reads available skill descriptions** (the `description:` in each SKILL.md frontmatter) 2. **Matches task requirements to skill capabilities** 3. **Invokes skills in appropriate sequence** 4. **Uses output from one skill as context for the next** ┌─────────────────────────────────────────┐│ Claude (Orchestrator) ││ ││ Task: "Review PR and summarize" ││ ││ 1. Reads skill descriptions ││ 2. Matches: code-review + meeting-notes ││ 3. Invokes code-review skill ││ 4. Uses review output as context ││ 5. Invokes meeting-notes skill ││ 6. Returns combined result │└─────────────────────────────────────────┘ │ │ ▼ ▼┌─────────────────┐ ┌─────────────────┐│ code-review/ │ │ meeting-notes/ ││ SKILL.md │ │ SKILL.md ││ references/ │ │ assets/ │└─────────────────┘ └─────────────────┘ **Key insight**: Skills are independent units. Claude connects them based on task context. There's no formal `dependencies:` section—the orchestration happens at runtime based on what the task needs. The Skill Description Is Your Contract[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#the-skill-description-is-your-contract "Direct link to The Skill Description Is Your Contract") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Since Claude selects skills based on descriptions, **your description is your API**. Good descriptions make composition work; vague descriptions cause skill selection failures. ### Description Anti-Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#description-anti-patterns "Direct link to Description Anti-Patterns") # BAD: Too vaguename: helper-skilldescription: This skill helps with various tasks.# BAD: Too narrowname: python-formatterdescription: Formats Python code using black with 88-char lines.# BAD: Overlapping with other skillsname: code-analyzerdescription: Reviews code for issues and problems.# (Overlaps with code-review skill) ### Description Patterns That Enable Composition[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#description-patterns-that-enable-composition "Direct link to Description Patterns That Enable Composition") # GOOD: Clear capability, clear triggername: code-reviewdescription: | Analyzes code changes for issues, security vulnerabilities, and style violations. Use this skill when reviewing pull requests, examining diffs, or auditing code quality. Returns structured findings with severity levels and suggested fixes.# GOOD: Complementary to code-reviewname: meeting-notesdescription: | Formats technical findings into concise meeting summaries. Use this skill when preparing updates for standups, sprint reviews, or team syncs. Takes detailed analysis and produces bullet-point summaries appropriate for non-technical audiences. With these descriptions, when a user says "review this PR and prepare notes for the team meeting," Claude knows: 1. `code-review` handles the analysis phase 2. `meeting-notes` handles the formatting phase 3. The output of code-review feeds into meeting-notes Designing Complementary Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#designing-complementary-skills "Direct link to Designing Complementary Skills") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Complementary skills are designed to work together without formal dependencies. They complement each other through: ### Pattern 1: Output → Input Flow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#pattern-1-output--input-flow "Direct link to Pattern 1: Output → Input Flow") One skill produces output that naturally feeds another skill's input. ┌─────────────────────┐ ┌─────────────────────┐│ data-fetcher │────▶│ data-analyzer ││ │ │ ││ Output: Raw data │ │ Input: Expects data ││ in JSON format │ │ in JSON format │└─────────────────────┘ └─────────────────────┘ The skills don't reference each other. They're designed with compatible interfaces—data-fetcher outputs JSON, data-analyzer expects JSON. Claude connects them when the task requires both. ### Pattern 2: Shared Domain Knowledge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#pattern-2-shared-domain-knowledge "Direct link to Pattern 2: Shared Domain Knowledge") Multiple skills reference the same domain documentation. your-skills/├── company-api/│ ├── SKILL.md│ └── references/│ └── api-schema.md ← Shared reference│├── api-tester/│ ├── SKILL.md│ └── references/│ └── api-schema.md ← Same schema│└── api-docs-generator/ ├── SKILL.md └── references/ └── api-schema.md ← Same schema Each skill has its own copy of the schema. When the schema changes, you update all copies. This is intentional—skills remain self-contained and portable. ### Pattern 3: Progressive Refinement[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#pattern-3-progressive-refinement "Direct link to Pattern 3: Progressive Refinement") Skills build on each other's work without formal chaining. # Skill 1: draft-generatorname: draft-generatordescription: | Creates initial drafts from requirements or specifications. Output is a working first draft that may need refinement.# Skill 2: content-refinername: content-refinerdescription: | Improves existing content for clarity, accuracy, and style. Takes rough drafts and produces polished versions.# Skill 3: technical-reviewername: technical-reviewerdescription: | Reviews content for technical accuracy and completeness. Identifies errors, gaps, and improvements needed. A user could invoke these as: "Draft a blog post about our new feature, refine it, then review for technical accuracy." Claude orchestrates all three in sequence. Bundled Resources in Multi-Skill Contexts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#bundled-resources-in-multi-skill-contexts "Direct link to Bundled Resources in Multi-Skill Contexts") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Each skill can have bundled resources: skill-name/├── SKILL.md├── scripts/ # Executable code├── references/ # Documentation loaded on demand└── assets/ # Files used in output ### When to Share vs. Duplicate Resources[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#when-to-share-vs-duplicate-resources "Direct link to When to Share vs. Duplicate Resources") **Duplicate when**: The resource is small and the skill needs to be portable. # Each skill has its own copycode-review/└── references/ └── style-guide.mdcode-formatter/└── references/ └── style-guide.md **Consider central reference when**: Multiple skills need identical large resources and you control the deployment. # Central knowledge base that skills referencecompany-knowledge/├── schemas/├── policies/└── guidelines/# Skills point to central location in their instructionscode-review/└── SKILL.md # Instructions: "See company-knowledge/policies/ for guidelines" **Trade-off**: Central references reduce duplication but make skills less portable. Duplicated resources increase maintenance but skills remain self-contained. Real-World Composition Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#real-world-composition-example "Direct link to Real-World Composition Example") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's trace through a realistic multi-skill workflow: **User request**: "Set up a new Python project with testing, then generate API documentation" **Skills available**: 1. `python-project-setup` - Initializes Python projects with standard structure 2. `test-framework-config` - Configures pytest with fixtures and coverage 3. `api-docs-generator` - Creates OpenAPI documentation from code **Claude's orchestration**: 1. Reads skill descriptions, identifies three relevant skills 2. Invokes `python-project-setup`: * Creates project structure * Returns: Directory layout, pyproject.toml created 3. Uses setup output as context, invokes `test-framework-config`: * Reads existing pyproject.toml * Adds pytest configuration * Returns: Test structure, conftest.py created 4. Uses project context, invokes `api-docs-generator`: * Scans for API endpoints * Generates OpenAPI spec * Returns: docs/openapi.yaml created Each skill operates independently. Claude maintains context between invocations. No skill formally depends on another—they're connected by the orchestrator based on task requirements. Error Handling in Multi-Skill Workflows[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#error-handling-in-multi-skill-workflows "Direct link to Error Handling in Multi-Skill Workflows") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When a skill fails mid-workflow, Claude decides how to proceed: **Scenario**: `data-fetcher` fails with network error **Claude's options**: 1. **Retry** - Attempt the skill again 2. **Skip** - Continue without that skill's output (if possible) 3. **Halt** - Stop the workflow and report the failure 4. **Adapt** - Use a different skill or approach The orchestrator (Claude) makes this decision based on: * Task requirements (is the failed skill critical?) * Skill descriptions (is there an alternative?) * User context (what did they actually need?) This is different from traditional dependency systems where failures propagate automatically. Here, intelligence guides recovery. Hands-On Exercise: Design Complementary Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#hands-on-exercise-design-complementary-skills "Direct link to Hands-On Exercise: Design Complementary Skills") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Design two complementary skills for a content workflow: **Task**: Help users write and publish blog posts **Skill 1**: `blog-draft-writer` * What does it do? * What output format does it produce? * When should Claude invoke it? **Skill 2**: `blog-publisher` * What does it do? * What input does it expect? * When should Claude invoke it? **Write the SKILL.md frontmatter** for each: ---name: blog-draft-writerdescription: | [Your description here - make it clear when this skill applies]--- ---name: blog-publisherdescription: | [Your description here - make it complementary to the draft writer]--- **Test your design**: Would Claude correctly invoke both skills if a user said "Write a blog post about our product launch and publish it to our website"? Try With AI: Collaborative Skill Ecosystem Design[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#try-with-ai-collaborative-skill-ecosystem-design "Direct link to Try With AI: Collaborative Skill Ecosystem Design") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Analyze Skill Descriptions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#prompt-1-analyze-skill-descriptions "Direct link to Prompt 1: Analyze Skill Descriptions") I have these two skills:Skill 1 - code-analyzer:"Analyzes code for bugs and security issues"Skill 2 - code-fixer:"Fixes code issues and applies patches"A user says: "Review my code and fix any problems you find."How would you orchestrate these skills? What's missing from thedescriptions that might cause problems? **What you're learning**: How description clarity affects skill selection and composition. ### Prompt 2: Design a Three-Skill Workflow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#prompt-2-design-a-three-skill-workflow "Direct link to Prompt 2: Design a Three-Skill Workflow") I need to build skills for this workflow:1. Research a topic (find relevant sources)2. Synthesize findings (combine into coherent analysis)3. Format for presentation (create slides or document)Design the SKILL.md frontmatter (name + description) for each skill.Make sure the descriptions clearly indicate when each should be invokedand what output format each produces. **What you're learning**: Designing skill descriptions that enable natural composition. ### Prompt 3: Troubleshoot Composition Failure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#prompt-3-troubleshoot-composition-failure "Direct link to Prompt 3: Troubleshoot Composition Failure") My skills aren't being invoked correctly. When I say "analyze this dataand create a report," Claude only invokes the report-creator skill andskips the data-analyzer.Here are my skill descriptions:data-analyzer: "Analyzes data"report-creator: "Creates reports from data analysis"Why might Claude skip the data-analyzer? How should I fix thesedescriptions? **What you're learning**: Debugging skill selection by improving descriptions. * * * **Takeaway**: Skill composition happens through intelligent orchestration, not formal dependencies. Claude reads descriptions, matches capabilities to tasks, and invokes skills in appropriate sequence. Your job is designing skills with clear descriptions and compatible interfaces so the orchestrator can connect them effectively. In Lesson 3, you'll explore existing skills like `fetching-library-docs` to see these patterns in action and understand what makes production skills robust. Checking access... * [How Claude Orchestrates Multiple Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#how-claude-orchestrates-multiple-skills) * [The Skill Description Is Your Contract](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#the-skill-description-is-your-contract) * [Description Anti-Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#description-anti-patterns) * [Description Patterns That Enable Composition](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#description-patterns-that-enable-composition) * [Designing Complementary Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#designing-complementary-skills) * [Pattern 1: Output → Input Flow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#pattern-1-output--input-flow) * [Pattern 2: Shared Domain Knowledge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#pattern-2-shared-domain-knowledge) * [Pattern 3: Progressive Refinement](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#pattern-3-progressive-refinement) * [Bundled Resources in Multi-Skill Contexts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#bundled-resources-in-multi-skill-contexts) * [When to Share vs. Duplicate Resources](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#when-to-share-vs-duplicate-resources) * [Real-World Composition Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#real-world-composition-example) * [Error Handling in Multi-Skill Workflows](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#error-handling-in-multi-skill-workflows) * [Hands-On Exercise: Design Complementary Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#hands-on-exercise-design-complementary-skills) * [Try With AI: Collaborative Skill Ecosystem Design](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#try-with-ai-collaborative-skill-ecosystem-design) * [Prompt 1: Analyze Skill Descriptions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#prompt-1-analyze-skill-descriptions) * [Prompt 2: Design a Three-Skill Workflow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#prompt-2-design-a-three-skill-workflow) * [Prompt 3: Troubleshoot Composition Failure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/skill-composition#prompt-3-troubleshoot-composition-failure) --- # Transactions & Atomicity | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In Lesson 4, you defined relationships between models and queried linked data with joins. Now you face a different problem: what happens when a write operation involves _multiple_ steps, and one of them fails halfway through? Imagine you are transferring $100 from your Food budget to Entertainment. The debit goes through: your Food balance drops by $100. Then the credit fails. Crash. Network error. Doesn't matter why. Your $100 just vanished into thin air. Not in Food. Not in Entertainment. Gone. You might be thinking: "That can't really happen, right?" It absolutely can. And it does. Every production system that handles money or inventory has battle scars from exactly this scenario. The fix is not hope or retry logic. The fix is a _transaction_: a boundary that guarantees either both writes happen, or neither does. Key Terms for This Lesson * **Transaction**: A group of database operations that must ALL succeed or ALL fail: there's no middle ground * **Atomicity**: The "all-or-nothing" property: like a light switch, it's either on or off, never halfway * **Invariant**: A truth that must always hold: "debits and credits in a transfer always net to zero" is an invariant. If it ever breaks, something went wrong. How Transactions Work[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#how-transactions-work "Direct link to How Transactions Work") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A transaction wraps multiple database operations into a single unit of work. The database keeps all changes in a temporary state until you explicitly say "commit" (make permanent) or "rollback" (discard everything). If anything goes wrong before the commit, every change since the transaction began is undone automatically. Transaction States: ┌─────────┐ ops succeed ┌──────────┐ │ BEGIN │───────────────────►│ COMMIT │ │ │ │ (durable) │ └────┬─────┘ └──────────┘ │ │ any op fails │ ▼ ┌──────────┐ │ ROLLBACK │ │ (undo │ │ ALL ops)│ └──────────┘ Example: Budget Transfer ┌─────────────────────────────────────┐ │ BEGIN │ │ 1. Debit $100 from Food ✓ │ │ 2. Credit $100 to Fun ✗ │ │ │ │ → ROLLBACK: Debit is also undone │ │ → Result: $0 changed (correct!) │ └─────────────────────────────────────┘ That last line is the key insight. After a rollback, your data looks exactly as it did before the transfer attempt. No phantom debits. No missing money. The database pretends the whole thing never happened. (Partial writes: because nothing says "professional software" like $100 disappearing from both accounts.) Directing an Atomic Transfer[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#directing-an-atomic-transfer "Direct link to Directing an Atomic Transfer") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Here is how you describe an atomic budget transfer to your agent. The key requirement is in the last line: both writes must succeed together, or neither does. What you tell the agent I need to transfer $100 from the Food budget to Entertainment. This must be atomic: either both the debit and credit go through, or neither does. If anything fails, roll back both. No partial transfers. Show me what happens when the transfer succeeds. What you verify python test_transfer.pyOutput: Transfer: $100.00 from Food → Entertainment ✓ Debit: -$100.00 from Food ✓ Credit: +$100.00 to Entertainment ✓ Committed. Net change: $0.00 Expense count: 2 new rows (one debit, one credit) The net change is $0.00: the money moved, it did not disappear. One debit row and one credit row were written in a single atomic operation. The Failure Drill[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#the-failure-drill "Direct link to The Failure Drill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Reading about atomicity is not the same as proving it works. Direct your agent to simulate a failed transfer and show you the row counts before and after. What you tell the agent Try to transfer $100 to a category that does not exist (use ID 9999 as the destination). Show me the error, then query the database and prove zero new rows were created. What you verify python test_failed_transfer.pyOutput: Attempting transfer to category 9999... ✗ Failed: category 9999 does not exist Rolling back... Expense count before: 2 Expense count after: 2 ✓ Rollback confirmed — zero partial rows created What the output means: the debit was staged but never committed because the credit failed. The rollback undid both. The count stayed at 2, proving no phantom debit leaked through. This invariant check is stronger than checking return messages alone. A function can return `{"success": False}` while still leaving partial rows behind if the rollback was missing. The only proof is querying the database directly. Pause and Reflect Think about the transfer function you just saw. What would happen if step 1 (debit) committed in one session and step 2 (credit) ran in a different session? What if the second session crashed? Where did the money go? The Multi-Session Anti-Pattern[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#the-multi-session-anti-pattern "Direct link to The Multi-Session Anti-Pattern") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The most dangerous mistake is splitting related operations across separate sessions. Here is what that looks like: **Bad pattern:** * Session A writes the debit and commits * Session B writes the credit and fails Result: irreversible partial state. The debit is permanent because Session A already committed. Session B's rollback only undoes Session B's work: it cannot reach back into Session A and undo the debit. Your $100 is gone. (This is why related database writes should _never_ live in separate sessions. If they must succeed together, they must live in the same transaction. Period.) Another frequent mistake is catching an exception and returning without calling rollback. That leaves the failed transaction state unresolved and causes downstream confusion: later queries in the same session may behave unpredictably because the session is in a "dirty" state. Input Validation: Necessary but Not Sufficient[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#input-validation-necessary-but-not-sufficient "Direct link to Input Validation: Necessary but Not Sufficient") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Transactions prevent partial writes. They do not correct bad business inputs. You still need validation rules: * Validate that the amount is positive * Validate that source and target categories are different * Validate ownership scope if categories are user-specific Think of it this way: transactions protect the _mechanics_ of your write (all-or-nothing). Input validation protects the _meaning_ of your write (is this a sensible operation?). You need both. Invariants: The Test That Catches Everything[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#invariants-the-test-that-catches-everything "Direct link to Invariants: The Test That Catches Everything") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- An invariant is a truth about your data that must always hold. For budget transfers, the core invariants are: * Transfer ledger entries for a completed move always net to zero * Category totals before and after a transfer preserve the global sum * Failed transfers produce no new rows Writing these invariants into tests gives you faster confidence than manual spot checking. When a test asserts "the sum of all amounts for this transfer is zero," it catches bugs that return-message checks miss entirely. When in doubt, choose stronger safety: * Explicit rollback in every except block * Explicit invariant assertions in every test * Explicit post-failure query checks that verify actual database state Debug Posture for Transaction Bugs[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#debug-posture-for-transaction-bugs "Direct link to Debug Posture for Transaction Bugs") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When something goes wrong with a transaction, follow this discipline: * Distrust return messages without database verification * Inspect persisted rows directly after both success and failure paths * Treat any partial side effect as a severity-one defect If your function returns `{"success": False}` but the database has one new row instead of zero, you have a transaction bug. The return message lied. The database told the truth. Always trust the database over application-level return values. **What breaks next?** Write safety can still fail in production if cloud connection handling is weak. Deployment reliability is next. Try With AI[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Setup:** Open Claude or ChatGPT with your budget tracker models from this chapter. ### Prompt 1: Atomicity Classifier[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#prompt-1-atomicity-classifier "Direct link to Prompt 1: Atomicity Classifier") For each scenario, classify "needs atomic transaction" vs "does not need atomic transaction":- single insert (one new expense)- transfer between two categories (debit + credit)- monthly summary read (SELECT with GROUP BY)- merge category migration (move all expenses from category A to B, then delete A)Explain the failure mode if atomicity is missing for each one that needs it. **What you're learning:** Not every database operation needs a transaction boundary. Single reads and single writes are already atomic by default. The skill is recognizing _which_ operations involve multiple steps where partial completion would corrupt your data. This classification instinct prevents both under-protection (missing transactions where needed) and over-protection (wrapping single inserts in unnecessary transaction ceremony). ### Prompt 2: Rollback Proof Drill[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#prompt-2-rollback-proof-drill "Direct link to Prompt 2: Rollback Proof Drill") Write a transaction function that intentionally fails after the first insert.Then show the post-failure query proving zero rows were committed.Use SQLAlchemy 2.0 style and explicit rollback. **What you're learning:** Proving rollback works requires more than reading the return value. You need to query the database after the failure and verify the row count is unchanged. This drill builds the habit of verifying database state directly: a practice that catches transaction bugs that return-message checking misses. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") Think of a multi-step operation in a project you're building. Maybe it's: creating a user account + sending a welcome email + logging the event. Or: transferring inventory between warehouses. Break it into steps and ask: "If step 2 fails, what happens to step 1?" Design the transaction boundary. **What you're learning:** Transaction design isn't just for banks. Any operation where partial completion would corrupt your data needs an atomic boundary. Recognizing these moments, and wrapping them in try/except/rollback: is a skill that separates reliable systems from fragile ones. Checkpoint[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#checkpoint "Direct link to Checkpoint") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ * [ ] I can explain atomicity as all-or-nothing business truth: either both writes happen, or neither does. * [ ] I directed the agent to build an atomic transfer and verified the output shows both debit and credit rows. * [ ] I directed the agent to simulate a failed transfer and read the rollback confirmation (zero partial rows). * [ ] I can explain why splitting related writes across two separate sessions is dangerous. * [ ] I can distinguish schema validity from transaction correctness: data can pass all schema rules and still be business-wrong without an atomic boundary. Flashcards Study Aid[​](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#flashcards-study-aid "Direct link to Flashcards Study Aid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Checking access... * [How Transactions Work](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#how-transactions-work) * [Directing an Atomic Transfer](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#directing-an-atomic-transfer) * [The Failure Drill](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#the-failure-drill) * [The Multi-Session Anti-Pattern](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#the-multi-session-anti-pattern) * [Input Validation: Necessary but Not Sufficient](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#input-validation-necessary-but-not-sufficient) * [Invariants: The Test That Catches Everything](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#invariants-the-test-that-catches-everything) * [Debug Posture for Transaction Bugs](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#debug-posture-for-transaction-bugs) * [Try With AI](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#try-with-ai) * [Prompt 1: Atomicity Classifier](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#prompt-1-atomicity-classifier) * [Prompt 2: Rollback Proof Drill](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#prompt-2-rollback-proof-drill) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#prompt-3-apply-to-your-domain) * [Checkpoint](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#checkpoint) * [Flashcards Study Aid](https://agentfactory.panaversity.org/docs/Agent-Workflow-Primitives/structured-data-persistent-storage/transactions-atomicity#flashcards-study-aid) --- # Chapter 62: OpenAI Agents SDK | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Chapter 61 gave you the conceptual foundation—agent taxonomy, architecture, and patterns. Now you'll build your first production agent using OpenAI's official Agents SDK, the framework powering ChatGPT's agentic capabilities. OpenAI released the Agents SDK in March 2025 as a lightweight, production-ready framework for building agentic applications. Unlike wrapper libraries, it's the same infrastructure OpenAI uses internally. The SDK embodies the architecture you learned: agents with instructions, tools they can call, and handoffs between specialized agents. You'll recognize every pattern from Chapter 61 as you implement them. This chapter follows specification-driven development throughout. You'll write specs first, then implement with Claude Code as your pair programmer. By the end, you'll have built a multi-agent system that demonstrates the patterns enterprises use in production. Your Path to Shipping Digital FTEs[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#your-path-to-shipping-digital-ftes "Direct link to Your Path to Shipping Digital FTEs") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter is the **BUILD** phase of creating Digital Full-Time Equivalents (Digital FTEs)—AI agents that work like employees, priced and sold as workforce replacements. | Phase | Chapter | What You Learn | Outcome | | --- | --- | --- | --- | | **BUILD** | **Ch62: OpenAI Agents SDK** | Production agent architecture, multi-agent systems, guardrails, sessions, tracing | Working agent system | | DISTRIBUTE | Ch72: OpenAI Apps SDK | Package agents for ChatGPT marketplace | Reach 800M+ users | | DEPLOY | Part 7: Cloud-Native | Kubernetes, scaling, monitoring | Production infrastructure | **By chapter's end, you'll have built a Customer Support Digital FTE** that: * Routes inquiries to specialist agents (FAQ, Booking, Escalation) * Maintains conversation context across handoffs * Validates inputs with guardrails (abuse detection, PII filtering) * Persists conversations with SQLite sessions * Provides full observability through tracing * Integrates external knowledge via MCP and RAG The capstone includes **monetization models** (subscription, success fee, hybrid) so you understand how to price your Digital FTE when you're ready to ship. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#what-youll-learn "Direct link to What You'll Learn") --------------------------------------------------------------------------------------------------------------------------------------------------------------- By the end of this chapter, you'll be able to: * **Configure the OpenAI Agents SDK**: Set up your development environment, manage API keys securely, and understand the SDK's core primitives (Agent, Runner, tools, handoffs) * **Build single agents with tools**: Create agents that use function tools to interact with external systems, implementing the Tool-Use pattern from Chapter 61 * **Implement multi-agent handoffs**: Design systems where specialized agents hand off to each other, applying the Coordinator and Sequential patterns * **Add guardrails and safety**: Implement input/output validation, content filtering, and the Human-in-the-Loop pattern for sensitive operations * **Handle streaming and async**: Build responsive agents that stream responses and handle concurrent operations efficiently * **Trace and debug agents**: Use OpenAI's tracing infrastructure to understand agent behavior and diagnose issues Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------------------------------------- * Chapter 61: Introduction to AI Agents (conceptual foundation) * Part 4: Python Fundamentals (async/await, type hints, dataclasses) * Part 4: Applied Domain Workflows (specification-driven workflow) * OpenAI API key with Agents SDK access * [Your Path to Shipping Digital FTEs](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#your-path-to-shipping-digital-ftes) * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#what-youll-learn) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk#prerequisites) --- # Your First Widget | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Theory from Lesson 1 gave you the mental model. Now let's get something working. In the next 45 minutes, you'll build a ChatGPT App that displays "Hello, World!" in a styled widget—and you'll do it in under 50 lines of code. The goal isn't to understand every detail yet. The goal is to see a widget render in ChatGPT. Once that works, you have a foundation to build on. Each subsequent lesson adds one feature to this widget until you have a complete TaskManager. Project Setup[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#project-setup "Direct link to Project Setup") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Create a new directory for your ChatGPT App: mkdir taskmanager-widgetcd taskmanager-widgetuv init Add dependencies: uv add "mcp[cli]>=1.9.2" "uvicorn>=0.32.0" **Output:** Resolved 12 packages in 1.2sInstalled 12 packages in 50ms + mcp==1.9.2 + uvicorn==0.32.0 ... The Minimal Server[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#the-minimal-server "Direct link to The Minimal Server") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the entire server—under 50 lines. Create `main.py`: import mcp.types as typesfrom mcp.server.fastmcp import FastMCP# The magic MIME type that triggers widget renderingMIME_TYPE = "text/html+skybridge"# Your widget HTML - simple and styledWIDGET_HTML = '''

Hello, World!

Your first ChatGPT App widget

'''mcp = FastMCP("TaskManager")@mcp.tool()def show_greeting() -> types.CallToolResult: """Display a greeting widget.""" return types.CallToolResult( content=[types.TextContent(type="text", text="Showing greeting")], _meta={ "openai.com/widget": types.EmbeddedResource( type="resource", resource=types.TextResourceContents( uri="ui://greeting", mimeType=MIME_TYPE, text=WIDGET_HTML, ) ) } )if __name__ == "__main__": import uvicorn app = mcp.sse_app() uvicorn.run(app, host="0.0.0.0", port=8001) ### What Makes This Work[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#what-makes-this-work "Direct link to What Makes This Work") Three things turn this MCP server into a ChatGPT App: 1. **`MIME_TYPE = "text/html+skybridge"`** — This tells ChatGPT "render this as a widget, not text" 2. **`_meta` with `"openai.com/widget"`** — This attaches the widget HTML to the tool response 3. **`EmbeddedResource`** — This packages the HTML with its MIME type Without any of these three, you get text output instead of a widget. Running the Server[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#running-the-server "Direct link to Running the Server") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Start your server: uv run main.py **Output:** INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)INFO: Started server process [12345] Your server is running locally. But ChatGPT needs a public URL. Exposing with ngrok[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#exposing-with-ngrok "Direct link to Exposing with ngrok") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In a second terminal, create a tunnel: ngrok http 8001 **Output:** Forwarding https://abc123.ngrok-free.app -> http://localhost:8001 Copy that `https://...ngrok-free.app` URL. Your MCP endpoint is: https://abc123.ngrok-free.app/mcp Note: The URL changes each time you restart ngrok (on the free plan). Registering in ChatGPT[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#registering-in-chatgpt "Direct link to Registering in ChatGPT") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [chatgpt.com](https://chatgpt.com/) 2. Click Settings (gear icon) → Toggle **Developer mode** ON 3. Click **Create app** 4. Enter: * **Name**: TaskManager * **MCP Server URL**: `https://abc123.ngrok-free.app/mcp` (your ngrok URL + `/mcp`) * **Authentication**: No Auth 5. Click **Create** Your app should show "Connected" status. Testing Your Widget[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#testing-your-widget "Direct link to Testing Your Widget") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Start a new chat 2. Type `@TaskManager` to select your app 3. Say: "Show me a greeting" **Output:** You should see your purple gradient card with "Hello, World!" displayed directly in the conversation. If it works—congratulations. You've built your first ChatGPT App. Troubleshooting[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#troubleshooting "Direct link to Troubleshooting") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Widget shows "Loading..." forever** * Check that `MIME_TYPE` is exactly `"text/html+skybridge"` * Verify `_meta` key is exactly `"openai.com/widget"` **App shows "Disconnected"** * ngrok URL may have changed—restart ngrok and update the app URL in ChatGPT settings **Tool not appearing** * Delete the app in ChatGPT and recreate it with the current ngrok URL What You Built[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#what-you-built "Direct link to What You Built") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- You now have: * A FastMCP server running on port 8001 * A widget that renders in ChatGPT * A development workflow (ngrok + Developer Mode) This is your foundation. In the next lesson, you'll add a "Refresh" button that triggers new conversation turns. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Add a Name Parameter[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#prompt-1-add-a-name-parameter "Direct link to Prompt 1: Add a Name Parameter") Modify my show_greeting tool to accept a "name" parameter.The widget should display "Hello, [name]!" instead of "Hello, World!".Use window.openai?.toolOutput to read the name in the widget JavaScript. **What you're learning:** How data flows from tool parameters through `structuredContent` to the widget via `window.openai.toolOutput`. This pattern is essential for the TaskManager. ### Prompt 2: Change the Style[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#prompt-2-change-the-style "Direct link to Prompt 2: Change the Style") Update my widget's CSS to use a different color gradient (green to teal instead of purple). Also add a subtle animation when the widget appears. **What you're learning:** Widget styling capabilities within the iframe sandbox. All CSS must be inline since external stylesheets don't reliably load. ### Prompt 3: Understand the MIME Type[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#prompt-3-understand-the-mime-type "Direct link to Prompt 3: Understand the MIME Type") Explain why text/html+skybridge is required for ChatGPT App widgets. What happens if I use text/html instead? Why did OpenAI create a custom MIME type? **What you're learning:** The architectural decision behind widget rendering. Understanding this helps debug when widgets fail to appear. **Safety Note**: Your ngrok tunnel is publicly accessible. Don't include real credentials or sensitive data in your test server. Checking access... * [Project Setup](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#project-setup) * [The Minimal Server](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#the-minimal-server) * [What Makes This Work](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#what-makes-this-work) * [Running the Server](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#running-the-server) * [Exposing with ngrok](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#exposing-with-ngrok) * [Registering in ChatGPT](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#registering-in-chatgpt) * [Testing Your Widget](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#testing-your-widget) * [Troubleshooting](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#troubleshooting) * [What You Built](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#what-you-built) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#try-with-ai) * [Prompt 1: Add a Name Parameter](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#prompt-1-add-a-name-parameter) * [Prompt 2: Change the Style](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#prompt-2-change-the-style) * [Prompt 3: Understand the MIME Type](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/first-chatgpt-app#prompt-3-understand-the-mime-type) --- # Agent Interoperability & Security | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Agents need to work together and stay secure. This lesson covers how agents communicate across systems and how to protect them from misuse. Agent-to-Agent Communication[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agent-to-agent-communication "Direct link to Agent-to-Agent Communication") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### The Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#the-problem "Direct link to The Problem") You build a customer support agent. It needs to: * Check order status (OrderAgent) * Process refunds (BillingAgent) * Schedule callbacks (CalendarAgent) How do these agents find each other? How do they communicate? How does one agent know what another can do? ### A2A Protocol[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#a2a-protocol "Direct link to A2A Protocol") The Agent-to-Agent (A2A) protocol standardizes agent communication. **Core concepts:** **Agent Cards**: JSON descriptions of capabilities { "name": "BillingAgent", "description": "Handles billing operations", "capabilities": ["process_refund", "query_balance", "update_payment"], "inputs": { "process_refund": { "customer_id": "string", "amount": "number", "reason": "string" } }, "limits": { "max_refund": 500, "rate_limit": "100/hour" }, "endpoint": "https://billing.internal/agent"} Agent Cards answer: "What can this agent do? What does it need? What are its limits?" **Task Delegation**: One agent asks another to perform work SupportAgent → BillingAgent: "Process refund $50 for customer 123, reason: damaged item"BillingAgent: {accepts task, validates inputs}BillingAgent: {processes refund}BillingAgent → SupportAgent: "Refund complete. Confirmation: RF-456" **Progress Updates**: Streaming status for long tasks SupportAgent → ResearchAgent: "Compile report on customer trends Q4"ResearchAgent: {status: "accepted"}ResearchAgent: {status: "gathering data", progress: 25%}ResearchAgent: {status: "analyzing", progress: 60%}ResearchAgent: {status: "complete", result: [report]} ### Why A2A Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#why-a2a-matters "Direct link to Why A2A Matters") Without standards, every agent integration is custom. With A2A: * Agents discover each other's capabilities automatically * Communication follows predictable patterns * Error handling and retries work consistently * New agents plug into existing ecosystems Agent Security[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agent-security "Direct link to Agent Security") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Agents as a New Principal Class[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agents-as-a-new-principal-class "Direct link to Agents as a New Principal Class") Traditional security has two principals: * **Users**: Humans with identities and permissions * **Services**: Software with API keys and scopes Agents are a third class. They: * Act autonomously on behalf of systems * Make decisions without human approval * Have their own identity and permissions * Can be compromised or manipulated **Why this matters**: A compromised agent can do damage proportional to its permissions. Security design limits that blast radius. ### The Trust Trade-Off[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#the-trust-trade-off "Direct link to The Trust Trade-Off") Every capability given to an agent introduces risk. | Capability | Utility | Risk | | --- | --- | --- | | Read customer data | Personalized service | Data exposure | | Issue refunds | Autonomous resolution | Financial loss | | Execute code | Powerful automation | System compromise | | Access external APIs | Rich integrations | Data exfiltration | Security design balances utility against risk. An agent with no capabilities is useless. An agent with unlimited capabilities is dangerous. ### Defense in Depth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#defense-in-depth "Direct link to Defense in Depth") Two layers of protection work together: **Layer 1: Deterministic Guardrails** Hard limits enforced by infrastructure. Can't be bypassed by clever prompting. Examples: * **Rate limits**: 100 requests per minute maximum * **Resource caps**: $500 maximum per refund * **Scope limits**: Can only access customer database, not employee database * **Action restrictions**: Can read files but not delete them BillingAgent attempts: process_refund($750)Infrastructure blocks: "Amount exceeds $500 limit" The agent can't talk its way past these limits. They're enforced at the infrastructure layer, not the reasoning layer. **Layer 2: AI-Powered Guard Models** A separate LLM evaluates agent actions for context-dependent risks. Agent wants to: Send email to customerGuard model checks: - Is the content appropriate? ✓ - Is the recipient a valid customer? ✓ - Does this look like spam/phishing? ✓Decision: Allow Agent wants to: Send email to 10,000 addressesGuard model checks: - Is this bulk send authorized? ✗ - Does this match normal patterns? ✗Decision: Block, escalate for review Guard models catch context-dependent issues that hard rules can't anticipate: * Social engineering attempts * Unusual patterns (agent suddenly accessing files it never touched before) * Output that's technically allowed but inappropriate ### Agent Identity[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agent-identity "Direct link to Agent Identity") Each agent needs verifiable identity: **Credentials**: How the agent proves who it is * API keys, certificates, or tokens * Rotated regularly * Scoped to specific capabilities **Permissions**: What the agent can do * Principle of least privilege: only what's needed * Explicit grants, not implicit access * Reviewed and audited regularly **Audit Log**: What the agent has done * Every action recorded * Traceable to specific agent identity * Enables forensics when things go wrong **Compromise response**: When an agent is compromised: 1. Revoke its credentials immediately 2. Review audit logs to assess damage 3. Blast radius limited to that agent's permissions 4. Other agents unaffected This is why least-privilege matters. An agent with access to everything makes a breach catastrophic. An agent with minimal access makes a breach containable. Designing Secure Agent Systems[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#designing-secure-agent-systems "Direct link to Designing Secure Agent Systems") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Follow this process for any agent you build: ### Step 1: Define Capabilities[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-1-define-capabilities "Direct link to Step 1: Define Capabilities") What does this agent need to do? List specific actions. Example for a Support Agent: * Read customer order history * Issue refunds up to $500 * Create escalation tickets * Send confirmation emails ### Step 2: Identify Risks[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-2-identify-risks "Direct link to Step 2: Identify Risks") For each capability, what could go wrong? | Capability | Risk | Severity | | --- | --- | --- | | Read order history | Data exposure to wrong customer | Medium | | Issue refunds | Fraudulent refunds | High | | Create tickets | Spam/abuse | Low | | Send emails | Phishing, spam | Medium | ### Step 3: Set Deterministic Guardrails[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-3-set-deterministic-guardrails "Direct link to Step 3: Set Deterministic Guardrails") Hard limits for high-severity risks: * Refund cap: $500 per transaction, $2000 per day * Order access: Only orders matching authenticated customer * Email rate: Maximum 10 emails per hour ### Step 4: Design Guard Model Checks[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-4-design-guard-model-checks "Direct link to Step 4: Design Guard Model Checks") Contextual evaluation for medium-severity risks: * Before sending email: Check content for phishing patterns * Before refund: Verify refund reason matches order issues * Unusual patterns: Alert if accessing orders outside normal hours ### Step 5: Plan for Compromise[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-5-plan-for-compromise "Direct link to Step 5: Plan for Compromise") What happens if this agent is compromised? * Credential rotation: Can we revoke and reissue quickly? * Blast radius: What's the maximum damage with current permissions? * Detection: How would we know if the agent is acting maliciously? * Recovery: How do we undo damage? Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude, ChatGPT, or Gemini to design agent security. > "A BillingAgent can issue refunds up to $500 and query account balances. Design its Agent Card: name, capabilities, inputs, outputs, and guardrails." **Expected**: Structured Agent Card with capabilities (process\_refund, query\_balance), inputs (customer\_id, amount, reason), outputs (confirmation\_id, status), guardrails ($500/transaction, $5000/day, rate limits). > "If BillingAgent is compromised, what's the maximum damage an attacker could do? How does agent identity help contain it?" **Expected**: Maximum damage = $500 × rate limit. Credential revocation stops attack immediately. Audit log identifies abuse. Other agents unaffected. > "A billing action over $250 should go through a guard model before execution. What should the guard model check? Give me 5 specific checks." **Expected**: Is amount reasonable for this customer's history? Is this a duplicate of recent request? Does refund reason match order data? Is request timing suspicious? Does the request come from a verified session? > "Design a defense-in-depth strategy for an agent that can execute code in a sandbox. What deterministic guardrails and guard model checks would you implement?" **Expected**: Guardrails: Sandboxed environment, no network access, execution timeout, memory limits. Guard checks: Does code match expected patterns? Is it trying to access sensitive data? Does it attempt to escape sandbox? **Key insight**: Every capability is a risk. Security design manages the trust trade-off—enough power to be useful, enough limits to be safe. Checking access... * [Agent-to-Agent Communication](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agent-to-agent-communication) * [The Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#the-problem) * [A2A Protocol](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#a2a-protocol) * [Why A2A Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#why-a2a-matters) * [Agent Security](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agent-security) * [Agents as a New Principal Class](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agents-as-a-new-principal-class) * [The Trust Trade-Off](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#the-trust-trade-off) * [Defense in Depth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#defense-in-depth) * [Agent Identity](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#agent-identity) * [Designing Secure Agent Systems](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#designing-secure-agent-systems) * [Step 1: Define Capabilities](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-1-define-capabilities) * [Step 2: Identify Risks](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-2-identify-risks) * [Step 3: Set Deterministic Guardrails](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-3-set-deterministic-guardrails) * [Step 4: Design Guard Model Checks](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-4-design-guard-model-checks) * [Step 5: Plan for Compromise](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#step-5-plan-for-compromise) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/introduction-to-ai-agents/agent-interoperability-security#try-with-ai) --- # Advanced Skill Patterns | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've already built basic skills in Chapter 3. You understand the Persona + Questions + Principles framework. But there's a critical distinction between skills you've created so far and the skills this chapter will focus on—one that changes how you design for autonomous execution. Consider this scenario: You've built an advisory skill that says "You are a Python expert. Answer questions about Python best practices." Students ask questions, the skill recommends approaches. Simple advice-giving. Now imagine a different skill: "You are a Python code orchestrator. Watch for problems requiring multi-step code generation. Estimate execution complexity. Generate code, run it, analyze output. If partial success, refine code and retry until complete." This skill doesn't advise—it acts. It orchestrates code execution autonomously. This is the jump from **advisory** to **execution** skills. And it requires rethinking every aspect of how you design: persona, questions, and principles become leverage points for autonomous behavior instead of helpful guidance. The Execution Shift: Advisory vs Execution Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#the-execution-shift-advisory-vs-execution-skills "Direct link to The Execution Shift: Advisory vs Execution Skills") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### What Advisory Skills Do[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#what-advisory-skills-do "Direct link to What Advisory Skills Do") Advisory skills answer questions and offer recommendations. The human makes the final decision. **Example**: The `fetching-library-docs` skill from Chapter 3 is advisory: * **Persona**: "You are a Python documentation specialist" * **Questions**: "What's the user's core need? Which modules are relevant?" * **Principles**: "Be thorough but concise. Only suggest modules the user asked about." * **Role**: Recommends which documentation to fetch * **Human's role**: Evaluates recommendation, decides whether to adopt The skill provides intelligent guidance. The human is the executor. ### What Execution Skills Do[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#what-execution-skills-do "Direct link to What Execution Skills Do") Execution skills orchestrate code, make decisions, and iterate toward results. They are autonomous agents that act, not just advise. **Execution Example** (this chapter's focus): A skill that processes data autonomously: * **Persona**: "You are a data orchestrator: Identify data problems, generate processing code, execute it, validate results. Iterate if partial success." * **Questions**: "What's the input data structure? What transformations matter? What edge cases exist? Are results complete?" * **Principles**: "Fail safe: Validate data before processing. Converge deliberately: Iterate max 3 times. Log all transformations." * **Role**: Generates code, executes it, analyzes results, retries on failure * **Human's role**: Provides initial specification, monitors execution, validates final output The skill orchestrates the full execution loop. The human sets direction; the skill handles execution. ### The Critical Difference: Persona[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#the-critical-difference-persona "Direct link to The Critical Difference: Persona") The **persona is the core difference** between advisory and execution skills. **Advisory Persona** (from Chapter 3): "You are a Python best practices expert" This activates recommendation mode. The skill analyzes context and suggests what you should do. It's passive regarding execution—the human must implement suggestions. **Execution Persona** (this chapter): "You are a Python execution orchestrator: Watch for code-generation tasks.For each task: (1) Write code matching spec, (2) Execute code, (3) Analyze output,(4) If partial success, refine and retry, (5) Log all steps" This activates orchestration mode. The skill sees a task and autonomously manages the full execution cycle. The human watches; the skill operates. Notice the difference: * Advisory: "You are an expert..." → Provides recommendations * Execution: "You are an orchestrator: watch for X, do Y, analyze Z, iterate W..." → Performs workflow The execution persona explicitly describes the workflow the skill manages. That workflow description becomes the basis for decision-making questions and behavioral principles. Designing Questions for Autonomous Decision-Making[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#designing-questions-for-autonomous-decision-making "Direct link to Designing Questions for Autonomous Decision-Making") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Advisory skills ask questions like "What does the user need?" Execution skills ask different questions—questions that encode decision logic about **when, how, and whether** to act. ### Three Types of Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#three-types-of-questions "Direct link to Three Types of Questions") **Type 1: Context Analysis Questions** (What's the situation?) These questions help the skill understand the problem space before acting. **Good Example**: "Is this a code generation task or a documentation task?What constraints exist on execution (time, resources, data sensitivity)?What indicates 'success' for this particular task?" Why these work: They force analysis of context. The skill can't answer them without understanding the problem deeply. The skill's decisions will depend on these answers. **Bad Example**: "What should I do?""Is this important?""Should I try my best?" Why these fail: They're vague. No meaningful analysis. The skill ends up guessing. **Type 2: Convergence Questions** (When should I stop?) Execution skills iterate. These questions help determine when iteration has achieved enough. **Good Example**: "Does the output match the specification completely?Are there edge cases the code doesn't handle?What is the minimal additional iteration needed to reach completeness?" Why these work: They force evaluation of output quality against specification. The skill can decide whether to iterate or declare success. **Bad Example**: "Is the output good enough?Should I keep trying?" Why these fail: "Good enough" is undefined. The skill will keep iterating endlessly or stop prematurely. **Type 3: Safety/Constraint Questions** (What should I NOT do?) Execution skills operate autonomously, so constraints are critical. **Good Example**: "Does this operation access restricted filesystems or APIs?Are there rate limits I should respect?What should happen if execution would exceed resource constraints?" Why these work: They identify explicit boundaries. The skill knows what's forbidden before attempting operations. **Bad Example**: "Is this safe?""Should I be careful?" Why these fail: Vague. The skill will operate without clear constraints. Principles That Enable Autonomous Behavior[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#principles-that-enable-autonomous-behavior "Direct link to Principles That Enable Autonomous Behavior") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Principles are the behavioral guidelines that govern how a skill operates when it encounters decisions. Unlike questions (which are analytical), principles are directives that constrain choices. ### Structure of Effective Principles[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#structure-of-effective-principles "Direct link to Structure of Effective Principles") An effective principle has three components: 1. **Constraint** (what's the boundary?) 2. **Reason** (why does this boundary matter?) 3. **Application** (how do you apply this in practice?) **Example Principle**: Principle: "Fail Safe: Always validate data before processing"Constraint: Before any code executes against input data, validate structure and contentReason: Processing malformed data produces incorrect results silently (subtle bugs)Application: "Every script starts with data validation assertions. If validation fails, log and halt rather than attempting recovery" This principle tells the skill: "When you see data, validate it first. Here's why. Here's how you know you've done it right." ### Three Categories of Principles for Execution Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#three-categories-of-principles-for-execution-skills "Direct link to Three Categories of Principles for Execution Skills") **Category 1: Convergence Principles** (How do you know you're done?) These prevent infinite loops and define success criteria. **Example**: Principle: "Converge Deliberately: Stop iterating when spec is met or max iterations reached"Constraint: Skill will retry failed code generation at most 3 times per taskReason: Infinite iteration indicates a problem that won't be solved by retry (e.g., spec is unclear). Three iterations is enough to fix transient errors; more suggests fundamental issueApplication: After each iteration, check: "Does output match spec? If yes, declare success. If no, has this specific error occurred before? If yes, stop and flag for human review" **Category 2: Efficiency Principles** (How do you minimize wasted effort?) These optimize execution for speed and resource efficiency. **Example**: Principle: "Filter Ruthlessly: Keep only essential data in context"Constraint: Don't load all available data. Load only what the current task requires.Reason: Token limits and execution speed. Irrelevant data fills context without value.Application: "Before loading data, ask: 'What's the minimum data this step needs?' Load that subset. When the subset is processed, load the next subset." **Category 3: Safety Principles** (What should never happen?) These prevent dangerous operations and define guardrails. **Example**: Principle: "Sandbox First: Validate before Executing"Constraint: Never execute generated code against production data on first runReason: Generated code may have bugs. Testing against production risks data corruptionApplication: "All generated code executes in isolated sandbox first with test data. Review execution results. Only after validation, run against real data" Skill Composition: How Skills Depend on Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#skill-composition-how-skills-depend-on-skills "Direct link to Skill Composition: How Skills Depend on Skills") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As skills become more sophisticated, some skills depend on other skills. A data processing skill might depend on a data validation skill. A code execution skill might depend on a syntax error recovery skill. This creates a **skill dependency graph**. Understanding how to design skills that compose cleanly is critical. ### Dependency Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#dependency-patterns "Direct link to Dependency Patterns") **Pattern 1: Sequential Dependency** (Skill A must complete before Skill B starts) Specification Flow Skill ↓Data Fetch Skill (uses flow from above) ↓Data Processing Skill (uses fetched data) ↓Validation Skill (confirms processing) Each skill waits for previous to complete, then uses its output. **Pattern 2: Conditional Dependency** (Skill B only activates if Skill A produces specific result) ↓ Success Error Detection Skill ─→ Error Recovery Skill ↓ Fail Escalate The error recovery skill only activates if error detection identifies a recoverable error. **Pattern 3: Referenced Dependency** (Skill B explicitly mentions it depends on Skill A) In a skill's spec.md: dependencies: - skill: "data-validation-skill" version: "1.0+" when: "Before any code execution against external data" This declaration makes the dependency explicit. Other developers (and the skill itself) know what other skills this skill requires. ### Why Composition Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#why-composition-matters "Direct link to Why Composition Matters") When skills are well-composed: 1. **Clarity**: Other developers understand prerequisites 2. **Reusability**: Skills designed for composition work in multiple projects 3. **Testability**: Each skill can be tested independently 4. **Maintainability**: When one skill changes, composed skills adapt gracefully When composition is poor: 1. **Hidden dependencies**: Skills fail mysteriously due to missing prerequisites 2. **Rigidity**: Skills only work in one specific context 3. **Integration hell**: Composing skills requires extensive debugging 4. **Maintenance nightmare**: Changing one skill breaks others unexpectedly Design Decision: Layer 3 vs Layer 4 Skills[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#design-decision-layer-3-vs-layer-4-skills "Direct link to Design Decision: Layer 3 vs Layer 4 Skills") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Understanding when a skill belongs in Layer 3 (Intelligence Design - reusable component) vs Layer 4 (Spec-Driven Integration - capstone project) affects how you structure the skill. ### Layer 3 Skill Characteristics[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#layer-3-skill-characteristics "Direct link to Layer 3 Skill Characteristics") A skill is Layer 3 when: 1. **It's a reusable component**: The pattern applies across 3+ different projects 2. **It solves a recurring problem**: You've encountered this workflow 2+ times 3. **Others will use it**: The skill is documented for broader adoption 4. **It's moderately complex**: 5-7 decision points, not trivial **Example**: A "write-execute-analyze" loop skill * Data analysis projects use it * Code refactoring projects use it * Report generation projects use it * Others in your organization will adopt it Design for **generality and documentation**. ### Layer 4 Skill Characteristics[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#layer-4-skill-characteristics "Direct link to Layer 4 Skill Characteristics") A skill is Layer 4 when: 1. **It's a capstone orchestration**: Combines 3+ existing skills into a cohesive system 2. **It solves a specific domain problem**: You're building a Digital FTE for a customer 3. **It's production-ready**: Includes error recovery, logging, monitoring 4. **It's potentially monetized**: You'll sell this as a Digital FTE product **Example**: A "legal-document-analyzer" Digital FTE * Orchestrates document parsing skill * Uses semantic analysis skill * Coordinates with extraction skill * Produces customer-ready output * You'll charge customers for it Design for **domain specificity, production robustness, and customer value**. Manual Exercise: Design Your Execution Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#manual-exercise-design-your-execution-skill "Direct link to Manual Exercise: Design Your Execution Skill") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Using the framework from this lesson, design a skill for your own domain. You won't code it yet—this is design thinking. ### Step 1: Identify the Execution Task[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-1-identify-the-execution-task "Direct link to Step 1: Identify the Execution Task") What problem does your skill solve by orchestrating code execution? **Example answers:** * "Analyze customer feedback data and generate insights" * "Process financial transactions and detect anomalies" * "Refactor legacy code toward modern patterns" **Your answer:** My execution skill will: ### Step 2: Design the Persona[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-2-design-the-persona "Direct link to Step 2: Design the Persona") Write a persona that describes the skill as an orchestrator, not an advisor. Start with: "You are a \[domain\] orchestrator: \[describe the workflow\]" Use this template: You are a [domain] orchestrator: When you see [situation],(1) [first action],(2) [second action],(3) [validation step],(4) [iteration or completion decision]. **Your persona:** You are a _____________ orchestrator: ### Step 3: Write Five Decision-Making Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-3-write-five-decision-making-questions "Direct link to Step 3: Write Five Decision-Making Questions") Write five questions the skill must answer to operate autonomously. Each question should be: * **Concrete** (not vague—"Is this important?" is too vague) * **Answerable** (the skill can actually figure out the answer) * **Decision-triggering** (the answer determines what the skill does next) **Your questions:** 1.2.3.4.5. ### Step 4: Articulate Three Principles[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-4-articulate-three-principles "Direct link to Step 4: Articulate Three Principles") For each principle, state the constraint, reason, and application: Principle 1: [Name]Constraint: [What's the boundary?]Reason: [Why does this matter?]Application: [How do you know you've done it right?] **Your principles:** Principle 1:Principle 2:Principle 3: Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Use your AI companion (Claude, ChatGPT, Gemini, or similar) to validate and refine your skill design. ### Prompt 1: Validate Persona Clarity[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#prompt-1-validate-persona-clarity "Direct link to Prompt 1: Validate Persona Clarity") I've designed a skill with this persona:[paste your persona from Step 2]Does this persona describe an execution workflow (orchestration) or just advisory guidance?Specifically, what workflow does this persona describe? Can you extract the steps?If the persona is vague, suggest how to make it more concrete with explicit workflow steps. **What you're learning**: How to articulate execution workflows clearly so skills (and humans) understand exactly what the skill will do. ### Prompt 2: Refine Decision Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#prompt-2-refine-decision-questions "Direct link to Prompt 2: Refine Decision Questions") I've written these decision-making questions for my skill:[paste your five questions from Step 3]For each question:1. Is this question concrete enough that the skill can actually answer it?2. Does the answer lead to a specific action, or is it just informational?3. If the question is vague, rewrite it to be more concreteThen ask: "Which of your original questions are most critical to the skill's autonomous behavior?" **What you're learning**: How to distinguish vague questions ("Is this important?") from decision questions that actually guide execution ("Does this output match the spec?"). ### Prompt 3: Strengthen Principles[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#prompt-3-strengthen-principles "Direct link to Prompt 3: Strengthen Principles") I've written these principles to guide my skill's execution:[paste your three principles from Step 4]For each principle, evaluate:1. Is the constraint clear and testable?2. Does the reason explain WHY this constraint matters?3. Is the application concrete enough that someone could validate compliance?If any principle is unclear, help me rewrite it with a concrete constraint, clear reason, and testable application. **What you're learning**: How to encode behavioral guidelines that actually constrain skill behavior, not vague aspirations. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#safety-note "Direct link to Safety Note") As you design skills that orchestrate code execution, remember: the skill will be running code autonomously. Design your principles with safety in mind. What data should never be accessed? What operations should never be attempted? What limits (time, resources, retries) prevent runaway execution? These constraints become the difference between a reliable Digital FTE and a system that fails unpredictably. Checking access... * [The Execution Shift: Advisory vs Execution Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#the-execution-shift-advisory-vs-execution-skills) * [What Advisory Skills Do](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#what-advisory-skills-do) * [What Execution Skills Do](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#what-execution-skills-do) * [The Critical Difference: Persona](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#the-critical-difference-persona) * [Designing Questions for Autonomous Decision-Making](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#designing-questions-for-autonomous-decision-making) * [Three Types of Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#three-types-of-questions) * [Principles That Enable Autonomous Behavior](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#principles-that-enable-autonomous-behavior) * [Structure of Effective Principles](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#structure-of-effective-principles) * [Three Categories of Principles for Execution Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#three-categories-of-principles-for-execution-skills) * [Skill Composition: How Skills Depend on Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#skill-composition-how-skills-depend-on-skills) * [Dependency Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#dependency-patterns) * [Why Composition Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#why-composition-matters) * [Design Decision: Layer 3 vs Layer 4 Skills](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#design-decision-layer-3-vs-layer-4-skills) * [Layer 3 Skill Characteristics](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#layer-3-skill-characteristics) * [Layer 4 Skill Characteristics](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#layer-4-skill-characteristics) * [Manual Exercise: Design Your Execution Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#manual-exercise-design-your-execution-skill) * [Step 1: Identify the Execution Task](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-1-identify-the-execution-task) * [Step 2: Design the Persona](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-2-design-the-persona) * [Step 3: Write Five Decision-Making Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-3-write-five-decision-making-questions) * [Step 4: Articulate Three Principles](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#step-4-articulate-three-principles) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#try-with-ai) * [Prompt 1: Validate Persona Clarity](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#prompt-1-validate-persona-clarity) * [Prompt 2: Refine Decision Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#prompt-2-refine-decision-questions) * [Prompt 3: Strengthen Principles](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#prompt-3-strengthen-principles) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/advanced-skill-patterns#safety-note) --- # MCP Architecture Overview | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Imagine you're building a Digital FTE that needs to help customers manage their work. It needs to read files from their computer, query their database, create issues in their project tracker, and pull documents from their knowledge base. Without a standard protocol, you'd build four separate integrations—each with its own authentication, data formats, and error handling. When you want your agent to work with ChatGPT instead of Claude? Rebuild all four integrations. A new customer uses Jira instead of GitHub? Build a fifth integration from scratch. This is the **integration problem** that MCP solves. Model Context Protocol (MCP) is the universal standard for connecting AI applications to external systems. Think of it as **USB-C for AI**—just as USB-C provides one port that works with chargers, monitors, and storage devices across all manufacturers, MCP provides one protocol that connects any AI application to any external service. Before MCP (custom integrations):┌─────────┐ ┌─────────┐│ App A │──custom code──▶ │ GitHub │└─────────┘ └─────────┘┌─────────┐ ┌─────────┐│ App B │──different code──▶ │ GitHub │└─────────┘ └─────────┘Same service, two integrations. Add App C? Write a third.With MCP (one server, many clients):┌─────────┐ ┌─────────────────┐│ App A │───MCP───┐ │ │└─────────┘ │ │ GitHub MCP │ ├──────────▶│ Server │┌─────────┐ │ │ (built once) ││ App B │───MCP───┘ │ │└─────────┘ └─────────────────┘Same server serves both. Add App C? It just connects. Released by Anthropic in November 2024, adopted by OpenAI in March 2025, and donated to the Linux Foundation's Agentic AI Foundation in December 2025, MCP has evolved from one company's solution to industry-wide infrastructure. When you understand MCP, you're not learning one vendor's approach—you're learning the protocol that's unifying the entire ecosystem. Why MCP Exists: The Integration Explosion Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#why-mcp-exists-the-integration-explosion-problem "Direct link to Why MCP Exists: The Integration Explosion Problem") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider what happens when AI applications need to connect to external systems: **The Math Without Standards** | AI Applications | External Systems | Custom Integrations Needed | | --- | --- | --- | | 3 applications | 5 systems | 3 × 5 = **15 integrations** | | 5 applications | 10 systems | 5 × 10 = **50 integrations** | | 10 applications | 20 systems | 10 × 20 = **200 integrations** | Every new AI application multiplies the work. Every new external system multiplies it again. This is **O(n × m) complexity**—unsustainable for an ecosystem. **A Concrete Example: Your Customer's Tech Stack** You're selling a Digital FTE to a consulting firm. They need your agent to: 1. **Read project files** from their shared drive 2. **Query client data** from their PostgreSQL database 3. **Create tasks** in their Asana project tracker 4. **Search documentation** in their Confluence wiki Without MCP, you build four custom integrations. Six months later, they switch from Asana to Monday.com. You rebuild that integration. A year later, they want to use your agent in Cursor instead of Claude Desktop. You rebuild all four integrations for the new host. **With MCP**, you write zero custom integrations. You configure four MCP servers (filesystem, postgres, asana, confluence) that already exist. When they switch to Monday.com, you swap one server configuration. When they want Cursor support, it works automatically—Cursor already speaks MCP. Without MCP (your code): With MCP (community servers):├── integrations/ ├── mcp-config.json│ ├── claude_filesystem.py │ {│ ├── claude_postgres.py │ "filesystem": { "path": "/projects" },│ ├── claude_asana.py │ "postgres": { "connection": "..." },│ ├── claude_confluence.py │ "asana": { "token": "..." },│ ├── cursor_filesystem.py │ "confluence": { "url": "..." }│ ├── cursor_postgres.py │ }│ ├── cursor_asana.py └──│ └── cursor_confluence.py└── 8 files, 2000+ lines 1 file, 10 lines **The Math With MCP** | AI Applications | MCP Servers | Total Components | | --- | --- | --- | | 3 applications | 5 servers | 3 + 5 = **8 components** | | 5 applications | 10 servers | 5 + 10 = **15 components** | | 10 applications | 20 servers | 10 + 20 = **30 components** | MCP transforms O(n × m) into **O(n + m)**. That's why the industry converged on it. Bonus: SDK Tool Schema Unification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#bonus-sdk-tool-schema-unification "Direct link to Bonus: SDK Tool Schema Unification") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Beyond the integration problem, MCP also standardizes tool schemas. In the previous SDK chapters, you saw each SDK uses different formats: | SDK | Schema Key for Parameters | | --- | --- | | OpenAI | `function.parameters` | | Anthropic | `input_schema` | | Google ADK | `input` | MCP provides ONE schema format (`inputSchema`) that works everywhere. But this is a secondary benefit—the real value is solving the integration explosion. The Host-Client-Server Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#the-host-client-server-architecture "Direct link to The Host-Client-Server Architecture") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- MCP uses a clean three-tier architecture that you need to visualize clearly: ┌──────────────────────────────────────────────────────────┐│ MCP Host ││ (Any AI application: IDE, chat app, API service) ││ ││ ┌─────────────────────────────────────────────────┐ ││ │ MCP Client │ ││ │ (Manages server connections, tool routing) │ ││ └────────────┬────────────────────────────────────┘ ││ │ JSON-RPC 2.0 (stdio or HTTP/SSE) ││ │ │└───────────────┼─────────────────────────────────────────┘ │ │ (Network or local pipe) │ ┌───────▼────────┐ │ MCP Server │ │ (Any external │ │ service) │ └────────────────┘ ### Host: Where Humans Work[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#host-where-humans-work "Direct link to Host: Where Humans Work") The **MCP Host** is your application—Claude Code, Cursor IDE, VS Code, or your custom API service. It's where humans (or other agents) initiate requests. * Claude Code hosting MCP Client → Users request "connect GitHub" * VS Code hosting MCP Client → Developers want file search across codebase * Your API service hosting MCP Client → Backend needs database access The Host doesn't directly implement tools. It delegates to the Client. ### Client: The Connection Manager[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#client-the-connection-manager "Direct link to Client: The Connection Manager") The **MCP Client** is a component INSIDE the Host that manages connections to servers. One Host can run one or more Clients, and each Client manages exactly one Server connection. **Important distinction**: Not every Agent SDK needs multiple Clients. But every Agent that wants to use MCP needs at least one MCP Client—the lightweight component that speaks JSON-RPC 2.0 to MCP Servers. The Client: * Discovers what a server offers (tools, resources, prompts) * Routes requests from Host to appropriate Server * Translates Host requests into JSON-RPC format * Deserializes Server responses back to Host format ### Server: The Tool Provider[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#server-the-tool-provider "Direct link to Server: The Tool Provider") The **MCP Server** is a standalone process (Python, Node.js, etc.) that exposes capabilities: * **Tools**: Executable functions with inputs and outputs * **Resources**: Files, database records, API data * **Prompts**: Template prompts that encode domain expertise MCP Servers are typically simple services. A GitHub MCP Server provides tools for creating issues, fetching PRs, and managing workflows. A Database MCP Server provides tools for querying and updating records. A Filesystem Server provides tools for reading files, searching directories, and listing folders. Each Server is stateless and decoupled from the Host. Communication: JSON-RPC 2.0[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#communication-json-rpc-20 "Direct link to Communication: JSON-RPC 2.0") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All MCP communication uses **JSON-RPC 2.0**—a simple protocol for calling functions remotely using JSON messages. **What is JSON-RPC?** It's a way to say "call this function with these parameters" using a JSON object. The "RPC" stands for Remote Procedure Call—the client sends a message asking a remote server to execute a function, the server executes it locally, and sends back the result. The client never runs the code; it just receives the answer. The "2.0" is the version that adds features like batch requests and named parameters. **Why JSON-RPC?** It's language-agnostic (works with Python, TypeScript, Go, anything), human-readable (easy to debug), and battle-tested (used by Ethereum, Discord, VS Code's Language Server Protocol, and thousands of other systems). Every JSON-RPC message has: * `jsonrpc`: Always "2.0" (the version) * `id`: A number to match requests with responses * `method`: The function to call (for requests) * `params`: The function arguments (for requests) * `result` or `error`: The return value (for responses) Example: Host wants to list files in a directory **Request** (Host → Client → Server): { "jsonrpc": "2.0", "id": 1, "method": "resources/list", "params": {}} **Response** (Server → Client → Host): { "jsonrpc": "2.0", "id": 1, "result": { "resources": [ { "uri": "file:///path/to/file.py", "name": "file.py", "mimeType": "text/x-python" } ] }} JSON-RPC is simple because it has to work across incompatible systems. It's universal—whether your MCP Server runs locally (stdio transport) or remotely (HTTP/SSE), the protocol is identical. The Three Primitives: What Servers Provide[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#the-three-primitives-what-servers-provide "Direct link to The Three Primitives: What Servers Provide") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every MCP Server exposes capabilities through three distinct types. This lesson provides an overview—upcoming lessons cover each primitive in depth. ### 1\. Tools: Actions the Server Can Execute[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#1-tools-actions-the-server-can-execute "Direct link to 1. Tools: Actions the Server Can Execute") Tools are **functions** the Server implements. The Host sends a request; the Server executes and returns results. Tool schema defines: * **name**: Unique identifier * **description**: What it does * **inputSchema**: JSON Schema for parameters Example: GitHub Server provides a `create_issue` tool { "name": "create_issue", "description": "Create a new GitHub issue", "inputSchema": { "type": "object", "properties": { "repo": {"type": "string"}, "title": {"type": "string"}, "body": {"type": "string"} } }} ### 2\. Resources: Data the Server Can Access[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#2-resources-data-the-server-can-access "Direct link to 2. Resources: Data the Server Can Access") Resources are **read-only data** exposed by the Server. The Host can list them and fetch their contents. Resource examples: * A Filesystem Server provides resources for every file in a directory * A Database Server provides resources representing tables or query results * A GitHub Server provides resources for issues, PRs, code comments You can't modify resources through MCP (that's what tools are for), but you can read them and search them. ### 3\. Prompts: Templates Encoding Domain Expertise[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#3-prompts-templates-encoding-domain-expertise "Direct link to 3. Prompts: Templates Encoding Domain Expertise") Prompts are **pre-written instruction templates** the Server provides. Instead of the Host writing prompts from scratch, it can request a Prompt template from the Server. Example: A Code Review Server might provide a "security\_review" prompt: Review this code for security vulnerabilities.Code:[inserted by client]Checklist:- SQL injection points- Authentication bypasses- Data exposure risks Rather than hardcoding this prompt in your Client, the Server provides it. If the Server's maintainers discover better security questions, they update the prompt once—and every Client using that Server immediately gets the improvement. MCP in the Agent Stack: Where It Fits[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#mcp-in-the-agent-stack-where-it-fits "Direct link to MCP in the Agent Stack: Where It Fits") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Previous chapters**: You used SDKs to call tools directly Your Code → SDK Client → Model.chat() → Tool Call → SDK Tool Handler → Response **This chapter onward (with MCP)**: You use SDKs to call MCP Servers Your Code → SDK Client → Model.chat() → Tool Call → MCP Client → MCP Server → Response MCP sits **between the Agent and external services**. The Agent doesn't know (or care) that it's using MCP—it just calls tools. The MCP Client translates those calls into MCP format, routes to the appropriate Server, and handles responses. This is why MCP is powerful: **Your agent code doesn't change.** You add an MCP Client, point it to MCP Servers, and suddenly your agent has access to any service that provides an MCP Server. Adoption: The Convergence Signal[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#adoption-the-convergence-signal "Direct link to Adoption: The Convergence Signal") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **November 2024**: Anthropic releases MCP publicly **December 2024**: Multiple MCP servers emerge (GitHub, filesystem, databases, Slack) **March 2025**: OpenAI officially adopts MCP in ChatGPT and Agents SDK **June 2025**: MCP 2025-06-18 release adds OAuth authorization and structured outputs **September 2025**: MCP Registry preview launched for server discovery **December 2025**: Anthropic donates MCP to the Agentic AI Foundation (AAIF) under Linux Foundation governance ### The Agentic AI Foundation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#the-agentic-ai-foundation "Direct link to The Agentic AI Foundation") On December 9, 2025, MCP became a founding project of the **Agentic AI Foundation (AAIF)**—a Linux Foundation initiative co-founded by Anthropic, Block, and OpenAI, with support from Google, Microsoft, AWS, Cloudflare, and Bloomberg. This matters because: * **Vendor neutrality**: MCP is no longer "Anthropic's protocol"—it's industry infrastructure * **Competing companies cooperating**: OpenAI (competitor) and Google (competitor) are now co-stewards * **Enterprise confidence**: Fortune 500 companies can adopt MCP knowing it won't be controlled by one vendor **MCP adoption by the numbers** (December 2025): * 10,000+ public MCP servers * 97 million+ monthly SDK downloads (Python + TypeScript) * Adopted by ChatGPT, Cursor, Gemini, Microsoft Copilot, VS Code, and Claude When direct competitors donate their standards to a neutral foundation and co-govern them together, you're not learning a framework—you're learning infrastructure. Why This Matters for Digital FTE Production[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#why-this-matters-for-digital-fte-production "Direct link to Why This Matters for Digital FTE Production") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As you build Digital FTEs in later chapters, you'll need agents that interact with real systems: * **Accessing your customer's codebase** → MCP Filesystem Server * **Querying customer databases** → MCP Database Server * **Creating tickets in their project management** → MCP GitHub/Jira Server * **Pulling documents from knowledge base** → MCP Resource Server MCP solves a critical problem: Your Digital FTE can't require customers to learn three different tool integration patterns. It needs to work the same way everywhere. MCP gives you that standardization. In the next chapter, you'll build your own MCP Servers. In later chapters, you'll compose multiple servers to create integrated workflows. Understanding MCP architecture now is the foundation for that capability. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude Code or your AI companion (ChatGPT, Gemini) for these exercises. ### Prompt 1: Visualize the Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#prompt-1-visualize-the-architecture "Direct link to Prompt 1: Visualize the Architecture") **Setup**: You're explaining MCP to a developer friend who's used only OpenAI SDK **What you're learning**: Distinguishing between architectural layers (Host/Client/Server) and understanding how scaling to multiple capabilities requires multiple Servers, not multiple Clients per Server. I've been using OpenAI's SDK for months, but I keep hearing aboutMCP. Help me understand the architecture by answering these questionsabout the system below:System: I want my agent to create GitHub issues using MCP- Where does the Host fit in?- Where does the Client fit in?- Where does the Server fit in?- If I wanted to also add a filesystem MCP Server, would I need multiple Clients or multiple Servers?Use a simple ASCII diagram showing how requests flow. ### Prompt 2: Compare Integration Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#prompt-2-compare-integration-patterns "Direct link to Prompt 2: Compare Integration Patterns") **Setup**: Deciding whether to use MCP or implement custom integration **What you're learning**: Recognizing when standardization creates value (reusability, multi-platform support) versus when custom integration is justified (one-off need, deep framework coupling). I need my agent to access my company's Postgres database.I could:1. Implement direct database tools in my agent using the Anthropic SDK2. Use an MCP Database ServerFor each option, walk me through:- How the agent requests data- What happens if I want the same agent to work in ChatGPT- What happens if another developer wants to reuse my database access- Maintenance burden when the schema changesWhich approach scales better? When would you pick custom integration over MCP? ### Prompt 3: Connect to Your Future Work[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#prompt-3-connect-to-your-future-work "Direct link to Prompt 3: Connect to Your Future Work") **Setup**: Looking ahead to when you'll build MCP Servers **What you're learning**: Understanding why MCP Servers are designed for reusability across multiple clients, not just your one agent. This prepares you for the implementation work ahead. I'm learning MCP now, and I know the next chapter teaches me to build my ownMCP Servers. Imagine I'm building a "code analysis" MCP Server that:- Provides tools for analyzing code quality- Provides resources showing analysis results- Provides prompts for common review scenariosFor this server, help me:1. Define the JSON schema for 2-3 tools (code quality analysis, style checking)2. Describe what resources this server would expose3. Explain what domain expertise a prompt from this server would encodeHow is this different from just writing these tools directly in my agent? ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#safety-note "Direct link to Safety Note") As you work with MCP, remember: MCP Servers can expose sensitive data (database credentials, private files, API keys). Always review Server implementations carefully before connecting them to production agents. MCP provides the protocol—security depends on correct implementation. Checking access... * [Why MCP Exists: The Integration Explosion Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#why-mcp-exists-the-integration-explosion-problem) * [Bonus: SDK Tool Schema Unification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#bonus-sdk-tool-schema-unification) * [The Host-Client-Server Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#the-host-client-server-architecture) * [Host: Where Humans Work](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#host-where-humans-work) * [Client: The Connection Manager](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#client-the-connection-manager) * [Server: The Tool Provider](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#server-the-tool-provider) * [Communication: JSON-RPC 2.0](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#communication-json-rpc-20) * [The Three Primitives: What Servers Provide](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#the-three-primitives-what-servers-provide) * [1\. Tools: Actions the Server Can Execute](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#1-tools-actions-the-server-can-execute) * [2\. Resources: Data the Server Can Access](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#2-resources-data-the-server-can-access) * [3\. Prompts: Templates Encoding Domain Expertise](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#3-prompts-templates-encoding-domain-expertise) * [MCP in the Agent Stack: Where It Fits](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#mcp-in-the-agent-stack-where-it-fits) * [Adoption: The Convergence Signal](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#adoption-the-convergence-signal) * [The Agentic AI Foundation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#the-agentic-ai-foundation) * [Why This Matters for Digital FTE Production](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#why-this-matters-for-digital-fte-production) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#try-with-ai) * [Prompt 1: Visualize the Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#prompt-1-visualize-the-architecture) * [Prompt 2: Compare Integration Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#prompt-2-compare-integration-patterns) * [Prompt 3: Connect to Your Future Work](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#prompt-3-connect-to-your-future-work) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/mcp-architecture-overview#safety-note) --- # Vector Embeddings Mental Model | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You're building a task management system. A user searches for "deployment issues" but your database contains tasks with descriptions like "container failed to start in production" and "Kubernetes pod keeps crashing after rollout." These are clearly related, but traditional keyword search finds nothing because no task contains the word "deployment." This is the fundamental limitation of keyword search: it matches what you typed, not what you meant. Vector embeddings solve this problem by converting text into numbers that represent meaning. Once text becomes numbers, "deployment issues" and "container failed to start in production" become mathematically close because they describe related concepts. Your search engine can now find semantically similar content, even when the words are completely different. This lesson builds the mental model you need to understand RAG systems. By the end, you'll understand why embeddings work, how similarity is measured, and what embedding models actually produce. Text as Numbers: The Core Insight[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#text-as-numbers-the-core-insight "Direct link to Text as Numbers: The Core Insight") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Traditional computers store text as character sequences. The word "dog" is stored as three characters: d-o-g. The word "puppy" is stored as five different characters: p-u-p-p-y. To a computer doing string comparison, these words have almost nothing in common. But you know immediately that "dog" and "puppy" are closely related. They're both young canines. If someone asks for information about dogs, content mentioning puppies is probably relevant. **Embeddings capture this human intuition mathematically.** An embedding model converts text into a list of numbers (a "vector") where semantically similar texts get similar numbers. Here's the mental model: imagine every piece of text has a unique location on a map. Related texts are located near each other. The embedding is the GPS coordinates for that location. | Text | Meaning | Embedding (simplified) | | --- | --- | --- | | "dog" | A canine animal | \[0.82, 0.15, 0.43, ...\] | | "puppy" | A young dog | \[0.79, 0.18, 0.45, ...\] | | "democracy" | A political system | \[0.12, 0.91, 0.03, ...\] | Notice: "dog" and "puppy" have similar coordinates. "democracy" has completely different coordinates. The numbers encode meaning, not spelling. What Embedding Models Actually Produce[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#what-embedding-models-actually-produce "Direct link to What Embedding Models Actually Produce") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you send text to an embedding model like OpenAI's `text-embedding-3-small`, you get back a list of 1,536 numbers. Each number is a dimension in a high-dimensional space. from openai import OpenAIclient = OpenAI()response = client.embeddings.create( input="Deploy the application to production", model="text-embedding-3-small")embedding = response.data[0].embeddingprint(f"Dimensions: {len(embedding)}")print(f"First 5 values: {embedding[:5]}") **Output:** Dimensions: 1536First 5 values: [0.0231, -0.0142, 0.0089, 0.0315, -0.0067] Each dimension captures some aspect of meaning. You can't interpret individual dimensions ("dimension 742 means 'technology-related'")—the meaning emerges from patterns across all 1,536 dimensions working together. **Why 1,536 dimensions?** More dimensions allow the model to capture more nuanced differences in meaning. OpenAI offers `text-embedding-3-large` with 3,072 dimensions for even finer distinctions, but `text-embedding-3-small` provides excellent quality for most RAG applications while being faster and cheaper. Visualizing the Embedding Space[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#visualizing-the-embedding-space "Direct link to Visualizing the Embedding Space") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Imagine taking those 1,536 dimensions and projecting them onto a 2D surface (like pressing a 3D globe flat onto a map). Similar texts cluster together: Embedding Space (2D Projection) Technology Cluster Business Cluster ┌─────────────────┐ ┌─────────────────┐ │ "Docker" │ │ "revenue" │ │ ● │ │ ● │ │ "container" ● │ │ ● "profit" │ │ ● "K8s" │ │ "sales" ● │ └─────────────────┘ └─────────────────┘ Nature Cluster ┌─────────────────┐ │ "forest" ● │ │ ● "tree" │ │ "leaves" ● │ └─────────────────┘ When a user searches for "containerization strategy," their query lands near the Technology Cluster. The system finds the nearest texts—"Docker," "container," "K8s"—and returns them as relevant results. This is why semantic search works even when the query contains no overlapping keywords with the documents. The query and documents occupy nearby regions in the embedding space. Measuring Similarity: Cosine Similarity[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#measuring-similarity-cosine-similarity "Direct link to Measuring Similarity: Cosine Similarity") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once texts become vectors, we need a way to measure how close they are. The standard approach is **cosine similarity**, which measures the angle between two vectors. Think of two arrows pointing from the same origin: * If they point in the exact same direction: similarity = 1.0 (identical meaning) * If they point at right angles: similarity = 0.0 (unrelated) * If they point in opposite directions: similarity = -1.0 (opposite meaning) In practice, most text embeddings have positive similarity scores between 0 and 1: | Comparison | Cosine Similarity | Interpretation | | --- | --- | --- | | "dog" vs "puppy" | 0.92 | Very similar | | "dog" vs "cat" | 0.78 | Related (both animals) | | "dog" vs "automobile" | 0.31 | Weakly related | | "dog" vs "quantum mechanics" | 0.08 | Essentially unrelated | The formula involves dot products and magnitudes, but the intuition is simple: higher scores mean closer meaning. import numpy as npdef cosine_similarity(vec1, vec2): """Calculate cosine similarity between two vectors.""" dot_product = np.dot(vec1, vec2) magnitude1 = np.linalg.norm(vec1) magnitude2 = np.linalg.norm(vec2) return dot_product / (magnitude1 * magnitude2) **Output (conceptual example):** similarity = cosine_similarity(dog_embedding, puppy_embedding)print(f"dog vs puppy: {similarity:.2f}") # dog vs puppy: 0.92 You rarely compute cosine similarity manually. Vector databases like Qdrant handle this automatically when you search. But understanding that similarity is measured by vector angles helps you interpret results and debug issues. Why This Enables RAG[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#why-this-enables-rag "Direct link to Why This Enables RAG") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Retrieval-Augmented Generation works because: 1. **Documents become vectors**: When you add documents to your knowledge base, each chunk gets converted to an embedding and stored 2. **Queries become vectors**: When a user asks a question, that query gets converted to an embedding 3. **Vector search finds matches**: The database finds documents whose embeddings are closest to the query embedding 4. **LLM uses context**: Those relevant documents become context for the LLM to generate an accurate answer The embedding model is the translator between human language and mathematical space. It enables finding relevant content by meaning, not just string matching. User Query: "Why is my container crashing?" ↓ Embedding Model ↓Query Vector: [0.23, -0.11, 0.45, ...] ↓ Vector Database Search ↓Found: "Kubernetes pod restart loop after OOM error" "Container exits with memory allocation failure" "Docker container terminated unexpectedly" ↓ These become context for LLM ↓Answer: "Your container is likely crashing due to memory issues..." Without embeddings, the query "Why is my container crashing?" wouldn't match documents containing "pod restart loop" or "OOM error." With embeddings, the semantic relationship is captured mathematically. Common Misconceptions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#common-misconceptions "Direct link to Common Misconceptions") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Misconception 1: Longer text = better embeddings** Not necessarily. Embedding models have token limits (typically 8,000 tokens for OpenAI models). More importantly, longer texts dilute specificity. A 50-word paragraph about Docker gets a more focused embedding than a 5,000-word document covering Docker, Kubernetes, security, and databases. This is why RAG systems chunk documents into smaller pieces—each chunk gets a focused embedding. **Misconception 2: Embeddings understand context** Embeddings capture semantic meaning at the moment of encoding, but they don't "understand" in the human sense. The embedding for "bank" encodes multiple meanings (financial institution, river edge) simultaneously. Context comes from how you structure your RAG system, not from the embedding alone. **Misconception 3: All embedding models are equivalent** Different models capture meaning differently. OpenAI's models are trained on broad internet text. Domain-specific models (legal, medical, code) may perform better for specialized content. Model selection matters for retrieval quality. Choosing an Embedding Model[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#choosing-an-embedding-model "Direct link to Choosing an Embedding Model") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ For most RAG applications, OpenAI's `text-embedding-3-small` provides excellent quality at reasonable cost: | Model | Dimensions | Best For | | --- | --- | --- | | `text-embedding-3-small` | 1,536 | General RAG, cost-effective | | `text-embedding-3-large` | 3,072 | High-precision retrieval | | Domain-specific models | Varies | Legal, medical, scientific domains | **Key consideration**: Once you embed documents with a specific model, you must use the same model for query embeddings. You cannot mix models—their embedding spaces are incompatible. Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#reflect-on-your-skill "Direct link to Reflect on Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You built a `rag-deployment` skill in Lesson 0. Now test whether it understands embeddings. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#test-your-skill "Direct link to Test Your Skill") Ask your skill: I have 1,000 documents about software deployment. Explain how embeddingswould help users find relevant documents when they search for "rollbackstrategy" but none of the documents contain the word "rollback." ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#identify-gaps "Direct link to Identify Gaps") Does your skill's response: * Explain that embeddings capture semantic meaning, not keywords? * Mention that synonyms like "revert," "undo," or "restore" would cluster nearby? * Distinguish semantic search from keyword search? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#improve-your-skill "Direct link to Improve Your Skill") If your skill's response was weak on embeddings, update it with the mental model from this lesson. Add these concepts: * Embeddings as "GPS coordinates for meaning" * Cosine similarity as "distance between concepts" * Why chunking matters (focused embeddings vs diluted embeddings) * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Use your AI companion (Claude, ChatGPT, Gemini, or similar). ### Prompt 1: Explore Semantic Similarity[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#prompt-1-explore-semantic-similarity "Direct link to Prompt 1: Explore Semantic Similarity") I'm learning about embeddings. Generate 5 pairs of sentences:3 pairs that would have HIGH cosine similarity (similar meaning, different words)and 2 pairs that would have LOW similarity (look similar but mean different things).Explain your reasoning for each pair. **What you're learning**: Developing intuition for what the embedding model captures. You'll discover that meaning, not surface similarity, determines embedding closeness. ### Prompt 2: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#prompt-2-apply-to-your-domain "Direct link to Prompt 2: Apply to Your Domain") I work with [your domain: legal documents / customer support / technical docs].Give me 3 examples of queries that would fail with keyword search but succeedwith semantic search in my domain. For each, explain what documents would befound and why the embeddings would cluster together. **What you're learning**: Translating the abstract embedding concept to practical relevance in your specific field. ### Prompt 3: Debug a Retrieval Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#prompt-3-debug-a-retrieval-problem "Direct link to Prompt 3: Debug a Retrieval Problem") Imagine my RAG system is returning irrelevant results. A user searches for"authentication problems" but gets documents about "database configuration"instead. Walk me through a debugging process: What could cause this?How would I diagnose whether it's an embedding issue, a chunking issue,or something else? **What you're learning**: The practical debugging mindset for RAG systems. You'll connect the conceptual model (embeddings as meaning) to operational troubleshooting. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#safety-note "Direct link to Safety Note") When experimenting with embeddings, be mindful of API costs. OpenAI charges per token embedded. For learning and testing, use short sample texts rather than embedding entire documents repeatedly. Checking access... * [Text as Numbers: The Core Insight](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#text-as-numbers-the-core-insight) * [What Embedding Models Actually Produce](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#what-embedding-models-actually-produce) * [Visualizing the Embedding Space](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#visualizing-the-embedding-space) * [Measuring Similarity: Cosine Similarity](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#measuring-similarity-cosine-similarity) * [Why This Enables RAG](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#why-this-enables-rag) * [Common Misconceptions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#common-misconceptions) * [Choosing an Embedding Model](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#choosing-an-embedding-model) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#improve-your-skill) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#try-with-ai) * [Prompt 1: Explore Semantic Similarity](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#prompt-1-explore-semantic-similarity) * [Prompt 2: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#prompt-2-apply-to-your-domain) * [Prompt 3: Debug a Retrieval Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#prompt-3-debug-a-retrieval-problem) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/vector-embeddings-mental-model#safety-note) --- # Why Agents Need External Knowledge | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your Task API from Chapter 70 stores hundreds of tasks. A user asks: "Show me tasks related to Docker deployment." Right now, you can only filter by exact fields like status or priority. You can search for "pending" but not for "anything about containers." What if your API could understand _meaning_? What if it could find tasks about "Docker deployment" even when the word "Docker" never appears - because it understands that "containerization," "image build," and "Kubernetes pod" are semantically related? This is what RAG enables. But before we build it, we need to understand _why_ agents need external knowledge in the first place. The Two Types of Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-two-types-of-memory "Direct link to The Two Types of Memory") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When you ask Claude or GPT a question, where does the answer come from? The model has two fundamentally different sources of knowledge: ### Parametric Memory: What the Model "Learned"[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#parametric-memory-what-the-model-learned "Direct link to Parametric Memory: What the Model "Learned"") During training, models process billions of documents. Patterns from this data get encoded into the model's parameters - the billions of numbers that define how it responds. This is **parametric memory**. Think of it like your own brain after years of education. You don't consciously remember reading every textbook, but the knowledge shaped how you think. When someone asks "What's the capital of France?", you don't retrieve a specific memory - you just _know_ it's Paris. **Strengths of parametric memory:** * Fast - no lookup required * Generalizes well - understands patterns beyond specific examples * Handles language nuance - knows "Paris" means France's capital, not Paris, Texas **Limitations of parametric memory:** * Frozen at training time - can't learn new information * No source attribution - can't tell you _where_ it learned something * Prone to confident errors - may "know" things that aren't true ### Non-Parametric Memory: What the Model Retrieves[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#non-parametric-memory-what-the-model-retrieves "Direct link to Non-Parametric Memory: What the Model Retrieves") The alternative is **non-parametric memory** - looking things up at query time. Instead of relying solely on what was learned during training, the model retrieves relevant documents and uses them to generate a response. Think of it like using a reference library. You don't need to memorize every fact - you know how to find the right book and extract the answer. **Strengths of non-parametric memory:** * Always current - retrieve the latest documents * Verifiable - can cite sources * Scalable - add new knowledge without retraining **Limitations of non-parametric memory:** * Slower - requires search before generation * Quality depends on retrieval - wrong documents produce wrong answers * Requires infrastructure - need to store and index documents | Memory Type | Source | Speed | Can Update? | Can Cite? | | --- | --- | --- | --- | --- | | **Parametric** | Training data encoded in weights | Fast | No (requires retraining) | No | | **Non-Parametric** | Retrieved documents | Slower | Yes (update documents) | Yes | The Knowledge Cutoff Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-knowledge-cutoff-problem "Direct link to The Knowledge Cutoff Problem") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's a concrete problem: Every model has a training cutoff date. Claude's training data ends at some point. So does GPT-4's. Everything that happened after that date - new APIs, updated documentation, recent events - doesn't exist in parametric memory. **Consequences for agents:** 1. **Outdated answers** - Ask about a library released after the cutoff, and the model doesn't know it exists 2. **Wrong version information** - The model might confidently describe an old API that has since changed 3. **Missing context** - If your company added a new product last month, the model has never heard of it This matters enormously for agent applications. Imagine an agent that helps customers with your product. If your product documentation changed after the model's training cutoff, the agent gives outdated guidance - confidently, without knowing it's wrong. The Hallucination Risk[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-hallucination-risk "Direct link to The Hallucination Risk") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When models don't know something, they don't say "I don't know." They generate plausible-sounding answers. This is called **hallucination**. **Why does this happen?** Language models are trained to predict what text should come next. When asked about something outside their training, they still produce text that _sounds_ like a good answer. The model has learned patterns like "When asked about a library, describe its main function and give an example." It applies these patterns even when it doesn't actually know the library. **Real-world impact:** * A legal research agent invents case citations that don't exist * A coding assistant suggests API methods that were never implemented * A customer support agent describes product features that aren't real The model isn't lying - it genuinely doesn't distinguish between what it knows and what it's generating to fill gaps. From the model's perspective, both feel the same: predicting the next likely token. **When hallucination risk is highest:** | Scenario | Risk Level | Why | | --- | --- | --- | | General knowledge (history, science) | Lower | Extensively covered in training data | | Recent events (last few months) | Higher | After knowledge cutoff | | Specific internal data (your company docs) | Very High | Never in training data | | Precise technical details (exact API syntax) | Higher | Easy to confuse similar patterns | RAG: The Solution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#rag-the-solution "Direct link to RAG: The Solution") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Retrieval-Augmented Generation (RAG)** solves these problems by combining both memory types. Instead of relying solely on parametric memory, RAG systems: 1. **Retrieve** relevant documents based on the user's query 2. **Augment** the prompt with retrieved context 3. **Generate** a response grounded in the retrieved information User Query: "How do I deploy our app to Kubernetes?" ↓ ┌────────────────────────────────────────────┐ │ 1. RETRIEVE │ │ Search your documentation for relevant │ │ pages about Kubernetes deployment │ └────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────┐ │ 2. AUGMENT │ │ Add retrieved docs to the prompt: │ │ "Using this context: [deployment guide] │ │ Answer: How do I deploy to Kubernetes?" │ └────────────────────────────────────────────┘ ↓ ┌────────────────────────────────────────────┐ │ 3. GENERATE │ │ LLM produces response based on YOUR │ │ documentation, not just training data │ └────────────────────────────────────────────┘ ↓Response grounded in your actual deployment guide **Why this works:** * **Current information** - Retrieved documents can be updated anytime * **Source attribution** - The response comes from specific documents you control * **Reduced hallucination** - The model has actual context instead of guessing * **Domain-specific** - Works with your internal knowledge, not just public data RAG for Your Task API[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#rag-for-your-task-api "Direct link to RAG for Your Task API") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's bring this back to our running example. You have a Task API with these tasks: [ {"id": 1, "title": "Set up Docker image build pipeline", "status": "pending"}, {"id": 2, "title": "Configure Kubernetes pod manifests", "status": "in_progress"}, {"id": 3, "title": "Optimize container resource limits", "status": "completed"}, {"id": 4, "title": "Review authentication flow", "status": "pending"}] **Without RAG (current state):** * User asks: "Show me tasks related to deployment" * Your API can only filter by exact fields: `status=pending` * No way to understand that tasks 1, 2, and 3 are all about deployment **With RAG (what you'll build):** * User asks: "Show me tasks related to deployment" * RAG retrieves tasks whose descriptions are semantically similar to "deployment" * Returns tasks 1, 2, and 3 - even though none contain the word "deployment" This is what the rest of this chapter teaches: how to add semantic understanding to your Task API. The Mental Model You Need[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-mental-model-you-need "Direct link to The Mental Model You Need") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before diving into implementation details (embeddings, vector databases, retrieval chains), internalize this mental model: **Parametric memory** is like your long-term memory after education. You absorbed knowledge during training (school), and now you just "know" things. But you can't easily update it, and you might misremember details. **Non-parametric memory** is like having a library at your fingertips. You haven't memorized every book, but you know how to find the right one and extract what you need. You can add new books anytime. **RAG** is using the library to supplement your education. When asked a question, you first check if there's a relevant book, then use that book plus your general knowledge to answer well. This mental model will guide you through the rest of the chapter. When you learn about embeddings (Lesson 2), you're learning how to organize the library so you can find the right book. When you learn about vector databases (Lesson 4), you're learning where to store the library. When you learn about retrieval chains (Lesson 5), you're learning how to read the book and combine it with what you already know. Key Takeaways for Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#key-takeaways-for-agents "Direct link to Key Takeaways for Agents") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you're building agents, RAG is essential because: 1. **Agents need current information** - Your agent advises users about your product. The product changes weekly. Without RAG, the agent gives outdated advice. 2. **Agents need internal knowledge** - Your company's processes aren't in any training data. Without RAG, the agent can't help with company-specific questions. 3. **Agents need verifiable responses** - When an agent makes a claim, users want to know where it came from. RAG enables source attribution. 4. **Agents need to avoid hallucination** - Confident wrong answers destroy user trust. RAG grounds responses in actual documents. The rest of this chapter gives you the practical skills to implement RAG. By Lesson 8, you'll have extended your Task API with semantic search and applied advanced RAG patterns. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you understand why agents need external knowledge, explore these concepts further with AI assistance. **Prompt 1: Identify Memory Types in Practice** I'm reading about how Claude and GPT work. When I ask "What year did Python 3 release?" - is that answer coming from parametric memory or non-parametric memory? What about when I paste my company's documentation into the chat and then ask about it? Walk me through the difference. **What you're learning:** This prompt helps you recognize how memory types work in practice. The first question (Python 3 release date) uses parametric memory - it's in training data. The second (your pasted docs) is essentially manual RAG - you're augmenting the prompt with retrieved context. Understanding this distinction prepares you for implementing automatic retrieval. **Prompt 2: Evaluate Hallucination Scenarios** I'm building an agent for [choose: customer support / code documentation / internal wiki search].For each of these three scenarios, rate the hallucination risk (low/medium/high) and explain why:1. User asks about general programming concepts2. User asks about our specific product features3. User asks about an API released last monthThen suggest which scenarios most urgently need RAG. **What you're learning:** This prompt develops your judgment about when RAG adds value. Not every query needs retrieval - general knowledge questions are often fine with parametric memory. But domain-specific and time-sensitive queries are high-risk without RAG. You're learning to prioritize where to apply the techniques you'll learn. **Prompt 3: Design a RAG System for Your Domain** I have a [describe your data source - product docs, internal wiki, customer support tickets, code repository].Before writing any code, help me think through:1. What types of questions will users ask?2. What documents should I retrieve for each question type?3. How would I know if retrieval found the right documents?4. What happens if no relevant documents exist?I want to understand the design before implementation. **What you're learning:** This prompt connects RAG concepts to your real work. You're not asking AI to build a system - you're thinking through design decisions that matter: what data sources, what queries, what success looks like. This prepares you to make informed choices when you implement RAG in Lessons 3-6. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built a `rag-deployment` skill in Lesson 0. Even before learning implementation details, you can improve it based on the conceptual understanding from this lesson. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#test-your-skill "Direct link to Test Your Skill") Using my rag-deployment skill, explain when a project should use RAG versus relying on the LLM's parametric memory alone. Give me a decision framework with specific criteria. ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill mention knowledge cutoffs and why they matter? * Did it explain the trade-offs between parametric and non-parametric memory? * Did it help identify hallucination risk scenarios? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#improve-your-skill "Direct link to Improve Your Skill") If you found gaps, update your skill: My rag-deployment skill is missing decision criteria for when to use RAG.Update the skill to include:1. A checklist for identifying when RAG is necessary (knowledge cutoff, domain-specific data, verifiability needs)2. Warning signs that an LLM-only approach will hallucinate3. The retrieve-augment-generate pattern as a core conceptShow me the updated SKILL.md with these additions. **Your skill now has conceptual foundations.** The next lessons will add implementation details. Checking access... * [The Two Types of Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-two-types-of-memory) * [Parametric Memory: What the Model "Learned"](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#parametric-memory-what-the-model-learned) * [Non-Parametric Memory: What the Model Retrieves](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#non-parametric-memory-what-the-model-retrieves) * [The Knowledge Cutoff Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-knowledge-cutoff-problem) * [The Hallucination Risk](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-hallucination-risk) * [RAG: The Solution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#rag-the-solution) * [RAG for Your Task API](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#rag-for-your-task-api) * [The Mental Model You Need](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#the-mental-model-you-need) * [Key Takeaways for Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#key-takeaways-for-agents) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#try-with-ai) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/why-agents-need-external-knowledge#improve-your-skill) --- # Memory Architecture Patterns | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C When we designed the first personal computers, we looked to the human brain for inspiration. Working memory, long-term storage, recall mechanisms—these cognitive science concepts shaped computer architecture. The same is happening now with AI agents. Effective agent memory isn't one thing—it's a system of specialized memory types, each serving a different purpose. Understanding these types helps you design agents that remember the right things in the right ways. This lesson introduces five memory types and shows how they work together. We'll also look at how systems like Letta (formerly MemGPT) implement a two-tier architecture that mirrors how the brain handles in-focus vs. background information. Five Types of Agent Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#five-types-of-agent-memory "Direct link to Five Types of Agent Memory") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Cognitive science distinguishes several memory systems in the human brain. Agent systems have parallel concepts: ┌────────────────────────────────────────────────────────────────────────────┐│ AGENT MEMORY SYSTEM │├────────────────┬─────────────────┬─────────────────────────────────────────┤│ Conversation │ Working │ Long-term ││ Memory │ Memory │ Memory ││ │ │ ││ Recent │ Current task │ ┌─────────────────────────────────┐ ││ messages │ context │ │ Episodic Memory │ ││ │ │ │ (specific events) │ ││ Sliding │ Active goal │ ├─────────────────────────────────┤ ││ window │ + state │ │ Semantic Memory │ ││ │ │ │ (facts & entities) │ ││ Volatile │ Task-scoped │ └─────────────────────────────────┘ ││ (session) │ (cleared) │ Persistent │└────────────────┴─────────────────┴─────────────────────────────────────────┘ ### 1\. Conversation Memory (Short-term)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#1-conversation-memory-short-term "Direct link to 1. Conversation Memory (Short-term)") **What it is:** The immediate context of the current conversation—recent messages between user and agent. **Human analogy:** Working memory in a conversation. You remember what was just said, but after a day you might forget the exact wording. **Characteristics:** * Ephemeral: exists only during active session * Typically a sliding window of recent turns * Automatically included in LLM context * Lost when session ends **Example:** Conversation Memory (last 5 turns):1. User: "Add 'Review PR #123' to my tasks"2. Agent: "Added 'Review PR #123' with normal priority"3. User: "Make it high priority"4. Agent: "Updated to high priority"5. User: "When is it due?" **Best for:** Maintaining coherence within a single conversation session. ### 2\. Working Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#2-working-memory "Direct link to 2. Working Memory") **What it is:** Active context about the current task being performed—goals, intermediate results, and state. **Human analogy:** The mental workspace when solving a problem. You hold relevant pieces in mind while working. **Characteristics:** * Task-scoped: exists for duration of a task * Cleared when task completes * Holds current goal and progress * More structured than conversation memory **Example:** Working Memory (current task):{ "goal": "Create weekly status report", "status": "in_progress", "steps_completed": [ "Gathered task completions from last week", "Identified blockers" ], "current_step": "Drafting summary", "context": { "report_format": "bullet_points", "audience": "engineering_team" }} **Best for:** Multi-step tasks where the agent needs to track progress and intermediate state. ### 3\. Long-term Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#3-long-term-memory "Direct link to 3. Long-term Memory") The umbrella term for persistent memories that survive across sessions. Subdivides into episodic and semantic memory. ### 4\. Episodic Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#4-episodic-memory "Direct link to 4. Episodic Memory") **What it is:** Records of specific events and interactions—what happened, when, with whom. **Human analogy:** Autobiographical memory. "Last Tuesday I met Sarah and we discussed the project deadline." **Characteristics:** * Time-stamped: knows when events occurred * Sequential: can recall order of events * Contextual: includes surrounding circumstances * Decays over time (older = less detailed) **Example:** Episodic Memory:[ { "timestamp": "2025-01-15T10:30:00Z", "event": "User asked about authentication patterns", "outcome": "Recommended OAuth 2.0 with PKCE", "user_reaction": "positive" }, { "timestamp": "2025-01-17T14:00:00Z", "event": "User reported OAuth implementation issues", "outcome": "Debugged token refresh logic together", "resolution": "Fixed by adding retry mechanism" }, { "timestamp": "2025-01-20T09:00:00Z", "event": "User confirmed OAuth working in production", "context": "Three-day project completed successfully" }] **Best for:** Building continuity across sessions. "Last time we discussed..." patterns. ### 5\. Semantic Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#5-semantic-memory "Direct link to 5. Semantic Memory") **What it is:** Factual knowledge about entities and their relationships—general knowledge extracted from experiences. **Human analogy:** General knowledge. You know Paris is in France without remembering when you learned it. **Characteristics:** * Entity-focused: people, projects, concepts * Relationship-aware: how entities connect * Abstracted: distilled from multiple episodes * Relatively stable over time **Example:** Semantic Memory:{ "entities": { "Alex": { "type": "user", "role": "senior_engineer", "team": "platform", "preferences": ["morning_meetings", "detailed_docs"], "expertise": ["python", "kubernetes", "auth_systems"] }, "Phoenix Project": { "type": "project", "status": "active", "deadline": "2025-01-31", "involves": ["auth-service", "user-service"], "owner": "Alex" } }, "relationships": [ {"from": "Alex", "relation": "owns", "to": "Phoenix Project"}, {"from": "Phoenix Project", "relation": "uses", "to": "OAuth 2.0"} ]} **Best for:** Understanding context without explicit reminders. "The project" → Phoenix Project. Memory Type Comparison[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#memory-type-comparison "Direct link to Memory Type Comparison") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Memory Type | Persistence | Scope | Updates | Example Query | | --- | --- | --- | --- | --- | | **Conversation** | Session | Current chat | Every turn | "What did they just say?" | | **Working** | Task | Current task | Task progress | "What's our current goal?" | | **Episodic** | Long-term | Past events | After interactions | "When did we last discuss auth?" | | **Semantic** | Long-term | Entities/facts | When facts change | "What projects is Alex working on?" | The Two-Tier Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#the-two-tier-architecture "Direct link to The Two-Tier Architecture") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Research systems like Letta (formerly MemGPT) formalize a two-tier memory model that's worth understanding conceptually. ### In-Context Memory (Core Memory)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#in-context-memory-core-memory "Direct link to In-Context Memory (Core Memory)") Information that stays **always visible** to the LLM—included in every prompt without retrieval. ┌─────────────────────────────────────────────────────────────────┐│ IN-CONTEXT MEMORY ││ ││ ┌──────────────────────┐ ┌────────────────────────────┐ ││ │ Persona Block │ │ Human Block │ ││ │ │ │ │ ││ │ "I am a task │ │ "User: Alex │ ││ │ management agent │ │ Role: Senior Engineer │ ││ │ that helps with │ │ Preferences: │ ││ │ planning and │ │ - Morning meetings │ ││ │ productivity." │ │ - Bullet point format" │ ││ │ │ │ │ ││ └──────────────────────┘ └────────────────────────────┘ ││ ││ Always visible to LLM. Updated via self-editing tools. │└─────────────────────────────────────────────────────────────────┘ **Key insight:** Core memory is small but high-value. It's the agent's "active awareness" of who it is and who it's talking to. ### Out-of-Context Memory (External Memory)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#out-of-context-memory-external-memory "Direct link to Out-of-Context Memory (External Memory)") Information stored **externally** and retrieved on demand—too large to keep in context but available when needed. ┌─────────────────────────────────────────────────────────────────┐│ OUT-OF-CONTEXT MEMORY ││ ││ ┌────────────────────────────────────────────────────────┐ ││ │ Archival Memory │ ││ │ (Vector DB) │ ││ │ │ ││ │ Searchable storage for past conversations, │ ││ │ documents, facts, and extended context. │ ││ │ │ ││ │ Retrieved via: archival_memory_search(query) │ ││ └────────────────────────────────────────────────────────┘ ││ ││ ┌────────────────────────────────────────────────────────┐ ││ │ Conversation History │ ││ │ (Full Log) │ ││ │ │ ││ │ Complete message history. Too large for context. │ ││ │ │ ││ │ Retrieved via: conversation_search(query) │ ││ └────────────────────────────────────────────────────────┘ ││ ││ Retrieved on-demand. Not always visible. │└─────────────────────────────────────────────────────────────────┘ ### Self-Editing Memory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#self-editing-memory "Direct link to Self-Editing Memory") A distinguishing feature of the Letta approach: agents don't just read from memory—they actively manage it. **Memory Tools:** * `memory_replace(block, old_text, new_text)` — Update specific content * `memory_insert(block, text)` — Add new information * `memory_rethink(block, new_content)` — Complete block rewrite **Example behavior:** User: "Actually, I prefer afternoon meetings now."Agent thinks: "This contradicts the human block. I should update it."Agent action: memory_replace("human", "Morning meetings", "Afternoon meetings")Agent says: "Got it—I've updated my notes. Afternoon meetings from now on." The agent maintains its own memory rather than relying on external systems to do it. Designing Memory for Task API[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#designing-memory-for-task-api "Direct link to Designing Memory for Task API") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's apply these concepts to the Task API agent from Chapter 70. ### Memory Requirements Analysis[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#memory-requirements-analysis "Direct link to Memory Requirements Analysis") | Information Type | Memory Type | Rationale | | --- | --- | --- | | Current task list | Conversation | Immediate context | | Active task being modified | Working | Task in progress | | "Last week you completed 15 tasks" | Episodic | Past events with timing | | "Alex prefers morning tasks" | Semantic | Stable user fact | | "Phoenix project deadline is Jan 31" | Semantic | Project fact | | "Task estimates typically 50% low" | Semantic | Learned pattern | ### Architecture Diagram[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#architecture-diagram "Direct link to Architecture Diagram") ┌─────────────────────────────────────────────────────────────────┐│ TASK API AGENT MEMORY │├─────────────────────────────────────────────────────────────────┤│ ││ ┌─────────────────────────────────────────────────────────┐ ││ │ CORE MEMORY (always visible) │ ││ │ │ ││ │ Agent: "Task management specialist" │ ││ │ User: Name, preferences, working patterns │ ││ └─────────────────────────────────────────────────────────┘ ││ │ ││ │ (retrieval) ││ ▼ ││ ┌─────────────────────────────────────────────────────────┐ ││ │ EXTERNAL MEMORY (on-demand) │ ││ │ │ ││ │ Episodic: Task completion history │ ││ │ Semantic: Projects, deadlines, patterns │ ││ │ Archival: Past conversations about tasks │ ││ └─────────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────────┘ ### Example Memory Content[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#example-memory-content "Direct link to Example Memory Content") **Core Memory (Human Block):** User: Alex ChenRole: Senior Platform EngineerPreferences:- Prefers tasks created in morning (9-11am)- Uses priority labels: critical, high, normal, low- Likes estimated_hours on complex tasks- Typical task completion: 2-4 per dayWorking on: Phoenix Project (auth migration, deadline Jan 31) **Semantic Memory:** { "projects": { "Phoenix": { "type": "migration", "deadline": "2025-01-31", "components": ["auth-service", "user-service"], "status": "active" } }, "patterns": { "estimation_accuracy": 0.65, "typical_tasks_per_day": 3.2, "peak_productivity": "morning" }} **Episodic Memory (recent):** [ { "date": "2025-01-20", "event": "Completed OAuth token refresh implementation", "duration_actual": "4 hours", "duration_estimated": "2 hours", "note": "Took longer due to edge cases" }] Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these prompts to practice memory architecture design with Claude or your preferred AI assistant. ### Prompt 1: Memory Type Mapping[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#prompt-1-memory-type-mapping "Direct link to Prompt 1: Memory Type Mapping") I'm building a coding assistant agent that helps developers with:- Code review suggestions- Bug diagnosis- Documentation writing- API design guidanceFor each of these five memory types, give me concrete examples of what the agent should store:1. Conversation memory2. Working memory3. Episodic memory4. Semantic memoryAlso: What information should be in "core memory" (always visible) vs "external memory" (retrieved on-demand)? **What you're learning:** How to apply the five memory types to a specific domain. A coding assistant has very different memory needs than a task manager—code patterns, repository structure, and past debugging sessions become critical. ### Prompt 2: Episodic Memory for PR Reviews[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#prompt-2-episodic-memory-for-pr-reviews "Direct link to Prompt 2: Episodic Memory for PR Reviews") An agent reviews pull requests for a development team. Design its episodic memory system.Questions to answer:1. What events should trigger episodic memory creation?2. What metadata should each episode include?3. How would the agent use episodic memory when reviewing a new PR from the same author?4. How would it use episodic memory when reviewing a PR that touches the same files as a previous PR?Give me a JSON schema for an episodic memory entry and 3 example entries. **What you're learning:** Episodic memory isn't just "what happened"—it's about capturing context that enables better future decisions. For PR reviews, past patterns predict future needs. ### Prompt 3: Semantic Memory as Knowledge Graph[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#prompt-3-semantic-memory-as-knowledge-graph "Direct link to Prompt 3: Semantic Memory as Knowledge Graph") Design a knowledge graph structure for an agent that manages a software project.The agent needs to understand:- Team members and their roles- Services and their dependencies- Current sprints and deadlines- Technical decisions and their rationaleCreate:1. Entity types (nodes)2. Relationship types (edges)3. Example queries the agent would run4. How the agent would update this graph as the project evolvesShow a visual representation (ASCII or description) of a sample project's knowledge graph. **What you're learning:** Semantic memory shines when relationships matter. A knowledge graph lets the agent answer "Who owns the service that depends on auth-service?" without searching through conversation logs. Checking access... * [Five Types of Agent Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#five-types-of-agent-memory) * [1\. Conversation Memory (Short-term)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#1-conversation-memory-short-term) * [2\. Working Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#2-working-memory) * [3\. Long-term Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#3-long-term-memory) * [4\. Episodic Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#4-episodic-memory) * [5\. Semantic Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#5-semantic-memory) * [Memory Type Comparison](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#memory-type-comparison) * [The Two-Tier Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#the-two-tier-architecture) * [In-Context Memory (Core Memory)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#in-context-memory-core-memory) * [Out-of-Context Memory (External Memory)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#out-of-context-memory-external-memory) * [Self-Editing Memory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#self-editing-memory) * [Designing Memory for Task API](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#designing-memory-for-task-api) * [Memory Requirements Analysis](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#memory-requirements-analysis) * [Architecture Diagram](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#architecture-diagram) * [Example Memory Content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#example-memory-content) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#try-with-ai) * [Prompt 1: Memory Type Mapping](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#prompt-1-memory-type-mapping) * [Prompt 2: Episodic Memory for PR Reviews](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#prompt-2-episodic-memory-for-pr-reviews) * [Prompt 3: Semantic Memory as Knowledge Graph](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-architecture-patterns#prompt-3-semantic-memory-as-knowledge-graph) --- # ChatKit Architecture Foundations | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've built REST APIs with FastAPI. You know the pattern: request comes in, handler processes it, response goes out. Each request is independent, stateless, one-shot. Now imagine building a chat interface for your TaskManager agent. Users have ongoing conversations. The agent needs to remember what was said five messages ago. Responses stream token-by-token, updating the UI progressively. The interaction feels continuous, not transactional. This is where ChatKitServer enters. It's not a REST API framework. It's a **conversation orchestration layer** purpose-built for stateful, streaming, event-driven chat interfaces. Let's understand what makes it fundamentally different from the request/response world you already know. The Paradigm Shift: Conversations vs Requests[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#the-paradigm-shift-conversations-vs-requests "Direct link to The Paradigm Shift: Conversations vs Requests") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Request/Response World (FastAPI)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#requestresponse-world-fastapi "Direct link to Request/Response World (FastAPI)") When you build a REST API with FastAPI, you're designing around **discrete transactions**: USER → HTTP POST /api/tasks → SERVER ↓ Process request ↓SERVER → HTTP 200 + JSON → USER Each request is: * **Stateless**: Server doesn't remember previous requests * **One-shot**: Request → Response → Done * **Independent**: Each call stands alone **Example**: Creating a task: @app.post("/api/tasks")async def create_task(task: TaskCreate): new_task = Task(title=task.title) db.add(new_task) db.commit() return {"id": new_task.id, "status": "created"} Request happens. Function executes. Response returns. Connection closes. No memory of the interaction. ### Conversation World (ChatKitServer)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#conversation-world-chatkitserver "Direct link to Conversation World (ChatKitServer)") ChatKitServer operates on a different model: **ongoing relationships**: USER → Message: "Create a task for groceries" → AGENT ↓ Agent streams response token-by-token ↓AGENT → "I've created..." [streaming] → USER ↓USER → Message: "What's its priority?" → AGENT ↓ Agent recalls context from Thread history ↓AGENT → "The task you created has default priority" → USER Each interaction is: * **Stateful**: Conversation history maintained across messages * **Streaming**: Responses update progressively (token-by-token) * **Contextual**: Each message builds on previous messages in Thread **The shift**: From isolated calls to persistent conversations. Core Architecture Primitives[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#core-architecture-primitives "Direct link to Core Architecture Primitives") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ChatKitServer introduces primitives designed for conversation management: ### 1\. Thread (Conversation Container)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#1-thread-conversation-container "Direct link to 1. Thread (Conversation Container)") A **Thread** is the conversation container—like an email thread, but for chat. **What it contains**: * Thread ID (unique identifier) * Metadata (timestamps, user context) * Message history (all ThreadItems in chronological order) **Mental model**: Think of a Thread like a running conversation at a coffee shop. When you return the next day and continue where you left off, that's the Thread persisting context. **Key difference from REST**: * REST: Each request is a new conversation * ChatKit: Thread maintains ongoing conversation state ### 2\. ThreadItem (Individual Message)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#2-threaditem-individual-message "Direct link to 2. ThreadItem (Individual Message)") A **ThreadItem** is a single message within a Thread. **Types**: * **User message**: Input from the human * **Assistant message**: Response from the agent * **Tool message**: Result from tool execution (if agent uses tools) **Structure**: ThreadItem { id: "msg_abc123" role: "user" | "assistant" | "tool" content: "Create a task for groceries" timestamp: "2025-12-31T10:00:00Z"} **Key difference from REST**: * REST: Request body is ephemeral (processed and discarded) * ChatKit: Every message is persisted as ThreadItem for future context ### 3\. RequestContext (User Identity + Session)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#3-requestcontext-user-identity--session "Direct link to 3. RequestContext (User Identity + Session)") **RequestContext** carries user identity and session information. **What it contains**: * User ID (who is sending this message?) * Session metadata (authentication, permissions) * Request-specific context (IP, user agent, etc.) **Why it matters**: Multi-tenant chat systems need to isolate conversations. Your TaskManager agent shouldn't show Alice's tasks to Bob. RequestContext enforces this boundary. **Mental model**: Like a security badge in an office building. It identifies who you are and what you're allowed to access. ### 4\. respond() Method Signature[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#4-respond-method-signature "Direct link to 4. respond() Method Signature") This is the **core method** you implement in ChatKitServer: async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any,) -> AsyncIterator[ThreadStreamEvent]: **Parameters**: * `thread`: ThreadMetadata - Information about the conversation (Thread ID, history) * `input`: UserMessageItem | None - The user's latest message (or None on initialization) * `context`: Any - RequestContext with user identity and session info **Returns**: AsyncIterator\[ThreadStreamEvent\] - A stream of events (messages, tool status, widgets) **Key difference from FastAPI route handler**: # FastAPI (request/response)@app.post("/api/chat")async def chat(message: str) -> dict: response = generate_response(message) return {"reply": response} # Single return# ChatKitServer (streaming)async def respond(self, thread, input, context): async for token in generate_stream(input.content): yield AssistantMessage(content=token) # Stream events FastAPI returns **once**. ChatKitServer **yields continuously**. Architecture Comparison: REST vs ChatKit[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#architecture-comparison-rest-vs-chatkit "Direct link to Architecture Comparison: REST vs ChatKit") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Aspect | FastAPI (REST) | ChatKitServer (Chat) | | --- | --- | --- | | **Pattern** | Request → Process → Response | Message → Stream → Events | | **State** | Stateless (each request independent) | Stateful (Thread maintains history) | | **Output** | Single response object | Async iterator of events | | **Lifecycle** | One-shot (request completes) | Ongoing (conversation persists) | | **Context** | No built-in memory | Thread history loaded automatically | | **Method** | Route handler (`@app.post()`) | `respond()` method override | | **Use Case** | CRUD operations, data APIs | Conversational AI, chat agents | Streaming vs Request/Response: When to Use Each[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#streaming-vs-requestresponse-when-to-use-each "Direct link to Streaming vs Request/Response: When to Use Each") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Request/Response Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#requestresponse-pattern "Direct link to Request/Response Pattern") **Best for**: * Simple data fetch ("Get task by ID") * CRUD operations ("Update task title") * Idempotent actions ("Delete task") **Characteristics**: * Fast, predictable response time * Full response available at once * No progressive UI updates needed **Example**: Fetching a task list @app.get("/api/tasks")async def get_tasks(): tasks = db.query(Task).all() return {"tasks": [t.dict() for t in tasks]} User gets complete task list immediately. No need for streaming. ### Streaming Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#streaming-pattern "Direct link to Streaming Pattern") **Best for**: * Long-running AI responses (token-by-token generation) * Progressive UI updates (chat bubbles filling in real-time) * Partial results useful before completion **Characteristics**: * Response builds over time * User sees progress immediately * Better perceived performance (no "waiting" spinner) **Example**: Streaming agent response async def respond(self, thread, input, context): # User sees tokens appear progressively: # "I've" ... "created" ... "a" ... "task" ... async for token in agent.stream(input.content): yield AssistantMessage(content=token) User watches response construct in real-time, like a human typing. Event-Driven Architecture: ThreadStreamEvent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#event-driven-architecture-threadstreamevent "Direct link to Event-Driven Architecture: ThreadStreamEvent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ChatKitServer streams **events**, not just text. **Event types**: * **AssistantMessage**: Text response from agent * **ToolStatus**: Tool execution progress ("Querying database...") * **Widget**: Interactive UI components (buttons, forms) * **Task**: Background work notifications **Why events matter**: Chat interfaces aren't just text exchanges. They're rich interactions. **Example**: Task creation with tool visibility async def respond(self, thread, input, context): # Event 1: Tool execution status yield ToolStatus(tool="create_task", status="running") # Execute tool task = await create_task(title="Groceries") # Event 2: Tool completion status yield ToolStatus(tool="create_task", status="completed") # Event 3: Assistant response yield AssistantMessage( content=f"I've created task '{task.title}' with ID {task.id}" ) User sees: 1. "Creating task..." (ToolStatus running) 2. "Task created ✓" (ToolStatus completed) 3. "I've created task 'Groceries' with ID 42" (AssistantMessage) This is event-driven communication—multiple signals, progressive updates. Conversation Persistence: The Memory Layer[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#conversation-persistence-the-memory-layer "Direct link to Conversation Persistence: The Memory Layer") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ChatKitServer maintains **Thread history** automatically. **What this means**: When user sends message #5, the agent can reference messages #1-4. The Thread acts as **working memory**. **Example conversation**: Message 1 (User): "Create a task for groceries"Message 2 (Agent): "Task created with ID 42"Message 3 (User): "What's its priority?"Message 4 (Agent): "The task you created has default priority" When processing Message 3, the agent has access to Messages 1-2. It knows "its" refers to the task from Message 1. **Contrast with REST**: In a stateless REST API, Message 3 would need to explicitly pass context: POST /api/tasks/42/priority User must remember task ID and include it. With ChatKitServer, the Thread remembers. The respond() Lifecycle[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#the-respond-lifecycle "Direct link to The respond() Lifecycle") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Understanding what happens when a user sends a message: 1. User sends message: "Create task for groceries" ↓2. ChatKitServer receives message ↓3. Loads Thread metadata (ID, history) ↓4. Creates UserMessageItem ↓5. Appends to Thread history ↓6. Calls your respond() method with: - thread (conversation context) - input (latest user message) - context (user identity) ↓7. Your respond() yields events: - ToolStatus (optional) - AssistantMessage (response) - Widgets (optional) ↓8. ChatKitServer streams events to user ↓9. Appends AssistantMessage to Thread history ↓10. Thread persisted for next interaction **Key insight**: Thread history accumulates. Each message builds context for the next. Multi-Tenant Isolation: RequestContext[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#multi-tenant-isolation-requestcontext "Direct link to Multi-Tenant Isolation: RequestContext") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ChatKitServer supports **multi-user chat systems** through RequestContext. **The problem**: If your TaskManager agent serves 100 users, each needs isolated Threads. **The solution**: RequestContext identifies the user making the request. **How it works**: async def respond(self, thread, input, context): # context contains user_id user_id = context.user_id # Load user-specific data tasks = db.query(Task).filter(Task.user_id == user_id).all() # Generate response scoped to this user yield AssistantMessage(content=f"You have {len(tasks)} tasks") **Safety implication**: Without RequestContext validation, users could access each other's Threads. ChatKitServer enforces boundaries by design. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#safety-note "Direct link to Safety Note") When building multi-user chat systems, always validate RequestContext before accessing data. Never trust Thread ID alone—verify the requesting user has permission to access that Thread. ChatKitServer provides isolation primitives, but you must enforce authorization in your respond() implementation. Why This Architecture for Agent Factory[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#why-this-architecture-for-agent-factory "Direct link to Why This Architecture for Agent Factory") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You're not just learning ChatKitServer for chat UIs. You're learning **the infrastructure layer for sellable Digital FTEs**. **The connection**: Your TaskManager Digital FTE needs: * **Conversation memory** (Thread history) * **Streaming responses** (professional UX) * **Multi-tenant isolation** (serve multiple customers) * **Event-driven updates** (tool execution visibility) ChatKitServer provides this infrastructure out-of-the-box. You focus on **agent logic** (what your Digital FTE does), not **plumbing** (how chat works). **The next lessons** will show you how to: 1. Implement your first respond() method 2. Stream agent responses token-by-token 3. Load Thread history for context-aware agents 4. Handle session management and authentication 5. Build the TaskManager Digital FTE with full chat UX This lesson established the **architectural foundation**. You now understand: * Thread and ThreadItem as conversation primitives * respond() as the central integration point * Streaming vs request/response decision framework * Event-driven architecture for rich interactions * Conversation persistence and multi-tenant isolation In the next lesson, you'll **implement** your first ChatKitServer agent using the OpenAI Agents SDK integration. * [The Paradigm Shift: Conversations vs Requests](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#the-paradigm-shift-conversations-vs-requests) * [Request/Response World (FastAPI)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#requestresponse-world-fastapi) * [Conversation World (ChatKitServer)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#conversation-world-chatkitserver) * [Core Architecture Primitives](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#core-architecture-primitives) * [1\. Thread (Conversation Container)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#1-thread-conversation-container) * [2\. ThreadItem (Individual Message)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#2-threaditem-individual-message) * [3\. RequestContext (User Identity + Session)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#3-requestcontext-user-identity--session) * [4\. respond() Method Signature](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#4-respond-method-signature) * [Architecture Comparison: REST vs ChatKit](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#architecture-comparison-rest-vs-chatkit) * [Streaming vs Request/Response: When to Use Each](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#streaming-vs-requestresponse-when-to-use-each) * [Request/Response Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#requestresponse-pattern) * [Streaming Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#streaming-pattern) * [Event-Driven Architecture: ThreadStreamEvent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#event-driven-architecture-threadstreamevent) * [Conversation Persistence: The Memory Layer](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#conversation-persistence-the-memory-layer) * [The respond() Lifecycle](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#the-respond-lifecycle) * [Multi-Tenant Isolation: RequestContext](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#multi-tenant-isolation-requestcontext) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#safety-note) * [Why This Architecture for Agent Factory](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/chatkit-architecture#why-this-architecture-for-agent-factory) --- # Debugging and Troubleshooting MCP | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've configured MCP servers. Tools are loaded. Your agent starts making requests. And then... nothing works the way you expected. Maybe the server won't start. Maybe it starts but tools don't appear. Maybe tools appear but fail when called. Maybe they succeed but return the wrong data. The failures can be subtle—a typo in a configuration path, an environment variable not expanding, an authentication header missing, a timeout exceeded. **In production environments, silent failures are worse than loud ones.** An MCP server that crashes on startup gives you an error message. An MCP server that returns wrong data while appearing to work successfully is harder to detect and more dangerous. This lesson teaches you to debug systematically. You'll learn the MCP Inspector—a tool designed specifically for testing MCP servers in isolation. You'll recognize common error patterns and their causes. You'll understand why debugging differs between stdio and HTTP servers. By the end, you'll have confidence that your MCP servers are working correctly before deploying them to production. Understanding the Debugging Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#understanding-the-debugging-problem "Direct link to Understanding the Debugging Problem") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before diving into tools, let's understand what makes MCP debugging different from traditional software debugging. ### Three Layers of Potential Failure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#three-layers-of-potential-failure "Direct link to Three Layers of Potential Failure") | Layer | What Fails | How You Know | What You Debug | | --- | --- | --- | --- | | **Server Startup** | Server process won't launch | Process exits immediately; exit code nonzero | Server installation, dependencies, configuration parsing | | **Tool Discovery** | Server starts but tools don't appear | Client lists tools; list is empty or incomplete | Environment variables, authentication, permissions | | **Tool Execution** | Tools exist but calls fail | Error on tool invocation or timeout | Tool arguments, external service connectivity, API credentials | Most debugging tools address one layer. **The MCP Inspector addresses all three**—it's designed to test the entire MCP pipeline systematically. ### Why stdio and HTTP Debugging Differ[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#why-stdio-and-http-debugging-differ "Direct link to Why stdio and HTTP Debugging Differ") **stdio servers** (local Python/Node processes): * Communication through stdin/stdout * Logs must go to stderr (stdout is reserved for protocol) * Failures are process-level (crashes, exit codes) * Testing requires launching subprocesses **HTTP servers** (remote or containerized): * Communication over HTTP/SSE * Logs go to server's logging infrastructure * Failures are request-level (timeouts, 500s, CORS) * Testing requires network inspection This fundamental difference means your debugging approach must adapt to the transport type. The MCP Inspector: Interactive Debugging[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#the-mcp-inspector-interactive-debugging "Direct link to The MCP Inspector: Interactive Debugging") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The MCP Inspector is a browser-based debugging tool built into the MCP ecosystem. It's the primary way to test MCP servers in isolation, before integrating them into Claude Code or other clients. ### Starting the Inspector[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#starting-the-inspector "Direct link to Starting the Inspector") The Inspector command differs depending on your server's language: **For any Node.js server**: npx @modelcontextprotocol/inspector /path/to/server.js **Output:** Model Context Protocol InspectorStarting inspector on http://localhost:5173Open http://localhost:5173 in your browser **For Python servers**: mcp dev your_server.py **Output:** MCP Inspector startingListening on http://localhost:5000Open browser to http://localhost:5000 The Inspector opens a web interface where you can: 1. Connect to a running MCP server 2. List all available tools 3. Call tools with test inputs 4. View request/response pairs 5. Inspect error messages **Output:** Output: {"result": "success", "data": [...]}Raw Response: {"jsonrpc": "2.0", "id": 1, "result": {...}} ### Inspector Workflow: The Five-Step Validation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#inspector-workflow-the-five-step-validation "Direct link to Inspector Workflow: The Five-Step Validation") Use this workflow to systematically validate any MCP server: **Step 1: Start Inspector and Connect** mcp dev your_server.py# Open http://localhost:5000# Click "Connect" **Check**: Does the Inspector connect and display "Connected to MCP Server"? If not, the server isn't running or the port is wrong. **Step 2: List Tools** Click "Tools" tab **Check**: Do your expected tools appear in the list? Count matches your server's tool definitions? | Expected Tool | Result | Status | | --- | --- | --- | | `list_files` | appears, schema correct | ✓ PASS | | `read_file` | appears, schema correct | ✓ PASS | | `write_file` | missing from list | ✗ FAIL | **Step 3: Test with Simple Input** Select a tool: list_filesInput: (leave empty or minimal)Click "Call Tool" **Check**: Does the tool execute? What's the response? * Success: Tool returns valid result, no errors * Failure: Tool returns error message explaining what went wrong **Step 4: Test with Complex Input** Select tool: read_fileInput: {"path": "/etc/passwd"}Click "Call Tool" **Check**: Does tool behave correctly with realistic inputs? Does it: * Accept valid arguments? * Reject invalid arguments with clear error? * Handle edge cases (empty paths, nonexistent files)? **Step 5: Verify Error Messages** Intentionally provide bad inputExample: read_file with {"path": ""}Click "Call Tool" **Check**: Is the error message helpful? Does it guide toward the fix? * Good: "Path cannot be empty. Expected: /absolute/path/to/file" * Bad: "Error" or "Invalid" ### Inspector Output: Understanding Response Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#inspector-output-understanding-response-structure "Direct link to Inspector Output: Understanding Response Structure") When you call a tool in the Inspector, you see both the formatted result and the raw JSON-RPC message: **Formatted Result:**{ "content": "File contents here"}**Raw JSON-RPC Response:**{ "jsonrpc": "2.0", "id": 1, "result": { "content": "File contents here" }} The raw response reveals the protocol layer: * `"jsonrpc": "2.0"` — Protocol version * `"id": 1` — Matches your request ID * `"result"` — Tool succeeded (vs. `"error"` if it failed) If you see an error instead: { "jsonrpc": "2.0", "id": 2, "error": { "code": -32602, "message": "Invalid params: location must be a string" }} The error tells you: * `code: -32602` — Parameter validation error (standard JSON-RPC error code) * `message` — Specific problem (location expected string, got something else) Common MCP Error Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#common-mcp-error-patterns "Direct link to Common MCP Error Patterns") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Most MCP debugging falls into one of these categories. Learning to recognize them saves hours of troubleshooting. ### Error Pattern Reference Table[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#error-pattern-reference-table "Direct link to Error Pattern Reference Table") | Error | What Happened | Most Likely Cause | How to Fix | | --- | --- | --- | --- | | **Connection refused** | Client can't reach server | Server not running, wrong port, firewall blocking | `mcp dev server.py` in correct directory; verify port not in use | | **Tool not found** | Tool appears in list but calls fail | Tool name mismatch between definition and call | Check exact tool name in Inspector vs. what client sends | | **Invalid arguments** | Tool returns parameter validation error | Arguments don't match `inputSchema` | Validate argument types match schema; use Inspector to test with correct format | | **Authentication failed** | Tool fails with auth error (e.g., GitHub token invalid) | Environment variable not set or wrong token | Check `echo $GITHUB_TOKEN`; verify token has required permissions | | **Silent failure (no output)** | Tool runs but produces no output or timeout | stdout logging instead of stderr; network timeout | Redirect logs to stderr; increase timeout; check network connectivity | | **CORS error (HTTP servers)** | Browser can't reach HTTP server | Server not configured for CORS; browser requesting from wrong origin | Add CORS headers; verify Origin header matches server allowlist | | **File path error** | Tool can't find file or directory | Relative path vs. absolute path mismatch | Use absolute paths or verify working directory; print actual resolved path in error | ### Detailed Examples[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#detailed-examples "Direct link to Detailed Examples") **Example 1: Tool Not Found** # Inspector shows no tools# But server code defines them# Wrong: Forgetting to register toolclass MyServer: def __init__(self): # Missing: self.register_tool(...) pass# Fix: Register tools in initializationclass MyServer: def __init__(self): self.register_tool("list_files", self.list_files) self.register_tool("read_file", self.read_file) **Example 2: Invalid Arguments** # Inspector: read_file tool fails with validation error# Expected schema: {"path": string}# Sent: {"path": 123} # number instead of string# Fix: Ensure arguments match schema# Schema says "path": {"type": "string"}# Must call with: {"path": "/absolute/path"} **Example 3: Environment Variable Not Set** # Inspector: GitHub tool fails with "Authentication failed"# Code tries to use: os.environ["GITHUB_TOKEN"]# Problem: Variable not in environment$ echo $GITHUB_TOKEN# (empty)# Fix: Set before running Inspector$ export GITHUB_TOKEN="ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"$ mcp dev github_server.py **Example 4: Logging in Wrong Stream** # Wrong: Logs go to stdout (breaks protocol)print("DEBUG: Attempting to authenticate...") # Goes to stdout!# Right: Logs go to stderr (doesn't interfere)import sysprint("DEBUG: Attempting to authenticate...", file=sys.stderr)# Or use logging module (defaults to stderr)import logginglogging.info("Attempting to authenticate...") Debugging stdio vs. HTTP Servers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#debugging-stdio-vs-http-servers "Direct link to Debugging stdio vs. HTTP Servers") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The debugging approach differs significantly based on transport type. ### stdio Servers (Python/Node local processes)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#stdio-servers-pythonnode-local-processes "Direct link to stdio Servers (Python/Node local processes)") **How they communicate**: stdin/stdout pipes **Why debugging differs**: The protocol uses stdout, so any logging to stdout breaks the protocol. Tools must send logs to stderr. **Debugging checklist**: 1. **Server startup issues** mcp dev your_server.py 2>&1 | grep -i error# Capture both stdout (protocol) and stderr (logs) 2. **Tool not appearing** # Check server initialization# Add debug logging to stderr:logging.debug("Registering tool: list_files", file=sys.stderr) 3. **Tool execution failures** # Add try/catch with detailed stderr loggingtry: result = handle_request()except Exception as e: logging.error(f"Tool failed: {e}", file=sys.stderr) raise **Example logging configuration for Python**: import loggingimport sys# Configure logging to stderrlogging.basicConfig( stream=sys.stderr, # CRITICAL: Not stdout! level=logging.DEBUG, format='%(asctime)s [%(levelname)s] %(message)s')# Now safe to use logginglogging.debug("Server starting...")logging.info("Tool registered: list_files")logging.error("Authentication failed") ### HTTP Servers (Remote or containerized)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#http-servers-remote-or-containerized "Direct link to HTTP Servers (Remote or containerized)") **How they communicate**: HTTP requests/responses **Why debugging differs**: Network involved, no direct access to stdout/stderr, CORS restrictions possible. **Debugging checklist**: 1. **Connection issues** # Test network connectivitycurl -X POST http://localhost:8000/mcp \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"initialize"}' 2. **CORS errors** # Error: "No 'Access-Control-Allow-Origin' header"# Fix: Ensure server includes CORS headers 3. **Request/response inspection** # Use browser DevTools → Network tab# Watch HTTP requests to server# Inspect request headers, response status, body 4. **Server-side logging** # Access server logs (depends on deployment)# Docker: docker logs # Cloud: provider's log viewer# Local: check log file configuration Systematic Troubleshooting Decision Tree[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#systematic-troubleshooting-decision-tree "Direct link to Systematic Troubleshooting Decision Tree") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When something breaks, follow this decision tree to isolate the problem: MCP Server Problem?│├─ Server starts?│ ├─ NO → Check configuration syntax (JSON validation)│ │ Check dependencies installed (pip install / npm install)│ │ Check file paths (absolute vs. relative)│ ││ └─ YES → Tools appear in Inspector?│ ├─ NO → Check environment variables (echo $VAR_NAME)│ │ Check server initialization (tools registered?)│ │ Check authentication (API keys set?)│ ││ └─ YES → Tools execute successfully?│ ├─ NO → Check inputSchema matches arguments│ │ Check external service connectivity│ │ Check tool implementation for bugs│ ││ └─ YES → Results correct?│ ├─ NO → Tool logic error (debug tool code)│ ││ └─ YES → Server ready for production Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Setup:** You have three MCP servers with different failure modes. Use Claude or another LLM to help diagnose and fix each one. **Prompt 1: Connection Diagnosis** I'm trying to debug an MCP server. When I run:mcp dev my_weather_server.pyI get:ModuleNotFoundError: No module named 'requests'What's wrong and how do I fix it?Then, ask me: What does "requests" module do? Why might it be used in an MCP weather server? **What you're learning**: Recognizing server startup failures vs. runtime failures; understanding dependency issues as a category of failure. **Prompt 2: Tool Discovery Investigation** I have an MCP server that starts successfully. When I open the Inspector and connect, the server appears connected but the Tools tab is empty. No tools are listed.Here's the server code structure:- Server class has __init__ method- __init__ calls super().__init__()- Server class has several methods like list_weather, get_temperatureWhy would tools not appear even though the server runs?Also tell me: What's the relationship between the methods in my server class and the tools that appear in the Inspector? **What you're learning**: Tool registration as distinct from tool definition; understanding why code can run without errors but still produce no tools. **Prompt 3: Error Message Interpretation** I'm using the Inspector to test my file_reader tool. When I call it with:{"path": "/home/user/test.txt"}The Inspector returns:{ "jsonrpc": "2.0", "id": 3, "error": { "code": -32602, "message": "Invalid params: 'path' is required but got string" }}This error doesn't make sense—I did provide path as a string. What's really happening?Then ask me: If you were designing an error message, how would you make this clearer? **What you're learning**: Reading JSON-RPC error codes; understanding schema validation failures; recognizing when error messages mislead rather than help. * * * ✓ **You're now equipped to debug MCP systematically.** You understand the Inspector workflow, recognize error patterns, and know how to approach different failure modes. In production, these skills prevent silent failures that would otherwise go undetected until a customer reports data corruption or missed workflow triggers. **In the capstone project, you'll orchestrate multiple MCP servers in a production configuration. Everything you've learned—architecture, configuration, and debugging—converges into a reliable, production-ready system.** Checking access... * [Understanding the Debugging Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#understanding-the-debugging-problem) * [Three Layers of Potential Failure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#three-layers-of-potential-failure) * [Why stdio and HTTP Debugging Differ](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#why-stdio-and-http-debugging-differ) * [The MCP Inspector: Interactive Debugging](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#the-mcp-inspector-interactive-debugging) * [Starting the Inspector](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#starting-the-inspector) * [Inspector Workflow: The Five-Step Validation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#inspector-workflow-the-five-step-validation) * [Inspector Output: Understanding Response Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#inspector-output-understanding-response-structure) * [Common MCP Error Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#common-mcp-error-patterns) * [Error Pattern Reference Table](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#error-pattern-reference-table) * [Detailed Examples](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#detailed-examples) * [Debugging stdio vs. HTTP Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#debugging-stdio-vs-http-servers) * [stdio Servers (Python/Node local processes)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#stdio-servers-pythonnode-local-processes) * [HTTP Servers (Remote or containerized)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#http-servers-remote-or-containerized) * [Systematic Troubleshooting Decision Tree](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#systematic-troubleshooting-decision-tree) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/debugging-and-troubleshooting#try-with-ai) --- # Chapter 74: Relational Databases for Agents with SQLModel | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C Build production-ready async database layers for AI agent backends using SQLModel with PostgreSQL. Overview[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#overview "Direct link to Overview") ------------------------------------------------------------------------------------------------------------------------------------------------- Your AI agents need persistent state. When users create tasks, assign projects, or track workers, that data must survive server restarts, scale across instances, and maintain integrity under concurrent access. This chapter teaches you to build the database layer that makes your agent backends reliable. You'll learn to **think like a data architect**—identifying entities, applying normal forms, and making informed design decisions—before implementing with SQLModel and async patterns. Learning Outcomes[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#learning-outcomes "Direct link to Learning Outcomes") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By completing this chapter, you will be able to: 1. **Design database schemas** by identifying entities, relationships, and applying normalization principles 2. **Evaluate normalization trade-offs** for agent-specific workloads (when to denormalize) 3. **Configure async database engines** for PostgreSQL with proper pooling and connection management 4. **Design SQLModel tables** with JSONB columns, foreign keys, and self-referential relationships 5. **Implement async CRUD operations** using AsyncSession with proper commit/flush patterns 6. **Prevent N+1 queries** using selectinload and other eager loading strategies 7. **Manage database transactions** with proper rollback and error handling 8. **Create and run migrations** using Alembic with async support 9. **Build a complete database layer** for the Task API running example Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#prerequisites "Direct link to Prerequisites") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapter 70: FastAPI for Agents** - You've built basic APIs and seen SQLModel briefly * **Python async/await** - You understand `async def`, `await`, and async context managers * **Docker** - You can run a PostgreSQL container locally Technology Stack[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#technology-stack "Direct link to Technology Stack") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Technology | Purpose | | --- | --- | | SQLModel | ORM combining Pydantic + SQLAlchemy | | SQLAlchemy 2.0+ | Async database engine | | asyncpg | PostgreSQL async driver | | PostgreSQL | Production database | | Alembic | Database migrations | Lessons[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#lessons "Direct link to Lessons") ---------------------------------------------------------------------------------------------------------------------------------------------- | # | Title | Duration | Focus | | --- | --- | --- | --- | | 0 | Build Your Database Skill | 25 min | Create your `relational-db-agent` skill | | 1 | Why Agents Need Structured Data | 15 min | Persistence, ACID, async motivation | | 2 | **Database Design & Normalization** | 40 min | Entity modeling, ERD diagrams, 1NF-3NF, when to denormalize | | 3 | SQLModel + Async Engine Setup | 35 min | Engine creation, pooling deep dive, security, URL conversion | | 4 | Implementing Data Models | 35 min | SQLModel syntax, JSONB, indexing deep dive (GIN, composite, partial) | | 5 | Async Session Management | 25 min | AsyncSession, dependencies, lifecycle | | 6 | CRUD Operations Pattern | 35 min | CRUD with async, soft delete pattern, query optimization | | 7 | **Testing Database Code** | 30 min | pytest-asyncio, fixtures, test isolation | | 8 | Relationships and Eager Loading | 40 min | N+1 problem, selectinload, common mistakes | | 9 | Transactions and Error Handling | 25 min | Transactions, rollback, error patterns | | 10 | Migrations with Alembic | 35 min | Async migrations, backup/recovery, autogenerate | | 11 | Capstone: Complete Database Layer | 50 min | Full Task API implementation, multi-tenancy patterns | | 12 | Chapter Quiz | 20 min | 18 questions including normalization | **Total time**: ~410 minutes (6.8 hours) The Design-First Approach[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#the-design-first-approach "Direct link to The Design-First Approach") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter follows a **design-first** methodology: 1. **L02: Think Like a Data Architect** - Before writing any SQLModel code, you'll learn to: * Identify entities from requirements * Create ERD diagrams with Mermaid * Map relationships and cardinalities * Apply normal forms (1NF, 2NF, 3NF) * Decide when to denormalize for performance 2. **L03-L10: Implementation** - With a solid design, implementation becomes straightforward: * Engine setup with production pooling and security * Models with advanced indexing (GIN, composite, partial) * Testing patterns for database code * Common mistakes and how to avoid them * Backup/recovery strategies 3. **L11: Integration** - Combine design and implementation into a production-ready layer with multi-tenancy patterns Running Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#running-example "Direct link to Running Example") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter deepens the Task API you started in Chapter 70. You'll build a complete database layer with: * **Task** model with JSONB columns and relationships * **Project** model for organizing tasks * **Worker** model (human and AI agents) * Self-referential parent/child task relationships * Eager loading for efficient queries * Transaction handling for complex operations Skill-First Approach[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#skill-first-approach "Direct link to Skill-First Approach") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You'll start by creating your own `relational-db-agent` skill in Lesson 0. This skill grows with you—each lesson ends with a "Reflect on Your Skill" section where you test and improve your skill based on what you learned. By chapter end, you'll have a production-ready skill that thinks like a data architect AND generates correct async SQLModel code. * [Overview](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#overview) * [Learning Outcomes](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#learning-outcomes) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#prerequisites) * [Technology Stack](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#technology-stack) * [Lessons](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#lessons) * [The Design-First Approach](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#the-design-first-approach) * [Running Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#running-example) * [Skill-First Approach](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel#skill-first-approach) --- # Chapter Quiz: Advanced MCP Server Development | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C This quiz assesses your mastery of advanced MCP patterns from Chapter 67. You should be able to answer these questions confidently after completing Lessons 1–8. **Time**: ~45 minutes **Passing Score**: 15/20 (75%) **Advanced Score**: 18/20 (90%) * * * Context Object Category[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#context-object-category "Direct link to Context Object Category") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Question 1: Dependency Injection Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-1-dependency-injection-pattern "Direct link to Question 1: Dependency Injection Pattern") **What is the primary advantage of Context being injected into tool functions rather than passed as a parameter?** A) It reduces the number of function parameters, making function signatures cleaner B) FastMCP can automatically wire dependencies without requiring manual setup boilerplate C) It allows multiple Context objects to exist simultaneously in different threads D) It ensures Context is never shared between concurrent tool invocations **Correct Answer**: B **Explanation**: Dependency injection enables the framework to automatically provide Context without explicit parameter passing. This is why FastMCP can wire complex tools without requiring setup boilerplate. While A is a nice side effect, it's not the primary advantage. C is false (Context is per-request, not per-thread). D describes a property of Context, but not why it's injected rather than passed. * * * ### Question 2: Context Logging Levels[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-2-context-logging-levels "Direct link to Question 2: Context Logging Levels") **You're building a tool that processes customer data. You want to log when processing starts (informational), when a non-critical issue occurs (e.g., missing optional field), and when data validation fails (problematic but recoverable). Which logging methods should you use?** A) `info()` for all three cases B) `info()` for start, `warning()` for missing field, `error()` for validation failure C) `warning()` for all three cases D) `error()` for all three cases **Correct Answer**: B **Explanation**: Logging levels match severity: `info()` for expected operational events, `warning()` for non-critical issues that clients should be aware of, `error()` for failures that prevent normal operation. Using all one level loses diagnostic value—clients won't know which issues are serious. * * * ### Question 3: Session-Based Context Usage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-3-session-based-context-usage "Direct link to Question 3: Session-Based Context Usage") **When would you use `context.session.session_id` to cache results across multiple tool invocations within the same client connection?** A) Every time you need to store any data persistently B) When a user has executed multiple tools in sequence and you want to avoid recomputing shared data C) When building stateless servers that must not retain any client-specific state D) When implementing database connection pooling **Correct Answer**: B **Explanation**: Session-based caching is appropriate when a single client connection includes multiple tool calls that could benefit from shared context (same user, same conversation). Stateless servers explicitly avoid this pattern. It's not for persistent storage or database pooling—those are different architectural concerns. * * * Sampling Category[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#sampling-category "Direct link to Sampling Category") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Question 4: Problem Sampling Solves[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-4-problem-sampling-solves "Direct link to Question 4: Problem Sampling Solves") **What fundamental problem does sampling (servers calling LLMs through clients) solve that servers couldn't do before?** A) It allows servers to perform computation without using local CPU B) It enables servers to access frontier LLM reasoning while staying focused on their domain expertise C) It guarantees cheaper token costs because servers don't pay for LLM calls D) It removes the need for error handling in servers **Correct Answer**: B **Explanation**: Sampling shifts reasoning to frontier models (which clients have access to) while servers focus on domain operations. This is the conceptual insight: servers are good at tools/resources, frontier models are good at reasoning. Sampling bridges this gap. A is wrong (servers still compute). C is wrong (cost depends on client pricing). D is obviously false. * * * ### Question 5: Implementing Sampling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-5-implementing-sampling "Direct link to Question 5: Implementing Sampling") **You want your server to ask an LLM for advice on which data processing strategy to use. Which method would you use to call the LLM through the connected client?** A) `context.llm.generate(prompt)` — Call local LLM B) `context.session.create_message([...])` — Call client's LLM C) `await context.sample(model="claude-3-5-sonnet")` — Direct sampling D) `context.request_reasoning(query)` — Special reasoning mode **Correct Answer**: B **Explanation**: `context.session.create_message()` is the correct method for sampling. The client connection has access to its LLM; the server calls through that connection. A and C are made up. D doesn't exist. Sampling always goes through the session to the client's connected model. * * * ### Question 6: Sampling Cost Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-6-sampling-cost-structure "Direct link to Question 6: Sampling Cost Structure") **Your server uses sampling to ask an LLM for advice during tool execution. Who bears the token cost of this LLM call?** A) The server owner (you) B) The client who called the server C) Shared equally between server and client D) Nobody—sampling is always free **Correct Answer**: B **Explanation**: Cost shifting is the key insight: the client's LLM account pays for sampling. This is a feature, not a bug—it incentivizes servers to use sampling responsibly (reasoning doesn't happen on server's dime). This makes reasoning-intensive servers economically sustainable. * * * ### Question 7: Client-Side Sampling Callback[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-7-client-side-sampling-callback "Direct link to Question 7: Client-Side Sampling Callback") **When a server invokes `context.session.create_message()` to perform sampling, what must happen on the client side?** A) The client automatically routes the request to its configured LLM B) The client must have a special `on_sampling_request()` handler registered C) The client opens a new connection to request LLM access D) Sampling cannot work—it's server-only **Correct Answer**: A **Explanation**: MCP clients (like Claude Code) are designed to handle sampling requests transparently. When a server calls `create_message()`, the client automatically routes the request to its configured model. No special registration is needed. The architecture assumes clients will handle sampling; servers just call it. * * * Notifications Category[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#notifications-category "Direct link to Notifications Category") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Question 8: Progress Notifications[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-8-progress-notifications "Direct link to Question 8: Progress Notifications") **You're processing a file with 10,000 records. You want to report progress every time you process 100 records. Which method would you call?** A) `await context.report_progress(100, 10000)` after each batch B) `await context.progress_percentage(1)` to increment by 1% C) `await context.log_progress("100 records processed")` D) `context.session.notify_progress(100, 10000)` **Correct Answer**: A **Explanation**: `report_progress(completed, total)` is the correct signature. You call it repeatedly with updated values as progress changes. B and C are made up. D is wrong (it's a context method, not session). Progress notifications keep clients informed during long operations. * * * ### Question 9: Logging Levels in Notifications[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-9-logging-levels-in-notifications "Direct link to Question 9: Logging Levels in Notifications") **Your tool reports informational messages during processing, warnings when data quality is questionable, and errors when processing fails. How should clients distinguish between these severity levels?** A) All notifications appear as plain text—clients can't distinguish severity B) Use `info()`, `warning()`, and `error()` methods; clients interpret logging level in UI C) Send all messages with the same method; severity is determined by message text D) Use separate notification methods but don't expect client-side behavior to differ **Correct Answer**: B **Explanation**: Logging levels communicate severity. Clients (like Claude Code) display these differently—errors might be highlighted, warnings might use different colors, info might be less prominent. This is intentional design. A and D are wrong because they miss the point of structured logging. * * * ### Question 10: Client-Side Notification Handling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-10-client-side-notification-handling "Direct link to Question 10: Client-Side Notification Handling") **When a server sends logging and progress notifications, what happens on the client if the client doesn't implement handlers for these notifications?** A) The client crashes because it received unexpected messages B) Notifications are silently ignored; processing continues C) The client returns an error to the server D) Notifications are buffered and displayed when processing completes **Correct Answer**: B **Explanation**: MCP design assumes optional notification support. If a client doesn't implement handlers, notifications are safely ignored—the server continues normally. This is graceful degradation. A, C, and D would break compatibility with older clients. * * * Roots Category[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#roots-category "Direct link to Roots Category") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Question 11: What Roots Solve[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-11-what-roots-solve "Direct link to Question 11: What Roots Solve") **Why is the roots feature necessary for MCP servers that access files?** A) To improve file read/write performance B) To define secure boundaries on which directories servers can access C) To replace traditional file permissions in operating systems D) To enable file caching across multiple server instances **Correct Answer**: B **Explanation**: Roots establish a security boundary. Without roots, a tool could access `/etc/passwd` or other sensitive files outside its scope. Roots tell the client "this server can only see files under `/data/projects`" (for example). This is a security mechanism. A is wrong (roots don't improve performance). C is wrong (roots complement, not replace, OS permissions). D is wrong (roots don't handle caching). * * * ### Question 12: Implementing Path Validation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-12-implementing-path-validation "Direct link to Question 12: Implementing Path Validation") **You're writing a tool that downloads files to a directory controlled by the user. Before allowing the download, how should you validate the target path?** A) Trust the client-provided path—the client verified it already B) Check if the path exists using `os.path.exists()` C) Use `is_path_allowed()` to verify the path is within allowed roots D) Validate against a hardcoded list of allowed directories **Correct Answer**: C **Explanation**: `is_path_allowed()` checks against declared roots, preventing path traversal attacks. A is wrong (don't trust client input). B doesn't validate against roots. D is inflexible and doesn't use the roots mechanism. Proper validation combines roots declaration with `is_path_allowed()` enforcement. * * * ### Question 13: Roots Security Implications[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-13-roots-security-implications "Direct link to Question 13: Roots Security Implications") **A user declares a root as `/home/user/projects`. Your tool constructs a path using user input: `/home/user/projects/../../../etc/passwd`. What happens if you call `is_path_allowed()` on this path?** A) It returns True because the path contains `/home/user/projects` as a substring B) It returns False because the path resolves outside the declared root C) It returns None (no answer provided) D) It crashes the server because path traversal is detected **Correct Answer**: B **Explanation**: `is_path_allowed()` canonicalizes paths (resolves `..` sequences) before checking against roots. The path traversal is detected and validation fails. A is wrong (substring matching is not how it works). C and D are wrong (proper validation returns False, doesn't crash). * * * Transport & Scaling Category[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#transport--scaling-category "Direct link to Transport & Scaling Category") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Question 14: StreamableHTTP vs Stdio[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-14-streamablehttp-vs-stdio "Direct link to Question 14: StreamableHTTP vs Stdio") **When should you choose StreamableHTTP transport instead of stdio?** A) When your server runs on the same machine as the client B) When your server needs to handle multiple concurrent connections from different clients C) When you're developing and testing locally D) When your server is simple and processes requests sequentially **Correct Answer**: B **Explanation**: StreamableHTTP handles concurrent clients (one connection per client). Stdio is designed for single-client scenarios (like Claude Code). If your server needs to serve multiple clients simultaneously, StreamableHTTP is required. A is wrong (same machine can use either). C and D suggest stdio is better, which is backwards. * * * ### Question 15: Stateless vs Stateful Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-15-stateless-vs-stateful-configuration "Direct link to Question 15: Stateless vs Stateful Configuration") **Your MCP server processes documents and stores intermediate results in memory. You want to scale it horizontally behind a load balancer. What configuration choice must you make?** A) Use `stateless_http=True` to ensure each request is independent B) Use `stateless_http=False` to preserve state across requests C) Use `json_response=True` to serialize state between instances D) Add session affinity to the load balancer so requests route to the same server instance **Correct Answer**: A and D (but if forced to choose one, A is the correct configuration choice; D is a load balancer setting that would be needed if you chose stateful) **Explanation**: For horizontal scaling, you should design stateless servers (`stateless_http=True`) where each request is fully independent. This allows load balancers to route requests to any instance. If you must maintain state, use session affinity so requests from the same client stick to one server. But stateless is the preferred pattern. B is wrong (mutable state breaks load balancing). C is wrong (it doesn't solve the problem). * * * ### Question 16: json\_response Implications[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-16-json_response-implications "Direct link to Question 16: json_response Implications") **You set `json_response=True` on your StreamableHTTP server. What does this affect?** A) How the server stores data internally (all data becomes JSON) B) Whether responses are wrapped in JSON envelope or sent as raw bytes C) The programming language the server can be written in (must support JSON) D) Whether clients must be written in JavaScript **Correct Answer**: B **Explanation**: `json_response` determines the response format. When True, responses are JSON-wrapped (structured metadata + payload). When False, raw bytes are sent. This is a transport encoding choice. A is wrong (internal representation is unchanged). C and D are nonsense. * * * ### Question 17: Scaling Configuration Decision[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-17-scaling-configuration-decision "Direct link to Question 17: Scaling Configuration Decision") **You're deploying a server that processes real-time sensor data. Clients expect to subscribe to updates from previous requests. What configuration maximizes both scalability and update delivery?** A) Stateless with no session support B) Stateful with session affinity to one server instance C) Stateless with WebSocket subscriptions D) Stateful with database to share state between instances **Correct Answer**: D (with caveats) or B as practical choice **Explanation**: This is a reasoning question without a perfect answer. Real-time subscriptions require some statefulness. Option B (stateful with affinity) is practical but limits scaling. Option D (shared database) enables true horizontal scaling but adds complexity. A is wrong (can't do subscriptions). C (WebSockets) isn't part of standard MCP transport. * * * Error Handling & Packaging Category[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#error-handling--packaging-category "Direct link to Error Handling & Packaging Category") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Question 18: JSON-RPC Error Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-18-json-rpc-error-structure "Direct link to Question 18: JSON-RPC Error Structure") **Your tool encounters an error: "Database connection failed." How should you return this to the client in MCP?** A) Raise a Python exception and let FastMCP convert it to JSON-RPC error B) Return an error code as a string: `"error: database_connection_failed"` C) Use structured JSON-RPC error with code and message D) Log the error and return an empty response **Correct Answer**: A (Python exception) or C (if using direct JSON-RPC) **Explanation**: FastMCP automatically converts Python exceptions to JSON-RPC error format. Alternatively, you can use the structured format directly. B is wrong (raw strings aren't structured errors). D is wrong (don't silently fail). The framework handles error translation. * * * ### Question 19: Graceful Error Recovery[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-19-graceful-error-recovery "Direct link to Question 19: Graceful Error Recovery") **Your server uses sampling to ask for LLM advice. The LLM call fails with a timeout. How should your server behave?** A) Crash and return a critical error to the client B) Retry the LLM call up to 3 times with exponential backoff C) Proceed without LLM advice if it's non-essential to the tool, or return error if it's critical D) Always retry forever until the LLM responds **Correct Answer**: C **Explanation**: Graceful degradation depends on whether LLM input is essential. If the tool can proceed without it (e.g., advice on optimization), retry or log and continue. If it's essential (e.g., validation), return an error. B might work but don't blindly retry forever. D is wrong (infinite retries cause hangs). * * * ### Question 20: Packaging for Distribution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-20-packaging-for-distribution "Direct link to Question 20: Packaging for Distribution") **You've built a production MCP server. How should you define the entry point so it can be installed and discovered by clients?** A) Add a `__main__.py` file and users manually run your server B) Document the command to run in a README (e.g., `python -m my_server`) C) Define the entry point in `pyproject.toml` under `[project.scripts]` D) Create an executable shell script that users can run **Correct Answer**: C **Explanation**: `pyproject.toml` entry points enable standard Python package discovery. Clients can install your server with `pip install my-server` and automatically have the entry point available. A and B are manual and don't integrate with the Python ecosystem. D might work locally but isn't standard packaging. C is the production-ready approach. * * * Quiz Completion[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#quiz-completion "Direct link to Quiz Completion") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Congratulations on completing the Chapter 67 Quiz!** ### Next Steps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#next-steps "Direct link to Next Steps") * **Passing (15+/20)**: You've demonstrated mastery of advanced MCP patterns. Move forward to building your own production server. * **Advanced (18+/20)**: You're ready for the capstone (Lesson 9) and real-world deployment scenarios. * **Review Needed**: Review the lesson(s) corresponding to lower-scoring categories and retry this quiz. ### Answer Key Summary[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#answer-key-summary "Direct link to Answer Key Summary") | Question | Category | Correct | Difficulty | | --- | --- | --- | --- | | 1 | Context | B | Apply | | 2 | Context | B | Apply | | 3 | Context | B | Analyze | | 4 | Sampling | B | Understand | | 5 | Sampling | B | Apply | | 6 | Sampling | B | Apply | | 7 | Sampling | A | Analyze | | 8 | Notifications | A | Understand | | 9 | Notifications | B | Apply | | 10 | Notifications | B | Analyze | | 11 | Roots | B | Understand | | 12 | Roots | C | Apply | | 13 | Roots | B | Analyze | | 14 | Transport & Scaling | B | Understand | | 15 | Transport & Scaling | A | Apply | | 16 | Transport & Scaling | B | Apply | | 17 | Transport & Scaling | D | Analyze | | 18 | Error Handling & Packaging | A | Understand | | 19 | Error Handling & Packaging | C | Apply | | 20 | Error Handling & Packaging | C | Apply | * * * Try With AI: Quiz Review & Clarification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#try-with-ai-quiz-review--clarification "Direct link to Try With AI: Quiz Review & Clarification") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use this section to deepen your understanding of challenging concepts with AI assistance. **Setup**: Open Claude Code or your preferred MCP client ### Part 1: Identify Weak Areas[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-1-identify-weak-areas "Direct link to Part 1: Identify Weak Areas") Review your quiz answers. Note which categories or specific questions you found challenging: Questions I struggled with: [Q#, Q#, ...]Topics I want to clarify: [topic, topic, ...] ### Part 2: Deep Dive on Sampling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-2-deep-dive-on-sampling "Direct link to Part 2: Deep Dive on Sampling") **Prompt to Claude Code**: I'm struggling with sampling in MCP servers. Explain:1) Why sampling shifts costs to clients (why is this good?)2) What code pattern I'd use to call an LLM from within an MCP tool3) A real scenario where sampling is essential **What you're learning**: * The economic incentive structure of sampling * Implementation pattern for `context.session.create_message()` * When sampling adds value vs adds complexity ### Part 3: Scaling Tradeoffs[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-3-scaling-tradeoffs "Direct link to Part 3: Scaling Tradeoffs") **Prompt to Claude Code**: I want to build a server that handles multiple clients and maintains some state(like caches). Should I use stateless_http=True or False? What are the tradeoffs?Walk me through an example where state must be preserved vs where it can be stateless. **What you're learning**: * Tradeoff between scalability and feature requirements * When session affinity is necessary * How database-backed state enables true scaling ### Part 4: Roots Security[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-4-roots-security "Direct link to Part 4: Roots Security") **Prompt to Claude Code**: Show me a vulnerable example where a tool doesn't properly validate file pathsusing is_path_allowed(). Then show the fixed version. Explain how path traversalattacks work and why roots + canonicalization prevents them. **What you're learning**: * Common path traversal vulnerabilities * How is\_path\_allowed() protects against them * Why you can't trust client-provided paths ### Part 5: Self-Assessment[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-5-self-assessment "Direct link to Part 5: Self-Assessment") Ask yourself: * Can I explain sampling to someone who hasn't taken this chapter? * Could I design a server that uses roots to control file access? * Would I know which transport (stdio vs StreamableHTTP) to choose for a given scenario? * Can I recognize when error handling is insufficient and add graceful recovery? **If you answered "yes" to all four**, you've mastered this chapter's core concepts. * * * Additional Resources[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#additional-resources "Direct link to Additional Resources") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapter 67 Lessons 1–9**: Review specific lessons for deeper practice * **MCP Official Docs**: Context patterns, sampling examples, roots configuration * **Capstone Project (Lesson 9)**: Build a full production server that demonstrates all patterns **Final Note**: Advanced MCP server development isn't just about knowing patterns—it's about making tradeoff decisions. You've learned WHAT each pattern does. Now practice deciding WHEN and WHY to apply each one in production scenarios. Checking access... * [Context Object Category](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#context-object-category) * [Question 1: Dependency Injection Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-1-dependency-injection-pattern) * [Question 2: Context Logging Levels](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-2-context-logging-levels) * [Question 3: Session-Based Context Usage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-3-session-based-context-usage) * [Sampling Category](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#sampling-category) * [Question 4: Problem Sampling Solves](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-4-problem-sampling-solves) * [Question 5: Implementing Sampling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-5-implementing-sampling) * [Question 6: Sampling Cost Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-6-sampling-cost-structure) * [Question 7: Client-Side Sampling Callback](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-7-client-side-sampling-callback) * [Notifications Category](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#notifications-category) * [Question 8: Progress Notifications](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-8-progress-notifications) * [Question 9: Logging Levels in Notifications](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-9-logging-levels-in-notifications) * [Question 10: Client-Side Notification Handling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-10-client-side-notification-handling) * [Roots Category](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#roots-category) * [Question 11: What Roots Solve](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-11-what-roots-solve) * [Question 12: Implementing Path Validation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-12-implementing-path-validation) * [Question 13: Roots Security Implications](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-13-roots-security-implications) * [Transport & Scaling Category](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#transport--scaling-category) * [Question 14: StreamableHTTP vs Stdio](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-14-streamablehttp-vs-stdio) * [Question 15: Stateless vs Stateful Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-15-stateless-vs-stateful-configuration) * [Question 16: json\_response Implications](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-16-json_response-implications) * [Question 17: Scaling Configuration Decision](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-17-scaling-configuration-decision) * [Error Handling & Packaging Category](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#error-handling--packaging-category) * [Question 18: JSON-RPC Error Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-18-json-rpc-error-structure) * [Question 19: Graceful Error Recovery](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-19-graceful-error-recovery) * [Question 20: Packaging for Distribution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#question-20-packaging-for-distribution) * [Quiz Completion](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#quiz-completion) * [Next Steps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#next-steps) * [Answer Key Summary](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#answer-key-summary) * [Try With AI: Quiz Review & Clarification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#try-with-ai-quiz-review--clarification) * [Part 1: Identify Weak Areas](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-1-identify-weak-areas) * [Part 2: Deep Dive on Sampling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-2-deep-dive-on-sampling) * [Part 3: Scaling Tradeoffs](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-3-scaling-tradeoffs) * [Part 4: Roots Security](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-4-roots-security) * [Part 5: Self-Assessment](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#part-5-self-assessment) * [Additional Resources](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/chapter-quiz#additional-resources) --- # Evals Are Exams for Reasoning | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You built an agent. It "works"—the code runs, no errors appear, and sometimes it produces exactly what you expected. But users are complaining. "It doesn't understand what I need." "The suggestions don't fit my situation." "It keeps missing the point." You run your test suite. Everything passes. Yet the agent clearly isn't performing well in the real world. What's happening? The problem isn't your code. The problem is that you're testing the wrong thing. You're checking whether the calculator works when you should be testing whether the student knows when to multiply. The Calculator vs Student Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-calculator-vs-student-problem "Direct link to The Calculator vs Student Problem") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Traditional software testing answers a simple question: **Does this code produce the correct output for a given input?** def add(a, b): return a + bdef test_add(): assert add(2, 3) == 5 # PASS or FAIL - deterministic This is Test-Driven Development (TDD). You define expected outputs. Code either produces them (PASS) or doesn't (FAIL). There's no ambiguity—no "mostly correct" or "pretty good." But consider a different kind of test. Not "does the calculator work?" but "does the student know when to use multiplication?" You present a word problem: "A farmer has 3 fields, each with 4 cows. How many cows total?" A student who writes "3 + 4 = 7" has the calculator working perfectly—addition is correct! But they made the wrong decision about WHICH operation to use. The reasoning was flawed, even though the calculation was accurate. **This is exactly the challenge with AI agents.** Your agent's code might execute flawlessly. API calls succeed. JSON parses correctly. But did the agent make the right DECISION about what to do? Did it correctly interpret what the user actually needed? Did it choose the appropriate tool for the situation? TDD can't answer these questions. You need a different approach: **Evals**. TDD vs Evals: The Core Distinction[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#tdd-vs-evals-the-core-distinction "Direct link to TDD vs Evals: The Core Distinction") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Andrew Ng, co-founder of Google Brain and former Chief Scientist at Baidu, has observed hundreds of teams building AI agents. His assessment is stark: > "One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." — Andrew Ng What separates successful agent builders from struggling ones isn't better prompts or fancier architectures. It's **systematic evaluation of reasoning quality**. Here's the fundamental distinction: | Aspect | TDD (Code Testing) | Evals (Agent Evaluation) | | --- | --- | --- | | **What it tests** | Does function return correct output? | Did agent make the right decision? | | **Outcome type** | PASS or FAIL (deterministic) | Scores (probabilistic) | | **Example question** | "Does `get_weather()` return valid JSON?" | "Did agent correctly interpret user intent?" | | **Analogy** | Testing if calculator works | Testing if student knows WHEN to use multiplication | | **When it fails** | Code bug or logic error | Reasoning failure, context misunderstanding | | **How to improve** | Fix the code | Improve prompts, add context, adjust guardrails | ### Why Probabilistic Outcomes?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#why-probabilistic-outcomes "Direct link to Why Probabilistic Outcomes?") TDD produces binary results because code execution is deterministic. Run the same function with the same inputs, you get the same output every time. Agent reasoning is different. The same prompt can produce varying quality responses: * Model randomness (temperature settings) * Context interpretation differences * Ambiguous user requests that allow multiple valid interpretations * Edge cases the prompt doesn't explicitly address When you ask an agent "Help me write a professional email to decline a meeting," there isn't ONE correct answer. There's a range of acceptable responses and a range of unacceptable ones. The evaluation question isn't "Is this exactly right?" but "How good is this?" That's why evals produce **scores**, not binary verdicts. The Exam Analogy: Initial vs Regression Evals[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-exam-analogy-initial-vs-regression-evals "Direct link to The Exam Analogy: Initial vs Regression Evals") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Think of evals like course exams for your agent: | Eval Type | Educational Analogy | Purpose | | --- | --- | --- | | **Initial Eval** | Final exam | Does the agent pass the course? Can it handle all the scenarios it's designed for? | | **Regression Eval** | Pop quiz | Did the latest update break something that was working? | | **Component Eval** | Subject test | Test individual skills in isolation (routing, tool selection, output formatting) | | **End-to-End Eval** | Comprehensive exam | Test the full experience from input to final output | When you first build an agent, you run Initial Evals—comprehensive tests that verify it can handle its designed purpose. These are your "final exams." But agents evolve. You improve prompts, add features, adjust guardrails. Each change might break something that was working. Regression Evals are your "pop quizzes"—quick checks that previous capabilities still work after changes. What Makes Evals Different From Tests[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#what-makes-evals-different-from-tests "Direct link to What Makes Evals Different From Tests") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's a concrete example. Imagine you've built a Task API agent that helps users manage their to-do items. **Traditional test (TDD approach):** def test_create_task(): response = agent.create_task( title="Buy groceries", due_date="2025-01-15" ) assert response.status_code == 200 assert "task_id" in response.json() This tests that the code works. It doesn't test whether the agent made good decisions. **Evaluation (Evals approach):** def eval_task_interpretation(): """ Evaluate: Does the agent correctly interpret ambiguous task requests? """ test_cases = [ { "input": "remind me to call mom", "expected_interpretation": "Creates a task with reminder, not a calendar event", "criteria": ["task_created", "reminder_set", "no_calendar_event"] }, { "input": "meeting prep for tomorrow", "expected_interpretation": "Creates task due before the meeting, not the meeting itself", "criteria": ["task_created", "due_before_meeting", "no_meeting_created"] } ] results = [] for case in test_cases: output = agent.process(case["input"]) score = grade_interpretation(output, case["criteria"]) results.append(score) return sum(results) / len(results) # Returns 0.0 to 1.0, not PASS/FAIL Notice the difference: * TDD asks: "Did the function return 200?" * Evals ask: "Did the agent understand that 'remind me to call mom' should create a task with a reminder, not schedule a phone call on a calendar?" The Core Insight: What Needs Testing vs What Needs Evaluation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-core-insight-what-needs-testing-vs-what-needs-evaluation "Direct link to The Core Insight: What Needs Testing vs What Needs Evaluation") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not everything about your agent needs evals. Some things are still TDD territory: **Use TDD (traditional tests) for:** * Does the API endpoint return valid JSON? * Does authentication work correctly? * Does the database connection succeed? * Does the function handle null inputs without crashing? **Use Evals for:** * Did the agent correctly interpret user intent? * Did it select the appropriate tool for the task? * Is the output quality acceptable for the use case? * Did it handle ambiguity appropriately? * Did it maintain appropriate context across turns? The rule of thumb: **If there's exactly one correct answer, use TDD. If there's a range of acceptable answers requiring judgment, use evals.** Exercise: Identify Evaluation Targets in Your Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#exercise-identify-evaluation-targets-in-your-agent "Direct link to Exercise: Identify Evaluation Targets in Your Agent") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Think about an agent you're building or planning to build (or use the Task API agent from earlier chapters as your example). List three behaviors that need **evaluation** rather than traditional testing: **Example answers for Task API agent:** 1. **Intent interpretation**: When a user says "handle my emails," does the agent correctly understand they want to create tasks from emails, not send emails? 2. **Priority inference**: When a user creates a task without specifying priority, does the agent make reasonable priority decisions based on content ("urgent meeting prep" should be high priority)? 3. **Due date interpretation**: When a user says "do this soon," does the agent set an appropriate near-term due date rather than leaving it blank or setting it for next year? Each of these involves JUDGMENT—there's no single correct answer, but there are clearly better and worse decisions. That's what makes them eval targets rather than test targets. The Eval-Driven Development Loop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-eval-driven-development-loop "Direct link to The Eval-Driven Development Loop") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Just as TDD follows "write test, write code, verify test passes," eval-driven development follows its own loop: Build agent v1 | vCreate eval dataset (10-20 cases) | vRun evals --> Find 70% pass rate | vError analysis --> "45% of errors are routing problems" | vFix routing --> Run evals --> 85% pass rate | vError analysis --> "30% of errors are output format issues" | vFix format --> Run evals --> 92% pass rate | vShip with regression protection The key insight: **Don't guess what's wrong—measure it.** When your agent underperforms, evals tell you WHERE it's failing so you can focus improvement efforts. Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you're building the agent-evals skill from Lesson 0, add this core distinction to your skill: **Key insight to encode**: TDD validates code correctness with deterministic PASS/FAIL. Evals measure reasoning quality with probabilistic scores. Use TDD for "does it work?" questions. Use evals for "did it decide correctly?" questions. **Decision framework to add**: When evaluating agent behavior, ask: "Is there exactly one correct answer?" If yes, use TDD. If there's a range of acceptable answers requiring judgment, use evals. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion (Claude, ChatGPT, Gemini, or similar). ### Prompt 1: Explore the Distinction[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#prompt-1-explore-the-distinction "Direct link to Prompt 1: Explore the Distinction") I'm learning about the difference between TDD (Test-Driven Development) andEvals for AI agents. Help me understand this better by giving me 3 examplesof agent behaviors that seem like they should be "tested" but actually need"evaluated." For each example, explain:1. Why a traditional test would miss the real quality issue2. What an eval would measure instead3. How the eval result would be a score rather than PASS/FAIL **What you're learning**: Deepening your understanding of what makes agent evaluation fundamentally different from code testing—the probabilistic nature of reasoning quality. ### Prompt 2: Classify Your Agent's Behaviors[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#prompt-2-classify-your-agents-behaviors "Direct link to Prompt 2: Classify Your Agent's Behaviors") I'm building an agent that [describe your agent's purpose]. Help me classifyits behaviors into two categories:1. Things that need traditional TDD tests (deterministic correctness)2. Things that need evals (reasoning quality judgment)For the eval category, suggest what criteria I might use to score eachbehavior from 0-5 or 0-100. **What you're learning**: Applying the TDD vs Evals framework to your own project—the first step toward building effective evaluation systems. ### Prompt 3: Challenge the Analogy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#prompt-3-challenge-the-analogy "Direct link to Prompt 3: Challenge the Analogy") The lesson used "calculator vs student" as an analogy for TDD vs Evals.Challenge this analogy: What are its limitations? When might it be misleading?Then suggest an alternative analogy that might work better for a specificaspect of agent evaluation. **What you're learning**: Critical evaluation of mental models—understanding both the strengths and limitations of analogies helps you know when they apply and when they don't. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#safety-note "Direct link to Safety Note") As you explore evals, remember that no evaluation system is perfect. Evals measure what you define as quality—if your criteria miss important dimensions, your agent can score well while still failing users. Always complement automated evals with real user feedback. Checking access... * [The Calculator vs Student Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-calculator-vs-student-problem) * [TDD vs Evals: The Core Distinction](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#tdd-vs-evals-the-core-distinction) * [Why Probabilistic Outcomes?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#why-probabilistic-outcomes) * [The Exam Analogy: Initial vs Regression Evals](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-exam-analogy-initial-vs-regression-evals) * [What Makes Evals Different From Tests](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#what-makes-evals-different-from-tests) * [The Core Insight: What Needs Testing vs What Needs Evaluation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-core-insight-what-needs-testing-vs-what-needs-evaluation) * [Exercise: Identify Evaluation Targets in Your Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#exercise-identify-evaluation-targets-in-your-agent) * [The Eval-Driven Development Loop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#the-eval-driven-development-loop) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#reflect-on-your-skill) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#try-with-ai) * [Prompt 1: Explore the Distinction](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#prompt-1-explore-the-distinction) * [Prompt 2: Classify Your Agent's Behaviors](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#prompt-2-classify-your-agents-behaviors) * [Prompt 3: Challenge the Analogy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#prompt-3-challenge-the-analogy) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/evals-are-exams-for-reasoning#safety-note) --- # Adding a Refresh Button | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your widget from Lesson 2 displays static content. But real applications need interactivity—buttons that do things. In this lesson, you'll add a "Refresh" button that triggers a new conversation turn when clicked. The pattern you'll learn here—`sendFollowUpMessage`—is how TaskManager's "Add Task" and "Complete Task" buttons will work. Master this pattern, and you've mastered widget-to-conversation communication. What We're Building[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#what-were-building "Direct link to What We're Building") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Starting from your Lesson 2 widget, you'll add: 1. A "Refresh" button 2. A timestamp showing when data was loaded 3. Button click that asks ChatGPT to refresh When complete, clicking the button inserts a message into the conversation like: "Refresh the greeting". The window.openai API[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#the-windowopenai-api "Direct link to The window.openai API") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Inside ChatGPT, your widget has access to `window.openai`—an API for communicating with the host. The method we need: | Method | Purpose | | --- | --- | | `sendFollowUpMessage({prompt})` | Insert a message into the conversation | When you call `sendFollowUpMessage`, ChatGPT receives the prompt as if the user typed it. The model processes it and may call your tools again. ### API Availability: The First Rule[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#api-availability-the-first-rule "Direct link to API Availability: The First Rule") Here's the catch: `window.openai` only exists inside ChatGPT. If you preview your widget HTML in a browser during development, this code crashes: // This crashes outside ChatGPTwindow.openai.sendFollowUpMessage({ prompt: "Refresh" }); **Output (in browser):** Uncaught TypeError: Cannot read properties of undefined The fix: always use optional chaining: // This works everywherewindow.openai?.sendFollowUpMessage?.({ prompt: "Refresh" }); **Output (in browser):** No error—the call is safely skipped. **Output (in ChatGPT):** Message appears in conversation. Updating Your Server[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#updating-your-server "Direct link to Updating Your Server") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Update `main.py` to include a timestamp and pass data to the widget: import mcp.types as typesfrom mcp.server.fastmcp import FastMCPfrom datetime import datetimeMIME_TYPE = "text/html+skybridge"# Updated widget with button and timestampWIDGET_HTML = '''

Hello!

Loading...

'''mcp = FastMCP("TaskManager")@mcp.tool()def show_greeting() -> types.CallToolResult: """Display a greeting widget with refresh capability.""" now = datetime.now().strftime("%H:%M:%S") return types.CallToolResult( content=[types.TextContent(type="text", text=f"Greeting updated at {now}")], structuredContent={ "message": "Hello, World!", "timestamp": now }, _meta={ "openai.com/widget": types.EmbeddedResource( type="resource", resource=types.TextResourceContents( uri="ui://greeting", mimeType=MIME_TYPE, text=WIDGET_HTML, ) ) } )if __name__ == "__main__": import uvicorn app = mcp.sse_app() uvicorn.run(app, host="0.0.0.0", port=8001) ### What Changed[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#what-changed "Direct link to What Changed") **1\. Added `structuredContent`:** structuredContent={ "message": "Hello, World!", "timestamp": now} This data is accessible in the widget via `window.openai.toolOutput`. **2\. Widget reads dynamic data:** const data = window.openai?.toolOutput;if (data) { document.getElementById('greeting').textContent = data.message; document.getElementById('time').textContent = 'Updated: ' + data.timestamp;} **3\. Button triggers refresh:** function refresh() { window.openai?.sendFollowUpMessage?.({ prompt: "Refresh the greeting" });} Testing the Refresh[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#testing-the-refresh "Direct link to Testing the Refresh") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Restart your server (`uv run main.py`) 2. Make sure ngrok is running 3. In ChatGPT, invoke your tool: "Show me a greeting" 4. See the widget with timestamp 5. Click the "Refresh" button 6. Watch a new message appear: "Refresh the greeting" 7. The model calls your tool again, showing a new timestamp Each refresh shows an updated timestamp, proving the round-trip works. Data Flow Summary[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#data-flow-summary "Direct link to Data Flow Summary") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's how data flows in a widget with `sendFollowUpMessage`: 1. User clicks button ↓2. Widget calls sendFollowUpMessage({ prompt: "..." }) ↓3. ChatGPT receives prompt as new conversation turn ↓4. Model decides to call your tool ↓5. Server returns structuredContent + widget ↓6. New widget renders with fresh data from toolOutput This is the foundation for TaskManager's "Add Task" button—it will send a prompt like "Add task: Buy groceries" and your server will update the task list. What You Built[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#what-you-built "Direct link to What You Built") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Building on Lesson 2, you added: * A "Refresh" button using `sendFollowUpMessage` * Dynamic data via `structuredContent` → `window.openai.toolOutput` * Safe API access with optional chaining Your widget now communicates bidirectionally with ChatGPT. In the next lesson, you'll display a list of tasks and learn to separate large data from what the model sees. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Add a Second Button[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#prompt-1-add-a-second-button "Direct link to Prompt 1: Add a Second Button") Add a "Say Hello Again" button next to the Refresh button.When clicked, it should send a different prompt like "Say hello to me again with enthusiasm".Style the buttons to sit side by side. **What you're learning:** Multiple buttons can trigger different prompts, enabling varied interactions from a single widget. ### Prompt 2: Add Loading State[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#prompt-2-add-loading-state "Direct link to Prompt 2: Add Loading State") When the Refresh button is clicked, disable it and change its text to "Loading..." until the widget is replaced by a new one. How can I detect that the button was clicked? **What you're learning:** UX patterns for async operations in widgets. Since the widget gets replaced on refresh, you'll discover why loading states work differently than in traditional SPAs. ### Prompt 3: Custom Prompt with Input[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#prompt-3-custom-prompt-with-input "Direct link to Prompt 3: Custom Prompt with Input") Replace the Refresh button with a text input and "Greet" button.When clicked, send a prompt like "Greet me by the name: [input value]".Update the server to accept a name parameter. **What you're learning:** Collecting user input in widgets and passing it through the conversation. This is exactly how TaskManager's "Add Task" form will work. Checking access... * [What We're Building](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#what-were-building) * [The window.openai API](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#the-windowopenai-api) * [API Availability: The First Rule](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#api-availability-the-first-rule) * [Updating Your Server](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#updating-your-server) * [What Changed](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#what-changed) * [Testing the Refresh](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#testing-the-refresh) * [Data Flow Summary](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#data-flow-summary) * [What You Built](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#what-you-built) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#try-with-ai) * [Prompt 1: Add a Second Button](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#prompt-1-add-a-second-button) * [Prompt 2: Add Loading State](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#prompt-2-add-loading-state) * [Prompt 3: Custom Prompt with Input](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/widget-interactivity#prompt-3-custom-prompt-with-input) --- # Apps SDK Architecture | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've built backend agents that reason, plan, and execute tasks. You've created MCP servers that expose tools and resources. Your agents work brilliantly—but they live on your server. How do you get them in front of users who need them? This is the distribution problem every software builder faces. You can build the most capable agent in the world, but if users can't find and use it, it doesn't matter. Traditional distribution requires building your own interface, marketing to acquire users, and competing with millions of other apps for attention. The Apps SDK solves this problem by putting your agents where users already are: inside ChatGPT. Why Apps SDK Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#why-apps-sdk-matters "Direct link to Why Apps SDK Matters") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ChatGPT has over **800 million weekly active users** as of late 2024. That's not a typo—800 million people use ChatGPT every week. OpenAI also reports that over **1 million businesses** now use ChatGPT, with **92% of Fortune 500 companies** having adopted the platform. These aren't casual users; they're professionals actively looking for tools to help them work. When you build a ChatGPT App using the Apps SDK, your application becomes available in the **ChatGPT App Directory** at [chatgpt.com/apps](https://chatgpt.com/apps) . Users can discover your app, add it to their ChatGPT, and start using it immediately—no separate website, no mobile app download, no user account creation on your end. But here's what makes this truly different: **AI-powered discovery**. When a ChatGPT Plus user describes a task, ChatGPT can recommend relevant apps from the directory. Your app doesn't just sit in a catalog waiting to be found—the AI actively suggests it to users who need it. ### The Distribution Advantage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#the-distribution-advantage "Direct link to The Distribution Advantage") Consider the traditional path to reach users: 1. Build your application 2. Build a web interface or mobile app 3. Set up hosting and infrastructure 4. Create marketing materials 5. Run ads or content marketing 6. Convert visitors to users 7. Handle user authentication 8. Maintain the entire stack With Apps SDK: 1. Build your MCP server (you already know how from Chapter 67) 2. Add widget responses for visual interfaces 3. Deploy and register in ChatGPT's App Directory 4. Users discover and use your app inside ChatGPT The launch partners for ChatGPT Apps include **Canva, Figma, DoorDash, Expedia, and Spotify**—companies that already have massive user bases chose to build ChatGPT Apps anyway. Why? Because being where users already work creates frictionless adoption. ### What Problem Does It Solve?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#what-problem-does-it-solve "Direct link to What Problem Does It Solve?") You've built agents that can reason. Now make them visual and accessible. The Apps SDK transforms your MCP servers into visual applications that render rich HTML widgets directly in ChatGPT. Instead of getting a text list of tasks, users see an interactive Kanban board. Instead of receiving JSON data, they see charts, tables, and action buttons. The conversation becomes a visual workspace—and it's available to 800 million potential users. This lesson establishes the mental model you need before writing any code. You'll understand how ChatGPT, widgets, and your server communicate—the foundation for everything you build in this chapter. The Three-Layer Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#the-three-layer-architecture "Direct link to The Three-Layer Architecture") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ChatGPT Apps consist of three distinct layers that work together: ┌─────────────────────────────────────────────────────────────────┐│ LAYER 1: ChatGPT UI ││ - Renders conversation messages ││ - Hosts widget iframes ││ - Routes tool calls to MCP servers │├─────────────────────────────────────────────────────────────────┤│ LAYER 2: Widget (iframe) ││ - Your HTML/CSS/JS running in sandboxed iframe ││ - Communicates with ChatGPT via window.openai API ││ - Displays rich UI: charts, forms, buttons, tables │├─────────────────────────────────────────────────────────────────┤│ LAYER 3: MCP Server ││ - Your FastMCP/Node server (just like Chapter 67) ││ - Exposes tools with special widget metadata ││ - Returns structuredContent for widgets + text for model │└─────────────────────────────────────────────────────────────────┘ **Layer 1: ChatGPT UI** is OpenAI's interface that users interact with. It handles the conversation, invokes your MCP tools, and creates sandboxed iframes to display your widgets. **Layer 2: Widget** is your HTML/CSS/JavaScript code running inside an iframe. The iframe is sandboxed for security, but your widget can communicate with ChatGPT through the `window.openai` API. This is where your visual UI lives. **Layer 3: MCP Server** is the backend you already know how to build from Chapters 66-67. The difference? Your tool responses include special metadata that tells ChatGPT to render a widget instead of just showing text. ### Why Three Layers?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#why-three-layers "Direct link to Why Three Layers?") This architecture provides security through isolation. Your widget code runs in a sandboxed iframe—it cannot access the parent ChatGPT page or other widgets. Your MCP server runs on your infrastructure, keeping sensitive data and API keys secure. ChatGPT acts as the trusted intermediary. Data Flow: From Prompt to Widget[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#data-flow-from-prompt-to-widget "Direct link to Data Flow: From Prompt to Widget") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Understanding the data flow is essential for debugging and design. Here's what happens when a user asks your app to show data: **Step 1: User Prompt** User: "Show me the task board for Project Alpha" **Step 2: Model Tool Call** ChatGPT's model decides to call your tool. It sends a request to your MCP server: Tool call: show_kanban_boardArguments: { "project": "Project Alpha" } **Step 3: Server Response** Your MCP server fetches data and returns a structured response: return CallToolResult( content=[TextContent(type="text", text="Here's your board.")], structuredContent={ "columns": [ {"id": "todo", "tasks": [...]}, {"id": "in-progress", "tasks": [...]}, {"id": "done", "tasks": [...]} ] }, _meta={ "openai.com/widget": embedded_widget_resource }) **Step 4: Widget Render** ChatGPT sees the widget metadata in `_meta` and creates an iframe. Your widget HTML loads and accesses the data through `window.openai.toolOutput`. **Step 5: Model Narration** The model reads `structuredContent` and generates a natural language response: "I've loaded the Kanban board for Project Alpha. You have 5 tasks in progress." **Complete Flow Diagram:** User Prompt │ ▼ChatGPT Model (decides which tool to call) │ ▼MCP Server (your code: fetches data, returns response) │ ▼ChatGPT (creates iframe, loads widget HTML) │ ▼Widget (reads window.openai.toolOutput, renders UI) │ ▼Model Narration (summarizes what happened) Apps SDK vs Agents SDK[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#apps-sdk-vs-agents-sdk "Direct link to Apps SDK vs Agents SDK") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Both SDKs come from OpenAI, but they solve different problems: | Aspect | Agents SDK (Chapter 62) | Apps SDK (This Chapter) | | --- | --- | --- | | **Purpose** | Build backend agent logic | Build user-facing widgets | | **Where it runs** | Your server (Python/Node) | ChatGPT interface (iframe) | | **User interaction** | Through conversation text | Direct clicks, inputs, buttons | | **Output type** | Text and tool results | Interactive HTML widgets | | **Primary skill** | Agent orchestration, tools, handoffs | Widget design, window.openai API | | **When to use** | Complex reasoning, multi-agent workflows | Visual dashboards, interactive forms | **The Key Insight**: These SDKs complement each other. You might use Agents SDK to build a sophisticated backend agent, then use Apps SDK to give that agent a visual interface inside ChatGPT. **Example Scenario**: Your Agents SDK agent analyzes sales data and produces recommendations. Your Apps SDK widget displays those recommendations as an interactive chart with drill-down capabilities. Widget vs Standard MCP Tool[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#widget-vs-standard-mcp-tool "Direct link to Widget vs Standard MCP Tool") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You already know how to build MCP tools from Chapter 67. Widget-enabled tools differ in three key ways: ### 1\. Response MIME Type[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#1-response-mime-type "Direct link to 1. Response MIME Type") Standard MCP tools return text or JSON that the model summarizes: # Standard MCP toolreturn CallToolResult( content=[TextContent(type="text", text="Found 15 tasks")]) Widget-enabled tools include HTML with a special MIME type: # Widget-enabled toolMIME_TYPE = "text/html+skybridge" # This is the magicreturn CallToolResult( content=[TextContent(type="text", text="Loading board...")], _meta={ "openai.com/widget": EmbeddedResource( type="resource", resource=TextResourceContents( uri="ui://widget/kanban.html", mimeType=MIME_TYPE, # Tells ChatGPT: render this as widget text=WIDGET_HTML, title="Kanban Board" ) ) }) The `text/html+skybridge` MIME type signals to ChatGPT: "Don't show this as text—render it as an interactive widget." ### 2\. Dual Response Channels[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#2-dual-response-channels "Direct link to 2. Dual Response Channels") Standard tools have one output channel (text for the model). Widget tools have two: | Channel | Visibility | Purpose | | --- | --- | --- | | `structuredContent` | Model + Widget | Concise data for model narration | | `_meta` | Widget only | Large/sensitive data hidden from model | Why separate channels? The model has limited context. If you pass a 10,000-task dataset through `structuredContent`, the model tries to process it all. By putting large data in `_meta`, your widget can access it while the model only sees a summary. ### 3\. Bidirectional Communication[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#3-bidirectional-communication "Direct link to 3. Bidirectional Communication") Standard MCP tools are one-way: server responds, conversation continues. Widget-enabled tools are bidirectional: * **Widget to ChatGPT**: `window.openai.sendFollowUpMessage()` inserts messages * **Widget to Server**: `window.openai.callTool()` invokes other MCP tools * **ChatGPT to Widget**: Data flows through `window.openai.toolOutput` This enables real interaction. A button in your widget can trigger a new conversation turn or refresh data from your server. Exploring Official Examples[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#exploring-official-examples "Direct link to Exploring Official Examples") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ OpenAI provides a repository of example ChatGPT Apps at: https://github.com/openai/openai-apps-sdk-examples The `kitchen_sink` example demonstrates most Apps SDK features in one project: ### kitchen\_sink Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#kitchen_sink-structure "Direct link to kitchen_sink Structure") kitchen_sink/├── server/│ └── main.py # FastMCP server with widget tools├── widgets/│ ├── index.html # Widget HTML with window.openai usage│ └── styles.css # Widget styling├── pyproject.toml # Python dependencies (managed by uv)└── README.md # Setup instructions ### Key Files to Examine[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#key-files-to-examine "Direct link to Key Files to Examine") **server/main.py** - Look for these patterns: 1. **Widget resource registration**: How the server defines widget HTML as an MCP resource 2. **Tool metadata**: The `_meta` dictionary with `openai/outputTemplate` 3. **Response structure**: How `structuredContent` and `_meta` are used **widgets/index.html** - Look for these patterns: 1. **window.openai access**: How the widget reads `toolOutput` 2. **API availability checks**: Defensive `window.openai?.methodName` patterns 3. **Button handlers**: How clicks trigger `sendFollowUpMessage` or `callTool` ### What Each Example Teaches[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#what-each-example-teaches "Direct link to What Each Example Teaches") | Example | Primary Concept | | --- | --- | | `kitchen_sink` | All core features: tools, widgets, state, display modes | | `todo_app` | CRUD operations with widget state persistence | | `oauth-protected` | Authentication flow with protected resources | | `file-upload` | Handling user file uploads in widgets | For this chapter, start with `kitchen_sink`. It's comprehensive and well-documented. What You Need to Remember[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#what-you-need-to-remember "Direct link to What You Need to Remember") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Before building your first ChatGPT App, internalize these concepts: **Three Layers**: 1. ChatGPT UI (hosts everything) 2. Widget iframe (your visual code) 3. MCP Server (your backend logic) **Data Flow**: * User prompt triggers model tool call * Server returns structured data + widget HTML * ChatGPT renders widget in iframe * Model narrates what happened **Key Differences from Standard MCP**: * MIME type: `text/html+skybridge` * Dual channels: `structuredContent` (model sees) + `_meta` (widget only) * Bidirectional: widgets can call tools and send messages Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Explore the Distribution Opportunity[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#prompt-1-explore-the-distribution-opportunity "Direct link to Prompt 1: Explore the Distribution Opportunity") I'm building an AI agent that helps users manage their tasks.Compare these two distribution strategies:1. Building a standalone web app with my own user acquisition2. Building a ChatGPT App using the Apps SDKWhat are the advantages and trade-offs of each approach?Consider: user acquisition cost, maintenance burden, and user experience. **What you're learning:** This prompt helps you articulate the business case for Apps SDK to stakeholders. Understanding the distribution advantage is as important as understanding the technical architecture. ### Prompt 2: Understand the Three-Layer Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#prompt-2-understand-the-three-layer-architecture "Direct link to Prompt 2: Understand the Three-Layer Architecture") Explain the three-layer architecture of ChatGPT Apps:- ChatGPT UI (Layer 1)- Widget iframe (Layer 2)- MCP Server (Layer 3)For each layer, explain:1. What runs there2. Who controls it (OpenAI vs developer)3. How it communicates with adjacent layersThen explain why this separation exists from a security perspective. **What you're learning:** This prompt reinforces the architectural model you'll use throughout the chapter. Understanding _why_ the layers are separated (security isolation) helps you make better design decisions. ### Prompt 3: Compare SDK Options[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#prompt-3-compare-sdk-options "Direct link to Prompt 3: Compare SDK Options") I'm confused about OpenAI's different SDKs. Compare:- OpenAI Agents SDK (what we learned in Chapter 62)- OpenAI Apps SDK (what we're learning now)When would I use each? Can they work together?Give me a concrete example of an application that uses both. **What you're learning:** This prompt clarifies the relationship between SDKs you've already learned and the new one. Many developers confuse these—you won't. Activities[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activities "Direct link to Activities") --------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Activity 1: Pitch the Distribution Advantage[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-1-pitch-the-distribution-advantage "Direct link to Activity 1: Pitch the Distribution Advantage") Imagine you're explaining Apps SDK to a non-technical colleague. Write a 2-3 sentence pitch that explains: 1. The problem it solves (distribution) 2. The opportunity (800M+ users, AI-powered discovery) 3. Why this matters for your specific domain Test your pitch: Would this convince someone to invest time learning the Apps SDK? ### Activity 2: Draw the Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-2-draw-the-architecture "Direct link to Activity 2: Draw the Architecture") Without looking at this lesson, draw the three-layer architecture from memory: 1. Draw three boxes representing the three layers 2. Label each layer with its name and primary responsibility 3. Draw arrows showing how data flows between layers 4. Label each arrow with what data travels through it Check your diagram against the architecture section above. What did you remember? What did you miss? ### Activity 3: Complete the Comparison Table[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-3-complete-the-comparison-table "Direct link to Activity 3: Complete the Comparison Table") Fill in this comparison table from memory: | Aspect | Agents SDK | Apps SDK | | --- | --- | --- | | Purpose | ? | ? | | Where it runs | ? | ? | | User interaction | ? | ? | | Output type | ? | ? | Then check your answers against the comparison table in this lesson. ### Activity 4: Clone and Explore kitchen\_sink[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-4-clone-and-explore-kitchen_sink "Direct link to Activity 4: Clone and Explore kitchen_sink") Clone the official examples repository and explore the kitchen\_sink example: git clone https://github.com/openai/openai-apps-sdk-examples.gitcd openai-apps-sdk-examples/kitchen_sink Answer these questions by exploring the code: 1. In `server/main.py`, find where the widget MIME type is defined. What is the exact string? 2. In `widgets/index.html`, find at least two `window.openai` methods being used. What are they? 3. How many tool definitions are in the server? List their names. Write your answers down before moving to the next lesson. Checking access... * [Why Apps SDK Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#why-apps-sdk-matters) * [The Distribution Advantage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#the-distribution-advantage) * [What Problem Does It Solve?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#what-problem-does-it-solve) * [The Three-Layer Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#the-three-layer-architecture) * [Why Three Layers?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#why-three-layers) * [Data Flow: From Prompt to Widget](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#data-flow-from-prompt-to-widget) * [Apps SDK vs Agents SDK](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#apps-sdk-vs-agents-sdk) * [Widget vs Standard MCP Tool](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#widget-vs-standard-mcp-tool) * [1\. Response MIME Type](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#1-response-mime-type) * [2\. Dual Response Channels](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#2-dual-response-channels) * [3\. Bidirectional Communication](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#3-bidirectional-communication) * [Exploring Official Examples](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#exploring-official-examples) * [kitchen\_sink Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#kitchen_sink-structure) * [Key Files to Examine](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#key-files-to-examine) * [What Each Example Teaches](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#what-each-example-teaches) * [What You Need to Remember](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#what-you-need-to-remember) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#try-with-ai) * [Prompt 1: Explore the Distribution Opportunity](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#prompt-1-explore-the-distribution-opportunity) * [Prompt 2: Understand the Three-Layer Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#prompt-2-understand-the-three-layer-architecture) * [Prompt 3: Compare SDK Options](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#prompt-3-compare-sdk-options) * [Activities](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activities) * [Activity 1: Pitch the Distribution Advantage](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-1-pitch-the-distribution-advantage) * [Activity 2: Draw the Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-2-draw-the-architecture) * [Activity 3: Complete the Comparison Table](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-3-complete-the-comparison-table) * [Activity 4: Clone and Explore kitchen\_sink](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/apps-sdk-architecture#activity-4-clone-and-explore-kitchen_sink) --- # TDD Philosophy for Agent Development | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your agent test suite costs $1,847 per month to run. Every time you push code, 50 test cases fire. Each test calls OpenAI, generating roughly 2,000 tokens at $0.003 per 1K tokens. That's $0.30 per test run. With 20 pushes per day, 30 days per month, you're burning $1,847 just to validate your code works. But here's the real problem: **tests that cost money don't run often**. When running tests feels expensive, developers skip them. They push untested code. They merge pull requests with failing coverage. The agent ships with bugs that a $0.30 test would have caught—bugs that now cost $3,000 in customer support and reputation damage. This is the $50 test suite problem: your testing infrastructure is so expensive that you can't afford to use it properly. There's a better way. And it costs exactly $0.00 per test run. The TDD Philosophy: Test First, Then Implement[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#the-tdd-philosophy-test-first-then-implement "Direct link to The TDD Philosophy: Test First, Then Implement") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Test-Driven Development flips the traditional coding workflow on its head. **Traditional approach:** 1. Write code 2. Manually test it 3. Hope it works 4. Maybe write tests later (but probably not) **TDD approach:** 1. Write a failing test that defines desired behavior 2. Write minimum code to make test pass 3. Refactor while keeping tests green 4. Repeat The key insight: **you write the test before the implementation exists**. This sounds backwards until you understand what it accomplishes: | Benefit | Why It Matters | | --- | --- | | **Specification clarity** | Writing a test forces you to define exactly what the code should do | | **Instant feedback** | You know immediately when something breaks | | **Confidence to refactor** | Tests catch regressions, so you can improve code structure freely | | **Documentation that runs** | Tests show how code is meant to be used—and they prove it works | For agent development, TDD provides an additional critical benefit: **isolation from expensive LLM calls**. When you test agent code with TDD, you mock the LLM responses. The test validates your code logic—endpoint behavior, database operations, tool execution—without ever calling OpenAI or Anthropic. This is how you get from $1,847/month to $0/month. The Critical Distinction: TDD vs Evals[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#the-critical-distinction-tdd-vs-evals "Direct link to The Critical Distinction: TDD vs Evals") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's where most developers get confused. TDD and Evals both involve testing AI agents. But they test fundamentally different things using fundamentally different approaches. | Aspect | TDD (This Chapter) | Evals (Chapter 77) | | --- | --- | --- | | **Question** | Does the code work correctly? | Does the LLM reason well? | | **Nature** | Deterministic | Probabilistic | | **Output** | Pass/Fail | Scores (0-1) | | **Tests** | Functions, APIs, DB operations | Response quality, faithfulness | | **Speed** | Fast (mocked LLM) | Slow (real LLM calls) | | **Cost** | Zero (no API calls) | High (API calls required) | Let me break this down with concrete examples. ### TDD Tests Code Correctness[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#tdd-tests-code-correctness "Direct link to TDD Tests Code Correctness") Consider an agent that manages tasks. TDD tests answer questions like: * Does the `/tasks` endpoint return a 201 status when creating a task? * Does the database constraint prevent duplicate task titles? * Does the `create_task` tool function parse arguments correctly? * Does the error handler return proper JSON when the LLM times out? These questions have **deterministic answers**. Given the same input, you always get the same output. The test either passes or fails—there's no score between 0 and 1. And critically: **none of these questions require actually calling an LLM**. You mock the LLM response. You provide fake JSON that represents what OpenAI would return. Your code processes that fake response exactly as it would process a real one. If the code logic is correct, the test passes. ### Evals Measure LLM Reasoning Quality[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#evals-measure-llm-reasoning-quality "Direct link to Evals Measure LLM Reasoning Quality") Now consider different questions: * When asked "What are my high-priority tasks?", does the agent give a helpful response? * Does the agent correctly interpret ambiguous requests? * Are the agent's explanations accurate and trustworthy? * Does the agent refuse inappropriate requests? These questions **require real LLM calls** because you're measuring how the LLM thinks—not how your code processes the LLM's output. There's no way to mock this. The whole point is to see what GPT-4 or Claude actually produces when given your prompt. And the answer isn't pass/fail—it's a quality score that might vary across runs. ### Why the Distinction Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#why-the-distinction-matters "Direct link to Why the Distinction Matters") If you conflate TDD and Evals, you make expensive mistakes: **Mistake 1: Testing code logic with real LLM calls** Your endpoint handler is broken—it doesn't properly extract the task title from the response. You run a test that calls OpenAI, gets a response, and fails... but is the bug in your code or in the LLM's output? You can't tell without mocking. **Mistake 2: Trying to mock LLM reasoning quality** You create a mock that returns "Here are your high-priority tasks: Task 1, Task 2." Your test passes. But when users hit the real system, the LLM returns rambling, unhelpful responses. You tested your mock, not your agent. **The rule**: TDD for code correctness (mocked, fast, free). Evals for LLM quality (real calls, slow, expensive). What TDD Tests for Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#what-tdd-tests-for-agents "Direct link to What TDD Tests for Agents") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's what belongs in your TDD test suite: ### 1\. API Endpoint Behavior[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#1-api-endpoint-behavior "Direct link to 1. API Endpoint Behavior") async def test_create_task_returns_201(client): response = await client.post( "/api/tasks", json={"title": "Test Task", "priority": "high"} ) assert response.status_code == 201 assert response.json()["title"] == "Test Task" **Output:** tests/test_tasks.py::test_create_task_returns_201 PASSED This test verifies your FastAPI endpoint handles requests correctly. No LLM involved. ### 2\. Database Operations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#2-database-operations "Direct link to 2. Database Operations") async def test_cascade_delete_removes_subtasks(session): project = Project(name="Test Project") session.add(project) await session.commit() task = Task(title="Test Task", project_id=project.id) session.add(task) await session.commit() await session.delete(project) await session.commit() result = await session.get(Task, task.id) assert result is None # Cascade deleted **Output:** tests/test_models.py::test_cascade_delete_removes_subtasks PASSED This test verifies your SQLModel relationships work correctly. No LLM involved. ### 3\. Tool Function Logic[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#3-tool-function-logic "Direct link to 3. Tool Function Logic") def test_validate_input_rejects_injection(): malicious = "'; DROP TABLE users; --" with pytest.raises(ValidationError): validate_input(malicious) **Output:** tests/test_tools.py::test_validate_input_rejects_injection PASSED This test verifies your tool's input validation catches security threats. No LLM involved. ### 4\. Pipeline Flow with Mocked LLM[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#4-pipeline-flow-with-mocked-llm "Direct link to 4. Pipeline Flow with Mocked LLM") @respx.mockasync def test_agent_creates_task_on_request(client): # Mock what OpenAI would return respx.post("https://api.openai.com/v1/chat/completions").mock( return_value=httpx.Response(200, json={ "choices": [{ "message": { "tool_calls": [{ "function": { "name": "create_task", "arguments": '{"title": "Buy groceries"}' } }] } }] }) ) response = await client.post( "/api/agent/chat", json={"message": "Create a task to buy groceries"} ) assert response.status_code == 200 # Verify task was actually created in database tasks = await client.get("/api/tasks") assert any(t["title"] == "Buy groceries" for t in tasks.json()) **Output:** tests/test_agent.py::test_agent_creates_task_on_request PASSED This test verifies your entire pipeline—from API to LLM processing to database—works correctly. The LLM is mocked, so the test is fast and free. What TDD Does NOT Test[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#what-tdd-does-not-test "Direct link to What TDD Does NOT Test") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ These questions require Evals, not TDD: | Question | Why It's an Eval | | --- | --- | | Is the agent's response helpful? | Requires human judgment or LLM-as-judge | | Does the agent interpret ambiguous requests correctly? | Depends on LLM reasoning, not code logic | | Are responses safe and appropriate? | Requires running against content filters | | Does the agent stay on topic? | Measures LLM behavior, not code behavior | | Is the explanation accurate and trustworthy? | Requires factuality evaluation against ground truth | You'll learn to build evaluation systems for these questions in Chapter 77. For now, the key insight: **don't try to TDD these qualities by mocking LLM responses**. When you mock the response, you're testing that your code handles _that specific mock_ correctly. You're not testing that your prompt produces good outputs when fed to the real LLM. The Cost Calculation: Why This Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#the-cost-calculation-why-this-matters "Direct link to The Cost Calculation: Why This Matters") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's revisit that $1,847/month test suite. **Without mocking:** * 50 tests per run * 2,000 tokens per test (request + response) * $0.003 per 1K tokens (GPT-4o pricing) * Cost per test: $0.006 * Cost per full run: $0.30 * Runs per day: 20 (CI/CD on every push) * Monthly cost: $0.30 x 20 x 30 = **$180** That seems manageable—until you consider: * Multiple developers running tests locally * Test runs during debugging (10x more frequent) * Integration tests that make 5-10 LLM calls each Realistic monthly cost: **$1,000-2,000** **With mocking:** * 50 tests per run * Zero API calls * Cost per test: $0.00 * Cost per full run: $0.00 * Monthly cost: **$0.00** But cost isn't even the main benefit. **Speed comparison:** | Approach | Time per Test | Full Suite (50 tests) | | --- | --- | --- | | Real LLM calls | 2-5 seconds | 2-4 minutes | | Mocked responses | 10-50 milliseconds | <3 seconds | When tests take 3 seconds, you run them constantly. After every change. Before every commit. During every code review. When tests take 4 minutes, you run them once before merging and hope nothing broke. **The difference:** catching bugs in 3 seconds versus discovering them in production. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI assistant to practice applying the TDD vs Evals framework. ### Prompt 1: Categorize Agent Behaviors[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#prompt-1-categorize-agent-behaviors "Direct link to Prompt 1: Categorize Agent Behaviors") I'm building a customer support agent that:1. Creates support tickets in our database2. Retrieves customer order history via API3. Generates helpful responses to customer questions4. Escalates complex issues to human agents5. Validates customer identity before accessing accountsHelp me categorize each behavior: Should it be tested with TDD(deterministic, mockable) or Evals (requires real LLM, measuresquality)? For each, explain your reasoning. **What you're learning:** The skill of categorizing behaviors is foundational to building an effective test strategy. You need to know which tests to mock and which tests require real LLM calls before you can build either. ### Prompt 2: Calculate Your Testing Cost[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#prompt-2-calculate-your-testing-cost "Direct link to Prompt 2: Calculate Your Testing Cost") I'm planning tests for an agent with these characteristics:- 30 API endpoints to test- 10 database operations to validate- 5 agent tools that need unit tests- 3 integration flows that test the full pipelineIf I use real LLM calls for all tests:- Average 1,500 tokens per call- GPT-4o at $0.0025 per 1K tokens- I run tests 15 times per day- 22 working days per monthCalculate my monthly testing cost. Then calculate what I save bymocking the LLM calls and only using real calls for the 3 integrationflows. **What you're learning:** Quantifying costs makes the case for test infrastructure investment. When you can say "mocking saves us $X per month," you can justify the time spent learning these patterns. ### Prompt 3: Apply to Your Domain[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#prompt-3-apply-to-your-domain "Direct link to Prompt 3: Apply to Your Domain") I'm building an agent for [describe your domain—e.g., "schedulingmedical appointments" or "processing insurance claims" or "tutoringstudents in math"].Based on the TDD vs Evals framework from this lesson, help me identify:1. Three code behaviors I should test with TDD (deterministic, mockable, tests code correctness)2. Two quality aspects I should measure with Evals (probabilistic, requires real LLM, measures reasoning quality)For each, suggest what a test or eval might look like at a high level. **What you're learning:** Applying abstract frameworks to concrete domains is how knowledge becomes skill. Your domain expertise combined with the TDD/Evals framework produces a test strategy tailored to your specific agent. Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#reflect-on-your-skill "Direct link to Reflect on Your Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you created the `agent-tdd` skill in Lesson 0, let's verify and improve it. **Test Your Skill:** Using my agent-tdd skill, explain when to use TDD versus Evals fortesting an AI agent. Does my skill correctly distinguish:- Deterministic tests from probabilistic evaluations?- Code correctness from LLM reasoning quality?- Zero-cost mocked tests from expensive real-call tests? **Identify Gaps:** After running that prompt, ask yourself: * Does my skill include the comparison table from this lesson? * Does it explain the cost implications of unmocked tests? * Does it list what TDD tests versus what it doesn't test? **Improve Your Skill:** If you found gaps, update your skill: My agent-tdd skill needs a clearer TDD vs Evals section. Add:1. The six-dimension comparison table (Question, Nature, Output, Tests, Speed, Cost)2. A "What TDD Tests" section listing: endpoint behavior, DB operations, tool logic, pipeline flow with mocked LLM3. A "What TDD Does NOT Test" section: response quality, interpretation accuracy, safety, topic adherence4. A cost analysis showing why mocked tests enable rapid iteration Your skill grows as you learn. By the end of this chapter, your `agent-tdd` skill will encode everything you've learned—a reusable asset for every future agent project. * [The TDD Philosophy: Test First, Then Implement](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#the-tdd-philosophy-test-first-then-implement) * [The Critical Distinction: TDD vs Evals](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#the-critical-distinction-tdd-vs-evals) * [TDD Tests Code Correctness](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#tdd-tests-code-correctness) * [Evals Measure LLM Reasoning Quality](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#evals-measure-llm-reasoning-quality) * [Why the Distinction Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#why-the-distinction-matters) * [What TDD Tests for Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#what-tdd-tests-for-agents) * [1\. API Endpoint Behavior](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#1-api-endpoint-behavior) * [2\. Database Operations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#2-database-operations) * [3\. Tool Function Logic](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#3-tool-function-logic) * [4\. Pipeline Flow with Mocked LLM](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#4-pipeline-flow-with-mocked-llm) * [What TDD Does NOT Test](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#what-tdd-does-not-test) * [The Cost Calculation: Why This Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#the-cost-calculation-why-this-matters) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#try-with-ai) * [Prompt 1: Categorize Agent Behaviors](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#prompt-1-categorize-agent-behaviors) * [Prompt 2: Calculate Your Testing Cost](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#prompt-2-calculate-your-testing-cost) * [Prompt 3: Apply to Your Domain](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#prompt-3-apply-to-your-domain) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/tdd-for-agents/tdd-philosophy-for-agents#reflect-on-your-skill) --- # Chapter 65: Anthropic Claude Agent SDK | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C You've built agents with OpenAI and Google frameworks. Now you'll learn Anthropic's Claude Agent SDK—the architecture that powers Claude Code, the tool you've been using throughout this book. Understanding how Claude Code works internally transforms you from user to architect. Anthropic's approach emphasizes constitutional AI principles, computer use capabilities, and the extended thinking that makes Claude distinctive. The SDK provides primitives for building agents that can use computers like humans do—clicking, typing, navigating—alongside traditional tool use. You'll see how the general agent (Claude Code) and custom agents (your creations) share the same architectural foundation. This chapter completes your framework survey. By the end, you'll understand three major approaches to agent development and know how to choose based on requirements, not vendor preference. What You'll Learn[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#what-youll-learn "Direct link to What You'll Learn") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ By the end of this chapter, you'll be able to: * **Configure the Claude Agent SDK**: Set up authentication, understand the SDK's architecture, and connect to Claude models * **Implement tool use patterns**: Define tools using Anthropic's schema format, handle tool results, and manage conversation context * **Build computer-use agents**: Create agents that interact with desktop applications, browsers, and GUIs * **Apply extended thinking**: Leverage Claude's reasoning capabilities for complex multi-step problems * **Understand Claude Code's architecture**: See how skills, subagents, and MCP servers compose into the agent you use daily * **Compare all three frameworks**: Synthesize learnings across OpenAI, Google, and Anthropic approaches Chapter Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#chapter-structure "Direct link to Chapter Structure") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. **Claude SDK Setup & Architecture** — Authentication, model selection, and SDK primitives 2. **Tool Use with Claude** — Tool definitions, parameter handling, and the tool-use conversation loop 3. **Computer Use Capabilities** — Screen reading, mouse/keyboard control, and GUI automation patterns 4. **Extended Thinking & Reasoning** — Leveraging Claude's chain-of-thought for complex problems 5. **Inside Claude Code** — Skills as reusable intelligence, subagent orchestration, and MCP integration 6. **Framework Synthesis** — Decision matrix for choosing frameworks based on requirements 7. **Capstone: Development Assistant** — Spec-driven agent that combines tool use, computer use, and reasoning for development workflows Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#prerequisites "Direct link to Prerequisites") ------------------------------------------------------------------------------------------------------------------------------------------------------------------- * Chapter 61: Introduction to AI Agents (conceptual foundation) * Chapter 62: OpenAI Agents SDK (first framework experience) * Chapter 63: Google ADK (second framework comparison) * Part 4: Python Fundamentals (async/await, type hints) * Anthropic API key with Claude access * [What You'll Learn](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#what-youll-learn) * [Chapter Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#chapter-structure) * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development#prerequisites) --- # Chapter 70: FastAPI for Agents | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#__docusaurus_skipToContent_fallback) On this page Copy as MarkdownCtrl+⇧+C **You will build a FastAPI skill BEFORE you learn FastAPI.** This chapter follows the **Skill-First Learning Pattern**: In Lesson 0, you create a `fastapi-agent-api` skill using tools from Chapter 3. Then you spend the chapter understanding what you built and making it better. By the end, you don't just "know FastAPI"—you OWN a production-ready skill. Prerequisites[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#prerequisites "Direct link to Prerequisites") ----------------------------------------------------------------------------------------------------------------------------------------------------- * **Chapter 3**: skill-creator and fetching-library-docs * **Chapter 62**: OpenAI Agents SDK (Agent, Runner, function\_tool) * **Chapters 66-67**: MCP experience (HTTP/SSE patterns) * **Part 4**: Python Fundamentals (async/await, type hints, Pydantic) The Running Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#the-running-example "Direct link to The Running Example") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Lessons 1-10**: Build a Task API—production-ready CRUD with testing, database, auth, and dependency injection. **Lessons 11-15**: Add infrastructure and agent integration—middleware, lifespan, streaming, and the key insight: **APIs are functions, functions become tools, agents use tools**. What You'll Own[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#what-youll-own "Direct link to What You'll Own") ---------------------------------------------------------------------------------------------------------------------------------------------------------- .claude/skills/├── skill-creator/ # From Chapter 3├── fetching-library-docs/ # From Chapter 3└── fastapi-agent-api/ # NEW - this chapter This skill joins your growing **Digital FTE toolkit**. The Mindset Shift[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#the-mindset-shift "Direct link to The Mindset Shift") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- **Traditional**: "Teach me FastAPI" **Agent Factory**: "I own a FastAPI skill—help me make it better" You're not here to learn. You're here to **build assets**. Start with **Lesson 0: Build Your FastAPI Skill**. * [Prerequisites](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#prerequisites) * [The Running Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#the-running-example) * [What You'll Own](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#what-youll-own) * [The Mindset Shift](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents#the-mindset-shift) --- # Build Your FastAPI Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning FastAPI—the Python framework for building production APIs—you'll **own** a FastAPI skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Copy and paste this prompt: Using your skill creator skill create a new skill for FastAPI. I will useit to build projects with FastAPI from hello world to professional productionAPIs. Use context7 skill to study official documentation and then build itso no self assumed knowledge. Claude will: 1. Fetch official FastAPI documentation via Context7 2. Ask you clarifying questions (database preferences, auth patterns, deployment target) 3. Create the complete skill with references, templates, and starter code Your skill appears at `.claude/skills/fastapi-dev/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#done "Direct link to Done") --------------------------------------------------------------------------------------------------------------------------------------------------- You now own a FastAPI skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — Hello FastAPI** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/fastapi-for-agents/build-your-fastapi-skill#done) --- # Why Agents Need Structured Data | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your Task Manager agent just helped a user create 50 tasks across 5 projects. The user assigned workers, set priorities, added tags. Then the server restarted. Everything is gone. This isn't a hypothetical disaster—it's what happens when agents store data only in memory. The moment your process ends, your users lose their work. The Persistence Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#the-persistence-problem "Direct link to The Persistence Problem") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Consider what your Task Manager agent tracks: | Data | Type | Lifetime | | --- | --- | --- | | Tasks | Structured records | Permanent | | Projects | Related records | Permanent | | Worker assignments | Relationships | Permanent | | Conversation history | Sequential | Session or permanent | | Search embeddings | Vectors | Permanent | Memory handles conversation flow during a session. But tasks, projects, assignments—these must survive restarts, scale across instances, and remain consistent when multiple users access them simultaneously. You need a database. Relational vs Vector: Different Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#relational-vs-vector-different-questions "Direct link to Relational vs Vector: Different Questions") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You learned about vector databases in Chapter 73. They answer semantic questions: * "Find tasks similar to 'fix authentication'" * "What documents discuss deployment?" * "Show me related concepts" Relational databases answer structured questions: * "List all pending tasks for Project X" * "Who is assigned to Task #42?" * "How many tasks did each worker complete this week?" | Question Type | Database | Why | | --- | --- | --- | | "Similar to..." | Vector | Semantic similarity search | | "Filter by..." | Relational | Exact matching, ordering | | "Count/sum..." | Relational | Aggregation queries | | "Related to..." | Both | Depends on relationship type | Your agent needs both. Vector DB for semantic search. Relational DB for structured queries and persistent state. ACID: Why Agents Need Guarantees[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#acid-why-agents-need-guarantees "Direct link to ACID: Why Agents Need Guarantees") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When your agent updates the database, things can go wrong. Network drops. Server crashes. Two users edit the same task simultaneously. ACID properties protect you: **Atomicity**: All-or-nothing operations. If your agent creates a task with 5 subtasks, either all 6 records exist or none do. No half-created task trees. # Either both succeed or both failasync with session.begin(): session.add(parent_task) for subtask in subtasks: session.add(subtask) **Consistency**: Rules always hold. If your schema says `project_id` must reference a valid project, the database rejects orphan tasks. Your agent can't accidentally create invalid data. **Isolation**: Concurrent operations don't conflict. Two agents updating different tasks on the same project don't corrupt each other's work. **Durability**: Once committed, data survives crashes. Your user's 50 tasks persist through server restarts, power failures, and deployments. Why Async Matters for Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#why-async-matters-for-agents "Direct link to Why Async Matters for Agents") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your agent does many things: * Receives user requests * Calls LLM APIs * Reads and writes database * Sends responses Synchronous database access blocks everything. While waiting for a query, your agent can't process other requests. # SYNC: Blocks entire processdef get_tasks(project_id): return db.query(Task).filter_by(project_id=project_id).all() # Nothing else happens until this returns# ASYNC: Process continues while waitingasync def get_tasks(project_id): result = await session.exec(select(Task).where(Task.project_id == project_id)) return result.all() # Other requests processed while waiting Async database access lets your agent: * Handle multiple concurrent users * Process API responses while queries run * Scale to production workloads This is why we use `create_async_engine` and `AsyncSession` exclusively in this chapter. The Task Manager Data Model[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#the-task-manager-data-model "Direct link to The Task Manager Data Model") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Throughout this chapter, you'll build a database layer for the Task Manager: Project └── Task (many) ├── Worker (assigned) ├── Worker (created_by) └── Task (subtasks - self-referential) This structure exercises: * One-to-many relationships (Project → Tasks) * Many-to-one with multiple foreign keys (Task → Worker for two purposes) * Self-referential relationships (Task → parent/subtasks) * JSONB columns (tags, metadata) By chapter end, you'll have a complete async database layer matching production patterns. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Classify Your Data[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#prompt-1-classify-your-data "Direct link to Prompt 1: Classify Your Data") I'm building an AI agent that manages customer support tickets.It tracks:- Ticket records with status, priority, assignee- Customer information- Similar past tickets for context- Conversation history with customersFor each data type, recommend: relational database, vector database,or both? Explain your reasoning based on how the data will be queried. **What you're learning:** Database selection based on query patterns—matching data characteristics to appropriate storage. ### Prompt 2: Design for Reliability[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#prompt-2-design-for-reliability "Direct link to Prompt 2: Design for Reliability") My agent creates a Project with 10 Tasks in a single user request.What ACID property ensures that if task #7 fails to save,the entire operation rolls back including the Project and tasks 1-6?Show me the Python code pattern that implements this guaranteeusing SQLModel's async session. **What you're learning:** Atomicity in practice—understanding how transactions protect data integrity. ### Prompt 3: Justify Async[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#prompt-3-justify-async "Direct link to Prompt 3: Justify Async") My agent currently uses synchronous database access and handlesabout 10 requests per second. I want to scale to 100 requestsper second without adding more servers.Explain how async database access helps, and what changesI need to make to my SQLModel code to achieve this. **What you're learning:** Async performance reasoning—understanding why async matters for scale. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#safety-note "Direct link to Safety Note") Database operations can fail. Always handle connection errors, query timeouts, and constraint violations. Your agent should gracefully inform users when database issues occur rather than crashing silently. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#reflect-on-your-skill "Direct link to Reflect on Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You built a `relational-db-agent` skill in Lesson 0. Test its understanding of these concepts. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#test-your-skill "Direct link to Test Your Skill") Using my relational-db-agent skill, explain when I shoulduse a relational database vs a vector database for my agent.Give me a decision framework. ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill distinguish structured queries from semantic search? * Did it mention ACID properties? * Did it recommend async patterns? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My relational-db-agent skill doesn't explain when to userelational vs vector databases. Add guidance that:- Relational: exact matching, filtering, aggregation, relationships- Vector: semantic similarity, fuzzy matching, context retrieval- Often use both in agent architectures Your skill now helps you make better architectural decisions. Checking access... * [The Persistence Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#the-persistence-problem) * [Relational vs Vector: Different Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#relational-vs-vector-different-questions) * [ACID: Why Agents Need Guarantees](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#acid-why-agents-need-guarantees) * [Why Async Matters for Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#why-async-matters-for-agents) * [The Task Manager Data Model](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#the-task-manager-data-model) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#try-with-ai) * [Prompt 1: Classify Your Data](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#prompt-1-classify-your-data) * [Prompt 2: Design for Reliability](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#prompt-2-design-for-reliability) * [Prompt 3: Justify Async](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#prompt-3-justify-async) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#safety-note) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/why-agents-need-structured-data#improve-your-skill) --- # Build Your Database Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You're about to learn async database patterns with SQLModel. But you won't start with "what is SQLModel?" You'll start by **building a skill that knows SQLModel**. This is the skill-first approach: instead of accumulating knowledge that fades, you create an asset that grows. By chapter end, your `relational-db-agent` skill will generate production-quality database code—and you'll own it. Why Skill-First for Databases?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#why-skill-first-for-databases "Direct link to Why Skill-First for Databases?") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Database patterns are precise. Off-by-one errors in async patterns cause `MissingGreenlet` crashes. Forgotten `await` keywords block your entire agent. Wrong relationship configurations cascade into N+1 query disasters. A skill that knows the correct patterns eliminates these errors. You consult your skill, it generates correct code, you ship faster. Step 1: Clone the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-1-clone-the-skills-lab "Direct link to Step 1: Clone the Skills Lab") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Start fresh. Every chapter begins with a clean skills-lab: cd ~/projectsgit clone https://github.com/panaversity/claude-code-skills-lab.git ch44-skills-labcd ch44-skills-lab **Output:** Cloning into 'ch44-skills-lab'...remote: Enumerating objects: 45, done.remote: Counting objects: 100% (45/45), done.Receiving objects: 100% (45/45), done. Step 2: Write Your Learning Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-2-write-your-learning-specification "Direct link to Step 2: Write Your Learning Specification") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before building the skill, define what it should know. Create `LEARNING-SPEC.md`: # Learning Specification: Relational Database Skill## What This Skill Should Know### Core Patterns- Async engine creation with `create_async_engine`- Connection pooling for production (`pool_pre_ping`, `pool_size`)- AsyncSession from `sqlmodel.ext.asyncio.session`- Session lifecycle as FastAPI dependency### Model Patterns- SQLModel with `table=True`- JSONB columns for PostgreSQL- Foreign keys and relationships- Self-referential relationships (parent/child)### Operations- Async CRUD (add, flush, commit, refresh)- Eager loading with `selectinload`- Transaction patterns (rollback on error)### Migrations- Alembic async template- Model import requirements- Autogenerate and upgrade commands## Success CriteriaThe skill generates code that:1. Compiles without syntax errors2. Uses correct async/await patterns3. Includes proper imports4. Follows SQLModel conventions **Output:** Created LEARNING-SPEC.md (32 lines) Step 3: Fetch Official Documentation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-3-fetch-official-documentation "Direct link to Step 3: Fetch Official Documentation") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use the `/fetching-library-docs` command to get accurate patterns: /fetching-library-docs sqlmodel --topic "async session" The skill retrieves official SQLModel patterns—imports, code examples, and usage notes. **What you'll receive:** * `create_async_engine` syntax * `AsyncSession` import path * Session lifecycle patterns * Code examples from official docs Save the key patterns for your skill. Don't memorize—capture. Step 4: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-4-create-your-skill "Direct link to Step 4: Create Your Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use `/skill-creator` to build the `relational-db-agent` skill: /skill-creator relational-db-agentThe skill should:- Generate async SQLModel code for FastAPI backends- Know proper async engine and session patterns- Include PostgreSQL-specific features like JSONB- Prevent common async pitfalls (MissingGreenlet, N+1 queries)- Reference SQLModel and SQLAlchemy 2.0 async patternsUse the patterns from LEARNING-SPEC.md and the fetched documentation. The skill creator generates: * `SKILL.md` with triggering description and instructions * `references/` directory for detailed patterns * `scripts/verify.py` for validation **Output:** ✓ Created .claude/skills/relational-db-agent/SKILL.md✓ Created .claude/skills/relational-db-agent/references/async-patterns.md✓ Created .claude/skills/relational-db-agent/scripts/verify.py Step 5: Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-5-test-your-skill "Direct link to Step 5: Test Your Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Verify your skill generates valid code: Using my relational-db-agent skill, create a Task model with:- id as primary key- title as string- status as string with default "pending"- created_at timestamp **Expected output:** from sqlmodel import SQLModel, Fieldfrom datetime import datetimeclass Task(SQLModel, table=True): id: int | None = Field(default=None, primary_key=True) title: str status: str = Field(default="pending") created_at: datetime = Field(default_factory=datetime.utcnow) If the output includes correct imports, `table=True`, and proper Field usage—your skill works. What Happens Next[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#what-happens-next "Direct link to What Happens Next") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Lesson | You Learn | Your Skill Improves | | --- | --- | --- | | L01 | Why agents need structured data | Understands relational vs vector DB | | L02 | Async engine setup | Generates correct engine config | | L03 | Model design | Handles JSONB and constraints | | L04 | Session management | Knows AsyncSession lifecycle | | L05 | CRUD operations | Generates proper async CRUD | | L06 | Relationships | Includes selectinload patterns | | L07 | Transactions | Handles rollback correctly | | L08 | Migrations | Knows Alembic async setup | | L09 | Complete layer | Generates full database layer | Each lesson ends with a "Reflect on Your Skill" section where you test what you learned against your skill and improve it. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Verify Your Skill Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#prompt-1-verify-your-skill-structure "Direct link to Prompt 1: Verify Your Skill Structure") Check my relational-db-agent skill at .claude/skills/relational-db-agent/Does it have:1. SKILL.md with proper YAML frontmatter2. A description starting with "Use when"3. References for async patternsSuggest any improvements to the structure. **What you're learning:** Skill validation—ensuring your skill follows the format that makes it discoverable and usable. ### Prompt 2: Test AsyncSession Generation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#prompt-2-test-asyncsession-generation "Direct link to Prompt 2: Test AsyncSession Generation") Using my relational-db-agent skill, generate a get_session()FastAPI dependency that yields an AsyncSession from an async engine.Include all necessary imports. **What you're learning:** Session management patterns—the foundation of async database operations. ### Prompt 3: Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#prompt-3-identify-gaps "Direct link to Prompt 3: Identify Gaps") I'm about to learn SQLModel async patterns. Based on myrelational-db-agent skill, what database concepts shouldI add to make it more complete? Consider:- What async pitfalls is it missing?- What PostgreSQL features should it know?- What relationship patterns should it include? **What you're learning:** Gap analysis—proactively identifying what your skill needs to become production-ready. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#safety-note "Direct link to Safety Note") Your skill will generate database code. Always review generated code before running it—especially CREATE, DROP, or migration operations. Test in development before production. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#reflect-on-your-skill "Direct link to Reflect on Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You created a `relational-db-agent` skill. Test it now. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#test-your-skill "Direct link to Test Your Skill") Using my relational-db-agent skill, generate a completeasync database setup for a FastAPI app with:- PostgreSQL connection- Engine with connection pooling- get_session dependency ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include `pool_pre_ping=True`? * Did it use `sqlmodel.ext.asyncio.session.AsyncSession`? * Did it structure the dependency correctly with `yield`? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My relational-db-agent skill is missing connection pool configuration.Update it to always include pool_pre_ping=True and configurablepool_size for production deployments. Your skill just got better. This pattern repeats every lesson. Checking access... * [Why Skill-First for Databases?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#why-skill-first-for-databases) * [Step 1: Clone the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-1-clone-the-skills-lab) * [Step 2: Write Your Learning Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-2-write-your-learning-specification) * [Step 3: Fetch Official Documentation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-3-fetch-official-documentation) * [Step 4: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-4-create-your-skill) * [Step 5: Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#step-5-test-your-skill) * [What Happens Next](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#what-happens-next) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#try-with-ai) * [Prompt 1: Verify Your Skill Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#prompt-1-verify-your-skill-structure) * [Prompt 2: Test AsyncSession Generation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#prompt-2-test-asyncsession-generation) * [Prompt 3: Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#prompt-3-identify-gaps) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#safety-note) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/build-your-database-skill#improve-your-skill) --- # Build Your Evals Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You have built agents that generate outputs, make decisions, and take actions. But how do you know if those outputs are actually good? How do you measure whether your agent made the right decision, not just a decision? This is where most developers struggle, and it separates effective AI builders from the rest. Andrew Ng, one of the pioneers of modern AI, identified the critical pattern: "One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." The developers who master evaluation build agents that improve systematically. The developers who skip evaluation build agents that seem to work until they fail in production. This lesson follows the Skill-First pattern. Instead of learning evaluation concepts first and building a skill later, you will create your `agent-evals` skill immediately. Why? Because building the skill forces you to engage with official documentation from the start. Your skill becomes grounded in authoritative sources rather than general knowledge. Every lesson in this chapter will then test and improve your skill, and by the capstone, you will own a production-ready evaluation capability. Why Skill-First for Evals[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#why-skill-first-for-evals "Direct link to Why Skill-First for Evals") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The traditional approach to learning agent evaluation looks like this: 1. Read about evaluation concepts 2. See some examples 3. Maybe build something later 4. Forget most of it within a week The Skill-First approach inverts this: 1. Define what you want to learn (specification) 2. Build a skill structure from official documentation 3. Learn concepts by improving your skill 4. End the chapter owning a valuable asset The difference is ownership. In traditional learning, knowledge lives in your head (temporarily). In Skill-First learning, knowledge lives in a reusable skill that you can apply to every future agent project. For evaluations specifically, this matters more than most topics. Evaluation requires discipline, and discipline requires structure. By encoding evaluation methodology into a skill, you create a forcing function that ensures you actually do the work rather than skipping it when deadlines pressure you. Clone skills-lab Fresh[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#clone-skills-lab-fresh "Direct link to Clone skills-lab Fresh") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Every chapter starts with a clean slate. This prevents confusion from leftover files and ensures reproducible results. Open your terminal and navigate to your workspace directory: cd ~/workspace If you have a previous `skills-lab` directory, remove it: rm -rf skills-lab Clone a fresh copy: git clone https://github.com/panaversity/skills-lab.gitcd skills-lab **Output:** Cloning into 'skills-lab'...remote: Enumerating objects: 156, done.remote: Counting objects: 100% (156/156), done.remote: Compressing objects: 100% (89/89), done.Receiving objects: 100% (156/156), 24.18 KiB | 1.21 MiB/s, done. Verify the directory structure: ls -la **Output:** total 16drwxr-xr-x 6 user staff 192 Dec 30 10:00 .drwxr-xr-x 12 user staff 384 Dec 30 10:00 ..drwxr-xr-x 8 user staff 256 Dec 30 10:00 .git-rw-r--r-- 1 user staff 234 Dec 30 10:00 README.mddrwxr-xr-x 3 user staff 96 Dec 30 10:00 skillsdrwxr-xr-x 2 user staff 64 Dec 30 10:00 specs You now have a clean environment ready for skill development. Write Your LEARNING-SPEC.md[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#write-your-learning-specmd "Direct link to Write Your LEARNING-SPEC.md") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before fetching documentation or building anything, articulate what you want to learn. This specification becomes your learning contract. It defines success criteria that you can measure at the end of the chapter. Create the specification file: mkdir -p specs/agent-evalstouch specs/agent-evals/LEARNING-SPEC.md Open the file in your editor and add: # LEARNING-SPEC: Agent Evaluations## IntentI want to learn how to systematically evaluate AI agent performance so that I can:1. Measure whether my agents make good decisions (not just any decisions)2. Catch quality regressions before they reach production3. Improve agent performance through data-driven iteration## Success CriteriaBy the end of Chapter 77, I will be able to:- [ ] Explain the difference between TDD (testing code) and evals (testing reasoning)- [ ] Design an evaluation dataset with typical, edge, and error cases- [ ] Create graders that turn subjective quality into measurable scores- [ ] Perform error analysis to find which component caused failures- [ ] Set up regression protection that runs on every agent change- [ ] Choose between end-to-end and component-level evals appropriately## Questions to Answer1. How do I define "good" for an agent output when there's no single right answer?2. Why do LLM graders fail with 1-5 scales, and what works instead?3. How do I trace errors back to specific agent components?4. When should I use end-to-end evals vs component-level evals?5. How many test cases do I actually need to start?## Constraints- Focus on framework-agnostic concepts (apply to any SDK)- Start with 10-20 test cases, not 1000- Prioritize quick-and-dirty evals over perfect ones- Use binary criteria instead of numeric scales## Non-Goals- Building a complete evaluation framework from scratch- Learning every evaluation library in detail- Achieving perfect agent performance (iterative improvement instead) **Output:** (No output - file created silently) Verify the file exists: cat specs/agent-evals/LEARNING-SPEC.md | head -20 **Output:** # LEARNING-SPEC: Agent Evaluations## IntentI want to learn how to systematically evaluate AI agent performance so that I can:1. Measure whether my agents make good decisions (not just any decisions)2. Catch quality regressions before they reach production3. Improve agent performance through data-driven iteration## Success CriteriaBy the end of Chapter 77, I will be able to:- [ ] Explain the difference between TDD (testing code) and evals (testing reasoning)- [ ] Design an evaluation dataset with typical, edge, and error cases- [ ] Create graders that turn subjective quality into measurable scores You now have a clear learning contract that defines what success looks like. Create Initial Skill Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#create-initial-skill-structure "Direct link to Create Initial Skill Structure") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ With your learning specification in place, create the skill directory and file: mkdir -p skills/agent-evalstouch skills/agent-evals/SKILL.md Add the initial skill structure. This is a skeleton that you will develop throughout the chapter: ---name: agent-evalsdescription: Design and implement evaluation frameworks for AI agents. Use when testing agent reasoning quality, building graders, performing error analysis, or establishing regression protection.---# Agent Evaluations Skill## Core Thesis"One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." - Andrew Ng## When to ActivateUse this skill when:- Building quality checks for any AI agent- Designing evaluation datasets- Creating graders to define "good" automatically- Performing error analysis to find failure patterns- Setting up regression protection for agent changes## Key Concepts (To Be Developed)### Evals vs TDD| Aspect | TDD (Code Testing) | Evals (Agent Evaluation) ||--------|-------------------|-------------------------|| **Tests** | Does function return correct output? | Did agent make the right decision? || **Outcome** | PASS or FAIL (deterministic) | Scores (probabilistic) || **Analogy** | Testing if calculator works | Testing if student knows WHEN to use multiplication |### Graders (TODO)- Binary criteria over 1-5 scales- LLM-as-Judge patterns- Position bias awareness### Error Analysis (TODO)- Build-Analyze loop- Trace and span terminology- Error classification patterns### Dataset Design (TODO)- Quality over quantity (10-20 cases to start)- Three categories: typical, edge, error- Use real data, not synthetic### Regression Protection (TODO)- Run evals on every change- Baseline comparison- Eval-driven development loop## IntegrationThis skill connects to:- SDK-specific evaluation modules (OpenAI, Claude, Google ADK)- Observability skills for trace analysis- CI/CD skills for automated eval runs---*Status: Initial skeleton - to be developed through Chapter 77* **Output:** (No output - file created silently) Verify the skill structure: cat skills/agent-evals/SKILL.md | head -30 **Output:** ---name: agent-evalsdescription: Design and implement evaluation frameworks for AI agents. Use when testing agent reasoning quality, building graders, performing error analysis, or establishing regression protection.---# Agent Evaluations Skill## Core Thesis"One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." - Andrew Ng## When to ActivateUse this skill when:- Building quality checks for any AI agent- Designing evaluation datasets Verify Your Skill Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#verify-your-skill-works "Direct link to Verify Your Skill Works") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A skill needs to be invokable. Test that your skill loads correctly by checking its structure: # Check YAML frontmatter is validhead -5 skills/agent-evals/SKILL.md **Output:** ---name: agent-evalsdescription: Design and implement evaluation frameworks for AI agents. Use when testing agent reasoning quality, building graders, performing error analysis, or establishing regression protection.--- The skill has valid frontmatter with `name` and `description` fields. This is the minimum requirement for a Claude Code skill. Check the file size to ensure content exists: wc -l skills/agent-evals/SKILL.md **Output:** 62 skills/agent-evals/SKILL.md You now have a working skill skeleton. Each subsequent lesson in this chapter will add content to specific sections, transforming this skeleton into a production-ready evaluation capability. What You Built Today[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#what-you-built-today "Direct link to What You Built Today") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Artifact | Purpose | Location | | --- | --- | --- | | Clean skills-lab | Reproducible starting point | `~/workspace/skills-lab/` | | LEARNING-SPEC.md | Learning contract with success criteria | `specs/agent-evals/LEARNING-SPEC.md` | | SKILL.md skeleton | Initial skill structure to develop | `skills/agent-evals/SKILL.md` | You have not learned evaluation concepts yet. That is intentional. You have built the container that will hold that knowledge. As you progress through this chapter, every concept you learn gets encoded into your skill. By the end, you own something valuable: a reusable evaluation methodology grounded in official documentation and tested through practice. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use Claude Code, Gemini CLI, or your preferred AI tool. These prompts help you verify understanding and begin developing your skill. ### Prompt 1: Validate Your Learning Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#prompt-1-validate-your-learning-specification "Direct link to Prompt 1: Validate Your Learning Specification") Review my LEARNING-SPEC.md for agent evaluations:[Paste your LEARNING-SPEC.md content]Questions:1. Are my success criteria measurable? Which ones are vague?2. What questions am I missing that I should add?3. Do my constraints make sense for a beginner learning evals? **What you're learning:** Specification quality matters. Clear success criteria let you measure progress. Vague criteria like "understand evaluations" cannot be measured. AI helps identify which criteria need sharpening before you invest time learning. ### Prompt 2: Expand Your Skill's "When to Activate" Section[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#prompt-2-expand-your-skills-when-to-activate-section "Direct link to Prompt 2: Expand Your Skill's "When to Activate" Section") I'm building an agent-evals skill. Here's my current "When to Activate" section:Use this skill when:- Building quality checks for any AI agent- Designing evaluation datasets- Creating graders to define "good" automatically- Performing error analysis to find failure patterns- Setting up regression protection for agent changesWhat specific scenarios am I missing? Give me 3-5 additional triggersthat would indicate someone should use this skill. Focus on situationsa developer might not immediately recognize as evaluation problems. **What you're learning:** Skill activation triggers determine when your skill gets used. Comprehensive triggers ensure you apply evaluation methodology in all relevant situations, not just obvious ones. AI suggests scenarios you might overlook. ### Prompt 3: Connect Evals to Your Domain[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#prompt-3-connect-evals-to-your-domain "Direct link to Prompt 3: Connect Evals to Your Domain") I'm learning agent evaluation. My domain is [describe your field:customer support, content generation, data analysis, etc.].Help me think through:1. What would "good" mean for an agent in my domain?2. What are 3 typical cases I should test?3. What are 2 edge cases that might break my agent?4. What's 1 error case where the agent should gracefully fail?Use these to help me draft the "Dataset Design" section of my skill. **What you're learning:** Evaluation is domain-specific. "Good" for a customer support agent differs from "good" for a code review agent. AI helps you translate abstract concepts to your specific context, making your skill immediately applicable. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#safety-note "Direct link to Safety Note") Evaluation skills require judgment about what constitutes "good" agent behavior. AI can suggest patterns and criteria, but you must validate that evaluation frameworks align with your actual requirements. Do not assume AI-suggested grading criteria match your domain's definition of quality. Test evaluation approaches with real examples from your use case before trusting them in production. Checking access... * [Why Skill-First for Evals](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#why-skill-first-for-evals) * [Clone skills-lab Fresh](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#clone-skills-lab-fresh) * [Write Your LEARNING-SPEC.md](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#write-your-learning-specmd) * [Create Initial Skill Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#create-initial-skill-structure) * [Verify Your Skill Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#verify-your-skill-works) * [What You Built Today](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#what-you-built-today) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#try-with-ai) * [Prompt 1: Validate Your Learning Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#prompt-1-validate-your-learning-specification) * [Prompt 2: Expand Your Skill's "When to Activate" Section](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#prompt-2-expand-your-skills-when-to-activate-section) * [Prompt 3: Connect Evals to Your Domain](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#prompt-3-connect-evals-to-your-domain) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/build-your-evals-skill#safety-note) --- # Designing Eval Datasets | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You have decided to build systematic evaluations for your agent. Now comes the question that stops most teams: How many test cases do you need? Your instinct says more is better. A thousand test cases must be more thorough than twenty. Enterprise software has thousands of unit tests. Why would agent evaluation be different? Here is why that instinct is wrong, and what to do instead. The Paralysis Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#the-paralysis-problem "Direct link to The Paralysis Problem") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Teams building agent evaluations often fall into one of two traps: **Trap 1: Perfection paralysis.** "We need a comprehensive dataset before we can start evaluating." The team spends weeks designing taxonomy, debating coverage, building elaborate labeling systems. Meanwhile, the agent ships without any evaluation at all because the "comprehensive" dataset is never finished. **Trap 2: Volume delusion.** "We generated 5,000 synthetic test cases, so our coverage must be good." The team runs evaluations, sees 94% pass rate, ships confidently. Then production failures emerge in patterns the synthetic data never captured. Both traps share the same root cause: confusing quantity with quality. Quality Over Quantity: The Andrew Ng Insight[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#quality-over-quantity-the-andrew-ng-insight "Direct link to Quality Over Quantity: The Andrew Ng Insight") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Andrew Ng, who has advised hundreds of teams building AI systems, offers a counterintuitive directive: > "Quick and dirty evals is fine to get started. 10-20 examples... You don't have to have a thousand examples to get started." Why does this work? Because the bottleneck in agent improvement is not data volume. It is understanding WHY things fail. **Ten thoughtful cases tell you:** * Which specific scenarios break your agent * What patterns of failure repeat * Where to focus improvement efforts **One thousand random cases tell you:** * A pass rate number * Very little about WHY failures occur * Almost nothing actionable about how to improve The purpose of evaluation is not to generate a score. It is to drive improvement. A small dataset that reveals failure patterns is infinitely more valuable than a large dataset that hides them in noise. The Three-Category Framework[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#the-three-category-framework "Direct link to The Three-Category Framework") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every eval dataset should cover three categories with different purposes: | Category | Count | Purpose | What it tests | | --- | --- | --- | --- | | **Typical** | 10 | Common use cases | Does the agent handle normal requests well? | | **Edge** | 5 | Unusual but valid | Does the agent handle uncommon situations gracefully? | | **Error** | 5 | Should fail gracefully | Does the agent recognize when it cannot help and respond appropriately? | ### Typical Cases (10)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#typical-cases-10 "Direct link to Typical Cases (10)") These are the bread-and-butter scenarios your agent was designed for. If it fails these, nothing else matters. **Characteristics:** * Represent the most common 80% of real usage * Clear user intent, reasonable complexity * Success criteria are well-defined * These should have HIGH pass rates (90%+) **Example for Task API agent:** "Create a task called 'Buy groceries' for tomorrow""Show me my tasks for this week""Mark the dentist appointment as done""What's my highest priority task?" **Why 10 cases?** Not arbitrary. Ten typical cases cover the core user journeys without redundancy. If you find yourself writing similar cases repeatedly, you have enough. ### Edge Cases (5)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#edge-cases-5 "Direct link to Edge Cases (5)") These are unusual but valid requests. Users do weird things. Your agent should handle them. **Characteristics:** * Uncommon but legitimate use cases * May require the agent to make judgment calls * Success might mean "reasonable response" not "perfect response" * Acceptable pass rate is lower (70-80%) **Example for Task API agent:** "Create a task with emoji title: '🎉 Birthday party planning'""Schedule something for 'next week sometime'""Add a task that repeats every other Thursday except holidays""Create 15 tasks at once from this list: ..." **Why 5 cases?** Edge cases have diminishing returns. The first few reveal whether your agent handles ambiguity. More than five typically find the same class of issues repeatedly. ### Error Cases (5)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#error-cases-5 "Direct link to Error Cases (5)") These are requests where your agent SHOULD fail gracefully. Not every request is valid. A good agent recognizes its limitations. **Characteristics:** * Requests outside the agent's capabilities * Malformed or nonsensical inputs * Should NOT succeed (success here is actually failure) * Tests graceful degradation **Example for Task API agent:** "Delete all my tasks" (if agent lacks delete capability)"What's the weather tomorrow?" (outside task domain)"asdfghjkl" (nonsense input)"Create task for yesterday" (impossible request)"Hack into the mainframe" (inappropriate request) **Why 5 cases?** Error handling has clear patterns. A handful of cases verify the agent recognizes different failure modes: out of scope, malformed, impossible, inappropriate. Using REAL Data[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#using-real-data "Direct link to Using REAL Data") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here is where most evaluation efforts go wrong: synthetic data. **Synthetic data** is test cases you imagine users might submit. It is clean, well-formed, and represents how you THINK users behave. **Real data** is actual user queries from production logs, support tickets, and customer feedback. It is messy, unexpected, and represents how users ACTUALLY behave. The gap between synthetic and real data is enormous: | Synthetic Data | Real Data | | --- | --- | | "Create a task for tomorrow" | "can u remind me 2 call mom tmrw" | | "What tasks are due this week?" | "whats due" | | "Mark task as complete" | "done with that thing I added yesterday" | | Clean grammar, clear intent | Typos, abbreviations, assumed context | ### Where to Find Real Data[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#where-to-find-real-data "Direct link to Where to Find Real Data") Your production system already contains gold for evaluation: **1\. Application logs** * Actual user inputs and agent responses * Filter for sessions with negative feedback or repeated attempts * Look for queries that led to support tickets **2\. Support tickets** * User complaints describe exactly what failed * "I asked for X but got Y" is a perfect eval case * Support volume indicates which failures matter most **3\. User feedback** * Thumbs down, negative ratings, explicit complaints * Each piece of negative feedback is a potential eval case * Positive feedback can validate typical cases **4\. Session recordings** * Multi-turn conversations where users gave up * Indicates where agent behavior frustrated users * Reveals context-dependent failures ### Mining Real Data: A Process[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#mining-real-data-a-process "Direct link to Mining Real Data: A Process") 1. Export last 30 days of user queries | v2. Filter for signals: - Negative feedback - Multiple attempts (retry pattern) - Support ticket creation - Abandoned sessions | v3. Sample 50-100 candidates | v4. Classify into categories: - Typical (should work) - Edge (unusual but valid) - Error (should fail gracefully) | v5. Select 20 diverse cases Eval Case Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#eval-case-structure "Direct link to Eval Case Structure") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every eval case needs three components: { "id": "unique-identifier", "category": "typical | edge | error", "input": { "user_message": "The actual user input", "context": { "prior_messages": [], "user_state": {} } }, "expected_behavior": { "should_succeed": true, "criteria": [ "criterion_1", "criterion_2" ], "expected_output_pattern": "optional regex or template" }, "rationale": "Why this case matters and what it tests"} ### Example: Complete Eval Case[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#example-complete-eval-case "Direct link to Example: Complete Eval Case") { "id": "typical-003", "category": "typical", "input": { "user_message": "remind me to call mom tomorrow at 3pm", "context": { "prior_messages": [], "user_state": { "timezone": "America/New_York" } } }, "expected_behavior": { "should_succeed": true, "criteria": [ "Creates a task (not a calendar event)", "Title contains 'call mom' or equivalent", "Due date is tomorrow", "Time is 3pm in user's timezone", "Reminder is set (not just due date)" ], "expected_output_pattern": null }, "rationale": "Tests natural language time parsing and task vs reminder distinction - common user pattern from production logs"} ### The Rationale Field[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#the-rationale-field "Direct link to The Rationale Field") The `rationale` field is often skipped but essential. It captures: * **Why** this case was included * **What** specific behavior it tests * **Where** it came from (production data reference) Without rationale, six months later you cannot remember why a case exists or whether it still matters. With rationale, your eval dataset is self-documenting. When to Grow Your Dataset[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#when-to-grow-your-dataset "Direct link to When to Grow Your Dataset") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Start with 20 cases. But when do you need more? **Signal that you need more cases:** You observe a production failure that NO eval case covers | vYou add eval cases that capture this failure pattern | vYou verify the new cases fail (confirming the bug) | vYou fix the agent | vNew cases now pass (confirming the fix) This is the eval-driven development loop. Your dataset grows organically as you discover gaps. **Do NOT add cases because:** * "We should have more coverage" (vague, not actionable) * "Competitive products have 10,000 cases" (irrelevant to YOUR agent) * "We haven't updated the dataset in a while" (age is not a problem) **DO add cases when:** * Production failure reveals an uncovered pattern * New feature requires new test scenarios * Error analysis shows a category is under-represented * User feedback identifies a failure mode Exercise: Design a 20-Case Dataset for Task API[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#exercise-design-a-20-case-dataset-for-task-api "Direct link to Exercise: Design a 20-Case Dataset for Task API") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Task API agent helps users manage tasks through natural language. Design a complete eval dataset. **Step 1: List typical cases (10)** Think about the core user journeys: * Creating tasks * Viewing tasks * Updating tasks * Querying tasks For each, write a realistic user input (not synthetic "Create a task" commands). **Step 2: List edge cases (5)** Think about unusual but valid scenarios: * Ambiguous time references * Special characters in titles * Multiple actions in one request * Missing information that requires inference **Step 3: List error cases (5)** Think about what should fail gracefully: * Requests outside task management domain * Impossible requests (past dates, contradictory requirements) * Missing required information with no reasonable default * Potentially harmful requests **Step 4: Add structure** For each case, fill in: * Input (user message + any context) * Expected behavior (criteria for success/graceful failure) * Rationale (why this case matters) **Sample answer structure:** | ID | Category | Input | Key Criteria | Rationale | | --- | --- | --- | --- | --- | | T-01 | Typical | "remind me to pick up dry cleaning tomorrow" | Creates task, tomorrow due date, reminder set | Common natural language pattern | | T-02 | Typical | "what do I have due this week" | Lists tasks, filters by date range | Basic query pattern | | E-01 | Edge | "add groceries and also schedule dentist for next tuesday" | Creates TWO tasks correctly | Multi-action request | | E-02 | Edge | "task for 'meeting with @john re: $$$ deal'" | Handles special chars in title | Real user data often has special chars | | R-01 | Error | "what's the weather" | Politely declines, stays in domain | Out of scope, should not attempt | | R-02 | Error | "delete all my tasks forever" | Does not delete (no capability) | Tests capability boundaries | Common Mistakes to Avoid[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#common-mistakes-to-avoid "Direct link to Common Mistakes to Avoid") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Mistake 1: All synthetic data** Your imagination of user behavior differs from reality. Mine production data. **Mistake 2: Only happy path** A dataset of only typical cases tells you nothing about robustness. Include edge and error cases. **Mistake 3: No rationale** Cases without rationale become mysterious artifacts. Document WHY each case exists. **Mistake 4: Waiting for completeness** You will never have "complete" coverage. Start with 20 cases. Add more when production failures reveal gaps. **Mistake 5: Ignoring error cases** How your agent fails matters as much as how it succeeds. Graceful failure is a feature. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Extract Cases from User Feedback[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#prompt-1-extract-cases-from-user-feedback "Direct link to Prompt 1: Extract Cases from User Feedback") I'm building eval cases for my [describe agent] from real user feedback.Here are some recent complaints from users:[Paste 5-10 real user complaints or negative feedback]Help me turn each complaint into a structured eval case with:- Input (what the user likely said)- Expected behavior (what should have happened)- Category (typical, edge, or error)- Rationale (what this case tests) **What you're learning**: Transforming unstructured user feedback into actionable evaluation cases. Real complaints reveal the gaps between your mental model and user reality. ### Prompt 2: Generate Edge Cases from Typical Cases[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#prompt-2-generate-edge-cases-from-typical-cases "Direct link to Prompt 2: Generate Edge Cases from Typical Cases") I have these typical eval cases for my Task API agent:[List your 10 typical cases]For each typical case, suggest ONE edge case variant that tests the samecore functionality but with added complexity or ambiguity. Explain whatadditional behavior the edge case tests. **What you're learning**: Systematic edge case generation. Starting from solid typical cases and adding complexity is more effective than imagining edge cases from scratch. ### Prompt 3: Audit Your Dataset Balance[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#prompt-3-audit-your-dataset-balance "Direct link to Prompt 3: Audit Your Dataset Balance") Here's my current eval dataset:[Paste your 20 cases]Audit this dataset for:1. Category balance: Do I have appropriate typical/edge/error distribution?2. Feature coverage: What agent capabilities are over/under-represented?3. Diversity: Are cases too similar to each other?4. Real vs synthetic: Which cases seem synthetic vs derived from real data?Suggest 3 specific improvements to make this dataset more useful. **What you're learning**: Critical evaluation of your own evaluation system. The meta-skill of auditing eval quality prevents the common trap of low-quality evals producing misleading confidence. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#safety-note "Direct link to Safety Note") When extracting eval cases from production data, ensure you handle user privacy appropriately. Remove personally identifiable information (PII) from user queries before including them in eval datasets. Never include actual user names, emails, or sensitive content in test cases. Anonymize and generalize while preserving the linguistic patterns that make real data valuable. Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After completing this lesson, update your `agent-evals` skill with dataset design patterns: **Add to your skill's Dataset Design section:** ### Dataset Design**Quality Over Quantity**: Start with 10-20 cases, not 1000.**Three-Category Framework**:| Category | Count | Purpose ||----------|-------|---------|| Typical | 10 | Common use cases - should have 90%+ pass rate || Edge | 5 | Unusual but valid - acceptable 70-80% pass rate || Error | 5 | Should fail gracefully - tests boundaries |**Real Data Over Synthetic**: Mine production logs, support tickets,user feedback for authentic test cases.**Growth Trigger**: Add cases when production failures reveal gaps,not based on arbitrary coverage targets.**Case Structure**: Every case needs input, expected_behavior, rationale. **Key insight to encode**: The purpose of evaluation is improvement, not scoring. Twenty cases that reveal WHY things fail enable faster improvement than a thousand cases that only report WHAT percentage passes. Checking access... * [The Paralysis Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#the-paralysis-problem) * [Quality Over Quantity: The Andrew Ng Insight](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#quality-over-quantity-the-andrew-ng-insight) * [The Three-Category Framework](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#the-three-category-framework) * [Typical Cases (10)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#typical-cases-10) * [Edge Cases (5)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#edge-cases-5) * [Error Cases (5)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#error-cases-5) * [Using REAL Data](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#using-real-data) * [Where to Find Real Data](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#where-to-find-real-data) * [Mining Real Data: A Process](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#mining-real-data-a-process) * [Eval Case Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#eval-case-structure) * [Example: Complete Eval Case](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#example-complete-eval-case) * [The Rationale Field](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#the-rationale-field) * [When to Grow Your Dataset](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#when-to-grow-your-dataset) * [Exercise: Design a 20-Case Dataset for Task API](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#exercise-design-a-20-case-dataset-for-task-api) * [Common Mistakes to Avoid](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#common-mistakes-to-avoid) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#try-with-ai) * [Prompt 1: Extract Cases from User Feedback](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#prompt-1-extract-cases-from-user-feedback) * [Prompt 2: Generate Edge Cases from Typical Cases](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#prompt-2-generate-edge-cases-from-typical-cases) * [Prompt 3: Audit Your Dataset Balance](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#prompt-3-audit-your-dataset-balance) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#safety-note) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/designing-eval-datasets#reflect-on-your-skill) --- # Memory for Claude Code | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've built agents with memory from scratch. Now experience memory from the other side—as the user of a memory-augmented assistant. The [claude-mem](https://github.com/thedotmack/claude-mem) plugin gives Claude Code persistent memory across sessions. It remembers your preferences, project context, and past decisions, making every conversation build on what came before. By the end of this lesson, you'll have a memory-augmented Claude Code that knows your development style. Installation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#installation "Direct link to Installation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### From Plugin Marketplace[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#from-plugin-marketplace "Direct link to From Plugin Marketplace") # In Claude Code terminal/plugin marketplace add thedotmack/claude-mem Or install directly: /plugin install claude-mem ### Verify Installation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#verify-installation "Direct link to Verify Installation") After installation, restart Claude Code and check: /plugins **Output:** Installed plugins:• claude-mem (v1.0.0) - Active Hooks: SessionStart, UserPromptSubmit, PostToolUse, Stop, SessionEnd MCP Tools: search, timeline, get_observations ### Configuration Location[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#configuration-location "Direct link to Configuration Location") The plugin stores configuration and data in: ~/.claude-mem/├── settings.json # Configuration├── memory.db # SQLite storage└── chroma/ # Vector embeddings How It Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#how-it-works "Direct link to How It Works") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Claude-mem uses **lifecycle hooks** to capture and retrieve memories automatically: ┌─────────────────────────────────────────────────────────────┐│ CLAUDE-MEM ARCHITECTURE │├─────────────────────────────────────────────────────────────┤│ ││ ┌─────────────────────────────────────────────────────┐ ││ │ Claude Code │ ││ │ │ ││ │ User ──→ Prompt ──→ Tools ──→ Response │ ││ │ │ │ │ │ │ ││ └─────┼────────┼─────────┼──────────┼─────────────────┘ ││ │ │ │ │ ││ ▼ ▼ ▼ ▼ ││ ┌─────────────────────────────────────────────────────┐ ││ │ Lifecycle Hooks │ ││ │ │ ││ │ SessionStart → Retrieve relevant context │ ││ │ UserPromptSubmit → Capture user intent │ ││ │ PostToolUse → Observe tool results │ ││ │ Stop → Extract session insights │ ││ │ SessionEnd → Finalize and store │ ││ │ │ ││ └──────────────────────────┬──────────────────────────┘ ││ │ ││ ┌──────────────────────────▼──────────────────────────┐ ││ │ Storage Layer │ ││ │ │ ││ │ SQLite (structured) + Chroma (semantic search) │ ││ │ │ ││ └─────────────────────────────────────────────────────┘ ││ │└─────────────────────────────────────────────────────────────┘ ### Hook Behavior[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#hook-behavior "Direct link to Hook Behavior") | Hook | When | What It Captures | | --- | --- | --- | | `SessionStart` | Conversation begins | Retrieves relevant past context | | `UserPromptSubmit` | User sends message | User intent and questions | | `PostToolUse` | After each tool | Tool results and patterns | | `Stop` | Response complete | Session insights and decisions | | `SessionEnd` | Session closes | Finalizes and indexes memories | Privacy Controls[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#privacy-controls "Direct link to Privacy Controls") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### The `` Tag[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-private-tag "Direct link to the-private-tag") Anything wrapped in `` tags is **never stored**: Can you help me debug this API call?API_KEY=sk-1234567890abcdefDB_PASSWORD=supersecret123The error is a 401 Unauthorized. Claude-mem will store: "User asked for help debugging API call, getting 401 Unauthorized error." It will **not** store: API keys, passwords, or anything between `` tags. ### Configuration Settings[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#configuration-settings "Direct link to Configuration Settings") Edit `~/.claude-mem/settings.json`: { "privacy": { "enabled": true, "excluded_patterns": [ "password", "api_key", "api-key", "secret", "token", "bearer", "credential", "private_key", "ssh_key", "-----BEGIN" ], "excluded_paths": [ "**/.env*", "**/secrets/**", "**/.ssh/**", "**/credentials/**" ] }, "storage": { "path": "~/.claude-mem", "max_observations": 10000, "retention_days": 365 }, "retrieval": { "max_results": 10, "relevance_threshold": 0.5, "recency_weight": 0.3 }} ### Privacy Audit[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#privacy-audit "Direct link to Privacy Audit") Run a privacy check on stored memories: # In terminal (not Claude Code)cd ~/.claude-memsqlite3 memory.db "SELECT content FROM observations WHERE content LIKE '%password%' OR content LIKE '%api_key%' OR content LIKE '%secret%';" If this returns results, review and clean up: sqlite3 memory.db "DELETE FROM observations WHERE content LIKE '%password%';" Web UI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#web-ui "Direct link to Web UI") ----------------------------------------------------------------------------------------------------------------------------------------------------- Claude-mem includes a web interface for exploring your memories. ### Starting the Web UI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#starting-the-web-ui "Direct link to Starting the Web UI") # The server starts automatically with the plugin# Access at:open http://localhost:37777 ### Web UI Features[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#web-ui-features "Direct link to Web UI Features") **Timeline View** ┌─────────────────────────────────────────────────────────────┐│ Memory Timeline 🔍 Search │├─────────────────────────────────────────────────────────────┤│ ││ 📅 Today ││ ├── 10:45 AM - Working on Phoenix authentication ││ ├── 10:32 AM - Prefers pytest over unittest ││ └── 10:15 AM - Using FastAPI + SQLModel stack ││ ││ 📅 Yesterday ││ ├── 4:30 PM - Fixed connection pool issue ││ ├── 2:15 PM - Decided to use Qdrant for vectors ││ └── 11:00 AM - Started Phoenix project ││ ││ 📅 Last Week ││ └── ... ││ │└─────────────────────────────────────────────────────────────┘ **Search View** Enter natural language queries: * "What do I prefer for testing?" * "Phoenix project context" * "Past debugging sessions" **Category Filters** Filter memories by: * Preferences * Projects * Decisions * Errors * Patterns **Memory Management** * Delete individual memories * Bulk delete by date range * Export memories as JSON MCP Tools[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#mcp-tools "Direct link to MCP Tools") -------------------------------------------------------------------------------------------------------------------------------------------------------------- Claude-mem adds MCP tools for memory operations within Claude Code. ### Available Tools[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#available-tools "Direct link to Available Tools") **`search`** - Semantic memory search Use the search tool to find memories about Python testing preferences **`timeline`** - Recent observations Show me my memory timeline from the last 7 days **`get_observations`** - Retrieve specific observations Get observations about the Phoenix project ### Using MCP Tools in Conversation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#using-mcp-tools-in-conversation "Direct link to Using MCP Tools in Conversation") **Example 1: Context Retrieval** You: What have I been working on lately?Claude: [Uses timeline tool]Based on your recent activity:- This week: Phoenix project authentication module- Focus: FastAPI endpoints and SQLModel integration- Testing: Using pytest with async fixturesYou mentioned yesterday that the connection pool issue wascaused by missing cleanup in the context manager. **Example 2: Preference Lookup** You: How do I usually structure my models?Claude: [Uses search tool with "model structure preferences"]From our past conversations, you prefer:- SQLModel for FastAPI projects (combines SQLAlchemy + Pydantic)- Repository pattern for data access- Separate Base classes from Table classesHere's the pattern you've used before:[Shows code from past context] ### The `mem-search` Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-mem-search-skill "Direct link to the-mem-search-skill") Claude-mem includes a built-in skill for natural language memory queries: /mem-search testing frameworks I've usedResults:1. [2025-01-20] Uses pytest, prefers fixtures over setup/teardown2. [2025-01-15] Tried pytest-asyncio for async test support3. [2025-01-10] Mentioned avoiding unittest in favor of pytest Experiencing Memory-Augmented Claude Code[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#experiencing-memory-augmented-claude-code "Direct link to Experiencing Memory-Augmented Claude Code") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### First Session[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#first-session "Direct link to First Session") You: Hi, I'm starting a new Python project using FastAPI.Claude: Great! I'll remember that you're working with FastAPI.What kind of project is it?You: It's a task management API. I prefer using SQLModel and pytest.Claude: Perfect! I've noted your preferences:- Framework: FastAPI- ORM: SQLModel (nice choice - combines SQLAlchemy with Pydantic)- Testing: pytestWant me to help set up the project structure? ### Second Session (Next Day)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#second-session-next-day "Direct link to Second Session (Next Day)") You: Let's continue working on the projectClaude: Welcome back! I remember you're building a task managementAPI with FastAPI + SQLModel. Yesterday we discussed the projectstructure. Where would you like to pick up?You: I need to add user authenticationClaude: For your FastAPI project, I'd suggest using python-josefor JWT tokens - it works well with SQLModel. Based on yourpreference for clean architecture, here's a pattern:[Shows auth structure consistent with past decisions] ### The Memory Difference[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-memory-difference "Direct link to The Memory Difference") | Without Memory | With Memory | | --- | --- | | "What framework?" | "Your FastAPI project..." | | Generic examples | Examples using your stack | | Repeated explanations | Builds on past context | | Starting fresh | Continuous relationship | Workflow Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#workflow-patterns "Direct link to Workflow Patterns") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Pattern 1: Project Onboarding[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-1-project-onboarding "Direct link to Pattern 1: Project Onboarding") When starting a new project, explicitly set context: I'm starting a new project called "PaymentGateway":- Stack: FastAPI, SQLModel, Redis for caching- Architecture: Hexagonal with ports and adapters- Testing: pytest with 80% coverage target- Deployment: Docker + KubernetesRemember this context for our future conversations. ### Pattern 2: Decision Documentation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-2-decision-documentation "Direct link to Pattern 2: Decision Documentation") When making important decisions: We decided to use Stripe over PayPal for payments because:- Better API documentation- Native Python SDK- Webhook reliabilityPlease remember this decision and the reasoning. ### Pattern 3: Error Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-3-error-patterns "Direct link to Pattern 3: Error Patterns") After debugging: Found the issue: SQLModel relationship lazy loading fails in async.Solution: Use selectinload() for async relationship queries.Remember this pattern - it's likely to come up again. ### Pattern 4: Privacy-Aware Sharing[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-4-privacy-aware-sharing "Direct link to Pattern 4: Privacy-Aware Sharing") When sharing sensitive context: The payment integration uses:STRIPE_SECRET_KEY=sk_live_...STRIPE_WEBHOOK_SECRET=whsec_...The webhook endpoint is /api/webhooks/stripe What Makes Memory Feel Right[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#what-makes-memory-feel-right "Direct link to What Makes Memory Feel Right") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After using memory-augmented Claude Code, these patterns create value: ### ✅ What Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#-what-works "Direct link to ✅ What Works") **Preference Persistence** Week 1: "I prefer async/await over .then() chains"Week 4: Claude automatically uses async/await in suggestions **Project Context** "The Phoenix project uses microservices with Kafka"Later: Claude suggests Kafka patterns for related features **Error Memory** "That connection pool bug was from missing cleanup"Later: Claude proactively mentions cleanup patterns ### ❌ What Feels Intrusive[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#-what-feels-intrusive "Direct link to ❌ What Feels Intrusive") **Over-remembering** ❌ "You mentioned liking pizza last month" → Not relevant to coding **Stale Context** ❌ "Based on your preference for React..." → You've since switched to Vue (update preferences!) **Wrong Confidence** ❌ "You always use X pattern" → You used it once (memory conflates frequency) ### The Balance[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-balance "Direct link to The Balance") Good memory feels like working with a colleague who knows your style. Bad memory feels like surveillance. The difference: **relevance** and **recency**. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these prompts to explore memory-augmented development. ### Prompt 1: Configure Your Memory Profile[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#prompt-1-configure-your-memory-profile "Direct link to Prompt 1: Configure Your Memory Profile") I just installed claude-mem and want to set up my memory profile.Help me create initial memories for:1. My development preferences: - Languages I use most - Frameworks I prefer - Testing approach - Code style preferences2. Current project context: - What I'm building - Architecture decisions - Key constraints3. Privacy settings I should configure: - Patterns to exclude - File paths to ignoreWalk me through each category and suggest specific memories to store. **What you're learning:** Explicit initial configuration sets the foundation for useful memory. Random accumulation leads to noise. ### Prompt 2: Memory Maintenance Routine[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#prompt-2-memory-maintenance-routine "Direct link to Prompt 2: Memory Maintenance Routine") I've been using claude-mem for 3 months and want to do maintenance.Design a monthly memory hygiene routine:1. How to identify stale memories (outdated preferences, old projects)2. How to find and remove potentially sensitive stored data3. How to consolidate redundant memories4. How to update preferences that have changed5. A checklist I can follow each monthInclude the actual commands and queries I should run. **What you're learning:** Memory systems need maintenance. Without periodic cleanup, they accumulate noise that degrades retrieval quality. ### Prompt 3: Project Memory Template[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#prompt-3-project-memory-template "Direct link to Prompt 3: Project Memory Template") I'm about to start a new project. Create a memory template that I canfill out at project start to give Claude Code the right context:Template should include:1. Project identity (name, type, goals)2. Technical stack (languages, frameworks, databases)3. Architecture decisions and rationale4. Coding standards and patterns to follow5. Testing requirements6. Known constraints or challengesFormat it so I can copy-paste and fill in the blanks, then sharewith Claude Code to establish project memory. **What you're learning:** Structured project onboarding creates better memory than organic accumulation. Templates ensure consistent, useful context. What You've Learned[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#what-youve-learned "Direct link to What You've Learned") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This chapter took you from understanding why agents need memory to using memory-augmented tools: | Lesson | Key Concept | | --- | --- | | L01 | Context window problem and stateless vs stateful | | L02 | Five memory types: conversation, working, episodic, semantic | | L03 | Relevance scoring and privacy-compliant forgetting | | L04 | Retrieval strategies: recency, relevance, entity, hybrid | | L05 | Context window management and summarization chains | | L06 | Mem0 SDK implementation for Task API | | L07 | Production patterns: injection, retrieval, conflicts | | L08 | Building complete memory-augmented agents | | L09 | Memory for Claude Code with claude-mem plugin | You now understand memory from both sides—as the builder and as the user. That dual perspective is essential for creating agents that people actually want to use. Next Steps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#next-steps "Direct link to Next Steps") ----------------------------------------------------------------------------------------------------------------------------------------------------------------- With memory in place, your agents can maintain context across sessions. The next chapter explores how to evaluate whether your agents actually work—testing strategies for AI systems that don't have deterministic outputs. Checking access... * [Installation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#installation) * [From Plugin Marketplace](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#from-plugin-marketplace) * [Verify Installation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#verify-installation) * [Configuration Location](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#configuration-location) * [How It Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#how-it-works) * [Hook Behavior](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#hook-behavior) * [Privacy Controls](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#privacy-controls) * [The `` Tag](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-private-tag) * [Configuration Settings](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#configuration-settings) * [Privacy Audit](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#privacy-audit) * [Web UI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#web-ui) * [Starting the Web UI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#starting-the-web-ui) * [Web UI Features](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#web-ui-features) * [MCP Tools](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#mcp-tools) * [Available Tools](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#available-tools) * [Using MCP Tools in Conversation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#using-mcp-tools-in-conversation) * [The `mem-search` Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-mem-search-skill) * [Experiencing Memory-Augmented Claude Code](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#experiencing-memory-augmented-claude-code) * [First Session](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#first-session) * [Second Session (Next Day)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#second-session-next-day) * [The Memory Difference](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-memory-difference) * [Workflow Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#workflow-patterns) * [Pattern 1: Project Onboarding](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-1-project-onboarding) * [Pattern 2: Decision Documentation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-2-decision-documentation) * [Pattern 3: Error Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-3-error-patterns) * [Pattern 4: Privacy-Aware Sharing](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#pattern-4-privacy-aware-sharing) * [What Makes Memory Feel Right](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#what-makes-memory-feel-right) * [✅ What Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#-what-works) * [❌ What Feels Intrusive](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#-what-feels-intrusive) * [The Balance](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#the-balance) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#try-with-ai) * [Prompt 1: Configure Your Memory Profile](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#prompt-1-configure-your-memory-profile) * [Prompt 2: Memory Maintenance Routine](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#prompt-2-memory-maintenance-routine) * [Prompt 3: Project Memory Template](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#prompt-3-project-memory-template) * [What You've Learned](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#what-youve-learned) * [Next Steps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-for-claude-code#next-steps) --- # Context Window Management | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Every LLM has a limit—a maximum number of tokens it can process at once. This is the **context window**, and managing it effectively is one of the most important skills in building memory-augmented agents. The context window seems large. GPT-4 offers 128,000 tokens. Claude offers 200,000 tokens. That's roughly 100,000 words—a full novel. But in practice, you'll find it fills up fast. System prompts, conversation history, retrieved memories, tools definitions—they all compete for space. This lesson teaches you how to manage that space wisely: where to inject memories, how to compress old information, and when to summarize versus retrieve in full. Understanding Context Windows[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#understanding-context-windows "Direct link to Understanding Context Windows") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Size Comparison[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#size-comparison "Direct link to Size Comparison") | Model | Context Window | Approximate Words | | --- | --- | --- | | GPT-4o | 128,000 tokens | ~100,000 words | | Claude 3.5 Sonnet | 200,000 tokens | ~150,000 words | | GPT-4o-mini | 128,000 tokens | ~100,000 words | | Gemini 1.5 Pro | 1,000,000 tokens | ~750,000 words | ### Why Large Isn't Enough[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#why-large-isnt-enough "Direct link to Why Large Isn't Enough") Even with 128k tokens, you face constraints: Context Window Allocation (128k tokens)├── System Prompt (instructions, persona) 3,000 tokens├── Tool Definitions (function calling) 2,000 tokens├── Conversation History (recent messages) 10,000 tokens├── Retrieved Memories 4,000 tokens├── Current User Message 1,000 tokens├── Reserved for Response 4,000 tokens└── TOTAL USED: 24,000 tokensProblems:1. Cost: You pay per token. 100k tokens x $0.01/1k = $1 per call2. Latency: More tokens = slower response (roughly linear)3. Attention: LLMs struggle with very long contexts (lost in middle) ### The "Lost in the Middle" Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#the-lost-in-the-middle-problem "Direct link to The "Lost in the Middle" Problem") Research shows LLMs attend less to information in the middle of long contexts. Information at the beginning and end gets more attention. Implication: Put your most important memories at the beginning or end of the injected context, not buried in the middle. Memory Injection Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#memory-injection-strategies "Direct link to Memory Injection Strategies") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Where you place memories in the prompt affects how the LLM uses them. ### 1\. Pre-Prompt Injection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#1-pre-prompt-injection "Direct link to 1. Pre-Prompt Injection") Inject memories **before** the user's message, typically after the system prompt. def build_prompt_with_memory(system_prompt: str, memories: list, user_message: str) -> str: """Pre-prompt memory injection.""" memory_context = format_memories(memories) return f"""{system_prompt}## Relevant Context from Previous Interactions{memory_context}## Current ConversationUser: {user_message}""" **When to use:** Most common approach. Works well for background context. ### 2\. Mid-Prompt Injection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#2-mid-prompt-injection "Direct link to 2. Mid-Prompt Injection") Inject memories **after** the user's message but **before** the assistant response. def build_prompt_mid_injection(system_prompt: str, user_message: str, memories: list) -> str: """Mid-prompt memory injection.""" memory_context = format_memories(memories) return f"""{system_prompt}User: {user_message}[System note: Relevant context for responding:{memory_context}]""" **When to use:** Query-specific context that shouldn't influence other parts of the conversation. ### 3\. Dynamic Injection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#3-dynamic-injection "Direct link to 3. Dynamic Injection") Retrieve and inject memories **during** reasoning, based on what the LLM discovers it needs. async def dynamic_retrieval_agent(user_message: str, user_id: str): """Agent that retrieves memories on-demand during reasoning.""" # Tool that agent can call to retrieve memories @function_tool async def recall_memory(query: str) -> str: """Search memories for relevant context.""" results = memory.search(query, filters={"user_id": user_id}, limit=3) return format_memories(results['results']) agent = Agent( name="MemoryAgent", instructions="You can recall memories about the user when needed.", tools=[recall_memory] ) return await Runner.run(agent, user_message) **When to use:** Complex queries where the agent needs to discover what context it needs. Summarization Chains[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#summarization-chains "Direct link to Summarization Chains") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Old detailed memories consume tokens. **Summarization chains** compress them hierarchically. ### The Hierarchy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#the-hierarchy "Direct link to The Hierarchy") Detailed Events (hours old) ↓ (daily summarization)Daily Summaries ↓ (weekly summarization)Weekly Summaries ↓ (monthly summarization)Monthly Summaries ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#implementation "Direct link to Implementation") async def run_summarization_chain(user_id: str): """Hierarchical summarization of old memories.""" # Daily: Summarize events older than 24 hours await summarize_tier( user_id=user_id, older_than_hours=24, source_type="event", target_type="daily_summary" ) # Weekly: Summarize daily summaries older than 7 days await summarize_tier( user_id=user_id, older_than_days=7, source_type="daily_summary", target_type="weekly_summary" ) # Monthly: Summarize weekly summaries older than 30 days await summarize_tier( user_id=user_id, older_than_days=30, source_type="weekly_summary", target_type="monthly_summary" )async def summarize_tier(user_id, older_than_days=None, older_than_hours=None, source_type=None, target_type=None): """Summarize memories at one tier into the next.""" # Calculate cutoff if older_than_hours: cutoff = datetime.now() - timedelta(hours=older_than_hours) else: cutoff = datetime.now() - timedelta(days=older_than_days) # Get memories to summarize old_memories = memory.search( query="", filters={ "user_id": user_id, "type": source_type, "created_at": {"lte": cutoff.isoformat()} } )['results'] if len(old_memories) < 3: # Don't summarize small groups return # Group by date/week/month grouped = group_by_period(old_memories, target_type) for period, memories in grouped.items(): # Generate summary using LLM summary_prompt = f""" Summarize these memories into a single concise paragraph: {format_memories(memories)} Focus on: key events, decisions, patterns, and outcomes. """ summary = await llm.generate(summary_prompt) # Store summary memory.add([{ "role": "system", "content": summary }], user_id=user_id, metadata={ "type": target_type, "period": period, "source_count": len(memories) }) # Delete original memories for mem in memories: memory.delete(mem['id']) ### Example[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#example "Direct link to Example") **Before (10 daily events):** - Jan 15: Created task "Review PR #123"- Jan 15: Completed task "Review PR #123"- Jan 16: Created task "Fix login bug"- Jan 16: Updated task priority to high- Jan 17: Completed "Fix login bug"- Jan 18: Created 3 tasks for Phoenix project- Jan 19: Completed 2 Phoenix tasks- Jan 20: Had meeting about deadline- Jan 20: Extended deadline to Feb 15- Jan 21: Started working on auth migration **After (1 weekly summary):** Week of Jan 15-21: User completed 4 tasks including PR review and login bug fix.Created multiple Phoenix project tasks. Had planning meeting that resulted indeadline extension to Feb 15. Began auth migration work. When to Retrieve vs. Summarize[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#when-to-retrieve-vs-summarize "Direct link to When to Retrieve vs. Summarize") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Decision Framework[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#decision-framework "Direct link to Decision Framework") | Scenario | Approach | Reason | | --- | --- | --- | | User asks about recent event | Retrieve full | Details matter | | User asks about patterns | Inject summary | Patterns, not details | | Reference to specific date | Retrieve full | Specific lookup | | Background context | Inject summary | General awareness | | User says "exactly what I said" | Retrieve full | Verbatim needed | | Agent needs general knowledge | Inject summary | Token efficiency | ### Decision Tree[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#decision-tree "Direct link to Decision Tree") def decide_retrieval_strategy(query: str, age_days: int) -> str: """Decide whether to retrieve full memories or summaries.""" # Recent events: always full retrieval if age_days < 7: return "full_retrieval" # Specific date reference: full retrieval if contains_date_reference(query): return "full_retrieval" # Pattern or trend questions: summary pattern_keywords = ["usually", "typically", "pattern", "trend", "often"] if any(kw in query.lower() for kw in pattern_keywords): return "summary" # Very old events: summary if age_days > 30: return "summary" # Default: full retrieval return "full_retrieval" Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these prompts to practice context window management with Claude or your preferred AI assistant. ### Prompt 1: Compression Strategy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#prompt-1-compression-strategy "Direct link to Prompt 1: Compression Strategy") You have an agent with 1000 memories but only 4000 tokens of context budget for memory injection.Memory distribution:- 500 memories from last week (avg 50 tokens each = 25,000 tokens)- 300 memories from last month (avg 40 tokens each = 12,000 tokens)- 200 memories from last year (avg 30 tokens each = 6,000 tokens)Design a compression strategy that:1. Stays within the 4000 token budget2. Prioritizes recent over old3. Preserves important patterns from older memories4. Includes specific counts and token allocationsShow your budget allocation and explain the trade-offs. **What you're learning:** Token budgets force hard choices. You can't include everything, so you must prioritize ruthlessly while preserving essential patterns through summarization. ### Prompt 2: Hierarchical Summarization[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#prompt-2-hierarchical-summarization "Direct link to Prompt 2: Hierarchical Summarization") Design a hierarchical summarization system for an agent that's been running for 1 year.The agent handles customer support tickets and has accumulated:- 50,000 individual interaction memories- Average 100 tokens per memory = 5 million tokens totalCreate a 4-tier summarization hierarchy:1. What goes in each tier?2. What triggers promotion to the next tier?3. What information is preserved vs lost at each compression?4. How would you reconstruct detailed information if needed later?Include example memories at each tier level. **What you're learning:** Hierarchical summarization is lossy compression. The art is choosing what to preserve at each level so patterns remain visible even as details fade. ### Prompt 3: Retrieve vs. Summarize Decision[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#prompt-3-retrieve-vs-summarize-decision "Direct link to Prompt 3: Retrieve vs. Summarize Decision") For each of these user queries, decide: should the agent retrieve full memories or inject summaries?1. "What did we discuss about the API yesterday?"2. "What's my typical task completion rate?"3. "Who was in that meeting last Tuesday?"4. "How have my priorities changed over time?"5. "What exactly did I say about the deadline?"6. "What projects am I usually working on?"7. "When did we first discuss the Phoenix migration?"8. "What are my general preferences for task scheduling?"For each:- State your decision (retrieve/summarize)- Explain why- Describe what the retrieved/summarized content would look like **What you're learning:** The decision isn't always obvious. Some queries need precision (full retrieval), others need patterns (summarization). The key is matching retrieval strategy to query intent. Checking access... * [Understanding Context Windows](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#understanding-context-windows) * [Size Comparison](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#size-comparison) * [Why Large Isn't Enough](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#why-large-isnt-enough) * [The "Lost in the Middle" Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#the-lost-in-the-middle-problem) * [Memory Injection Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#memory-injection-strategies) * [1\. Pre-Prompt Injection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#1-pre-prompt-injection) * [2\. Mid-Prompt Injection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#2-mid-prompt-injection) * [3\. Dynamic Injection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#3-dynamic-injection) * [Summarization Chains](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#summarization-chains) * [The Hierarchy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#the-hierarchy) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#implementation) * [Example](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#example) * [When to Retrieve vs. Summarize](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#when-to-retrieve-vs-summarize) * [Decision Framework](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#decision-framework) * [Decision Tree](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#decision-tree) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#try-with-ai) * [Prompt 1: Compression Strategy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#prompt-1-compression-strategy) * [Prompt 2: Hierarchical Summarization](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#prompt-2-hierarchical-summarization) * [Prompt 3: Retrieve vs. Summarize Decision](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/context-window-management#prompt-3-retrieve-vs-summarize-decision) --- # Your First ADK Agent | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Imagine you're building a news research system that autonomously gathers information about emerging technologies. You don't want to write imperative code orchestrating every API call and data flow. Instead, you declare what the agent should do—research, synthesize, and report—and let the framework handle the execution. This is where Google's Agent Development Kit shines. You've built agents with OpenAI's procedural SDK in Chapter 62, where you construct agents imperatively through code. Google ADK takes a different philosophy: **declarative agent design**. You specify agent structure through configuration and type annotations, and ADK handles the orchestration. This approach scales better to complex multi-agent systems and integrates deeply with Google Cloud's production infrastructure. This lesson walks you through installation, your first ADK agent powered by Gemini, and how the declarative approach differs from the SDK pattern you already know. By the end, you'll understand when to choose ADK's structured approach over OpenAI's flexibility. Installation and Project Setup[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#installation-and-project-setup "Direct link to Installation and Project Setup") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before building agents, you need Google ADK installed and your development environment configured. ### Step 1: Install Google ADK[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-1-install-google-adk "Direct link to Step 1: Install Google ADK") Open your terminal and install the google-adk package: pip install google-adk This installs the core ADK framework with dependencies. Verify the installation worked: pip show google-adk **Output:** Name: google-adkVersion: X.Y.Z # Check PyPI for current versionLocation: /path/to/your/python/site-packages If installation succeeds, you're ready to scaffold a project. ### Step 2: Create Your First Agent Project[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-2-create-your-first-agent-project "Direct link to Step 2: Create Your First Agent Project") ADK provides a scaffolding tool to initialize agent projects with best practices. Run: adk create --type=code my_news_agent This creates a project directory with the following structure: my_news_agent/├── agent.py # Main agent code├── agent.yaml # Agent configuration (optional, for declarative definition)├── .env # Environment variables (API keys, project IDs)├── __init__.py # Package marker└── requirements.txt # Python dependencies Navigate into your project: cd my_news_agent ### Step 3: Configure Your API Key[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-3-configure-your-api-key "Direct link to Step 3: Configure Your API Key") ADK supports two authentication backends: **Option A: Google AI (Free Tier)** Get a free API key from [https://ai.google.dev/](https://ai.google.dev/) . Then create a `.env` file in your project: GOOGLE_API_KEY=your-api-key-here **Option B: Vertex AI (Google Cloud)** If you have a Google Cloud project with Vertex AI enabled, set: GOOGLE_GENAI_USE_VERTEXAI=trueGOOGLE_CLOUD_PROJECT=your-project-id For this lesson, Option A (Google AI with free key) is simplest. ADK automatically loads `.env` when you run agents. ### Step 4: Verify Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-4-verify-configuration "Direct link to Step 4: Verify Configuration") Create a verification script `verify_setup.py`: import osfrom google.genai import Clientapi_key = os.getenv('GOOGLE_API_KEY')if not api_key: print("ERROR: GOOGLE_API_KEY not set in .env")else: print(f"API Key found: {api_key[:20]}...") # Optional: Test client initialization client = Client(api_key=api_key) print("Google AI client initialized successfully") **Output:** API Key found: AIzaSyD...Google AI client initialized successfully Your First Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#your-first-agent "Direct link to Your First Agent") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now let's build a working agent. This introduces ADK's declarative approach. ### The Declarative Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#the-declarative-pattern "Direct link to The Declarative Pattern") Unlike OpenAI's imperative SDK (define agent → run → extract result), ADK uses a **declarative model**: 1. **Define the Agent** — Specify agent properties (name, instructions, tools) through code or YAML 2. **Add Tools** — Equip agent with capabilities (google\_search, custom functions) 3. **Run with CLI** — Use `adk run` or `adk web` to interact The key difference: ADK separates agent **definition** from **execution**, making configuration reusable across environments. ### Hello World Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#hello-world-agent "Direct link to Hello World Agent") Create a file `agent.py`: from google.adk.agents import Agentfrom google.adk.tools import google_search# Define the agent declarativelyroot_agent = Agent( name="ai_news_agent", model="gemini-2.5-flash", instruction="You are an AI News Assistant. Use Google Search to find recent AI news and summarize findings clearly.", tools=[google_search]) That's it. No runner object needed. No manual orchestration. ADK handles execution. ### Running Your Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#running-your-agent "Direct link to Running Your Agent") Now use ADK's CLI tools to interact with your agent. **Terminal Interaction:** adk run agent.py This starts an interactive terminal session. Type your queries: You: What are the latest developments in AI agents?Agent: I'll search for recent AI agent developments for you.[Searching for "latest AI agent developments 2025"...]Based on recent news, here are key developments:1. Multi-agent systems becoming mainstream with frameworks like ADK2. Voice agents gaining adoption with Gemini's Live API3. Enterprise deployment through Vertex AI Agent Engine... **Web Interface (Recommended for Learning):** adk web This opens a web UI at `http://localhost:8000` with a chat interface AND an Events tab showing agent reasoning. The Events tab is crucial for understanding how your agent works: * **Tool calls**: See which tools the agent used and when * **Model reasoning**: Watch the agent's thinking process * **Parameters**: Inspect what arguments were passed to each tool * **Results**: See what each tool returned This transparency is one of ADK's core strengths for production agents. Understanding the Declarative Approach[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#understanding-the-declarative-approach "Direct link to Understanding the Declarative Approach") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's break down why ADK's design matters. ### Agent Definition (Code)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#agent-definition-code "Direct link to Agent Definition (Code)") root_agent = Agent( name="ai_news_agent", # Human-readable identifier model="gemini-2.5-flash", # Gemini model to use instruction="You are an AI News Assistant...", # Agent behavior (singular!) tools=[google_search] # Agent capabilities) **Compare to OpenAI SDK** (Chapter 62): # OpenAI: Imperative approachagent = Agent( name="NewsResearcher", instruction="You are..." # Tools added separately through definitions or context)# Then you orchestrate:result = Runner.run_sync(agent, "What's new in AI?")print(result.final_output) **ADK difference:** * Agent configuration is self-contained (name, instructions, tools together) * Execution is abstracted into CLI commands, not code * This separates concerns: developers define agents; operators run them ### Tool Integration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#tool-integration "Direct link to Tool Integration") Google ADK provides built-in tools: * **google\_search** — Web search without needing API keys * **code\_execution** — Python code running in sandboxed environment * **file\_access** — Read/write files safely You simply add them to the `tools` list: from google.adk.tools import google_searchfrom google.adk.code_executors import BuiltInCodeExecutorroot_agent = Agent( name="research_assistant", model="gemini-2.5-flash", instruction="Research questions and validate answers with code.", tools=[google_search], code_executor=BuiltInCodeExecutor() # Enables code execution) Custom tools (your own functions) come later in Lesson 2. ### Gemini Model Selection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#gemini-model-selection "Direct link to Gemini Model Selection") ADK defaults to `gemini-2.5-flash`, a fast, efficient model suitable for most agent tasks. If you want different models: from google.adk.agents import Agentfrom google.adk.tools import google_searchroot_agent = Agent( name="ai_news_agent", model="gemini-2.0-flash-live-001", # For voice/streaming support instruction="You are an AI News Assistant.", tools=[google_search]) **Key model choices:** * `gemini-2.5-flash` — Fast, good for tooling (default) * `gemini-2.0-flash-live-001` — Voice/video streaming support * `gemini-1.5-pro` — Higher reasoning capability (slower) Comparing ADK to OpenAI SDK[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#comparing-adk-to-openai-sdk "Direct link to Comparing ADK to OpenAI SDK") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By now you've seen both approaches. Here's how they differ: | Aspect | OpenAI SDK | Google ADK | | --- | --- | --- | | **Design Pattern** | Imperative (code orchestrates execution) | Declarative (config specifies agent) | | **Tool Definition** | Pydantic models, explicit function wrapping | Type annotations, auto-wrapped functions | | **Execution** | `Runner.run_sync()` in code | `adk run` / `adk web` CLI | | **State Management** | `context_variables` parameter passing | `ToolContext` for persistent state (Lesson 3) | | **Multi-Agent** | Handoffs (flexible but manual) | Workflow agents (deterministic pipelines) | | **Safety** | Input/output validation | 6 callback types for granular control | | **Debugging** | Print statements or external logging | Events tab in `adk web` UI | | **Deployment** | Self-hosted anywhere | Vertex AI Agent Engine (managed) | **When to choose ADK:** * Building production agents at scale (Vertex AI integration) * Needing transparent debugging (Events tab) * Creating deterministic workflows (Workflow agents) * Working in Google Cloud ecosystem **When to choose OpenAI SDK:** * Building simple, one-off agents * Needing maximum flexibility for custom orchestration * Integrating with non-Google services exclusively * Team familiar with Python imperative patterns Common Setup Issues[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#common-setup-issues "Direct link to Common Setup Issues") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Issue: "ModuleNotFoundError: No module named 'google.adk'"** Your environment doesn't have ADK installed. Install it: pip install google-adk **Issue: "GOOGLE\_API\_KEY not found" or authentication fails** Your `.env` file isn't set up or isn't in the right location. Verify: cat .env # Check if file exists and has GOOGLE_API_KEY=... If missing, create `.env` in your project root (same directory as `agent.py`). **Issue: "google\_search tool not found"** You're importing from the wrong module. Use: from google.adk.tools import google_search NOT `from google.genai.tools import ...` **Issue: adk web doesn't open or shows "Connection refused"** The web server is running on `localhost:8000` but your browser can't reach it. Try: # Check if server is runningcurl http://localhost:8000# If that fails, restart adk webadk web --port=8001 # Use different port **Issue: Agent takes 30+ seconds to respond** This is expected. ADK agents: 1. Send your prompt to Google's servers 2. Call google\_search tool (which makes HTTP requests) 3. Process search results 4. Generate response Network latency compounds. This is normal for cloud-based agents. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Domain-Specific News Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#prompt-1-domain-specific-news-agent "Direct link to Prompt 1: Domain-Specific News Agent") Create an ADK agent that specializes in researching one specific domain(choose: climate technology, space exploration, cybersecurity, healthcare AI, or renewable energy).Write the agent definition with:- Clear instructions emphasizing the domain focus- The google_search tool- Specific constraints (e.g., "only report peer-reviewed sources" or "focus on recent breakthroughs")Run the agent with: adk runAsk it: "What's the most significant news in [your domain] this month?"What does the agent prioritize in its search results? How do the instructions shape its focus? **What you're learning:** How declarative agent specifications drive search behavior and information prioritization ### Prompt 2: Comparing ADK vs OpenAI Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#prompt-2-comparing-adk-vs-openai-pattern "Direct link to Prompt 2: Comparing ADK vs OpenAI Pattern") Think about an agent you might build: a customer feedback analyzer that researchescompany reputation and competitive positioning.Write the agent definition for ADK:- Name and instructions- Tools needed (google_search, maybe others)- Model choice (explain why)Then, imagine writing the same agent with OpenAI SDK from Chapter 62.What would be different?Specifically consider:- How would you orchestrate the search (manual loop vs declarative definition)?- How would you see the agent's reasoning (Events tab vs print statements)?- How would you deploy it (Vertex AI vs self-hosted)?Which would you choose for this use case? Why? **What you're learning:** The architectural tradeoffs between declarative (ADK) and imperative (OpenAI) agent design ### Prompt 3: Debugging with adk web[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#prompt-3-debugging-with-adk-web "Direct link to Prompt 3: Debugging with adk web") Create a simple agent that researches AI news.Run it with: adk webOpen http://localhost:8000 in your browser and click the Events tab.Ask the agent: "What are the latest breakthroughs in AI agent architectures?"In the Events tab, examine:1. How many times did google_search get called?2. What search queries did the agent construct?3. What results did search return?4. How did the agent synthesize results into its response?Compare this visibility to what you'd see with OpenAI SDK (just final_output).Why is this transparency important for production agents? **What you're learning:** Production agent debugging through declarative tool transparency and reasoning visibility * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built a `google-adk` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#test-your-skill "Direct link to Test Your Skill") Using my google-adk skill, create a basic ADK agent with google_search tool.Does my skill correctly demonstrate LlmAgent creation, model configuration, and tool integration? ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include the declarative agent definition pattern (Agent class with name, model, instruction, tools)? * Did it explain how ADK's Runner pattern differs from OpenAI's imperative SDK approach? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My google-adk skill is missing guidance on declarative agent setup and the Agent class pattern.Update it to include:- Agent class instantiation with proper parameters- Tool registration (both built-in like google_search and custom tools)- adk run vs adk web usage patterns- How declarative config separates agent definition from execution * * * Checking access... * [Installation and Project Setup](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#installation-and-project-setup) * [Step 1: Install Google ADK](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-1-install-google-adk) * [Step 2: Create Your First Agent Project](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-2-create-your-first-agent-project) * [Step 3: Configure Your API Key](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-3-configure-your-api-key) * [Step 4: Verify Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#step-4-verify-configuration) * [Your First Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#your-first-agent) * [The Declarative Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#the-declarative-pattern) * [Hello World Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#hello-world-agent) * [Running Your Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#running-your-agent) * [Understanding the Declarative Approach](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#understanding-the-declarative-approach) * [Agent Definition (Code)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#agent-definition-code) * [Tool Integration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#tool-integration) * [Gemini Model Selection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#gemini-model-selection) * [Comparing ADK to OpenAI SDK](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#comparing-adk-to-openai-sdk) * [Common Setup Issues](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#common-setup-issues) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#try-with-ai) * [Prompt 1: Domain-Specific News Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#prompt-1-domain-specific-news-agent) * [Prompt 2: Comparing ADK vs OpenAI Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#prompt-2-comparing-adk-vs-openai-pattern) * [Prompt 3: Debugging with adk web](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#prompt-3-debugging-with-adk-web) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/your-first-adk-agent#improve-your-skill) --- # Transport Layers: How MCP Messages Travel | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've configured an MCP server on your laptop. It works perfectly—Claude Code can read files, query databases, create issues. But what happens when you want to deploy that server to the cloud so your whole team can use it? Or run it on a customer's infrastructure? The good news: **your MCP server code doesn't change.** Only the _transport_—how messages physically travel between client and server—changes. Think of it like phone calls. Whether you call someone via landline, cellular, or WiFi calling, the conversation is the same. Only the underlying network changes. MCP works the same way: the protocol (JSON-RPC messages) stays identical whether messages travel through local process streams or across the internet via HTTP. This lesson explores that separation. You'll understand **when to use stdio transport** (perfect for development and single-client scenarios), **when to use Streamable HTTP transport** (necessary for remote servers and multi-client products), and most importantly, **why your server code doesn't need to know the difference**. Understanding Transport Agnosticism[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#understanding-transport-agnosticism "Direct link to Understanding Transport Agnosticism") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before diving into specific transports, let's establish why this architecture choice matters. **Traditional approach**: Lock tools to a single communication mechanism. GraphQL typically uses HTTP. gRPC requires its own protocol. WebSocket connections have different semantics than REST APIs. **MCP's approach**: Define the protocol (JSON-RPC message structure, capability discovery, tool schemas) completely separately from the transport. A tool definition works identically whether it arrives via subprocess streams or HTTP requests. This separation means: * **Developers focus on business logic** (what tools do, what resources expose) without transport concerns * **Deployers choose optimal transport** based on infrastructure (local for testing, remote for production) * **Systems scale horizontally** with zero application changes—just switch to HTTP transport You'll see the same pattern across professional systems: email protocol (SMTP, POP, IMAP) works over different underlying transports; web browsers speak HTTP whether connected via broadband or wireless; APIs remain unchanged whether accessed from local network or cloud. MCP's transport independence is not convenient—it's architectural maturity. stdio Transport: Local Communication via Streams[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#stdio-transport-local-communication-via-streams "Direct link to stdio Transport: Local Communication via Streams") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### How It Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#how-it-works "Direct link to How It Works") When you add an MCP server to Claude Code's configuration, Claude **launches that server as a subprocess**. The two programs then communicate through standard input/output streams: * **Client writes** JSON-RPC requests to the server's stdin * **Server reads** from stdin, processes, and writes responses to stdout * **Client reads** responses from the server's stdout This is the simplest possible IPC (inter-process communication) mechanism. No network layer. No HTTP headers. No authentication negotiation. Just process streams. ┌─────────────────────────────────────────┐│ MCP Host (Client) ││ ││ Launches: mcp-server ││ │ ││ ←─────────────→ stdin/stdout ││ │└─────────────────────────────────────────┘ ▲ │ │ subprocess streams │ ▼┌─────────────────────────────────────────┐│ MCP Server (subprocess) ││ ││ Reads from stdin ││ Processes tool calls ││ Writes to stdout ││ │└─────────────────────────────────────────┘ ### Critical Detail: Log Redirection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#critical-detail-log-redirection "Direct link to Critical Detail: Log Redirection") Here's where many developers stumble: **stdio has ONE output stream**. Both regular output and error messages use stdout. This creates a problem—how does the client distinguish message protocol from debug logs? MCP solves this with **strict stream discipline**: * **Messages go to stdout** (JSON-RPC request/response protocol) * **Logs go to stderr** (diagnostic information, warnings, errors) This means if your MCP server implementation prints to stdout instead of stderr, you corrupt the protocol stream. The client tries to parse "Starting server..." as JSON-RPC and fails. This is not theoretical—it's a common debugging scenario: # WRONG - breaks stdio transportprint("Server starting") # Goes to stdout, corrupts protocolresult = tool_function() # CORRECT - preserves protocol streamimport sysprint("Server starting", file=sys.stderr) # Goes to stderrresult = tool_function() ### When to Use stdio[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#when-to-use-stdio "Direct link to When to Use stdio") **Best for**: * **Local development**: Server and client on same machine * **Desktop applications**: Claude Code, Cursor, Zed * **Single-client scenarios**: One user, one instance of the server * **Simplicity**: Minimal configuration, no networking **Limitations**: * **Not remote**: Can't reach server on different machine * **Not multi-client**: Each client needs its own subprocess * **Process dependent**: Server must be executable in client's environment HTTP Fundamentals (Optional Background)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#http-fundamentals-optional-background "Direct link to HTTP Fundamentals (Optional Background)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Already Know HTTP? If you've built web APIs, used `fetch()` or `requests`, or understand how browsers load websites, skip to **Streamable HTTP Transport** below. This section provides foundation for those new to web protocols. When you type a URL into your browser, you're using **HTTP (HyperText Transfer Protocol)**—the communication standard that powers the web. Understanding HTTP basics is essential for MCP's remote transport. ### The Request-Response Model[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#the-request-response-model "Direct link to The Request-Response Model") HTTP follows a simple pattern: 1. **Client sends a request** — "I want something from you" 2. **Server sends a response** — "Here's what you asked for" (or an error) Every HTTP interaction is one request producing one response. The client always initiates; the server always responds. ### HTTP Methods: What You Want to Do[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#http-methods-what-you-want-to-do "Direct link to HTTP Methods: What You Want to Do") | Method | Purpose | Example | | --- | --- | --- | | **GET** | Retrieve data | Load a webpage, fetch user profile | | **POST** | Send data to create/process | Submit a form, call an API | | **PUT** | Update existing data | Edit a user's settings | | **DELETE** | Remove data | Delete a comment | MCP uses **POST** exclusively—every MCP message is sent via POST request because you're sending JSON-RPC data for the server to process. ### Headers: Metadata About Your Request[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#headers-metadata-about-your-request "Direct link to Headers: Metadata About Your Request") Headers are key-value pairs that travel with requests and responses. Think of them as the envelope around your letter—they describe the contents without being the contents. Common headers you'll encounter: * `Content-Type: application/json` — "My request body is JSON" * `Authorization: Bearer abc123` — "Here's my access token" * `Accept: text/event-stream` — "I can receive streaming data" ### Status Codes: Did It Work?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#status-codes-did-it-work "Direct link to Status Codes: Did It Work?") Servers respond with a three-digit code indicating what happened: | Code | Meaning | What It Tells You | | --- | --- | --- | | **200** | OK | Request succeeded | | **201** | Created | Resource was created | | **400** | Bad Request | Your request was malformed | | **401** | Unauthorized | Authentication required/failed | | **404** | Not Found | Resource doesn't exist | | **500** | Server Error | Something broke on the server | When MCP uses HTTP transport, you'll see these codes in logs and error messages. ### Server-Sent Events (SSE): One-Way Streaming[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#server-sent-events-sse-one-way-streaming "Direct link to Server-Sent Events (SSE): One-Way Streaming") Standard HTTP is request-response: one request, one response, done. But what if the server needs to send multiple messages over time—like progress updates during a long operation? **Server-Sent Events (SSE)** solves this. The client makes one request, and the server keeps the connection open, streaming multiple messages: Client: POST /mcp (start operation)Server: data: {"progress": 10}Server: data: {"progress": 50}Server: data: {"progress": 100}Server: data: {"result": "done"} SSE is one-way (server to client only) and works over standard HTTP, no special protocols needed. MCP's stateful HTTP transport uses SSE for streaming responses. * * * With these fundamentals in place, let's see how MCP applies them. Streamable HTTP Transport: Remote Communication[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#streamable-http-transport-remote-communication "Direct link to Streamable HTTP Transport: Remote Communication") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### How It Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#how-it-works-1 "Direct link to How It Works") When an MCP server runs on a remote machine (or needs to serve multiple clients), the architecture shifts. The server becomes a **persistent service** listening on an HTTP endpoint. Clients connect via HTTP POST requests: 1. **Client sends** HTTP POST request with JSON-RPC message 2. **Server processes** the request 3. **Server responds** with HTTP response body containing JSON-RPC result ### Stateless vs Stateful Modes[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#stateless-vs-stateful-modes "Direct link to Stateless vs Stateful Modes") Streamable HTTP supports **two operational modes**: | Mode | Response Type | Session | Best For | | --- | --- | --- | --- | | **Stateless** | JSON or streaming | None | Cloud scaling (multiple nodes), serverless, no MCP advanced features needed | | **Stateful** | SSE stream | Maintained | Long-running operations, progress updates, complex workflows | **Stateless mode** (recommended for cloud deployments): * Client sends HTTP POST with JSON-RPC request * Server can return single JSON OR streaming response * No session state between requests * Essential for horizontal scaling (load balancers, multi-node, serverless) * Use when you don't need MCP advanced features (sampling, roots, etc.) Client → POST /mcp {"method": "tools/call", ...}Server → 200 OK {"result": {...}} (single response) **Stateful mode** (for streaming scenarios): * Server maintains session across requests * Uses Server-Sent Events (SSE) for streaming responses * Required for long-running tool executions with progress updates * More complex to deploy (sticky sessions, connection management) Client → POST /mcp {"method": "tools/call", ...}Server → 200 OK (SSE stream) data: {"progress": 25} data: {"progress": 50} data: {"result": {...}} For most use cases, **stateless JSON is simpler and scales better**—only use stateful SSE when you genuinely need streaming progress or real-time updates. ### Multi-Client Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#multi-client-architecture "Direct link to Multi-Client Architecture") Unlike stdio (one client per server process), Streamable HTTP allows multiple clients to connect to a single server: ┌──────────────────────────────────────┐│ Client A ││ ││ HTTP POST /mcp ││ {tool request...} ││ ↓ ││ [JSON or SSE response] │└──────────────────────────────────────┘┌──────────────────────────────────────┐│ Client B ││ ││ HTTP POST /mcp ││ {tool request...} ││ ↓ ││ [JSON or SSE response] │└──────────────────────────────────────┘ ▲ ▲ │ │ HTTP requests (multiple clients) │ │ ▼ ▼┌──────────────────────────────────────┐│ MCP Server (remote service) ││ ││ Listens on :8000 ││ Handles multiple clients ││ Returns JSON or streams SSE ││ │└──────────────────────────────────────┘ ### Authentication[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#authentication "Direct link to Authentication") HTTP transport requires securing the connection. MCP supports standard HTTP authentication mechanisms: * **Bearer tokens**: Client includes `Authorization: Bearer ` header * **OAuth 2.1**: Full authentication flow with token exchange * **Custom headers**: Implementation-specific authentication The server validates credentials before processing requests. This isn't extra security layering—it's integral to HTTP transport, unlike stdio which assumes trusted localhost communication. ### When to Use Streamable HTTP[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#when-to-use-streamable-http "Direct link to When to Use Streamable HTTP") **Best for**: * **Remote servers**: Server on different machine/cloud service * **Multi-client products**: Multiple users/clients accessing same server * **Cloud deployment**: Serverless functions, managed services * **Standard infrastructure**: Works with any HTTP client library **Complexity**: * Requires running server as persistent service (not subprocess) * Network configuration (firewall, routing) * Authentication setup * Monitoring and health checks Transport Comparison: When to Choose Which[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#transport-comparison-when-to-choose-which "Direct link to Transport Comparison: When to Choose Which") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's how the transports compare across practical dimensions: | Criterion | stdio | Streamable HTTP (Stateless) | Streamable HTTP (Stateful) | | --- | --- | --- | --- | | **Deployment** | Local only | Local or remote | Local or remote | | **Client count** | Single | Multiple simultaneous | Multiple (with session mgmt) | | **Launch mechanism** | Subprocess (automatic) | External service | External service | | **Latency** | under 1ms (local IPC) | 10-100ms (network) | 10-100ms + stream overhead | | **Setup complexity** | Simple JSON config | Service + networking | Service + sessions + SSE | | **Suitable for** | Development, desktop | Production, serverless | Long-running operations | | **Authentication** | None (trusted localhost) | Required (HTTP auth) | Required (HTTP auth) | | **Scalability** | Linear (one per client) | Horizontal (stateless) | Requires sticky sessions | | **State management** | In-process | None | Server-side sessions | ### Decision Framework[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#decision-framework "Direct link to Decision Framework") **Choose stdio when**: * Building features for Claude Code, Cursor, or similar local IDEs * Prototyping and testing MCP servers * Your MCP server runs on developer's machine * Working in single-client context **Choose Streamable HTTP (Stateless) when**: * Server runs on cloud infrastructure or different machine * Multiple users/applications need to access same MCP server * Building production MCP services with serverless (Lambda, Cloud Run) * Server needs to scale horizontally with load balancers * Tools complete quickly (< few seconds) **Choose Streamable HTTP (Stateful) when**: * Tools take significant time (code generation, large file processing) * Users need progress updates during execution * Workflow involves multiple back-and-forth exchanges * Building interactive experiences requiring real-time feedback The Data Layer Remains Transport-Independent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#the-data-layer-remains-transport-independent "Direct link to The Data Layer Remains Transport-Independent") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is the crucial insight: **your JSON-RPC message structure doesn't change**. Whether a tool request travels through stdio or HTTP, the actual message is identical: { "jsonrpc": "2.0", "id": "123", "method": "tools/call", "params": { "name": "search_codebase", "arguments": { "pattern": "async" } }} The transport layer handles **how** this message gets from client to server. Your tool implementation, parameter validation, and business logic remain unchanged. This is why experienced developers care about this architecture: it means **switching a server from local development (stdio) to production deployment (HTTP) requires updating configuration, not rewriting code**. An upcoming lesson covers how to configure MCP clients in various hosts—you'll see how transport selection is purely a configuration concern. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion to explore transport layer concepts and decision-making. ### Prompt 1: Discover Your Current Transport (No Setup Required)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#prompt-1-discover-your-current-transport-no-setup-required "Direct link to Prompt 1: Discover Your Current Transport (No Setup Required)") I use Claude Code with several MCP servers configured. Help me understandwhich transport each one uses. Here's my config (show your .claude/config.jsonor relevant configuration snippet):[paste your MCP configuration]For each server, explain:1. Is it stdio or HTTP based on the configuration?2. How do you know from the config?3. What would need to change if we moved it from local to cloud? **What you're learning**: Recognizing transport choice from configuration structure; understanding what configuration reveals about deployment architecture. ### Prompt 2: Diagnose a Transport Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#prompt-2-diagnose-a-transport-problem "Direct link to Prompt 2: Diagnose a Transport Problem") My MCP server works fine locally in Claude Code, but when we try torun it on our remote server, clients can't connect. The server startswithout errors but responds with blank responses.The server uses stdio transport locally. I moved it to HTTP by:1. Changed from subprocess to Flask service2. Removed stdin/stdout communication3. Added HTTP endpointsWhat's probably wrong? What questions would you ask to diagnose this? **What you're learning**: Connecting transport configuration to communication flow; identifying common mistakes when switching transports. ### Prompt 3: Evaluate Transport for a Scenario[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#prompt-3-evaluate-transport-for-a-scenario "Direct link to Prompt 3: Evaluate Transport for a Scenario") We're building an internal tool that helps engineers refactor Python codeusing AI. The tool needs to:- Run AI-powered code analysis- Access files on the engineer's machine- Provide results back in the editor- Eventually: serve 50+ engineers simultaneously from cloudShould we use stdio or HTTP transport initially? What about after launch?What changes when we scale from 1 user to 50+?Walk me through the trade-offs and explain your recommendation. **What you're learning**: Making deployment architecture decisions based on requirements; understanding how scalability drives transport choices. ### Safety Note on Transport Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#safety-note-on-transport-configuration "Direct link to Safety Note on Transport Configuration") When setting up HTTP transport in production, always verify authentication is enabled—never expose MCP endpoints without requiring bearer tokens or OAuth. Local stdio transport is inherently safer because it requires the server to be running as a subprocess on your machine, but HTTP services are network-reachable and need explicit access control. Checking access... * [Understanding Transport Agnosticism](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#understanding-transport-agnosticism) * [stdio Transport: Local Communication via Streams](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#stdio-transport-local-communication-via-streams) * [How It Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#how-it-works) * [Critical Detail: Log Redirection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#critical-detail-log-redirection) * [When to Use stdio](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#when-to-use-stdio) * [HTTP Fundamentals (Optional Background)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#http-fundamentals-optional-background) * [The Request-Response Model](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#the-request-response-model) * [HTTP Methods: What You Want to Do](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#http-methods-what-you-want-to-do) * [Headers: Metadata About Your Request](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#headers-metadata-about-your-request) * [Status Codes: Did It Work?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#status-codes-did-it-work) * [Server-Sent Events (SSE): One-Way Streaming](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#server-sent-events-sse-one-way-streaming) * [Streamable HTTP Transport: Remote Communication](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#streamable-http-transport-remote-communication) * [How It Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#how-it-works-1) * [Stateless vs Stateful Modes](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#stateless-vs-stateful-modes) * [Multi-Client Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#multi-client-architecture) * [Authentication](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#authentication) * [When to Use Streamable HTTP](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#when-to-use-streamable-http) * [Transport Comparison: When to Choose Which](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#transport-comparison-when-to-choose-which) * [Decision Framework](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#decision-framework) * [The Data Layer Remains Transport-Independent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#the-data-layer-remains-transport-independent) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#try-with-ai) * [Prompt 1: Discover Your Current Transport (No Setup Required)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#prompt-1-discover-your-current-transport-no-setup-required) * [Prompt 2: Diagnose a Transport Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#prompt-2-diagnose-a-transport-problem) * [Prompt 3: Evaluate Transport for a Scenario](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#prompt-3-evaluate-transport-for-a-scenario) * [Safety Note on Transport Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/transport-layers#safety-note-on-transport-configuration) --- # Prompts: The User-Controlled Primitive | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You know how Claude Code has slash commands? `/summarize`, `/review`, `/test`—quick prompts that appear when you need them, encoding expertise into one-click workflows. That's exactly what MCP prompts are. Pre-crafted instruction templates that domain experts create once and users apply whenever needed. Unlike tools (where the AI decides when to use them) and resources (where the app fetches them automatically), **prompts put the user in control: "I want to apply the legal contract review prompt now."** The insight is subtle but profound: Some of your most valuable expertise isn't code. It's the way you ask questions, structure analysis, or guide thinking. Prompts let you package that expertise and distribute it. The User-Controlled Paradigm[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-user-controlled-paradigm "Direct link to The User-Controlled Paradigm") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In the three MCP primitives you've learned, control flows different directions: **Tools**: The LLM decides. You provide tools, AI chooses when to call them. You: "What's the weather?"AI: [decides] → Calls get_weather tool → Returns result **Resources**: The app decides. You fetch data when needed. You: "Show me document X"App: [decides] → Fetches resource:///documents/X → Returns data **Prompts**: The user decides. You select when to apply an expert-crafted template. You: [consciously chooses] → Applies contract_review_prompt → AI receives structured instruction This isn't subtle. Compare these workflows: | Primitive | Who Controls? | When Applied? | Use Case | | --- | --- | --- | --- | | Tools | LLM decides autonomously | During conversation | Actions (file ops, API calls, calculations) | | Resources | App fetches automatically | Before/during conversation | Data access (files, databases, APIs) | | Prompts | User selects explicitly | Before conversation | Domain expertise templates | Prompts are the way domain experts encode their judgment into systems. A contract lawyer designs the prompt once; clients use it thousands of times, getting expert-quality review every time. Prompt Discovery: prompts/list[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#prompt-discovery-promptslist "Direct link to Prompt Discovery: prompts/list") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Before a user can apply a prompt, they need to know what prompts exist. MCP handles this through `prompts/list`. ### The Sequence[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-sequence "Direct link to The Sequence") Client (Claude Code) Server (Your MCP) | | |-----> prompts/list ------>| | | |<--- Prompt definitions ----| | | ### The Request[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-request "Direct link to The Request") The client sends a simple discovery request: { "jsonrpc": "2.0", "id": 1, "method": "prompts/list", "params": {}} Simple discovery request: "What prompts do you have?" (The spec supports optional pagination via `cursor` parameter for large prompt collections.) ### The Response[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-response "Direct link to The Response") Your server responds with available prompts: { "jsonrpc": "2.0", "id": 1, "result": { "prompts": [ { "name": "contract_review", "description": "Legal contract analysis with risk assessment", "arguments": [ { "name": "contract_type", "description": "Type of contract (NDA, employment, licensing, etc.)", "required": true } ] }, { "name": "code_security_audit", "description": "Security-focused code review using OWASP principles", "arguments": [ { "name": "language", "description": "Programming language (python, javascript, go, etc.)", "required": true }, { "name": "framework", "description": "Framework context (fastapi, express, gin, optional)", "required": false } ] } ] }} Each prompt includes: * **name**: Unique identifier (snake\_case) * **description**: Human-readable summary * **arguments**: List of parameters the prompt accepts (optional) Notice: Unlike tools, prompts don't have input schemas. Arguments are simpler—just name and description. This is intentional. Prompts are instruction templates, not computational functions. Prompt Retrieval: prompts/get[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#prompt-retrieval-promptsget "Direct link to Prompt Retrieval: prompts/get") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once the user selects a prompt, the client requests the actual prompt text with arguments: ### The Request[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-request-1 "Direct link to The Request") { "jsonrpc": "2.0", "id": 2, "method": "prompts/get", "params": { "name": "contract_review", "arguments": { "contract_type": "NDA" } }} The client passes: * **name**: Which prompt to retrieve * **arguments**: Values to substitute into the template ### The Response[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-response-1 "Direct link to The Response") { "jsonrpc": "2.0", "id": 2, "result": { "messages": [ { "role": "user", "content": "You are a contract review specialist focusing on NDA agreements.\n\nAnalyze the following NDA for:\n\n1. **Confidentiality Scope**: What information is protected? How broad is the definition?\n\n2. **Term & Termination**: How long does the NDA last? Can it be terminated early?\n\n3. **Return/Destruction**: What happens to information after termination?\n\n4. **Allowed Disclosures**: What exceptions exist (legal requirement, public domain)?\n\n5. **Remedies**: What's the penalty for breach?\n\nProvide risk rating (LOW/MEDIUM/HIGH) for each section." } ] }} The response is a `messages` array—the exact instructions to give the AI. The server has: 1. Loaded the template 2. Substituted arguments (contract\_type = "NDA") 3. Returned the ready-to-use instruction Implementing Prompts with FastMCP[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#implementing-prompts-with-fastmcp "Direct link to Implementing Prompts with FastMCP") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's how to implement the contract review prompt: from mcp.server.fastmcp import FastMCPfrom pydantic import Fieldmcp = FastMCP("legal-assistant")@mcp.prompt()def contract_review( contract_type: str = Field( description="Type of contract (NDA, employment, licensing, etc.)" )) -> str: """Legal contract analysis with risk assessment""" return f"""You are a contract review specialist focusing on {contract_type} agreements.Analyze the following {contract_type} for:1. **Confidentiality Scope**: What information is protected? How broad is the definition?2. **Term & Termination**: How long does the {contract_type} last? Can it be terminated early?3. **Return/Destruction**: What happens to information after termination?4. **Allowed Disclosures**: What exceptions exist (legal requirement, public domain)?5. **Remedies**: What's the penalty for breach?Provide risk rating (LOW/MEDIUM/HIGH) for each section."""@mcp.prompt()def code_security_audit( language: str = Field(description="Programming language (python, javascript, go)"), framework: str = Field( description="Framework context (fastapi, express, gin, optional)", default=None )) -> str: """Security-focused code review using OWASP principles""" framework_context = f" in {framework}" if framework else "" return f"""You are a security engineer reviewing {language} code{framework_context}.Analyze for OWASP Top 10 vulnerabilities:1. **Injection**: SQL injection, command injection, template injection2. **Authentication**: Weak credential handling, session management flaws3. **Sensitive Data**: Exposed secrets, unencrypted transmission4. **XML External Entities**: XXE attacks in XML parsing5. **Broken Access Control**: Authorization bypasses6. **Misconfiguration**: Security headers, debug flags, unnecessary services7. **XSS**: Cross-site scripting (reflected, stored, DOM)8. **Insecure Deserialization**: Object injection risks9. **Components**: Known vulnerabilities in dependencies10. **Insufficient Logging**: Missing security event trackingFor each finding, provide:- Vulnerability class- Risk level (CRITICAL/HIGH/MEDIUM/LOW)- Proof of concept- Remediation steps""" The `@mcp.prompt()` decorator transforms a Python function into a discoverable prompt. Arguments become `Field()` descriptions. Static vs Dynamic Prompts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#static-vs-dynamic-prompts "Direct link to Static vs Dynamic Prompts") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Static prompts** have no arguments—they're the same every time: @mcp.prompt()def daily_standup() -> str: """Morning standup meeting template""" return """Generate a standup summary covering: 1. What did you complete yesterday? 2. What will you complete today? 3. What blockers exist?""" **Dynamic prompts** accept arguments that customize the template: @mcp.prompt()def api_design_review( framework: str = Field(description="API framework (REST, GraphQL, gRPC)"), context: str = Field(description="Domain context (ecommerce, healthcare, finance)")) -> str: """API design review for specified framework and domain""" return f"""Review this {framework} API design for {context} applications. Evaluate: - Endpoint design patterns - Error handling standards - Security controls for {context} - Versioning strategy - Rate limiting approach""" Dynamic prompts are more powerful. They let one template serve thousands of use cases. The Three Control Models Summary[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-three-control-models-summary "Direct link to The Three Control Models Summary") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's how the three MCP primitives compare: | **Primitive** | **Controller** | **Timing** | **Best For** | **Example** | | --- | --- | --- | --- | --- | | **Tools** | LLM decides | During conversation | Autonomous actions | `save_file()`, `run_test()` | | **Resources** | App fetches | Before/during conversation | Data access | Database queries, file reads | | **Prompts** | User selects | Before conversation | Domain expertise | Contract review, security audit | **How to choose**: * Something the AI should decide autonomously? → **Tool** * Data the app should fetch when needed? → **Resource** * Expert-crafted instruction the user applies? → **Prompt** A domain expert (lawyer, architect, engineer) creates prompts once. Users apply them thousands of times, getting consistent, expert-quality guidance without needing the expert present. Why Prompts Matter[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#why-prompts-matter "Direct link to Why Prompts Matter") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Three reasons prompts are revolutionary: **1\. Expertise Distribution** A legal expert spends 10 hours designing the contract review prompt. 10,000 non-lawyers use it and get expert-quality reviews. The bottleneck is eliminated. **2\. Consistency** When the expert is a person, they're inconsistent (tired, busy, ill, on vacation). When the expert is a prompt, every application is identical. Quality is predictable. **3\. Knowledge Preservation** An expert leaves the company. If their knowledge was in their head, it's gone. If it's in a prompt, it lives on. The organization retains the expertise. This is why MCP prompts are part of the Agent Factory vision. You build agents that distribute your expertise globally. Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#safety-note "Direct link to Safety Note") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When designing prompts, remember: **The user sees the prompt, but they don't see the implementation.** A security-focused prompt that makes a dangerous recommendation can cause harm. Test prompts thoroughly with edge cases and adversarial inputs before distributing them. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Setup**: You'll use Claude Code to explore prompts in your existing MCP servers. **Prompt 1: Understanding Prompt Discovery** Ask Claude: "In the learning context, explain how `prompts/list` discovers available prompts in an MCP server. What information does the response include and why?" **What you're learning**: You're examining the protocol structure for prompt discovery—the request that populates the user interface with available prompts. **Prompt 2: Designing a Domain-Expert Prompt** Ask Claude: "I'm a data scientist who wants to create an MCP prompt that guides users through feature engineering decisions. The prompt should accept arguments for data type (categorical, numerical, time-series) and problem context (classification, regression, clustering). Design this prompt with clear questions that guide systematic feature selection." Record Claude's suggestion. Notice: * How arguments customize the template * How the prompt guides thinking (questions vs directives) * Whether the prompt encodes expert decision-making **Prompt 3: Comparing Control Models** Ask Claude: "In MCP, when would you use each primitive? I want to: 1. Let Claude automatically save analysis results (which primitive?) 2. Have Claude analyze a file I select (which primitive?) 3. Apply a legal-review checklist I designed (which primitive?) For each, explain why that primitive matches the control flow." **What emerged from iteration**: Through these three interactions, you've seen: * How prompts differ from tools and resources in control flow * How to design prompts that encode expertise through questions * How to select the right primitive for your use case This is the foundation for designing MCP systems that distribute domain expertise globally. Checking access... * [The User-Controlled Paradigm](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-user-controlled-paradigm) * [Prompt Discovery: prompts/list](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#prompt-discovery-promptslist) * [The Sequence](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-sequence) * [The Request](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-request) * [The Response](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-response) * [Prompt Retrieval: prompts/get](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#prompt-retrieval-promptsget) * [The Request](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-request-1) * [The Response](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-response-1) * [Implementing Prompts with FastMCP](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#implementing-prompts-with-fastmcp) * [Static vs Dynamic Prompts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#static-vs-dynamic-prompts) * [The Three Control Models Summary](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#the-three-control-models-summary) * [Why Prompts Matter](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#why-prompts-matter) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#safety-note) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/prompts-the-user-controlled-primitive#try-with-ai) --- # Conversation History Management | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your agent responds to user messages but treats each interaction as brand new—no memory of previous questions, no continuity between sessions. The user asks, "What was that task we discussed earlier?" and your agent replies, "I don't see any previous discussion." The conversation context exists in the database, but it never reaches the agent's prompt. This isn't a bug in ChatKit. It's a missing step in your respond() implementation. ChatKit provides the storage API—`store.load_thread_items()`—but leaves history serialization to you. This lesson shows you how to load conversation history, format it for agent consumption, and extract the recurring pattern into a reusable skill. By the end, you'll have built the `conversation-history` skill that transforms ChatKit's thread storage into agent memory. * * * Loading Thread History from Store[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#loading-thread-history-from-store "Direct link to Loading Thread History from Store") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ChatKit stores every message in the thread database. The `store.load_thread_items()` method retrieves conversation history with control over ordering, pagination, and filtering. **Method Signature**: from chatkit.types import Page, ThreadItemasync def load_thread_items( self, thread_id: str, *, after: str | None = None, limit: int | None = None, order: Literal["asc", "desc"] = "asc",) -> Page[ThreadItem]: **Parameters**: * `thread_id`: Conversation identifier (from `thread.id` in respond()) * `after`: Load items after this item ID (pagination cursor) * `limit`: Maximum number of items to return (default: all) * `order`: `"asc"` (oldest first) or `"desc"` (newest first) **Returns**: `Page[ThreadItem]` object containing `.data` (list of items) and pagination metadata ### Loading Recent History[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#loading-recent-history "Direct link to Loading Recent History") Most agents don't need the entire conversation history—recent context suffices. Load the last 20 messages: from chatkit.server import ChatKitServerfrom chatkit.types import ThreadMetadata, UserMessageItem, ThreadItemfrom typing import AsyncIterator, Anyclass MemoryAwareServer(ChatKitServer): async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: # Load last 20 messages (newest first) page = await self.store.load_thread_items( thread.id, after=None, limit=20, order="desc", # Newest messages first context=context, ) # Extract items from Page object history_items = page.data # Reverse to chronological order (oldest → newest) history_items.reverse() # Now serialize history_items for agent... **Output** (when thread has 5 messages): Loaded 5 thread items from thread_abc123Items in chronological order: 1. user: "Create a task for the demo" 2. assistant: "Task created: Demo preparation" 3. user: "What's the deadline?" 4. assistant: "The deadline is December 31st" 5. user: "Can you extend it by one week?" **Why reverse after loading?** ChatKit's `order="desc"` gives newest messages first (efficient pagination), but agents expect chronological order (oldest → newest) for conversation flow. * * * Serializing History for Agent Context[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#serializing-history-for-agent-context "Direct link to Serializing History for Agent Context") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Agents consume conversation history as formatted strings in their system prompt or as message arrays. ChatKit's `ThreadItem` objects need conversion to agent-consumable format. **ThreadItem Structure** (simplified): @dataclassclass ThreadItem: id: str # Message identifier role: Literal["user", "assistant", "tool"] content: str # Message text created_at: int # Unix timestamp # ... additional fields ### Building History String[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#building-history-string "Direct link to Building History String") Convert `ThreadItem` list to formatted conversation: def serialize_history(items: list[ThreadItem]) -> str: """Convert thread items to conversation history string.""" lines = [] for item in items: if item.role == "user": lines.append(f"User: {item.content}") elif item.role == "assistant": lines.append(f"Assistant: {item.content}") # Skip tool messages (internal agent operations) return "\n".join(lines)# Usage in respond()history_string = serialize_history(history_items)# Include in agent system promptsystem_prompt = f"""You are a TaskManager assistant.Previous conversation:{history_string}Current request follows. Maintain continuity with conversation history.""" **Output**: You are a TaskManager assistant.Previous conversation:User: Create a task for the demoAssistant: Task created: Demo preparationUser: What's the deadline?Assistant: The deadline is December 31stUser: Can you extend it by one week?Current request follows. Maintain continuity with conversation history. ### Alternative: Message Array Format[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#alternative-message-array-format "Direct link to Alternative: Message Array Format") Some agent frameworks (like OpenAI SDK) accept message arrays: def serialize_to_messages(items: list[ThreadItem]) -> list[dict]: """Convert thread items to message array format.""" messages = [] for item in items: if item.role in ("user", "assistant"): messages.append({ "role": item.role, "content": item.content }) return messages# Usage with OpenAI SDKfrom agents import Agent, Runnerhistory_messages = serialize_to_messages(history_items)result = Runner.run_streamed( agent=agent, messages=[ *history_messages, # Previous conversation {"role": "user", "content": input.content} # Current input ]) **Output** (message array): [ {"role": "user", "content": "Create a task for the demo"}, {"role": "assistant", "content": "Task created: Demo preparation"}, {"role": "user", "content": "What's the deadline?"}, {"role": "assistant", "content": "The deadline is December 31st"}, {"role": "user", "content": "Can you extend it by one week?"}] **Choosing serialization format**: Use string format for simple agents (concatenate into system prompt), use message array for SDK-based agents (Runner.run\_streamed() accepts messages parameter). * * * Context Window Management Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#context-window-management-strategies "Direct link to Context Window Management Strategies") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Not all conversation history fits in the agent's context window. Long threads require pruning strategies to stay within token limits. | Strategy | When to Use | Trade-offs | | --- | --- | --- | | **Recent Window** | General conversations | Loses older context but maintains recent continuity | | **Token-Aware Pruning** | Production systems | Prevents context overflow, requires token counting | | **Semantic Summarization** | Long-running support threads | Preserves key information, adds summarization cost | | **Conversation Branching** | Multi-topic threads | Isolates topics, requires explicit branching UI | ### Recent Window Strategy (Simplest)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#recent-window-strategy-simplest "Direct link to Recent Window Strategy (Simplest)") Load fixed number of recent messages: # Load last 20 messagespage = await self.store.load_thread_items( thread.id, after=None, limit=20, order="desc", context=context,)history_items = page.datahistory_items.reverse() **Pros**: Simple, predictable memory usage **Cons**: Hard cutoff may lose important context mid-conversation ### Token-Aware Pruning Strategy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#token-aware-pruning-strategy "Direct link to Token-Aware Pruning Strategy") Calculate token count and truncate to model limits: import tiktoken # OpenAI tokenizerdef prune_to_token_limit( items: list[ThreadItem], max_tokens: int = 8000 # Leave room for response) -> list[ThreadItem]: """Keep most recent messages within token budget.""" encoding = tiktoken.encoding_for_model("gpt-4") pruned_items = [] total_tokens = 0 # Process newest → oldest, keep until budget exceeded for item in reversed(items): item_tokens = len(encoding.encode(item.content)) if total_tokens + item_tokens > max_tokens: break # Stop before exceeding budget pruned_items.insert(0, item) # Maintain chronological order total_tokens += item_tokens return pruned_items# Usagepage = await self.store.load_thread_items( thread.id, after=None, order="desc", context=context,)history_items = prune_to_token_limit(page.data, max_tokens=8000) **Output** (with 15,000 token conversation): Original: 42 messages (15,247 tokens)Pruned: 28 messages (7,983 tokens)Removed 14 oldest messages to fit context window **When to use**: Production systems where context overflow causes failures. Requires `tiktoken` library (`uv add tiktoken`). * * * Extract Your conversation-history Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#extract-your-conversation-history-skill "Direct link to Extract Your conversation-history Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've implemented history loading, serialization, and context management. This pattern will recur across every ChatKit server you build—customer support, code review, content moderation. Time to harvest this intelligence. **Layer 3 trigger**: Pattern recurs 2+ times across projects → Create reusable intelligence ### Step: Extract the Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#step-extract-the-skill "Direct link to Step: Extract the Skill") Copy and paste this prompt: Using your skill-creator skill, study the conversation history implementationcode I just wrote in this lesson and extract it into a reusableconversation-history skill.Encapsulate: loading thread history with Page.data extraction, serializationpatterns (string/array formats), token-aware pruning with tiktoken, andcommon pitfalls with fixes. Claude will: 1. Review the implementation code from this lesson 2. Use `skill-creator` to generate the skill structure 3. Populate patterns: load history, serialize, prune tokens 4. Create reference docs with string/array serialization formats 5. Add verification checklist matching canonical format Your skill appears at `.claude/skills/conversation-history/`. * * * Integrating History into Agent Workflow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#integrating-history-into-agent-workflow "Direct link to Integrating History into Agent Workflow") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With the skill created, apply it to your ChatKit server: from chatkit.server import ChatKitServerfrom chatkit.agents import stream_agent_responsefrom agents import Agent, Runnerclass TaskManagerServer(ChatKitServer): async def respond(self, thread, input, context): # STEP 1: Load conversation history (conversation-history skill) page = await self.store.load_thread_items( thread.id, after=None, limit=20, order="desc", context=context, ) history_items = page.data history_items.reverse() # STEP 2: Serialize to message array format history_messages = [ {"role": item.role, "content": item.content} for item in history_items if item.role in ("user", "assistant") ] # STEP 3: Create agent with history context agent = Agent( name="TaskManager", instructions="You are a task management assistant. Maintain conversation continuity.", tools=[] # Add task management tools ) # STEP 4: Run agent with history result = Runner.run_streamed( agent=agent, messages=[ *history_messages, # Previous conversation {"role": "user", "content": input.content} # Current input ] ) # STEP 5: Stream response async for event in stream_agent_response(context, result): yield event **Output** (user continues conversation from previous session): User: Can you extend it by one week?[Agent loads history, sees previous deadline discussion]Agent: Extended deadline from December 31st to January 7th. Task updated. Agent remembers the December 31st deadline from history and calculates the one-week extension correctly. **Safety Note**: Always validate thread ownership in production (check `context.user_id` matches thread owner) before loading history. Multi-tenant systems must enforce thread access controls to prevent unauthorized history access. * * * Try With AI: Apply Your conversation-history Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#try-with-ai-apply-your-conversation-history-skill "Direct link to Try With AI: Apply Your conversation-history Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've extracted the conversation history patterns into a reusable skill. Now test it on new scenarios. ### Prompt 1: Semantic Conversation Branching[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#prompt-1-semantic-conversation-branching "Direct link to Prompt 1: Semantic Conversation Branching") Using my conversation-history skill, implement conversation branching thatloads history from a specific message ID (not just the most recent 20).This enables users to "rewind" the conversation to an earlier point andcontinue from there, creating a conversation tree. **What you're learning**: How to extend skill patterns beyond the basic implementation. Your skill provides the foundation (load history, serialize), and you add branching logic on top. ### Prompt 2: Multi-User Conversation Filtering[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#prompt-2-multi-user-conversation-filtering "Direct link to Prompt 2: Multi-User Conversation Filtering") Using my conversation-history skill, modify the history loading to filter byuser_id. This enables group chat scenarios where each user only sees messagesrelevant to them (private DMs mixed with group messages). **What you're learning**: How to compose skills with security requirements. Your conversation-history skill handles loading/serialization; you add access control filtering. ### Prompt 3: Semantic Summarization[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#prompt-3-semantic-summarization "Direct link to Prompt 3: Semantic Summarization") Using my conversation-history skill, add a summarization layer that embedsrecent messages verbatim but summarizes older messages to save tokens.Recent 10 messages: Full textMessages 11-50: Summarized to 2 sentences eachMessages 51+: Single paragraph summary of entire history **What you're learning**: How skills become platforms for experimentation. Your conversation-history skill provides the primitives (load, serialize, prune); you layer advanced features (embedding, summarization) on top without rewriting the foundation. * [Loading Thread History from Store](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#loading-thread-history-from-store) * [Loading Recent History](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#loading-recent-history) * [Serializing History for Agent Context](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#serializing-history-for-agent-context) * [Building History String](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#building-history-string) * [Alternative: Message Array Format](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#alternative-message-array-format) * [Context Window Management Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#context-window-management-strategies) * [Recent Window Strategy (Simplest)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#recent-window-strategy-simplest) * [Token-Aware Pruning Strategy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#token-aware-pruning-strategy) * [Extract Your conversation-history Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#extract-your-conversation-history-skill) * [Step: Extract the Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#step-extract-the-skill) * [Integrating History into Agent Workflow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#integrating-history-into-agent-workflow) * [Try With AI: Apply Your conversation-history Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#try-with-ai-apply-your-conversation-history-skill) * [Prompt 1: Semantic Conversation Branching](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#prompt-1-semantic-conversation-branching) * [Prompt 2: Multi-User Conversation Filtering](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#prompt-2-multi-user-conversation-filtering) * [Prompt 3: Semantic Summarization](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/conversation-history-management#prompt-3-semantic-summarization) --- # Build Your Claude Agent SDK Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Before learning Anthropic's Claude Agent SDK—the architecture behind Claude Code itself—you'll **own** a Claude Agent SDK skill. * * * Step 1: Get the Skills Lab[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#step-1-get-the-skills-lab "Direct link to Step 1: Get the Skills Lab") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Go to [github.com/panaversity/claude-code-skills-lab](https://github.com/panaversity/claude-code-skills-lab) 2. Click the green **Code** button 3. Select **Download ZIP** 4. Extract the ZIP file 5. Open the extracted folder in your terminal cd claude-code-skills-labclaude * * * Step 2: Create Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#step-2-create-your-skill "Direct link to Step 2: Create Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Copy and paste this prompt: Using your skill creator skill create a new skill for Claude Agent SDK(Anthropic's agent framework). I will use it to build AI agents fromhello world to professional production systems. Use context7 skill tostudy official documentation and then build it so no self assumed knowledge. Claude will: 1. Fetch official Claude Agent SDK documentation via Context7 2. Ask you clarifying questions (tool patterns, subagent needs, hook requirements) 3. Create the complete skill with references and templates Your skill appears at `.claude/skills/claude-agent-sdk/`. * * * Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#done "Direct link to Done") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- You now own a Claude Agent SDK skill built from official documentation. The rest of this chapter teaches you what it knows—and how to make it better. **Next: Lesson 1 — SDK Architecture** Checking access... * [Step 1: Get the Skills Lab](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#step-1-get-the-skills-lab) * [Step 2: Create Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#step-2-create-your-skill) * [Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/anthropic-agents-kit-development/build-your-claude-agent-skill#done) --- # Displaying Tasks | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Time to transition from greeting to tasks. In this lesson, you'll build a widget that displays a list of tasks—the core of TaskManager. Along the way, you'll learn a critical pattern: separating what the model sees from what the widget sees. Why does this matter? Imagine your task list has 100 items. If you put all 100 in `structuredContent`, the model might try to summarize every single one, producing a verbose response. By using `_meta`, you give the widget the full list while the model only sees "You have 100 tasks." The Two Data Channels[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#the-two-data-channels "Direct link to The Two Data Channels") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your tool response has two ways to send data: | Field | Who Sees It | Best For | | --- | --- | --- | | `structuredContent` | Model + Widget | Summary counts, status info | | `_meta` (via `toolResponseMetadata`) | Widget only | Full data lists, sensitive info | Both are accessible in the widget, but only `structuredContent` influences the model's response. Updating to a Task List[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#updating-to-a-task-list "Direct link to Updating to a Task List") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Replace your greeting server with a task list server. Update `main.py`: import mcp.types as typesfrom mcp.server.fastmcp import FastMCPfrom datetime import datetimeMIME_TYPE = "text/html+skybridge"# In-memory task storage (we'll persist later)TASKS = [ {"id": 1, "title": "Buy groceries", "done": False}, {"id": 2, "title": "Review pull request", "done": False}, {"id": 3, "title": "Call mom", "done": True},]WIDGET_HTML = '''

TaskManager

Loading...

    '''mcp = FastMCP("TaskManager")@mcp.tool()def show_tasks() -> types.CallToolResult: """Display the task list widget.""" pending = len([t for t in TASKS if not t["done"]]) completed = len([t for t in TASKS if t["done"]]) return types.CallToolResult( content=[types.TextContent( type="text", text=f"Showing {len(TASKS)} tasks ({pending} pending)" )], # Model sees: just the summary structuredContent={ "total": len(TASKS), "pending": pending, "completed": completed, }, _meta={ # Widget sees: full task list "tasks": TASKS, # Plus the embedded widget "openai.com/widget": types.EmbeddedResource( type="resource", resource=types.TextResourceContents( uri="ui://tasks", mimeType=MIME_TYPE, text=WIDGET_HTML, ) ) } )if __name__ == "__main__": import uvicorn app = mcp.sse_app() uvicorn.run(app, host="0.0.0.0", port=8001) Understanding the Data Split[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#understanding-the-data-split "Direct link to Understanding the Data Split") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **What the model sees (`structuredContent`):** { "total": 3, "pending": 2, "completed": 1} The model can narrate: "You have 3 tasks. 2 are pending and 1 is completed." **What the widget sees (`_meta`):** { "tasks": [ {"id": 1, "title": "Buy groceries", "done": false}, {"id": 2, "title": "Review pull request", "done": false}, {"id": 3, "title": "Call mom", "done": true} ]} The widget renders the full list with titles and checkboxes. ### Accessing Both in the Widget[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#accessing-both-in-the-widget "Direct link to Accessing Both in the Widget") // Summary from structuredContentconst summary = window.openai?.toolOutput;// Full data from _metaconst meta = window.openai?.toolResponseMetadata;const tasks = meta?.tasks || []; Two different API properties: * `toolOutput` → `structuredContent` * `toolResponseMetadata` → `_meta` Why This Pattern Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#why-this-pattern-matters "Direct link to Why This Pattern Matters") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With 3 tasks, the difference is subtle. But imagine 100 tasks: **Without separation (all in structuredContent):** The model might respond: > "Here are your tasks: 1. Buy groceries (pending), 2. Review pull request (pending), 3. Call mom (done), 4. Schedule dentist (pending), 5. Fix bug #123 (pending)..." \[continues for 100 items\] This wastes tokens and annoys users. **With separation:** The model responds: > "You have 100 tasks. 73 are pending and 27 are completed." The widget shows the full interactive list. Best of both worlds. Testing Your Task List[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#testing-your-task-list "Direct link to Testing Your Task List") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Restart your server 2. In ChatGPT: "Show my tasks" 3. You should see: * The widget with 3 tasks (checkboxes and titles) * Model narrating the summary: "3 tasks, 2 pending" 4. Click "Refresh" to reload What You Built[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#what-you-built "Direct link to What You Built") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Building on previous lessons: * Switched from greeting to task list * Used `structuredContent` for model summary * Used `_meta` for full widget data * Rendered dynamic list from server data Your TaskManager now displays tasks. In the next lesson, you'll add Complete and Delete buttons using `callTool`. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#try-with-ai "Direct link to Try With AI") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Add Empty State[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#prompt-1-add-empty-state "Direct link to Prompt 1: Add Empty State") When there are no tasks, the widget should show "No tasks yet! Add one to get started."Update the JavaScript to handle the empty tasks array case. **What you're learning:** Defensive UI programming. Real apps need to handle edge cases gracefully. ### Prompt 2: Style Pending vs Done[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#prompt-2-style-pending-vs-done "Direct link to Prompt 2: Style Pending vs Done") Make pending tasks have a purple left border.Make completed tasks have a green left border and lighter background.Keep the checkbox styling as is. **What you're learning:** Visual hierarchy in task management UIs. Users should instantly see what needs attention. ### Prompt 3: Add Task Count to Title[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#prompt-3-add-task-count-to-title "Direct link to Prompt 3: Add Task Count to Title") Change the widget title from "TaskManager" to "TaskManager (3 tasks)" where the number updates based on the actual task count. Use the summary data from structuredContent. **What you're learning:** Combining summary data with UI elements. The header becomes dynamic while keeping the full list in \_meta. Checking access... * [The Two Data Channels](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#the-two-data-channels) * [Updating to a Task List](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#updating-to-a-task-list) * [Understanding the Data Split](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#understanding-the-data-split) * [Accessing Both in the Widget](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#accessing-both-in-the-widget) * [Why This Pattern Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#why-this-pattern-matters) * [Testing Your Task List](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#testing-your-task-list) * [What You Built](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#what-you-built) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#try-with-ai) * [Prompt 1: Add Empty State](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#prompt-1-add-empty-state) * [Prompt 2: Style Pending vs Done](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#prompt-2-style-pending-vs-done) * [Prompt 3: Add Task Count to Title](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/response-payload-design#prompt-3-add-task-count-to-title) --- # Build Your MCP-Wrapping Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C In the previous lesson, you analyzed how skills layer intelligence on top of MCP servers. You saw how `fetching-library-docs` achieves ~77% token reduction through careful filtering. You understood the persona, questions, and principles that guide intelligent decision-making. Now you're going to build one. This lesson is different from most programming tutorials. We're not starting with code. We're starting with a **specification**—a clear statement of what your skill should do, why it matters, and how you'll know it works. Once that specification is solid, the implementation becomes straightforward. You'll work collaboratively with AI to build the skill, discovering patterns together rather than following pre-written steps. By the end of this lesson, you'll have a production-ready MCP-wrapping skill that: * Wraps an MCP server from Chapter 67 with intelligent filtering * Reduces token consumption by 30%+ through targeted result filtering * Handles errors gracefully with fallback strategies * Works across multiple query scenarios from your domain Let's build it together. * * * Step 1: Specification First[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-1-specification-first "Direct link to Step 1: Specification First") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The most common failure in skill development happens here: developers skip the specification and jump to coding. They think they know what they're building, start writing code, and discover halfway through that their approach doesn't match their actual needs. We're not doing that. Before touching code, you'll write a **specification** that answers four critical questions: 1. **What MCP server are you wrapping?** (intent, library, use cases) 2. **What intelligent decisions does your skill make?** (persona, questions, filtering logic) 3. **How will you measure success?** (token efficiency targets, test coverage) 4. **What happens when something fails?** (error recovery strategy) ### Your First Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#your-first-specification "Direct link to Your First Specification") Choose an MCP server from Chapter 67 that you've built or studied. This could be: * A documentation fetcher (like Context7 in Chapter 67 examples) * A database query tool * A file processor * A code analysis service * Any tool that returns results larger than typical Claude context Now write your spec. Use this template: # Specification: [Your Skill Name]## Intent[What problem does this skill solve? Who uses it? When do they invoke it?]## MCP Server Reference[Which MCP server does this wrap? What tool does it expose?]## Success Criteria- Token efficiency: Achieve 30%+ reduction vs raw MCP output- Accuracy: Returned results must satisfy 100% of query requirements- Coverage: Works across [describe domain variety: "3+ different documentation libraries", "5+ query patterns", etc.]- Reliability: Graceful error handling for [describe failure modes: "missing resources", "timeout", "malformed queries"]## Filtering Strategy[What logic will you use to filter MCP results?- Token counting before/after- Relevance scoring criteria- Content extraction rules- Edge case handling]## Test Scenarios[List 3-5 concrete test cases you'll validate against:1. Scenario: [user query] → Expected output: [what should be returned]2. Scenario: [edge case] → Expected output: [how should skill handle it] ...] Take 10 minutes to write this. Don't overthink it—your specification will evolve as you implement. The goal is clarity, not perfection. **Example Specification** (You'll write your own): # Specification: Python-API-Documentation-Fetcher## IntentFetch Python standard library and popular package documentation with automatic filtering for code examples and signatures. Activated when user asks "How do I use X?" or "Show me examples of X."## MCP Server ReferenceDocumentation-Context MCP (Chapter 67) exposing `fetch_docs` tool.## Success Criteria- Token efficiency: Achieve 40%+ reduction vs raw Context7 output (target: 800 tokens → 480 tokens)- Accuracy: Returned examples match user's experience level (beginner vs advanced)- Coverage: Works across Python stdlib, NumPy, Pandas, Requests- Reliability: Graceful "Not found" message when documentation unavailable## Filtering Strategy1. Extract code blocks (marked with ```python)2. Extract function signatures (def keyword + parameters)3. Extract usage notes (marked with # Note: or # Warning:)4. Score by relevance to query keywords5. Return top 5 highest-scoring segments## Test Scenarios1. "How do I use list comprehensions?" → Returns 3-4 examples, signature, one warning2. "What's NumPy reshape?" → Returns reshape() signature, 2 examples, constraint notes3. "Pandas concat invalid" → Returns "Not found" with suggestion to try "merge"4. "Show me decorators" → Returns decorator patterns, syntax, common mistakes5. "urllib3 connection pooling" → Returns class structure, pool configuration, timeout example * * * Step 2: Persona and Questions Design[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-2-persona-and-questions-design "Direct link to Step 2: Persona and Questions Design") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you have a specification, let's translate it into the persona and questions your skill will use. Your **persona** is the expert identity that activates your skill's decision-making. It's not "You are a Python expert" (generic). It's the specific stance your skill takes. For the documentation fetcher: **Persona**: "You are an API documentation specialist focused on code clarity. Your role is to (1) identify which library and feature the user is asking about, (2) fetch documentation via MCP, and (3) extract and return ONLY code examples, function signatures, and critical constraints that match the user's apparent experience level." This persona tells us exactly what the skill optimizes for: clarity through selective extraction. Now write the **questions** your skill asks itself to make decisions: 1. **Which library is the user asking about?** (Determines which MCP context to fetch) 2. **What specific feature or concept?** (Narrows filtering scope) 3. **What's the user's apparent experience level?** (Filters for beginner vs advanced examples) 4. **Are they looking for examples, syntax, or both?** (Determines extraction priority) 5. **What constraints matter most?** (Performance, security, compatibility?) These questions drive your filtering logic. Each question you answer shapes what gets filtered and what gets returned. **Your turn**: Write a persona and 5 questions for your skill. Make them specific to your domain, not generic. The persona should describe what decisions the skill makes. The questions should be answerable from examining the user's query. * * * Step 3: Building the Skill Collaboratively[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-3-building-the-skill-collaboratively "Direct link to Step 3: Building the Skill Collaboratively") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is where the real learning happens. You're not going to build this skill alone. You and AI are going to collaborate—discovering execution patterns together, incorporating your domain constraints, and iterating toward a solution neither of you would have produced independently. Here's how the collaboration unfolds: ### Part 1: Learning Execution Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-1-learning-execution-patterns "Direct link to Part 1: Learning Execution Patterns") Ask AI to show you how to initialize an MCP client and call a tool. Don't just ask "how do I use MCP client?" Ask a specific question that reveals the pattern: **Your prompt**: I'm building a skill that wraps an MCP server.I need to initialize an MCP StdIO client that connects to my custom MCP server,then call a specific tool and process the response.Show me the pattern for:1. Initializing the StdIO client2. Calling a tool with parameters3. Extracting the tool result4. Counting tokens in the response before and after filteringInclude error handling for "tool not found" and "timeout" scenarios. AI will show you patterns you might not have discovered—maybe using context managers, error recovery strategies, token counting libraries. You'll see approaches that expand your thinking. ### Part 2: Refining With Your Domain Knowledge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-2-refining-with-your-domain-knowledge "Direct link to Part 2: Refining With Your Domain Knowledge") Now AI has suggested an approach. But your domain has specific requirements that AI doesn't know about. You refine the implementation: **Your response to AI**: This pattern helps, but I need to adjust for my domain constraints:1. My queries often include comparison requests ("Compare X and Y libraries"). Your current filtering returns single-library results. How do we modify the filtering to support multi-library comparison?2. My users range from beginners (want simple examples) to advanced (want performance notes). I need to detect experience level from the query and adjust filtering accordingly. How would you score relevance differently for beginner vs advanced?3. The MCP server timeout is 5 seconds. If it times out, I should return a cached result from previous queries on the same library (if available). How would you implement caching without storing full responses? You're articulating your actual requirements, not accepting generic solutions. This is where your domain expertise shapes the implementation. ### Part 3: Iterating Toward a Robust Solution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-3-iterating-toward-a-robust-solution "Direct link to Part 3: Iterating Toward a Robust Solution") AI responds to your constraints. You test the solution. It doesn't work quite right. You iterate: **Round 1**: * AI suggests filtering by keyword scoring * You test: some irrelevant results still pass through **Round 2**: * You point out: "The scoring weights aren't working. Simple examples are being filtered out for advanced queries." * AI refines: suggests weighted scoring where query keywords get 60% weight, content type (example vs note) gets 40% * You test: better, but still not right **Round 3**: * You notice: "The experience level detection is too simplistic. 'decorator' is beginner in one context, advanced in another." * AI proposes: use question marks and "how do I" as beginner signals, "performance", "optimization" as advanced signals * You test: now working well for 80% of queries **Round 4**: * You discover: "The remaining 20% are comparison queries that don't fit either pattern." * AI suggests: detection for "vs" and "compared to" keywords triggers comparison mode with different filtering * You test: all patterns working What emerged is a solution neither of you had at the start. The robustness came through iteration, with each round addressing a specific limitation. * * * Step 4: Implementation and Testing[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-4-implementation-and-testing "Direct link to Step 4: Implementation and Testing") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now let's build the actual skill. The specification and persona guide our implementation, so we know exactly what we're building. ### Skeleton Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#skeleton-implementation "Direct link to Skeleton Implementation") Your skill has this structure: # skill.md (SKILL.md structure from Chapter 3)persona: | [Your persona from Step 2]questions: - [Question 1] - [Question 2] - [Question 3] - [Question 4] - [Question 5]principles: - Fail gracefully: Return meaningful error messages if MCP unavailable - Filter ruthlessly: Prioritize clarity over comprehensiveness - Verify early: Check MCP connection before processing queries - Cache strategically: Store successful results for timeout recovery# Implementation follows: ### Key Implementation Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#key-implementation-patterns "Direct link to Key Implementation Patterns") **Pattern 1: MCP Client Initialization** from mcp.client.stdio import StdIOClient# Initialize client with your MCP serverclient = StdIOClient( command="python", args=["-m", "your_mcp_server_module"], timeout=5.0)# Connect and verifywith client: # Tool calls happen inside this context result = client.call_tool("fetch_docs", {"library": "numpy", "topic": "reshape"}) The context manager ensures proper cleanup. The timeout prevents hanging. Keep this pattern consistent. **Pattern 2: Token Counting Before/After** import tiktokendef count_tokens(text: str) -> int: """Count tokens in text using OpenAI's tokenizer.""" encoding = tiktoken.encoding_for_model("gpt-4") return len(encoding.encode(text))# Before filteringraw_response = client.call_tool(...)tokens_before = count_tokens(raw_response)# After filteringfiltered_response = filter_results(raw_response, user_query)tokens_after = count_tokens(filtered_response)# Calculate savingssavings_percent = ((tokens_before - tokens_after) / tokens_before) * 100print(f"Token savings: {savings_percent:.1f}%") **Pattern 3: Result Filtering Logic** Your filtering logic depends on your domain, but here's a template: def filter_results(raw_response: str, user_query: str) -> str: """ Filter raw MCP response based on user query and experience level. Args: raw_response: Full response from MCP server user_query: Original user question Returns: Filtered response containing only relevant segments """ # Step 1: Detect experience level experience_level = detect_experience_level(user_query) # "beginner" or "advanced" # Step 2: Extract candidates code_examples = extract_code_blocks(raw_response) signatures = extract_function_signatures(raw_response) notes = extract_notes(raw_response, experience_level) # Step 3: Score by relevance scored_examples = [ (example, relevance_score(example, user_query)) for example in code_examples ] scored_examples.sort(key=lambda x: x[1], reverse=True) # Step 4: Return top N results selected = scored_examples[:5] # Top 5 by relevance return format_output(signatures, selected, notes) ### Your Implementation Task[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#your-implementation-task "Direct link to Your Implementation Task") Using your specification and persona as a guide: 1. **Write the skill.md structure** (persona, questions, principles) 2. **Implement MCP client initialization** (handle the specific MCP server you're wrapping) 3. **Implement filtering logic** matching your specification's success criteria 4. **Add error handling** for timeout, missing resources, invalid queries 5. **Test with your test scenarios** from Step 1 specification Use the patterns above as templates. Adapt them to your specific MCP server and filtering needs. * * * Step 5: Documentation for Skill Consumers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-5-documentation-for-skill-consumers "Direct link to Step 5: Documentation for Skill Consumers") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ After testing, you'll write documentation so others can use your skill. This is the final validation: if you can't explain clearly how to use your skill, you haven't understood it well enough. ### Usage Guide Template[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#usage-guide-template "Direct link to Usage Guide Template") # [Skill Name] Usage Guide## What This Skill Does[Explain the skill's purpose in 1-2 sentences. Who uses it? When?]## When to Activate This SkillYour skill activates automatically when:- User query contains keywords matching your activation conditions- Context suggests the user needs [your domain]- [Other trigger conditions specific to your skill]## Success MetricsThis skill is working well when:- Token efficiency: Results show 30%+ reduction vs raw MCP output- Accuracy: Returned results directly answer the user's question- Completeness: All critical information is included (not overly filtered)- Speed: MCP call completes within timeout (handle gracefully if not)## Example Queries| Query | What Skill Should Do | Example Output || ----------------------- | -------------------- | --------------- || [Example 1] | [Expected behavior] | [Sample result] || [Example 2] | [Expected behavior] | [Sample result] || [Example 3 - edge case] | [Expected behavior] | [Sample result] |## Constraints and Limitations- [List what this skill does NOT do]- [List known limitations]- [List scenarios where fallback behavior triggers] Write your usage guide after testing. It forces you to articulate what your skill actually does vs what you thought it would do. * * * Try With AI: Building an MCP-Wrapping Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#try-with-ai-building-an-mcp-wrapping-skill "Direct link to Try With AI: Building an MCP-Wrapping Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now it's your turn to build. This "Try With AI" section walks you through the complete process using AI collaboration. ### Part 1: Specification Development[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-1-specification-development "Direct link to Part 1: Specification Development") Start by clarifying your skill's purpose: **Prompt 1 - Specification Planning**: I'm building a skill that wraps an MCP server. I want to wrap [your chosen MCP server]to provide [your domain/use case].Help me develop a clear specification covering:1. What problem does this skill solve?2. What MCP tool does it call?3. What filtering logic will reduce token consumption?4. What failure modes might occur?5. How will I measure success?Let me be specific about my domain:[Describe your specific use case, domain, and target users] This step clarifies what you're building before you start coding. Specification quality directly determines implementation quality. * * * ### Part 2: Pattern Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-2-pattern-implementation "Direct link to Part 2: Pattern Implementation") Once you have your specification, ask AI to show you the execution patterns: **Prompt 2 - MCP Client Pattern**: I have this specification for my MCP-wrapping skill:[Paste your spec here]Now show me the pattern for implementing this:1. How do I initialize an MCP client that connects to my server?2. How do I call the MCP tool and handle the response?3. How do I count tokens before and after filtering?4. How do I implement the filtering logic to achieve my token efficiency target?Include error handling for:- MCP server not available- Tool execution timeout- Tool returns empty result- Malformed queryGive me complete, testable code that I can run immediately. This step reveals practical patterns you can adapt to your domain. You see error recovery, token measurement, and client initialization working together. * * * ### Part 3: Constraint Refinement[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-3-constraint-refinement "Direct link to Part 3: Constraint Refinement") Now test the AI's suggested approach and refine based on your actual needs: **Prompt 3 - Domain Constraints**: I implemented the pattern you showed, and I discovered that my domain has specificconstraints that the generic pattern doesn't handle:[Describe 2-3 specific constraints from testing:- Example: "Queries often ask for comparisons between two libraries"- Example: "My users range from beginner to advanced, and filtering should adapt"- Example: "The MCP server is slow for large libraries, so I need caching"]How do I modify the filtering logic to handle these constraints while maintainingmy token efficiency target? Show me concrete code changes that address each constraint.Here's what I've tried so far:[Share a code snippet or description of your attempt]What's not working:[Describe the specific failure mode or limitation you hit] This step addresses the gap between generic patterns and your domain reality. Real testing reveals which constraints matter most. * * * ### Part 4: Testing and Iteration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-4-testing-and-iteration "Direct link to Part 4: Testing and Iteration") Run your skill against your test scenarios and discover what needs refinement: **Prompt 4 - Testing and Debugging**: I tested my skill with these scenarios from my specification:Test 1: [Your test scenario]Expected: [What should happen]Actual: [What actually happened]Problem: [Why it didn't work]Test 2: [Your test scenario]Expected: [What should happen]Actual: [What actually happened]Problem: [Why it didn't work]I can see the pattern: [Describe the common issue you've discovered]How would you adjust the filtering heuristics or error handling to address this?Show me specific code changes and explain the reasoning. This step shows how testing surfaces patterns that guide refinement. Theory meets practice when your assumptions collide with actual behavior. * * * ### Part 5: Final Validation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-5-final-validation "Direct link to Part 5: Final Validation") Once your skill is working across test scenarios, validate that it meets your specification: **Prompt 5 - Specification Validation**: I've built a working MCP-wrapping skill. Now I need to validate it against myoriginal specification.My spec required:- Token efficiency: [Your target - example: 30%+ reduction]- Accuracy: [Your target - example: 100% of queries answered directly]- Coverage: [Your target - example: Works across 3+ libraries]- Reliability: [Your target - example: Graceful error handling for timeouts]Here are my actual results:- Token efficiency: [Measured - example: 42% reduction achieved]- Accuracy: [Measured - example: 95% of queries directly answered]- Coverage: [Tested - example: Works across React, Vue, Svelte]- Reliability: [Tested - example: Handles timeouts with fallback][For each metric that falls short, ask]:How would you improve this? What's the tradeoff between [metric A] and [metric B]?[For each metric that exceeds target, ask]:Was there an unexpected benefit? What drove the better-than-expected result? This step validates your implementation against your original intent. The differences between specification and reality reveal what you've learned. * * * ### Success Criteria[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#success-criteria "Direct link to Success Criteria") You've successfully completed this lesson when: ✓ **Specification written**: You have a clear spec.md that answers the four core questions (what MCP, what intelligent decisions, how to measure success, error recovery) ✓ **Skill implemented**: Your skill successfully initializes an MCP client, calls the target tool, and returns filtered results ✓ **Token efficiency demonstrated**: Your filtered output achieves 30%+ token reduction compared to raw MCP output (measure with tiktoken) ✓ **Error handling validated**: Your skill gracefully handles at least 3 failure scenarios (timeout, missing resource, malformed query) ✓ **Tests passing**: Your skill produces correct results for all test scenarios in your specification ✓ **Documentation complete**: You can explain to another developer how to use your skill and what it optimizes for ✓ **Collaboration evident**: Your implementation shows how specification, pattern discovery, domain constraints, testing, and validation shaped your solution The skill you've built isn't just code. It's an encoding of expertise in your domain. It knows when to invoke an MCP server, what to ask for, how to filter the results, and how to recover from failure. That's the pattern you'll see again and again as you build more complex skills in Lessons 5-8. * * * ### Reference: Skill-Building Prompts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#reference-skill-building-prompts "Direct link to Reference: Skill-Building Prompts") Use these prompts in sequence, adapting based on your specific MCP server and domain: **Prompt 1: Specification Planning** (What are we building and why?) I'm building a skill that wraps an MCP server [your server name].The problem I'm solving: [your domain problem]The target users: [who needs this]The MCP server capability: [what does it provide]Help me write a specification covering intent, success criteria, filtering strategy,and test scenarios. I want to achieve [your efficiency target]% token reduction. **What you're learning**: Specification clarity—translating vague ideas into measurable requirements before writing any code. **Prompt 2: Pattern Teaching** (How does MCP client work?) Show me the complete pattern for:- Initializing an MCP StdIO client- Calling a tool with parameters- Counting tokens before/after with tiktoken- Implementing filtering that achieves my token efficiency target- Handling errors: timeout, not found, malformed queryInclude the full working code, not just explanations. **What you're learning**: Implementation patterns—recognizing reusable code structures for MCP interaction that you can adapt to any domain. **Prompt 3: Constraint Teaching** (What's specific about my domain?) I tested the pattern you showed, and I need to adapt it for my domain:Constraint 1: [Describe a specific requirement from your domain]Constraint 2: [Describe another specific requirement]Constraint 3: [Describe a third specific requirement]How do I modify the filtering logic to handle these while maintaining token efficiency? **What you're learning**: Domain constraint translation—adapting generic patterns to your specific needs while maintaining efficiency goals. **Prompt 4: Testing and Debugging** (Why isn't it working?) I'm testing against these scenarios:Test case: [Your test]Expected: [What should happen]Actual: [What happened]Problem: [Why it's failing]How do I fix the filtering to handle this case correctly? **What you're learning**: Iterative refinement—using test results to identify and fix design gaps systematically. **Prompt 5: Specification Validation** (Does it meet requirements?) I've built a working skill. Does it match my specification?Spec requirement: [Your requirement]Measured result: [What you measured]Gap (if any): [If it falls short, what's the gap?][Ask for each requirement separately]Overall: Does this skill successfully encode the intelligence layer I specified? **What you're learning**: Validation thinking—measuring whether your implementation meets original intent and identifying any gaps. **Expected outcome**: You have a tested, documented MCP-wrapping skill that demonstrates 30%+ token efficiency and handles error scenarios gracefully. Checking access... * [Step 1: Specification First](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-1-specification-first) * [Your First Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#your-first-specification) * [Step 2: Persona and Questions Design](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-2-persona-and-questions-design) * [Step 3: Building the Skill Collaboratively](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-3-building-the-skill-collaboratively) * [Part 1: Learning Execution Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-1-learning-execution-patterns) * [Part 2: Refining With Your Domain Knowledge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-2-refining-with-your-domain-knowledge) * [Part 3: Iterating Toward a Robust Solution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-3-iterating-toward-a-robust-solution) * [Step 4: Implementation and Testing](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-4-implementation-and-testing) * [Skeleton Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#skeleton-implementation) * [Key Implementation Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#key-implementation-patterns) * [Your Implementation Task](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#your-implementation-task) * [Step 5: Documentation for Skill Consumers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#step-5-documentation-for-skill-consumers) * [Usage Guide Template](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#usage-guide-template) * [Try With AI: Building an MCP-Wrapping Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#try-with-ai-building-an-mcp-wrapping-skill) * [Part 1: Specification Development](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-1-specification-development) * [Part 2: Pattern Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-2-pattern-implementation) * [Part 3: Constraint Refinement](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-3-constraint-refinement) * [Part 4: Testing and Iteration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-4-testing-and-iteration) * [Part 5: Final Validation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#part-5-final-validation) * [Success Criteria](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#success-criteria) * [Reference: Skill-Building Prompts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-mcp-wrapping-skill#reference-skill-building-prompts) --- # Async Session Management | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C The engine connects to the database. Sessions manage individual operations—queries, inserts, updates, deletes. Each request to your FastAPI app gets its own session. Getting sessions right prevents subtle bugs that crash your agent. The Correct Import[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#the-correct-import "Direct link to The Correct Import") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- SQLModel provides an async session wrapper. The import path matters: # CORRECT - SQLModel's async sessionfrom sqlmodel.ext.asyncio.session import AsyncSession# WRONG - SQLAlchemy's directly (works but loses SQLModel features)from sqlalchemy.ext.asyncio import AsyncSession **Why it matters:** SQLModel's `AsyncSession` extends SQLAlchemy's with `exec()` method that works with `select()` statements properly. Session as FastAPI Dependency[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#session-as-fastapi-dependency "Direct link to Session as FastAPI Dependency") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Create a dependency that yields sessions: from sqlmodel.ext.asyncio.session import AsyncSessionfrom collections.abc import AsyncGeneratorasync def get_session() -> AsyncGenerator[AsyncSession]: """Dependency that yields async database sessions.""" async with AsyncSession(engine) as session: yield session **The pattern explained:** 1. `async with AsyncSession(engine)` creates a session 2. `yield session` provides it to the endpoint 3. After endpoint completes, `async with` closes the session Use in endpoints: from fastapi import Depends@router.get("/tasks/{task_id}")async def get_task( task_id: int, session: AsyncSession = Depends(get_session),): task = await session.get(Task, task_id) if not task: raise HTTPException(status_code=404, detail="Task not found") return task **Output:** { "id": 42, "title": "Fix authentication", "status": "pending"} Session Lifecycle[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#session-lifecycle "Direct link to Session Lifecycle") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Each session has a clear lifecycle: Create → Use → Commit/Rollback → Close ↑ ↓ └──── Managed by async with ───┘ **Within a session:** async with AsyncSession(engine) as session: # 1. Create - session starts task = Task(title="New task") session.add(task) # 2. Use - operations queued await session.flush() # Writes to DB, gets ID # 3. Commit - makes permanent await session.commit() # 4. Close - handled by async with **After `async with` exits:** * Session is closed * Connection returns to pool * Any uncommitted changes are rolled back The MissingGreenlet Error[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#the-missinggreenlet-error "Direct link to The MissingGreenlet Error") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This error crashes agents that access relationships incorrectly: # This CRASHES in async codetask = await session.get(Task, task_id)print(task.assignee.name) # MissingGreenlet! **Why it happens:** `task.assignee` triggers a "lazy load"—a database query. But in async code, lazy loads need special handling that isn't automatic. **Solutions:** ### Solution 1: Eager Loading (Recommended)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#solution-1-eager-loading-recommended "Direct link to Solution 1: Eager Loading (Recommended)") Load relationships upfront: from sqlalchemy.orm import selectinloadstmt = ( select(Task) .options(selectinload(Task.assignee)) .where(Task.id == task_id))result = await session.exec(stmt)task = result.one()print(task.assignee.name) # Works! Already loaded ### Solution 2: Explicit Query[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#solution-2-explicit-query "Direct link to Solution 2: Explicit Query") Fetch related data separately: task = await session.get(Task, task_id)assignee = await session.get(Worker, task.assignee_id)print(assignee.name) # Works! Explicit query ### Solution 3: AsyncAttrs (Advanced)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#solution-3-asyncattrs-advanced "Direct link to Solution 3: AsyncAttrs (Advanced)") Use SQLAlchemy's AsyncAttrs mixin: from sqlalchemy.ext.asyncio import AsyncAttrsclass Task(AsyncAttrs, SQLModel, table=True): assignee: "Worker" = Relationship(back_populates="tasks")# Usagetask = await session.get(Task, task_id)assignee = await task.awaitable_attrs.assignee # Awaitable lazy loadprint(assignee.name) expire\_on\_commit=False[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#expire_on_commitfalse "Direct link to expire_on_commit=False") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, SQLAlchemy expires (invalidates) object attributes after commit. This triggers lazy loads when you access them: # Default behavior (expire_on_commit=True)session.add(task)await session.commit()print(task.title) # Triggers lazy load → MissingGreenlet! **Solution:** Disable expiration: from sqlalchemy.ext.asyncio import async_sessionmakerasync_session = async_sessionmaker(engine, expire_on_commit=False)async def get_session() -> AsyncGenerator[AsyncSession]: async with async_session() as session: yield session **Or:** Always refresh after commit: session.add(task)await session.commit()await session.refresh(task) # Reloads from DBprint(task.title) # Works! Complete Session Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#complete-session-configuration "Direct link to Complete Session Configuration") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Production-ready session setup. Create `database.py`: from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmakerfrom sqlmodel import SQLModelfrom sqlmodel.ext.asyncio.session import AsyncSessionfrom collections.abc import AsyncGeneratorengine = create_async_engine( DATABASE_URL, pool_pre_ping=True, pool_size=5,)# Session factory with expire_on_commit=Falseasync_session_factory = async_sessionmaker( engine, class_=AsyncSession, expire_on_commit=False,)async def get_session() -> AsyncGenerator[AsyncSession]: """Dependency that yields async database sessions.""" async with async_session_factory() as session: yield session **Output (using in endpoint):** @router.post("/tasks")async def create_task( data: TaskCreate, session: AsyncSession = Depends(get_session),): task = Task(**data.model_dump()) session.add(task) await session.commit() # No refresh needed - expire_on_commit=False return task Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Debug MissingGreenlet[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#prompt-1-debug-missinggreenlet "Direct link to Prompt 1: Debug MissingGreenlet") I'm getting this error in my FastAPI endpoint:"sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called"The error happens on this line: return task.project.nameMy endpoint code:async def get_task(task_id: int, session: AsyncSession = Depends(get_session)): task = await session.get(Task, task_id) return {"title": task.title, "project": task.project.name}What's wrong and how do I fix it? **What you're learning:** MissingGreenlet debugging—recognizing and fixing async relationship access issues. ### Prompt 2: Session Factory Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#prompt-2-session-factory-configuration "Direct link to Prompt 2: Session Factory Configuration") I want to configure my AsyncSession to:1. Not expire attributes after commit2. Use a specific pool size3. Log all SQL statements in developmentShow me the complete database.py with async_sessionmakerand get_session dependency. **What you're learning:** Session factory—centralizing session configuration for consistency. ### Prompt 3: Request Scoping[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#prompt-3-request-scoping "Direct link to Prompt 3: Request Scoping") Explain why each FastAPI request should get its own database session.What problems occur if I share a session across requests?Include code examples showing the wrong and right approach. **What you're learning:** Request scoping—understanding isolation requirements for concurrent access. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#safety-note "Direct link to Safety Note") Never share a session across async tasks. Each `asyncio.gather()` coroutine needs its own session. Sharing causes race conditions and data corruption. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#reflect-on-your-skill "Direct link to Reflect on Your Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built a `relational-db-agent` skill in Lesson 0. Test its session management knowledge. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#test-your-skill "Direct link to Test Your Skill") Using my relational-db-agent skill, generate:1. A get_session() dependency with expire_on_commit=False2. An endpoint that creates a Task and returns it immediately3. Proper error handling if creation fails ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill use `sqlmodel.ext.asyncio.session.AsyncSession`? * Did it configure `expire_on_commit=False`? * Did it structure the dependency with `async with` and `yield`? * Did it include proper imports? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My relational-db-agent skill uses wrong AsyncSession import.Update it to always use:from sqlmodel.ext.asyncio.session import AsyncSessionAnd when creating session factory, use:from sqlalchemy.ext.asyncio import async_sessionmakerasync_session_factory = async_sessionmaker( engine, class_=AsyncSession, expire_on_commit=False,) Your skill now generates correct async session patterns. Checking access... * [The Correct Import](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#the-correct-import) * [Session as FastAPI Dependency](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#session-as-fastapi-dependency) * [Session Lifecycle](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#session-lifecycle) * [The MissingGreenlet Error](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#the-missinggreenlet-error) * [Solution 1: Eager Loading (Recommended)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#solution-1-eager-loading-recommended) * [Solution 2: Explicit Query](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#solution-2-explicit-query) * [Solution 3: AsyncAttrs (Advanced)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#solution-3-asyncattrs-advanced) * [expire\_on\_commit=False](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#expire_on_commitfalse) * [Complete Session Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#complete-session-configuration) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#try-with-ai) * [Prompt 1: Debug MissingGreenlet](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#prompt-1-debug-missinggreenlet) * [Prompt 2: Session Factory Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#prompt-2-session-factory-configuration) * [Prompt 3: Request Scoping](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#prompt-3-request-scoping) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#safety-note) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/async-session-management#improve-your-skill) --- # Chapter 74 Quiz | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Test your understanding of async database patterns. Choose the best answer for each question. * * * Question 1: Async Engine Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-1-async-engine-configuration "Direct link to Question 1: Async Engine Configuration") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What's wrong with this engine configuration for a production PostgreSQL database? engine = create_async_engine( "postgresql://user:pass@localhost/db", pool_size=5,) **A)** Missing `echo=True` for logging **B)** Wrong URL format - should be `postgresql+asyncpg://` **C)** Missing `pool_pre_ping=True` **D)** Both B and C Answer **D) Both B and C** The URL must use the async driver (`postgresql+asyncpg://`), and `pool_pre_ping=True` is essential for managed databases to prevent stale connection errors. * * * Question 2: First Normal Form (1NF)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-2-first-normal-form-1nf "Direct link to Question 2: First Normal Form (1NF)") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Which table violates First Normal Form (1NF)? **A)** A table with `id`, `name`, and `email` columns **B)** A table with a `tags` column containing "bug, urgent, backend" as a single string **C)** A table with a nullable `description` column **D)** A table with an auto-incrementing primary key Answer **B) A table with a `tags` column containing "bug, urgent, backend" as a single string** 1NF requires atomic values. Storing multiple tags in a single string violates atomicity. Use a separate tags table or JSONB array (acceptable in modern databases for simple lists). * * * Question 3: Third Normal Form (3NF)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-3-third-normal-form-3nf "Direct link to Question 3: Third Normal Form (3NF)") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What's wrong with this table from a normalization perspective? | task_id | title | assignee_id | assignee_email ||---------|----------|-------------|-------------------|| 1 | Fix bug | 5 | john@example.com | **A)** Missing primary key **B)** `assignee_email` depends on `assignee_id`, not `task_id` (3NF violation) **C)** Column names are too short **D)** Nothing wrong - this is properly normalized Answer **B) `assignee_email` depends on `assignee_id`, not `task_id` (3NF violation)** 3NF requires no transitive dependencies. `assignee_email` depends on `assignee_id`, which depends on `task_id`. Move email to a separate Workers table. * * * Question 4: When to Denormalize[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-4-when-to-denormalize "Direct link to Question 4: When to Denormalize") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When is denormalization acceptable? **A)** Always - normalized tables are too slow **B)** Never - denormalization causes data anomalies **C)** For audit logs that capture state at time of action **D)** When you don't understand foreign keys Answer **C) For audit logs that capture state at time of action** Audit logs should capture data as it was when an action occurred. If you normalize and the worker's name changes later, your audit log would show the wrong name for historical events. * * * Question 5: Session Import[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-5-session-import "Direct link to Question 5: Session Import") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Which import gives you the correct AsyncSession for SQLModel? **A)** `from sqlalchemy.ext.asyncio import AsyncSession` **B)** `from sqlmodel.ext.asyncio.session import AsyncSession` **C)** `from sqlmodel import AsyncSession` **D)** `from sqlalchemy.orm import AsyncSession` Answer **B) `from sqlmodel.ext.asyncio.session import AsyncSession`** SQLModel's AsyncSession extends SQLAlchemy's with the `exec()` method that works properly with SQLModel's `select()`. * * * Question 3: JSONB Columns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-3-jsonb-columns "Direct link to Question 3: JSONB Columns") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- How do you correctly define a JSONB list column in SQLModel for PostgreSQL? **A)** `tags: list[str] = Field(default_factory=list)` **B)** `tags: list[str] = Field(sa_column=Column(JSONB))` **C)** `tags: list[str] = Field(default_factory=list, sa_column=Column(JSONB, nullable=False, server_default="[]"))` **D)** `tags: JSONB = Field(default=[])` Answer **C) `tags: list[str] = Field(default_factory=list, sa_column=Column(JSONB, nullable=False, server_default="[]"))`** You need both the Python default (`default_factory=list`) and the SQLAlchemy column configuration (`sa_column=Column(JSONB, ...)`) with a server default. * * * Question 4: flush() vs commit()[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-4-flush-vs-commit "Direct link to Question 4: flush() vs commit()") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When should you use `await session.flush()` instead of `await session.commit()`? **A)** Always use flush(), never commit() **B)** When you need the database-assigned ID before the transaction ends **C)** When you want to roll back changes **D)** When using SQLite instead of PostgreSQL Answer **B) When you need the database-assigned ID before the transaction ends** `flush()` writes to the database (assigning IDs) but keeps the transaction open. This lets you use the ID for related records before the final commit. * * * Question 5: MissingGreenlet Error[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-5-missinggreenlet-error "Direct link to Question 5: MissingGreenlet Error") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What causes the MissingGreenlet error in async SQLModel code? **A)** Missing `await` keyword **B)** Accessing a lazy-loaded relationship in async context **C)** Using sync engine instead of async engine **D)** Forgetting to call `session.refresh()` Answer **B) Accessing a lazy-loaded relationship in async context** When you access `task.assignee` without eager loading, SQLAlchemy tries to load it lazily, which requires special greenlet handling that isn't available in standard async contexts. * * * Question 6: N+1 Prevention[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-6-n1-prevention "Direct link to Question 6: N+1 Prevention") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ What's the correct pattern to prevent N+1 queries when loading tasks with their assignees? **A)** `await session.exec(select(Task).join(Worker))` **B)** `await session.exec(select(Task).options(selectinload(Task.assignee)))` **C)** `await session.exec(select(Task)).all(); [await session.get(Worker, t.assignee_id) for t in tasks]` **D)** `await session.exec(select(Task, Worker))` Answer **B) `await session.exec(select(Task).options(selectinload(Task.assignee)))`** `selectinload` loads the related workers in a single additional query using an IN clause, preventing N+1 queries. * * * Question 7: unique() Requirement[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-7-unique-requirement "Direct link to Question 7: unique() Requirement") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Why must you call `result.unique().all()` when using selectinload? **A)** To remove null values **B)** To deduplicate parent objects that may appear multiple times **C)** To sort results **D)** To convert to a list Answer **B) To deduplicate parent objects that may appear multiple times** `selectinload` can create duplicate parent objects in the result set. `unique()` removes these duplicates. * * * Question 8: Self-Referential Relationships[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-8-self-referential-relationships "Direct link to Question 8: Self-Referential Relationships") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What's required in `sa_relationship_kwargs` for a self-referential parent relationship? **A)** `{"lazy": "selectin"}` **B)** `{"remote_side": "Task.id"}` **C)** `{"cascade": "all, delete-orphan"}` **D)** `{"uselist": False}` Answer **B) `{"remote_side": "Task.id"}`** For self-referential relationships, `remote_side` tells SQLAlchemy which side is the "one" in the one-to-many relationship. * * * Question 9: Multiple Foreign Keys[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-9-multiple-foreign-keys "Direct link to Question 9: Multiple Foreign Keys") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When a model has two foreign keys to the same table, what must you specify? **A)** Different table names **B)** `sa_relationship_kwargs={"foreign_keys": "[Model.field_id]"}` **C)** `Relationship(secondary=...)` **D)** `back_populates` with unique names Answer **B) `sa_relationship_kwargs={"foreign_keys": "[Model.field_id]"}`** When there are multiple FKs to the same table, SQLAlchemy can't determine which FK each relationship uses without explicit specification. * * * Question 10: Transaction Rollback[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-10-transaction-rollback "Direct link to Question 10: Transaction Rollback") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What's wrong with this error handling pattern? try: session.add(task) await session.commit()except IntegrityError: raise HTTPException(400, "Error") **A)** Missing `await` before `session.add()` **B)** Should catch `SQLAlchemyError` instead **C)** Missing `await session.rollback()` before raising **D)** HTTPException should be 500 Answer **C) Missing `await session.rollback()` before raising** After an error, you must rollback to return the session to a clean state. Without rollback, subsequent operations on the session may fail or behave unexpectedly. * * * Question 11: Alembic Model Import[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-11-alembic-model-import "Direct link to Question 11: Alembic Model Import") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Why must all models be imported in Alembic's env.py? **A)** To make migrations run faster **B)** So autogenerate can detect tables by comparing metadata **C)** To enable downgrade operations **D)** For type checking Answer **B) So autogenerate can detect tables by comparing metadata** Alembic compares `SQLModel.metadata` to the database schema. If models aren't imported, they're not registered in metadata, and Alembic won't see them. * * * Question 12: Async Alembic[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-12-async-alembic "Direct link to Question 12: Async Alembic") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- What command initializes Alembic with async support? **A)** `alembic init alembic` **B)** `alembic init -t async alembic` **C)** `alembic init --async alembic` **D)** `alembic init alembic --driver=asyncpg` Answer **B) `alembic init -t async alembic`** The `-t async` flag uses the async template, which configures env.py for async database operations. * * * Question 13: expire\_on\_commit[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-13-expire_on_commit "Direct link to Question 13: expire_on_commit") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ What problem does `expire_on_commit=False` solve? **A)** Prevents database connections from expiring **B)** Allows attribute access after commit without MissingGreenlet **C)** Makes commits faster **D)** Prevents automatic rollback Answer **B) Allows attribute access after commit without MissingGreenlet** By default, SQLAlchemy expires attributes after commit, requiring a reload that triggers lazy loading. With `expire_on_commit=False`, attributes remain accessible. * * * Question 14: Session Scope[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-14-session-scope "Direct link to Question 14: Session Scope") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Why should each FastAPI request get its own database session? **A)** Performance optimization **B)** Isolation - concurrent requests don't share transaction state **C)** Memory management **D)** Type safety Answer **B) Isolation - concurrent requests don't share transaction state** Sharing sessions across requests means one request's uncommitted changes are visible to another, and errors in one request can corrupt another's session state. * * * Question 15: Pool Pre-Ping[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-15-pool-pre-ping "Direct link to Question 15: Pool Pre-Ping") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Why is `pool_pre_ping=True` essential for cloud databases? **A)** Improves query performance **B)** Enables connection pooling **C)** Detects and replaces stale connections before use **D)** Reduces connection count Answer **C) Detects and replaces stale connections before use** Cloud databases often close idle connections. `pool_pre_ping` tests connections before use, replacing dead ones so your first query doesn't fail with "connection closed." * * * Scoring[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#scoring "Direct link to Scoring") ----------------------------------------------------------------------------------------------------------------------------------------------------------- | Score | Assessment | | --- | --- | | 16-18 | Excellent - ready for production database work | | 13-15 | Good - review weak areas | | 10-12 | Fair - revisit core lessons | | Below 10 | Review chapter and practice exercises | Next Steps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#next-steps "Direct link to Next Steps") -------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you scored below 16, revisit these lessons: * **Questions 1 wrong**: Review L03 (Engine Setup) * **Questions 2-4 wrong**: Review L02 (Database Design & Normalization) * **Questions 5-7 wrong**: Review L04 (Models) and L05 (Sessions) * **Questions 8-9 wrong**: Review L06 (CRUD Operations) * **Questions 10-13 wrong**: Review L08 (Relationships) * **Questions 14-16 wrong**: Review L09 (Transactions) and L10 (Migrations) * **Questions 17-18 wrong**: Review L03 (Engine Setup) and L05 (Sessions) * [Question 1: Async Engine Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-1-async-engine-configuration) * [Question 2: First Normal Form (1NF)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-2-first-normal-form-1nf) * [Question 3: Third Normal Form (3NF)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-3-third-normal-form-3nf) * [Question 4: When to Denormalize](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-4-when-to-denormalize) * [Question 5: Session Import](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-5-session-import) * [Question 3: JSONB Columns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-3-jsonb-columns) * [Question 4: flush() vs commit()](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-4-flush-vs-commit) * [Question 5: MissingGreenlet Error](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-5-missinggreenlet-error) * [Question 6: N+1 Prevention](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-6-n1-prevention) * [Question 7: unique() Requirement](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-7-unique-requirement) * [Question 8: Self-Referential Relationships](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-8-self-referential-relationships) * [Question 9: Multiple Foreign Keys](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-9-multiple-foreign-keys) * [Question 10: Transaction Rollback](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-10-transaction-rollback) * [Question 11: Alembic Model Import](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-11-alembic-model-import) * [Question 12: Async Alembic](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-12-async-alembic) * [Question 13: expire\_on\_commit](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-13-expire_on_commit) * [Question 14: Session Scope](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-14-session-scope) * [Question 15: Pool Pre-Ping](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#question-15-pool-pre-ping) * [Scoring](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#scoring) * [Next Steps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/chapter-quiz#next-steps) --- # Transactions and Error Handling | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Database transactions ensure your agent's operations succeed completely or fail cleanly. When creating a task with audit logs, either both records exist or neither does. Getting transactions wrong causes data corruption that's hard to debug. flush() vs commit()[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#flush-vs-commit "Direct link to flush() vs commit()") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- These two operations are often confused: | Operation | What It Does | When to Use | | --- | --- | --- | | `flush()` | Writes to DB but doesn't commit | Need auto-assigned values (IDs) | | `commit()` | Makes changes permanent | Operation complete, persist data | async def create_task_with_audit(session: AsyncSession, data: dict): # Create task task = Task(**data) session.add(task) # flush() writes to DB - task.id is now available await session.flush() print(f"Task ID (not yet committed): {task.id}") # Use task.id to create audit log audit = AuditLog( entity_type="task", entity_id=task.id, # Available after flush action="created", ) session.add(audit) # commit() makes both permanent await session.commit() print(f"Transaction committed: Task {task.id} and audit log") **Output:** Task ID (not yet committed): 42Transaction committed: Task 42 and audit log **If you skip flush():** task = Task(**data)session.add(task)# task.id is None here - database hasn't assigned it yetaudit = AuditLog(entity_id=task.id) # None! Wrong! Multi-Operation Transactions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#multi-operation-transactions "Direct link to Multi-Operation Transactions") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Group related operations into a single transaction: async def create_project_with_tasks( session: AsyncSession, project_data: dict, task_titles: list[str],) -> Project: # Create project project = Project(**project_data) session.add(project) await session.flush() # Get project.id # Create all tasks tasks = [ Task(title=title, project_id=project.id) for title in task_titles ] session.add_all(tasks) # Single commit for everything await session.commit() await session.refresh(project) return project **Atomicity guarantee:** If any task fails to insert, the project is also rolled back. Rollback on Error[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#rollback-on-error "Direct link to Rollback on Error") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When operations fail, rollback to previous state: async def safe_create_task(session: AsyncSession, data: dict) -> Task: try: task = Task(**data) session.add(task) await session.commit() await session.refresh(task) return task except Exception: await session.rollback() raise **What rollback does:** * Discards all uncommitted changes in the session * Returns connection to pool in clean state * Prevents partial data from persisting Handling IntegrityError[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#handling-integrityerror "Direct link to Handling IntegrityError") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Database constraints raise `IntegrityError`: from sqlalchemy.exc import IntegrityErrorfrom fastapi import HTTPExceptionasync def create_project( session: AsyncSession, data: ProjectCreate,) -> Project: try: project = Project(**data.model_dump()) session.add(project) await session.commit() await session.refresh(project) return project except IntegrityError as e: await session.rollback() if "unique constraint" in str(e).lower(): raise HTTPException( status_code=400, detail="Project with this name already exists" ) if "foreign key constraint" in str(e).lower(): raise HTTPException( status_code=400, detail="Referenced record does not exist" ) raise HTTPException(status_code=500, detail="Database error") **Common IntegrityError causes:** | Constraint | Example | Response | | --- | --- | --- | | Unique | Duplicate email | 400: Already exists | | Foreign key | Invalid project\_id | 400: Parent not found | | Not null | Missing required field | 400: Field required | Context Manager Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#context-manager-pattern "Direct link to Context Manager Pattern") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Use `async with session.begin()` for automatic commit/rollback: async def transactional_update(session: AsyncSession): async with session.begin(): # All operations in this block are in one transaction task = await session.get(Task, 42) task.status = "completed" audit = AuditLog(entity_id=42, action="completed") session.add(audit) # Auto-commit on successful exit # Auto-rollback on exception **Benefits:** * Commit happens automatically on clean exit * Rollback happens automatically on exception * No need for try/except/rollback boilerplate **Caution:** Don't mix with manual `commit()` calls: # WRONG - double commitasync with session.begin(): session.add(task) await session.commit() # Error! begin() manages the transaction Nested Operations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#nested-operations "Direct link to Nested Operations") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When calling functions that might use the session: async def outer_operation(session: AsyncSession): # Start transaction project = Project(name="New Project") session.add(project) await session.flush() # Call inner function - shares the transaction await create_initial_tasks(session, project.id) # Single commit for everything await session.commit()async def create_initial_tasks(session: AsyncSession, project_id: int): # These adds are part of the outer transaction tasks = [ Task(title="Setup", project_id=project_id), Task(title="Planning", project_id=project_id), ] session.add_all(tasks) # Don't commit here - outer function handles it Complete Error Handling Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#complete-error-handling-pattern "Direct link to Complete Error Handling Pattern") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Production-ready service with error handling: from sqlalchemy.exc import IntegrityError, SQLAlchemyErrorfrom fastapi import HTTPExceptionimport logginglogger = logging.getLogger(__name__)class TaskService: def __init__(self, session: AsyncSession): self.session = session async def create(self, data: TaskCreate, created_by_id: int) -> Task: try: task = Task( **data.model_dump(), created_by_id=created_by_id, ) self.session.add(task) await self.session.flush() # Create audit log audit = AuditLog( entity_type="task", entity_id=task.id, action="created", actor_id=created_by_id, ) self.session.add(audit) await self.session.commit() await self.session.refresh(task) return task except IntegrityError as e: await self.session.rollback() logger.warning(f"IntegrityError creating task: {e}") if "project_id" in str(e): raise HTTPException(400, "Project not found") if "created_by_id" in str(e): raise HTTPException(400, "Creator not found") raise HTTPException(400, "Invalid data") except SQLAlchemyError as e: await self.session.rollback() logger.error(f"Database error creating task: {e}") raise HTTPException(500, "Database error") **Output (on FK violation):** WARNING: IntegrityError creating task: insert or update on table "task" violates foreign key constraintHTTP 400: Project not found Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Atomic Bulk Update[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#prompt-1-atomic-bulk-update "Direct link to Prompt 1: Atomic Bulk Update") I need to update all tasks in a project to "archived" status,and create a single audit log entry for the bulk action.If any task update fails, I want the entire operation to rollbackincluding the audit log.Show me the transactional pattern for this. **What you're learning:** Bulk transaction patterns—ensuring atomicity across multiple records. ### Prompt 2: Retry on Deadlock[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#prompt-2-retry-on-deadlock "Direct link to Prompt 2: Retry on Deadlock") My agent occasionally gets "deadlock detected" errors whenmultiple agents update the same project simultaneously.Show me a retry pattern that:1. Catches deadlock errors2. Retries up to 3 times with exponential backoff3. Rolls back and re-starts the transaction on each retry **What you're learning:** Concurrency handling—recovering from database contention. ### Prompt 3: Partial Success Handling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#prompt-3-partial-success-handling "Direct link to Prompt 3: Partial Success Handling") I'm importing tasks from a CSV. Some rows might have invalid data.I want to:1. Insert all valid rows2. Skip invalid rows3. Return a report of what succeeded and what failedShould I use one transaction or many? Show me the pattern. **What you're learning:** Batch processing—balancing atomicity with partial success requirements. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#safety-note "Direct link to Safety Note") Never catch exceptions without rollback. A session with uncommitted changes after an error can corrupt subsequent operations. Always rollback in exception handlers. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#reflect-on-your-skill "Direct link to Reflect on Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ You built a `relational-db-agent` skill in Lesson 0. Test its transaction knowledge. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#test-your-skill "Direct link to Test Your Skill") Using my relational-db-agent skill, generate a function that:1. Creates a Project2. Creates 3 initial Tasks for the project3. Creates an AuditLog entry4. Uses flush() to get IDs before commit5. Rolls back everything if any step fails ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill use `flush()` to get project.id before creating tasks? * Did it include `try/except` with `rollback()` in the except block? * Did it handle `IntegrityError` specifically? * Did it use a single `commit()` at the end? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My relational-db-agent skill doesn't handle IntegrityError.Add this pattern for all create/update operations:from sqlalchemy.exc import IntegrityErrortry: await session.commit()except IntegrityError: await session.rollback() raise HTTPException(400, "Constraint violation") Your skill now generates robust transaction patterns. Checking access... * [flush() vs commit()](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#flush-vs-commit) * [Multi-Operation Transactions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#multi-operation-transactions) * [Rollback on Error](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#rollback-on-error) * [Handling IntegrityError](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#handling-integrityerror) * [Context Manager Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#context-manager-pattern) * [Nested Operations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#nested-operations) * [Complete Error Handling Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#complete-error-handling-pattern) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#try-with-ai) * [Prompt 1: Atomic Bulk Update](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#prompt-1-atomic-bulk-update) * [Prompt 2: Retry on Deadlock](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#prompt-2-retry-on-deadlock) * [Prompt 3: Partial Success Handling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#prompt-3-partial-success-handling) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#safety-note) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/relational-databases-sqlmodel/transactions-and-error-handling#improve-your-skill) --- # Packaging & Distribution | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Throughout this chapter, you've built powerful MCP servers: injecting Context, calling LLMs through clients, reporting progress, controlling file access, and handling errors gracefully. Now comes a critical step that separates toy projects from **production-ready components**: packaging your server so others can install and use it. A packaged MCP server is fundamentally different from code in a git repository. With packaging, users run **one command** to install your entire server, including dependencies, and immediately use it in Claude Desktop or other MCP clients. No cloning repos, no installing dependencies manually, no configuration headaches. This lesson teaches you the packaging patterns that turn your MCP server code into an **installable Digital FTE component** that integrates seamlessly into Claude and other workflows. From Code to Package: The Transformation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#from-code-to-package-the-transformation "Direct link to From Code to Package: The Transformation") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ When you have working MCP server code, the journey to distribution involves these steps: 1. **Define metadata** (pyproject.toml) — What is this package? Who made it? What's required? 2. **Create entry points** (command-line starters) — How does the server start after installation? 3. **Build the package** (wheel creation) — Compress code and dependencies into installable form 4. **Test locally** (installation verification) — Does it install cleanly? Does it work? 5. **Configure in clients** (Claude Desktop) — Register the server so Claude can use it 6. **Distribute** (PyPI, custom repos) — Make it available for others to install Let's work through each step. The pyproject.toml: Project Metadata and Dependencies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#the-pyprojecttoml-project-metadata-and-dependencies "Direct link to The pyproject.toml: Project Metadata and Dependencies") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your `pyproject.toml` file is the **contract** between your code and the outside world. It answers fundamental questions: * What is this project called? * What version is it? * What Python versions does it support? * What dependencies must be installed? * How do users run it after installation? ### Minimal but Complete pyproject.toml[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#minimal-but-complete-pyprojecttoml "Direct link to Minimal but Complete pyproject.toml") Here's the structure for an MCP server: [build-system]requires = ["hatchling"]build-backend = "hatchling.build"[project]name = "my-mcp-server"version = "0.1.0"description = "MCP server for domain-specific tasks"readme = "README.md"requires-python = ">=3.11"authors = [ {name = "Your Name", email = "you@example.com"},]keywords = ["mcp", "agent", "tools"]classifiers = [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12",]dependencies = [ "mcp>=1.6.0", "httpx>=0.27.0", "pydantic>=2.0.0",][project.scripts]my-mcp-server = "my_mcp_server:main"[project.urls]Homepage = "https://github.com/yourusername/my-mcp-server"Repository = "https://github.com/yourusername/my-mcp-server"Documentation = "https://github.com/yourusername/my-mcp-server/blob/main/README.md" **What each section does:** | Section | Purpose | Example | | --- | --- | --- | | `[build-system]` | Specifies how to build the package | `hatchling` is modern, lightweight builder | | `[project]` | Core metadata (name, version, description) | Used by PyPI, pip, and installation tools | | `dependencies` | Runtime requirements (what pip installs) | MCP, httpx for HTTP requests, pydantic for validation | | `[project.scripts]` | Entry points (CLI commands created on install) | `my-mcp-server` command becomes available after `pip install` | | `[project.urls]` | Project links (documentation, repository) | Help users find source code and docs | ### Understanding the Entry Point Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#understanding-the-entry-point-pattern "Direct link to Understanding the Entry Point Pattern") The `[project.scripts]` section is how your installed server becomes executable: [project.scripts]my-mcp-server = "my_mcp_server:main" This line says: * When user runs `my-mcp-server` command, execute the `main()` function from module `my_mcp_server` * During installation, pip creates a CLI wrapper script that calls this function ### Your Server's Entry Point Module[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#your-servers-entry-point-module "Direct link to Your Server's Entry Point Module") To make this work, you need a `main()` function in your package: # my_mcp_server/__init__.pyfrom mcp.server.fastmcp import FastMCPmcp = FastMCP("my-server")# ... define your tools, resources, prompts ...def main(): """Entry point for installed server.""" mcp.run()if __name__ == "__main__": main() **Why this structure?** 1. **FastMCP global**: Your `mcp` object is defined at module level, so all decorators (@mcp.tool, @mcp.resource) can attach to it 2. **main() function**: Called by the entry point when user runs `my-mcp-server` command 3. **if **name** == "**main**"**: Allows testing by running the module directly: `python -m my_mcp_server` ### Specifying Dependencies Precisely[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#specifying-dependencies-precisely "Direct link to Specifying Dependencies Precisely") Your `dependencies` list should include only what's **essential** for the server to run: dependencies = [ "mcp>=1.6.0", # MCP framework (minimum version) "httpx>=0.27.0", # For HTTP requests in your tools "pydantic>=2.0.0", # Data validation (MCP requires this)] **Dependency version specifications:** | Format | Meaning | Use Case | | --- | --- | --- | | `mcp>=1.6.0` | At least 1.6.0 | Framework is stable; new minor versions are safe | | `mcp~=1.6.0` | 1.6.x (not 1.7) | Stricter; only patch version changes allowed | | `mcp==1.6.0` | Exactly 1.6.0 | Most restrictive; pinned version | | `mcp>=1.6.0,<2.0` | 1.6.0 to 1.x | Common pattern; avoid major version jumps | For MCP servers, use `>=1.6.0` for framework dependencies. MCP follows semantic versioning, so minor versions are backward compatible. Building Your Package[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#building-your-package "Direct link to Building Your Package") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The `uv build` command creates a **wheel** (`.whl` file)—a compressed archive containing your code and a manifest. ### Step 1: Prepare Project Structure[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-1-prepare-project-structure "Direct link to Step 1: Prepare Project Structure") Ensure your project layout matches this pattern: my-mcp-server/├── my_mcp_server/ # Package directory (matches [project] name with underscores)│ ├── __init__.py # Contains FastMCP instance and main()│ ├── tools.py # Tool implementations│ ├── resources.py # Resource implementations│ └── prompts.py # Prompt definitions├── pyproject.toml # Package metadata├── README.md # Documentation└── LICENSE # License (e.g., MIT) **Critical note**: The directory name (`my_mcp_server`) must match your `[project] name` with hyphens converted to underscores. ### Step 2: Build the Wheel[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-2-build-the-wheel "Direct link to Step 2: Build the Wheel") # Navigate to project directorycd my-mcp-server# Build the packageuv build# Output:# Building sdist (source distribution)...# Building wheel...# Successfully built my_mcp_server-0.1.0-py3-none-any.whl The `uv build` command creates two artifacts: * `*.tar.gz` (source distribution) — for pip to install from source if needed * `*.whl` (wheel) — binary distribution, faster to install ### Step 3: Inspect Package Contents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-3-inspect-package-contents "Direct link to Step 3: Inspect Package Contents") # List files in the wheelunzip -l dist/my_mcp_server-0.1.0-py3-none-any.whl | head -30# Output shows your code, metadata, entry points:# my_mcp_server/__init__.py# my_mcp_server/tools.py# my_mcp_server-0.1.0.dist-info/entry_points.txt ← Entry point manifest# my_mcp_server-0.1.0.dist-info/METADATA ← Project metadata from pyproject.toml Local Testing: Installation and Verification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#local-testing-installation-and-verification "Direct link to Local Testing: Installation and Verification") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Before distributing to users, test your package locally: ### Step 1: Install the Built Package[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-1-install-the-built-package "Direct link to Step 1: Install the Built Package") # Install the wheel (requires pip or uv)uv pip install dist/my_mcp_server-0.1.0-py3-none-any.whl# Or with pip:pip install dist/my_mcp_server-0.1.0-py3-none-any.whl After installation, the entry point is available as a command: # Test that the command existswhich my-mcp-server# Output: /path/to/venv/bin/my-mcp-server# Check what it doesmy-mcp-server --help# Output: Usage: my-mcp-server [options]# MCP server for domain-specific tasks ### Step 2: Verify the Server Runs[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-2-verify-the-server-runs "Direct link to Step 2: Verify the Server Runs") # Start the servermy-mcp-server# Expected output:# Starting MCP server...# Server running on stdio transport# (Server waits for client connections; press Ctrl+C to stop) The server should start without errors. If it crashes: 1. **Check imports**: Verify all modules in `my_mcp_server/` can be imported 2. **Check dependencies**: Confirm all `dependencies` from pyproject.toml are installed 3. **Check main()**: Ensure `main()` function is defined and calls `mcp.run()` ### Step 3: Verify Entry Point Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-3-verify-entry-point-works "Direct link to Step 3: Verify Entry Point Works") # List installed packagespip show my-mcp-server# Expected output:# Name: my-mcp-server# Version: 0.1.0# Summary: MCP server for domain-specific tasks# Location: /path/to/site-packages# Requires: mcp, httpx, pydantic# List available entry pointspip show --files my-mcp-server | grep entry_points# Or check directlycat /path/to/site-packages/my_mcp_server-0.1.0.dist-info/entry_points.txt# Expected output:# [console_scripts]# my-mcp-server = my_mcp_server:main Claude Desktop Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#claude-desktop-configuration "Direct link to Claude Desktop Configuration") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After installing your MCP server, register it with Claude Desktop so Claude can discover and use its tools. ### Step 1: Locate Claude Desktop Config[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-1-locate-claude-desktop-config "Direct link to Step 1: Locate Claude Desktop Config") Claude Desktop stores server configurations in a JSON file: **macOS/Linux:** ~/.config/Claude/claude_desktop_config.json **Windows:** %APPDATA%\Claude\claude_desktop_config.json ### Step 2: Add Your Server to the Config[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-2-add-your-server-to-the-config "Direct link to Step 2: Add Your Server to the Config") Edit the config file (or create it if it doesn't exist): { "mcpServers": { "my-server": { "command": "my-mcp-server" } }} **Configuration breakdown:** { "mcpServers": { // Top-level section for all MCP servers "my-server": { // Server ID (internal name, can be anything) "command": "my-mcp-server" // Exact command from [project.scripts] } }} ### Step 3: Restart Claude Desktop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-3-restart-claude-desktop "Direct link to Step 3: Restart Claude Desktop") Close and reopen Claude Desktop. After restart, Claude will: 1. Start your MCP server via the `my-mcp-server` command 2. Discover all tools, resources, and prompts your server defines 3. Make them available in the Claude interface ### Step 4: Verify Tools Appear[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-4-verify-tools-appear "Direct link to Step 4: Verify Tools Appear") In Claude Desktop, check the bottom-left corner. You should see: * A "Tools" or "Integrations" menu * Your server name (e.g., "my-server") * List of available tools from your server If tools don't appear: 1. Restart Claude Desktop completely 2. Check that `my-mcp-server` command is accessible (can you run it from terminal?) 3. Review your server's tool definitions (@mcp.tool decorators) Distribution Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#distribution-strategies "Direct link to Distribution Strategies") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once your package is built and tested locally, you have options for distribution: ### Option 1: Share as Wheel File (Direct Installation)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#option-1-share-as-wheel-file-direct-installation "Direct link to Option 1: Share as Wheel File (Direct Installation)") Users can install directly from your wheel: pip install my-mcp-server-0.1.0-py3-none-any.whl **Advantages:** * No PyPI account needed * Users can install from GitHub releases, S3, etc. **Disadvantages:** * Users need full filename (no `pip install my-mcp-server` without version) * No automatic updates ### Option 2: Publish to PyPI (Official Python Package Index)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#option-2-publish-to-pypi-official-python-package-index "Direct link to Option 2: Publish to PyPI (Official Python Package Index)") Upload your package to PyPI so users can install with a single command: # User installation (after you publish):pip install my-mcp-server **Advantages:** * Standard Python installation experience * Automatic dependency resolution * `pip install my-mcp-server` without version numbers * PyPI hosts documentation **Disadvantages:** * Requires PyPI account * Public name registration (first come, first served) * Version management responsibility Publishing to PyPI involves: 1. Creating a PyPI account (pypi.org) 2. Building and signing your package 3. Uploading with `twine` or `uv publish` (Detailed PyPI publishing is beyond this lesson's scope; it's typically handled in a "Distribution & Publishing" chapter at the end of the course.) ### Option 3: Private Distribution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#option-3-private-distribution "Direct link to Option 3: Private Distribution") For organizational use, distribute through: * **GitHub releases**: Users download `.whl` from your releases page * **Private PyPI**: Enterprise-grade package repository * **Internal packages repo**: Organization-hosted mirror Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- The patterns in this lesson are straightforward, but attention to detail matters. Work with AI to verify your packaging is correct. ### Prompt 1: Validate Your pyproject.toml[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#prompt-1-validate-your-pyprojecttoml "Direct link to Prompt 1: Validate Your pyproject.toml") **What you're learning**: How to identify potential configuration issues before they cause installation failures. Ask Claude: I've created a pyproject.toml for my MCP server. Please review it for completeness and correctness:[paste your pyproject.toml]Check:1. Is [build-system] correctly specified?2. Does [project] have all required fields (name, version, description, requires-python)?3. Are dependencies correctly specified with version constraints?4. Is the [project.scripts] entry point correctly formatted?5. Are there any common mistakes or missing fields? **Expected result**: Claude identifies any structural issues, missing metadata, or version constraint problems before you try to build. ### Prompt 2: Debug Installation Failures[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#prompt-2-debug-installation-failures "Direct link to Prompt 2: Debug Installation Failures") **What you're learning**: How to interpret installation errors and fix underlying packaging issues. If installation fails, ask Claude: My MCP server installation failed with this error:[paste error message]My pyproject.toml is:[paste pyproject.toml]My package structure is:[paste directory listing]My __init__.py contains:[paste __init__.py code]What's wrong, and how do I fix it? **Expected result**: Claude traces from the error to the root cause (missing file, incorrect module path, dependency issue, etc.) and suggests specific fixes. ### Prompt 3: Generate Complete pyproject.toml[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#prompt-3-generate-complete-pyprojecttoml "Direct link to Prompt 3: Generate Complete pyproject.toml") **What you're learning**: How to leverage AI to generate correctly-formatted packaging boilerplate while you focus on your domain logic. Ask Claude: Generate a complete pyproject.toml for an MCP server with these specifications:- Name: research-assistant- Version: 0.2.1- Description: MCP server that performs research tasks- Python requirement: 3.11+- Dependencies: mcp>=1.6.0, httpx>=0.27.0, pydantic>=2.0.0, requests>=2.31.0- Entry point command: research-assistant- Author: Your Name, email@example.com- License: MIT- GitHub: https://github.com/yourusername/research-assistantFormat it ready to copy into a new pyproject.toml file. **Expected result**: Claude generates a complete, properly-formatted pyproject.toml that you can immediately use as your project's packaging configuration. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built an `mcp-server` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#test-your-skill "Direct link to Test Your Skill") Using my mcp-server skill, create a pyproject.toml configuration for packaging an MCP server.Does my skill include guidance on entry points, dependency specifications, and build system configuration? ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include pyproject.toml structure (\[project.scripts\] entry points)? * Did it explain how to build with uv, test locally, and configure in Claude Desktop? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My mcp-server skill is missing packaging and distribution patterns.Update it to include pyproject.toml configuration, [project.scripts] entry point patterns, dependency version specifications, building with uv build, local installation testing, and Claude Desktop configuration. * * * Checking access... * [From Code to Package: The Transformation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#from-code-to-package-the-transformation) * [The pyproject.toml: Project Metadata and Dependencies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#the-pyprojecttoml-project-metadata-and-dependencies) * [Minimal but Complete pyproject.toml](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#minimal-but-complete-pyprojecttoml) * [Understanding the Entry Point Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#understanding-the-entry-point-pattern) * [Your Server's Entry Point Module](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#your-servers-entry-point-module) * [Specifying Dependencies Precisely](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#specifying-dependencies-precisely) * [Building Your Package](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#building-your-package) * [Step 1: Prepare Project Structure](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-1-prepare-project-structure) * [Step 2: Build the Wheel](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-2-build-the-wheel) * [Step 3: Inspect Package Contents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-3-inspect-package-contents) * [Local Testing: Installation and Verification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#local-testing-installation-and-verification) * [Step 1: Install the Built Package](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-1-install-the-built-package) * [Step 2: Verify the Server Runs](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-2-verify-the-server-runs) * [Step 3: Verify Entry Point Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-3-verify-entry-point-works) * [Claude Desktop Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#claude-desktop-configuration) * [Step 1: Locate Claude Desktop Config](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-1-locate-claude-desktop-config) * [Step 2: Add Your Server to the Config](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-2-add-your-server-to-the-config) * [Step 3: Restart Claude Desktop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-3-restart-claude-desktop) * [Step 4: Verify Tools Appear](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#step-4-verify-tools-appear) * [Distribution Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#distribution-strategies) * [Option 1: Share as Wheel File (Direct Installation)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#option-1-share-as-wheel-file-direct-installation) * [Option 2: Publish to PyPI (Official Python Package Index)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#option-2-publish-to-pypi-official-python-package-index) * [Option 3: Private Distribution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#option-3-private-distribution) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#try-with-ai) * [Prompt 1: Validate Your pyproject.toml](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#prompt-1-validate-your-pyprojecttoml) * [Prompt 2: Debug Installation Failures](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#prompt-2-debug-installation-failures) * [Prompt 3: Generate Complete pyproject.toml](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#prompt-3-generate-complete-pyprojecttoml) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/packaging-distribution#improve-your-skill) --- # The Two Evaluation Axes | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've built an agent that summarizes customer support tickets. How do you know if it's doing a good job? Your first instinct might be to run some test cases and see if the outputs "look right." But "looks right" doesn't scale. You need systematic evaluation—and here's the problem: **not all evals work the same way**. Some agent behaviors have clear right answers you can check with code. Others require judgment calls that only a human (or another AI) can make. Some behaviors have reference outputs you can compare against. Others have no single "correct" answer—just better and worse ones. Understanding these distinctions changes how you design evaluations. The wrong approach gives you unreliable scores. The right approach gives you actionable signal about agent quality. Two Questions That Define Every Eval[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#two-questions-that-define-every-eval "Direct link to Two Questions That Define Every Eval") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every evaluation you'll ever write falls somewhere on two independent axes: **Axis 1: How do you check correctness?** * **Objective (Code-checkable)**: A deterministic function can verify correctness * **Subjective (LLM-judged)**: Requires reasoning about quality, not just matching **Axis 2: Do you have ground truth?** * **Per-example ground truth**: Each test case has a known "correct" answer * **No per-example ground truth**: Quality is assessed against criteria, not reference outputs These two axes create four distinct quadrants—each requiring different evaluation strategies. Axis 1: Objective vs Subjective Scoring[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#axis-1-objective-vs-subjective-scoring "Direct link to Axis 1: Objective vs Subjective Scoring") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Objective Evals (Code Can Check)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#objective-evals-code-can-check "Direct link to Objective Evals (Code Can Check)") Objective evaluations use deterministic functions to verify correctness. The grader runs code that returns pass/fail or a numeric score—no AI reasoning required. **Characteristics**: * Same input always produces same evaluation result * Implemented as pure functions (Python, regex, parsers) * Fast and cheap to run (no LLM API calls for grading) * Zero ambiguity—the code decides **Examples of objective criteria**: | Criterion | How Code Checks It | | --- | --- | | Response under 500 tokens | `len(tokenize(response)) < 500` | | Contains required JSON fields | `set(required_keys).issubset(response.keys())` | | Called the expected tool | `"search_database" in trace.tool_calls` | | Output matches regex pattern | `re.match(pattern, output) is not None` | | No PII in response | `pii_detector.scan(response) == []` | **When to use objective scoring**: When success criteria can be expressed as computable predicates. If you can write a function that returns True/False, you have an objective eval. ### Subjective Evals (LLM Must Judge)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#subjective-evals-llm-must-judge "Direct link to Subjective Evals (LLM Must Judge)") Subjective evaluations require reasoning about quality—something code alone cannot do. You use another LLM as the judge. **Characteristics**: * Results may vary slightly between runs (LLM non-determinism) * Implemented as prompts to a grader LLM * Slower and more expensive (requires inference) * Captures nuanced quality dimensions **Examples of subjective criteria**: | Criterion | Why Code Can't Check | | --- | --- | | Response is helpful | "Helpful" requires understanding context and user needs | | Explanation is clear | Clarity depends on reader comprehension model | | Tone is professional | Professionalism involves subtle linguistic choices | | Summary captures key points | "Key" requires understanding importance hierarchy | | Response addresses user's actual question | Intent interpretation requires reasoning | **When to use subjective scoring**: When quality depends on semantic understanding that cannot be reduced to pattern matching. If the criterion requires "reading comprehension" to evaluate, you need an LLM judge. ### The Grading Implementation Difference[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#the-grading-implementation-difference "Direct link to The Grading Implementation Difference") Here's how the same evaluation target—a customer support response—might use both approaches: **Objective graders (code)**: def grade_format_compliance(response: dict) -> dict: """Check structural requirements with code.""" checks = { "has_greeting": response.get("greeting") is not None, "has_resolution": response.get("resolution_steps") is not None, "under_300_words": len(response.get("body", "").split()) < 300, "no_profanity": not contains_profanity(response.get("body", "")) } return { "score": sum(checks.values()), "max_score": len(checks), "details": checks } **Output:** {"score": 3, "max_score": 4, "details": {"has_greeting": true, "has_resolution": true, "under_300_words": false, "no_profanity": true}} **Subjective grader (LLM judge)**: JUDGE_PROMPT = """Evaluate this customer support response on these criteria.For EACH criterion, answer YES or NO:1. Does the response acknowledge the customer's frustration?2. Does it provide actionable next steps?3. Is the tone empathetic but professional?4. Does it avoid making promises the company can't keep?5. Would you be satisfied receiving this response?Response to evaluate:{response}Return JSON: {"criteria": [true/false, ...], "total_yes": N}"""def grade_quality(response: str, llm_client) -> dict: """Use LLM to evaluate quality dimensions.""" result = llm_client.complete( JUDGE_PROMPT.format(response=response) ) return json.loads(result) **Output:** {"criteria": [true, true, true, false, true], "total_yes": 4} Notice the difference: the objective grader runs instantly with deterministic results. The subjective grader calls an LLM, costs money, and captures dimensions (empathy, actionability) that code cannot assess. Axis 2: Ground Truth vs No Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#axis-2-ground-truth-vs-no-ground-truth "Direct link to Axis 2: Ground Truth vs No Ground Truth") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Per-Example Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#per-example-ground-truth "Direct link to Per-Example Ground Truth") Some evaluations have reference answers for each test case. You compare agent output against the known correct answer. **Characteristics**: * Each test case includes expected output(s) * Grader compares agent output to reference * Can measure exact match, partial overlap, or semantic similarity * Test data requires curation (someone must produce ground truth) **Examples with ground truth**: | Task | Ground Truth | How You Got It | | --- | --- | --- | | Invoice date extraction | The actual date from the invoice | Human labeled the invoice | | Named entity recognition | List of entities that should be found | Annotated training data | | QA from documents | The answer according to the document | Human read and answered | | Code that should compile | The code compiles successfully | Verified by compiler | | Translation | Professional human translation | Paid translator | **When you have ground truth**: Your grader compares agent output to expected output. The comparison might be exact match, fuzzy match, or semantic similarity—but there's always a reference to compare against. ### No Per-Example Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#no-per-example-ground-truth "Direct link to No Per-Example Ground Truth") Other evaluations have no single correct answer. Quality is assessed against criteria or rubrics, not reference outputs. **Characteristics**: * Test cases include inputs but not expected outputs * Grader evaluates quality against criteria, not against reference * Multiple "correct" answers may exist * Rubric-based scoring defines quality dimensions **Examples without ground truth**: | Task | Why No Ground Truth | How to Evaluate | | --- | --- | --- | | Creative writing | Many valid stories exist | Rubric: coherence, engagement, style | | Open-ended advice | Context-dependent, no single right answer | Rubric: relevance, actionability, safety | | Conversation quality | Good responses vary by personality/context | Rubric: appropriateness, helpfulness | | Code review feedback | Multiple valid critiques exist | Rubric: accuracy, specificity, tone | | Summary quality | Many valid summaries of same document | Rubric: coverage, conciseness, accuracy | **When you lack ground truth**: Your grader applies a rubric—a set of criteria that define quality. No comparison to reference. Instead: does this output meet our quality standards? The Four Quadrants[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#the-four-quadrants "Direct link to The Four Quadrants") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Combining both axes creates four distinct evaluation types: | | **Objective (Code)** | **Subjective (LLM Judge)** | | --- | --- | --- | | **Per-example ground truth** | **Q1**: Exact extraction, expected tool calls | **Q2**: Gold standard talking points, reference comparisons | | **No per-example ground truth** | **Q3**: Format rules, length limits, constraint satisfaction | **Q4**: Rubric-based quality, helpfulness, clarity | ### Quadrant 1: Objective + Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-1-objective--ground-truth "Direct link to Quadrant 1: Objective + Ground Truth") **What it looks like**: You have the correct answer. Code can verify if agent's answer matches. **Real examples**: * Invoice date extraction (expected: "2024-03-15", agent output must match) * Tool call verification (expected: `search(query="user query")`, agent must call this) * JSON schema validation (expected schema, output must conform) * Math problem solving (expected: 42, agent output must equal) **Grader pattern**: def grade_extraction(agent_output: str, expected: str) -> bool: """Direct comparison against ground truth.""" return normalize(agent_output) == normalize(expected) **Characteristics**: Cheapest, fastest, most reliable. Use whenever possible. ### Quadrant 2: Subjective + Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-2-subjective--ground-truth "Direct link to Quadrant 2: Subjective + Ground Truth") **What it looks like**: You have reference content, but comparison requires judgment. **Real examples**: * Summary covers gold standard talking points (reference: 5 key points that must appear) * Response addresses required topics (reference: checklist of must-mention items) * Generated code implements specified functionality (reference: test cases it must pass) * Report includes required findings (reference: expert-identified key findings) **Grader pattern**: GRADER_PROMPT = """How many of these required talking points appear in the response?Required talking points:{talking_points}Response to evaluate:{response}For each talking point, determine if it's PRESENT or MISSING.Return JSON: {"present": [...], "missing": [...], "score": N}""" **Characteristics**: More expensive (LLM calls), but can check semantic coverage, not just string matching. ### Quadrant 3: Objective + No Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-3-objective--no-ground-truth "Direct link to Quadrant 3: Objective + No Ground Truth") **What it looks like**: No single correct answer, but constraints are code-checkable. **Real examples**: * Response under 500 tokens (no "right" response, but length is measurable) * Output contains no PII (any valid output works, PII detection is deterministic) * Response is valid JSON (structure verifiable, content varies) * No profanity in output (content can vary, profanity detection is code-based) **Grader pattern**: def grade_constraints(output: str) -> dict: """Check constraint satisfaction without ground truth.""" return { "under_limit": len(output.split()) < 200, "valid_json": is_valid_json(output), "no_pii": not detect_pii(output), "has_structure": contains_required_sections(output) } **Characteristics**: Fast and cheap like Q1, but evaluates constraints rather than correctness. ### Quadrant 4: Subjective + No Ground Truth[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-4-subjective--no-ground-truth "Direct link to Quadrant 4: Subjective + No Ground Truth") **What it looks like**: Quality judgment against criteria, no reference answer. **Real examples**: * Response is helpful and addresses user needs (no single "helpful" response) * Explanation is clear and well-structured (clarity is judgment call) * Tone is appropriate for context (appropriateness requires reasoning) * Creative output is engaging (engagement is subjective) **Grader pattern**: RUBRIC_PROMPT = """Evaluate this response against our quality rubric.For EACH criterion, answer YES or NO:1. Does it directly address what the user asked?2. Is the information accurate (to your knowledge)?3. Is it appropriately detailed for the question?4. Is the tone suitable for a professional context?5. Would a reasonable user find this helpful?Response: {response}Return JSON with boolean for each criterion and total count.""" **Characteristics**: Most expensive, captures nuanced quality, requires careful rubric design. Choosing Your Quadrant[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#choosing-your-quadrant "Direct link to Choosing Your Quadrant") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When designing an eval, ask these questions in order: **Question 1: Can you get ground truth for test cases?** If you can label expected outputs for your test cases (or already have labeled data): * You can use Quadrant 1 or 2 * Your evals will measure "correctness" against references If labeling expected outputs is impossible or impractical: * You must use Quadrant 3 or 4 * Your evals will measure "quality" against criteria **Question 2: Can code verify your criteria?** If success is deterministic (string match, format check, constraint satisfaction): * Use Quadrant 1 (with GT) or Quadrant 3 (without GT) * Your graders will be fast, cheap, and consistent If success requires semantic understanding (helpfulness, clarity, relevance): * Use Quadrant 2 (with GT) or Quadrant 4 (without GT) * Your graders will use LLM-as-judge **Decision flow**: Ground Truth Available? / \ YES NO / \ Code Can Verify? Code Can Verify? / \ / \ YES NO YES NO | | | | Q1 Q2 Q3 Q4 (Objective (Subjective (Objective (Subjective + Ground + Ground + No GT) + No GT) Truth) Truth) Exercise: Classify Task API Evals[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#exercise-classify-task-api-evals "Direct link to Exercise: Classify Task API Evals") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Task API agent helps users manage tasks. For each evaluation scenario below, identify the quadrant and explain your reasoning. **Scenario 1**: Verify that when a user says "create a task called groceries", the agent calls `create_task(title="groceries")`. **Scenario 2**: Check that task descriptions are under 500 characters. **Scenario 3**: Evaluate whether the agent's task suggestions are relevant to the user's stated goals. **Scenario 4**: Verify that the agent correctly extracts due dates from natural language (e.g., "next Tuesday" should resolve to the correct date). **Scenario 5**: Assess whether the agent's responses are friendly and professional in tone. **Work through each scenario**: * Can you provide the "right answer" for each test case? * Can code verify the criterion, or does it need judgment? (Answers at end of lesson) Why This Classification Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#why-this-classification-matters "Direct link to Why This Classification Matters") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Understanding quadrants prevents two costly mistakes: **Mistake 1: Using LLM judges when code suffices** If your criterion is code-checkable (Q1 or Q3), using an LLM judge wastes money and adds noise. A token counter is faster, cheaper, and more reliable than asking an LLM "is this under 500 tokens?" **Mistake 2: Expecting code to check what requires judgment** If your criterion needs semantic understanding (Q2 or Q4), code-based graders will miss the mark. Regex cannot determine if a response is "helpful"—you need an LLM judge with a well-designed rubric. **The practical impact**: * Misclassified Q1/Q3 as Q4 → Slow, expensive evals that should be instant * Misclassified Q2/Q4 as Q1/Q3 → Brittle graders that miss quality issues * Correct classification → Right tool for each job, reliable signal, manageable costs Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Classify Your Agent's Behaviors[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#prompt-1-classify-your-agents-behaviors "Direct link to Prompt 1: Classify Your Agent's Behaviors") I'm building an agent that [describe your agent]. Help me create an evaluation planby classifying these behaviors into the four quadrants:1. [Behavior 1 - e.g., "extracts customer name from email"]2. [Behavior 2 - e.g., "writes helpful responses"]3. [Behavior 3 - e.g., "stays under token limits"]4. [Behavior 4 - e.g., "handles angry customers appropriately"]For each, tell me:- Which quadrant (Q1-Q4)?- Why? (ground truth available? code can check?)- What grader approach should I use? **What you're learning**: Applying the two-axis framework to your own agent, seeing how different behaviors require different evaluation strategies. ### Prompt 2: Design a Rubric for Q4 Evals[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#prompt-2-design-a-rubric-for-q4-evals "Direct link to Prompt 2: Design a Rubric for Q4 Evals") I need to evaluate whether my agent's [describe output, e.g., "task suggestions"]are high quality. There's no single right answer, so I need a rubric.Help me design 5 binary criteria (yes/no) that capture quality dimensions like:- Relevance to user needs- Actionability- Appropriate detail level- [Add your own]Format as an LLM judge prompt I can use. **What you're learning**: Rubric design for subjective evaluations—converting vague "quality" into measurable binary criteria that LLM judges can assess consistently. ### Prompt 3: Challenge the Classification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#prompt-3-challenge-the-classification "Direct link to Prompt 3: Challenge the Classification") I classified this eval as [your quadrant]:[Describe the eval]Challenge my classification. Could it fit a different quadrant?What would need to change to move it to a "cheaper" quadrant(Q4 → Q3, Q2 → Q1)? Would that sacrifice important signal? **What you're learning**: The tradeoffs between evaluation approaches—sometimes you can simplify without losing signal, sometimes "cheap" evals miss what matters. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#safety-note "Direct link to Safety Note") When using LLM-as-judge for subjective evals, remember that LLM judges have biases (like position bias when comparing two outputs). Test your graders on known-quality examples to ensure they produce sensible scores. Don't blindly trust LLM judgments any more than you'd blindly trust agent outputs. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After completing this lesson, consider updating your agent-evals skill with quadrant classification: **Add to your skill's decision framework**: * When analyzing a new eval need, first classify it into a quadrant * Use quadrant to determine grader implementation approach * Prefer lower quadrants (Q1 > Q2, Q3 > Q4) when possible for cost/reliability **Key insight to encode**: The cheapest reliable eval is the best eval. Move toward Q1/Q3 whenever the criterion allows it. * * * Exercise Answers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#exercise-answers "Direct link to Exercise Answers") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Scenario 1**: Q1 (Objective + Ground Truth) * Ground truth: Yes (the expected tool call is `create_task(title="groceries")`) * Code can verify: Yes (compare actual tool call to expected) * Grader: Exact match on tool name and arguments **Scenario 2**: Q3 (Objective + No Ground Truth) * Ground truth: No (there's no single "correct" description, just a constraint) * Code can verify: Yes (`len(description) < 500`) * Grader: Character/word count function **Scenario 3**: Q4 (Subjective + No Ground Truth) * Ground truth: No (relevance depends on context, many valid suggestions) * Code can verify: No (requires understanding user goals and suggestion relevance) * Grader: LLM judge with relevance rubric **Scenario 4**: Q1 (Objective + Ground Truth) * Ground truth: Yes (each natural language date has a correct resolution) * Code can verify: Yes (compare resolved date to expected date) * Grader: Date comparison function **Scenario 5**: Q4 (Subjective + No Ground Truth) * Ground truth: No (no single "correct" friendly response) * Code can verify: No ("friendly" and "professional" require judgment) * Grader: LLM judge with tone rubric Checking access... * [Two Questions That Define Every Eval](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#two-questions-that-define-every-eval) * [Axis 1: Objective vs Subjective Scoring](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#axis-1-objective-vs-subjective-scoring) * [Objective Evals (Code Can Check)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#objective-evals-code-can-check) * [Subjective Evals (LLM Must Judge)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#subjective-evals-llm-must-judge) * [The Grading Implementation Difference](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#the-grading-implementation-difference) * [Axis 2: Ground Truth vs No Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#axis-2-ground-truth-vs-no-ground-truth) * [Per-Example Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#per-example-ground-truth) * [No Per-Example Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#no-per-example-ground-truth) * [The Four Quadrants](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#the-four-quadrants) * [Quadrant 1: Objective + Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-1-objective--ground-truth) * [Quadrant 2: Subjective + Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-2-subjective--ground-truth) * [Quadrant 3: Objective + No Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-3-objective--no-ground-truth) * [Quadrant 4: Subjective + No Ground Truth](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#quadrant-4-subjective--no-ground-truth) * [Choosing Your Quadrant](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#choosing-your-quadrant) * [Exercise: Classify Task API Evals](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#exercise-classify-task-api-evals) * [Why This Classification Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#why-this-classification-matters) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#try-with-ai) * [Prompt 1: Classify Your Agent's Behaviors](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#prompt-1-classify-your-agents-behaviors) * [Prompt 2: Design a Rubric for Q4 Evals](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#prompt-2-design-a-rubric-for-q4-evals) * [Prompt 3: Challenge the Classification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#prompt-3-challenge-the-classification) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#safety-note) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#reflect-on-your-skill) * [Exercise Answers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/the-two-evaluation-axes#exercise-answers) --- # Stateful vs Stateless Servers | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your MCP server is working beautifully. You've implemented sampling (Lesson 2), progress notifications (Lesson 3), secure file access (Lesson 4), and deployed it with StreamableHTTP (Lesson 5). But then success happens: Hundreds of clients want to use your server simultaneously. You deploy multiple instances behind a load balancer. Then something strange occurs. A client's request sometimes works, sometimes fails mysteriously. Progress messages never arrive even though you see them in logs. Sampling calls hang with timeouts. Welcome to the **horizontal scaling problem**. It's not your code—it's your architecture. The Horizontal Scaling Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#the-horizontal-scaling-problem "Direct link to The Horizontal Scaling Problem") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Single Instance: Everything Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#single-instance-everything-works "Direct link to Single Instance: Everything Works") When your server runs as a single instance, everything is straightforward: Client A ──────────────┐ ├─→ Server Instance (Single)Client B ──────────────┘ When Client A calls a tool that needs sampling: 1. Client A connects via SSE and gets a **session ID** 2. Tool needs LLM inference → Calls `context.session.create_message()` 3. Server has the session → Can route response back through Client A's SSE connection 4. Client A receives response → Tool completes successfully ### Multiple Instances Behind Load Balancer: The Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#multiple-instances-behind-load-balancer-the-problem "Direct link to Multiple Instances Behind Load Balancer: The Problem") Now you scale horizontally: Client A ──────────────┐ ├─→ [Load Balancer] ──┐Client B ──────────────┘ ├─→ Server A │ Server B └─→ Server C Here's the failure scenario: **Timeline of requests from Client A:** 1. Client A connects → Load balancer routes to **Server A** 2. Client A gets session ID from Server A's connection 3. Client A calls a tool that needs sampling 4. Load balancer receives the request → Routes to **Server B** (round-robin) 5. Server B doesn't have Client A's session (different instance) 6. Server B's tool can't find the client's SSE connection 7. `context.session.create_message()` fails → Tool returns error 8. Client A frustrated: "Why did it work once and fail the next time?" **The core issue:** Each server instance maintains sessions **in memory**. When a request goes to a different instance, that instance has no knowledge of the client's session. ### Feature Loss in Horizontal Scaling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#feature-loss-in-horizontal-scaling "Direct link to Feature Loss in Horizontal Scaling") With stateful servers and load balancers, you lose: | Feature | Why It Breaks | Solution | | --- | --- | --- | | **Sampling** (`create_message()`) | Different server can't find client's SSE | Sticky sessions OR stateless mode | | **Progress Notifications** | Different server can't send via client's SSE | Sticky sessions OR stateless mode | | **Subscription Streams** | SSE connection bound to one server | Sticky sessions OR stateless mode | | **Session-Based State** | Cache/state lost when request goes to different server | Stateless design | Solution 1: Stateless Mode (`stateless_http=True`)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#solution-1-stateless-mode-stateless_httptrue "Direct link to solution-1-stateless-mode-stateless_httptrue") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The simplest scaling solution: **Disable features that require server ↔ client communication.** ### What Stateless Mode Does[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#what-stateless-mode-does "Direct link to What Stateless Mode Does") from mcp.server.fastmcp import FastMCPmcp = FastMCP( name="simple_integration", stateless_http=True # ← Enable stateless mode) When `stateless_http=True`: 1. **No SSE connections** → Server can't initiate requests to clients 2. **No session IDs** → No server ↔ client state 3. **No sampling** → Tools can't call `context.session.create_message()` 4. **No progress** → `context.report_progress()` is silently ignored 5. **No subscriptions** → Resource subscriptions don't work 6. **Plain HTTP POST → JSON Response** → Only client-initiated requests work ### Trade-off Analysis: Stateless Mode[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#trade-off-analysis-stateless-mode "Direct link to Trade-off Analysis: Stateless Mode") **What you gain:** * Horizontal scaling works perfectly (any server instance handles any request) * Load balancing becomes trivial (no sticky sessions needed) * Infinite scale potential * Stateless design is cloud-native and resilient **What you lose:** * No server calling client (`create_message()` unavailable) * No progress feedback for long operations * No real-time notifications * No sampling (AI inside server, not through client) * No subscription streams ### When to Use Stateless Mode[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#when-to-use-stateless-mode "Direct link to When to Use Stateless Mode") ✓ Use stateless when: - Simple tool integrations (read file, transform data, call API) - No sampling needed (server doesn't need LLM from client) - No real-time feedback required - Horizontal scaling is critical - Integration just needs "request → response"✗ Avoid stateless when: - Server needs to call LLM through client (sampling) - Users need progress feedback (long operations) - Real-time bidirectional communication required - Notifications must be sent to client ### Code Example: Stateless Tool[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#code-example-stateless-tool "Direct link to Code Example: Stateless Tool") from mcp.server.fastmcp import FastMCPfrom pydantic import Fieldmcp = FastMCP( name="file_processor", stateless_http=True)@mcp.tool()async def process_file( file_path: str = Field(description="Path to file"), operation: str = Field(description="Operation: count-lines, reverse-words, etc")) -> str: """Process a file with specified operation. NOTE: Stateless mode - no sampling, no progress. This tool works independently without client communication. """ # Read file with open(file_path, 'r') as f: content = f.read() # Process if operation == "count-lines": return f"{len(content.splitlines())} lines" elif operation == "reverse-words": words = content.split() return " ".join(reversed(words)) else: return f"Unknown operation: {operation}"# ✗ This WILL NOT WORK in stateless mode:@mcp.tool()async def analyze_with_ai( text: str = Field(description="Text to analyze"), *, context: Context) -> str: """This tool requires sampling - incompatible with stateless mode.""" # ❌ This call will FAIL silently in stateless mode response = await context.session.create_message( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{"role": "user", "content": f"Analyze: {text}"}] ) return response.content[0].text **Output comparison:** When request hits **Server A** in stateless mode: Client: POST /tool/process_file {file_path: "/data/users.csv", operation: "count-lines"}Server A: ✓ Processes file → Returns: "1,250 lines" Works the same if request hits **Server B**—because no server state needed. Solution 2: Reducing Payload Size (`json_response=True`)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#solution-2-reducing-payload-size-json_responsetrue "Direct link to solution-2-reducing-payload-size-json_responsetrue") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ A complementary approach: **Disable streaming responses, use plain JSON instead.** ### What json\_response Does[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#what-json_response-does "Direct link to What json_response Does") mcp = FastMCP( name="simple_api", json_response=True # ← Enable JSON-only responses) When `json_response=True`: 1. **POST responses return JSON** → `{"result": value}` instead of streaming 2. **No intermediate streaming** → Tool must complete fully before response 3. **No log statements during execution** → Only final output sent 4. **Simpler integration** → Clients expect plain JSON, not SSE/streams ### Trade-off Analysis: json\_response[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#trade-off-analysis-json_response "Direct link to Trade-off Analysis: json_response") **What you gain:** * Compatible with ANY HTTP client (cURL, JavaScript fetch, etc.) * No streaming complexity * Standard REST API semantics * Easy to cache, load-balance, and CDN **What you lose:** * No real-time progress (nothing until tool finishes) * No streaming data (results fully buffered) * No log messages shown live * Large results may timeout if tool takes too long ### When to Use json\_response[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#when-to-use-json_response "Direct link to When to Use json_response") ✓ Use json_response when: - Tools complete quickly (< 5 seconds) - Results fit in memory - Simple HTTP clients expected - Standard JSON REST integration desired✗ Avoid json_response when: - Tools run long (need progress feedback) - Streaming large results (memory pressure) - Clients need live logging - Real-time feedback essential Configuration Matrix: Which Combination?[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#configuration-matrix-which-combination "Direct link to Configuration Matrix: Which Combination?") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's how these two settings combine: | Config | Stateless | JSON Response | Use Case | | --- | --- | --- | --- | | **Default** | `False` | `False` | Powerful servers with sampling, progress, streams. Single instance or sticky sessions. | | **Stateless** | `True` | `False` | Horizontal scaling, SSE available, but no sampling/progress. | | **JSON Response** | `False` | `True` | Single instance, JSON responses, but still supports sampling/progress. | | **Full Simple** | `True` | `True` | Maximum simplicity. Horizontal scaling, plain JSON. No advanced features. | **Real-world decision tree:** Does tool need LLM inference? (sampling)├─ YES → Stateless=False├─ NO → Can use Stateless=TrueDoes tool output large/streaming data?├─ YES → json_response=False (streaming)├─ NO → Can use json_response=TrueDo you need real-time progress?├─ YES → Stateless=False, json_response=False├─ NO → Can use eitherIs horizontal scaling critical?├─ YES → Stateless=True (with json_response=True for simplicity)├─ NO → Stateless=False (more features) Feature Limitations Matrix[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#feature-limitations-matrix "Direct link to Feature Limitations Matrix") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **Complete comparison of what works where:** | Capability | Stateful + Streaming | Stateless + Streaming | Stateless + JSON | | --- | --- | --- | --- | | Basic tools (text processing) | ✓ | ✓ | ✓ | | File I/O tools | ✓ | ✓ | ✓ | | API calls | ✓ | ✓ | ✓ | | `context.info()` / `context.warning()` | ✓ (sent live) | ✓ (sent live) | ✗ (lost) | | `context.report_progress()` | ✓ (sent live) | ✗ (ignored) | ✗ (ignored) | | `context.session.create_message()` | ✓ (works) | ✗ (can't route) | ✗ (can't route) | | Resources + subscriptions | ✓ (streaming) | ✗ (no SSE) | ✗ (no SSE) | | Horizontal scaling | ✗ (needs sticky) | ✓ (trivial) | ✓ (trivial) | | Requires load balancer config | ✗ | ✓ | ✓ | **Key insight:** Each option removes features to enable simpler scaling. Choose based on what your tools actually need. Decision Framework: Choosing Your Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#decision-framework-choosing-your-configuration "Direct link to Decision Framework: Choosing Your Configuration") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Decision question 1: Do you need sampling?** Does tool call context.session.create_message()?├─ YES → Must use stateless=False (stateful server)└─ NO → Can use stateless=True **Decision question 2: Do you need real-time progress?** Do users need progress updates while tool runs?├─ YES → Must use json_response=False (streaming)└─ NO → Can use json_response=True **Decision question 3: What's your scale requirement?** Expected concurrent clients?├─ < 100 → Stateful (default) works fine, simpler├─ 100-1000 → Evaluate. Stateless saves ops complexity└─ > 1000 → Stateless becomes essential **Decision question 4: What's your operational complexity budget?** Can your ops team manage:- Sticky sessions (session affinity)?- Client-side session reconstruction?- Cross-instance caching?├─ YES → Can use stateful with load balancer└─ NO → Must use stateless Production Architecture Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#production-architecture-patterns "Direct link to Production Architecture Patterns") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Pattern 1: Single Stateful Instance[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#pattern-1-single-stateful-instance "Direct link to Pattern 1: Single Stateful Instance") **When:** < 100 concurrent clients, features matter Clients → Reverse Proxy → MCP Server (stateful, streaming) └─ Single instance **Configuration:** mcp = FastMCP(name="my_server") # Default: stateful, streaming **Pros:** * All features available (sampling, progress, streams) * Simple operations * No session coordination **Cons:** * Vertical scaling only * Single point of failure ### Pattern 2: Horizontal Stateless with Load Balancer[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#pattern-2-horizontal-stateless-with-load-balancer "Direct link to Pattern 2: Horizontal Stateless with Load Balancer") **When:** 1000+ concurrent clients, simplicity important Clients → Load Balancer → MCP Server A (stateless, JSON) └─ MCP Server B (stateless, JSON) └─ MCP Server C (stateless, JSON) **Configuration:** mcp = FastMCP( name="my_server", stateless_http=True, json_response=True) **Pros:** * Infinite horizontal scaling * No sticky sessions needed * Any server handles any request * Cloud-native resilience **Cons:** * No sampling, progress, or streaming * Limited to request-response tools ### Pattern 3: Hybrid with Session Affinity[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#pattern-3-hybrid-with-session-affinity "Direct link to Pattern 3: Hybrid with Session Affinity") **When:** Need features + scaling, can manage sticky sessions Clients ──→ Load Balancer (sticky sessions) ├─→ MCP Server A (stateful, streaming) ├─→ MCP Server B (stateful, streaming) └─→ MCP Server C (stateful, streaming) **Configuration:** mcp = FastMCP(name="my_server") # Default: stateful# Operations: Configure load balancer sticky sessions# (different per load balancer: nginx, AWS ALB, etc.) **Pros:** * All features available * Horizontal scaling possible * More operationally complex **Cons:** * Requires load balancer sticky session support * Session affinity reduces load distribution * Must handle session failover Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Scenario 1: Deciding Architecture for a Customer Service Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#scenario-1-deciding-architecture-for-a-customer-service-agent "Direct link to Scenario 1: Deciding Architecture for a Customer Service Agent") You're building an MCP server that helps customer service teams. The tool: * Analyzes customer messages * Calls AI through the client to suggest responses * Shows real-time progress (30% analyzed, 60% drafted, etc.) * Streams large help articles Your product roadmap shows 1,000+ customers in 6 months. **Setup:** You're planning deployment. You have three options: **Option A:** Use default stateful server with load balancer sticky sessions * Pro: All features work perfectly * Con: Ops complexity managing session affinity **Option B:** Use stateless mode, remove sampling and progress * Pro: Simple horizontal scaling, no ops overhead * Con: Users lose real-time feedback and AI assistance **Option C:** Split into two services: * Stateful service for sampling/progress work (runs on dedicated infrastructure) * Stateless simple tools behind load balancer * Pro: Features where needed, scaling where needed * Con: More complex service architecture **Your decision:** Ask AI which approach best matches your constraints: "I'm building an MCP server for customer service agents.Requirements:- Tool needs sampling (AI through connected client)- Needs progress notifications (users see real-time feedback)- Streams help articles (large results)- Expected 1,000+ concurrent users in 6 months- Team of 3 engineers (limited ops capacity)What configuration should I recommend: stateful with sticky sessions,stateless with feature removal, or service split? Justify the choiceconsidering engineering complexity vs user experience." **What you're learning:** This scenario teaches architectural tradeoff analysis—a key skill for production MCP design. You'll consider user needs, team capacity, operational complexity, and scaling requirements simultaneously. ### Scenario 2: Evaluating Existing System Problems[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#scenario-2-evaluating-existing-system-problems "Direct link to Scenario 2: Evaluating Existing System Problems") Your team deployed a stateful MCP server last month. It's behind an AWS Application Load Balancer, but requests randomly fail. Sometimes tools work, sometimes they timeout with "session not found" errors. Logs show the errors happen inconsistently. **Setup:** You suspect the load balancer is routing requests to different instances. Your current configuration: mcp = FastMCP( name="analytics_engine", # Using defaults (stateful, streaming)) The server implements: * Text analytics tool (no sampling needed) * Report generation (large streaming results) * The issue: Sometimes works, sometimes fails with session errors **Your diagnosis:** Ask AI to help you diagnose and recommend fixes: "Our MCP server behind AWS ALB randomly fails with 'session not found' errors.Current setup:- FastMCP with defaults (stateful, streaming)- AWS ALB without sticky sessions- Tools: text analytics, report generation (large results)- Error pattern: Random failures, not consistent per toolDiagnosis:1. What's causing the random failures?2. Can we use stateless mode (analyze feature loss)?3. Should we enable sticky sessions instead?4. What's the simplest fix?For each option, show pros/cons and implementation complexity." **What you're learning:** This teaches diagnostic reasoning—understanding how architecture decisions create production problems, and evaluating solutions based on tradeoffs rather than just "fix the error." ### Scenario 3: Feature Negotiation with Product Team[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#scenario-3-feature-negotiation-with-product-team "Direct link to Scenario 3: Feature Negotiation with Product Team") Your product lead wants to add sampling to an existing stateless server. "Users want AI-assisted responses," she says. "Add the feature." But your server runs stateless across 10 instances with 5,000 concurrent users. **Setup:** Current server: mcp = FastMCP( name="document_processor", stateless_http=True, json_response=True) Tools process documents (no LLM needed). Adding sampling would require calling LLM through client—incompatible with stateless mode. **Your challenge:** Ask AI to help you prepare the business case for this decision: "Product wants to add AI-assisted document analysis (sampling required).Current system: stateless with 5,000 concurrent users, 10 server instances.Scenario options:1. Switch to stateful + sticky sessions (adds ops complexity)2. Keep stateless, remove AI feature3. Split services: stateless for basic tools, stateful for AI-assisted tools4. Client-side workaround: Clients call AI separately, send to toolFor each option, create a summary showing:- User impact (what changes?)- Ops impact (what's harder?)- Timeline (how long to implement?)- Cost (engineering, infrastructure)What would you recommend to the product team and why?" **What you're learning:** This teaches business reasoning—how architectural constraints affect product decisions. You're learning to communicate technical tradeoffs to non-technical stakeholders, and to negotiate solutions that balance engineering constraints with product goals. **Key decisions to carry forward:** 1. **Stateful (default)** → More features, more operational complexity 2. **Stateless (`True`)** → Simpler scaling, fewer features 3. **JSON Response (`True`)** → Standard HTTP integration, no streaming 4. **Choose based on:** Sampling needs, progress requirements, scale target, ops capacity Your choice here determines whether you build a system that scales elegantly or collapses under load. Make it thoughtfully, using the scenarios above as your guide. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#reflect-on-your-skill "Direct link to Reflect on Your Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built an `mcp-server` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#test-your-skill "Direct link to Test Your Skill") Using my mcp-server skill, decide whether to use stateful or stateless configuration for a production deployment.Does my skill include guidance on the tradeoffs between stateful (sampling, progress) and stateless (horizontal scaling) modes? ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include decision criteria for stateless\_http and json\_response configuration? * Did it explain the horizontal scaling problem and load balancer session routing? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My mcp-server skill is missing scaling architecture guidance.Update it to include when to use stateless_http=True vs False, json_response tradeoffs, feature limitations matrix, and decision frameworks for production scaling requirements. * * * Checking access... * [The Horizontal Scaling Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#the-horizontal-scaling-problem) * [Single Instance: Everything Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#single-instance-everything-works) * [Multiple Instances Behind Load Balancer: The Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#multiple-instances-behind-load-balancer-the-problem) * [Feature Loss in Horizontal Scaling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#feature-loss-in-horizontal-scaling) * [Solution 1: Stateless Mode (`stateless_http=True`)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#solution-1-stateless-mode-stateless_httptrue) * [What Stateless Mode Does](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#what-stateless-mode-does) * [Trade-off Analysis: Stateless Mode](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#trade-off-analysis-stateless-mode) * [When to Use Stateless Mode](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#when-to-use-stateless-mode) * [Code Example: Stateless Tool](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#code-example-stateless-tool) * [Solution 2: Reducing Payload Size (`json_response=True`)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#solution-2-reducing-payload-size-json_responsetrue) * [What json\_response Does](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#what-json_response-does) * [Trade-off Analysis: json\_response](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#trade-off-analysis-json_response) * [When to Use json\_response](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#when-to-use-json_response) * [Configuration Matrix: Which Combination?](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#configuration-matrix-which-combination) * [Feature Limitations Matrix](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#feature-limitations-matrix) * [Decision Framework: Choosing Your Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#decision-framework-choosing-your-configuration) * [Production Architecture Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#production-architecture-patterns) * [Pattern 1: Single Stateful Instance](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#pattern-1-single-stateful-instance) * [Pattern 2: Horizontal Stateless with Load Balancer](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#pattern-2-horizontal-stateless-with-load-balancer) * [Pattern 3: Hybrid with Session Affinity](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#pattern-3-hybrid-with-session-affinity) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#try-with-ai) * [Scenario 1: Deciding Architecture for a Customer Service Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#scenario-1-deciding-architecture-for-a-customer-service-agent) * [Scenario 2: Evaluating Existing System Problems](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#scenario-2-evaluating-existing-system-problems) * [Scenario 3: Feature Negotiation with Product Team](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#scenario-3-feature-negotiation-with-product-team) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/stateful-vs-stateless#improve-your-skill) --- # Finalize Your Evals Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You started this chapter by creating a skeleton. Now it is time to see what you actually built. A skill only proves its worth when it works on something you did not learn it on. You developed your agent-evals skill using Task API examples throughout this chapter. Every dataset design, grader pattern, and error analysis method came from that context. The question is: does your skill transfer? This lesson has one purpose. You will take your completed skill and apply it to a completely different agent. Not Task API. Not anything you have seen in this chapter. A fresh domain where your skill must stand on its own. If your skill helps you design evaluations for this new agent without returning to the chapter content, you own something valuable. If you find yourself confused or missing patterns, you know exactly where your skill needs strengthening. What Your Skill Should Include[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#what-your-skill-should-include "Direct link to What Your Skill Should Include") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before testing portability, verify your skill is complete. Review your `skills/agent-evals/SKILL.md` and check for these sections: | Section | Purpose | Completeness Check | | --- | --- | --- | | **Core Thesis** | Why evals matter | Andrew Ng quote + your interpretation | | **When to Activate** | Trigger patterns | 5+ specific scenarios | | **Evals vs TDD** | Foundational distinction | Table comparing tests and evals | | **Dataset Design** | Creating test cases | Three categories: typical, edge, error | | **Graders** | Defining "good" | Binary criteria pattern with examples | | **Error Analysis** | Finding failure patterns | Spreadsheet method with component columns | | **Component vs E2E** | Choosing eval scope | 5-step decision flow | | **Regression Protection** | Preventing quality drops | Workflow and threshold guidance | | **Framework Integration** | SDK-specific details | Table mapping concepts to frameworks | If any section is missing or incomplete, address it now. A skill with gaps will fail when you need it most. Skill Validation Checklist[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#skill-validation-checklist "Direct link to Skill Validation Checklist") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your skill is ready for validation if it can help you with these four core tasks: **Task 1: Dataset Design** Can your skill help you design an eval dataset for any agent? Criteria: * [ ] Identifies the three categories (typical, edge, error) * [ ] Provides guidance on starting with 10-20 cases * [ ] Explains how to use real data instead of synthetic * [ ] Includes patterns for growing datasets over time **Task 2: Grader Creation** Can your skill help you build graders that define "good" automatically? Criteria: * [ ] Explains why binary criteria beat 1-5 scales * [ ] Provides grader code template (binary checks pattern) * [ ] Covers LLM-as-Judge for subjective criteria * [ ] Warns about position bias **Task 3: Error Analysis** Can your skill help you find which component caused failures? Criteria: * [ ] Includes spreadsheet method structure * [ ] Lists trace terminology (trace, span, error classification) * [ ] Provides prioritization guidance (frequency times feasibility) * [ ] Explains how to focus effort where errors cluster **Task 4: Component vs E2E Decision** Can your skill help you choose the right eval scope? Criteria: * [ ] Includes the 5-step decision flow * [ ] Explains when E2E is appropriate (ship decisions, production monitoring) * [ ] Explains when component-level is better (debugging, tuning) * [ ] Provides guidance on moving between scopes If your skill satisfies all four task areas, proceed to validation. If not, return to the relevant lessons and extract the missing patterns. Testing on a Different Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#testing-on-a-different-agent "Direct link to Testing on a Different Agent") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your skill was developed using Task API examples. Now test it on something completely different. **Hypothetical Agent: Customer Support Bot** A customer support agent that: * Answers product questions from a knowledge base * Handles returns and refunds * Escalates complex issues to human agents * Maintains a helpful and professional tone This agent shares no code with Task API. It operates in a different domain with different success criteria. If your skill transfers, you can design evaluations for it without returning to chapter content. ### Apply Your Skill: Dataset Design[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#apply-your-skill-dataset-design "Direct link to Apply Your Skill: Dataset Design") Using only your skill, design an eval dataset for the customer support agent. **Typical Cases (5 examples)**: | Input | Expected Behavior | | --- | --- | | "Where is my order?" | Ask for order number, provide tracking information | | "What's your return policy?" | Quote return policy from knowledge base | | "Can I get a refund?" | Clarify reason, initiate refund process if valid | | "Product X isn't working" | Troubleshoot with standard questions, offer solutions | | "How do I cancel my subscription?" | Verify identity, process cancellation, confirm | **Edge Cases (3 examples)**: | Input | Expected Behavior | | --- | --- | | "I'm really frustrated" + valid complaint | Acknowledge emotion, solve problem, maintain professionalism | | Vague complaint without details | Ask clarifying questions, don't guess | | Request for competitor comparison | Decline politely, redirect to product benefits | **Error Cases (2 examples)**: | Input | Expected Behavior | | --- | --- | | Request to access other customer's data | Refuse firmly, explain privacy policy | | Abusive language without valid request | Maintain professionalism, offer to help when ready | If you designed these categories using your skill's guidance, the skill is transferring. ### Apply Your Skill: Grader Creation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#apply-your-skill-grader-creation "Direct link to Apply Your Skill: Grader Creation") Design a grader for customer support responses using binary criteria. def grader_support_response(response: str, case: dict) -> dict: """ Binary criteria grader for customer support agent. Each criterion is yes/no. Sum them for score. """ checks = { # Criterion 1: Did it address the customer's issue? "addressed_issue": ( case["expected_topic"] in response.lower() ), # Criterion 2: Did it maintain professional tone? "professional_tone": not any( word in response.lower() for word in ["rude", "stupid", "whatever"] ), # Criterion 3: Did it provide actionable next steps? "has_next_steps": any( phrase in response.lower() for phrase in ["please", "you can", "next step", "i'll help"] ), # Criterion 4: Did it avoid making things up? "no_hallucination": not ( "our policy is" in response.lower() and case.get("has_no_policy", False) ), # Criterion 5: Did it know when to escalate? "appropriate_escalation": ( case.get("should_escalate", False) == ("human agent" in response.lower() or "escalate" in response.lower()) ) } score = sum(checks.values()) return { "passed": score == 5, "score": score, "max_score": 5, "checks": checks, "explanation": f"Passed {score}/5 support criteria" } **Output:** # Test the gradertest_case = { "input": "Where is my order?", "expected_topic": "order", "should_escalate": False}response = "I'd be happy to help you track your order. Please provide your order number and I'll look that up for you right away."result = grader_support_response(response, test_case)print(f"Score: {result['score']}/5")print(f"Checks: {result['checks']}") Score: 4/5Checks: {'addressed_issue': True, 'professional_tone': True, 'has_next_steps': True, 'no_hallucination': True, 'appropriate_escalation': True} If you created this grader using your skill's binary criteria pattern, the skill is working in a new domain. Documenting Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#documenting-your-skill "Direct link to Documenting Your Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your skill needs documentation that allows future you to use it without rereading the chapter. Complete this template in your `SKILL.md`: ---name: agent-evalsdescription: Design and implement evaluation frameworks for AI agents. Use when testing agent reasoning quality, building graders, doing error analysis, or establishing regression protection. Framework-agnostic concepts that apply to any SDK.---# Agent Evaluations: Measuring Reasoning Quality**Core Thesis**: "One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." - Andrew Ng## When to ActivateUse this skill when:- Building systematic quality checks for any AI agent- Designing evaluation datasets (typical, edge, error categories)- Creating graders to define "good" automatically- Performing error analysis to find failure patterns- Setting up regression protection for agent changes- Deciding when to use end-to-end vs component-level evals- Debugging why an agent's output quality is inconsistent- Preparing an agent for production deployment## Core Patterns### Pattern: Dataset Design (10-20 cases to start)Categories:- Typical (60%): Common use cases the agent will handle daily- Edge (25%): Unusual but valid inputs that test boundaries- Error (15%): Cases where agent should fail gracefullyUse REAL data from production logs when possible. Synthetic data misses the messiness of reality.### Pattern: Binary Criteria GradersDO NOT use 1-5 scales (LLMs are poorly calibrated).DO use binary criteria:1. Define 3-7 yes/no criteria2. Check each criterion independently3. Sum to get total score4. Threshold for pass/failTemplate:def grader(response, case) -> dict: checks = {"criterion_1": bool_check_1, "criterion_2": bool_check_2} score = sum(checks.values()) return {"passed": score == len(checks), "score": score, "checks": checks}### Pattern: Error Analysis (Spreadsheet Method)Columns: Case | Routing | Tool Selection | Output Format | Content QualityProcess:1. Run failing cases through eval suite2. Trace each failure to component3. Count which component fails most often4. Prioritize: frequency x feasibility### Pattern: Component vs E2E Decision5-step flow:1. Start with E2E evals to find overall quality2. Use error analysis to identify problem component3. Build component-level eval for that component4. Tune component using component eval5. Verify improvement with E2E eval### Pattern: Regression ProtectionWorkflow:Before change -> Run eval suite -> Establish baselineAfter change -> Run eval suite -> Compare to baselineIf drop > threshold -> Investigate before shippingThresholds by criticality:- High-stakes (medical, financial): Any drop = block- Normal (support, productivity): 5% drop = investigate- Experimental (prototypes): 10% drop = investigate## Framework Application| Framework | Trace Access | Grader Integration ||-----------|-------------|-------------------|| OpenAI Agents SDK | Built-in tracing | Custom graders || Claude Agent SDK | Hooks for tracing | Custom graders || Google ADK | Evaluation module | Built-in graders || LangChain | LangSmith traces | LangSmith evals || Custom | Logging middleware | Custom graders |## Anti-Patterns to Avoid| Anti-Pattern | Why It's Bad | What to Do Instead ||-------------|--------------|-------------------|| 1000+ test cases first | Quantity without quality | Start with 20 thoughtful cases || 1-5 scale ratings | LLMs poorly calibrated | Binary criteria summed || Ignoring traces | Miss root cause | Read intermediate outputs || End-to-end only | Too noisy for debugging | Add component-level evals || Synthetic test data | Misses real-world messiness | Use actual user queries |---*Skill Version: 1.0.0 | Created: Chapter 77 | Owner: [Your Name]* The Portable Thinking[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#the-portable-thinking "Direct link to The Portable Thinking") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The concepts you learned transfer across any agent framework because they address universal problems: | Concept | Why It Transfers | | --- | --- | | Evals vs TDD | Agents reason probabilistically everywhere | | Binary criteria | LLM calibration issues exist in all systems | | Error analysis | Multi-component agents fail similarly across frameworks | | Regression protection | Quality degradation happens regardless of SDK | | Dataset categories | Typical/edge/error applies to any domain | What does NOT transfer directly: | Concept | Framework-Specific Adaptation | | --- | --- | | Trace access | Each SDK has different tracing APIs | | Built-in graders | Google ADK has them; others don't | | Dataset storage | Varies by infrastructure | | CI/CD integration | Depends on deployment pipeline | Your skill should contain the portable patterns. Framework-specific details get added when you apply the skill to a particular SDK. Exercise: Test Your Skill on a Third Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#exercise-test-your-skill-on-a-third-agent "Direct link to Exercise: Test Your Skill on a Third Agent") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your skill has now been validated on Customer Support. For your final exercise, test it on one more agent type to confirm the patterns truly generalize. **Choose one:** 1. **Content Generation Agent**: Creates blog posts, social media content, marketing copy 2. **Code Review Agent**: Reviews pull requests, suggests improvements, catches bugs 3. **Data Analysis Agent**: Answers questions about datasets, creates visualizations, identifies trends Using only your skill: 1. Design a 10-case eval dataset (5 typical, 3 edge, 2 error) 2. Write 5 binary criteria for a grader 3. Identify which component would be hardest to evaluate 4. Decide: E2E eval or component-level first? Why? If you complete this exercise without returning to chapter content, your skill is production-ready. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Test Your Skill on a New Agent Type[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#prompt-1-test-your-skill-on-a-new-agent-type "Direct link to Prompt 1: Test Your Skill on a New Agent Type") I'm validating that my agent-evals skill is portable. I just learnedevaluation methodology using a Task API agent. Now I need to apply itto a completely different domain.My new agent is: [describe an agent in your actual domain]Using evaluation methodology (not any specific framework), help me:1. Design a 10-case eval dataset with typical, edge, and error categories2. Define 5 binary criteria for grading responses3. Identify which component would be hardest to trace errors back toI want to verify my evaluation thinking transfers, not learn new concepts. **What you're learning:** Skill portability requires active testing. Your evaluation thinking should work in any domain because you learned patterns, not examples. AI helps you apply those patterns to verify they transfer. ### Prompt 2: Generate a New Domain's Eval Dataset[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#prompt-2-generate-a-new-domains-eval-dataset "Direct link to Prompt 2: Generate a New Domain's Eval Dataset") I have an agent that [describe your real agent]. I need to design anevaluation dataset using the three-category approach:- Typical (60%): Common cases- Edge (25%): Unusual but valid- Error (15%): Should fail gracefullyGenerate 15 test cases for my agent following this structure. For each case:1. Input: What the user says/does2. Expected behavior: What the agent should do3. Category: Typical, edge, or error4. Why this category: Brief justificationUse realistic examples from my domain, not generic ones. **What you're learning:** Dataset design transfers across domains when you use the category framework. The specific cases differ by domain, but the structure remains constant. AI helps you generate domain-specific cases using your portable framework. ### Prompt 3: Create a Grader for Different Criteria[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#prompt-3-create-a-grader-for-different-criteria "Direct link to Prompt 3: Create a Grader for Different Criteria") I need to evaluate a [describe your agent type] using binary criteria.The subjective quality I care about is: [describe what "good" means]Help me:1. Break this subjective quality into 5-7 binary yes/no criteria2. For each criterion, suggest how to check it (string matching, keyword presence, LLM judge)3. Identify which criteria need LLM-as-Judge vs can be checked with codeRemember: No 1-5 scales. Each criterion must be decidable as true/false. **What you're learning:** The binary criteria pattern applies to any subjective quality you need to measure. Breaking "good" into checkable components is a skill that transfers across every agent you will ever build. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#safety-note "Direct link to Safety Note") Skills evolve through use. The version you finalize today is not the final version. As you apply this skill to more agents, you will encounter patterns not covered here. When that happens, update your skill. A living skill grows stronger with each use. A frozen skill becomes obsolete. Build the habit of returning to your skills and adding what you learn in practice. Checking access... * [What Your Skill Should Include](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#what-your-skill-should-include) * [Skill Validation Checklist](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#skill-validation-checklist) * [Testing on a Different Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#testing-on-a-different-agent) * [Apply Your Skill: Dataset Design](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#apply-your-skill-dataset-design) * [Apply Your Skill: Grader Creation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#apply-your-skill-grader-creation) * [Documenting Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#documenting-your-skill) * [The Portable Thinking](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#the-portable-thinking) * [Exercise: Test Your Skill on a Third Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#exercise-test-your-skill-on-a-third-agent) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#try-with-ai) * [Prompt 1: Test Your Skill on a New Agent Type](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#prompt-1-test-your-skill-on-a-new-agent-type) * [Prompt 2: Generate a New Domain's Eval Dataset](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#prompt-2-generate-a-new-domains-eval-dataset) * [Prompt 3: Create a Grader for Different Criteria](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#prompt-3-create-a-grader-for-different-criteria) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/finalize-your-evals-skill#safety-note) --- # Sampling — Servers Calling LLMs | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your research tool needs to summarize documents. Your code analysis tool needs to explain complex patterns. Your customer support server needs to reason about edge cases. These tools require AI inference. But you face a problem: **If the server manages its own API key**, it's tightly coupled to a specific model provider. It becomes expensive (running continuously), complex (error handling, rate limiting), and insecure (keys embedded everywhere). **If the server asks the client to call Claude**, costs shift to the client, complexity vanishes, and the server stays provider-agnostic. This is sampling. Sampling is the MCP mechanism that lets servers request LLM inference from clients without managing keys, credentials, or provider relationships. The server asks, "Hey client, can you run Claude on this?" The client responds, "Sure, here's what Claude says." Deterministic operations stay on the server. Reasoning operations route through the client's models. This is how you build hybrid systems: fast operations on the server, reasoning on the frontier model, scaling without complexity. The Sampling Problem and Solution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#the-sampling-problem-and-solution "Direct link to The Sampling Problem and Solution") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's look at what you need vs what sampling gives you: ### The Naive Approach (Don't Do This)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#the-naive-approach-dont-do-this "Direct link to The Naive Approach (Don't Do This)") import anthropicimport os@mcp.tool()async def summarize(text_to_summarize: str): """Summarize research findings.""" # PROBLEM: Server needs its own API key client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY")) message = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1000, messages=[{ "role": "user", "content": f"Summarize: {text_to_summarize}" }] ) return message.content[0].text **Problems**: * Server has API key embedded (security risk) * Server pays for inference (cost leakage) * Server depends on Anthropic specifically (no provider flexibility) * Server must handle Claude errors directly (complexity) * Server can't use client's context window (isolated reasoning) ### The Sampling Approach (Do This Instead)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#the-sampling-approach-do-this-instead "Direct link to The Sampling Approach (Do This Instead)") @mcp.tool()async def summarize(text_to_summarize: str, ctx: Context): """Summarize research findings using sampling.""" # Client provides inference—server just specifies intent result = await ctx.session.create_message( messages=[ SamplingMessage( role="user", content=TextContent(type="text", text=f"Summarize: {text_to_summarize}") ) ], max_tokens=1000, system_prompt="You are a research expert summarizing academic findings" ) if result.content.type == "text": return result.content.text else: raise ValueError("Sampling failed") **Benefits**: * No API keys on server (secure) * Client pays for inference (aligned costs) * Client determines model (flexible) * Client handles errors (centralized) * Server can leverage client's context (powerful) How Sampling Works: The Complete Flow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#how-sampling-works-the-complete-flow "Direct link to How Sampling Works: The Complete Flow") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The pattern involves two sides working together: ### Server Side: Request Inference[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#server-side-request-inference "Direct link to Server Side: Request Inference") from mcp import Contextfrom mcp.types import SamplingMessage, TextContent@mcp.tool()async def analyze_code(code_snippet: str, ctx: Context): """Analyze Python code using Claude.""" # Server specifies WHAT to do result = await ctx.session.create_message( messages=[ SamplingMessage( role="user", content=TextContent( type="text", text=f"Identify bugs and suggest fixes:\n{code_snippet}" ) ) ], max_tokens=2000, system_prompt="You are an expert Python code reviewer. Focus on bugs, security issues, and performance." ) # Unpack response if result.content and result.content[0].type == "text": analysis = result.content[0].text return { "analysis": analysis, "source": "claude-3-5-sonnet-20241022" } else: raise ValueError("Code analysis failed") **What's happening**: * Server creates `SamplingMessage` (not a regular message—signals "route to client") * Calls `ctx.session.create_message()` (special method for sampling) * Specifies `system_prompt` (instructions for Claude) * Awaits response from client's Claude ### Client Side: Handle Sampling Callback[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#client-side-handle-sampling-callback "Direct link to Client Side: Handle Sampling Callback") from mcp.client.session import ClientSessionfrom mcp.types import CreateMessageRequestParams, CreateMessageResultasync def sampling_callback( request_id: str, params: CreateMessageRequestParams): """Intercept server's sampling requests and route through Claude.""" # Client has API key and model access—server doesn't client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY")) # Convert sampling request to standard Claude API call response = client.messages.create( model="claude-3-5-sonnet-20241022", # Client decides model max_tokens=params.max_tokens, system=params.system_prompt, messages=params.messages # Already formatted by server ) # Return result back to server return CreateMessageResult( model="claude-3-5-sonnet-20241022", content=[TextContent( type="text", text=response.content[0].text )], role="assistant" )# Register callback with clientsession = ClientSession( transport, init_options={ "sampling_callback": sampling_callback }) **What's happening**: * Client defines `sampling_callback` (intercepts `create_message` requests) * Client owns API key (server never sees it) * Client calls Claude (determines model, handles errors) * Client returns structured response to server Comparison: Three Approaches[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#comparison-three-approaches "Direct link to Comparison: Three Approaches") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Factor | Direct Server Call | Sampling | API Pass-Through | | --- | --- | --- | --- | | **Server complexity** | High (API management) | Low (just request) | Medium | | **Cost management** | Server pays | Client pays | Hidden | | **Security** | Keys on server (risky) | Keys on client only (safe) | Keys duplicated | | **Model flexibility** | Fixed to server's choice | Client controls model | Fixed upstream | | **Error handling** | Server handles | Client handles | Mixed | | **Context awareness** | Server context only | Client's full context | Limited | | **Use case** | Never (anti-pattern) | Inference requests | Data transform | **Key insight**: Sampling shifts both cost and responsibility to the client (where they belong). The server focuses on what it knows best; the client focuses on inference. Real-World Sampling Example: Research Assistant[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#real-world-sampling-example-research-assistant "Direct link to Real-World Sampling Example: Research Assistant") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's a complete example combining research operations (server) with AI reasoning (sampling): **Server: Research Tool** import httpxfrom mcp import Contextfrom mcp.types import SamplingMessage, TextContent@mcp.tool()async def research_topic(topic: str, ctx: Context): """Research a topic and synthesize findings.""" # Step 1: Server fetches raw research (deterministic) async with httpx.AsyncClient() as client: response = await client.get( f"https://api.arxiv.org/query", params={"search_query": f"all:{topic}", "max_results": 5} ) papers = response.json() # Step 2: Server extracts key information (deterministic) research_data = { "papers": [ { "title": p["title"], "abstract": p["summary"][:500] # First 500 chars } for p in papers[:5] ] } # Step 3: Server asks Claude to synthesize (sampling) synthesis = await ctx.session.create_message( messages=[ SamplingMessage( role="user", content=TextContent( type="text", text=f"""Synthesize these research papers into a coherent summary.Papers:{json.dumps(research_data['papers'], indent=2)}Create:1. Key findings2. Research gaps3. Future directions""" ) ) ], max_tokens=1500, system_prompt="You are a research analyst synthesizing academic findings for executives." ) # Step 4: Return synthesized findings if synthesis.content and synthesis.content[0].type == "text": return { "raw_papers": len(research_data['papers']), "synthesis": synthesis.content[0].text } else: raise ValueError("Synthesis failed") **Client Usage**: # Client registers callbackasync def handle_research_sampling( request_id: str, params: CreateMessageRequestParams): """Route research synthesis through Claude.""" response = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=params.max_tokens, system=params.system_prompt, messages=params.messages ) return CreateMessageResult( model="claude-3-5-sonnet-20241022", content=response.content, role="assistant" )# When client calls the research toolresponse = await session.call_tool( name="research_topic", arguments={"topic": "transformer architectures"})# Output:# {# "raw_papers": 5,# "synthesis": "Transformers have evolved from..."# } **What you see**: * Server fetches data, formats it * Server asks Claude to synthesize (sampling) * Claude responds through client * Server returns synthesized result to caller The server never needed an API key. The client never needed to manage fetching. Clean separation. When to Use Sampling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#when-to-use-sampling "Direct link to When to Use Sampling") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Use sampling when**: * Server needs to reason about data it collected * Tool requires natural language understanding * You want client to bear inference costs * Tool should work with any frontier model (not tied to one provider) * You need the client's broader context **Examples**: * Summarizing documents (server retrieves, Claude synthesizes) * Analyzing code (server parses, Claude explains) * Content moderation (server checks patterns, Claude evaluates intent) * Decision support (server fetches data, Claude recommends action) **Don't use sampling when**: * Operation is purely deterministic (pure computation, data lookup) * Operation must complete in under 500ms (sampling has latency) * Server can't format data properly (client shouldn't guess intent) * Cost must be server-controlled (sampling routes to client) Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Part 1: Review the Code** You have a document processing server that needs to classify documents. Read this code snippet: @mcp.tool()async def classify_document(content: str, ctx: Context): """Classify document using sampling.""" classification = await ctx.session.create_message( messages=[ SamplingMessage( role="user", content=TextContent( type="text", text=f"Classify this document:\n{content}" ) ) ], max_tokens=200 ) return classification.content[0].text Ask Claude: **"This server tool uses sampling to classify documents. What's missing from the implementation that would make it production-ready?"** Pay attention to Claude's suggestions about: * System prompts (does Claude have guidance?) * Error handling (what if sampling fails?) * Response validation (is the classification in the expected format?) **Part 2: Implement Error Handling** Claude will likely suggest handling sampling failures. Based on its suggestions, ask: **"Show me how to add retry logic and graceful fallback if sampling fails."** Review the code Claude generates. Ask yourself: * Does this handle the case where sampling times out? * Does this prevent infinite retry loops? * Does the fallback return sensible output? **Part 3: Design a Sampling Workflow** You're building a code review server. It needs to: 1. Parse Python code (deterministic) 2. Ask Claude for security issues (sampling) 3. Ask Claude for performance suggestions (sampling) 4. Return both analyses Ask Claude: **"Design a tool that does code review using two separate sampling calls—one for security, one for performance. Should these be separate tools or one tool with two sampling requests?"** Compare Claude's recommendation to your instinct. Ask: **"Why did you recommend this approach? What's the tradeoff?"** This question forces you to think about sampling composition—when to use multiple samples vs when to combine them. **Part 4: Evaluate Tradeoffs** You're deciding: should your research server call Claude for each paper (5 sampling calls) or fetch all papers then call Claude once? Ask Claude: **"Compare these two approaches: (A) sample Claude for each paper's summary individually, or (B) fetch all papers, then sample Claude once for synthesis. What are the latency, cost, and quality tradeoffs?"** Notice what emerges: Claude will explain reasoning you might not have considered (parallelization, context window efficiency, response quality). Ask yourself: **"Which tradeoff matters most for my use case?"** This is sampling in action—not just moving code around, but reasoning about when AI involvement creates value vs when it adds latency without benefit. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#reflect-on-your-skill "Direct link to Reflect on Your Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built an `mcp-server` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#test-your-skill "Direct link to Test Your Skill") Using my mcp-server skill, create a tool that uses sampling to call an LLM through the client.Does my skill explain when to use sampling vs direct API calls, and how to implement context.session.create_message()? ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include sampling patterns and SamplingMessage structures? * Did it explain the tradeoffs between server-side API calls and client-side sampling? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My mcp-server skill is missing sampling implementation patterns.Update it to include when sampling is appropriate, how to use context.session.create_message(), and the architectural benefits of delegating LLM calls to clients. * * * Checking access... * [The Sampling Problem and Solution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#the-sampling-problem-and-solution) * [The Naive Approach (Don't Do This)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#the-naive-approach-dont-do-this) * [The Sampling Approach (Do This Instead)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#the-sampling-approach-do-this-instead) * [How Sampling Works: The Complete Flow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#how-sampling-works-the-complete-flow) * [Server Side: Request Inference](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#server-side-request-inference) * [Client Side: Handle Sampling Callback](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#client-side-handle-sampling-callback) * [Comparison: Three Approaches](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#comparison-three-approaches) * [Real-World Sampling Example: Research Assistant](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#real-world-sampling-example-research-assistant) * [When to Use Sampling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#when-to-use-sampling) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#try-with-ai) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/sampling-servers-calling-llms#improve-your-skill) --- # What to Remember and What to Forget | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C A memory that stores everything is as useless as one that stores nothing. When you ask your agent about a preference, it shouldn't wade through years of irrelevant conversations to find the answer. And when a user says "forget everything about me," the agent must comply—completely and immediately. Good memory systems are selective. They prioritize what matters, compress what's old, and delete what shouldn't exist. This lesson teaches you how to design those selection mechanisms. The Memory Prioritization Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#the-memory-prioritization-problem "Direct link to The Memory Prioritization Problem") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Agents interact with users constantly. Every conversation generates potential memories. Without prioritization, you face three problems: | Problem | Impact | | --- | --- | | **Storage costs** | Memory grows unbounded. Vector databases cost money. | | **Retrieval noise** | Too many memories = wrong memories retrieved. | | **Contradiction chaos** | Old information conflicts with new information. | The solution is **selective memory**—storing what's valuable and actively managing what isn't. Relevance Scoring[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#relevance-scoring "Direct link to Relevance Scoring") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not all memories are equally valuable. A relevance scoring system helps prioritize which memories to keep, retrieve, and surface. ### Scoring Factors[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#scoring-factors "Direct link to Scoring Factors") Three factors typically influence memory relevance: **1\. Semantic Similarity** How closely does this memory match the current query? Query: "What are Alex's task preferences?"Memory A: "Alex prefers morning tasks" → High similarity (0.92)Memory B: "Alex completed 5 tasks yesterday" → Medium similarity (0.65)Memory C: "Weather was sunny last Tuesday" → Low similarity (0.12) **2\. Recency** How recently was this memory created or accessed? Memory A: Created 2 days ago → High recencyMemory B: Created 30 days ago → Medium recencyMemory C: Created 1 year ago → Low recency **3\. Access Frequency** How often is this memory retrieved? Memory A: Retrieved 15 times → Frequently accessedMemory B: Retrieved 3 times → Occasionally accessedMemory C: Retrieved 0 times → Never accessed ### The Scoring Formula[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#the-scoring-formula "Direct link to The Scoring Formula") A weighted combination of these factors produces a relevance score: def calculate_relevance(memory, query, current_time): """ Score = w1 * semantic_similarity + w2 * recency_decay + w3 * access_frequency Weights should sum to 1.0 """ # Semantic similarity (0 to 1) semantic = cosine_similarity( embed(memory.text), embed(query) ) # Recency decay (exponential, 0 to 1) days_old = (current_time - memory.created_at).days recency = math.exp(-days_old / 30) # Half-life of ~30 days # Access frequency (normalized, 0 to 1) max_accesses = get_max_access_count() frequency = memory.access_count / max_accesses if max_accesses > 0 else 0 # Weighted combination w1, w2, w3 = 0.5, 0.3, 0.2 return w1 * semantic + w2 * recency + w3 * frequency ### Tuning Weights for Use Cases[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#tuning-weights-for-use-cases "Direct link to Tuning Weights for Use Cases") Different agents need different weight distributions: | Agent Type | Semantic (w1) | Recency (w2) | Frequency (w3) | | --- | --- | --- | --- | | Customer Support | 0.6 | 0.3 | 0.1 | | Personal Assistant | 0.4 | 0.4 | 0.2 | | Research Agent | 0.7 | 0.1 | 0.2 | | Task Manager | 0.5 | 0.3 | 0.2 | A customer support agent cares most about matching the current issue (semantic). A personal assistant cares about recent context (recency). A research agent cares about topic relevance regardless of when it was learned. Memory Consolidation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#memory-consolidation "Direct link to Memory Consolidation") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Old detailed memories consume space and add noise. **Consolidation** compresses them into summaries without losing essential information. ### The Consolidation Process[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#the-consolidation-process "Direct link to The Consolidation Process") BEFORE (10 detailed memories):├── "Jan 1: Alex created task 'Fix login bug', priority high"├── "Jan 2: Alex completed 'Fix login bug' in 3 hours"├── "Jan 3: Alex created task 'Add password reset', priority normal"├── "Jan 4: Alex completed 'Add password reset' in 2 hours"├── "Jan 5: Alex created task 'Review PR #45', priority high"├── "Jan 6: Alex completed 'Review PR #45' in 1 hour"├── "Jan 7: Alex created task 'Update docs', priority low"├── "Jan 8: Alex completed 'Update docs' in 4 hours"├── "Jan 9: Alex created task 'Deploy to staging', priority critical"└── "Jan 10: Alex completed 'Deploy to staging' in 2 hours"AFTER (1 consolidated memory):└── "Week of Jan 1-10: Alex completed 5 tasks averaging 2.4 hours each. Priority distribution: 1 critical, 2 high, 1 normal, 1 low. Pattern: Tasks consistently completed same day as created." ### Consolidation Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#consolidation-strategies "Direct link to Consolidation Strategies") **1\. Time-based Consolidation** Daily details → Weekly summary → Monthly summary → Yearly summaryTrigger: When memory age exceeds threshold- After 7 days: Daily → Weekly- After 30 days: Weekly → Monthly- After 365 days: Monthly → Yearly **2\. Count-based Consolidation** When memories in a category exceed N, consolidate oldest half.Trigger: Memory count exceeds threshold- If task_memories > 100: consolidate oldest 50 into summary **3\. Importance-based Retention** Never consolidate high-importance memories.Only consolidate routine, low-value memories.High value (keep detailed): Decisions, preferences, errorsLow value (consolidate): Routine completions, minor updates ### Consolidation Example Code[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#consolidation-example-code "Direct link to Consolidation Example Code") def consolidate_old_memories(user_id: str, days_threshold: int = 30): """Consolidate memories older than threshold into summaries.""" cutoff = datetime.now() - timedelta(days=days_threshold) # Get old memories old_memories = memory.search( query="", filters={ "user_id": user_id, "created_at": {"lte": cutoff.isoformat()} } ) # Group by category by_category = group_memories_by_category(old_memories) for category, memories in by_category.items(): if len(memories) < 5: # Don't consolidate small groups continue # Generate summary using LLM summary = generate_summary(memories) # Store summary as new memory memory.add([{ "role": "system", "content": f"Consolidated summary of {category}: {summary}" }], user_id=user_id, metadata={ "type": "summary", "category": category, "consolidated_count": len(memories), "date_range": f"{memories[0].created_at} to {memories[-1].created_at}" }) # Delete original detailed memories for mem in memories: memory.delete(mem.id) Active Forgetting[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#active-forgetting "Direct link to Active Forgetting") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Some information must be deleted, not just deprioritized. Active forgetting is a critical capability. ### When to Forget[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#when-to-forget "Direct link to When to Forget") **1\. User Request** User: "Forget that I mentioned my salary."Agent: Deletes all memories containing salary information. **2\. Privacy Compliance (GDPR Article 17)** The "right to be forgotten" requires complete deletion upon user request: async def handle_deletion_request(user_id: str): """GDPR-compliant complete user deletion.""" # Get ALL user memories all_memories = memory.search( query="", filters={"user_id": user_id}, limit=10000 # Get everything ) # Delete each one deleted_count = 0 for mem in all_memories['results']: memory.delete(mem['id']) deleted_count += 1 # Audit log (required for compliance) log_deletion_event( user_id=user_id, deleted_count=deleted_count, timestamp=datetime.now(), reason="user_request_gdpr" ) return {"status": "deleted", "count": deleted_count} **3\. Outdated Information** When new information supersedes old: Old memory: "Alex's phone number is 555-1234"New statement: "My new phone number is 555-5678"Action: Delete old memory, store new one **4\. Time-based Expiration** Some memories should auto-expire: # Store with expirationmemory.add( messages, user_id=user_id, metadata={ "expires_at": (datetime.now() + timedelta(days=90)).isoformat(), "type": "temporary_context" })# Cleanup job (run daily)def cleanup_expired_memories(): expired = memory.search( query="", filters={"expires_at": {"lte": datetime.now().isoformat()}} ) for mem in expired['results']: memory.delete(mem['id']) ### Privacy Requirements Summary[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#privacy-requirements-summary "Direct link to Privacy Requirements Summary") | Requirement | Implementation | | --- | --- | | User deletion request | Delete all user memories immediately | | Consent withdrawal | Delete memories created after consent given | | Purpose limitation | Only store memories for stated purpose | | Data minimization | Don't store more than necessary | | Accuracy | Correct or delete inaccurate memories | | Audit trail | Log all deletions with timestamps | Contradiction Resolution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#contradiction-resolution "Direct link to Contradiction Resolution") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Users change. Preferences evolve. Facts update. Your memory system must handle contradictions gracefully. ### Detection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#detection "Direct link to Detection") def detect_contradiction(new_memory: str, existing_memories: list) -> list: """Find memories that contradict the new information.""" contradictions = [] for existing in existing_memories: # Use LLM to detect contradiction prompt = f""" Existing memory: {existing.content} New information: {new_memory} Do these contradict each other? If yes, explain how. Respond with: CONTRADICTION: [explanation] or NO_CONTRADICTION """ result = llm.generate(prompt) if result.startswith("CONTRADICTION"): contradictions.append({ "existing": existing, "explanation": result.split(": ", 1)[1] }) return contradictions ### Resolution Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#resolution-strategies "Direct link to Resolution Strategies") **1\. Timestamp-based (Newer Wins)** def resolve_by_timestamp(old_memory, new_memory): """Most recent information wins.""" if new_memory.created_at > old_memory.created_at: memory.delete(old_memory.id) memory.add(new_memory) return "replaced_with_newer" return "kept_older" **2\. Explicit Update** User: "I'm vegetarian now"Old memory: "Loves steak dinners"Agent: "I notice this contradicts an earlier memory about food preferences. I'll update my notes to reflect that you're now vegetarian. Should I remove all previous food preference memories?" **3\. Version History** def update_with_history(memory_id, new_content): """Update memory while preserving history.""" existing = memory.get(memory_id) # Store version history history = existing.metadata.get("history", []) history.append({ "content": existing.content, "valid_until": datetime.now().isoformat() }) # Update with new content memory.update(memory_id, new_content, metadata={ "history": history, "updated_at": datetime.now().isoformat() }) ### Example Contradiction Flow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#example-contradiction-flow "Direct link to Example Contradiction Flow") Turn 1 (January):User: "I prefer morning meetings"→ Store: {"preference": "morning_meetings", "created": "2025-01"}Turn 2 (March):User: "Actually, afternoon works better for me now"Detection:→ Search finds: "morning meetings" preference→ LLM identifies: CONTRADICTIONResolution:→ Option A: Replace silently (timestamp wins)→ Option B: Confirm with user (explicit update)→ Option C: Keep both with dates (version history)Agent chooses Option A:→ Delete: "morning meetings" memory→ Store: "afternoon meetings" preference with March date→ Reply: "Got it—I've updated my notes. Afternoons from now on." Safety Note: Memory Privacy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#safety-note-memory-privacy "Direct link to Safety Note: Memory Privacy") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Memory systems handle sensitive personal information. Always implement: 1. **Encryption at rest** — Memories stored in encrypted databases 2. **Access controls** — Only authorized services read memories 3. **Audit logging** — Track all access and modifications 4. **Consent management** — Store proof of user consent 5. **Data minimization** — Don't store what you don't need Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these prompts to practice memory prioritization design with Claude or your preferred AI assistant. ### Prompt 1: Design a Relevance Scoring System[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#prompt-1-design-a-relevance-scoring-system "Direct link to Prompt 1: Design a Relevance Scoring System") I'm building an e-commerce customer support agent. Design a relevance scoring system for its memories.The agent handles:- Order inquiries- Return requests- Product questions- Shipping issuesFor each of these factors, explain how you'd weight them and why:1. Semantic similarity to current query2. Recency of the memory3. Access frequencyCreate a scoring formula and show how it would rank these 4 memories when the user asks "Where is my order?":- Memory A: "Order #123 shipped 2 days ago" (created 2 days ago, accessed 5 times)- Memory B: "User prefers email updates" (created 30 days ago, accessed 12 times)- Memory C: "User had delivery issue last month" (created 30 days ago, accessed 3 times)- Memory D: "User's name is Alex" (created 60 days ago, accessed 20 times) **What you're learning:** How to tune scoring weights for a specific domain. E-commerce support cares heavily about order recency—a 2-day-old shipping memory is gold when the user asks about delivery. ### Prompt 2: Handle Contradictions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#prompt-2-handle-contradictions "Direct link to Prompt 2: Handle Contradictions") A user's stated preferences have evolved over time:Timeline:- January: "I love steak, it's my favorite food"- March: "I'm trying to eat less red meat"- June: "I'm fully vegetarian now"- September: User asks: "Can you recommend a restaurant for my birthday dinner?"Questions:1. How should the agent handle this preference evolution?2. Should it keep all three memories, or consolidate them?3. What should it recommend, and how should it reference the user's dietary journey?4. What if in November the user says "I'm eating meat again"—how does that affect the memory?Design a contradiction resolution strategy for dietary preferences that handles:- Gradual evolution- Explicit reversals- Temporary vs permanent changes **What you're learning:** Contradictions aren't always "wrong"—sometimes they're evolution. A good memory system captures change over time rather than just keeping the latest value. ### Prompt 3: Consolidation Strategy for Customer Support[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#prompt-3-consolidation-strategy-for-customer-support "Direct link to Prompt 3: Consolidation Strategy for Customer Support") A customer support agent has accumulated these memories about a user over 6 months:[15 memories about individual support tickets][8 memories about product preferences][12 memories about billing inquiries][5 memories about feature requests]Design a consolidation strategy that answers:1. Which memories should be consolidated and which kept detailed?2. What triggers consolidation (time? count? both?)3. What information must the summaries preserve?4. How do you handle a situation where a consolidated ticket becomes relevant again?Create example "before and after" for the support ticket category, showing:- The 15 individual ticket memories- The consolidated summary- What was preserved vs. lost **What you're learning:** Consolidation is lossy compression—you choose what to keep. For support tickets, patterns matter more than details: "User has had 3 billing issues" is more useful than 3 separate ticket summaries after 6 months. Checking access... * [The Memory Prioritization Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#the-memory-prioritization-problem) * [Relevance Scoring](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#relevance-scoring) * [Scoring Factors](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#scoring-factors) * [The Scoring Formula](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#the-scoring-formula) * [Tuning Weights for Use Cases](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#tuning-weights-for-use-cases) * [Memory Consolidation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#memory-consolidation) * [The Consolidation Process](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#the-consolidation-process) * [Consolidation Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#consolidation-strategies) * [Consolidation Example Code](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#consolidation-example-code) * [Active Forgetting](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#active-forgetting) * [When to Forget](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#when-to-forget) * [Privacy Requirements Summary](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#privacy-requirements-summary) * [Contradiction Resolution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#contradiction-resolution) * [Detection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#detection) * [Resolution Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#resolution-strategies) * [Example Contradiction Flow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#example-contradiction-flow) * [Safety Note: Memory Privacy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#safety-note-memory-privacy) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#try-with-ai) * [Prompt 1: Design a Relevance Scoring System](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#prompt-1-design-a-relevance-scoring-system) * [Prompt 2: Handle Contradictions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#prompt-2-handle-contradictions) * [Prompt 3: Consolidation Strategy for Customer Support](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/what-to-remember-and-forget#prompt-3-consolidation-strategy-for-customer-support) --- # Connecting Your First Agent | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Chapter 70 built TaskManager as a REST API. But conversations aren't request/response cycles—they're ongoing relationships. ChatKit Server transforms your agents from stateless endpoints into conversational interfaces. In this lesson, you'll connect the agents you built in Chapters 62-65 to ChatKit's conversation infrastructure. The result: token-by-token streaming, automatic history management, and session persistence—all without writing streaming logic yourself. The Conversation Architecture Shift[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#the-conversation-architecture-shift "Direct link to The Conversation Architecture Shift") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### FastAPI Pattern (Chapter 70)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#fastapi-pattern-chapter-70 "Direct link to FastAPI Pattern (Chapter 70)") @app.post("/tasks")def create_task(task: TaskCreate) -> TaskResponse: # Execute operation result = process_task(task) # Return single response return result **Characteristics:** * One request → One response * Stateless (no conversation memory) * Synchronous (wait for completion) * Client handles all state management ### ChatKit Pattern (This Chapter)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#chatkit-pattern-this-chapter "Direct link to ChatKit Pattern (This Chapter)") class MyAgentServer(ChatKitServer): async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: # Stream events progressively async for event in stream_agent_response(...): yield event **Characteristics:** * Ongoing conversation → Stream of events * Stateful (ChatKit manages history) * Asynchronous (progressive updates) * Server manages conversation state **The key insight**: ChatKit handles the conversation mechanics (history, streaming, sessions). You implement one method: `respond()`. The respond() Method Contract[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#the-respond-method-contract "Direct link to The respond() Method Contract") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every ChatKit server extends `ChatKitServer` and implements `respond()`: from chatkit import ChatKitServer, ThreadMetadata, UserMessageItem, ThreadStreamEventfrom typing import AsyncIterator, Anyclass TaskAgentServer(ChatKitServer): async def respond( self, thread: ThreadMetadata, # Conversation container input: UserMessageItem | None, # User's message context: Any, # Session/user context ) -> AsyncIterator[ThreadStreamEvent]: # Your agent logic here pass ### Parameter Breakdown[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#parameter-breakdown "Direct link to Parameter Breakdown") **`thread: ThreadMetadata`** * Contains conversation metadata (thread ID, timestamps) * Provides access to conversation history * Managed by ChatKit automatically **`input: UserMessageItem | None`** * The user's message content * Can be None (for example, on conversation initialization) * Contains text, attachments, metadata **`context: Any`** * RequestContext with user identity and session information * Used for multi-tenant isolation * Typically contains user ID, permissions, session data **Returns: `AsyncIterator[ThreadStreamEvent]`** * Stream of events (assistant messages, tool status, widgets, tasks) * Token-by-token output for real-time UI updates * Yielded progressively, not returned all at once Connecting an OpenAI Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#connecting-an-openai-agent "Direct link to Connecting an OpenAI Agent") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the complete pattern for integrating an OpenAI Agents SDK agent into ChatKit: from chatkit import ChatKitServer, ThreadMetadata, UserMessageItem, ThreadStreamEventfrom chatkit.agents import stream_agent_responsefrom agents import Agent, Runnerfrom typing import AsyncIterator, Anyclass TaskAgentServer(ChatKitServer): def __init__(self): super().__init__() # Define your agent once self.agent = Agent( name="TaskManager", instructions="""You are a task management assistant. Help users create, organize, and track their tasks.""", tools=[ # Your tools from Chapter 62-36 ], ) async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: # Extract user message user_message = input.content if input else "" # Run agent with streaming result = Runner.run_streamed( self.agent, user_message, ) # Stream agent response through ChatKit async for event in stream_agent_response(context, result): yield event **Output:** User: Create a task "Review pull requests" for tomorrowAssistant: I've created the task "Review pull requests" scheduled for tomorrow...[Token-by-token streaming visible in ChatKit UI] ### How It Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#how-it-works "Direct link to How It Works") **Step 1**: User sends message → ChatKit calls `respond()` **Step 2**: `respond()` extracts message from `input` parameter **Step 3**: Agent processes message using `Runner.run_streamed()` (streaming mode) **Step 4**: `stream_agent_response()` converts agent output to ChatKit events **Step 5**: Events yielded progressively → ChatKit UI updates in real-time **The magic**: `stream_agent_response()` is a helper that bridges agent SDKs to ChatKit's event format. You don't write streaming logic—you call the helper. Connecting a Google ADK Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#connecting-a-google-adk-agent "Direct link to Connecting a Google ADK Agent") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Google ADK uses the `Runner` pattern with session services. While Google ADK has different patterns, ChatKit provides `stream_agent_response()` to handle event conversion: from chatkit import ChatKitServer, ThreadMetadata, UserMessageItem, ThreadStreamEventfrom chatkit.agents import AgentContext, stream_agent_responsefrom google.adk.runners import Runnerfrom google.adk.sessions import InMemorySessionServicefrom google.genai import typesfrom typing import AsyncIterator, Anyclass TaskAgentServer(ChatKitServer): def __init__(self, adk_agent): super().__init__() # Store your pre-built Google ADK agent from Chapter 63 self.agent = adk_agent # Initialize ADK session service self.session_service = InMemorySessionService() # Initialize ADK runner self.runner = Runner( app_name='chatkit_task_agent', agent=self.agent, session_service=self.session_service, ) async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: user_message = input.content if input else "" user_id = getattr(context, 'user_id', 'default-user') # Create AgentContext for ChatKit integration agent_context = AgentContext( thread=thread, store=self.store, request_context=context ) # Create or retrieve session for this thread session = self.session_service.create_session( state={}, app_name='chatkit_task_agent', user_id=user_id ) # Format message for ADK message = types.Content( role='user', parts=[types.Part.from_text(text=user_message)] ) # Stream agent response - ChatKit helper handles event conversion result = self.runner.run_async( session_id=session.id, user_id=user_id, new_message=message ) # stream_agent_response() converts ADK events to ChatKit format async for event in stream_agent_response(agent_context, result): yield event **Output:** User: List my tasks for this weekAssistant: Here are your tasks for this week:1. Review pull requests (Tomorrow)2. Team meeting (Wednesday)...[Streaming in real-time] **Key differences from OpenAI**: 1. **Runner pattern**: Google ADK uses `Runner` with `run_async()` instead of direct agent calls 2. **Session management**: Requires `InMemorySessionService` for conversation state 3. **Content objects**: Messages wrapped in `types.Content` with `types.Part` 4. **Event conversion**: `stream_agent_response()` helper handles ChatKit event formatting automatically Connecting a Claude Agent SDK Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#connecting-a-claude-agent-sdk-agent "Direct link to Connecting a Claude Agent SDK Agent") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Claude Agent SDK uses the standalone `query()` function with options. ChatKit's `stream_agent_response()` handles event conversion: from chatkit import ChatKitServer, ThreadMetadata, UserMessageItem, ThreadStreamEventfrom chatkit.agents import AgentContext, stream_agent_responsefrom claude_agent_sdk import query, ClaudeAgentOptionsfrom typing import AsyncIterator, Anyclass TaskAgentServer(ChatKitServer): def __init__(self): super().__init__() # Configure Claude agent options self.agent_options = ClaudeAgentOptions( allowed_tools=["Read", "Write", "Bash"], system_prompt="You are a task management assistant. Help users create, organize, and track their tasks.", model="claude-sonnet-4-20250514", max_turns=10, include_partial_messages=True, # Enable streaming ) async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: user_message = input.content if input else "" # Create AgentContext for ChatKit integration agent_context = AgentContext( thread=thread, store=self.store, request_context=context ) # Query Claude Agent SDK with streaming result = query(prompt=user_message, options=self.agent_options) # stream_agent_response() handles all event conversion # Including thinking blocks, tool calls, and text responses async for event in stream_agent_response(agent_context, result): yield event **Output:** User: What's the status of my high-priority tasks?[Hidden: Claude's thinking about task prioritization logic]Assistant: Let me check your high-priority tasks...[Tool call executes]You have 3 high-priority tasks:- Review security audit (Overdue)...[Streaming progressively] **Key differences from OpenAI**: 1. **Standalone function**: Uses `query()` function, not `Agent` class or API client 2. **ClaudeAgentOptions**: Configure system prompt, tools, and model through options object 3. **Message types**: Handle `AssistantMessage` with typed content blocks 4. **ThinkingBlock handling**: Extended thinking tokens visible in stream (can hide as `HiddenContextItem`) 5. **No helper needed**: Manual type checking instead of `stream_agent_response()` Running Your ChatKit Server[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#running-your-chatkit-server "Direct link to Running Your ChatKit Server") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Once you've implemented `respond()`, run the server: # server.pyfrom chatkit import ChatKitServerclass TaskAgentServer(ChatKitServer): # ... respond() implementation from aboveif __name__ == "__main__": import uvicorn server = TaskAgentServer() uvicorn.run(server.app, host="0.0.0.0", port=8000) **Terminal:** python server.py# Output:# INFO: Started server process# INFO: Uvicorn running on http://0.0.0.0:8000 **Testing:** Open browser at `http://localhost:8000` → ChatKit's built-in UI loads → Start conversing with your agent. Common Integration Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#common-integration-patterns "Direct link to Common Integration Patterns") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Pattern 1: Reusing Chapter 62-36 Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#pattern-1-reusing-chapter-62-36-agents "Direct link to Pattern 1: Reusing Chapter 62-36 Agents") You already built agents. Don't rebuild—integrate: # Import your existing agentfrom chapter_34_openai_agents.task_agent import create_task_agentclass TaskAgentServer(ChatKitServer): def __init__(self): super().__init__() # Reuse existing agent definition self.agent = create_task_agent() async def respond(self, thread, input, context): result = Runner.run_streamed(self.agent, input.content) async for event in stream_agent_response(context, result): yield event **Why this works**: Agent logic is separate from conversation infrastructure. ChatKit handles the conversation; your agent handles the domain logic. ### Pattern 2: Handling Empty Input[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#pattern-2-handling-empty-input "Direct link to Pattern 2: Handling Empty Input") First message might be empty (conversation initialization): from chatkit.types import AssistantMessageEventasync def respond(self, thread, input, context): if not input or not input.content: # Send welcome message yield AssistantMessageEvent( content="Hello! I'm your task management assistant. How can I help?" ) return # Normal agent processing result = Runner.run_streamed(self.agent, input.content) async for event in stream_agent_response(context, result): yield event **Output:** [User opens chat - no message sent yet]Assistant: Hello! I'm your task management assistant. How can I help? ### Pattern 3: Error Handling in Stream[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#pattern-3-error-handling-in-stream "Direct link to Pattern 3: Error Handling in Stream") Wrap agent execution to catch errors gracefully: from chatkit.types import AssistantMessageEventasync def respond(self, thread, input, context): try: result = Runner.run_streamed(self.agent, input.content) async for event in stream_agent_response(context, result): yield event except Exception as e: # Send error message to user yield AssistantMessageEvent( content=f"I encountered an error: {str(e)}. Please try again." ) **Output:** User: Create a task with invalid date "tomorrow yesterday"Assistant: I encountered an error: Invalid date format. Please try again. **Why this matters**: Unhandled exceptions break the conversation stream. Users see frozen UI. Error handling keeps conversation flowing. Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#safety-note "Direct link to Safety Note") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- When testing locally, ChatKit runs without authentication by default. This is great for development but **never deploy to production without authentication** (covered in Lesson 6). Anyone with the URL can access your agent. For local testing, this is fine. For production: authentication is mandatory. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion (Claude, ChatGPT, or Gemini). ### Prompt 1: Integration Verification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#prompt-1-integration-verification "Direct link to Prompt 1: Integration Verification") I just connected my [OpenAI/Google/Anthropic] agent to ChatKit using therespond() method. Help me verify the integration is correct.My agent from Chapter [34/35/36] does [describe functionality].Here's my respond() implementation: [paste code]Ask me:1. Does stream_agent_response() receive the right agent output format?2. Am I handling the case where input is None?3. What happens if my agent throws an error mid-stream? **What you're learning:** Critical verification through AI-guided code review—identifying integration gaps before runtime failures. ### Prompt 2: Streaming Troubleshooting[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#prompt-2-streaming-troubleshooting "Direct link to Prompt 2: Streaming Troubleshooting") My ChatKit server runs, but I don't see token-by-token streaming.The entire response appears at once.My setup:- Agent SDK: [OpenAI/Google/Anthropic]- Using stream_agent_response(): [yes/no]- Agent call method: [paste your Runner.run_streamed() or equivalent]What's wrong with my streaming implementation? **What you're learning:** Pattern debugging—AI helps you identify whether the issue is in agent streaming setup or ChatKit integration. ### Prompt 3: Multi-Agent Extension[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#prompt-3-multi-agent-extension "Direct link to Prompt 3: Multi-Agent Extension") I have multiple agents from Chapters 62-65:- TaskManager (OpenAI)- ResearchAgent (Google)- CodeReviewer (Anthropic)How would I extend my ChatKitServer to let users choose which agent to talk toin the same conversation?Don't write the full code—ask me:1. Should agent selection happen in respond() or outside it?2. How should users specify which agent they want?3. Where should I store the "active agent" state for this conversation? **What you're learning:** Architectural exploration—using AI to discover design patterns for multi-agent ChatKit servers before implementing. * [The Conversation Architecture Shift](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#the-conversation-architecture-shift) * [FastAPI Pattern (Chapter 70)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#fastapi-pattern-chapter-70) * [ChatKit Pattern (This Chapter)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#chatkit-pattern-this-chapter) * [The respond() Method Contract](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#the-respond-method-contract) * [Parameter Breakdown](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#parameter-breakdown) * [Connecting an OpenAI Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#connecting-an-openai-agent) * [How It Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#how-it-works) * [Connecting a Google ADK Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#connecting-a-google-adk-agent) * [Connecting a Claude Agent SDK Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#connecting-a-claude-agent-sdk-agent) * [Running Your ChatKit Server](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#running-your-chatkit-server) * [Common Integration Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#common-integration-patterns) * [Pattern 1: Reusing Chapter 62-36 Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#pattern-1-reusing-chapter-62-36-agents) * [Pattern 2: Handling Empty Input](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#pattern-2-handling-empty-input) * [Pattern 3: Error Handling in Stream](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#pattern-3-error-handling-in-stream) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#safety-note) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#try-with-ai) * [Prompt 1: Integration Verification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#prompt-1-integration-verification) * [Prompt 2: Streaming Troubleshooting](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#prompt-2-streaming-troubleshooting) * [Prompt 3: Multi-Agent Extension](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/connecting-your-first-agent#prompt-3-multi-agent-extension) --- # LangChain Document Processing | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your Task API stores task descriptions as plain text. But when you want to find semantically similar tasks—like "show me tasks related to container deployment" when the actual text says "configure Kubernetes pods"—you need to prepare documents for vector search. The first step: loading your content and splitting it into chunks that preserve meaning. This lesson teaches you the document processing pipeline that sits between your raw content and the vector store. You'll learn to load documents from multiple sources, split them intelligently, and preserve the metadata that makes retrieval useful. Why Document Processing Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#why-document-processing-matters "Direct link to Why Document Processing Matters") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Raw documents don't go directly into vector databases. Consider a 50-page PDF about your API documentation. If you embed the entire document as one chunk, you'll get one vector representing the average meaning of everything. Search quality suffers because the vector can't represent specific sections accurately. The solution: split documents into smaller chunks that each represent a focused concept. But splitting naively—every 500 characters regardless of content—breaks sentences mid-thought and loses context. LangChain solves this with intelligent document loaders and text splitters that: * Extract content while preserving structure * Split at natural boundaries (paragraphs, sentences) * Maintain overlap so context isn't lost between chunks * Preserve metadata for filtering and citation Document Loaders: Getting Content In[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#document-loaders-getting-content-in "Direct link to Document Loaders: Getting Content In") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- LangChain provides specialized loaders for different content types. Each returns a list of `Document` objects—the universal format for content in LangChain. ### The Document Object[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#the-document-object "Direct link to The Document Object") Every document loader produces `Document` objects with two properties: from langchain_core.documents import Documentdoc = Document( page_content="The actual text content goes here", metadata={"source": "api-docs.pdf", "page": 3, "section": "Authentication"}) **Output:** Document(page_content='The actual text content goes here', metadata={'source': 'api-docs.pdf', 'page': 3, 'section': 'Authentication'}) The `page_content` holds the text. The `metadata` dictionary carries source information—essential for citing where answers came from. ### TextLoader: Plain Text Files[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#textloader-plain-text-files "Direct link to TextLoader: Plain Text Files") The simplest loader reads text files directly: from langchain_community.document_loaders import TextLoaderloader = TextLoader("task_descriptions.txt", encoding="utf-8")docs = loader.load()print(f"Loaded {len(docs)} document(s)")print(f"Content preview: {docs[0].page_content[:100]}...")print(f"Metadata: {docs[0].metadata}") **Output:** Loaded 1 document(s)Content preview: # Task Descriptions## Setup Docker EnvironmentConfigure Docker for development including...Metadata: {'source': 'task_descriptions.txt'} TextLoader automatically adds the source file path as metadata. You'll use this to trace search results back to their origin. ### PyPDFLoader: PDF Documents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pypdfloader-pdf-documents "Direct link to PyPDFLoader: PDF Documents") PDF files require specialized parsing. PyPDFLoader extracts text page by page: from langchain_community.document_loaders import PyPDFLoaderloader = PyPDFLoader("api_reference.pdf")docs = loader.load()print(f"Loaded {len(docs)} pages")print(f"Page 1 preview: {docs[0].page_content[:150]}...")print(f"Page 1 metadata: {docs[0].metadata}") **Output:** Loaded 24 pagesPage 1 preview: Task API Reference GuideVersion 2.0This document describes the REST API endpoints for task management including creation, retrieval, update...Metadata: {'source': 'api_reference.pdf', 'page': 0} Notice how each page becomes a separate Document. The metadata includes both source and page number—invaluable when users ask "where did this answer come from?" ### WebBaseLoader: Web Pages[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#webbaseloader-web-pages "Direct link to WebBaseLoader: Web Pages") Web content requires HTML parsing. WebBaseLoader fetches pages and extracts text: import bs4from langchain_community.document_loaders import WebBaseLoader# Parse only specific elements to avoid navigation/footer noisebs4_strainer = bs4.SoupStrainer(class_=("post-content", "article-body"))loader = WebBaseLoader( web_paths=("https://docs.example.com/task-api/quickstart",), bs_kwargs={"parse_only": bs4_strainer},)docs = loader.load()print(f"Loaded {len(docs)} document(s)")print(f"Source: {docs[0].metadata.get('source')}") **Output:** Loaded 1 document(s)Source: https://docs.example.com/task-api/quickstart The `bs4_strainer` filters HTML elements—you get article content without navigation menus, footers, and sidebars that would pollute your embeddings. Text Splitting: Chunking for Retrieval[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#text-splitting-chunking-for-retrieval "Direct link to Text Splitting: Chunking for Retrieval") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Documents loaded, now what? A 10,000-character document won't embed well. You need chunks—but smart chunks that preserve meaning. ### The Chunking Challenge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#the-chunking-challenge "Direct link to The Chunking Challenge") Consider this paragraph being split at exactly 50 characters: "Configure Docker for development including network settings and volume mounts." Naive split at character 50: * Chunk 1: "Configure Docker for development including netwo" * Chunk 2: "rk settings and volume mounts." The word "network" is broken. Neither chunk makes complete sense. This damages embedding quality. ### RecursiveCharacterTextSplitter: The Solution[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#recursivecharactertextsplitter-the-solution "Direct link to RecursiveCharacterTextSplitter: The Solution") LangChain's `RecursiveCharacterTextSplitter` splits intelligently using a hierarchy of separators: from langchain_text_splitters import RecursiveCharacterTextSplittertext_splitter = RecursiveCharacterTextSplitter( chunk_size=1000, # Target size in characters chunk_overlap=200, # Overlap between chunks add_start_index=True, # Track position in original separators=["\n\n", "\n", " ", ""] # Split hierarchy) **How it works:** 1. First tries to split on `"\n\n"` (paragraph breaks) 2. If chunks are still too large, splits on `"\n"` (line breaks) 3. Then on `" "` (spaces) 4. Only uses `""` (characters) as last resort This preserves semantic boundaries. Paragraphs stay together when possible. ### Applying the Splitter[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#applying-the-splitter "Direct link to Applying the Splitter") from langchain_text_splitters import RecursiveCharacterTextSplitterfrom langchain_community.document_loaders import TextLoader# Load a documentloader = TextLoader("task_descriptions.txt", encoding="utf-8")docs = loader.load()# Configure splittertext_splitter = RecursiveCharacterTextSplitter( chunk_size=500, chunk_overlap=100, add_start_index=True,)# Split documentssplits = text_splitter.split_documents(docs)print(f"Original: {len(docs)} document(s)")print(f"After splitting: {len(splits)} chunk(s)")print(f"\nFirst chunk ({len(splits[0].page_content)} chars):")print(splits[0].page_content[:200] + "...")print(f"\nMetadata: {splits[0].metadata}") **Output:** Original: 1 document(s)After splitting: 12 chunk(s)First chunk (487 chars):# Task Descriptions## Setup Docker EnvironmentConfigure Docker for development including network settings and volume mounts. This task covers:- Installing Docker Desktop- Configuring daemon.json- Setting up Docker Compose...Metadata: {'source': 'task_descriptions.txt', 'start_index': 0} Notice `start_index: 0`—this tracks where in the original document each chunk came from. Essential for context expansion later. ### Understanding Chunk Overlap[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#understanding-chunk-overlap "Direct link to Understanding Chunk Overlap") Why overlap? Imagine searching for "Docker network configuration": Without overlap: * Chunk 1 ends: "...Docker for development including" * Chunk 2 starts: "network settings and volume mounts..." The relevant content spans both chunks, but neither chunk alone captures the full context. With 20% overlap (200 chars for 1000-char chunks), both chunks contain the transitional content. **Visualizing overlap:** Document: [==========CHUNK 1==========] [==========CHUNK 2==========] [==========CHUNK 3==========] |__________________| overlap ### Choosing Parameters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#choosing-parameters "Direct link to Choosing Parameters") Parameter selection depends on your use case: | Parameter | Small Values | Large Values | | --- | --- | --- | | `chunk_size` | More chunks, precise retrieval, higher cost | Fewer chunks, more context per chunk, lower cost | | `chunk_overlap` | Risk losing boundary context | Better continuity, more redundancy | **Guidelines for starting values:** * **chunk\_size=1000**: Good default. Fits most embedding model context windows. * **chunk\_overlap=200**: 20% overlap balances continuity vs redundancy. * **add\_start\_index=True**: Always enable for debugging and context expansion. Adjust based on experimentation—which you'll do in the Try With AI section. Metadata Preservation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#metadata-preservation "Direct link to Metadata Preservation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Metadata flows through the entire pipeline. This matters more than you might expect. ### Why Metadata Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#why-metadata-matters "Direct link to Why Metadata Matters") When your RAG system answers "How do I configure task priorities?", users want to know: * Which document contained that information? * What page or section? * When was it written? Without metadata, you have answers without sources. With metadata, you can cite and link back. ### Metadata Through Splitting[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#metadata-through-splitting "Direct link to Metadata Through Splitting") Watch how metadata propagates: from langchain_core.documents import Documentfrom langchain_text_splitters import RecursiveCharacterTextSplitter# Document with rich metadatadoc = Document( page_content="Docker configuration guide. First, install Docker Desktop...", metadata={ "source": "devops-guide.pdf", "page": 5, "section": "Container Setup", "author": "DevOps Team", "updated": "2025-01-15" })splitter = RecursiveCharacterTextSplitter( chunk_size=200, chunk_overlap=50, add_start_index=True)splits = splitter.split_documents([doc])for i, split in enumerate(splits): print(f"Chunk {i}: {split.metadata}") **Output:** Chunk 0: {'source': 'devops-guide.pdf', 'page': 5, 'section': 'Container Setup', 'author': 'DevOps Team', 'updated': '2025-01-15', 'start_index': 0}Chunk 1: {'source': 'devops-guide.pdf', 'page': 5, 'section': 'Container Setup', 'author': 'DevOps Team', 'updated': '2025-01-15', 'start_index': 178} Every chunk inherits the parent's metadata, plus gets `start_index`. This enables: * **Filtering**: "Search only DevOps Team documents" * **Citation**: "Answer found on page 5 of devops-guide.pdf" * **Context expansion**: "Get 500 chars before and after this chunk" Complete Processing Pipeline[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#complete-processing-pipeline "Direct link to Complete Processing Pipeline") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Let's combine everything into a reusable pipeline: from langchain_community.document_loaders import TextLoader, PyPDFLoaderfrom langchain_text_splitters import RecursiveCharacterTextSplitterfrom pathlib import Pathdef process_documents(file_paths: list[str], chunk_size: int = 1000, chunk_overlap: int = 200) -> list: """Load and split documents from multiple sources.""" all_docs = [] for path in file_paths: file_path = Path(path) # Choose loader based on file type if file_path.suffix == ".pdf": loader = PyPDFLoader(str(file_path)) elif file_path.suffix in [".txt", ".md"]: loader = TextLoader(str(file_path), encoding="utf-8") else: print(f"Skipping unsupported file: {path}") continue docs = loader.load() all_docs.extend(docs) print(f"Loaded {len(docs)} document(s) from {path}") # Split all documents splitter = RecursiveCharacterTextSplitter( chunk_size=chunk_size, chunk_overlap=chunk_overlap, add_start_index=True, ) splits = splitter.split_documents(all_docs) print(f"\nTotal chunks: {len(splits)}") return splits# Example usagefiles = ["task_descriptions.txt", "api_reference.pdf"]chunks = process_documents(files, chunk_size=800, chunk_overlap=150) **Output:** Loaded 1 document(s) from task_descriptions.txtLoaded 24 document(s) from api_reference.pdfTotal chunks: 89 This pipeline handles mixed content types, applies consistent splitting, and preserves all metadata. The chunks are ready for embedding and storage in Qdrant (next lesson). Common Pitfalls and Solutions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#common-pitfalls-and-solutions "Direct link to Common Pitfalls and Solutions") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Pitfall 1: Chunks Too Small[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pitfall-1-chunks-too-small "Direct link to Pitfall 1: Chunks Too Small") If `chunk_size=100`, you get many tiny chunks that lack context: * "Configure Docker" * "for development" * "including network" Each chunk is too fragmented to represent a useful concept. **Solution**: Start with `chunk_size=1000` and decrease only if chunks contain too many unrelated concepts. ### Pitfall 2: No Overlap[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pitfall-2-no-overlap "Direct link to Pitfall 2: No Overlap") With `chunk_overlap=0`, boundary content gets lost. Searches for concepts that span chunk boundaries fail. **Solution**: Use 10-20% overlap. For 1000-char chunks, 100-200 overlap. ### Pitfall 3: Ignoring Metadata[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pitfall-3-ignoring-metadata "Direct link to Pitfall 3: Ignoring Metadata") Loading documents without preserving source information makes debugging impossible and removes citation capability. **Solution**: Always keep metadata from loaders. Add custom metadata when loading programmatically: doc = Document( page_content=content, metadata={"source": "manual_entry", "category": "tasks", "priority": "high"}) Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#reflect-on-your-skill "Direct link to Reflect on Your Skill") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built a `rag-deployment` skill in Lesson 0. Now test it with document processing concepts. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#test-your-skill "Direct link to Test Your Skill") Ask your skill: > "I have a 500-page PDF manual. What chunk\_size and chunk\_overlap would you recommend, and why?" ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#identify-gaps "Direct link to Identify Gaps") Does your skill know about: * Different document loaders for different file types? * The separator hierarchy in RecursiveCharacterTextSplitter? * Why add\_start\_index enables context expansion? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#improve-your-skill "Direct link to Improve Your Skill") If gaps exist, update your skill with document processing patterns from this lesson. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#try-with-ai "Direct link to Try With AI") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Document processing parameters significantly affect RAG quality. Experiment to understand the tradeoffs. ### Prompt 1: Compare Chunking Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#prompt-1-compare-chunking-strategies "Direct link to Prompt 1: Compare Chunking Strategies") I have a markdown file with 15 sections, each about 2000 characters.Compare these chunking approaches:1. chunk_size=500, chunk_overlap=1002. chunk_size=2000, chunk_overlap=4003. chunk_size=1000, chunk_overlap=200For each, explain:- How many chunks I'd get approximately- What retrieval behavior to expect- When this configuration makes sense **What you're learning:** How parameter choices create different retrieval characteristics. Smaller chunks = precise but fragmented. Larger chunks = more context but less focused. ### Prompt 2: Debug a Loading Pipeline[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#prompt-2-debug-a-loading-pipeline "Direct link to Prompt 2: Debug a Loading Pipeline") I'm loading documents with this code but getting empty results:loader = TextLoader("docs/README.md")docs = loader.load()print(len(docs)) # Shows 0The file definitely exists and has content. Help me debug:1. What could cause empty results?2. How do I verify the file is being read correctly?3. What encoding issues might occur? **What you're learning:** Troubleshooting document loading failures—encoding mismatches, path issues, and verification techniques. ### Prompt 3: Design for Your Domain[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#prompt-3-design-for-your-domain "Direct link to Prompt 3: Design for Your Domain") I'm building RAG for my Task API documentation. The content includes:- API endpoint descriptions (200-500 chars each)- Code examples (50-300 chars)- Conceptual explanations (1000-3000 chars)Design a chunking strategy that:1. Keeps API endpoints as complete units2. Preserves code examples intact3. Handles long explanations appropriatelyInclude specific parameter recommendations and explain your reasoning. **What you're learning:** Applying chunking strategy to real-world content with varying structure—the core skill for production RAG systems. * * * **Safety Note**: When loading documents from external URLs with WebBaseLoader, only fetch from trusted sources. Malicious content could affect your embeddings and RAG outputs. * * * Checking access... * [Why Document Processing Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#why-document-processing-matters) * [Document Loaders: Getting Content In](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#document-loaders-getting-content-in) * [The Document Object](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#the-document-object) * [TextLoader: Plain Text Files](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#textloader-plain-text-files) * [PyPDFLoader: PDF Documents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pypdfloader-pdf-documents) * [WebBaseLoader: Web Pages](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#webbaseloader-web-pages) * [Text Splitting: Chunking for Retrieval](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#text-splitting-chunking-for-retrieval) * [The Chunking Challenge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#the-chunking-challenge) * [RecursiveCharacterTextSplitter: The Solution](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#recursivecharactertextsplitter-the-solution) * [Applying the Splitter](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#applying-the-splitter) * [Understanding Chunk Overlap](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#understanding-chunk-overlap) * [Choosing Parameters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#choosing-parameters) * [Metadata Preservation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#metadata-preservation) * [Why Metadata Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#why-metadata-matters) * [Metadata Through Splitting](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#metadata-through-splitting) * [Complete Processing Pipeline](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#complete-processing-pipeline) * [Common Pitfalls and Solutions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#common-pitfalls-and-solutions) * [Pitfall 1: Chunks Too Small](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pitfall-1-chunks-too-small) * [Pitfall 2: No Overlap](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pitfall-2-no-overlap) * [Pitfall 3: Ignoring Metadata](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#pitfall-3-ignoring-metadata) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#improve-your-skill) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#try-with-ai) * [Prompt 1: Compare Chunking Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#prompt-1-compare-chunking-strategies) * [Prompt 2: Debug a Loading Pipeline](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#prompt-2-debug-a-loading-pipeline) * [Prompt 3: Design for Your Domain](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/vector-databases-rag-langchain/langchain-document-processing#prompt-3-design-for-your-domain) --- # Coordinator Patterns | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Imagine your AI agent researches breaking news across dozens of sources, compiles market data, analyzes trends, and synthesizes findings into a structured report. If you ask the agent to do all this in real-time and stream everything back to you immediately, you'll be drowning in verbose output while waiting for the final analysis. The **coordinator-dispatcher pattern** solves this by separating user-facing communication from background work. The agent acknowledges your request immediately, processes everything silently in the background, saves structured results to persistent storage, and finally confirms completion with the output location. This pattern is essential for production agents because it: 1. **Improves UX**: Users see immediate feedback and a final summary, not a firehose of intermediate steps 2. **Enables persistence**: Work persists across sessions (files, databases, APIs) 3. **Scales processing**: Complex multi-step work runs without blocking user interaction 4. **Provides evidence**: Structured output files serve as audit trails and re-usable artifacts In this lesson, you'll implement a coordinator agent that researches AI news, enriches it with financial data, and saves results to a markdown report—all while keeping the user experience clean and focused. The Coordinator-Dispatcher Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#the-coordinator-dispatcher-problem "Direct link to The Coordinator-Dispatcher Problem") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Raw Output Overwhelms Users[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#raw-output-overwhelms-users "Direct link to Raw Output Overwhelms Users") When agents execute research directly and stream results back, users see everything: Agent: Searching for AI news...[Tool: google_search | "latest AI news 2025"]Result: 47 headlines found...Result: Processing headline 1 of 47...Result: Extracting company info...[Tool: get_financial_context | "OpenAI stock price"]Result: OPENAI (NVDA): $890.25 (+2.3%)...Result: Analyzing market impact...Result: Writing report... This is technically transparent, but pedagogically terrible for users. They don't care about the plumbing—they want the final research report. ### Solution: Two-Message Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#solution-two-message-architecture "Direct link to Solution: Two-Message Architecture") The coordinator pattern uses exactly two user-facing messages: 1. **Acknowledgment** (immediate): "I'll research that for you. This might take a moment." 2. **Completion** (after silent processing): "Done! Results saved to ai\_research\_report.md" Everything between acknowledgment and completion happens silently, invisible to the user. User: "Research the latest AI news and save a report" ↓[Acknowledgment Message]Agent: "Okay, I'll research that. This might take a moment." ↓[Silent Background Processing — All work happens here]Agent internally: - Searches for news (invisible) - Enriches with financial data (invisible) - Formats structured report (invisible) - Saves to file (invisible) ↓[Completion Message]Agent: "All done. Report saved to ai_research_report.md" This keeps the user interaction clean while the agent does heavy lifting silently. File Persistence Tool[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#file-persistence-tool "Direct link to File Persistence Tool") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The coordinator pattern requires a tool that saves structured output to files. Let's build one. ### Design Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#design-specification "Direct link to Design Specification") Before writing code, specify what this tool should do: **Intent**: Save markdown-formatted content to files in the current working directory **Success Criteria**: * Accepts filename and markdown content * Validates filename (adds .md extension if missing) * Writes to current working directory using pathlib for cross-platform compatibility * Returns status dict with success/error message * Handles exceptions gracefully (full path in output, error messages clear) **Constraints**: * Always writes to current directory (no path traversal) * Overwrites existing files silently * Uses UTF-8 encoding ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#implementation "Direct link to Implementation") Here's the file persistence tool: import pathlibfrom typing import Dictdef save_news_to_markdown(filename: str, content: str) -> Dict[str, str]: """ Saves markdown content to a file in the current directory. This tool is designed for coordinator agents that process work silently and save structured results. It enables persistent storage of agent output. Args: filename: Name of file to save (e.g., 'ai_news.md'). If filename doesn't end with .md, extension is added automatically. content: Markdown-formatted string to write to file Returns: Dictionary with operation status: { "status": "success" | "error", "message": Human-readable status message, "filepath": Absolute path to saved file (on success only) } Example: >>> result = save_news_to_markdown( ... "research_report", ... "# AI News\n\n- Breaking: New model released" ... ) >>> print(result["message"]) Successfully saved to /Users/you/project/research_report.md """ try: # Validate and normalize filename if not filename: return { "status": "error", "message": "Filename cannot be empty" } # Add .md extension if missing if not filename.endswith(".md"): filename += ".md" # Get current directory using pathlib (cross-platform safe) current_directory = pathlib.Path.cwd() file_path = current_directory / filename # Write content to file with UTF-8 encoding file_path.write_text(content, encoding="utf-8") # Return success with absolute path return { "status": "success", "message": f"Successfully saved to {file_path.resolve()}" } except Exception as e: # Catch any IO errors and return error status return { "status": "error", "message": f"Failed to save file: {str(e)}" } ### What This Tool Does[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#what-this-tool-does "Direct link to What This Tool Does") **Specification → Implementation**: * "Validates filename" → Checks for empty string, adds .md extension * "Uses pathlib" → `pathlib.Path.cwd()` and `/` operator (cross-platform safe) * "Returns status dict" → Always returns `{"status": "...", "message": "..."}` * "Handles exceptions" → try/except catches IO errors **Key design decisions**: 1. **Always returns dict, never raises exceptions** → Agent can check status and adapt 2. **Absolute path in message** → User knows exactly where file was saved 3. **Auto-adds .md extension** → Reduces user burden (they can pass "report" or "report.md") 4. **UTF-8 encoding** → Handles special characters in markdown consistently ### Using This Tool in ADK[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#using-this-tool-in-adk "Direct link to Using This Tool in ADK") Add the tool to your agent: from google.adk.agents import Agentfrom google.adk.tools import google_search, get_financial_context# Import your custom toolfrom your_module import save_news_to_markdownagent = Agent( name="ai_news_coordinator", model="gemini-2.5-flash", instruction="""...""", tools=[google_search, get_financial_context, save_news_to_markdown]) The agent will now have access to the persistence tool. Coordinator Agent Instructions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#coordinator-agent-instructions "Direct link to Coordinator Agent Instructions") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now let's write the coordinator agent that uses the two-message pattern plus silent processing. ### Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#specification "Direct link to Specification") **Intent**: Research AI news and financial data, then save structured report **Required Behavior**: 1. First message: Acknowledge request 2. Silent phase: Execute all research tools (invisible to user) 3. Final message: Confirm completion with file location **Key constraint**: No streaming or intermediate output between acknowledgment and completion ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#implementation-1 "Direct link to Implementation") Here's the coordinator agent with proper instructions: from google.adk.agents import Agentfrom google.adk.tools import google_search, get_financial_contextfrom your_module import save_news_to_markdownroot_agent = Agent( name="ai_news_research_coordinator", model="gemini-2.5-flash", instruction="""You are a Background AI Research Coordinator. Your job is to research and save reports silently.## Your Identity and ConstraintsYou operate in the background, not as a conversational assistant. You accept one research request,process everything silently, and save results to a file. The user should see exactly two messagesfrom you: acknowledgment and completion.## Required Two-Message InteractionIMPORTANT: Follow this pattern strictly.1. **First Message (Acknowledgment)**: Immediately acknowledge the request "Okay, I'll research [topic]. This might take a moment."2. **Silent Processing Phase**: Execute all your work here (invisible to user) - Use google_search to find recent news - Use get_financial_context to gather market data - Format results into structured markdown - Call save_news_to_markdown to persist output - Do NOT send any intermediate messages or updates3. **Final Message (Completion)**: Confirm the work is done "All done. Report saved to [filepath]"## Work Guidelines### Research Phase (Silent)- Search for: "AI industry news 2025", "AI startup funding", "AI model releases"- Gather context on the top 3-5 stories by relevance- For each story, note: Company, Date, Impact### Formatting Phase (Silent)Use this markdown structure: # AI Industry News Report ## Top Headlines ### 1. [Headline] - **Company:** [Name] ([Ticker]) - **Date:** [YYYY-MM-DD] - **Summary:** [2-3 sentences on impact and significance] ### 2. [Headline] - **Company:** [Name] - **Date:** [YYYY-MM-DD] - **Summary:** [Summary] ## Market Context - Key sector trends - Investment activity - Regulatory news### Persistence Phase (Silent)- Call save_news_to_markdown("ai_research_report", formatted_content)- Check the returned status- If status=="success", use the filepath in completion message- If status=="error", report the error in completion message## Safety Notes- Never print intermediate steps- Never stream partial research results- Never ask for clarification once research begins- Always structure output as markdown (never raw text or JSON)- Always persist results to file before completing """, tools=[google_search, get_financial_context, save_news_to_markdown]) ### Why This Instruction Design Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#why-this-instruction-design-works "Direct link to Why This Instruction Design Works") **Two-message constraint**: The instruction explicitly says "two messages: acknowledgment and completion." This teaches the agent the pattern through explicit direction. **Silent processing emphasis**: Phrases like "invisible to user," "Do NOT send any intermediate messages," and repeated "Silent" headings reinforce that work happens without streaming. **Structured format**: The markdown template removes ambiguity about output format. The agent doesn't have to decide what to write—it follows the template. **Tool-aware workflow**: The instruction references specific tools (google\_search, get\_financial\_context, save\_news\_to\_markdown) in the order they should be used, guiding the agent's reasoning. Structured Report Schema[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#structured-report-schema "Direct link to Structured Report Schema") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The coordinator saves output using a markdown report template. Let's understand the structure: ### The Template[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#the-template "Direct link to The Template") # AI Industry News ReportGenerated: 2025-01-15 | Focus: Recent developments## Executive SummaryBrief overview of key themes from this research period.## Top Headlines### 1. [Headline]- **Company:** [Name] ([Stock Ticker])- **Date:** [YYYY-MM-DD]- **Category:** [Model Release | Funding | Partnership | Regulation]- **Summary:** [2-3 sentences on what happened and why it matters]- **Market Impact:** [Stock movement, funding valuation, industry implications]### 2. [Next Headline]- **Company:** [Name]- **Date:** [YYYY-MM-DD]- **Category:** [Category]- **Summary:** [Summary]- **Market Impact:** [Impact]## Market Context### Funding Activity- [Fund round]: $X million for [company] ([brief note])### Regulatory News- [Event]: [Brief description of regulatory action]### Technology Trends- [Trend]: [What's emerging, adoption rate]## Key Metrics| Metric | This Period | Previous | Trend ||--------|------------|----------|-------|| News stories | 12 | 8 | ↑ 50% || Funding announcements | 5 | 3 | ↑ 67% || Model releases | 2 | 1 | ↑ 100% |## Next StepsBased on these developments, watch for:- [Upcoming expected development]- [Regulatory event to monitor] ### Design Rationale[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#design-rationale "Direct link to Design Rationale") **Why markdown?** * Human readable (not JSON) * Versioning friendly (diffs work well) * GitHub-renderable (can be committed to repos) * No special tools needed to view (just a text editor) **Why this specific structure?** * **Headlines first** → User's most important information is at the top * **Metadata per story** → Date, company, ticker enable follow-up research * **Market Impact section** → Connects news to practical consequences * **Metrics table** → Shows comparative trends (up/down from previous) **Why these fields?** * **Company + Ticker** → User can quickly look up stock performance * **Category** → Helps user prioritize (funding vs regulation affect decisions differently) * **Date** → Essential for news (recency affects relevance) * **Summary** → Explains significance, not just facts Putting It Together: Complete Coordinator[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#putting-it-together-complete-coordinator "Direct link to Putting It Together: Complete Coordinator") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's a complete, working coordinator that processes research silently: from google.adk.agents import Agentfrom google.adk.tools import google_search, get_financial_contextimport pathlibfrom typing import Dict# File persistence tooldef save_news_to_markdown(filename: str, content: str) -> Dict[str, str]: try: if not filename.endswith(".md"): filename += ".md" file_path = pathlib.Path.cwd() / filename file_path.write_text(content, encoding="utf-8") return { "status": "success", "message": f"Successfully saved to {file_path.resolve()}" } except Exception as e: return {"status": "error", "message": f"Failed: {str(e)}"}# Coordinator agentcoordinator = Agent( name="ai_news_research_coordinator", model="gemini-2.5-flash", instruction="""You are a Background AI Research Coordinator that processes research requests silently.## Two-Message Pattern1. **Acknowledgment**: "Okay, I'll research the latest AI news and trends. This might take a moment."2. **Silent Processing**: Use google_search and get_financial_context to research. Format markdown. Save file.3. **Completion**: "All done. Report saved to ai_research_report.md"Do NOT send intermediate messages or ask for clarification.## Work InstructionsSearch for:1. Latest AI model releases (this month)2. Major funding announcements (last 30 days)3. Enterprise AI adoption newsStructure as markdown with Headlines, Market Context, and Key Metrics.Call save_news_to_markdown("ai_research_report", formatted_content) to persist.Always follow the two-message pattern. """, tools=[google_search, get_financial_context, save_news_to_markdown])# Use the coordinator:# result = coordinator.run("Research the latest AI news and trends")# The agent will acknowledge, process silently, save the report, and complete. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now you'll apply the coordinator pattern to a real research scenario. You'll experience how the two-message interaction keeps the user experience clean while the agent processes complex work silently. ### Setup[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#setup "Direct link to Setup") You'll research emerging AI capabilities from the last 90 days. The coordinator will gather news, enrich with financial context, structure the findings, and save a markdown report. **What you're learning:** * How coordinator-dispatcher separates user-facing communication from background work * When to use silent processing vs streaming results (and why the choice matters) * How structured output (markdown templates) removes ambiguity from agent work ### Prompt 1: Understand the Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#prompt-1-understand-the-pattern "Direct link to Prompt 1: Understand the Pattern") Ask AI: "I'm building a research coordinator agent that needs to research AI news, gather financial data, and save a report. The user should only see two messages: acknowledgment and completion. What are the key challenges in this design, and how do I structure the agent instructions to enforce the two-message pattern?" **What you're learning:** * AI articulates the pattern problem (intermediate messages disrupt UX) * AI explains how instructions can guide the two-message behavior * You internalize why structure matters in agent design ### Prompt 2: Design the File Persistence Tool[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#prompt-2-design-the-file-persistence-tool "Direct link to Prompt 2: Design the File Persistence Tool") Ask AI: "Write a Python function that saves markdown content to a file with proper error handling. It should return a status dict so the agent can know if the save succeeded or failed. The function should handle edge cases like missing filenames or permission errors gracefully." Compare AI's response to the implementation we provided above: * Does AI include error handling? * Does it return status information (not raise exceptions)? * Does it use pathlib for cross-platform paths? * Are the docstring and type hints clear enough for agent use? **What you're learning:** * How to design tools that agents can reason about (returning status vs raising exceptions) * Why error handling is essential when agents make decisions based on tool output * How documentation helps agents use tools correctly ### Prompt 3: Validate Your Understanding[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#prompt-3-validate-your-understanding "Direct link to Prompt 3: Validate Your Understanding") Ask AI: "I have a coordinator agent that searches for news and saves reports. Between the acknowledgment and completion messages, it should search, enrich with financial data, format a markdown report, and persist to a file. Write the agent instructions that make this two-message pattern explicit and prevent the agent from sending intermediate messages." Compare AI's instructions to the one provided in this lesson: * Does it explicitly mention "two messages"? * Does it say "silent" or "invisible" for the processing phase? * Does it reference the specific tools in order? * Does it guide markdown formatting? * Does it mention checking tool status? Evaluate: Which approach is clearer for the agent? Where does AI's version improve on this lesson's pattern? Where does this lesson's pattern provide better guidance? **What you're learning:** * How explicit instruction structure guides agent behavior * The importance of naming patterns ("two-message," "silent processing") in agent prompts * How to balance prescriptive guidance (follow this format) with flexibility (implement as you see fit) ### Reflection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#reflection "Direct link to Reflection") After working with AI on these prompts: * **Pattern recognition**: Can you identify coordinator-dispatcher in other agent workflows you've seen? * **Trade-offs**: When would you use coordinator (silent processing) vs streaming (real-time feedback)? When do each serve the user better? * **Implementation**: If you added a third tool (like audio generation), how would you maintain the two-message pattern while the agent does more work silently? * **Generalization**: This pattern works for research, report generation, and file persistence. What other workflows could benefit from the coordinator pattern? * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#reflect-on-your-skill "Direct link to Reflect on Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built a `google-adk` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#test-your-skill "Direct link to Test Your Skill") Using my google-adk skill, implement a coordinator agent with the two-message pattern.Does my skill demonstrate silent background processing and file persistence tools? ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include the two-message interaction pattern (acknowledgment → silent processing → completion)? * Did it show how to create file persistence tools using pathlib for cross-platform paths? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My google-adk skill is missing coordinator-dispatcher patterns.Update it to include:- Two-message pattern in agent instructions (acknowledgment, silent work, completion)- File persistence tool with pathlib.Path and UTF-8 encoding- Structured markdown report templates for agent output- Agent instructions that prevent intermediate message streaming- Error handling in file tools (return status dicts, not exceptions) * * * Checking access... * [The Coordinator-Dispatcher Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#the-coordinator-dispatcher-problem) * [Raw Output Overwhelms Users](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#raw-output-overwhelms-users) * [Solution: Two-Message Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#solution-two-message-architecture) * [File Persistence Tool](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#file-persistence-tool) * [Design Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#design-specification) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#implementation) * [What This Tool Does](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#what-this-tool-does) * [Using This Tool in ADK](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#using-this-tool-in-adk) * [Coordinator Agent Instructions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#coordinator-agent-instructions) * [Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#specification) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#implementation-1) * [Why This Instruction Design Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#why-this-instruction-design-works) * [Structured Report Schema](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#structured-report-schema) * [The Template](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#the-template) * [Design Rationale](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#design-rationale) * [Putting It Together: Complete Coordinator](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#putting-it-together-complete-coordinator) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#try-with-ai) * [Setup](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#setup) * [Prompt 1: Understand the Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#prompt-1-understand-the-pattern) * [Prompt 2: Design the File Persistence Tool](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#prompt-2-design-the-file-persistence-tool) * [Prompt 3: Validate Your Understanding](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#prompt-3-validate-your-understanding) * [Reflection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#reflection) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/google-adk-reliable-agents/coordinator-patterns#improve-your-skill) --- # Using Community MCP Servers | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You understand MCP architecture. You can configure clients. Now comes the practical question: **What servers actually exist, and how do you choose between them?** The MCP ecosystem has grown rapidly. There are reference servers maintained by Anthropic, community-created servers for specialized tasks, and enterprise servers from companies like Sentry and Datadog. This abundance is powerful—but it creates a discovery and evaluation challenge. How do you find servers? How do you know if a community server is maintained or abandoned? What security risks come with installing code from unknown authors? When should you build a custom server versus adapting an existing one? This lesson covers the entire server lifecycle: discovery through the official registry, evaluation criteria, installation and configuration, and composition into powerful multi-server workflows. By the end, you'll know how to build development environments that combine filesystem access, GitHub integration, database queries, and web search into a unified agent ecosystem. The MCP Ecosystem: Official, Community, and Enterprise[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#the-mcp-ecosystem-official-community-and-enterprise "Direct link to The MCP Ecosystem: Official, Community, and Enterprise") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The MCP server landscape has three tiers: ### Official Reference Servers (Maintained by Anthropic / The Linux Foundation)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#official-reference-servers-maintained-by-anthropic--the-linux-foundation "Direct link to Official Reference Servers (Maintained by Anthropic / The Linux Foundation)") These servers are vetted, well-documented, and receive security updates. They demonstrate MCP capabilities and serve as reference implementations. **Key official servers**: | Server | Purpose | Status | Best For | | --- | --- | --- | --- | | **Filesystem** | Read/write local files, create directories, search | Reference implementation | File-based workflows, development environments | | **GitHub** | Repository operations, issues, PRs, workflows | Official (owned by GitHub) | Version control, issue tracking, CI/CD integration | | **Git** | Generic git operations (different from GitHub) | Reference implementation | Low-level repository manipulation | | **Fetch** | HTTP requests, web content fetching | Reference implementation | API integration, web scraping | | **Memory** | Persistent knowledge graph across sessions | Reference implementation | Long-term context retention | | **Everything** | Test/reference server with all capabilities | Reference implementation | Development, testing, learning | **Where to find them**: [github.com/modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) **Why use official servers**: * Security reviews and vulnerability patches * Stable APIs and well-defined behavior * Complete documentation * Community adoption (you're not alone debugging issues) ### Community Servers (Created and Maintained by Developers)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#community-servers-created-and-maintained-by-developers "Direct link to Community Servers (Created and Maintained by Developers)") The community has created thousands of MCP servers for specialized domains: databases, monitoring tools, AI platforms, analytics, and more. **Popular community servers** (maintained as of 2025): | Server | Purpose | Maintenance | Language | | --- | --- | --- | --- | | **SQLite** / **PostgreSQL** | Database query and manipulation | Active | Python | | **Puppeteer** / **Playwright** | Browser automation | Active | Node.js / Python | | **Brave Search** | Web search without tracking | Active | Node.js | | **Slack** | Send messages, manage channels | Community-maintained | Python | | **Linear** | Issue tracking, project management | Active | Python | | **AWS** | EC2, S3, Lambda operations | Community-maintained | Python | | **Sentry** | Error monitoring and debugging | Active | Python | | **Datadog** | Metrics, logs, monitoring | Community-maintained | Python | | **Google Docs** | Read/write documents | Community-maintained | Python | | **Jira** | Project management, ticket operations | Community-maintained | Python | **Where to find community servers**: 1. **Official MCP Registry** at [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io/) — The authoritative source with metadata and ratings 2. **Curated lists** on GitHub (e.g., [wong2/awesome-mcp-servers](https://github.com/wong2/awesome-mcp-servers) , [punkpeye/awesome-mcp-servers](https://github.com/punkpeye/awesome-mcp-servers) ) 3. **GitHub search** for "mcp-server" repository topic **Important caveat**: Community servers are not officially vetted. Security, maintenance, and documentation vary widely. ### Enterprise Servers (Built by Companies for Their Platforms)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#enterprise-servers-built-by-companies-for-their-platforms "Direct link to Enterprise Servers (Built by Companies for Their Platforms)") Companies like Sentry, Datadog, Linear, and others have built official MCP servers for their platforms. These sit between community (unsupported) and official (fully maintained) servers. **Characteristics**: * Built and maintained by the company (not community volunteers) * Focus on their specific platform * Commercial support available (usually) * Security and stability important (these companies are accountable) Evaluating Server Quality and Trust[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#evaluating-server-quality-and-trust "Direct link to Evaluating Server Quality and Trust") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Not all community servers are created equal. Some are production-ready; others are abandoned experiments. Here's how to evaluate: ### Evaluation Checklist[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#evaluation-checklist "Direct link to Evaluation Checklist") Before installing any server, assess: **1\. Official vs Community Status** * Official (Anthropic/Linux Foundation maintained) = Highest trust * Enterprise (Sentry/Datadog/etc.) = High trust (company accountability) * Community = Variable (evaluate further) **2\. Maintenance Activity** Check the repository: * **Last commit date**: Within last 3 months = Active; Within last year = Maintained but slow; >1 year = Likely abandoned * **Open issues**: How many? Are they being addressed? * **Pull request velocity**: Are contributions being reviewed and merged? **3\. Security Considerations** Review what the server can do: * **What system resources does it access?** (files, network, environment variables) * **What operations are destructive?** (writes, deletes, modifications) * **Can it expose secrets?** (in logs, error messages, returned data) * **What permissions does it require?** (API keys, database credentials) **Risk framework**: Low Risk: Read-only operations on non-sensitive dataMedium Risk: Modifications or access to non-secret resourcesHigh Risk: System access, secret management, destructive operations **4\. Code Review Red Flags** If you can read the code: * Does it validate inputs before using them in system commands? * Does it log secrets or sensitive data? * Does it handle errors gracefully or expose internal details? * Does it limit scope to documented capabilities? **5\. Documentation Quality** * Clear README with examples? * List of all available tools/resources? * Configuration examples for each platform? * Troubleshooting guide? **Example**: GitHub MCP Server:- Official: YES (GitHub owns it now, Anthropic created it)- Maintenance: Active (multiple commits per week)- Risk: Medium (requires GitHub token, can create/modify issues)- Documentation: Excellent (full tool reference, examples)Verdict: SAFE TO USE Hypothetical "SuperAI-Master" server:- Official: NO (random GitHub user)- Maintenance: Last commit 18 months ago- Risk: UNKNOWN (code is obfuscated/minified)- Documentation: Missing (just one-liner description)Verdict: AVOID (too many unknowns) Installation: npm vs uvx vs Docker[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#installation-npm-vs-uvx-vs-docker "Direct link to Installation: npm vs uvx vs Docker") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- MCP servers are distributed as packages. Installation depends on runtime: ### Node.js Servers (via npx)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#nodejs-servers-via-npx "Direct link to Node.js Servers (via npx)") Most official servers are Node.js: # Install and run directly (no global install needed)npx -y @modelcontextprotocol/server-filesystem /path/to/serve# In configuration (JSON), reference the package"command": "npx","args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/serve"] **Advantages**: * No global installation needed (npx downloads on-demand) * Works across platforms (macOS, Linux, Windows) * Easy to upgrade (npx always gets latest) **Disadvantages**: * First run is slow (downloads package) * Requires Node.js installed * Internet needed for first run ### Python Servers (via uvx)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#python-servers-via-uvx "Direct link to Python Servers (via uvx)") Python-based servers (common in community): # Install and run via uvxuvx mcp-sqlite# With Python version specificationuvx --python 3.12 mcp-github# In configuration"command": "uvx","args": ["mcp-sqlite"]# Or with Python version"command": "uvx","args": ["--python", "3.12", "mcp-github"] **Advantages**: * Works with isolated Python environments (no global pollution) * Automatic dependency management * Clean uninstall (just delete directory) **Disadvantages**: * Slower than pre-installed (downloads/installs on first run) * Requires uvx (from uv package manager) ### Docker Servers (for Complex Setups)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#docker-servers-for-complex-setups "Direct link to Docker Servers (for Complex Setups)") For servers with heavy dependencies: # Build and run Docker containerdocker run -e GITHUB_TOKEN=$GITHUB_TOKEN mcp-github:latest **When to use Docker**: * Server has complex dependencies (database drivers, system libraries) * You want guaranteed environment consistency * Deploying to production (Kubernetes, etc.) Popular Servers: Configuration Examples[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#popular-servers-configuration-examples "Direct link to Popular Servers: Configuration Examples") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Let's configure the most commonly-used community servers. ### Filesystem Server (Official)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#filesystem-server-official "Direct link to Filesystem Server (Official)") Already covered in Lesson 6, but listed here for completeness: { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/projects"] } }} ### GitHub Server (Official, owned by GitHub)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#github-server-official-owned-by-github "Direct link to GitHub Server (Official, owned by GitHub)") { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } }} **What it enables**: * Create issues, pull requests * Fetch repository information * Manage workflows and deployments * Search across repositories **Setup**: 1. Create GitHub Personal Access Token (Settings → Developer Settings → Personal Access Tokens) 2. Create with scopes: `repo`, `read:org`, `workflow` 3. Store in `.env`: `GITHUB_TOKEN=ghp_xxxx` ### Brave Search Server (Community, Actively Maintained)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#brave-search-server-community-actively-maintained "Direct link to Brave Search Server (Community, Actively Maintained)") Web search without tracking: { "mcpServers": { "search": { "command": "npx", "args": ["-y", "mcp-brave-search"], "env": { "BRAVE_SEARCH_API_KEY": "${BRAVE_SEARCH_API_KEY}" } } }} **What it enables**: * Web search (news, web results) * Query summarization * Fact verification **Setup**: 1. Get API key from [api.search.brave.com](https://api.search.brave.com/) 2. Free tier available (limited requests) 3. Store in `.env`: `BRAVE_SEARCH_API_KEY=xxx` ### SQLite Server (Community, Popular)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#sqlite-server-community-popular "Direct link to SQLite Server (Community, Popular)") Local database queries: { "mcpServers": { "sqlite": { "command": "uvx", "args": ["mcp-sqlite", "/path/to/database.db"] } }} **What it enables**: * Query tables * Create/modify records * Schema inspection **Setup**: 1. Point to existing SQLite database file 2. No authentication needed (file-based) 3. Restricted to single database ### PostgreSQL Server (Community)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#postgresql-server-community "Direct link to PostgreSQL Server (Community)") Remote database operations: { "mcpServers": { "database": { "command": "uvx", "args": ["mcp-postgres"], "env": { "DATABASE_URL": "${DATABASE_URL}" } } }} **What it enables**: * Query multiple tables * Create/update/delete records * Run stored procedures **Setup**: 1. Create connection string: `postgresql://user:password@host:5432/database` 2. Store in `.env`: `DATABASE_URL=postgresql://...` 3. Ensure network access (firewall rules, VPN) ### Puppeteer Server (Community: Browser Automation)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#puppeteer-server-community-browser-automation "Direct link to Puppeteer Server (Community: Browser Automation)") Automated browser interaction: { "mcpServers": { "browser": { "command": "npx", "args": ["mcp-puppeteer"] } }} **What it enables**: * Navigate to URLs * Fill forms * Click buttons * Take screenshots * Extract content **No authentication needed** (but can slow down operations). Composing Multiple Servers: Building Complex Workflows[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#composing-multiple-servers-building-complex-workflows "Direct link to Composing Multiple Servers: Building Complex Workflows") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The power of MCP emerges when you combine servers. A single agent can work across filesystem, GitHub, databases, and web search simultaneously. ### Example: Multi-Server Development Environment[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#example-multi-server-development-environment "Direct link to Example: Multi-Server Development Environment") { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/project"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }, "sqlite": { "command": "uvx", "args": ["mcp-sqlite", "/Users/you/project/data.db"] }, "search": { "command": "npx", "args": ["-y", "mcp-brave-search"], "env": { "BRAVE_SEARCH_API_KEY": "${BRAVE_SEARCH_API_KEY}" } } }} **What this enables**: An agent using this configuration can: 1. **Read project files** (filesystem server) → Understand codebase structure 2. **Check GitHub issues** (GitHub server) → See what needs fixing 3. **Query local database** (SQLite server) → Understand data schema 4. **Search the web** (Brave server) → Find API documentation or examples 5. **Execute multi-step workflows**: * "Read the TODO.md file, search for examples of how others solved this, create a GitHub issue with the findings, update the database schema" ### Tool Routing: How Agents Know Which Server to Use[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#tool-routing-how-agents-know-which-server-to-use "Direct link to Tool Routing: How Agents Know Which Server to Use") When you ask an agent to perform a task, the client automatically routes it to the right server: Agent: "Search for OAuth 2.0 examples in Node.js"↓Client checks all servers for search-related tools↓Brave Search server has `search()` tool↓Client calls Brave server with query↓Returns resultsAgent: "Create an issue in the GitHub repo"↓Client checks all servers for GitHub-related tools↓GitHub server has `create_issue()` tool↓Client calls GitHub server with issue data↓Returns issue URL Each server exposes its own tools. The agent sees ALL tools from ALL connected servers. This creates a powerful unified interface. ### Workflow Example: Bug Report → GitHub Issue → Database Update[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#workflow-example-bug-report--github-issue--database-update "Direct link to Workflow Example: Bug Report → GitHub Issue → Database Update") Here's what a multi-server workflow looks like: **Agent request**: "I found a bug in the user authentication. Help me:1. Search for similar issues already reported2. Create a new GitHub issue with detailed description3. Update the bugs table in our database with this issue4. Draft a fix proposal based on code review" **Workflow execution**: 1. **Search** (Brave Search server) → Find similar issues + relevant code patterns 2. **Create Issue** (GitHub server) → Open issue with title, description, labels 3. **Update Database** (SQLite server) → Add bug report to tracking table 4. **Code Review** (Filesystem server) → Read relevant files to understand auth logic 5. **Synthesis** → Agent combines all information to propose fix None of this requires explicit server selection. The agent requests capabilities (search, create issue, database write, file read) and the client routes to the right servers. Security Considerations for Community Servers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#security-considerations-for-community-servers "Direct link to Security Considerations for Community Servers") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Community servers require more caution than official servers. ### Risk Assessment Framework[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#risk-assessment-framework "Direct link to Risk Assessment Framework") **Before installing any community server, ask**: 1. **What does it do?** (Read code or documentation) 2. **What could go wrong?** (Worst-case execution) 3. **How is it sandboxed?** (What limits prevent damage?) **Examples**: Scenario 1: Community Filesystem Server (Hypothetical)- What: Reads/writes files to specific directory- Could go wrong: Server exploit → attacker writes malicious code- Sandbox: Restricted to single directory (limited scope)- Risk: MEDIUM (containable)- Mitigation: Use official filesystem server instead (better security review)Scenario 2: Database Server- What: Connects to PostgreSQL with stored credentials- Could go wrong: Leaked database URL → attacker accesses all data- Sandbox: Database user permissions (e.g., read-only)- Risk: HIGH (contains sensitive data)- Mitigation: Use database user with minimal privileges; rotate credentials regularlyScenario 3: Web Search Server- What: Makes HTTP requests to search API- Could go wrong: None (read-only operation, no state change)- Sandbox: Search server can only return results (no side effects)- Risk: LOW (benign operation)- Mitigation: Verify API key not exposed in logs ### Security Checklist for Community Servers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#security-checklist-for-community-servers "Direct link to Security Checklist for Community Servers") * [ ] Is this server still maintained? (Check last commit date) * [ ] Does it have security vulnerabilities? (Check GitHub issues and releases) * [ ] What permissions does it need? (API keys, file access, database credentials) * [ ] Can it expose secrets? (Check if it logs credentials or returns them) * [ ] Have you code-reviewed it? (Read the source for obvious issues) * [ ] Is there a less-risky alternative? (Official vs community, simpler vs complex) * [ ] Can you sandbox it? (Restricted permissions, isolated environment) Discovering New Servers and Staying Updated[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#discovering-new-servers-and-staying-updated "Direct link to Discovering New Servers and Staying Updated") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The MCP ecosystem evolves constantly. How do you stay informed? ### Official MCP Registry[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#official-mcp-registry "Direct link to Official MCP Registry") **Best source**: [registry.modelcontextprotocol.io](https://registry.modelcontextprotocol.io/) The registry launched in preview (September 8, 2025) and entered API freeze v0.1 (October 24, 2025). It's the canonical source for: * Server metadata (author, version, capabilities) * Community ratings and reviews * Installation instructions * Links to repositories ### Community Curated Lists[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#community-curated-lists "Direct link to Community Curated Lists") Popular GitHub repositories maintain curated lists: * [wong2/awesome-mcp-servers](https://github.com/wong2/awesome-mcp-servers) * [punkpeye/awesome-mcp-servers](https://github.com/punkpeye/awesome-mcp-servers) These lists are community-maintained (not official) but useful for discovery and recommendations. ### Following Updates[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#following-updates "Direct link to Following Updates") Once you identify servers you use: * **Watch the repository** on GitHub (get notifications for releases) * **Subscribe to release notes** (many servers publish security advisories) * **Monitor CVE databases** (for security vulnerabilities in dependencies) Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion to explore the MCP server ecosystem, evaluate options, and design multi-server configurations. ### Security Note on Community Servers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#security-note-on-community-servers "Direct link to Security Note on Community Servers") When installing and configuring community MCP servers, remember: * **Official servers are vetted**: Use them by default (Filesystem, GitHub, Fetch, Memory) * **Community servers require judgment**: Check maintenance status, code review, and security implications before deploying * **Credentials are high-risk**: Community server vulnerability → leaked API keys → compromised accounts * **Sandbox principles apply**: Give servers only the permissions they need (read-only when possible, restricted database users, limited file access) * **Monitor actively**: Set up alerts for security issues in servers you depend on; rotate credentials regularly if using community servers The convenience of community servers must be balanced against security responsibility. Start with official servers. Only add community servers after evaluating alternatives and understanding risks. ### Prompt 1: Discover Servers for Your Use Case[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-1-discover-servers-for-your-use-case "Direct link to Prompt 1: Discover Servers for Your Use Case") **Setup**: Finding the right server for your project needs I'm working on a project that needs to:1. Read and modify code files in /my/project2. Interact with GitHub (create issues, comment on PRs)3. Query a local SQLite database of project metadataHelp me:1. Identify which MCP servers would be useful2. For each server, explain what it does and what I need to set it up3. List any authentication/credentials I need to create4. Explain how these servers work together in a single configuration **What you're learning**: Server discovery and selection based on project requirements. ### Prompt 2: Evaluate Server Quality[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-2-evaluate-server-quality "Direct link to Prompt 2: Evaluate Server Quality") **Setup**: Comparing options for the same task I need database access for my AI agent. I found three servers:1. Official SQLite server from modelcontextprotocol/servers2. Community PostgreSQL server (last updated 2 weeks ago, 50 GitHub stars)3. Another community PostgreSQL server (last updated 1 year ago, 200 stars)Help me evaluate:1. What are the trust/maintenance/security differences?2. What questions would you ask about each before using it in production?3. Which would you recommend and why?4. What are the risks of picking the "wrong" one? **What you're learning**: Evaluating server quality and making risk-based decisions. ### Prompt 3: Configure Multiple Servers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-3-configure-multiple-servers "Direct link to Prompt 3: Configure Multiple Servers") **Setup**: Building a multi-server configuration I want to set up a development environment with these servers:- Filesystem (to read project files)- GitHub (to check issues and create PRs)- Brave Search (to find examples)- SQLite (for project database)Help me:1. Write the complete JSON configuration (.claude/settings.json)2. List all the environment variables I need and where to get them3. Show me how to create a .env file with these secrets4. Explain how to test each server to verify it's working5. What should I NEVER commit to git? Why? **What you're learning**: Multi-server configuration, authentication management, and security best practices. ### Prompt 4: Design a Complex Workflow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-4-design-a-complex-workflow "Direct link to Prompt 4: Design a Complex Workflow") **Setup**: Using multiple servers together I have a feature request that requires:1. Reading code from my project (filesystem)2. Searching for similar implementations online (web search)3. Creating a GitHub issue with the requirements4. Adding task records to my SQLite database5. Drafting implementation codeFor this workflow:1. Which servers would you use? Why?2. In what order would you invoke them?3. What could go wrong? How would you handle failures?4. How would you test this end-to-end? **What you're learning**: Multi-server workflow design and failure mode analysis. Checking access... * [The MCP Ecosystem: Official, Community, and Enterprise](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#the-mcp-ecosystem-official-community-and-enterprise) * [Official Reference Servers (Maintained by Anthropic / The Linux Foundation)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#official-reference-servers-maintained-by-anthropic--the-linux-foundation) * [Community Servers (Created and Maintained by Developers)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#community-servers-created-and-maintained-by-developers) * [Enterprise Servers (Built by Companies for Their Platforms)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#enterprise-servers-built-by-companies-for-their-platforms) * [Evaluating Server Quality and Trust](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#evaluating-server-quality-and-trust) * [Evaluation Checklist](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#evaluation-checklist) * [Installation: npm vs uvx vs Docker](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#installation-npm-vs-uvx-vs-docker) * [Node.js Servers (via npx)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#nodejs-servers-via-npx) * [Python Servers (via uvx)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#python-servers-via-uvx) * [Docker Servers (for Complex Setups)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#docker-servers-for-complex-setups) * [Popular Servers: Configuration Examples](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#popular-servers-configuration-examples) * [Filesystem Server (Official)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#filesystem-server-official) * [GitHub Server (Official, owned by GitHub)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#github-server-official-owned-by-github) * [Brave Search Server (Community, Actively Maintained)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#brave-search-server-community-actively-maintained) * [SQLite Server (Community, Popular)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#sqlite-server-community-popular) * [PostgreSQL Server (Community)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#postgresql-server-community) * [Puppeteer Server (Community: Browser Automation)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#puppeteer-server-community-browser-automation) * [Composing Multiple Servers: Building Complex Workflows](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#composing-multiple-servers-building-complex-workflows) * [Example: Multi-Server Development Environment](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#example-multi-server-development-environment) * [Tool Routing: How Agents Know Which Server to Use](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#tool-routing-how-agents-know-which-server-to-use) * [Workflow Example: Bug Report → GitHub Issue → Database Update](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#workflow-example-bug-report--github-issue--database-update) * [Security Considerations for Community Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#security-considerations-for-community-servers) * [Risk Assessment Framework](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#risk-assessment-framework) * [Security Checklist for Community Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#security-checklist-for-community-servers) * [Discovering New Servers and Staying Updated](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#discovering-new-servers-and-staying-updated) * [Official MCP Registry](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#official-mcp-registry) * [Community Curated Lists](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#community-curated-lists) * [Following Updates](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#following-updates) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#try-with-ai) * [Security Note on Community Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#security-note-on-community-servers) * [Prompt 1: Discover Servers for Your Use Case](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-1-discover-servers-for-your-use-case) * [Prompt 2: Evaluate Server Quality](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-2-evaluate-server-quality) * [Prompt 3: Configure Multiple Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-3-configure-multiple-servers) * [Prompt 4: Design a Complex Workflow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/using-community-mcp-servers#prompt-4-design-a-complex-workflow) --- # Capstone: Shippable Agent Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've now learned every pattern this chapter teaches: persona design (Lesson 1), skill composition (Lesson 2), MCP-wrapping (Lessons 3-4), script execution (Lessons 5-6), and workflow orchestration (Lesson 7). But learning patterns is not the same as building products. This capstone is different. You're not following steps or solving a predetermined problem. You're doing what every AI-native entrepreneur must do: **articulate a problem clearly enough that AI can build a solution to it, then validate that solution works in the real world.** You're building a **specification-first Digital FTE**—a complete, shippable execution skill that solves a real customer problem. This skill will orchestrate the accumulated intelligence from all previous lessons, handle production edge cases, and be ready to monetize. Why This Capstone Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#why-this-capstone-matters "Direct link to Why This Capstone Matters") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In the agent economy, the developers who win are those who can: 1. **Identify customer problems** that are specific enough to solve but broad enough to charge for 2. **Write specifications clearly** so AI can implement without ambiguity 3. **Compose existing intelligence** rather than reinventing patterns 4. **Validate ruthlessly** that the product works before claiming it's production-ready 5. **Price strategically** to capture value without leaving money on the table This capstone exercises all five. You're not learning to code faster. You're learning to build products that customers pay for. Phase 1: Domain Specification (Spec FIRST)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-1-domain-specification-spec-first "Direct link to Phase 1: Domain Specification (Spec FIRST)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before any implementation, define what problem you're solving for whom. ### Select Your Domain[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#select-your-domain "Direct link to Select Your Domain") Choose a domain where you have expertise or strong interest. This skill should solve a real problem you've experienced or observed: **Domain Categories**: * **Professional Services**: Legal document review, financial analysis, code review, technical writing * **Content Operations**: Social media scheduling, newsletter writing, content moderation, SEO optimization * **Data Processing**: Sales pipeline analysis, customer segmentation, report generation, data validation * **System Operations**: Log analysis, configuration validation, performance monitoring, security compliance * **Creative**: Image description, metadata generation, caption writing, style adaptation **Real Examples**: * **Legal Brief Analyzer**: Ingest case documents, extract relevant precedents, summarize arguments, flag contradictions * **Sales Data Processor**: Import sales data, identify anomalies, segment customers, forecast pipeline * **Code Review Enforcer**: Check PRs against team standards, catch security issues, suggest improvements * **Content Quality Inspector**: Validate blog posts for SEO, readability, brand consistency, factual accuracy Pick something specific. "Data processing tool" is too vague. "CSV sales data analyzer that identifies top performers, flags stagnant accounts, and generates forecasts" is right-sized. ### Write Your Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#write-your-specification "Direct link to Write Your Specification") Create a `skill-spec.md` that answers these questions: #### Intent (The Customer Problem)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#intent-the-customer-problem "Direct link to Intent (The Customer Problem)") ## Intent**What customer problem does this skill solve?**A [company type] needs to [specific business outcome] but currently [current friction]. This skill [specific solution] enabling them to [measurable business result].**Example:**Sales teams need to identify which accounts are growing vs stagnating to prioritize outreach. Currently, managers spend 4-6 hours weekly manually reviewing spreadsheets. This skill analyzes customer transaction history, flags trends, and generates weekly reports in 5 minutes, freeing managers for strategy.**Why customers would buy:**- [Time saved per week/month]- [Cost reduction: (current cost) → (new cost)]- [Risk reduced: (current failure rate) → (new failure rate)]- [Revenue impact: (new revenue opportunity OR cost avoidance)] #### Success Criteria (How You'll Know It Works)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#success-criteria-how-youll-know-it-works "Direct link to Success Criteria (How You'll Know It Works)") Define 5-7 measurable success criteria that a customer would use to evaluate if the skill works: ## Success Criteria- Accuracy: [specific metric] achieved on test data (e.g., "Correctly identifies 95%+ of anomalies in clean data")- Speed: [completes in X time] (e.g., "Processes 1000 records in under 30 seconds")- Reliability: [uptime or error rate] (e.g., "Handles 99%+ of real-world data formats without crashes")- Completeness: [coverage metric] (e.g., "Generates all required report fields with no missing values")- Safety: [risk mitigation] (e.g., "Flags confidence levels <80% for manual review before acting")- Integration: [system compatibility] (e.g., "Imports from Salesforce, HubSpot, and CSV formats")- Learning: [improvement mechanism] (e.g., "Generates error logs enabling continuous pattern improvement") #### Constraints (What's NOT Included)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#constraints-whats-not-included "Direct link to Constraints (What's NOT Included)") Define explicit boundaries—what this skill does NOT do: ## Constraints (Non-Goals)- Does NOT: Real-time decision-making (only batch processing)- Does NOT: Modify source data (analysis only, read-only)- Does NOT: Handle unstructured text (CSV/database inputs only)- Does NOT: Provide legal interpretation (flags issues, humans decide)Why: [reasoning for each boundary] #### Acceptance Tests (Proof of Success)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#acceptance-tests-proof-of-success "Direct link to Acceptance Tests (Proof of Success)") Write concrete test cases proving each success criterion: ## Acceptance Tests### Test 1: Accuracy on Clean DataInput: [clean_data.csv with known patterns]Expected: [specific output meeting criterion]Pass/Fail Criteria: [measurable validation rule]### Test 2: Robustness to Missing ValuesInput: [data_with_nulls.csv]Expected: [skill either fills intelligently or flags for review]Pass/Fail Criteria: [specific error handling rule]### Test 3: Performance at ScaleInput: [1000_records.csv]Expected: [completes in <30 seconds]Pass/Fail Criteria: [timing measurement]### Test 4: Integration with MCPInput: [real-world data from Salesforce MCP]Expected: [processes without format errors]Pass/Fail Criteria: [data shape validation]### Test 5: Error RecoveryInput: [malformed_data.csv]Expected: [skill detects issue, suggests fix, asks for confirmation]Pass/Fail Criteria: [user can recovery without restarting] #### Architecture (Which Skills Compose)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#architecture-which-skills-compose "Direct link to Architecture (Which Skills Compose)") Map which skills from Lessons 1-7 you'll use: ## Architecture### Component 1: Data Ingestion (Lesson 4 - MCP Wrapping)- Which MCP? [e.g., "Salesforce connector MCP from Chapter 66"]- Skill wrapping it: [e.g., "sales-data-fetcher skill"]- What it does: Fetches data, validates format, transforms to internal schema### Component 2: Data Analysis (Lesson 6 - Script Execution)- Script type: [Python, Bash, SQL]- Skill orchestrating it: [e.g., "anomaly-detector skill"]- What it does: Runs analysis script, catches errors, retries on failure### Component 3: Workflow Coordination (Lesson 7 - Orchestration)- Master skill: [e.g., "sales-analyzer orchestrator"]- Coordination: Calls Component 1 → Component 2 → generates report- Error recovery: If Component 2 fails, Component 3 retries with constraints### Data Flow[Diagram showing: MCP Input → Component 1 → Component 2 → Component 3 → Output] Phase 2: Skill Composition[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-2-skill-composition "Direct link to Phase 2: Skill Composition") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With specification complete, design how components integrate. ### Map Component Dependencies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#map-component-dependencies "Direct link to Map Component Dependencies") Which skills call which? orchestrator-skill (master coordinator)├─ mcp-wrapping-skill (data source)│ └─ Context7 MCP (external data)├─ script-execution-skill (analysis)│ └─ Python code generation└─ validation-skill (error checking) └─ Test data against spec ### Design Data Contracts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#design-data-contracts "Direct link to Design Data Contracts") How does each skill accept/produce data? ## Data Contracts### MCP-Wrapping Skill Input/OutputInput: {source: "salesforce", filters: {...}}Output: {records: [...], schema: {...}, validation_passed: bool}### Script-Execution Skill Input/OutputInput: {data: [...], analysis_type: "anomaly_detection", params: {...}}Output: {results: [...], errors: [...], iterations: n}### Orchestrator Skill Input/OutputInput: {customer_id: "...", report_type: "weekly", confidence_threshold: 0.8}Output: {report: {...}, success: bool, issues_requiring_human_review: [...]} ### Define Error Recovery Paths[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#define-error-recovery-paths "Direct link to Define Error Recovery Paths") What happens when each component fails? ## Error Recovery Strategy### If MCP Fails (Network timeout, auth issue)→ Retry 3x with exponential backoff→ If still failing, return cached data (if available) with staleness warning→ If no cache, escalate to human with actionable error### If Script Execution Fails (Bad data format)→ Log specific error message→ Generate corrected input (remove nulls, handle encoding)→ Retry script execution with modified input→ If still failing after 3 iterations, flag for manual review### If Orchestrator Detects Inconsistency→ Example: Analysis claims "high growth" but raw data shows decline→ Generate diagnostic to investigate root cause→ Present both interpretations to user→ Ask user to validate which interpretation is correct Phase 3: Specification → Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-3-specification--implementation "Direct link to Phase 3: Specification → Implementation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now you have a clear spec. Time to build. ### Create the Skill Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#create-the-skill-implementation "Direct link to Create the Skill Implementation") Work with AI to implement your SKILL.md file that orchestrates all components: ---name: "your-domain-skill"version: "1.0.0"description: "[From your spec intent]"proficiency_level: "B2"category: "Applied"---# PersonaYou are a [domain] execution orchestrator. Your job is to:1. Accept customer specifications from intent above2. Invoke appropriate data sources (MCP-wrapping skill)3. Execute analysis (script-execution skill)4. Validate results against success criteria5. Generate human-readable output6. Flag confidence issues for human review7. Iterate if partial success until spec satisfaction... ### Test Each Component[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#test-each-component "Direct link to Test Each Component") Before testing the full orchestration, validate each component works: 1. **MCP Component**: Does it fetch data correctly? 2. **Script Component**: Does it analyze data without errors? 3. **Validation Component**: Does it catch actual problems? 4. **Integration**: When combined, does data flow correctly between components? ### Document Implementation Decisions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#document-implementation-decisions "Direct link to Document Implementation Decisions") As you build, document why you made each architectural choice: ## Implementation Notes### Why we chose [MCP X] over [MCP Y]- Comparison: [criteria]- Decision: X provides [advantage] that Y lacks### Why error recovery retries 3x maximum- Risk: Infinite loops would timeout- Benefit: 3x usually sufficient for transient failures- Tradeoff: Some failures might need manual intervention### Why we validate output against original spec- Purpose: Ensure analysis actually answers customer question- Example: If spec asked for "anomalies" and we found "seasonal patterns", both valid but different Phase 4: Acceptance Testing & Validation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-4-acceptance-testing--validation "Direct link to Phase 4: Acceptance Testing & Validation") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Production readiness means: **Every acceptance test passes**. No exceptions. ### Run Full Acceptance Test Suite[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#run-full-acceptance-test-suite "Direct link to Run Full Acceptance Test Suite") Execute each test case from your spec: ## Test Results### Test 1: Accuracy on Clean Data- Input: clean_data.csv (100 records)- Expected: 95%+ anomaly detection accuracy- Actual: ✓ PASS (97.2% accuracy)### Test 2: Robustness to Missing Values- Input: data_with_nulls.csv (50 records, 15% nulls)- Expected: Handles nulls intelligently- Actual: ✓ PASS (Imputes using mean, documents assumptions)### Test 3: Performance at Scale- Input: 1000_records.csv- Expected: <30 seconds- Actual: ✓ PASS (24.3 seconds)### Test 4: Integration with MCP- Input: Real Salesforce data- Expected: Processes without format errors- Actual: ✓ PASS (5 test runs, 0 format failures)### Test 5: Error Recovery- Input: malformed_data.csv (bad encoding)- Expected: Detects error, asks user- Actual: ✓ PASS (Error caught, user prompted, recovery successful) ### Document Any Spec Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#document-any-spec-gaps "Direct link to Document Any Spec Gaps") If testing reveals issues, update specification: ## Discovered During Validation### Issue 1: Specification was unclear about timezone handling- Original: "Process timestamps from multiple regions"- Realized: Didn't specify how to normalize timezones- Updated Spec: "Normalize all timestamps to UTC before analysis, document source timezone in output"- Test Added: Verify timezone conversion accuracy### Issue 2: Edge case not covered: Empty dataset- Original spec didn't mention: "What if data has 0 records?"- Solution: Return empty results with explanatory message- Updated spec: "If input has <1 record, return {success: true, records_processed: 0, message: '...'}" Phase 5: Production Packaging[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-5-production-packaging "Direct link to Phase 5: Production Packaging") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Now package the skill for customers to use. ### Create Customer Documentation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#create-customer-documentation "Direct link to Create Customer Documentation") Write a simple guide customers will use: # [Your Skill] User Guide## What This Does[2-3 sentence summary of business value]## What You Need- Input data format: [CSV / database connection / API]- Required fields: [column names or schema]- Recommended data size: [e.g., "Works best with 100-10,000 records"]## How to Use It1. [Step 1]2. [Step 2]3. [Step 3]## Understanding Results[Explain key fields in output]## Troubleshooting**Problem**: [Common failure scenario]**Solution**: [How to fix]## SupportContact: [your email]Response time: [e.g., "24 hours"] ### Version Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#version-your-skill "Direct link to Version Your Skill") Use semantic versioning: version: "1.0.0" # Major.Minor.Patch# Major: Breaking changes to input/output format# Minor: New features that are backward compatible# Patch: Bug fixes ### Create Installation Instructions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#create-installation-instructions "Direct link to Create Installation Instructions") How does a customer deploy this skill? ## Installation### Option 1: Claude Code (Recommended)1. Save this SKILL.md file to ~/.claude/skills/your-domain-skill/SKILL.md2. In Claude Code, type: /skill-list to see your new skill3. Activate it: /skill-activate your-domain-skill4. Test it: Ask Claude "Help me analyze my data with your-domain-skill"### Option 2: Custom Integration[For customers integrating with their own LLM provider] Phase 6: Digital FTE Positioning[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-6-digital-fte-positioning "Direct link to Phase 6: Digital FTE Positioning") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Finally, position this skill as a product. ### Identify Your Customer[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#identify-your-customer "Direct link to Identify Your Customer") Who would pay for this? ## Customer Profile### Primary Segment- Title: [e.g., "Sales Manager at B2B SaaS companies"]- Pain point: [specific problem they face]- Current solution: [what they do today, poorly]- Budget: [How much would they pay monthly/annually?]### Secondary Segments[Other customer types who might benefit] ### Articulate Business Value[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#articulate-business-value "Direct link to Articulate Business Value") Why should customers choose your skill over alternatives? ## Value Proposition**Problem**: [Customer's current pain]**Our solution**: [What this skill does]**Outcome**: [Specific business result]**Differentiation**: [Why ours is better than]- Alternative 1: [comparison on key dimension]- Alternative 2: [comparison on key dimension]### ROI Calculation- Time saved per week: [X hours]- Cost per hour: [$Y]- Monthly savings: [X × Y × 4.33 weeks = $Z]- Skill cost: [$P/month]- Payback period: [Z/P months] ### Select Monetization Model[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#select-monetization-model "Direct link to Select Monetization Model") Choose how you'll charge: ## Monetization Model### Option 1: Subscription- Price: $[X]/month or $[Y]/year- Users included: [1 / unlimited]- Data volume: [up to X records/month]- Pros: Predictable revenue, customer lock-in- Cons: Requires ongoing support### Option 2: Success Fee- Price: [X%] of business value generated- Example: "2% of monthly savings" or "5% of revenue increase"- Pros: Aligned incentives, higher ceiling- Cons: Requires tracking customer results### Option 3: License- Price: $[X] one-time fee for unlimited use- Pros: Simple, customer owns it- Cons: No recurring revenue### Recommendation[Which model fits this skill best and why?] ### Design Go-to-Market[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#design-go-to-market "Direct link to Design Go-to-Market") How will customers find and adopt your skill? ## Go-to-Market Strategy### Positioning"[Your skill] is the [category] for [audience] that [specific benefit]."Example: "sales-analyzer is the revenue intelligence tool for B2B SaaS sales leaders that identifies growth opportunities in 5 minutes instead of 5 hours."### Customer Acquisition1. [Channel 1]: [Tactic] - Target: [Who you'll reach] - Cost: [acquisition cost] - Conversion: [estimated %]2. [Channel 2]: [Tactic]### Success Metrics- Month 1: [X customers]- Month 3: [X customers]- Month 6: [X customers]- Target: [$Y MRR by month 12] Try With AI: Build Your Capstone Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#try-with-ai-build-your-capstone-skill "Direct link to Try With AI: Build Your Capstone Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is where you move from learning patterns to building products. ### Prompt 1: Specification Refinement[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-1-specification-refinement "Direct link to Prompt 1: Specification Refinement") Ask AI to review your domain choice and help sharpen your specification: I'm building a Digital FTE in [domain] that solves [customer problem].My current specification:- Intent: [your intent statement]- Success criteria: [your success criteria]- Non-goals: [your constraints]Review my specification. Is it:1. Specific enough that you could implement without asking clarifying questions?2. Focused enough (not trying to solve everything)?3. Measurable (could you write a test that proves it works)?If any answer is "no", what's missing from my specification? **What you're learning**: How clear specifications prevent implementation rework. Vague specs require clarification loops; clear specs compile cleanly. ### Prompt 2: Architecture Design[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-2-architecture-design "Direct link to Prompt 2: Architecture Design") Have AI help you design which skills compose together: I need to compose skills from Chapter 68 into an orchestrator that implements my specification above.The skills I have available:- Lesson 1: [Advanced Skill Patterns]- Lesson 2: [Skill Composition patterns]- Lesson 3-4: [MCP-wrapping skill for my data source]- Lesson 5-6: [Script-execution skill for my analysis]- Lesson 7: [Orchestration skill combining them]Design an architecture showing:1. Which skills I should compose2. Data flow between them3. Error recovery if any component fails4. How each component validates its output before passing to nextDraw a diagram using ASCII or describe in text. **What you're learning**: Composition thinking—reusing tested components rather than building from scratch. This is Digital FTE production practice. ### Prompt 3: Implementation & Testing[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-3-implementation--testing "Direct link to Prompt 3: Implementation & Testing") Ask AI to help implement while you validate each decision: Implement my orchestrator skill (SKILL.md) that composes the architecture above.As you implement, explain:1. Why you chose this persona for the orchestrator2. What decision questions activate autonomous behavior3. How error recovery paths will work in practice4. Which parts need explicit safety constraintsThen, let's test it against my acceptance criteria:[Paste your 5-7 success criteria]Create a test plan showing which tests validate which criteria. **What you're learning**: How specifications drive implementation. AI uses your spec to make decisions without ambiguity. If it needs to guess, your spec was incomplete. ### Prompt 4: Validation & Gap Analysis[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-4-validation--gap-analysis "Direct link to Prompt 4: Validation & Gap Analysis") Run your tests and document what happens: I ran my skill against my test data with these results:[Paste your test results]Issues discovered:1. [Issue 1 and what caused it]2. [Issue 2 and what caused it]For each issue, should I:- Fix the skill implementation?- Update the specification to be more accurate?- Both?Help me update either the spec or the implementation to make all tests pass. **What you're learning**: The specification ↔ implementation feedback loop. Tests reveal gaps in both. Fixing only the code misses the root cause (underspecified requirements). ### Prompt 5: Business Positioning[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-5-business-positioning "Direct link to Prompt 5: Business Positioning") Finally, articulate why someone would pay for this: I've built a working Digital FTE that [what it does]. Now help me position it as a product:1. What customer segment has the most acute version of this problem?2. What's the annual cost of them NOT having this solution? (time wasted, missed revenue, etc.)3. What would be a fair price that captures 20-30% of that value?4. What should my go-to-market strategy focus on (direct sales, self-serve, marketplace)?5. What's one surprising use case for this skill that customers might not think of?Use this to help me write a positioning statement and pricing proposal. **What you're learning**: Digital FTEs aren't just technical products—they're customer solutions. This prompt teaches the business side of the agent economy. ### Success Criteria (Validation Checklist)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#success-criteria-validation-checklist "Direct link to Success Criteria (Validation Checklist)") Your capstone is complete when: **Specification Phase:** * [ ] Spec clearly describes a customer problem, not a technology exercise * [ ] Success criteria are measurable (could write automated tests) * [ ] Constraints explicitly define what's NOT included * [ ] Architecture diagram shows 3+ composed skills from Lessons 1-7 **Implementation Phase:** * [ ] Skill implementation based on spec (not invented during coding) * [ ] Each component (MCP, script, orchestration) tested independently * [ ] Implementation decisions documented with reasoning * [ ] Skill version defined using semantic versioning **Validation Phase:** * [ ] All acceptance tests designed before implementation * [ ] All acceptance tests run and documented (pass/fail) * [ ] Any spec gaps discovered during testing are documented and remediated * [ ] Results prove skill meets original specification **Production Phase:** * [ ] Customer-facing documentation complete (not internal jargon) * [ ] Installation instructions for both Claude Code and custom integrations * [ ] Troubleshooting guide addresses common failure modes * [ ] Version control / release notes prepared **Business Phase:** * [ ] Customer profile clearly defined (title, problem, budget) * [ ] ROI calculation shows when customers break even on investment * [ ] Monetization model selected with reasoning (subscription/success-fee/license) * [ ] Go-to-market positioning statement (single sentence describing what/for-whom/why) ### Reference: Capstone Skill Template[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#reference-capstone-skill-template "Direct link to Reference: Capstone Skill Template") If you get stuck, use this template: ---name: "my-digital-fte-skill"version: "1.0.0"description: "Production-ready skill orchestrating code execution pattern"proficiency_level: "B2"---# Specification## Intent[2-3 sentences describing customer problem and solution]## Success Criteria1. [Measurable criterion 1]2. [Measurable criterion 2]3. [Measurable criterion 3]## Non-Goals- Does NOT: [explicit boundary 1]- Does NOT: [explicit boundary 2]## Architecture[Component 1] → [Component 2] → [Component 3] → Output# PersonaYou are a [domain] orchestrator that:1. [Action 1]2. [Action 2]3. [Action 3]# Questions- [Question 1: What analysis is needed?]- [Question 2: What data validates success?]- [Question 3: What errors require human intervention?]# Principles- [Principle 1: Safety/Reliability]- [Principle 2: Completeness]# Implementation[Your skill implementation details]# Business Positioning- **Customer**: [Who pays]- **Problem**: [What pain]- **Value**: [Specific outcome] * * * ### What This Capstone Teaches[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#what-this-capstone-teaches "Direct link to What This Capstone Teaches") This capstone represents mastery of one critical truth: **In the agent economy, your value is not your code. Your value is your specification and your composition skills.** You've now learned that the developers who build billion-dollar AI businesses are exactly those who: 1. **Write clear specifications** that prevent ambiguity 2. **Compose reusable intelligence** rather than reinventing patterns 3. **Validate ruthlessly** through acceptance testing 4. **Position strategically** as products with customer value You just completed all four. This skill is now deployable—it could be sold as-is through subscription, license, or success-fee model. Scale this process across multiple skills, and you're building the Digital FTE empire that will define the next decade of software. Checking access... * [Why This Capstone Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#why-this-capstone-matters) * [Phase 1: Domain Specification (Spec FIRST)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-1-domain-specification-spec-first) * [Select Your Domain](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#select-your-domain) * [Write Your Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#write-your-specification) * [Phase 2: Skill Composition](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-2-skill-composition) * [Map Component Dependencies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#map-component-dependencies) * [Design Data Contracts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#design-data-contracts) * [Define Error Recovery Paths](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#define-error-recovery-paths) * [Phase 3: Specification → Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-3-specification--implementation) * [Create the Skill Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#create-the-skill-implementation) * [Test Each Component](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#test-each-component) * [Document Implementation Decisions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#document-implementation-decisions) * [Phase 4: Acceptance Testing & Validation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-4-acceptance-testing--validation) * [Run Full Acceptance Test Suite](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#run-full-acceptance-test-suite) * [Document Any Spec Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#document-any-spec-gaps) * [Phase 5: Production Packaging](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-5-production-packaging) * [Create Customer Documentation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#create-customer-documentation) * [Version Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#version-your-skill) * [Create Installation Instructions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#create-installation-instructions) * [Phase 6: Digital FTE Positioning](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#phase-6-digital-fte-positioning) * [Identify Your Customer](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#identify-your-customer) * [Articulate Business Value](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#articulate-business-value) * [Select Monetization Model](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#select-monetization-model) * [Design Go-to-Market](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#design-go-to-market) * [Try With AI: Build Your Capstone Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#try-with-ai-build-your-capstone-skill) * [Prompt 1: Specification Refinement](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-1-specification-refinement) * [Prompt 2: Architecture Design](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-2-architecture-design) * [Prompt 3: Implementation & Testing](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-3-implementation--testing) * [Prompt 4: Validation & Gap Analysis](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-4-validation--gap-analysis) * [Prompt 5: Business Positioning](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#prompt-5-business-positioning) * [Success Criteria (Validation Checklist)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#success-criteria-validation-checklist) * [Reference: Capstone Skill Template](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#reference-capstone-skill-template) * [What This Capstone Teaches](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/capstone-shippable-skill#what-this-capstone-teaches) --- # Build Script-Execution Skill | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've learned the pattern (Lesson 5): write code from specification → execute it → analyze errors → iterate. Now you're going to build a skill that orchestrates this loop autonomously. But here's what makes this different from following a tutorial: You'll specify what problem you're solving FIRST, then let AI help you build the skill while you validate each decision. You're not just learning a pattern—you're learning to think about error recovery, convergence criteria, and edge cases the way production systems demand. Step 1: Write Your Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-1-write-your-specification "Direct link to Step 1: Write Your Specification") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before touching any skill code, write a specification for the problem you're solving. You'll use a CSV data processing task because it's concrete and has natural edge cases. Choose one of these: * **CSV Analysis**: Analyze customer or sales data for patterns * **CSV Transformation**: Clean and restructure messy CSV data * **CSV Aggregation**: Group data by dimensions and calculate metrics Or define your own data processing task. ### Your Specification[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#your-specification "Direct link to Your Specification") Write this to a file (skill-spec.md) or document: # CSV Analysis Skill Specification## Intent[What does this skill do? Be specific about the business problem it solves]## Input- data_file: [type and format, e.g., "CSV with columns: customer_id, purchase_date, amount"]- parameters: [what configuration does skill accept?]## Output- format: [JSON, CSV, report?]- required_fields: [exact fields that must be in output]- validation_rules: [how to verify output is correct]## Success Criteria- All data processed without loss- Output format exactly matches specification- Edge cases handled gracefully (malformed rows, missing values, etc.)- Execution completes within 30 seconds## Edge Cases to Handle- [Case 1: e.g., "Empty CSV file"]- [Case 2: e.g., "Missing column header"]- [Case 3: e.g., "Non-numeric values in amount field"] **Key principle**: Your specification must be complete enough that AI can generate correct code without additional context. If your spec is vague, the generated code will be equally vague. Step 2: Design Your Skill's Persona and Questions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-2-design-your-skills-persona-and-questions "Direct link to Step 2: Design Your Skill's Persona and Questions") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before building, define how your skill thinks about this problem. # Skill Personapersona: | You are a data orchestrator: your role is to write Python scripts that process data robustly. When you encounter errors, you read the error message carefully, understand why the code failed, and generate corrected code. You validate results against the specification. You handle edge cases explicitly rather than hoping they don't occur.questions: - "What does the data structure look like? (columns, data types, edge cases)" - "What transformation or analysis does the specification require?" - "What output format must the code produce?" - "What validation proves the output is correct?" - "What edge cases are most likely to occur in real data?"principles: - "Validate data before processing: Check columns exist, types are correct" - "Fail explicitly: Raise errors with clear messages rather than silently producing wrong results" - "Test assumptions: Don't assume column names; inspect actual data first" - "Document the transformation: Add comments explaining the logic" Step 3: Build the Skill Core with AI Collaboration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-3-build-the-skill-core-with-ai-collaboration "Direct link to Step 3: Build the Skill Core with AI Collaboration") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now you're going to build this skill with AI. You'll test code, discover error patterns you hadn't anticipated, and learn what your actual data requires. As you iterate, the skill improves—not because you're following a formula, but because specification-driven feedback drives real improvements. ### Part A: Generate Initial Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-a-generate-initial-implementation "Direct link to Part A: Generate Initial Implementation") **Your prompt to AI:** I'm building a skill that processes CSV data. Here's my specification:[PASTE YOUR SPEC]Generate a Python skill implementation that:1. Reads the CSV file2. Validates the data structure (check columns, types)3. Performs the required transformation/analysis4. Returns results in the specified format5. Includes error handling for common CSV issuesThe code should be production-quality (defensive, not assuming data format). AI will generate code. Study it. Does it match your specification? **Critical evaluation**: * Does the code check for expected columns before using them? * Does it handle missing/null values? * Does it validate the output format matches your spec? **Document what you notice**: Things I observe:- [Good pattern in the approach]- [Assumption that might not hold]- [Edge case not addressed yet] Use these observations to guide your feedback to AI when iterating. ### Part B: Test with Real Data[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-b-test-with-real-data "Direct link to Part B: Test with Real Data") Get or create sample CSV data that matches your specification's expected format. **Run the generated code**: # Save AI-generated code to analysis.py# Create test_data.csv with sample data# Run itpython analysis.py test_data.csv **What happens**? * ✓ Success: Output matches specification → Great! Move to Part C * ✗ Syntax Error: Code won't even parse * ✗ Runtime Error: Code runs but crashes (KeyError, TypeError, etc.) * ✗ Logic Error: Code runs, output is wrong or incomplete ### Part C: Recover from Errors[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-c-recover-from-errors "Direct link to Part C: Recover from Errors") This is where error recovery becomes visible. **If you got a syntax error:** Show AI the error:"Here's my code:[show problematic section]Error: [paste error message]What's wrong and how do I fix it?" AI explains and provides corrected code. **If you got a runtime error:** Show AI the error:"The code crashed with:[error message and traceback]What does this error mean?What assumption did the code make that's wrong?How should I fix it to handle real data?" **If output is wrong/incomplete:** "My spec requires [required output].My code produces [what it actually produces].What's missing? How should the code be changed to match the spec?" ### Part D: Iterate Until Convergence[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-d-iterate-until-convergence "Direct link to Part D: Iterate Until Convergence") Keep improving until: ✓ Code runs without errors ✓ Output matches your specification exactly ✓ Edge cases are handled (test with malformed data) ✓ Execution completes within time limit **Test with multiple scenarios:** # Test 1: Clean data (happy path)python analysis.py clean_data.csv# Test 2: Missing columnspython analysis.py missing_columns.csv# Test 3: Non-numeric values where numeric expectedpython analysis.py malformed_data.csv# Test 4: Empty filepython analysis.py empty.csv# Test 5: Large file (check performance)python analysis.py large_data.csv For each test, document: * Did it run without error? (Yes/No) * Does output match spec format? (Yes/No) * Are edge cases handled gracefully? (Yes/No) Step 4: Build the Iteration Loop (The Skill Automating the Pattern)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-4-build-the-iteration-loop-the-skill-automating-the-pattern "Direct link to Step 4: Build the Iteration Loop (The Skill Automating the Pattern)") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Now that you've manually gone through the loop, you're going to build a skill that does this automatically. **Your skill needs these components:** def build_analysis_skill(): """ The full script-execution skill that orchestrates: 1. Generate code from spec 2. Execute the code 3. Check for errors 4. Generate fixes if needed 5. Iterate until convergence """ # Component 1: Code Generation def generate_code(specification: str) -> str: """Generate Python code from specification using AI""" # Prompt AI with: "Given this spec: [spec], # write complete Python code that implements it" # Return the generated code pass # Component 2: Code Execution def execute_code(code: str, input_file: str, timeout: int = 30) -> tuple[bool, str, str]: """Execute code, return (success, output, error_message)""" # Run code with subprocess # Capture stdout, stderr # Return results with timeout protection pass # Component 3: Error Analysis def analyze_error(error_message: str, code: str) -> str: """Understand what went wrong""" # Parse error type (SyntaxError, RuntimeError, etc.) # Extract the problematic line # Return clear analysis of the issue pass # Component 4: Fix Generation def generate_fix(error_analysis: str, code: str, spec: str) -> str: """Generate corrected code""" # Prompt AI: "This code failed with: [error] # Here's the problem: [analysis] # The spec is: [spec] # Generate corrected code that fixes this" pass # Component 5: Convergence Check def check_convergence(output: str, spec: dict) -> bool: """Does output satisfy the specification?""" # Validate: all required fields present # Validate: output format correct # Validate: no error messages in output # Return True if spec is satisfied pass # Component 6: Main Iteration Loop def execute_skill(specification: str, input_file: str) -> str: """Main skill that orchestrates everything""" max_iterations = 5 iteration = 0 code = None while iteration < max_iterations: iteration += 1 if iteration == 1: # First iteration: generate from spec code = generate_code(specification) # Execute the code success, output, error = execute_code(code, input_file) if success and check_convergence(output, spec): # ✓ Converged! Specification is satisfied return output if not success: # ✗ Error occurred analysis = analyze_error(error, code) code = generate_fix(analysis, code, specification) # Loop continues, retry with fixed code elif not check_convergence(output, spec): # ✗ Output doesn't match spec fix_request = f"Output is incomplete: {output}. Required by spec: {spec}. Generate code that adds missing parts." code = generate_fix(fix_request, code, specification) # Loop continues, retry with improved code # If we get here, max iterations reached without converging raise RuntimeError(f"Failed to converge after {max_iterations} iterations") Step 5: Implementation Guidance with AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-5-implementation-guidance-with-ai "Direct link to Step 5: Implementation Guidance with AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You're going to build this skill using AI, but testing and validating each component. ### Get AI Help Building the Iteration Loop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#get-ai-help-building-the-iteration-loop "Direct link to Get AI Help Building the Iteration Loop") I'm building a Python skill that generates code, executes it, and iteratesuntil a specification is satisfied.Here's my specification:[PASTE YOUR SPEC]Here's my first attempt at code generation and execution:[PASTE YOUR MANUAL CODE FROM STEP 3-4]Now I need to build an automated loop that:1. Generates code once (given spec)2. Executes code (capture output/errors, 30-second timeout)3. If error: analyze error, prompt you to generate fixed code4. If output doesn't match spec: prompt you to improve code5. Check convergence (spec is satisfied) → Stop6. Repeat until convergence or 5 iterations maxShow me how to structure this as a Python class/functions.Include error handling, timeout protection, and convergence checking. ### Build Convergence Validation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#build-convergence-validation "Direct link to Build Convergence Validation") This is critical. Your skill must STOP when the specification is satisfied. def convergence_check(output: str, specification: dict) -> dict: """ Validate whether output satisfies specification. Returns: { 'converged': bool, 'missing': [list of unsatisfied requirements], 'issues': [any problems found] } """ results = { 'converged': True, 'missing': [], 'issues': [] } # Check all required fields are present for field in specification.get('output', {}).get('required_fields', []): if field not in output: results['missing'].append(f"Field missing: {field}") results['converged'] = False # Check output format (if JSON specified) if specification.get('output', {}).get('format') == 'JSON': try: json.loads(output) except: results['issues'].append("Output is not valid JSON") results['converged'] = False # Add domain-specific validation based on your spec # Example: if analyzing customers, verify segments exist if 'required_segments' in specification: for segment in specification['required_segments']: if segment not in output: results['missing'].append(f"Segment missing: {segment}") results['converged'] = False return results ### Add Timeout and Resource Protection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#add-timeout-and-resource-protection "Direct link to Add Timeout and Resource Protection") import subprocessimport signaldef execute_code_safely(code: str, input_file: str, timeout: int = 30) -> tuple[bool, str, str]: """ Execute Python code with timeout and error capture. Returns: (success: bool, output: str, error: str) """ # Write code to temporary file with open('_temp_analysis.py', 'w') as f: f.write(code) try: # Run with timeout result = subprocess.run( ['python', '_temp_analysis.py', input_file], capture_output=True, text=True, timeout=timeout ) if result.returncode == 0: # Success return (True, result.stdout, '') else: # Execution failed return (False, result.stdout, result.stderr) except subprocess.TimeoutExpired: return (False, '', 'TimeoutError: Execution exceeded 30 seconds') except Exception as e: return (False, '', f'ExecutionError: {str(e)}') Step 6: Test Your Skill Against Edge Cases[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-6-test-your-skill-against-edge-cases "Direct link to Step 6: Test Your Skill Against Edge Cases") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your skill should handle: ### Test 1: Clean Data (Happy Path)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-1-clean-data-happy-path "Direct link to Test 1: Clean Data (Happy Path)") skill = ScriptExecutionSkill( specification=your_spec, input_file='clean_data.csv')result = skill.execute()assert result is not Noneassert 'error' not in result.lower() **Expected**: Succeeds on first iteration ### Test 2: Malformed Data (Edge Case)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-2-malformed-data-edge-case "Direct link to Test 2: Malformed Data (Edge Case)") # CSV with missing columns, non-numeric values, etc.result = skill.execute(input_file='malformed_data.csv')# Skill should detect error, fix code, retryassert 'error' not in result.lower() # After recovery, still valid **Expected**: Skill generates fix after detecting error ### Test 3: Empty File (Non-Recoverable)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-3-empty-file-non-recoverable "Direct link to Test 3: Empty File (Non-Recoverable)") result = skill.execute(input_file='empty.csv')# This SHOULD fail (non-recoverable)assert result is None or 'error' in result.lower() **Expected**: Skill recognizes this is non-recoverable, stops gracefully ### Test 4: Timeout Scenario[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-4-timeout-scenario "Direct link to Test 4: Timeout Scenario") # Spec with large data processing that might timeoutresult = skill.execute(input_file='large_data.csv', timeout=5)# Skill should timeout gracefully, not hangassert 'timeout' in result.lower() or result is None **Expected**: Skill times out, reports clearly Step 7: Document Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-7-document-your-skill "Direct link to Step 7: Document Your Skill") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Real skills are documented for others to use. # CSV Analysis Skill## Purpose[What problem does this solve?]## Usage```pythonfrom my_skill import ScriptExecutionSkillskill = ScriptExecutionSkill( specification={ 'input': ['customers.csv'], 'output': {'format': 'JSON', 'required_fields': [...]}, 'success_criteria': [...] }, input_file='customers.csv')result = skill.execute()print(result) How It Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#how-it-works "Direct link to How It Works") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Specification defines what code must do 2. Skill generates Python code from spec 3. Code executes against input file 4. Errors trigger automatic fix generation 5. Iteration continues until spec is satisfied or max retries reached Success Metrics[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#success-metrics "Direct link to Success Metrics") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * Execution time: < 30 seconds * Convergence rate: 95%+ (passes with clean data) * Edge case handling: Gracefully recovers or fails clearly Known Limitations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#known-limitations "Direct link to Known Limitations") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- * \[What doesn't it handle?\] * \[When should you use something else?\] ---## Try With AINow you'll refine your skill with AI collaboration, focused on error recovery and robustness.### Prompt 1: Design Error Recovery Patterns I've built a skill that generates Python code from specifications and executes it. It encounters three types of errors: 1. Syntax errors (code won't parse) 2. Runtime errors (code crashes during execution) 3. Logic errors (code runs but output is wrong) For each error type, help me design the recovery strategy: **Syntax errors**: * How should I prompt you to generate fixed code? * What context should I provide? **Runtime errors**: * How should I parse the error message? * What information helps you generate a better fix? **Logic errors**: * How do I detect these (they don't produce error messages)? * How should I describe the problem to you? Show me the exact prompts I should use for each type. **What you're learning**: How to design prompts that help AI generate fixes, not just resuggest the same broken code.### Prompt 2: Implement Convergence Testing My specification requires these success criteria: \[PASTE YOUR CRITERIA FROM YOUR SPEC\] I need a function that validates whether code output satisfies these criteria. For each criterion, what should the validation check? * How do I verify the output format is correct? * How do I verify all required fields are present? * How do I detect if the output is incomplete or wrong? Show me a Python function that validates all criteria and returns which ones passed, which ones failed, and what's missing. **What you're learning**: How to translate specification requirements into automated validation that tells you exactly when to stop iterating.### Prompt 3: Test Your Skill with Intentional Failures I want to test my skill's error recovery. Help me design test cases: **Test Case 1: Missing column** * Create CSV data where a required column is missing * Show me what error the generated code will produce * What should my skill do to recover? **Test Case 2: Wrong data type** * Create data where a numeric column contains text * Show the error this produces * How should the skill fix this? **Test Case 3: Timeout scenario** * What operation would cause a timeout? * How should my skill handle timeouts gracefully? For each test case, show me: 1. The test data 2. The error produced 3. How my skill should recover **What you're learning**: Testing is not about success cases—it's about understanding how your skill behaves when things break.### Prompt 4: Validate Convergence Against Diverse Inputs My skill has processed the following test scenarios: **Test 1 - Clean data**: PASSED **Test 2 - Missing column**: RECOVERED (3 iterations) **Test 3 - Empty file**: FAILED (non-recoverable) **Test 4 - Malformed values**: RECOVERED (2 iterations) Based on these results: * Is my skill ready for production? * What patterns suggest robustness? * What edge cases might still break it? * What should I test next? Help me evaluate the skill's readiness. **What you're learning**: Testing isn't a binary pass/fail. It's about understanding your skill's behavior patterns and building confidence in its robustness.---## Success CriteriaYour skill is complete when:✓ **Specification is clear and complete** — AI can generate code from it without asking questions✓ **Code executes successfully on clean data** — Happy path works✓ **Error recovery works** — Syntax and runtime errors trigger fixes✓ **Convergence is detected** — Skill stops when spec is satisfied✓ **Edge cases are handled** — Tested with malformed, empty, large data✓ **Iteration limits work** — Skill stops after 5 attempts or timeout✓ **Skill is documented** — Someone else could use itYour skill will become a reusable component in Lesson 7 (orchestration) when you combine it with MCP-wrapping skills to create complete workflows.---**Takeaway**: You didn't just learn the write-execute-analyze loop—you built a skill that automates it. You discovered that error recovery isn't magic; it's specification clarity + intelligent prompting + convergence validation. In Lesson 7, you'll orchestrate this skill with MCP-wrapping skills to build complex workflows that combine code execution with external tools. Checking access... * [Step 1: Write Your Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-1-write-your-specification) * [Your Specification](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#your-specification) * [Step 2: Design Your Skill's Persona and Questions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-2-design-your-skills-persona-and-questions) * [Step 3: Build the Skill Core with AI Collaboration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-3-build-the-skill-core-with-ai-collaboration) * [Part A: Generate Initial Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-a-generate-initial-implementation) * [Part B: Test with Real Data](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-b-test-with-real-data) * [Part C: Recover from Errors](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-c-recover-from-errors) * [Part D: Iterate Until Convergence](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#part-d-iterate-until-convergence) * [Step 4: Build the Iteration Loop (The Skill Automating the Pattern)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-4-build-the-iteration-loop-the-skill-automating-the-pattern) * [Step 5: Implementation Guidance with AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-5-implementation-guidance-with-ai) * [Get AI Help Building the Iteration Loop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#get-ai-help-building-the-iteration-loop) * [Build Convergence Validation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#build-convergence-validation) * [Add Timeout and Resource Protection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#add-timeout-and-resource-protection) * [Step 6: Test Your Skill Against Edge Cases](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-6-test-your-skill-against-edge-cases) * [Test 1: Clean Data (Happy Path)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-1-clean-data-happy-path) * [Test 2: Malformed Data (Edge Case)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-2-malformed-data-edge-case) * [Test 3: Empty File (Non-Recoverable)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-3-empty-file-non-recoverable) * [Test 4: Timeout Scenario](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#test-4-timeout-scenario) * [Step 7: Document Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#step-7-document-your-skill) * [How It Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#how-it-works) * [Success Metrics](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#success-metrics) * [Known Limitations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/agent-skills-mcp-code-execution/build-script-execution-skill#known-limitations) --- # Capstone: Building a Customer Support Digital FTE | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You have learned every pattern in the OpenAI Agents SDK toolkit. Now you prove mastery by building something you could sell. Customer support is a $15+ billion market where businesses spend 60-70% of their support budget on labor. A Digital FTE that handles Tier 1 support---FAQs, billing questions, basic troubleshooting---at $500-2,000/month versus $4,000-8,000/month for a human representative represents genuine value. This capstone is different from a tutorial. There's no step-by-step code to copy. Instead, you'll receive: * **Architecture** to guide your design * **Requirements** to specify what you must build * **Lesson references** pointing back to the patterns you learned * **Validation checklist** to verify your implementation * **Business strategy** to monetize your creation By the end, you'll have a production-ready Customer Support Digital FTE that demonstrates mastery of this chapter. System Architecture[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#system-architecture "Direct link to System Architecture") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Digital FTE follows this architecture: User Message │ ▼┌─────────────────────────────────────────┐│ INPUT GUARDRAILS ││ ┌─────────────┐ ┌─────────────────┐ ││ │ PII Check │ │ Injection Check │ ││ └─────────────┘ └─────────────────┘ │└─────────────────────────────────────────┘ │ (if passes) ▼┌─────────────────────────────────────────┐│ TRIAGE AGENT ││ Routes based on intent analysis ││ Tools: lookup_customer │└─────────────────────────────────────────┘ │ ├──── FAQ? ────► FAQAgent ─────────────────►┐ │ │ │ │ ▼ │ │ ┌──────────────────┐ │ │ │ KNOWLEDGE BASE │ │ │ │ (FileSearchTool)│ │ │ │ - Policies │ │ │ │ - FAQs │ │ │ │ - Product docs │ │ │ └──────────────────┘ │ │ │ ├── Billing? ──► BillingAgent ─────────────►│ │ │ │ │ └─► EscalationAgent ────►│ │ │ └─ Technical? ─► TechnicalAgent ───────────►│ │ │ └─► EscalationAgent ────►│ │ ▼┌─────────────────────────────────────────┐│ OUTPUT GUARDRAILS ││ ┌──────────────────────────────────┐ ││ │ Secrets/PII Leakage Detection │ ││ └──────────────────────────────────┘ │└─────────────────────────────────────────┘ │ ▼┌─────────────────────────────────────────┐│ SESSION STORAGE ││ SQLiteSession for persistence │└─────────────────────────────────────────┘ │ ▼User Response (with tracing) Component Requirements[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#component-requirements "Direct link to Component Requirements") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Build each component using patterns from the specified lessons: ### 1\. Context Model[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#1-context-model "Direct link to 1. Context Model") **Requirements:** * Track customer identification (ID, email, plan) * Track session metadata (ID, start time) * Track routing history (handoffs, agents involved) * Track metrics (tokens, estimated cost) * Track resolution status (resolved, escalated, reason) **Reference:** Lesson 2 - Context objects with Pydantic BaseModel * * * ### 2\. Input Guardrails[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#2-input-guardrails "Direct link to 2. Input Guardrails") **Requirements:** * PII detection: Block credit card numbers, SSN patterns, bank account numbers * Prompt injection detection: Block "ignore previous instructions", "you are now", "pretend you are" patterns * Return user-friendly error messages when triggered **Reference:** Lesson 5 - `@input_guardrail` decorator, `GuardrailFunctionOutput`, `tripwire_triggered` * * * ### 3\. Output Guardrails[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#3-output-guardrails "Direct link to 3. Output Guardrails") **Requirements:** * Detect API keys, internal IDs, database queries, passwords in output * Block responses that would leak sensitive data **Reference:** Lesson 5 - `@output_guardrail` decorator * * * ### 4\. Agent Tools[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#4-agent-tools "Direct link to 4. Agent Tools") **Requirements:** | Tool | Purpose | Returns | | --- | --- | --- | | `lookup_customer` | Find customer by email | Customer info or "not found" | | `check_billing_history` | Get recent orders | List of orders | | `process_refund` | Refund orders under $100 | Confirmation or escalation needed | | `check_support_tickets` | Get open tickets | List of tickets | | `create_escalation_ticket` | Create human handoff | Ticket ID with SLA | **Reference:** Lesson 2 - `@function_tool` decorator, `RunContextWrapper[T]` * * * ### 5\. Specialist Agents[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#5-specialist-agents "Direct link to 5. Specialist Agents") **Requirements:** | Agent | Responsibilities | Tools | Handoffs To | | --- | --- | --- | --- | | **FAQAgent** | Answer pricing, policies, features | None | None | | **BillingAgent** | Handle payments, refunds | `check_billing_history`, `process_refund`, `check_support_tickets` | EscalationAgent | | **TechnicalAgent** | Resolve product issues | `check_support_tickets` | EscalationAgent | | **EscalationAgent** | Prepare cases for humans | `create_escalation_ticket` | None | **Reference:** Lesson 1 (basic agents), Lesson 2 (tools), Lesson 4 (handoffs) * * * ### 6\. Triage Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#6-triage-agent "Direct link to 6. Triage Agent") **Requirements:** * Entry point for all conversations * Identify customer using `lookup_customer` * Route to appropriate specialist based on intent * Apply input guardrails * Apply output guardrails **Routing Rules:** * General questions (pricing, policies, features) → FAQAgent * Billing issues (charges, refunds, payments) → BillingAgent * Technical problems (errors, bugs, API) → TechnicalAgent **Reference:** Lesson 4 - `handoff()` function, `on_handoff` callbacks, `handoff_filters` * * * ### 7\. Observability Hooks[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#7-observability-hooks "Direct link to 7. Observability Hooks") **Requirements:** * Log agent start/end with timing * Log tool start/end * Log handoffs * Track which agents were involved * Output structured JSON logs **Reference:** Lesson 7 - `RunHooks` class, lifecycle methods * * * ### 8\. Session Management[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#8-session-management "Direct link to 8. Session Management") **Requirements:** * Create sessions with unique IDs * Persist conversations across turns * Enable multi-user support **Reference:** Lesson 6 - `SQLiteSession`, session parameter in `Runner.run()` * * * ### 9\. Knowledge Base (RAG)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#9-knowledge-base-rag "Direct link to 9. Knowledge Base (RAG)") **Requirements:** * Upload internal documents (policies, FAQs, product guides) * Create vector store for semantic search * Integrate `FileSearchTool` with FAQAgent for policy lookups * Enable agents to cite sources from knowledge base **Documents to include:** | Document | Purpose | Used By | | --- | --- | --- | | `return-policy.md` | Refund and return rules | FAQAgent, BillingAgent | | `pricing-guide.md` | Plan features and pricing | FAQAgent | | `troubleshooting.md` | Common technical issues | TechnicalAgent | | `escalation-criteria.md` | When to escalate to humans | All agents | **Reference:** Lesson 9 - `FileSearchTool`, vector stores, `file_search` tool type * * * ### 10\. MCP Integration (Optional)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#10-mcp-integration-optional "Direct link to 10. MCP Integration (Optional)") **Requirements:** * Connect to external documentation server for live product docs * Enable real-time knowledge updates without redeployment * Use `async with` pattern for proper lifecycle management **Reference:** Lesson 8 - `MCPServerStreamableHttp`, `params` dictionary, agent creation inside context * * * ### 11\. Main Handler[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#11-main-handler "Direct link to 11. Main Handler") **Requirements:** * Accept message, session, context, hooks * Generate trace ID for each request * Handle guardrail exceptions with user-friendly messages * Track token usage and costs * Use `RunConfig` with `max_turns` to prevent infinite loops **Reference:** Lesson 7 - `gen_trace_id()`, `trace()`, `RunConfig` * * * Validation Checklist[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#validation-checklist "Direct link to Validation Checklist") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your implementation passes when: ### Routing[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#routing "Direct link to Routing") * [ ] FAQ questions route to FAQAgent * [ ] Billing questions route to BillingAgent * [ ] Technical questions route to TechnicalAgent * [ ] Complex issues escalate properly ### Guardrails[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#guardrails "Direct link to Guardrails") * [ ] Credit card numbers are blocked * [ ] SSN patterns are blocked * [ ] Prompt injection attempts are blocked * [ ] API keys don't appear in output ### Tools[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#tools "Direct link to Tools") * [ ] Customer lookup updates context * [ ] Billing history returns order list * [ ] Refunds under $100 process successfully * [ ] Refunds over $100 trigger escalation * [ ] Escalation tickets include priority and SLA ### Sessions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#sessions "Direct link to Sessions") * [ ] Conversations persist across turns * [ ] Different users have isolated sessions * [ ] Context survives session reconnection ### Knowledge Base (RAG)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#knowledge-base-rag "Direct link to Knowledge Base (RAG)") * [ ] Vector store created with policy documents * [ ] FAQAgent retrieves relevant policies * [ ] Responses cite sources from knowledge base * [ ] Policy questions answered accurately ### Observability[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#observability "Direct link to Observability") * [ ] Agent lifecycle events are logged * [ ] Tool calls are logged * [ ] Handoffs are logged * [ ] Session summary shows metrics ### Demo Scenarios[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#demo-scenarios "Direct link to Demo Scenarios") **Scenario 1: Billing Issue (Test routing + tools)** Turn 1: "Hi, I'm alice@example.com and I was charged twice this month."Expected: Triage identifies customer, routes to BillingAgentTurn 2: "Yes, I see ORD-1001 and ORD-1002 on the same day for $99 each."Expected: BillingAgent confirms duplicate chargesTurn 3: "Please process the refund for the duplicate charge."Expected: BillingAgent processes refund, provides confirmation **Scenario 2: Policy Question (Test RAG)** Turn 1: "What is your refund policy for annual subscriptions?"Expected: Routes to FAQAgent, retrieves from knowledge base, cites return-policy.mdTurn 2: "Can I get a prorated refund if I cancel mid-year?"Expected: FAQAgent answers with specific policy details from knowledge base **Scenario 3: Guardrail Test** Turn 1: "My credit card is 4532-1234-5678-9012, can you check my account?"Expected: Input guardrail blocks, returns user-friendly message about PII * * * Monetization Models[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#monetization-models "Direct link to Monetization Models") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Building the agent is half the journey. The other half is turning it into a business. ### Model 1: Subscription (Managed Service)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#model-1-subscription-managed-service "Direct link to Model 1: Subscription (Managed Service)") | Tier | Monthly Price | Included | Best For | | --- | --- | --- | --- | | Starter | $500/month | 1,000 conversations | Small businesses | | Growth | $1,500/month | 5,000 conversations | Growing teams | | Enterprise | $3,000+/month | Unlimited + SLA | Large organizations | **Margin calculation** (Growth tier): * Revenue: $1,500/month * Token costs: ~$300/month (5K conversations × ~$0.06 each) * Infrastructure: ~$100/month * **Gross margin: ~73%** ### Model 2: Success Fee (Per Resolution)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#model-2-success-fee-per-resolution "Direct link to Model 2: Success Fee (Per Resolution)") | Metric | Price | Rationale | | --- | --- | --- | | Per conversation | $0.50-2.00 | Volume-based | | Per resolution | $2.00-5.00 | Value-based | | Per escalation avoided | $5.00-15.00 | Cost savings | **Advantage:** Aligns incentives. You only get paid when the Digital FTE delivers value. ### Model 3: Hybrid (Base + Success)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#model-3-hybrid-base--success "Direct link to Model 3: Hybrid (Base + Success)") | Component | Price | | --- | --- | | Base platform fee | $200/month | | Per conversation | $0.25 | | Per escalation avoided | $3.00 | **Why hybrid works:** Predictable base revenue with upside for performance. ### Pricing Calculator[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#pricing-calculator "Direct link to Pricing Calculator") Calculate your minimum viable price: Monthly conversations: [X]Avg tokens per conversation: ~2,000Input token cost: (X × 1,200 × $2.50/M) = $AOutput token cost: (X × 800 × $10.00/M) = $BInfrastructure: $50 + (X × $0.01) = $CTotal cost = $A + $B + $CRequired revenue (65% margin) = Total cost ÷ 0.35Per conversation price = Required revenue ÷ X **Example at 5,000 conversations:** * Token cost: $275 * Infrastructure: $100 * Total: $375 * Required revenue: ~$1,070 * Per conversation: ~$0.21 * * * What's Next: Distribution and Deployment[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#whats-next-distribution-and-deployment "Direct link to What's Next: Distribution and Deployment") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Digital FTE is built. Now you need customers and infrastructure: | Next Step | Chapter | What You'll Learn | | --- | --- | --- | | **Distribution** | Ch72: OpenAI Apps SDK | Package your agent for ChatGPT's 800M+ users | | **Containerization** | Ch49: Docker | Package your agent as a deployable container | | **Orchestration** | Ch50: Kubernetes | Scale to handle thousands of concurrent users | | **Monitoring** | Ch51: Helm Charts | Production monitoring and auto-scaling | The BUILD phase is complete. The DISTRIBUTE and DEPLOY phases transform your working prototype into a business. * * * Progressive Project: Complete Your Support Desk[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#progressive-project-complete-your-support-desk "Direct link to Progressive Project: Complete Your Support Desk") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've built the Support Desk progressively through 9 lessons: | Lesson | Capability Added | Key Pattern | | --- | --- | --- | | L01 | Basic agent | `Agent()`, `Runner.run()` | | L02 | Function tools | `@function_tool`, `RunContextWrapper` | | L03 | Sub-agents | `.as_tool()` pattern | | L04 | Handoffs | `handoff()`, routing, callbacks | | L05 | Guardrails | `@input_guardrail`, `@output_guardrail` | | L06 | Sessions | `SQLiteSession`, persistence | | L07 | Tracing | `RunHooks`, `trace()`, metrics | | L08 | MCP | `MCPServerStreamableHttp`, live docs | | L09 | RAG | `FileSearchTool`, knowledge base | ### Your Task[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#your-task "Direct link to Your Task") **Integrate all 9 versions into a single production system.** You already have the code. The capstone proves you understand how the pieces fit together. **Step 1: Gather your components** Open each version (v1.0-v9.0) of your Support Desk. Identify the imports, classes, and functions you'll need. **Step 2: Design the integration** Sketch how components connect: * Which agents need which tools? * Where do guardrails attach? * How does session data flow? * What gets traced? **Step 3: Build incrementally** Don't try to integrate everything at once: 1. First: Triage → Specialists (no guardrails, no sessions) 2. Then: Add guardrails 3. Then: Add sessions 4. Then: Add observability 5. Finally: Add MCP and RAG (if time permits) **Step 4: Test each integration** After each step, run a test conversation. Fix issues before adding more complexity. **Step 5: Run the validation scenario** Use the demo scenario in the [Validation Checklist](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#validation-checklist) to verify your complete system. * * * ### Applying to Your Domain[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#applying-to-your-domain "Direct link to Applying to Your Domain") Once your Support Desk works, adapt these patterns to **your domain**: | Domain | Specialists | Key Guardrails | | --- | --- | --- | | Legal | Intake, Research, Document Review | Attorney-client privilege | | Healthcare | Triage, Scheduling, Billing, Clinical | HIPAA, PHI detection | | Finance | Account Services, Trading, Compliance | PII, investment disclaimers | | Education | Admissions, Registration, Financial Aid | FERPA, student records | **Domain Adaptation Checklist:** * [ ] Identify distinct workflows → specialist agents * [ ] Map client language → routing rules * [ ] Research compliance requirements * [ ] Design domain-specific guardrails * [ ] Calculate pricing (human cost vs. your value) * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion to refine your implementation. ### Prompt 1: Architecture Review[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#prompt-1-architecture-review "Direct link to Prompt 1: Architecture Review") Review my Customer Support Digital FTE implementation:[Paste your code]Evaluate:1. Are agent responsibilities clearly separated?2. Are handoff conditions non-overlapping?3. What edge cases might cause routing failures?4. How would you improve error handling? **What you're learning:** Critical analysis of multi-agent systems. ### Prompt 2: Compliance Enhancement[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#prompt-2-compliance-enhancement "Direct link to Prompt 2: Compliance Enhancement") I'm deploying my Support Digital FTE to handle [INDUSTRY] clients.Help me:1. Identify compliance requirements (GDPR, HIPAA, PCI, SOC2)2. Design additional guardrails for compliance3. Implement audit logging for compliance evidence4. Create data retention policies **What you're learning:** Compliance-first design for regulated industries. ### Prompt 3: Monetization Strategy[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#prompt-3-monetization-strategy "Direct link to Prompt 3: Monetization Strategy") I've built a [DOMAIN] Digital FTE that handles [USE CASE].Help me develop:1. Ideal customer profile (size, pain points)2. Pricing model (subscription, usage, hybrid)3. Value calculation vs. human agent4. ROI metrics for sales conversations5. Common objections and responses **What you're learning:** Business model development for AI products. * * * Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#safety-note "Direct link to Safety Note") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Production deployment requires careful consideration: * **Legal review**: Have counsel review guardrails and disclaimers * **Data handling**: Ensure session storage complies with GDPR, CCPA * **Liability**: Clarify what happens when the agent gives incorrect information * **Human escalation**: Always provide a path to human support * **Monitoring**: Alert on guardrail triggers and unusual patterns * **Load testing**: Agents behave differently under pressure * **Rollback plan**: Have a way to disable the Digital FTE if issues arise * * * Chapter Complete[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#chapter-complete "Direct link to Chapter Complete") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've mastered the OpenAI Agents SDK: | Skill | Lesson | Status | | --- | --- | --- | | Basic agents | L01 | ✓ | | Function tools | L02 | ✓ | | Agents-as-tools | L03 | ✓ | | Handoffs | L04 | ✓ | | Guardrails | L05 | ✓ | | Sessions | L06 | ✓ | | Tracing | L07 | ✓ | | MCP | L08 | ✓ | | RAG | L09 | ✓ | | **Integration** | L10 | ✓ | You now have the skills to build production-grade Digital FTEs. The next chapters show you how to distribute them (Apps SDK) and deploy them at scale (Cloud-Native). **Your Digital FTE journey continues in Part 7: AI Cloud-Native Development.** * [System Architecture](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#system-architecture) * [Component Requirements](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#component-requirements) * [1\. Context Model](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#1-context-model) * [2\. Input Guardrails](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#2-input-guardrails) * [3\. Output Guardrails](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#3-output-guardrails) * [4\. Agent Tools](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#4-agent-tools) * [5\. Specialist Agents](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#5-specialist-agents) * [6\. Triage Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#6-triage-agent) * [7\. Observability Hooks](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#7-observability-hooks) * [8\. Session Management](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#8-session-management) * [9\. Knowledge Base (RAG)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#9-knowledge-base-rag) * [10\. MCP Integration (Optional)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#10-mcp-integration-optional) * [11\. Main Handler](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#11-main-handler) * [Validation Checklist](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#validation-checklist) * [Routing](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#routing) * [Guardrails](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#guardrails) * [Tools](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#tools) * [Sessions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#sessions) * [Knowledge Base (RAG)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#knowledge-base-rag) * [Observability](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#observability) * [Demo Scenarios](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#demo-scenarios) * [Monetization Models](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#monetization-models) * [Model 1: Subscription (Managed Service)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#model-1-subscription-managed-service) * [Model 2: Success Fee (Per Resolution)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#model-2-success-fee-per-resolution) * [Model 3: Hybrid (Base + Success)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#model-3-hybrid-base--success) * [Pricing Calculator](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#pricing-calculator) * [What's Next: Distribution and Deployment](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#whats-next-distribution-and-deployment) * [Progressive Project: Complete Your Support Desk](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#progressive-project-complete-your-support-desk) * [Your Task](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#your-task) * [Applying to Your Domain](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#applying-to-your-domain) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#try-with-ai) * [Prompt 1: Architecture Review](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#prompt-1-architecture-review) * [Prompt 2: Compliance Enhancement](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#prompt-2-compliance-enhancement) * [Prompt 3: Monetization Strategy](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#prompt-3-monetization-strategy) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#safety-note) * [Chapter Complete](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-agents-sdk/capstone-customer-support-fte#chapter-complete) --- # Task Actions with callTool | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your task list displays tasks, but clicking a checkbox does nothing. In this lesson, you'll add real actions—Complete and Delete buttons that modify tasks on the server. The key difference from Lesson 3: `sendFollowUpMessage` goes through the conversation (the model sees it), while `callTool` talks directly to your server (no conversation turn). For fast, silent updates like toggling checkboxes, `callTool` is what you need. sendFollowUpMessage vs callTool[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#sendfollowupmessage-vs-calltool "Direct link to sendFollowUpMessage vs callTool") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Method | Goes Through Model | Creates Turn | Best For | | --- | --- | --- | --- | | `sendFollowUpMessage` | Yes | Yes | User-initiated actions, adding tasks | | `callTool` | No | No | Silent updates, toggling, deleting | When a user clicks "Complete Task", they don't need the model to narrate "I've marked task 1 as complete." They just want the checkbox to update. That's `callTool`. Adding Action Tools[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#adding-action-tools "Direct link to Adding Action Tools") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Add two new tools to your server. Update `main.py`: import mcp.types as typesfrom mcp.server.fastmcp import FastMCPMIME_TYPE = "text/html+skybridge"TASKS = [ {"id": 1, "title": "Buy groceries", "done": False}, {"id": 2, "title": "Review pull request", "done": False}, {"id": 3, "title": "Call mom", "done": True},]mcp = FastMCP("TaskManager")@mcp.tool( annotations={ "openai/widgetAccessible": True, # Required for callTool })def complete_task(task_id: int) -> types.CallToolResult: """Toggle a task's completion status.""" for task in TASKS: if task["id"] == task_id: task["done"] = not task["done"] return types.CallToolResult( content=[types.TextContent(type="text", text="Task toggled")], structuredContent={"success": True, "task_id": task_id, "done": task["done"]} ) return types.CallToolResult( content=[types.TextContent(type="text", text="Not found")], structuredContent={"success": False} )@mcp.tool( annotations={ "openai/widgetAccessible": True, })def delete_task(task_id: int) -> types.CallToolResult: """Delete a task by ID.""" global TASKS original_len = len(TASKS) TASKS = [t for t in TASKS if t["id"] != task_id] success = len(TASKS) < original_len return types.CallToolResult( content=[types.TextContent(type="text", text="Deleted" if success else "Not found")], structuredContent={"success": success} ) ### The Critical Annotation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#the-critical-annotation "Direct link to The Critical Annotation") @mcp.tool( annotations={ "openai/widgetAccessible": True, }) Without `openai/widgetAccessible: True`, the tool cannot be called from widgets. This is a security feature—only explicitly marked tools are accessible via `callTool`. Updating the Widget[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#updating-the-widget "Direct link to Updating the Widget") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Add action buttons to each task. Update `WIDGET_HTML`: WIDGET_HTML = '''

    TaskManager

    Loading...

      ''' ### How callTool Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#how-calltool-works "Direct link to How callTool Works") async function toggleTask(taskId) { await window.openai?.callTool?.('complete_task', { task_id: taskId }); refresh();} 1. `callTool('complete_task', { task_id: taskId })` calls your server tool directly 2. The tool toggles the task and returns immediately 3. `refresh()` triggers `sendFollowUpMessage` to redraw the widget The user sees: click checkbox → instant update. No model narration interrupts the flow. The Pattern: callTool + Refresh[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#the-pattern-calltool--refresh "Direct link to The Pattern: callTool + Refresh") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- For any action that modifies data: 1. `callTool` to make the change 2. `sendFollowUpMessage` to refresh the display This is the standard pattern for interactive widgets. Complete Server Code[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#complete-server-code "Direct link to Complete Server Code") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Here's the full `main.py`: import mcp.types as typesfrom mcp.server.fastmcp import FastMCPMIME_TYPE = "text/html+skybridge"TASKS = [ {"id": 1, "title": "Buy groceries", "done": False}, {"id": 2, "title": "Review pull request", "done": False}, {"id": 3, "title": "Call mom", "done": True},]WIDGET_HTML = '''

      TaskManager

      Loading...

        '''mcp = FastMCP("TaskManager")@mcp.tool(annotations={"openai/widgetAccessible": True})def complete_task(task_id: int) -> types.CallToolResult: """Toggle a task's completion status.""" for task in TASKS: if task["id"] == task_id: task["done"] = not task["done"] return types.CallToolResult( content=[types.TextContent(type="text", text="Task toggled")], structuredContent={"success": True, "task_id": task_id, "done": task["done"]} ) return types.CallToolResult( content=[types.TextContent(type="text", text="Not found")], structuredContent={"success": False} )@mcp.tool(annotations={"openai/widgetAccessible": True})def delete_task(task_id: int) -> types.CallToolResult: """Delete a task by ID.""" global TASKS original_len = len(TASKS) TASKS = [t for t in TASKS if t["id"] != task_id] success = len(TASKS) < original_len return types.CallToolResult( content=[types.TextContent(type="text", text="Deleted" if success else "Not found")], structuredContent={"success": success} )@mcp.tool()def show_tasks() -> types.CallToolResult: """Display the task list widget.""" pending = len([t for t in TASKS if not t["done"]]) return types.CallToolResult( content=[types.TextContent(type="text", text=f"Showing {len(TASKS)} tasks")], structuredContent={"total": len(TASKS), "pending": pending, "completed": len(TASKS) - pending}, _meta={"tasks": TASKS, "openai.com/widget": types.EmbeddedResource( type="resource", resource=types.TextResourceContents(uri="ui://tasks", mimeType=MIME_TYPE, text=WIDGET_HTML) )} )if __name__ == "__main__": import uvicorn app = mcp.sse_app() uvicorn.run(app, host="0.0.0.0", port=8001) Testing Task Actions[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#testing-task-actions "Direct link to Testing Task Actions") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. Restart your server 2. In ChatGPT: "Show my tasks" 3. Click a checkbox—it should toggle 4. Click the × button—task should disappear 5. The widget refreshes after each action If actions don't work, check: * `openai/widgetAccessible: True` is set on action tools * Tool names match exactly: `complete_task`, `delete_task` * Parameter name matches: `task_id` (not `taskId`) What You Built[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#what-you-built "Direct link to What You Built") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Building on previous lessons: * Added `complete_task` and `delete_task` tools * Used `openai/widgetAccessible` for widget access * Implemented `callTool` for direct server communication * Pattern: `callTool` → `refresh()` for all mutations Your TaskManager now has working Complete and Delete actions. In the next lesson, you'll add state persistence and display modes. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#try-with-ai "Direct link to Try With AI") ---------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Prompt 1: Add an "Add Task" Input[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#prompt-1-add-an-add-task-input "Direct link to Prompt 1: Add an "Add Task" Input") Add an input field and "Add" button to the widget.When clicked, it should send a prompt like "Add task: [input value]".Create an add_task tool on the server that creates a new task with an auto-incrementing ID. **What you're learning:** Combining `sendFollowUpMessage` (for adding, which benefits from model understanding) with `callTool` (for toggling/deleting, which doesn't). ### Prompt 2: Confirmation for Delete[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#prompt-2-confirmation-for-delete "Direct link to Prompt 2: Confirmation for Delete") Before deleting a task, show a confirmation prompt.Only call delete_task if the user confirms.Use window.confirm() for simplicity. **What you're learning:** Destructive action patterns. Users should confirm before losing data. ### Prompt 3: Error Handling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#prompt-3-error-handling "Direct link to Prompt 3: Error Handling") What happens if callTool fails (network error, server down)?Add error handling that shows a message to the user instead of silently failing. **What you're learning:** Defensive programming for widget actions. Users need feedback when things go wrong. Checking access... * [sendFollowUpMessage vs callTool](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#sendfollowupmessage-vs-calltool) * [Adding Action Tools](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#adding-action-tools) * [The Critical Annotation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#the-critical-annotation) * [Updating the Widget](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#updating-the-widget) * [How callTool Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#how-calltool-works) * [The Pattern: callTool + Refresh](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#the-pattern-calltool--refresh) * [Complete Server Code](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#complete-server-code) * [Testing Task Actions](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#testing-task-actions) * [What You Built](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#what-you-built) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#try-with-ai) * [Prompt 1: Add an "Add Task" Input](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#prompt-1-add-an-add-task-input) * [Prompt 2: Confirmation for Delete](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#prompt-2-confirmation-for-delete) * [Prompt 3: Error Handling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/openai-apps-sdk/state-display-modes#prompt-3-error-handling) --- # Memory Retrieval Strategies | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Storing memories is only half the problem. The other half—often the harder half—is getting the right memories back when you need them. Ask the wrong question of your memory system and you get irrelevant context. Ask the right question the wrong way and you miss crucial information. This lesson covers four retrieval strategies: recency-based (what happened recently), relevance-based (what matches the query), entity-based (what involves specific entities), and hybrid (combining all three). You'll learn when to use each and how to manage the constraint that makes it all tricky: the token budget. The Retrieval Challenge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#the-retrieval-challenge "Direct link to The Retrieval Challenge") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When a user asks "What about the project?"—which memories should the agent retrieve? Available Memories:├── "Alex completed OAuth implementation" (yesterday)├── "Phoenix project deadline is Jan 31" (2 weeks ago)├── "Alex prefers morning meetings" (1 month ago)├── "Sprint planning discussed Phoenix blockers" (3 days ago)├── "User's favorite color is blue" (2 months ago)└── "Phoenix project uses auth-service" (2 weeks ago) Different strategies give different answers: | Strategy | Selected Memories | | --- | --- | | Recency | OAuth implementation, Sprint planning | | Relevance | Phoenix deadline, Phoenix uses auth-service | | Entity ("Phoenix") | Phoenix deadline, Sprint planning, Phoenix uses auth-service | | Hybrid | Phoenix deadline, Sprint planning, OAuth implementation | The right choice depends on context. Let's explore each strategy. Recency-Based Retrieval[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#recency-based-retrieval "Direct link to Recency-Based Retrieval") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Principle:** Most recent memories are most likely relevant to current context. ### When to Use[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use "Direct link to When to Use") * Continuing an ongoing conversation * Following up on recent events * When the user references "what we just discussed" ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation "Direct link to Implementation") def get_recent_memories(user_id: str, limit: int = 10): """Retrieve most recent memories regardless of content.""" return memory.search( query="", # Empty query = no semantic filtering filters={"user_id": user_id}, sort_by="created_at", sort_order="desc", limit=limit ) **Output:** [ {"memory": "Alex completed OAuth implementation", "created_at": "2025-01-20"}, {"memory": "Sprint planning discussed Phoenix blockers", "created_at": "2025-01-18"}, {"memory": "Phoenix project deadline is Jan 31", "created_at": "2025-01-06"}, ...] ### Limitations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#limitations "Direct link to Limitations") * Misses old but highly relevant memories * Recency doesn't equal importance * Fails for queries about past events Relevance-Based Retrieval (Semantic Search)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#relevance-based-retrieval-semantic-search "Direct link to Relevance-Based Retrieval (Semantic Search)") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Principle:** Memories semantically similar to the query are most relevant. ### When to Use[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use-1 "Direct link to When to Use") * Answering specific questions * Looking up facts * When the query clearly states what's needed ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation-1 "Direct link to Implementation") def get_relevant_memories(user_id: str, query: str, limit: int = 10, threshold: float = 0.5): """Retrieve memories semantically similar to query.""" results = memory.search( query=query, filters={"user_id": user_id}, limit=limit ) # Filter by similarity threshold return [r for r in results['results'] if r['score'] >= threshold] **Example:** query = "What is the project deadline?"results = get_relevant_memories("alex", query) **Output:** [ {"memory": "Phoenix project deadline is Jan 31", "score": 0.91}, {"memory": "Sprint planning discussed Phoenix blockers", "score": 0.67}, {"memory": "Phoenix project uses auth-service", "score": 0.54},] ### How It Works[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#how-it-works "Direct link to How It Works") 1. Query is embedded into a vector 2. Memory vectors are compared using cosine similarity 3. Highest-similarity memories are returned Query: "What is the project deadline?" ↓ (embedding) [0.1, 0.8, 0.3, 0.5, ...]Compare to each memory vector:Memory A: "Phoenix project deadline is Jan 31" [0.15, 0.75, 0.28, 0.48, ...] → similarity: 0.91Memory B: "Alex prefers morning meetings" [0.7, 0.1, 0.9, 0.2, ...] → similarity: 0.23 ### Limitations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#limitations-1 "Direct link to Limitations") * May miss memories with different wording * Semantic similarity isn't always relevance * Requires good embeddings Entity-Based Retrieval[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#entity-based-retrieval "Direct link to Entity-Based Retrieval") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Principle:** When a specific entity (person, project, concept) is mentioned, retrieve all memories involving that entity. ### When to Use[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use-2 "Direct link to When to Use") * User mentions a name or project * Building context around a specific topic * Answering "What do you know about X?" ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation-2 "Direct link to Implementation") def get_entity_memories(user_id: str, entity: str, limit: int = 10): """Retrieve memories involving a specific entity.""" # Option 1: Semantic search for entity name return memory.search( query=entity, filters={"user_id": user_id}, limit=limit ) # Option 2: Metadata filter (if entities are tagged) # return memory.search( # query="", # filters={ # "user_id": user_id, # "entities": {"contains": entity} # } # ) ### Entity Extraction[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#entity-extraction "Direct link to Entity Extraction") First, extract entities from the user's message: def extract_entities(message: str) -> list: """Extract named entities from message.""" # Using LLM for extraction prompt = f""" Extract named entities from this message. Categories: people, projects, companies, products, dates Message: {message} Return as JSON: {{"entities": [{{"name": "...", "type": "..."}}]}} """ result = llm.generate(prompt) return json.loads(result)['entities'] **Example:** message = "What's happening with the Phoenix project that Alex is working on?"entities = extract_entities(message)# [{"name": "Phoenix", "type": "project"}, {"name": "Alex", "type": "person"}]# Retrieve memories for each entityfor entity in entities: memories = get_entity_memories(user_id, entity['name']) ### Limitations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#limitations-2 "Direct link to Limitations") * Entity extraction can miss or misidentify entities * Same entity may have multiple names ("Phoenix", "the project") * Requires entity disambiguation Hybrid Retrieval[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#hybrid-retrieval "Direct link to Hybrid Retrieval") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Principle:** Combine multiple strategies to get the best of all worlds. ### When to Use[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use-3 "Direct link to When to Use") * Most production scenarios * When query type is unclear * When maximum recall is important ### Implementation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation-3 "Direct link to Implementation") def hybrid_retrieve(user_id: str, query: str, limit: int = 10): """Combine recency, relevance, and entity retrieval.""" # 1. Get relevant memories (semantic) relevant = memory.search( query=query, filters={"user_id": user_id}, limit=limit )['results'] # 2. Extract entities and get entity-based memories entities = extract_entities(query) entity_memories = [] for entity in entities: entity_results = memory.search( query=entity['name'], filters={"user_id": user_id}, limit=5 )['results'] entity_memories.extend(entity_results) # 3. Get recent memories recent = memory.search( query="", filters={"user_id": user_id}, sort_by="created_at", sort_order="desc", limit=5 )['results'] # 4. Combine and deduplicate all_memories = deduplicate(relevant + entity_memories + recent) # 5. Re-score with weighted formula scored = [] for mem in all_memories: score = calculate_hybrid_score( memory=mem, query=query, semantic_weight=0.5, recency_weight=0.3, entity_weight=0.2 ) scored.append({**mem, "hybrid_score": score}) # 6. Sort by hybrid score and return top scored.sort(key=lambda x: x['hybrid_score'], reverse=True) return scored[:limit] ### Weighted Scoring[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#weighted-scoring "Direct link to Weighted Scoring") def calculate_hybrid_score(memory, query, semantic_weight, recency_weight, entity_weight): """Calculate weighted hybrid score.""" # Semantic similarity (from vector search) semantic = memory.get('score', 0) # Recency decay days_old = (datetime.now() - memory['created_at']).days recency = math.exp(-days_old / 30) # Entity match boost entities = extract_entities(query) entity_match = 1.0 if any(e['name'].lower() in memory['memory'].lower() for e in entities) else 0.0 return ( semantic_weight * semantic + recency_weight * recency + entity_weight * entity_match ) The Token Budget Constraint[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#the-token-budget-constraint "Direct link to The Token Budget Constraint") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You can retrieve 100 relevant memories, but you can only inject so many into the context window. ### The Problem[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#the-problem "Direct link to The Problem") Context Window: 128,000 tokensSystem Prompt: 2,000 tokensUser Message: 500 tokensAvailable for Memories: 4,000 tokens (budget decision)Reserved for Response: 2,000 tokensRemaining for Conversation: 119,500 tokensAverage Memory Size: 200 tokensMaximum Memories: 4,000 / 200 = 20 memories ### Token-Aware Selection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#token-aware-selection "Direct link to Token-Aware Selection") def select_within_budget(memories: list, token_budget: int) -> list: """Select highest-scored memories that fit within token budget.""" selected = [] current_tokens = 0 # Memories should already be sorted by score for memory in memories: mem_tokens = estimate_tokens(memory['memory']) if current_tokens + mem_tokens <= token_budget: selected.append(memory) current_tokens += mem_tokens else: # Check if we can fit a smaller memory continue return selecteddef estimate_tokens(text: str) -> int: """Estimate token count. Rule of thumb: ~4 chars per token.""" return len(text) // 4 ### Budget Allocation Strategies[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#budget-allocation-strategies "Direct link to Budget Allocation Strategies") **Fixed Budget:** # Always use same budgetmemory_budget = 4000 # tokens **Dynamic Budget:** # Adjust based on query complexityif is_simple_query(query): memory_budget = 2000elif is_complex_query(query): memory_budget = 6000else: memory_budget = 4000 **Priority Tiers:** # Allocate budget across tiersbudgets = { "critical": 1500, # Must-have context "relevant": 2000, # Highly relevant "background": 500 # Nice to have}for tier, budget in budgets.items(): tier_memories = get_memories_by_priority(tier) selected.extend(select_within_budget(tier_memories, budget)) Strategy Selection Guide[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#strategy-selection-guide "Direct link to Strategy Selection Guide") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Scenario | Recommended Strategy | | --- | --- | | "What did we just discuss?" | Recency | | "What's the project deadline?" | Relevance | | "Tell me about Alex" | Entity | | "Help me with the next step" | Hybrid (recency + relevance) | | "What do you know about me?" | Entity (user) + Relevance | | Complex, unclear query | Hybrid | Complete Retrieval Pipeline[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#complete-retrieval-pipeline "Direct link to Complete Retrieval Pipeline") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- class MemoryRetriever: def __init__(self, memory_client, token_budget: int = 4000): self.memory = memory_client self.token_budget = token_budget def retrieve(self, user_id: str, query: str) -> list: """Full retrieval pipeline.""" # 1. Classify query type query_type = self.classify_query(query) # 2. Choose strategy if query_type == "recent": memories = self.recency_retrieve(user_id, query) elif query_type == "specific": memories = self.relevance_retrieve(user_id, query) elif query_type == "entity": memories = self.entity_retrieve(user_id, query) else: memories = self.hybrid_retrieve(user_id, query) # 3. Apply token budget selected = self.select_within_budget(memories, self.token_budget) # 4. Format for injection return self.format_for_context(selected) def format_for_context(self, memories: list) -> str: """Format memories for prompt injection.""" if not memories: return "" lines = ["Relevant context from previous interactions:"] for mem in memories: lines.append(f"- {mem['memory']}") return "\n".join(lines) Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use these prompts to practice retrieval strategy design with Claude or your preferred AI assistant. ### Prompt 1: Hybrid Strategy Design[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#prompt-1-hybrid-strategy-design "Direct link to Prompt 1: Hybrid Strategy Design") Design a hybrid retrieval strategy for a project management agent.The agent needs to answer questions like:- "What's the status of Project Alpha?" (entity + relevance)- "What did we discuss yesterday?" (recency)- "Who's responsible for the API?" (entity + relevance)- "What are my priorities?" (entity + recency + relevance)For each query type:1. Which retrieval strategies should be combined?2. What weights would you assign to each strategy?3. Show example memories that would be retrievedThen design a query classifier that determines which strategy to use. **What you're learning:** Different queries need different strategy mixes. A hybrid approach with query classification adapts to user needs dynamically. ### Prompt 2: Token Budget Optimization[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#prompt-2-token-budget-optimization "Direct link to Prompt 2: Token Budget Optimization") Your agent has 20 relevant memories but only 2000 tokens of budget.Memories (with token counts):1. "User prefers Python over JavaScript" (50 tokens) - relevance: 0.922. "Project Phoenix deadline Jan 31" (60 tokens) - relevance: 0.883. "User completed 5 tasks yesterday" (80 tokens) - relevance: 0.754. "Phoenix uses microservices architecture" (150 tokens) - relevance: 0.855. "User had meeting with Sarah about auth" (100 tokens) - relevance: 0.70... (15 more memories)Design an algorithm that:1. Maximizes total relevance within the token budget2. Ensures diversity (not all memories about same topic)3. Handles the case where a high-relevance memory is also very longShow which memories would be selected and the reasoning. **What you're learning:** Token budget management is an optimization problem. Sometimes a shorter, less relevant memory is better than a longer, more relevant one. Diversity constraints prevent over-concentration on one topic. ### Prompt 3: Entity-Based Retrieval System[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#prompt-3-entity-based-retrieval-system "Direct link to Prompt 3: Entity-Based Retrieval System") Build an entity-based retrieval system for a customer relationship management (CRM) agent.Entity types:- Companies (customers)- Contacts (people at companies)- Deals (sales opportunities)- Products- Issues (support tickets)Design:1. How to extract these entities from user queries2. How to link related entities (e.g., Contact → works at → Company)3. A retrieval strategy that follows entity relationshipsExample query: "What's happening with the Acme deal?"Show the retrieval steps:- Entity extraction- Relationship traversal- Memory retrieval- Final context assembly **What you're learning:** Entity-based retrieval shines in domain-specific agents. A CRM agent needs to understand relationships—when you ask about a deal, you probably also need context about the company and contacts involved. Checking access... * [The Retrieval Challenge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#the-retrieval-challenge) * [Recency-Based Retrieval](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#recency-based-retrieval) * [When to Use](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation) * [Limitations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#limitations) * [Relevance-Based Retrieval (Semantic Search)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#relevance-based-retrieval-semantic-search) * [When to Use](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use-1) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation-1) * [How It Works](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#how-it-works) * [Limitations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#limitations-1) * [Entity-Based Retrieval](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#entity-based-retrieval) * [When to Use](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use-2) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation-2) * [Entity Extraction](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#entity-extraction) * [Limitations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#limitations-2) * [Hybrid Retrieval](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#hybrid-retrieval) * [When to Use](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#when-to-use-3) * [Implementation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#implementation-3) * [Weighted Scoring](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#weighted-scoring) * [The Token Budget Constraint](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#the-token-budget-constraint) * [The Problem](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#the-problem) * [Token-Aware Selection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#token-aware-selection) * [Budget Allocation Strategies](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#budget-allocation-strategies) * [Strategy Selection Guide](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#strategy-selection-guide) * [Complete Retrieval Pipeline](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#complete-retrieval-pipeline) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#try-with-ai) * [Prompt 1: Hybrid Strategy Design](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#prompt-1-hybrid-strategy-design) * [Prompt 2: Token Budget Optimization](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#prompt-2-token-budget-optimization) * [Prompt 3: Entity-Based Retrieval System](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/augmented-memory/memory-retrieval-strategies#prompt-3-entity-based-retrieval-system) --- # Session Lifecycle Management | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Users close the chat. When they return, conversation is lost. They type "What did we discuss?" and your agent says "I'm sorry, I don't have that context." Professional ChatKit servers manage session lifecycle—create, resume, timeout, cleanup—so conversations persist across browser refreshes, device switches, and days between messages. By the end of this lesson, you'll implement complete session lifecycle management and extract the patterns into a reusable `session-lifecycle` skill. * * * The Problem: Stateless Browsers, Stateful Conversations[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#the-problem-stateless-browsers-stateful-conversations "Direct link to The Problem: Stateless Browsers, Stateful Conversations") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **REST APIs are stateless** (each request independent). **Conversations are stateful** (context builds over time). **Users are unreliable**: * Close browser tabs mid-conversation * Return hours later expecting continuity * Switch from desktop to mobile * Open multiple simultaneous conversations **Without session management**: User: "Create a task: Review Q4 budget"Agent: ✓ Created[User closes tab][User reopens 2 hours later]User: "Mark it complete"Agent: "Mark what complete?" ❌ Lost context **With session management**: User: "Create a task: Review Q4 budget"Agent: ✓ Created [Session: abc123][User closes tab][User reopens 2 hours later, session resumed]User: "Mark it complete"Agent: ✓ Marked complete [Session: abc123, resumed] Sessions bridge the gap between stateless HTTP and stateful conversations. * * * Session Lifecycle States[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#session-lifecycle-states "Direct link to Session Lifecycle States") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ┌─────────────┐│ CREATED │──────┐└─────────────┘ │ │ │ │ First │ Timeout │ Message │ (30 min inactive) ▼ │┌─────────────┐ ││ ACTIVE │◄─────┤└─────────────┘ │ │ │ │ Resumed │ │ │ ▼ ▼┌─────────────┐ ┌─────────────┐│ RESUMED │ │ EXPIRED │└─────────────┘ └─────────────┘ │ │ │ Cleanup │ Cleanup │ │ ▼ ▼┌─────────────────────────────┐│ CLEANED UP │└─────────────────────────────┘ **States**: * **CREATED**: Session initialized, no messages yet * **ACTIVE**: User actively sending messages * **RESUMED**: Session loaded from store after inactivity * **EXPIRED**: Timeout reached, session marked for cleanup * **CLEANED UP**: Session data removed from store * * * RequestContext: The Session Container[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#requestcontext-the-session-container "Direct link to RequestContext: The Session Container") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ChatKit provides `context` parameter in `respond()` containing session data: from chatkit.server import ChatKitServerclass MyServer(ChatKitServer): async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, # ← RequestContext with session info ) -> AsyncIterator[ThreadStreamEvent]: # Extract session data user_id = context.user_id session_id = context.session_id metadata = context.metadata **RequestContext fields**: * `user_id`: Unique user identifier (from authentication) * `session_id`: Unique session identifier (generated or resumed) * `metadata`: Custom key-value data (user preferences, feature flags) * `timestamp`: Request timestamp **Why this matters**: RequestContext provides user isolation (multi-tenant) and session continuity. * * * Pattern 1: Session Creation[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-1-session-creation "Direct link to Pattern 1: Session Creation") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **When**: First message from new user or new conversation. **Implementation**: from datetime import datetime, timezonefrom typing import Dict, Anyfrom chatkit.server import ChatKitServer, ThreadMetadata, UserMessageItemfrom chatkit.types import ThreadStreamEventclass SessionServer(ChatKitServer): def __init__(self): super().__init__() self.sessions: Dict[str, Dict[str, Any]] = {} # In-memory session store async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: session_id = context.session_id user_id = context.user_id # Create session if new if session_id not in self.sessions: await self.create_session(session_id, user_id) # Process message async for event in self.process_message(thread, input, context): yield event async def create_session(self, session_id: str, user_id: str) -> None: """Create new session with initial state.""" self.sessions[session_id] = { "session_id": session_id, "user_id": user_id, "created_at": datetime.now(timezone.utc), "last_activity": datetime.now(timezone.utc), "state": "CREATED", "message_count": 0, "metadata": {}, } print(f"✓ Created session {session_id} for user {user_id}") **Output**: ✓ Created session abc123 for user alice@example.com * * * Pattern 2: Session Resumption[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-2-session-resumption "Direct link to Pattern 2: Session Resumption") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **When**: User returns after browser refresh or inactivity. **Implementation**: async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any,) -> AsyncIterator[ThreadStreamEvent]: session_id = context.session_id user_id = context.user_id # Create or resume session if session_id not in self.sessions: await self.create_session(session_id, user_id) else: await self.resume_session(session_id) # Process message async for event in self.process_message(thread, input, context): yield eventasync def resume_session(self, session_id: str) -> None: """Resume existing session, update activity timestamp.""" session = self.sessions[session_id] session["last_activity"] = datetime.now(timezone.utc) session["state"] = "RESUMED" print(f"✓ Resumed session {session_id} (inactive for {self.get_inactive_duration(session)})")def get_inactive_duration(self, session: Dict[str, Any]) -> str: """Calculate time since last activity.""" now = datetime.now(timezone.utc) delta = now - session["last_activity"] if delta.total_seconds() < 60: return f"{int(delta.total_seconds())}s" elif delta.total_seconds() < 3600: return f"{int(delta.total_seconds() / 60)}m" else: return f"{int(delta.total_seconds() / 3600)}h" **Output**: ✓ Resumed session abc123 (inactive for 15m) **Why this matters**: Resumption restores conversation context without forcing users to repeat themselves. * * * Pattern 3: Timeout Handling[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-3-timeout-handling "Direct link to Pattern 3: Timeout Handling") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **When**: Session inactive beyond configured threshold. **Timeout Strategies**: | Strategy | Threshold | Use Case | | --- | --- | --- | | **Aggressive** | 5-10 minutes | High-security environments, limited resources | | **Standard** | 30-60 minutes | Most applications, balance between UX and resources | | **Lenient** | 2-4 hours | Casual apps, infrequent usage patterns | | **Persistent** | 24+ hours | Research tools, long-form conversations | **Implementation**: from datetime import timedeltaclass SessionServer(ChatKitServer): SESSION_TIMEOUT = timedelta(minutes=30) # Configurable threshold async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any, ) -> AsyncIterator[ThreadStreamEvent]: session_id = context.session_id # Check timeout before resuming if session_id in self.sessions: if self.is_session_expired(session_id): await self.expire_session(session_id) # Create new session await self.create_session(session_id, context.user_id) else: await self.resume_session(session_id) else: await self.create_session(session_id, context.user_id) async for event in self.process_message(thread, input, context): yield event def is_session_expired(self, session_id: str) -> bool: """Check if session exceeded timeout threshold.""" session = self.sessions[session_id] now = datetime.now(timezone.utc) inactive_duration = now - session["last_activity"] return inactive_duration > self.SESSION_TIMEOUT async def expire_session(self, session_id: str) -> None: """Mark session as expired (ready for cleanup).""" session = self.sessions[session_id] session["state"] = "EXPIRED" session["expired_at"] = datetime.now(timezone.utc) print(f"⚠ Session {session_id} expired after {self.SESSION_TIMEOUT}") **Output**: ⚠ Session abc123 expired after 0:30:00✓ Created session abc123 for user alice@example.com **Why this matters**: Timeout prevents stale sessions from consuming server resources. * * * Pattern 4: Session Cleanup[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-4-session-cleanup "Direct link to Pattern 4: Session Cleanup") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **When**: Session expired or explicitly closed by user. **Cleanup Strategies**: | Strategy | When | Resources Freed | | --- | --- | --- | | **Immediate** | On expiration | Memory, cache entries | | **Deferred** | Background task (hourly) | Batch cleanup, lower overhead | | **Lazy** | On next access attempt | Simple, no cron jobs needed | **Implementation (Immediate Cleanup)**: async def cleanup_session(self, session_id: str) -> None: """Remove session data from store.""" if session_id in self.sessions: session = self.sessions[session_id] # Log final state print(f"🧹 Cleaning up session {session_id}:") print(f" - Messages: {session['message_count']}") print(f" - Duration: {datetime.now(timezone.utc) - session['created_at']}") print(f" - State: {session['state']}") # Remove from store del self.sessions[session_id] print(f"✓ Session {session_id} cleaned up")async def expire_session(self, session_id: str) -> None: """Mark session expired and clean up immediately.""" session = self.sessions[session_id] session["state"] = "EXPIRED" session["expired_at"] = datetime.now(timezone.utc) # Immediate cleanup await self.cleanup_session(session_id) **Output**: 🧹 Cleaning up session abc123: - Messages: 5 - Duration: 0:45:23 - State: EXPIRED✓ Session abc123 cleaned up * * * Pattern 5: State Persistence[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-5-state-persistence "Direct link to Pattern 5: State Persistence") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Problem**: In-memory sessions lost on server restart. **Solution**: Persist to database or Redis. **Implementation (SQLModel + PostgreSQL)**: from sqlmodel import Field, Session, SQLModel, create_engine, selectfrom datetime import datetime, timezoneclass SessionModel(SQLModel, table=True): __tablename__ = "chatkit_sessions" session_id: str = Field(primary_key=True) user_id: str created_at: datetime last_activity: datetime state: str message_count: int metadata: dict = Field(default_factory=dict, sa_column_kwargs={"type_": "JSONB"})class PersistentSessionServer(ChatKitServer): def __init__(self, db_url: str): super().__init__() self.engine = create_engine(db_url) SQLModel.metadata.create_all(self.engine) async def create_session(self, session_id: str, user_id: str) -> None: """Create session in database.""" with Session(self.engine) as session: db_session = SessionModel( session_id=session_id, user_id=user_id, created_at=datetime.now(timezone.utc), last_activity=datetime.now(timezone.utc), state="CREATED", message_count=0, metadata={}, ) session.add(db_session) session.commit() print(f"✓ Created session {session_id} (persisted to DB)") async def resume_session(self, session_id: str) -> None: """Resume session, update last_activity in database.""" with Session(self.engine) as session: statement = select(SessionModel).where(SessionModel.session_id == session_id) db_session = session.exec(statement).first() if db_session: db_session.last_activity = datetime.now(timezone.utc) db_session.state = "RESUMED" session.add(db_session) session.commit() print(f"✓ Resumed session {session_id} from DB") **Output**: ✓ Created session abc123 (persisted to DB)✓ Resumed session abc123 from DB **Why this matters**: Persistence enables session recovery after server crashes or deployments. * * * Pattern 6: Concurrent Session Management[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-6-concurrent-session-management "Direct link to Pattern 6: Concurrent Session Management") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Problem**: User opens chat in multiple browser tabs. **Strategies**: | Strategy | Behavior | Use Case | | --- | --- | --- | | **Single Session** | All tabs share one session | Most apps (simplest) | | **Per-Tab Session** | Each tab independent session | Power users, complex workflows | | **Active Tab Wins** | Most recent tab takes control | Prevent conflicting updates | **Implementation (Single Session Strategy)**: async def respond( self, thread: ThreadMetadata, input: UserMessageItem | None, context: Any,) -> AsyncIterator[ThreadStreamEvent]: session_id = context.session_id user_id = context.user_id # Lock session for this request async with self.session_lock(session_id): if session_id not in self.sessions: await self.create_session(session_id, user_id) else: await self.resume_session(session_id) # Update activity timestamp self.sessions[session_id]["last_activity"] = datetime.now(timezone.utc) self.sessions[session_id]["message_count"] += 1 async for event in self.process_message(thread, input, context): yield event **Why this matters**: Concurrent access without locking can corrupt session state. * * * Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#safety-note "Direct link to Safety Note") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ⚠ **Session cleanup prevents memory leaks in production**. Without cleanup: * Long-running servers accumulate expired sessions * Memory usage grows unbounded * Server performance degrades over time **Best practice**: Combine timeout expiration with periodic background cleanup (e.g., hourly cron job removing sessions older than 24 hours). * * * Extract Your session-lifecycle Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#extract-your-session-lifecycle-skill "Direct link to Extract Your session-lifecycle Skill") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've implemented session creation, resumption, timeout handling, and cleanup. These patterns will recur in every stateful ChatKit application. Time to harvest this intelligence. **Layer 3 trigger**: Pattern recurs 2+ times across projects → Create reusable intelligence ### Step: Extract the Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#step-extract-the-skill "Direct link to Step: Extract the Skill") Copy and paste this prompt: Using your skill-creator skill, study the session lifecycle implementationcode I just wrote in this lesson and extract it into a reusablesession-lifecycle skill.Encapsulate all 6 patterns: session creation, resumption, timeout handling,cleanup, state persistence, and concurrent session management. Claude will: 1. Review the 6 session patterns from this lesson 2. Use `skill-creator` to generate the skill structure 3. Extract patterns: create, resume, timeout, cleanup, persistence, concurrency 4. Create reference docs with state diagrams and cleanup strategies 5. Follow canonical format from building-chat-interfaces Your skill appears at `.claude/skills/session-lifecycle/`. * * * Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Setup[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#setup "Direct link to Setup") Open `claude-code-skills-lab` terminal with your chatkit-server skill loaded. ### Prompts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#prompts "Direct link to Prompts") **Prompt 1: Extend Timeout Strategy** Using my session-lifecycle skill, add a dynamic timeout strategy that:- Uses 5 minutes for high-security mode- Uses 30 minutes for standard mode- Uses 4 hours for casual mode- Reads mode from session metadataShow implementation with switching logic. **What you're learning**: Configurable session policies based on context. * * * **Prompt 2: Session Analytics** Using my session-lifecycle skill, add analytics tracking:- Count active sessions- Calculate average session duration- Track sessions per user- Export metrics as JSONShow implementation with metrics endpoint. **What you're learning**: Observability patterns for session management. * * * **Prompt 3: Background Cleanup Job** Using my session-lifecycle skill, create a background cleanup task that:- Runs every hour- Removes sessions expired >24 hours ago- Logs cleanup statistics- Uses asyncio for schedulingShow implementation with asyncio task. **What you're learning**: Deferred cleanup strategy with scheduled tasks. * [The Problem: Stateless Browsers, Stateful Conversations](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#the-problem-stateless-browsers-stateful-conversations) * [Session Lifecycle States](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#session-lifecycle-states) * [RequestContext: The Session Container](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#requestcontext-the-session-container) * [Pattern 1: Session Creation](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-1-session-creation) * [Pattern 2: Session Resumption](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-2-session-resumption) * [Pattern 3: Timeout Handling](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-3-timeout-handling) * [Pattern 4: Session Cleanup](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-4-session-cleanup) * [Pattern 5: State Persistence](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-5-state-persistence) * [Pattern 6: Concurrent Session Management](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#pattern-6-concurrent-session-management) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#safety-note) * [Extract Your session-lifecycle Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#extract-your-session-lifecycle-skill) * [Step: Extract the Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#step-extract-the-skill) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#try-with-ai) * [Setup](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#setup) * [Prompts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/chatkit-server/session-lifecycle-management#prompts) --- # Configuring MCP Clients | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C You've understood MCP architecture: the Host-Client-Server model, transport layers, and the three primitives (tools, resources, prompts). Now comes the practical question: **How do you actually connect MCP servers to your tools so they can use them?** Configuration is where theory meets reality. And it's deceptively straightforward—JSON files that define which servers to launch, how to authenticate, what parameters to pass. But getting it wrong breaks everything silently. A misconfigured environment variable means your agent can't access GitHub. A wrong path means the server won't start. A typo means no error message—the tool simply fails to load. This lesson covers configuration across every major platform: Claude Code (where you probably started), Claude Desktop (the standalone application), VS Code (for developers integrating MCP), and programmatic clients (when you build agents in Python or Node.js). By the end, you'll understand how to set up production-ready MCP configurations that work reliably. Configuration Locations: Where MCP Looks for Servers[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-locations-where-mcp-looks-for-servers "Direct link to Configuration Locations: Where MCP Looks for Servers") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- MCP clients search for configuration in predictable locations: ### Claude Code (Project-Level Configuration)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#claude-code-project-level-configuration "Direct link to Claude Code (Project-Level Configuration)") **Path**: `.claude/settings.json` (in your project root) This is **project-specific**. Each project can have different MCP servers configured. { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/project"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } }} **Advantages**: * Project isolation (different projects, different servers) * Committed to git (team visibility) * Workspace-specific setup **Use when**: * Building agents for specific projects * Team collaboration (configuration in version control) * Different projects need different server sets ### Claude Desktop (Global Configuration)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#claude-desktop-global-configuration "Direct link to Claude Desktop (Global Configuration)") **Path**: * **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` * **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` * **Linux**: `~/.config/Claude/claude_desktop_config.json` This is **global**—applies to all Claude Desktop sessions. { "mcpServers": { "github": { "command": "uvx", "args": ["--python", "3.12", "mcp-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }, "filesystem": { "command": "uvx", "args": ["mcp-filesystem", "/Users/yourname/Documents"] } }} **Advantages**: * Always available across sessions * Centralized configuration * Perfect for frequently-used servers **Use when**: * Tools you use in every conversation * Personal assistant-style MCP servers * Development tools needed across projects ### VS Code / Cursor (User Settings)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#vs-code--cursor-user-settings "Direct link to VS Code / Cursor (User Settings)") **Path**: * **User settings** (global): Via VS Code settings UI or `settings.json` * **Workspace settings** (project): `.vscode/settings.json` VS Code integrates MCP through extensions. Configuration looks slightly different: { "mcpServers": { "github": { "command": "uvx", "args": ["mcp-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } }} Settings are managed through VS Code's configuration system, not directly as files (though the underlying format is JSON). **Use when**: * IDE-integrated MCP usage * VS Code extensions that speak MCP * Cursor IDE integration Configuration Structure: The mcpServers Object[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-structure-the-mcpservers-object "Direct link to Configuration Structure: The mcpServers Object") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Every MCP client configuration follows the same structure: { "mcpServers": { "server-name": { "command": "executable or npx", "args": ["arg1", "arg2", "arg3"], "env": { "VAR_NAME": "${VAR_NAME}" } } }} Let's break down each field: ### `command`: How to Start the Server[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#command-how-to-start-the-server "Direct link to command-how-to-start-the-server") **For local executables**: "command": "/usr/local/bin/mcp-server" **For npm packages** (via npx): "command": "npx","args": ["-y", "@modelcontextprotocol/server-github"] **For Python packages** (via uvx): "command": "uvx","args": ["--python", "3.12", "mcp-github"] The client executes this command as a subprocess and communicates via stdio. ### `args`: Command Arguments[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#args-command-arguments "Direct link to args-command-arguments") Arguments passed to the command. For npm packages: "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/serve"] Breaking this down: * `-y`: Auto-answer yes to npm prompts * `@modelcontextprotocol/server-filesystem`: Package name * `/path/to/serve`: Path the server will expose **For HTTP servers** (if you're pointing to remote MCP): "args": ["--host", "localhost", "--port", "8000"] ### `env`: Environment Variables[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#env-environment-variables "Direct link to env-environment-variables") Passes environment variables to the server process. **This is critical for secrets**. "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}", "DATABASE_URL": "${DATABASE_URL}", "API_KEY": "${API_KEY}"} The `${VAR_NAME}` syntax tells the client: "Look up VAR\_NAME from the environment where you're running and substitute it here." This protects you from accidentally committing secrets to git: * Configuration file contains `${GITHUB_TOKEN}` (safe) * Your shell environment has `GITHUB_TOKEN=ghp_xxxx` (secret) * Client substitutes at runtime Environment Variables: Where Secrets Come From[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#environment-variables-where-secrets-come-from "Direct link to Environment Variables: Where Secrets Come From") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Configuration references environment variables. The client retrieves their actual values from: 1. **Shell environment** (what you have in your terminal) 2. **.env files** (if the client supports them) 3. **System variables** (machine-wide settings) ### .env Files (For Development)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#env-files-for-development "Direct link to .env Files (For Development)") Create a `.env` file in your project root: GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxDATABASE_URL=postgresql://user:pass@localhost/dbAPI_KEY=sk-1234567890abcdef **Don't commit this to git.** Add to `.gitignore`: .env.env.local Some MCP clients automatically load `.env` files. Others require you to `source` the file: export $(cat .env | xargs) Then start Claude Code or your application. ### System Environment Variables[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#system-environment-variables "Direct link to System Environment Variables") For production, set variables in your deployment environment: # In your shellexport GITHUB_TOKEN=ghp_xxxx# Or in deployment platform (Docker, Kubernetes, etc.)ENV GITHUB_TOKEN=ghp_xxxx The client reads these at runtime. Common Configuration Patterns[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#common-configuration-patterns "Direct link to Common Configuration Patterns") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Pattern 1: Filesystem Server (Local Files)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-1-filesystem-server-local-files "Direct link to Pattern 1: Filesystem Server (Local Files)") { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/project"] } }} **What it does**: Exposes files in `/Users/you/project` as MCP resources. The agent can read files, search directories. **No environment variables needed**: Filesystem doesn't require secrets. ### Pattern 2: GitHub Server (Requires Authentication)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-2-github-server-requires-authentication "Direct link to Pattern 2: GitHub Server (Requires Authentication)") { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } }} **What it does**: Enables MCP tools for creating issues, fetching PRs, managing workflows. **Requires**: GitHub Personal Access Token with appropriate permissions. **Get token**: 1. Go to GitHub Settings → Developer Settings → Personal Access Tokens 2. Create token with `repo`, `read:org` scopes 3. Store in `.env`: `GITHUB_TOKEN=ghp_xxxx` ### Pattern 3: Database Server (PostgreSQL, MySQL, etc.)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-3-database-server-postgresql-mysql-etc "Direct link to Pattern 3: Database Server (PostgreSQL, MySQL, etc.)") { "mcpServers": { "database": { "command": "uvx", "args": ["mcp-database"], "env": { "DATABASE_URL": "${DATABASE_URL}" } } }} **What it does**: Exposes database tables as MCP resources; provides tools for querying and updates. **Requires**: Database connection string. **Format**: `postgresql://user:password@host:port/database` **Get connection string**: # For local developmentDATABASE_URL=postgresql://user:pass@localhost:5432/mydb# For managed services (Supabase, Railway, etc.)DATABASE_URL=postgresql://user:xxx@server.replication.supabase.co:5432/postgres?sslmode=require ### Pattern 4: Multiple Servers in One Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-4-multiple-servers-in-one-configuration "Direct link to Pattern 4: Multiple Servers in One Configuration") { "mcpServers": { "filesystem": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "/project"] }, "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }, "database": { "command": "uvx", "args": ["mcp-database"], "env": { "DATABASE_URL": "${DATABASE_URL}" } } }} **Important**: Each server is independent. The client launches one subprocess per server. When you ask the agent to execute a tool: 1. Agent requests tool `create_issue` (GitHub tool) 2. Client routes to GitHub server 3. GitHub server executes; returns result The agent doesn't know (or care) about database server—it's not involved unless you explicitly request a database tool. Configuration for Programmatic Clients (Python)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-for-programmatic-clients-python "Direct link to Configuration for Programmatic Clients (Python)") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- When building agents programmatically, you configure MCP differently—through code instead of JSON files. ### Using Claude SDK with Stdio Transport[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#using-claude-sdk-with-stdio-transport "Direct link to Using Claude SDK with Stdio Transport") from mcp import ClientSession, StdioClientTransport# Configure servertransport = StdioClientTransport( command="npx", args=[ "-y", "@modelcontextprotocol/server-github" ], env={ "GITHUB_TOKEN": os.getenv("GITHUB_TOKEN") })# Create sessionasync with ClientSession(transport) as session: await session.initialize() tools = await session.list_tools() print(tools) **Key differences from JSON**: * Direct Python code instead of configuration files * Environment variables via `os.getenv()` instead of `${VAR_NAME}` * Explicit session initialization ### Using Anthropic SDK with MCP Integration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#using-anthropic-sdk-with-mcp-integration "Direct link to Using Anthropic SDK with MCP Integration") from anthropic import Anthropicclient = Anthropic()# Configure MCP serversmcp_config = { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": os.getenv("GITHUB_TOKEN") } } }}# Pass to clientresponse = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, tools=[...], # MCP tools discovered via mcp_config messages=[...]) Configuration Comparison: Which to Use When[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-comparison-which-to-use-when "Direct link to Configuration Comparison: Which to Use When") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Scenario | Location | Format | Best For | | --- | --- | --- | --- | | **Claude Code** | `.claude/settings.json` | JSON | Project-specific agents, team workflows | | **Claude Desktop** | `~/.config/Claude/claude_desktop_config.json` (Linux) | JSON | Personal use, always-available tools | | **VS Code** | `.vscode/settings.json` or user settings | JSON | IDE integration, extension support | | **Python Agent** | Programmatic (code) | Python objects | Custom agent applications, deployment | | **Node.js Agent** | Programmatic (code) | JavaScript objects | Backend services, serverless functions | **Decision framework**: * **Team collaboration**: Use Claude Code with `.claude/settings.json` (in git) * **Personal tools**: Use Claude Desktop global config * **IDE workflows**: Use VS Code settings integration * **Production agents**: Programmatic configuration in Python/Node Security Best Practices for Production[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#security-best-practices-for-production "Direct link to Security Best Practices for Production") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Never Hardcode Secrets[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#never-hardcode-secrets "Direct link to Never Hardcode Secrets") **WRONG**: { "env": { "GITHUB_TOKEN": "ghp_xxxxxxxxxxxxxxxxxxxx" }} Commit this to git, and it's compromised forever. GitHub revokes it, but the token is in your git history permanently. **RIGHT**: { "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" }} Let the deployment platform provide the secret. ### Use Secret Management for Production[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#use-secret-management-for-production "Direct link to Use Secret Management for Production") **For Kubernetes**: apiVersion: v1kind: Secretmetadata: name: mcp-secretsdata: GITHUB_TOKEN: base64encodedtoken---apiVersion: v1kind: ConfigMapmetadata: name: mcp-configdata: settings.json: | { "mcpServers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } } } **For Docker**: FROM node:20COPY . /appWORKDIR /app# Read secret from build arg (passed at build time, not stored in image)ARG GITHUB_TOKENENV GITHUB_TOKEN=${GITHUB_TOKEN}CMD ["node", "agent.js"] Build with: `docker build --build-arg GITHUB_TOKEN=ghp_xxx .` ### Rotate Credentials Regularly[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#rotate-credentials-regularly "Direct link to Rotate Credentials Regularly") MCP agents running for weeks/months should use credentials that can be rotated without restarting: * GitHub: Regularly create new Personal Access Tokens * Database: Use database users with limited permissions * API keys: Implement key rotation policies ### Principle of Least Privilege[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#principle-of-least-privilege "Direct link to Principle of Least Privilege") Only grant credentials the minimum permissions needed: * **GitHub**: If MCP server only reads PRs, create token with read-only scopes * **Database**: Create database user with SELECT on specific tables (not admin) * **API Keys**: Use API keys with rate limits, specific resource access This limits damage if credentials leak. Troubleshooting Configuration Issues[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#troubleshooting-configuration-issues "Direct link to Troubleshooting Configuration Issues") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Server Won't Start[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#server-wont-start "Direct link to Server Won't Start") **Error**: `Command not found: mcp-server` **Causes**: * Executable doesn't exist at path * npx/uvx not installed * Path is wrong **Fix**: # Verify command works outside confignpx -y @modelcontextprotocol/server-github# If it works manually, config path might be relative# Use absolute paths instead"command": "/usr/local/bin/mcp-server"# Or use npx/uvx (which finds packages)"command": "npx" ### Server Starts But No Tools Available[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#server-starts-but-no-tools-available "Direct link to Server Starts But No Tools Available") **Cause**: Environment variable not found **Error**: `GITHUB_TOKEN not provided` **Fix**: # Set variable before startingexport GITHUB_TOKEN=ghp_xxxx# Then start Claude Code# Or add to .env and source itsource .env ### Configuration Not Being Loaded[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-not-being-loaded "Direct link to Configuration Not Being Loaded") **Cause**: Wrong file path or location **Check**: * Claude Code: `./.claude/settings.json` in project root * Claude Desktop (Linux): `~/.config/Claude/claude_desktop_config.json` * VS Code: Settings UI or `.vscode/settings.json` **Verify**: # List Claude Code configcat .claude/settings.json | jq .mcpServers# Check Claude Desktop location (macOS)cat ~/Library/Application\ Support/Claude/claude_desktop_config.json | jq . ### JSON Syntax Error[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#json-syntax-error "Direct link to JSON Syntax Error") **Error**: `Invalid JSON in configuration` **Check**: * Missing commas between properties * Trailing commas * Unquoted keys **Validate**: # Use jq to validatecat .claude/settings.json | jq .# Or use online JSON validator Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#try-with-ai "Direct link to Try With AI") --------------------------------------------------------------------------------------------------------------------------------------------------------------------- Use your AI companion to explore MCP configuration scenarios and troubleshoot setup issues. ### Safety Note on Configuration Security[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#safety-note-on-configuration-security "Direct link to Safety Note on Configuration Security") When configuring MCP servers, remember: * **Never commit `.env` files** or configuration with hardcoded secrets to git * **Use environment variable references** (`${VAR_NAME}`) in configuration files * **Rotate credentials** regularly—if a secret leaks, create new credentials immediately * **Principle of least privilege**—only grant each MCP server the permissions it needs * **Verify SSL/TLS** when connecting to remote MCP servers—don't accept unencrypted connections in production Configuration security determines whether your agents can be compromised. Get this right. ### Prompt 1: Analyze Your Current Configuration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-1-analyze-your-current-configuration "Direct link to Prompt 1: Analyze Your Current Configuration") **Setup**: Understanding what's already configured in your environment I have MCP servers configured in Claude Code. Help me understand what I have set up.Here's my .claude/settings.json:[paste your configuration]For each server in my configuration:1. Identify what tool this server provides2. What permissions or secrets does it need?3. How would I know if this server is working correctly?4. What would break if I misconfigured the environment variables? **What you're learning**: Analyzing existing configuration to understand server requirements and potential failure modes. ### Prompt 2: Configure a New Server[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-2-configure-a-new-server "Direct link to Prompt 2: Configure a New Server") **Setup**: Adding GitHub server to an existing setup I want to add a GitHub MCP server to my configuration. I already have:[paste your current mcpServers object]I want to be able to create issues and fetch pull requests. Help me:1. Write the JSON configuration for a GitHub MCP server2. Tell me what permissions I need on my GitHub token3. Explain how to securely provide the token (I should never commit it to git)4. Walk me through testing to verify it's workingHere's what my environment looks like:- I use Claude Code- My project is at /Users/me/project- I have a GitHub Personal Access Token already created **What you're learning**: Configuring new servers, managing environment variables securely, and validating setup through testing. ### Prompt 3: Troubleshoot Configuration Problems[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-3-troubleshoot-configuration-problems "Direct link to Prompt 3: Troubleshoot Configuration Problems") **Setup**: Debugging a configuration that isn't working I configured a database MCP server but when I try to query the database,I get this error: "[ENOENT] DATABASE_URL environment variable not set"Here's my configuration:[paste your configuration]Here's what I did to set up the environment:1. Created a .env file with DATABASE_URL=postgresql://...2. Didn't source the .env file3. Started Claude Code directlyWhat's wrong? Walk me through:1. Why this error is happening2. How environment variable substitution works3. The correct way to provide DATABASE_URL4. How to debug this in the future **What you're learning**: Understanding why configuration fails, how to diagnose environment variable issues, and implementing proper secret management. ### Prompt 4: Design Configuration for Team Collaboration[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-4-design-configuration-for-team-collaboration "Direct link to Prompt 4: Design Configuration for Team Collaboration") **Setup**: Setting up MCP for a team project I'm setting up MCP configuration for my team's project. Here's what we need:- 5 developers, each with their own GitHub token- Shared project codebase (all on same git repo)- Database server accessible from cloud (not local)- Filesystem access to /project directoryFor this scenario:1. Should we use .claude/settings.json or global Claude Desktop config?2. How do we handle different GITHUB_TOKEN values per developer?3. Should DATABASE_URL go in .env, environment variables, or somewhere else?4. What do we commit to git? What do we keep local?Help me design a configuration strategy that works for a team. **What you're learning**: Scaling configuration from personal use to team collaboration; understanding security implications of shared configuration. Checking access... * [Configuration Locations: Where MCP Looks for Servers](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-locations-where-mcp-looks-for-servers) * [Claude Code (Project-Level Configuration)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#claude-code-project-level-configuration) * [Claude Desktop (Global Configuration)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#claude-desktop-global-configuration) * [VS Code / Cursor (User Settings)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#vs-code--cursor-user-settings) * [Configuration Structure: The mcpServers Object](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-structure-the-mcpservers-object) * [`command`: How to Start the Server](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#command-how-to-start-the-server) * [`args`: Command Arguments](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#args-command-arguments) * [`env`: Environment Variables](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#env-environment-variables) * [Environment Variables: Where Secrets Come From](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#environment-variables-where-secrets-come-from) * [.env Files (For Development)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#env-files-for-development) * [System Environment Variables](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#system-environment-variables) * [Common Configuration Patterns](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#common-configuration-patterns) * [Pattern 1: Filesystem Server (Local Files)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-1-filesystem-server-local-files) * [Pattern 2: GitHub Server (Requires Authentication)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-2-github-server-requires-authentication) * [Pattern 3: Database Server (PostgreSQL, MySQL, etc.)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-3-database-server-postgresql-mysql-etc) * [Pattern 4: Multiple Servers in One Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#pattern-4-multiple-servers-in-one-configuration) * [Configuration for Programmatic Clients (Python)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-for-programmatic-clients-python) * [Using Claude SDK with Stdio Transport](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#using-claude-sdk-with-stdio-transport) * [Using Anthropic SDK with MCP Integration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#using-anthropic-sdk-with-mcp-integration) * [Configuration Comparison: Which to Use When](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-comparison-which-to-use-when) * [Security Best Practices for Production](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#security-best-practices-for-production) * [Never Hardcode Secrets](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#never-hardcode-secrets) * [Use Secret Management for Production](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#use-secret-management-for-production) * [Rotate Credentials Regularly](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#rotate-credentials-regularly) * [Principle of Least Privilege](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#principle-of-least-privilege) * [Troubleshooting Configuration Issues](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#troubleshooting-configuration-issues) * [Server Won't Start](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#server-wont-start) * [Server Starts But No Tools Available](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#server-starts-but-no-tools-available) * [Configuration Not Being Loaded](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#configuration-not-being-loaded) * [JSON Syntax Error](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#json-syntax-error) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#try-with-ai) * [Safety Note on Configuration Security](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#safety-note-on-configuration-security) * [Prompt 1: Analyze Your Current Configuration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-1-analyze-your-current-configuration) * [Prompt 2: Configure a New Server](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-2-configure-a-new-server) * [Prompt 3: Troubleshoot Configuration Problems](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-3-troubleshoot-configuration-problems) * [Prompt 4: Design Configuration for Team Collaboration](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/mcp-fundamentals/configuring-mcp-clients#prompt-4-design-configuration-for-team-collaboration) --- # StreamableHTTP Transport: Remote MCP Over HTTP | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C The Remote Deployment Challenge[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#the-remote-deployment-challenge "Direct link to The Remote Deployment Challenge") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You've built a powerful MCP server using FastMCP that processes documents, analyzes code, and handles system operations. It works perfectly on your local machine—stdio transport handles all communication seamlessly. But now you face a real-world problem: you need to deploy this server to a remote machine. Your clients (Claude instances, browser agents, mobile apps) are distributed across the network. Stdio is a local pipe—it only works when the server and client run on the same machine. You need **remote MCP**. The problem isn't just "how do we send messages over the network?" Any HTTP client can do that. The problem is that **HTTP is fundamentally asymmetric**: * **Client → Server**: Easy. Client sends request, server responds. * **Server → Client**: Hard. HTTP has no native way for servers to initiate messages. But MCP requires bidirectional communication. Consider these scenarios: **Server → Client (required by MCP):** * Progress notifications ("Processing file 3 of 10...") * Logging messages ("Tool execution took 2.3 seconds") * Sampling requests ("I need another file to make this decision") **Without server-initiated messaging**, these workflows fail. The client must continuously poll ("Are you done yet? Are you done yet?"), wasting bandwidth and adding latency. **StreamableHTTP solves this.** It transforms HTTP from request-response into a streaming protocol using Server-Sent Events (SSE), enabling true bidirectional MCP communication over the network. How StreamableHTTP Works: The Dual-Connection Pattern[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#how-streamablehttp-works-the-dual-connection-pattern "Direct link to How StreamableHTTP Works: The Dual-Connection Pattern") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ StreamableHTTP operates on a principle: **one connection for client→server requests, another for server→client messages.** ### Architecture Overview[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#architecture-overview "Direct link to Architecture Overview") Client Application ↓ ├─ POST /sse/message → Server (client sends requests) │ └─ GET /sse/{sessionId} (server streams responses + notifications) ↓ Server-Sent Events (long-lived HTTP response) ↓ Receives: Tool results, progress updates, logging, sampling requests ### The Session ID: Binding Connections Together[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#the-session-id-binding-connections-together "Direct link to The Session ID: Binding Connections Together") Here's the key insight: HTTP connections are stateless. When your client makes a POST request and a separate GET request for SSE, **the server needs to know these connections belong to the same session**. That's what the **session ID** does. **Session Lifecycle:** 1. **Initialize**: Client sends `Initialize` request → Server responds with `mcp-session-id` header 2. **Acknowledge**: Client sends `Initialized` notification **with that session ID** 3. **Stream**: Client opens SSE connection with `?session_id=[ID]` 4. **Communicate**: Client sends requests via POST, server streams responses via SSE 5. **Cleanup**: Either side closes connections, server cleans up session ### Example Request Flow[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#example-request-flow "Direct link to Example Request Flow") CLIENT SERVER──────────────────────────────────────────1. POST /initialize {"protocol": "mcp"} ────────────────────→ ← ─ ─ ─ ─ 200 OK + header: mcp-session-id: abc1232. POST /notifications { "jsonrpc": "2.0", "method": "notifications/initialized", "params": {"sessionId": "abc123"} } ────────────────────→ ← ─ ─ ─ ─ 200 OK3. GET /sse?session_id=abc123 (keeps connection open) ────────────────────→ (connection stays open - server can stream to it)4. POST /request { "jsonrpc": "2.0", "method": "resources/read", "params": {...}, "sessionId": "abc123" } ────────────────────→ ← ─ ─ ─ ─ 200 OK (acknowledge receipt)5. Server streams through SSE connection (from step 3): data: {"jsonrpc": "2.0", "result": {...}} ← ─ ─ ─ ─ (data arrives through GET connection) Problem 1: Multiple SSE Streams for Different Contexts[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#problem-1-multiple-sse-streams-for-different-contexts "Direct link to Problem 1: Multiple SSE Streams for Different Contexts") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- MCP servers often handle multiple tool calls simultaneously. Each tool might need: * Sampling requests (asking the client for more data) * Progress notifications (keeping the client informed) * Logging (debugging what happened) **Naive approach**: Send everything through one SSE stream. **Problem**: Ordering becomes ambiguous. If Tool A sends a sampling request and Tool B logs a message, how does the client know which tool issued which request? **StreamableHTTP solution**: Create **tool-specific SSE connections**. ### Primary SSE Connection[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#primary-sse-connection "Direct link to Primary SSE Connection") Carries: * General notifications * Tool results (for simple tools) * Errors @app.get("/sse/{session_id}")async def primary_sse(session_id: str): """Primary stream for all non-tool-specific messages""" session = session_store.get(session_id) if not session: return {"error": "invalid session"} async def event_generator(): while session.is_active: message = await session.queue.get() yield f"data: {json.dumps(message)}\n\n" return StreamingResponse(event_generator(), media_type="text/event-stream") ### Tool-Specific SSE Connections[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#tool-specific-sse-connections "Direct link to Tool-Specific SSE Connections") Created dynamically for tools that need interactive communication: @app.get("/sse/{session_id}/tool/{tool_name}")async def tool_sse(session_id: str, tool_name: str): """Tool-specific stream for sampling requests and progress""" session = session_store.get(session_id) if not session: return {"error": "invalid session"} tool_queue = session.get_tool_queue(tool_name) async def event_generator(): while session.is_active: message = await tool_queue.get() yield f"data: {json.dumps(message)}\n\n" return StreamingResponse(event_generator(), media_type="text/event-stream") | Connection Type | Purpose | Lifecycle | | --- | --- | --- | | **Primary SSE** | General notifications, tool results, errors | Stays open entire session | | **Tool-Specific SSE** | Sampling requests, progress for one tool | Created per tool call, closes when tool completes | Problem 2: Session ID Routing Across Requests[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#problem-2-session-id-routing-across-requests "Direct link to Problem 2: Session ID Routing Across Requests") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Every client request must include the session ID so the server routes it correctly: @app.post("/messages")async def handle_message(request: Request): body = await request.json() session_id = body.get("sessionId") # Extract from request session = session_store.get(session_id) if not session: return {"error": "Invalid session ID"} # Route to correct session's handler result = await session.handle_request(body) return result **Client responsibility**: Include `sessionId` in every POST request body. // Example client codeasync function makeRequest(method, params) { const response = await fetch(`${SERVER_URL}/messages`, { method: "POST", body: JSON.stringify({ jsonrpc: "2.0", method: method, params: params, sessionId: SESSION_ID // Critical }) }); return response.json();} **Why this matters**: Without explicit session ID routing, requests from different clients could collide. Session IDs ensure isolation. Implementing StreamableHTTP with FastMCP[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#implementing-streamablehttp-with-fastmcp "Direct link to Implementing StreamableHTTP with FastMCP") ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### Step 1: Specification (The Deployment Pattern)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-1-specification-the-deployment-pattern "Direct link to Step 1: Specification (The Deployment Pattern)") Before code, clarify what we're building: ## StreamableHTTP MCP Server Specification**Intent**: Enable remote MCP clients to connect to FastMCP servers over HTTPwith streaming support for progress notifications and sampling requests.**Success Criteria**:- ✅ Client can initialize and establish session- ✅ Client can send requests (via POST) and receive results- ✅ Server can push notifications (via SSE) without client polling- ✅ Server can request samples from client (sampling pattern)- ✅ Multiple clients can maintain independent sessions- ✅ Session timeout cleans up stale connections**Constraints**:- Use FastMCP's StreamableHTTP mixin- Configure both primary and tool-specific SSE streams- Validate session IDs on every request- Implement connection heartbeat to detect dead clients ### Step 2: Configure FastMCP for HTTP[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-2-configure-fastmcp-for-http "Direct link to Step 2: Configure FastMCP for HTTP") FastMCP provides a `StreamableHTTP` transport that handles the complexity: from fastmcp import FastMCPfrom fastmcp.server import StreamableHTTPimport json# Initialize FastMCP with HTTP transportmcp = FastMCP( "document-analyzer", transports=[ StreamableHTTP( host="0.0.0.0", port=8000, session_timeout=300, # 5 minutes enable_tool_sse=True, # Create tool-specific SSE streams ) ])# Your tools work exactly as before@mcp.tool()async def analyze_document(file_path: str) -> str: """Analyze document and return insights""" # Tool implementation unchanged return f"Analysis of {file_path}: ..." **Key configuration options:** | Option | Purpose | | --- | --- | | `host` | Bind to all interfaces (0.0.0.0) for remote access | | `port` | HTTP listening port | | `session_timeout` | Minutes before stale sessions are cleaned | | `enable_tool_sse` | Create separate SSE streams per tool call | ### Step 3: Handle Session Initialization[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-3-handle-session-initialization "Direct link to Step 3: Handle Session Initialization") StreamableHTTP handles session creation automatically, but you should understand the flow: # FastMCP automatically provides these endpoints:# POST /initialize → Creates session, returns mcp-session-id# POST /messages → Client sends requests# GET /sse/{session_id} → Primary SSE stream# GET /sse/{session_id}/tool/{tool_name} → Tool-specific SSE When a client connects: 1. Client POST to `/initialize` with MCP protocol version 2. Server creates new session, returns `mcp-session-id` header 3. Client POST to `/notifications/initialized` with session ID 4. Client opens SSE connection with `GET /sse/{session_id}` 5. Server is now ready to stream responses ### Step 4: Sending Notifications Through SSE[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-4-sending-notifications-through-sse "Direct link to Step 4: Sending Notifications Through SSE") Your tools can send notifications that stream to clients: @mcp.tool()async def long_running_operation(data: str) -> str: """Demonstrate progress notifications""" # Process in steps, notify after each for i in range(5): await mcp.emit_notification({ "type": "progress", "current": i + 1, "total": 5, "message": f"Processing batch {i+1}/5..." }) # Do work... await asyncio.sleep(1) return f"Completed processing {data}" The notification arrives on the client's SSE stream in real-time—no polling required. ### Step 5: Sampling Requests (Interactive Tools)[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-5-sampling-requests-interactive-tools "Direct link to Step 5: Sampling Requests (Interactive Tools)") Tools can request additional context from the client: @mcp.tool()async def intelligent_analysis(file_path: str, context_needed: bool = False) -> str: """Tool that can request additional context""" # Initial analysis result = analyze(file_path) # If we need more context, ask client if context_needed: context = await mcp.request_sample({ "type": "context_request", "message": "I need additional context. What domain is this file from?", "options": ["Finance", "Healthcare", "E-commerce", "Other"] }) # Use provided context result = refine_with_context(result, context) return result The client sees this request stream through SSE and can respond interactively. Real-World Deployment: Connecting Remote Claude[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#real-world-deployment-connecting-remote-claude "Direct link to Real-World Deployment: Connecting Remote Claude") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Once your StreamableHTTP server is running: # Server running on remote host at api.example.com:8000uvicorn your_server:app --host 0.0.0.0 --port 8000 Connect it to Claude: from anthropic import Anthropicclient = Anthropic()# Configure MCP connection to remote serverclient.configure_mcp_connection( transport="streamable-http", url="http://api.example.com:8000",)# Now Claude can call your remote toolsresponse = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, tools=client.get_tools(), # Discovers tools from remote MCP messages=[ { "role": "user", "content": "Analyze the document at /data/report.pdf and provide insights" } ]) Claude automatically: 1. Initializes the session 2. Opens the SSE stream 3. Calls your tools 4. Receives streaming responses and notifications 5. Cleans up when done The Session ID Pattern: Why It Matters[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#the-session-id-pattern-why-it-matters "Direct link to The Session ID Pattern: Why It Matters") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This pattern is critical for production systems: **Single-Client Scenario (Appears to Work Without Session ID):** Client 1├─ POST /initialize → Session ABC├─ GET /sse/ABC└─ POST /messages with SessionId: ABC ✓ **Multi-Client Scenario (Fails Without Session ID):** Client 1 Client 2├─ POST /initialize → Session ABC ├─ POST /initialize → Session DEF├─ GET /sse/ABC ├─ GET /sse/DEF├─ POST /messages (SessionId: ABC) ├─ POST /messages (SessionId: DEF)│ "Convert video1" │ "Convert video2"│ │├ Request routes to Session ABC ✓ ├ Request routes to Session DEF ✓└─ Result streams to correct client└─ Result streams to correct client Without explicit session IDs, requests get mixed up. With them, isolation is guaranteed. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **Setup**: You're designing a document processing service that connects to remote Claude instances. You need to decide between keeping everything on one connection vs using the dual-stream pattern. **Part 1: Understanding the Constraint** Ask AI: I'm building an MCP server that processes documents and provides real-time progress updates.The client needs to:1. Send a request ("Process this file")2. Receive progress notifications ("50% complete...")3. Get the final resultWhy can't I just use HTTP POST for all communication? What makes this pattern differentfrom a normal request-response API? Explain the technical constraint and why SSE is the solution. **What you're learning**: The fundamental difference between request-response (HTTP's native model) and bidirectional streaming (what MCP requires). Understanding this gap is crucial for designing remote agents. You're discovering why the architecture exists. **Part 2: Designing the Connection Pattern** Based on the explanation above, ask AI: Given the HTTP limitation, why does StreamableHTTP use TWO separate connections(one POST for requests, one GET for SSE responses) instead of trying to createone true bidirectional connection?What are the advantages of this split architecture? What are potential downsides?Which problems does this solve, and which remain? **What you're learning**: The tradeoff between simplicity (one connection for everything) and clarity (separate concerns). This informs architectural decisions you'll make when deploying real systems. You're refining the pattern based on HTTP's actual constraints. **Part 3: Session Management Under Load** Now consider a real deployment scenario: I'm deploying my MCP server to handle 50 concurrent clients. Each client:- Initializes a session (gets unique session ID)- Opens an SSE stream with that ID- Sends multiple requests (each includes the session ID)- Can disconnect/reconnect at any timeGiven this scenario, what could go wrong with session ID management?How would I detect and clean up abandoned sessions? What happens if a clientcrashes without closing their SSE stream? How would I prevent session ID collisions? **What you're learning**: Production resilience patterns. Session timeouts, heartbeats, and cleanup logic aren't optional—they prevent resource leaks in long-running systems. You're thinking like a distributed systems engineer considering failure modes before they happen. * * * Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#reflect-on-your-skill "Direct link to Reflect on Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- You built an `mcp-server` skill in Lesson 0. Test and improve it based on what you learned. ### Test Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#test-your-skill "Direct link to Test Your Skill") Using my mcp-server skill, configure a server for remote deployment with StreamableHTTP.Does my skill include guidance on session ID management, SSE connection patterns, and the dual-connection architecture? ### Identify Gaps[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#identify-gaps "Direct link to Identify Gaps") Ask yourself: * Did my skill include StreamableHTTP configuration and session lifecycle management? * Did it explain the difference between primary SSE and tool-specific SSE streams? ### Improve Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#improve-your-skill "Direct link to Improve Your Skill") If you found gaps: My mcp-server skill is missing remote deployment patterns with StreamableHTTP.Update it to include StreamableHTTP transport configuration, session ID routing, SSE connection management, and the architectural patterns for bidirectional HTTP communication. * * * Checking access... * [The Remote Deployment Challenge](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#the-remote-deployment-challenge) * [How StreamableHTTP Works: The Dual-Connection Pattern](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#how-streamablehttp-works-the-dual-connection-pattern) * [Architecture Overview](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#architecture-overview) * [The Session ID: Binding Connections Together](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#the-session-id-binding-connections-together) * [Example Request Flow](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#example-request-flow) * [Problem 1: Multiple SSE Streams for Different Contexts](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#problem-1-multiple-sse-streams-for-different-contexts) * [Primary SSE Connection](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#primary-sse-connection) * [Tool-Specific SSE Connections](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#tool-specific-sse-connections) * [Problem 2: Session ID Routing Across Requests](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#problem-2-session-id-routing-across-requests) * [Implementing StreamableHTTP with FastMCP](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#implementing-streamablehttp-with-fastmcp) * [Step 1: Specification (The Deployment Pattern)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-1-specification-the-deployment-pattern) * [Step 2: Configure FastMCP for HTTP](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-2-configure-fastmcp-for-http) * [Step 3: Handle Session Initialization](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-3-handle-session-initialization) * [Step 4: Sending Notifications Through SSE](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-4-sending-notifications-through-sse) * [Step 5: Sampling Requests (Interactive Tools)](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#step-5-sampling-requests-interactive-tools) * [Real-World Deployment: Connecting Remote Claude](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#real-world-deployment-connecting-remote-claude) * [The Session ID Pattern: Why It Matters](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#the-session-id-pattern-why-it-matters) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#try-with-ai) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#reflect-on-your-skill) * [Test Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#test-your-skill) * [Identify Gaps](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#identify-gaps) * [Improve Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/custom-mcp-servers/streamable-http-transport#improve-your-skill) --- # Systematic Error Analysis | Agent Factory [Skip to main content](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#__docusaurus_skipToContent_fallback) On this page Teach Me Sign in to access Teach Me mode Ask Sign in to ask questions Copy as MarkdownCtrl+⇧+C Your agent fails 30% of the time. You have graders that detect failures. But which failures matter most? Which component is causing them? And what should you fix first? Andrew Ng identified what separates effective AI builders from the rest: "Less experienced teams spend a lot of time building and probably much less time analyzing." When your agent fails, the instinct is to start fixing immediately. You see an error, you have a theory about the cause, and you start coding. But that theory is often wrong. Consider this scenario: Your web research agent produces poor results. You think "the LLM prompt must be unclear," so you spend two days rewriting prompts. Performance improves slightly. Then you actually count the errors and discover: 45% of failures came from the web search component returning low-quality sources. The prompt was fine. You fixed the wrong component. Analysis time is not overhead. Analysis time is investment that prevents wasted effort. The developers who spend 30 minutes counting errors before fixing anything outperform those who spend 30 hours fixing the wrong component. The Build-Analyze Loop[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#the-build-analyze-loop "Direct link to The Build-Analyze Loop") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Effective agent development follows a cycle: Build agent v1 | vRun evaluations | vAnalyze errors <-- This is where most developers skip | vIdentify which component failed most | vFix that specific component | vRun evaluations again | v(Repeat until quality is acceptable) The trap is moving directly from "run evaluations" to "fix something." Without analysis, you're guessing which component to fix. Even smart guesses are often wrong because agent failures have multiple causes, and the one that comes to mind first is not necessarily the one causing most failures. Traces and Spans: The Vocabulary of Analysis[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#traces-and-spans-the-vocabulary-of-analysis "Direct link to Traces and Spans: The Vocabulary of Analysis") -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Before analyzing errors, you need language to describe what you're examining. **Trace**: All intermediate outputs from a single agent run. When your agent processes a query, it might call an LLM, search the web, select sources, and generate a response. The trace captures everything: every input, every intermediate output, every decision point. **Span**: The output of a single step within a trace. If the trace is the complete journey, each span is one leg of that journey. The web search span contains query terms and results. The source selection span contains which sources were chosen and why. The final output span contains the response shown to the user. Think of debugging a flight delay. The trace is the entire trip: airport, plane, connections, destination. Each span is one segment: "check-in took 45 minutes," "boarding delayed 30 minutes," "connection missed due to late arrival." To fix delays, you examine spans to find which segment caused the problem. **The relationship in practice**: | Term | What It Contains | When to Examine | | --- | --- | --- | | **Trace** | Complete agent run from input to output | Understanding overall failure pattern | | **Span** | Single step's input, processing, and output | Identifying which component failed | When your agent produces a poor response, you examine the trace to see what happened. You look at each span to identify where the problem originated. Maybe the web search span returned only blog posts when you needed academic sources. Maybe the source selection span picked the wrong articles from good search results. Maybe the output generation span ignored the best sources. This vocabulary matters because error analysis requires precision. "The agent failed" doesn't tell you what to fix. "The source selection span selected low-quality blogs despite high-quality academic sources being available in the search results" tells you exactly what to fix. The Spreadsheet Method[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#the-spreadsheet-method "Direct link to The Spreadsheet Method") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The most effective error analysis tool is embarrassingly simple: a spreadsheet. When your agent fails, don't just note "it failed." Break down the trace into spans and mark which spans produced problematic outputs. After analyzing 20-30 failures, patterns emerge from the data rather than from your intuition. **The structure**: | Case | Input | Search Terms | Search Results | Source Selection | Final Output | Error Location | | --- | --- | --- | --- | --- | --- | --- | | Q1 | "Black holes" | OK | Too many blogs | Based on poor input | Missing key points | Search Results | | Q2 | "Seattle rent" | OK | OK | Missed relevant blog | OK | Source Selection | | Q3 | "Robot farming" | Too generic | Poor results | Based on poor input | Missing company | Search Terms | | Q4 | "Climate 2024" | OK | OK | OK | OK | None | | Q5 | "AI agents" | OK | Outdated sources | Based on poor input | Stale information | Search Results | After completing the table, count the Error Location column: | Error Location | Count | Percentage | | --- | --- | --- | | Search Results | 2 | 40% | | Source Selection | 1 | 20% | | Search Terms | 1 | 20% | | None (success) | 1 | 20% | | **Total** | **5** | **100%** | Now you have data instead of intuition. If you had guessed, you might have focused on the LLM prompt generating final outputs. The data shows the real problem: the web search component returns low-quality results 40% of the time. Fix that first. Why Counting Beats Intuition[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#why-counting-beats-intuition "Direct link to Why Counting Beats Intuition") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your intuition about error causes is biased. You remember the dramatic failures, not the common ones. You remember the failures you understand, not the ones that confuse you. You remember recent failures more than older ones. Andrew Ng describes teams that "spend a lot of time building and probably much less time analyzing." These teams chase the last error they saw. They fix the failure that frustrated them most. They work on the component they understand best, regardless of whether it's the component that fails most. Counting corrects for these biases: | Bias | How It Misleads | How Counting Corrects | | --- | --- | --- | | **Availability bias** | Recent or dramatic errors feel more common | All errors counted equally | | **Confirmation bias** | You notice errors matching your theory | Data shows all patterns | | **Expertise bias** | You focus on components you understand | Data reveals unfamiliar problems | | **Anchoring bias** | First error you saw dominates thinking | Percentages show true distribution | When someone says "I think the routing is the problem," ask them: "What percentage of errors come from routing?" If they can't answer with data, they're guessing. Maybe they're right, maybe not. The spreadsheet tells you for certain. The Prioritization Formula[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#the-prioritization-formula "Direct link to The Prioritization Formula") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Knowing which component fails most doesn't automatically tell you what to fix first. A component might fail frequently but be nearly impossible to fix. Another might fail rarely but be trivially fixable. The prioritization formula balances both factors: **Priority = Frequency x Feasibility** | Factor | What It Measures | Scale | | --- | --- | --- | | **Frequency** | How often this error type occurs | 0-100% of failures | | **Feasibility** | How easily you can fix it | 0 (impossible) to 1 (trivial) | **Example prioritization**: | Error Type | Frequency | Feasibility | Priority Score | | --- | --- | --- | --- | | Search returns blogs | 45% | 0.8 (add filters) | 36 | | Routing misclassifies | 25% | 0.4 (needs new training data) | 10 | | Output format wrong | 15% | 0.9 (fix template) | 13.5 | | Source timeout | 10% | 0.3 (infrastructure change) | 3 | | Unknown errors | 5% | 0.2 (need investigation) | 1 | By priority score, you should fix in this order: 1. Search returns blogs (score: 36) 2. Output format wrong (score: 13.5) 3. Routing misclassifies (score: 10) 4. Source timeout (score: 3) 5. Unknown errors (score: 1) The routing errors occur more often than format errors (25% vs 15%), but format errors are much easier to fix (0.9 vs 0.4). Fix the format first, then tackle routing. **How to estimate feasibility**: | Feasibility | Description | Example | | --- | --- | --- | | 0.9 - 1.0 | Trivial fix, minutes to implement | Change a config value, fix a regex | | 0.7 - 0.8 | Clear solution, hours to implement | Add search filters, update prompt | | 0.5 - 0.6 | Known approach, days to implement | Retrain classifier, add new component | | 0.3 - 0.4 | Uncertain solution, research needed | New architecture, external dependency | | 0.0 - 0.2 | Unknown cause, investigation required | Intermittent failures, vendor issues | Generating Error Analysis Data[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#generating-error-analysis-data "Direct link to Generating Error Analysis Data") --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- While spreadsheets work for small-scale analysis, Python code systematizes the process for larger evaluations: import csvfrom dataclasses import dataclassfrom collections import Counter@dataclassclass AnalyzedCase: """A single analyzed test case with error attribution.""" case_id: str input_query: str search_terms_ok: bool search_results_ok: bool source_selection_ok: bool output_ok: bool error_location: str # Which span failed, or "None"def analyze_trace(case_id: str, trace: dict) -> AnalyzedCase: """ Analyze a single trace and determine which span caused failure. Args: case_id: Identifier for this test case trace: Dictionary containing span outputs from agent run Returns: AnalyzedCase with error attribution """ # Extract span quality from trace # Your actual logic depends on trace structure search_terms_ok = trace.get("search_terms_quality", "OK") == "OK" search_results_ok = trace.get("search_results_quality", "OK") == "OK" source_selection_ok = trace.get("source_selection_quality", "OK") == "OK" output_ok = trace.get("output_quality", "OK") == "OK" # Attribute error to first failing span if not search_terms_ok: error_location = "Search Terms" elif not search_results_ok: error_location = "Search Results" elif not source_selection_ok: error_location = "Source Selection" elif not output_ok: error_location = "Output" else: error_location = "None" return AnalyzedCase( case_id=case_id, input_query=trace.get("input_query", ""), search_terms_ok=search_terms_ok, search_results_ok=search_results_ok, source_selection_ok=source_selection_ok, output_ok=output_ok, error_location=error_location )def generate_error_report(cases: list[AnalyzedCase]) -> dict: """ Generate error analysis report from analyzed cases. Returns: Dictionary with error counts, percentages, and recommendations """ error_counts = Counter(case.error_location for case in cases) total = len(cases) # Calculate percentages error_percentages = { location: (count / total) * 100 for location, count in error_counts.items() } # Sort by frequency sorted_errors = sorted( error_percentages.items(), key=lambda x: x[1], reverse=True ) # Identify highest-frequency error (excluding "None" which means success) non_success = [(loc, pct) for loc, pct in sorted_errors if loc != "None"] recommendation = non_success[0][0] if non_success else "No errors detected" return { "total_cases": total, "error_counts": dict(error_counts), "error_percentages": error_percentages, "sorted_by_frequency": sorted_errors, "recommendation": f"Focus on: {recommendation} ({error_percentages.get(recommendation, 0):.1f}% of failures)" }def export_to_csv(cases: list[AnalyzedCase], filename: str) -> None: """Export analyzed cases to CSV for spreadsheet analysis.""" with open(filename, 'w', newline='') as f: writer = csv.writer(f) writer.writerow([ "Case ID", "Input Query", "Search Terms OK", "Search Results OK", "Source Selection OK", "Output OK", "Error Location" ]) for case in cases: writer.writerow([ case.case_id, case.input_query, "OK" if case.search_terms_ok else "ERROR", "OK" if case.search_results_ok else "ERROR", "OK" if case.source_selection_ok else "ERROR", "OK" if case.output_ok else "ERROR", case.error_location ]) **Output:** # Example usage with sample datatraces = [ {"case_id": "Q1", "input_query": "Black holes", "search_results_quality": "ERROR", "output_quality": "ERROR"}, {"case_id": "Q2", "input_query": "Seattle rent", "source_selection_quality": "ERROR"}, {"case_id": "Q3", "input_query": "Robot farming", "search_terms_quality": "ERROR", "search_results_quality": "ERROR"}, {"case_id": "Q4", "input_query": "Climate 2024"}, # All OK {"case_id": "Q5", "input_query": "AI agents", "search_results_quality": "ERROR"}]# Analyze all tracescases = [analyze_trace(t["case_id"], t) for t in traces]# Generate reportreport = generate_error_report(cases)print(f"Total cases analyzed: {report['total_cases']}")print(f"\nError distribution:")for location, percentage in report['sorted_by_frequency']: print(f" {location}: {percentage:.1f}%")print(f"\n{report['recommendation']}")# Export for spreadsheet reviewexport_to_csv(cases, "error_analysis.csv")print(f"\nExported to error_analysis.csv") Total cases analyzed: 5Error distribution: Search Results: 40.0% None: 20.0% Source Selection: 20.0% Search Terms: 20.0%Focus on: Search Results (40.0% of failures)Exported to error_analysis.csv Exercise: Analyze Task API Agent Failures[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#exercise-analyze-task-api-agent-failures "Direct link to Exercise: Analyze Task API Agent Failures") ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Your Task API agent helps users manage their tasks. You've run 20 test cases and collected traces. Here are the summarized results: | Case | Input | Intent Recognition | Database Query | Task Matching | Response Generation | Overall | | --- | --- | --- | --- | --- | --- | --- | | 1 | "What's next?" | OK | OK | OK | OK | PASS | | 2 | "Add dentist call" | OK | OK | N/A | OK | PASS | | 3 | "Overdue tasks" | OK | ERROR (timeout) | N/A | ERROR | FAIL | | 4 | "Mark groceries done" | ERROR (ambiguous) | N/A | N/A | ERROR | FAIL | | 5 | "High priority only" | OK | OK | OK | OK | PASS | | 6 | "Delete old tasks" | OK | ERROR (timeout) | N/A | ERROR | FAIL | | 7 | "What did I finish?" | OK | OK | OK | OK | PASS | | 8 | "Tasks for Monday" | OK | OK | OK | OK | PASS | | 9 | "Add buy milk" | ERROR (truncated) | N/A | N/A | ERROR | FAIL | | 10 | "Show everything" | OK | OK | OK | OK | PASS | | 11 | "Complete the report" | ERROR (ambiguous) | N/A | N/A | ERROR | FAIL | | 12 | "Overdue items" | OK | OK | OK | OK | PASS | | 13 | "What's urgent?" | OK | OK | OK | OK | PASS | | 14 | "Delete meeting prep" | OK | OK | ERROR (matched wrong task) | ERROR | FAIL | | 15 | "Add task call mom" | OK | OK | N/A | OK | PASS | | 16 | "Show today's tasks" | OK | ERROR (timeout) | N/A | ERROR | FAIL | | 17 | "Mark done: email" | OK | OK | ERROR (matched wrong task) | ERROR | FAIL | | 18 | "What needs attention?" | OK | OK | OK | OK | PASS | | 19 | "Clear completed" | OK | OK | OK | OK | PASS | | 20 | "Add reminder walk dog" | ERROR (truncated) | N/A | N/A | ERROR | FAIL | **Your task**: 1. Count errors by component 2. Calculate percentages 3. Apply the prioritization formula (estimate feasibility yourself) 4. Determine which component to fix first **Work through this before reading the solution below.** * * * **Solution**: **Step 1: Count errors by component** | Component | Error Count | | --- | --- | | Intent Recognition | 4 (Cases 4, 9, 11, 20) | | Database Query | 3 (Cases 3, 6, 16) | | Task Matching | 2 (Cases 14, 17) | | Response Generation | 0 (errors are downstream from other failures) | | None (success) | 11 | **Step 2: Calculate percentages** (of 20 total, 9 failures) | Component | Percentage of Failures | | --- | --- | | Intent Recognition | 4/9 = 44% | | Database Query | 3/9 = 33% | | Task Matching | 2/9 = 22% | **Step 3: Apply prioritization formula** | Error Type | Frequency | Feasibility | Priority Score | | --- | --- | --- | --- | | Intent Recognition (ambiguous/truncated) | 44% | 0.6 (prompt engineering, add examples) | 26.4 | | Database Query (timeout) | 33% | 0.4 (infrastructure, caching, indices) | 13.2 | | Task Matching (wrong task) | 22% | 0.7 (improve matching algorithm) | 15.4 | **Step 4: Recommended fix order** 1. **Intent Recognition** (score: 26.4) - Fix the prompt to handle ambiguous and truncated inputs 2. **Task Matching** (score: 15.4) - Improve the matching logic for similar task names 3. **Database Query** (score: 13.2) - Address timeout issues (lower priority due to infrastructure complexity) Even though database timeouts are frustrating, they're harder to fix than prompt improvements. Start with intent recognition. Reflect on Your Skill[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#reflect-on-your-skill "Direct link to Reflect on Your Skill") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ After practicing systematic error analysis, add these patterns to your agent-evals skill: **Pattern: The Spreadsheet Method** When analyzing agent failures:1. Create a table with columns for each component/span2. Mark each span as OK or ERROR3. Identify which span FIRST introduced the error4. Count error locations across all failures5. Percentages reveal where to focus **Pattern: Prioritization Formula** Priority = Frequency x FeasibilityFrequency: Percentage of failures from this componentFeasibility: How easily you can fix it (0 to 1)Fix high-priority items first, even if lower-frequencyerrors are more frustrating to debug. **Pattern: Trace and Span Vocabulary** Trace: Complete record of one agent run - Contains all intermediate outputs - Shows the full journey from input to outputSpan: Output of single step within trace - Web search span: query + results - Selection span: which sources chosen - Output span: final responseError attribution: Which span first produced bad output?Downstream spans inherit upstream errors. **Key insight to encode**: Don't go by gut. Count errors systematically. The time spent analyzing is an investment that prevents wasted effort fixing the wrong component. Andrew Ng's observation that "less experienced teams spend a lot of time building and probably much less time analyzing" is the difference between methodical improvement and thrashing. Try With AI[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#try-with-ai "Direct link to Try With AI") ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ### Prompt 1: Design Error Categories for Your Agent[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#prompt-1-design-error-categories-for-your-agent "Direct link to Prompt 1: Design Error Categories for Your Agent") I'm building error analysis for my [describe agent type] agent.The agent has these components:- [Component 1, e.g., "intent classification"]- [Component 2, e.g., "data retrieval"]- [Component 3, e.g., "response generation"]Help me design a spreadsheet structure for error analysis:1. What columns should I track for each test case?2. What error categories make sense for each component?3. How should I attribute errors when multiple components fail?Give me a template I can use with 20 test cases. **What you're learning**: Error categories must match your specific agent architecture. Generic categories like "LLM error" don't help you fix anything. AI helps you design categories specific to your component structure. ### Prompt 2: Estimate Feasibility for Your Error Types[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#prompt-2-estimate-feasibility-for-your-error-types "Direct link to Prompt 2: Estimate Feasibility for Your Error Types") I've identified these error types in my agent (with frequency):1. [Error type 1] - [X]% of failures2. [Error type 2] - [Y]% of failures3. [Error type 3] - [Z]% of failuresFor each error type, help me estimate feasibility:- What would fixing it involve?- Is it a code change, prompt change, or infrastructure change?- What unknowns would require investigation?Use the 0-1 feasibility scale where 0.9-1.0 is trivial and 0.0-0.2 needs investigation.Then calculate priority scores and recommend my fix order. **What you're learning**: Feasibility estimation requires understanding the fix. You might think "improve intent recognition" is easy, but it could require new training data (hard) or just adding examples to the prompt (easy). AI helps you think through what the fix actually involves. ### Prompt 3: Generate Error Analysis Code for Your Framework[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#prompt-3-generate-error-analysis-code-for-your-framework "Direct link to Prompt 3: Generate Error Analysis Code for Your Framework") I'm using [OpenAI Agents SDK / Claude SDK / Google ADK / custom framework]for my agent. My agent has these spans:- [Span 1]- [Span 2]- [Span 3]Write Python code that:1. Extracts span outputs from my framework's trace format2. Classifies each span as OK or ERROR based on [describe your criteria]3. Produces error analysis CSV with component attribution4. Calculates error percentages and recommends focus areaInclude example output showing what the analysis would look likefor 5 sample failures. **What you're learning**: Error analysis automation is framework-specific. The trace format differs between SDKs. AI helps you bridge from generic patterns to your specific implementation. ### Safety Note[​](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#safety-note "Direct link to Safety Note") Error analysis reveals patterns in your agent's failures, but patterns are not always causes. A component might fail frequently because it receives bad input from an earlier component, not because it's broken. Always trace errors back to their root cause by examining full traces, not just counting which span flagged the error. The goal is systematic improvement, not blame attribution. Checking access... * [The Build-Analyze Loop](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#the-build-analyze-loop) * [Traces and Spans: The Vocabulary of Analysis](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#traces-and-spans-the-vocabulary-of-analysis) * [The Spreadsheet Method](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#the-spreadsheet-method) * [Why Counting Beats Intuition](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#why-counting-beats-intuition) * [The Prioritization Formula](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#the-prioritization-formula) * [Generating Error Analysis Data](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#generating-error-analysis-data) * [Exercise: Analyze Task API Agent Failures](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#exercise-analyze-task-api-agent-failures) * [Reflect on Your Skill](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#reflect-on-your-skill) * [Try With AI](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#try-with-ai) * [Prompt 1: Design Error Categories for Your Agent](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#prompt-1-design-error-categories-for-your-agent) * [Prompt 2: Estimate Feasibility for Your Error Types](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#prompt-2-estimate-feasibility-for-your-error-types) * [Prompt 3: Generate Error Analysis Code for Your Framework](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#prompt-3-generate-error-analysis-code-for-your-framework) * [Safety Note](https://agentfactory.panaversity.org/docs/Building-Agent-Factories/evals-agent-performance/systematic-error-analysis#safety-note) ---