Build AI Agents with Three Fundamentals

Start with a useful mental model

An agent is not a mysterious autonomous employee hidden behind a chat box. It is a software loop that combines a model with context, tools, policies, and feedback. The model proposes the next step. The harness decides what information to provide, which actions are allowed, how those actions run, and whether the result is good enough to continue.

That distinction matters because most agent failures are not solved by simply choosing a larger model. They are solved by improving the information and capabilities around the model.

System map

The capability stack

1. Memory: the desk and the filing cabinet

An agent uses at least two different kinds of information. The active context is the limited desk where the current task is processed. Persistent storage is the filing cabinet beside it: documents, prior observations, customer records, or other data that can be retrieved when needed.

The filing cabinet does not help simply because it exists. The harness needs a retrieval strategy that identifies relevant information, puts it into context, and keeps it within the available token budget. As the desk fills, the system may need to trim, summarize, or prioritize content. Each choice can change what the model notices and what it forgets.

System map

Context is a working surface

Useful questions for the memory layer include:

• What information is needed for the current decision?
• How is relevance measured?
• What happens when retrieval returns too much or too little?
• Which facts can become stale, and how are they refreshed?
• Can a person inspect or delete what was stored about them?

2. Inference: next-step prediction

The model receives the active context and predicts a response. In an agent system, that response may be a user-facing answer, a structured decision, or a tool call. The model does not have direct access to the entire business or codebase. It sees the representation that the harness constructs for that step.

This is why vague instructions create drift. A role description, procedure, tool schema, and output format act as steering tracks for the model’s prediction. They reduce the number of plausible next steps and make behavior easier to evaluate.

goal + rules + retrieved context + tool definitions
  -> model inference
  -> validated response or tool call

Inference still needs boundaries. Validate structured outputs, reject malformed arguments, cap retries, and make failure a visible state instead of silently asking the model to improvise forever.

3. Capabilities: equip the body

The model can suggest an action, but the local system determines what is possible. A read-only search tool, an email sender, and a production database writer represent very different risk profiles. Each should have a narrow contract and an explicit authorization boundary.

Good capability design includes:

• Small tools with specific names and typed inputs.
• Permission checks outside the model’s control.
• Timeouts, retries, and idempotency for external actions.
• Clear error messages that can be added to the next context.
• Logs that make the decision and action traceable.
• Human approval for high-impact or irreversible operations.

System map

The agent execution loop

Make the three fundamentals testable

Before calling a system production-ready, create a small evaluation set that represents real work. Include normal tasks, incomplete information, ambiguous requests, tool failures, permission failures, and attempts to exceed the intended scope.

For each task, record:

1. What context was assembled.
2. What action the model proposed.
3. What the tool actually did.
4. What result was observed.
5. Whether the final outcome met the acceptance criteria.

This gives you a way to compare changes to retrieval, prompts, tools, or models without relying on a handful of demos.

A practical build order

Start with a deterministic workflow and one narrow capability. Add retrieval only when the task needs information that cannot fit safely in the initial context. Add additional tools after the first tool has clear contracts and evaluation coverage. Add autonomy gradually, with a human checkpoint wherever an incorrect action would be costly.

The strongest agent systems are usually less magical than their demos. They make the model’s inputs, possible actions, and failure states visible enough for a team to improve them.