Designing chat agents

What the broader industry has converged on for building production chat agents that can reach into internal systems via tools. Framework- and vendor-neutral. The AgencyCore-specific shape lives in Chat agent.

Where the quality lives

Modern chat agents are mostly a loop around one model call, with tools and memory bolted on. That part is easy. What separates a brittle demo from a system you can put in front of customers is context engineering: deciding what the model sees on every turn, in what order, and how state survives between turns.

The dominant 2026 consensus is to treat the prompt as a layered envelope assembled per turn, with a stable cacheable prefix on top and fresh evidence near the user's question.

┌─────────────────────────────────────────────────────────────────┐
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 1. System prompt   (stable, cacheable across turns)       │  │
│  │    identity, scope, tool-use rules, output format          │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 2. Tool definitions   (stable, cacheable)                  │  │
│  │    name, when to use, args, example call                   │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 3. Persistent context   (slow-changing)                    │  │
│  │    user role, org glossary, long-term memory summary       │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 4. Session state   (per-conversation)                      │  │
│  │    rolling summary of older turns + last N raw turns       │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 5. Retrieved evidence   (per-turn, just-in-time)           │  │
│  │    RAG hits, scoped documents, each tagged with source id  │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 6. Tool results   (current loop iteration only)            │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ 7. User message   (closest to generation)                  │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
              ▲                                       ▲
        cache-friendly                          fresh per turn

Five rules carry most of the weight:

1. Retrieve before you reason, cite before you recommend. Push this into the system prompt; have tools return source IDs so citations are mechanical.
2. Keep the cacheable prefix byte-stable. Sections 1 through 3 must not vary turn to turn. Anything dynamic — page state, today's date, document scope — lives below, otherwise you bust the provider's prompt cache on every message and pay for it on every active conversation.
3. Summarize old turns, do not truncate. A rolling summary preserves intent. Hard truncation throws away the user's actual goal.
4. Scope tools per request. Surface five tools relevant to the user instead of forty general ones. Fewer tools means fewer wrong tool calls.
5. Filter tool results before they re-enter context. Raw API responses are noisy. Map them to compact, schema-stable summaries the model can read as text, not as JSON archaeology.

Memory: short-term lives in the window, long-term lives outside

Short-term memory is the prompt itself: recent turns, the rolling summary, the current tool results. Long-term memory is outside the model — a vector store of prior conversations, a KV store of user preferences, an episodic log of past decisions. The agent fetches from long-term memory on demand and pulls the result into short-term context just for this turn.

   ┌──── short-term ────┐         ┌──── long-term (outside model) ────┐
   │ recent turns       │         │ vector store: prior chats, docs   │
   │ tool results       │  ◄──►   │ KV store: user prefs, facts       │
   │ rolling summary    │ fetch   │ episodic log: events, decisions   │
   │ scratchpad / plan  │         │ skills / SOP library              │
   └────────────────────┘         └───────────────────────────────────┘

Crucially, the rolling summary is short-term but its content is long-lived — it represents older turns that have already scrolled out of the live window. The trick is to persist it so the next turn can replay it without re-deriving it from raw history.

Tool design

The model reads tool definitions the same way a junior engineer reads an API doc. Treat the docstring as user-facing copy.

       Hostile                       Friendly
  ┌──────────────┐                ┌──────────────────────┐
  │ get_data()   │                │ search_customer(     │
  │   → 4MB JSON │                │   query: str,        │
  └──────────────┘                │   limit: int = 10    │
                                  │ ) → [{id,name,plan}] │
                                  │                      │
                                  │ # when to use:       │
                                  │ # finding a customer │
                                  │ # by free-text       │
                                  │ # when NOT to use:   │
                                  │ # listing all-       │
                                  │ # use search w/      │
                                  │ # empty query        │
                                  └──────────────────────┘

• One tool, one verb. Overlapping tools confuse the model more than missing ones limit it.
• Docstrings always include when to use, when not to use, and one example call.
• Return shapes are stable, small, and human-readable. The model reads them as text.
• Errors are messages the model can act on ("customer not found, try search_customer with a partial name"), not stack traces.
• Compact large results before they re-enter context. The model is reading text, not parsing JSON archaeology.

Tool granularity: workflow shape, not API parity

Three workflow-shaped tools beat thirty CRUD wrappers. The granularity heuristic is that a tool corresponds to a user intent (read this thing, write that thing, read these things in parallel), not to a backend endpoint. A chat agent that has to choose between forty look-alike tools picks the wrong one more often than an agent with five sharply distinct ones, and pays for the full schema list on every turn either way.

When parallelism matters, expose it as its own tool. A separate read_many beats inviting the model to schedule N sequential single-reads. The model does not enjoy planning fan-out; making it part of the tool surface removes a class of "called the read tool four times when one batch call would have worked" bugs.

When something needs procedural knowledge or many heuristics to use correctly, it's not a tool — it's a skill. See the next section.

Skills are the layer below tools, and they progressively disclose

Tools are functions. Skills are bundles of procedural knowledge that tell the agent how to use those tools well: entity-resolution rules, common patterns, troubleshooting, page-specific behaviour. They live as Markdown files with frontmatter, not as Python.

The load-bearing property is progressive disclosure. Skill metadata (name + one-line description, ~tens of tokens) loads up front so the model knows which skills exist. The skill body loads on demand when the model decides it needs it. The trigger lives in the system prompt: "for CRM work, load the ac-cli-crm skill"; "if Current route: matches /launchpad, load the launchpad skill". The skill body never enters context until earned — a fifteen-skill registry costs maybe a kilobyte of metadata.

This is the dial that makes the cacheable prefix small without sacrificing depth. A 5KB skill body that loads on 1 in 10 turns costs less, on average, than 1KB of trivia that loads on every turn. Treat skill descriptions as the user-facing copy: third-person, "what + when", under a kilobyte. The model uses them to choose.

The pattern decomposes naturally:

• Shared rules (ac-cli) — entity resolution, action bias, output conventions; always loaded once.
• Domain skills (ac-cli-crm, ac-cli-envoy, …) — command names, flags, worked examples; loaded by intent.
• Route-aware skills (launchpad, sonar, current-view) — page-specific behaviour; loaded when the route matches.

Error recovery is a tool contract, not the model's problem

Tool errors return short, actionable strings the model can act on directly — "argument X not allowed — reload the domain skill and retry", not stack traces, error codes, or {"error": {"type": "...", "details": {...}}} archaeology. The agent should be able to retry without a human in the loop.

Two failure modes deserve explicit handling at the tool layer rather than getting handed to the model raw:

• Empty success. When the contract says every success carries data and the tool got back nothing, that's a failure — not a Done. Surface it ("Command produced no output; cannot verify the action succeeded — re-read the entity") so the model re-reads instead of confabulating success.
• Oversized success. Compact large payloads server-side before they re-enter context. A single "list everything" call should not bloat this turn and every subsequent turn via conversation history. Return {summarized: true, total_count: N, preview: [...], note: "..."} and let the model drill in if it needs to.

Treat the tool envelope as a contract the model is allowed to depend on: ok: bool, stable field names, predictable error semantics. Random shape changes are how silent regressions creep in.

Cache stability is a load-bearing invariant

Modern providers cache long input prefixes. A byte-stable prefix on every turn means every active conversation pays the cached-input rate (a fraction of full input); a prefix that drifts means every conversation pays full price every turn. On a chat product this is the difference between a sustainable cost line and one that scales linearly with engagement.

Two rules buy this property:

• Anything dynamic (page state, today's date, document scope) rides in a per-turn user-message preamble below the cached prefix. Never spliced into the system instructions.
• The static prefix (system prompt + tool schemas + skill registry) is pinned in CI. Hash the rendered system prompt; hash the rendered tool schemas; fail the build on silent drift. Pair the hashes with explicit version constants (PROMPT_VERSION, TOOL_VERSION) and a bump ceremony: edit → bump version → run snapshot test → paste new digest → re-run until green.

The ceremony is tedious by design. The cost of skipping it is that an innocent-looking copy edit rotates the cache prefix and every active conversation pays full input price until the cache rewarms. Snapshot tests catch this at PR time instead of in next month's invoice.

When to add complexity

The temptation with agents is to reach for multi-agent orchestration too early. The honest progression is the other way around:

 well-defined  ───────────►  fluid / exploratory
 ┌──────────┐  ┌──────────┐  ┌──────────────┐  ┌──────────────┐
 │  Chain   │  │  Router  │  │ Orchestrator │  │ Agent loop   │
 │ (fixed)  │  │ (branch) │  │ + workers    │  │ (model-led)  │
 └──────────┘  └──────────┘  └──────────────┘  └──────────────┘
       Workflows (you decide the flow)         Agents (LLM decides)

Start with a single agent loop behind a streaming chat endpoint. Promote to orchestrator-workers only when a concrete failure mode demands it — usually because one agent's context is mixing concerns it can't handle cleanly. Multi-agent systems multiply the surface area of your bugs by the number of agents; do not pay that cost speculatively.

Rendering parity is part of the agent contract

A chat agent that emits a correct ac_suggested_actions JSON block but renders a blank chip row is broken from the user's point of view — even though every behaviour eval passes. The agent's contract is "what the user sees," not "what bytes the agent produces."

Two consequences for how to test:

• The renderer is in scope. Treat the frontend that consumes ac_* blocks as part of the agent boundary. A frontend regression that swallows a content block is an agent bug.
• One layer of evals isn't enough. Stub-toolkit behaviour evals catch decision and prose regressions. They cannot catch a renderer that throws away actions because the schema expects options. The cheapest remedy is a live-browser eval loop (see System prompt architecture › What to test) on the small set of cases where rendering parity matters — every content-block change, every per-turn-preamble change, every page-context payload change.

Knowing where the boundary sits also tells you where to fix gaps. A missing body preview in an approval card is rarely an agent problem — the page-context payload didn't ship the body field. A blank chip row when the JSON looks fine is a renderer problem. The shape of the failure tells you which side of the boundary owns the fix.

Key takeaways

• Context engineering is where chat-agent quality lives. Treat the prompt as a layered envelope, keep the cacheable prefix stable, and put per-turn dynamism in a preamble below the cache boundary.
• Single agent loop is the right default. Multi-agent only when a concrete failure mode demands it.
• Memory is two stores. Short-term lives in the window, long-term lives outside. The rolling summary is the bridge; persist it so you stop paying to re-derive it.
• Tool design is API design for the model. One tool one verb, docstrings with when-to-use and when-not, return shapes the model can read as text.
• Rendering parity is in scope. A correct JSON block that renders blank is broken. Reserve a live-browser eval loop for the cases where rendering parity matters.