Chat agent

High-level system design of the AgencyCore chat agent — core components, data flow, and the two abstractions that hold it together.

1 min read Updated May 13, 2026

Chat agent: system design

The AgencyCore chat agent is a single-agent loop sitting behind a streaming HTTP endpoint. It receives a user message, assembles everything the model needs to see, runs a tool-using loop against an LLM, and streams the result back. This article describes the components and how a turn flows through them. Framework choices (Agno, OpenAI Responses, Supabase, Redis) are recorded in code; this is the layer above.

For the industry-wide playbook behind these choices — context engineering, memory split, tool design, when to add complexity — see Designing chat agents.

Core components

TEXT
┌───────────────────────────────────────────────────────────────────┐
│                          ac-frontend                              │
│                        (SSE chat client)                          │
└───────────────────────────────────────────────────────────────────┘
                    │  POST /chat (stream)
                    ▼
┌───────────────────────────────────────────────────────────────────┐
│  Router               auth, rate limit, idempotency               │
│  Guardrails           input length, topic checks                  │
├───────────────────────────────────────────────────────────────────┤
│  Agent loop           model → tool → model → ... → final text     │
│       │                                                           │
│       ├─► Context builder  ────────► ChatContext (envelope)       │
│       │                                                           │
│       ├─► Tools            scoped CLI toolkit                     │
│       │                                                           │
│       └─► Tool pipeline    summarize / map tool results           │
├───────────────────────────────────────────────────────────────────┤
│  Persistence          threads, messages, rolling summary          │
│  Tool budget          per-thread call cap                         │
└───────────────────────────────────────────────────────────────────┘
        │                       │                       │
        ▼                       ▼                       ▼
   Supabase                 Redis                   LLM provider
   (threads, msgs,          (budget,                (model + tool
    summary)                 dedupe keys)            execution)

Three boundaries do the load-bearing work:

  • Context builder owns everything the model sees. It has one entry point and returns a ChatContext envelope. Changing what reaches the model is a one-function edit.
  • Tool pipeline owns what we do with each tool result before it re-enters the conversation. Filtering noisy payloads happens here, not in the agent loop.
  • Persistence + memory split keeps short-term state in the prompt and long-term state in stores the agent fetches from on demand.

Data flow of one turn

TEXT
POST /chat
   │
   ▼
agent.stream_chat_response
   │
   ├── 1. pre-flight (concurrent)
   │     ownership · history load · doc scope · tool-budget
   │
   ├── 2. persist user message
   │
   ├── 3. build_chat_context()  ──►  ChatContext
   │        • load rolling summary from Supabase
   │        • compose history tail after summary cutoff
   │        • pre-retrieve scoped documents
   │        • build per-turn preamble (page state, date)
   │
   ├── 4. stream model run
   │        loop:  model → tool call → tool pipeline → model
   │                                                    │
   │                                                    ▼
   │                                              final text (SSE)
   │
   ├── 5. finalize context
   │        persist new rolling summary if history was compacted
   │
   └── 6. persist assistant message
            background: usage logging, eval, escalations

Step 3 is the hinge. The agent loop never assembles a prompt directly — it asks the context layer for an envelope and feeds it whole to the model. To inspect what the model saw, log the envelope.

Context construction

The context builder is the single most important component in the system. Quality lives here. The builder assembles a layered prompt per turn — a stable cacheable prefix on top, fresh per-turn evidence near the user's question. The design principles article explains why this shape; the diagram below is how we implement it.

TEXT
┌─────────────────────────────────────────────────────────────────┐
│  CACHEABLE PREFIX  (byte-stable across turns)                   │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ system prompt          context/system_prompt.py           │  │
│  │   identity, scope, tool-use rules, output format          │  │
│  │   hash-pinned · rotates only via PROMPT_VERSION bump      │  │
│  ├───────────────────────────────────────────────────────────┤  │
│  │ tool schema            tools.py + tool_schema_fingerprint │  │
│  │   names, args, when-to-use docstrings                     │  │
│  │   hash-pinned · rotates only via TOOL_VERSION bump        │  │
│  └───────────────────────────────────────────────────────────┘  │
│ ─────────────────── cache boundary ─────────────────────────── │
│  FRESH PER TURN                                                 │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ rolling summary        context/history.py                 │  │
│  │   synthetic recap of turns older than cutoff              │  │
│  ├───────────────────────────────────────────────────────────┤  │
│  │ history tail           context/history.py                 │  │
│  │   raw messages newer than the summary cutoff              │  │
│  ├───────────────────────────────────────────────────────────┤  │
│  │ scoped documents       context/references.py              │  │
│  │   pre-retrieved RAG hits for this thread's doc scope      │  │
│  ├───────────────────────────────────────────────────────────┤  │
│  │ per-turn preamble      context/preamble.py                │  │
│  │   page state · doc scope hint · today's date              │  │
│  │   deduped by preamble_hash if unchanged from last turn    │  │
│  ├───────────────────────────────────────────────────────────┤  │
│  │ user message                                              │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
                          ChatContext envelope

Five rules govern the layering:

  1. The cacheable prefix never varies turn to turn. All dynamic content lives below the cache boundary. Anything that drifts into the prefix silently doubles per-turn cost across every active conversation.
  2. History is summarized, not truncated. When the live window crosses ~80% of budget, the builder generates a synthetic summary and carries it on the envelope. finalize_context() persists it to chat_threads.summary_text with a cutoff timestamp. The next turn loads summary + messages newer than the cutoff — no recomputation.
  3. Evidence is retrieved before the model sees the turn. Doc scope checks and RAG hits happen in pre-flight, not via a tool call. The model gets the citations as text, not as a search problem.
  4. The preamble is dedupe-aware. A short hash on page state + scope lets the builder skip re-injection if nothing changed, saving tokens on quick follow-up turns.
  5. Token attribution is tracked. token_breakdown records per-source token cost on the envelope for telemetry, so context regressions show up in dashboards instead of in customer complaints.

The ChatContext envelope

The builder's output is one dataclass passed whole to the agent loop:

TEXT
ChatContext
├── agent_input            what gets handed to the model
├── history_snapshot       (role, content) pairs for retrieval rewriting
├── preamble               page state + doc scope (per turn)
├── preamble_hash          short SHA, dedupes re-injection
├── token_breakdown        per-source token estimate (telemetry)
└── new_summary            set when this turn freshly compacted history

The agent loop never assembles a prompt directly. To change what the model sees: edit build_chat_context(). To inspect what the model saw: log the envelope.

Memory model

TEXT
   ┌──── short-term (in prompt) ────┐    ┌──── long-term (outside model) ────┐
   │ recent turns                   │    │ vector store: prior chats, docs   │
   │ tool results (current loop)    │◄──►│ KV store:    user prefs, facts    │
   │ rolling summary                │    │ Supabase:    threads + messages   │
   └────────────────────────────────┘    └───────────────────────────────────┘

The rolling summary is short-term in shape (lives in the prompt this turn) but long-lived in content (represents turns that scrolled out of the live window). Persisting it is the bridge between the two stores.

Tool result hygiene

When a CLI tool returns 100 rows, the raw payload would flow into history and re-cost on every follow-up turn. The tool pipeline intercepts large list responses and replaces the body with {summarized, total_count, preview: first 10, hint: "narrow with filters"} before the result re-enters context. The model keeps the signal, history stays small.

This is the second-most-important boundary in the system after context construction. Filtering happens once, in tool_pipeline/output_summary.py — not scattered across each tool implementation.

What we deliberately did not build

  • No multi-agent orchestration. One loop with a sharp toolset handles everything we ship today. If we ever split, agent.py is the orchestrator boundary — the context layer stays as-is.
  • No physical split of the system prompt into per-section files. One triple-quoted string + a module-load regex emits PROMPT_SECTIONS for diff attribution. Splitting would have hurt review for no production payoff.

Where to look in the code

TopicFile
Streaming agent loopac-python-api/src/domains/chat/agent.py
HTTP surfaceac-python-api/src/domains/chat/router.py
Context envelopeac-python-api/src/domains/chat/context/envelope.py
Context builderac-python-api/src/domains/chat/context/builder.py
History + rolling summaryac-python-api/src/domains/chat/context/history.py
Per-turn preambleac-python-api/src/domains/chat/context/preamble.py
System prompt (cacheable)ac-python-api/src/domains/chat/context/system_prompt.py
Tool definitionsac-python-api/src/domains/chat/tools.py
Tool schema byte-pinac-python-api/src/domains/chat/tool_schema_fingerprint.py
Tool result post-processingac-python-api/src/domains/chat/tool_pipeline/
Persistenceac-python-api/src/domains/chat/service.py
Rolling summary columnsac-backend/supabase/migrations/20260512120000_add_chat_thread_rolling_summary.sql
agents
architecture
chat
system-design