Observability and operations

Observability is a plane. Every layer writes into it, and no layer needs it to function.

Policy decides before the work.
Observability records what the work did.

Two systems, and one sentence each.

A failed vendor call is normal agent work, so it belongs in a span. A KeyError in our handler is a defect, so it belongs in Sentry.

One run, one span tree

agent.runs is the product run table. A agent.spans row is one unit of work inside a run.

agent.spans
  span_id
  organization_id      the tenancy boundary; RLS reads it
  run_id
  root_run_id          copied from the run, so one query builds a whole tree
  parent_span_id
  kind
  name
  status               running | ok | error
  started_at / ended_at
  duration_ms          generated from the two above; never written
  updated_at           a BEFORE UPDATE trigger writes it; the reconnect filter reads it
  input  jsonb         bounded and redacted
  output jsonb         bounded and redacted
  usage_id             the usage row this work produced, when there is one
  attributes jsonb     attempt, replayed, policy, step_path, and similar
  error jsonb

Two fields are easy to leave out and expensive to add later.

• organization_id is the tenancy boundary. Every product table in this platform carries it, and a span holds tool arguments, so it needs the same protection as the data it describes.
• root_run_id is what makes the run explorer one query. A workflow with three child runs is one tree to a person. Without this field the explorer walks the run graph first, then queries spans per run.

A child run hangs from the node that started it

spans_parent_same_tree_fk allows a parent in another Run of the same tree, and the run span of a child Run is the one span that uses it. Without that edge the tree has as many roots as it has Runs, and a person reading a workflow cannot see which node produced which child.

The parent span is the span that was current at the node, and the node opens none of its own. The eight kinds below are closed, and none of them names an agent node or a subworkflow node. So parent_span_id is current_span_id() at the node: the enclosing parallel span, or the run's own run span. The child's run span carries the node ID as its name, so a person still reads which node produced which child, and no ninth kind is needed. Read "the node span" below as that span.

The child worker learns the parent span from the Run row. RunSource.parent_span_id reaches RunManager on the start command, and the child executes in another process, sometimes minutes later and sometimes after a crash. So the value is a column:

agent.runs
  parent_span_id  uuid   the node span that started this Run; null for every other source

It carries no foreign key, and retention is the reason. agent.spans is kept 90 days and agent.runs is kept 13 months, so the parent span is deleted while the child Run is still the product record. ON DELETE CASCADE would delete that Run, and ON DELETE SET NULL would break the shape rule below.

The shape rule is an equivalence. A workflow_step Run carries a parent span, and every other source carries none. Written as an implication it forbids the value elsewhere and never requires it here, so every child Run would be free to land with a null parent and sever its own tree with nothing raising.

A trigger proves the span at the INSERT, and freezes it after. Without it the value is proved one write later, when the child's own run span meets spans_parent_same_tree_fk. By then the Run row is committed and its dispatch event is sent, so the child fails mid flight instead of never starting. The same check refuses a span of another tenant.

Span kinds

run         the root span of one run
segment     one agent segment
llm         one model call
tool        one tool call
approval    one human wait
wait        one event wait or delay
branch      one condition choice
parallel    one parallel container

A sequence container writes no span, because ordering adds no timing a person can use.

There is no policy kind. An allowed action is an attribute on the span of the work it gated, and a refusal already writes a agent.policy_decisions row. An admission decision writes a row whatever it answers, because admission runs once per run. A span per decision would double the writes of the hot path and explain nothing new.

There is no eval kind either. An evaluation run is a run. It writes the same kinds, with its source in the run row.

Do not add a durable event table beside spans. Events such as tool.called and tool.failed are projections of the span lifecycle.

Who opens a span

The code doing the work owns the span. Application code uses one scoped context manager and nothing else.

async with recorder.span(kind='tool', name=tool.name, input=args) as span:
    result = await handler.execute(ctx, args)
    span.set_output(result)

The recorder opens the span, completes it, or fails it and re-raises the original exception. Tracing never decides a retry and never changes control flow.

A span that wraps more than one Inngest step needs an open and a close. The block above closes where it opens, and a parallel span or a run span cannot: the work between them is several durable steps, and the SDK re-executes the function body once per step. So SpanRecorder also offers open_span(), which answers a span ID, and close_span(), which takes it. Each half runs inside its own step, so a replay writes neither again. Everything that fits in one step keeps the block, and that is almost everything.

The input is an argument of span(), and there is no set_input. The input of a unit of work is known before the work starts, and it is written by the INSERT that opens the span. A setter would offer a second moment to write it, and the only span that reached that moment would be one the crash already left open.

A payload that is neither a dict nor a list is wrapped before it is bounded. bound() raises TypeError on any other type, and a span output is often a string or a number: a model turn answers text, and a tool answers a count. The recorder wraps such a value as {'value': payload} and bounds the wrapper. It does not pass the bare value, and it does not skip the boundary.

The recorder also writes the heartbeat

RunManager stamps heartbeat_at at the insert and at each lifecycle write, and a segment may then approach its 90 second wall clock. Nothing writes it between those two moments, so the reaper would have no safe stale_after.

SpanRecorder is the only writer between the claim and the finalize. It is not the only writer of the column. build_transition_values stamps heartbeat_at on every lifecycle write, because the insert and each transition are the two moments no span covers. Read "one writer" as "one writer per moment", and never as "one statement in the code base".

span opened -> the process wrote this run more than 10s ago
               -> UPDATE agent.runs SET heartbeat_at = now() WHERE id = :run_id
            -> the process wrote this root more than 60s ago
               -> UPDATE agent.runs SET heartbeat_at = now() WHERE id = :root_run_id

The window is tested in the process and never in the statement. See Write cost below for the measurement that removed the SQL guard.

The recorder is the right owner because it already fires on every unit of work and already holds both IDs.

The root row is written because a workflow parent suspended on a child writes no spans of its own. Its child's work is what proves the tree is alive.

The two windows are different numbers on purpose. A run's own row is touched by one worker, so 10 seconds is free. A root row is touched by every worker in its tree, and an email sequence has five hundred of them. At 10 seconds those five hundred contend on one row for a write that almost always changes nothing. At 60 seconds the root is touched at most once a minute per process however wide the tree grows, and stale_after is 120 seconds, so the reaper is still nowhere near it.

This is the one place where tracing carries a liveness duty. It is still not control flow: a failed heartbeat write reports to Sentry and changes nothing. See runtime execution.

Current span identity lives in a context variable, read through three accessors: current_run_id(), current_root_run_id() and current_span_id(). No function grows another trace parameter, and no type is named Context here.

One writer binds the run, and it is a scope rather than a setter. run_scope(run_id, root_run_id, organization_id, parent_span_id=None) is a context manager. A bare setter has no exit, so a worker that serves the next run reads the identity of the previous one. span() binds current_span_id() the same way, for the body of its own block.

The scope wraps the whole Inngest function body, not the claim step. A context variable set inside step.run('claim', ...) is gone when segment 1 starts, and every span of segments 1 to n then writes a null run id. The scope needs root_run_id and parent_span_id, which only the Run row carries, so the order is fixed: claim, then enter the scope, then run the loop inside it.

parent_span_id is how a child Run joins its parent's tree. The child executes in another process, so it reads the value from its own Run row rather than inheriting a context variable. Seeding the scope with it makes the child's own run span parent into the node, and every span below nests as usual.

Reset the token in the task that set it. ContextVar.reset raises ValueError in another context. async with satisfies this rule, and a token stored on an object does not.

That last part is deliberate. In this platform context means what a model may know, and ContextBrief is the only thing that carries it. Ambient execution identity is a different idea, so it gets accessors and no noun.

There is no trace_tool or trace_llm helper hierarchy in V1. kind plus name is enough.

Durable first, live second

open     -> INSERT agent.spans (status running) -> publish span.started
complete -> UPDATE agent.spans (status ok)      -> publish span.completed
fail     -> UPDATE agent.spans (status error)   -> publish span.failed

Live publishing is best effort. When delivery fails, the durable span is still correct, and the client recovers by refetching.

Runtime execution owns the live event vocabulary, the Redis and SSE path, and the reconnect rule. This page owns only the ordering guarantee above.

If a span write itself fails, report it to Sentry and let the business operation succeed. A tracing fault must never turn a completed send into a failure. Usage is the exception, and the next section says why.

A failed open leaves the parent span current. The span has no row, so a child that took it as a parent writes a parent_span_id that no row satisfies, and spans_parent_same_tree_fk answers 23503 on every child under it. One lost span becomes a lost subtree, and the rule above says a tracing fault costs the caller nothing. The block still runs, and its close is skipped.

The live publisher arrived with Phase 2. The ordering above is the rule the recorder is built to. Phase 1 shipped the seam and a default that published nothing, so no span write waited on a component that did not exist yet. Phase 2 added RedisRunEventPublisher, and a recorder built with None still publishes nothing.

Retries, replays and orphans

A crash leaves marks in the tree. The design chooses honest marks over tidy ones.

The third row needs an owner, or a tree keeps a span that spins for ever. The run.reaper already fails runs with a stale heartbeat, and it closes their open spans in the same pass, with status = error and error.code = worker_lost.

The fourth row has its own owner, and it is a third sweep in the same cron. A waiting run writes no heartbeat, so the read above never sees it. The wait sweep reads agent.runs.waiting_expires_at instead and ends a run past that clock plus INNGEST_AGENTIC_WAIT_GRACE_SECONDS, under error.code = wait_abandoned. It runs before the second span sweep below, so close_orphans() stamps that same code on the spans it leaves, rather than the generic run_ended. See runtime execution for the deadline, the grace window and the one ending it writes.

The two span sweeps need two methods, because the second is not scoped to a run. close_orphans(run_ids) closes what the reaper just failed, and it reads agent.spans (run_id) where status = 'running'. It takes a list, because one reaper pass fails many Runs and a list is one statement rather than one per Run. The second sweep below has no Run to be given: it looks for spans under runs that ended by some other path, and the reaper does not know which those are. It is its own query, and it takes a limit because a bad deploy can leave a lot of them.

⚠️ The second sweep is a read and then a write. It is never one UPDATE with an inner join. PostgREST ignores an embedded resource filter on an UPDATE, and it still applies that filter to the returned representation. The response then shows exactly the rows the author expected, and the statement has already closed every other running span in the schema, in every organization. Measured on the local stack, with one orphan under an ended Run and one healthy span under a live Run:

PATCH /spans?select=span_id,runs!inner(ended_at)
             &status=eq.running&runs.ended_at=not.is.null
  response  orphan-under-ended                  the join was applied here
  database  orphan-under-ended  -> error
            healthy-under-live  -> error        the join was not applied here

So the sweep reads the span IDs with the join and the limit, then writes them by ID:

GET   /spans?select=span_id,runs!inner(ended_at)
             &status=eq.running&runs.ended_at=not.is.null&limit=:limit
PATCH /spans?span_id=in.(...)&status=eq.running

status = 'running' stays on the write, because a span may close honestly between the two calls and a write without it would overwrite ok with the sweep code. The ID list travels in the URL, so it is written in pages of 200, which is the same bound RunRepository already measured against Kong.

It sweeps a second set as well: any span still running under a run that has ended. Inngest cancels a run by stopping its function, so the worker never reaches a finalize, and the run leaves running before its spans do. The heartbeat clause cannot find those, because the run is no longer queued or running. Without the second clause the orphan span alert below would fire for ever on every cancelled run.

The second sweep stamps error.code = run_ended, and never worker_lost. It reads every open span under a run that has ended, whatever ended it, so it cannot name a cause: a person cancels one run, run.failed ends a second, and the reaper pass above ends a third. worker_lost here sends a person to look for a worker that no run lost. The scoped close_orphans() still carries the true code of each run the pass failed, so no cause is lost. A cancel adds none: it runs while the worker is still alive, and a span it stamped would take the outcome the worker is about to write.

A replayed call still writes a span on purpose. Leaving it out makes a resumed run look as if it never touched the CRM. Marking it keeps the tree complete and keeps cost attribution exact, because the replay costs nothing.

Payload bounds and redaction

A span stores tool input and output, because that is what an auditor and an engineer both need. It stores them under three limits.

1. Bounded. bound(payload, 8 KB) each, and attributes.truncated comes from what it returns. It is the one payload boundary, defined in the platform contract; a span uses a smaller limit than a result, and the same algorithm. Storing the head of a serialized body is the thing it exists to prevent.
   One flag covers two payloads written at two moments, so the close merges it. The input is bounded at the open and the output at the close. A close that writes attributes whole sets truncated to what the output alone measured, and a trimmed input then reads as intact. The close reads the flag it is about to replace and keeps a true.
2. Redacted. Tool input and output pass the same boundary the tool layer applies. Credentials never reach a span. A field the model must not see is absent from the schema, so it is absent here too.
3. No prompts by default. A full prompt or a full context brief is not stored. Store a hash when a comparison needs one. An optional per-account debug capture must expire on its own, within days.

Business content such as an email body is legitimate span input. It is business data the product already holds.

Cost and usage

ai_usage_log is the cost truth. ai_usage_daily is its rollup.

• One row per model call, and one row per metered vendor call.
• A span points at a usage row with usage_id. A span never carries a second price.
• Run totals may be denormalized for the UI. The meter wins any disagreement.
• A run tree total sums by root_run_id. A parent therefore shows what its children spent.
• Policy accrual reads this same meter. There is no second counter.

ai_usage_log carries the root run, and the meter stamps it on every row. The table's existing workflow_run_id is a foreign key to public.workflow_runs, so it cannot hold an agent.runs id. Without a column of its own, a tree total has one path left: read every usage_id off the span tree, then filter the meter by that set. A discovery run of twenty thousand model turns sends twenty thousand identifiers back as a filter, which is a 740 KB PostgREST URL, and both readers of this number run on a hot path.

ai_usage_log
  agent_root_run_id  uuid   the root of the tree that spent this; null for a live-stack row
  index (agent_root_run_id) where agent_root_run_id is not null

The column is a soft reference with no foreign key, exactly as agent.spans.usage_id is, and for the same reason: a real key would tie the live table back into the isolated schema. usage_id on the span stays, because it answers the other question: which unit of work produced this row.

The sum is a database function, because PostgREST refuses an aggregate

⚠️ ai_usage_log cannot be summed over PostgREST. Aggregate functions are disabled on this instance, and authenticator carries no pgrst.db_aggregates_enabled setting to turn them on.

GET /ai_usage_log?select=cost.sum()
  -> 400  PGRST123  Use of aggregate functions is not allowed

Reading the rows and summing them in Python is not the fallback. max_rows is 1,000, and PostgREST truncates past it silently, so a discovery run of twenty thousand model turns would report the cost of the first thousand. Both readers of this number are on a hot path, and one of them is a ceiling.

So one read only SQL function answers it, in the agent schema beside the tables that need it:

agent.run_usage(p_root_run_id uuid, p_organization_id uuid)
  returns (input_tokens bigint, output_tokens bigint,
           total_tokens bigint, cost_cents bigint)
  -- STABLE. One indexed aggregate over ai_usage_log.agent_root_run_id.
  -- REVOKE EXECUTE FROM PUBLIC, per the rule of the agent schema.

agent.organization_day_cost(p_organization_id uuid)
  returns (spent_cents bigint, cap_cents bigint)
  -- STABLE. One indexed aggregate over (organization_id, created_at), plus
  -- the agent.cost_ceilings row of kind daily_cost. cap_cents is null when
  -- the organization set no ceiling.
  -- REVOKE EXECUTE FROM PUBLIC, per the rule of the agent schema.

An empty answer from either function is a fault, and never a zero.run_usage aggregates with no GROUP BY and coalesces every column. organization_day_cost selects two scalar subqueries and reads no table in its outer query. Each therefore answers exactly one row for every input. A tree that spent nothing answers a row of zeroes, and so does an organization.

Read as zero, an empty answer reports no spend and no cap. Accrual reads that as headroom, and the ceiling stays unenforced for the whole fault window. safe_execute_query makes the shape reachable: it turns a PostgREST 204 and an unparseable body into an empty result. The meter raises MissingRunUsage and MissingDayCost, and AccrualChecker turns either into deny under metering_unavailable.

A column the row does not carry is the same fault. PostgREST keeps the key of a null column, so a key lookup catches a renamed column and misses a dropped coalesce. Every column is BIGINT, so a float or a string means the function lost a cast. UsageSummary and DayCost validate nothing, and accrual compares their fields outside the block that catches a meter fault, so a null would raise out of a checker that promises never to raise. Each read therefore refuses an absent, a null and a non integer value. cap_cents is the one exception: null there means the organization set no ceiling.

A run detail answers a null usage, and not a 500. The governance half must fail closed. The read half must not. Three endpoints build a detail, and the start and the cancel build it after they committed their effect, so a 5xx there tells the caller that a live run did not start. A total the meter did not answer is not a run that spent nothing. RunDetail.usage is therefore nullable and it never falls back to zeroes.

The day function reads the ceiling in the same statement, so the accrual check costs one round trip and not two. The UTC day boundary is computed inside the function. Two workers must agree which day a call belongs to, and a boundary a worker passes in puts every worker clock on the correctness path.

The day sum filters on the organization and the day, and on no source. See policy and governance.

The function reads and returns. It decides nothing, so it does not break the rule that the database is storage. It carries its own organization_id for the same reason every statement in this schema does: the caller holds the service role, so no policy filters it.

Cost is summed before it is converted. ai_usage_log.cost is numeric(12,6) and it holds a currency amount, not cents. Thirty model calls of 0.004 each convert to 0 cents apiece and to 12 cents together. SUM first, multiply by 100, then round once.

The two usage types

@dataclass(frozen=True)
class UsageRecord:
    """One metered call, on its way to ai_usage_log."""
    organization_id: UUID
    root_run_id: UUID          # stamped into agent_root_run_id
    model_id: str
    model_provider: str
    cost: Decimal              # the currency amount the vendor charged
    input_tokens: int = 0
    output_tokens: int = 0
    total_tokens: int = 0
    cache_read_tokens: int = 0
    cache_write_tokens: int = 0
    reasoning_tokens: int = 0
    duration_ms: int | None = None
    user_id: UUID | None = None
    source: str = 'agentic'
    metadata: dict | None = None


@dataclass(frozen=True)
class UsageSummary:
    """What one run tree spent. RunDetail.usage carries it."""
    input_tokens: int
    output_tokens: int
    total_tokens: int
    cost_cents: int

UsageRecord writes no agent_run_id and no workflow_run_id. Both belong to the live stack, and agent_run_id is unique per row.

UsageSummary is the whole read. Accrual takes cost_cents off it rather than calling a second method, because a second method is a second query shape over the same function.

Who prices the call

The vendor reports tokens. It reports no price, and neither does the framework RunOutput. So the caller of UsageMeter.record prices the call, and the platform has one pricer: src/shared/metering/model_pricing.calculate_run_cost. It is not a new rate table. ai_usage_log is one table for the whole product, so a second rate card would put two prices in one column.

cost = Decimal(str(calculate_run_cost(model_id, input_tokens, output_tokens,
                                      cache_read_tokens, cache_write_tokens)))

The function answers a float and UsageRecord.cost is a Decimal, so the conversion goes through str. Decimal(0.0009) carries seventeen digits of binary noise into a numeric(12,6) column.

⚠️ The input token count means two different things. calculate_run_cost wants the count that excludes the cache reads, because it prices a cache read at its own discounted rate. Agno reports Anthropic's input_tokens, which already excludes them, and OpenAI's prompt_tokens, which includes them. Pass the raw number for OpenAI and the cached tokens are priced two times.

anthropic   input_tokens                              already the non cached count
openai      max(0, input_tokens - cache_read_tokens)

A model the rate table does not hold prices at zero and logs at ERROR. Zero is a wrong number and not an error, and max_cost_cents reads this column. See cross-document invariant 40.

Metering is not best effort

Policy accrual depends on the meter, so a run whose spend is not counted cannot be governed.

the vendor or model call returns
  -> write the usage row at once, in its own transaction
  -> a transient failure retries inline, three times
  -> still failing -> fail the run with metering_unavailable, and do not retry

The failure is deliberately terminal. A segment retry would spend more money that also cannot be counted, so retrying makes the hole larger. An uncounted run is worse than a failed run.

The usage row is written from the vendor response, so the amount is already known when the write starts. The only failure mode is the write itself.

An asynchronous provider job moves the frame, not the rule. The amount is still known when the write starts, because the provider reports it in the callback. Only the writer changes. The Run that submitted is often terminal by then. So provider_job.settle writes the row on an Inngest frame, and Inngest retries it. An exhausted retry reports to Sentry and leaves agent.provider_jobs.usage_id null, which is the query that finds an unsettled charge. It ends no Run, because there is no Run left to end. See asynchronous provider jobs.

The meter raises, and the caller ends the run. UsageMeter.record retries three times and then raises MeteringUnavailable. It does not call RunManager, because a component that both writes the meter and ends runs is two owners of the lifecycle. The caller maps that exception to a terminal failure with the code metering_unavailable, and to a non retryable error for Inngest.

record() never swallows. The live stack's log_usage is fire and forget and it catches every exception; this meter is the opposite of it, and it does not call it.

Retention

A scheduled job deletes past the window. retention.sweeper is that job for agent.policy_decisions, agent.spans and agent.runs: an Inngest cron beside run.reaper, on the half hour, which reads the oldest rows past each window and deletes them by id. Each table carries its own bound, DECISION_SWEEP_LIMIT, SPAN_SWEEP_LIMIT and RUN_SWEEP_LIMIT. A pass whose read fills its bound logs a warning. A run older than its span window keeps its result and its cost, and loses its step detail.

The order is decisions, then spans, then runs. agent.policy_decisions.run_id is ON DELETE SET NULL, so deleting a run updates every decision that names it. A decision under a 13 month run is already past its own 12 month window, so the log goes first and that update then touches nothing. The spans go before the runs while a backlog drains: both sweeps read the oldest trees first, so during a drain they read the same trees and the span sweep leaves the run cascade less to carry. At steady state the two cohorts sit ten months apart and never meet, and what helps the run sweep then is that the span sweep runs at all — a tree reaches 13 months with its spans already gone.

A sweep that cannot reach its table does not stop the sweeps after it. Each sweep records its own failure, the pass finishes every sweep, and the error is raised at the end. A span tree that refuses every hour would otherwise stop agent.runs from ever being swept again.

A run is deleted by its root, one root to one statement. root_run_id and parent_run_id both carry ON DELETE CASCADE, so one root row takes its whole tree, and the tree takes its spans, its sessions, its approvals and its cancellation row. A statement naming a page of roots would cascade into millions of rows and meet the statement timeout, which rolls back every root it named. A child run is created after its root, so a child past the window always has a root past the same window and no tree is left half deleted. A tree the database refuses is counted and skipped, because the read is ascending and a raise there would stall the sweep on its oldest rows for ever.

A span tree is chosen by its ROOT span, and the run stays. The read tests the span that carries no parent. A child span opens after its parent, so a tree whose root span is past the window holds no span inside it, except the live tree case below. agent.runs.parent_span_id carries no foreign key, and this window is the reason it does not: the run outlives the node span it hung from.

The delete is newest first, one bounded page to one statement. It does not name the root span. spans_parent_same_tree_fk carries ON DELETE CASCADE, so a statement naming the root cascades the whole subtree inside itself. A tree of 150,000 spans then meets the 8 second statement timeout on every pass. The widest trees hold the most content, so those are the trees that would never delete.

started_at comes from the column default at the insert, and SpanRecorder awaits a parent open before it opens a child. So the parent carries the earlier value. A page is a prefix of the descending order, so every descendant of a row in the page is in the page as well. The order rests on that await and not on the foreign key, which needs the parent committed first and not started first.

Three consequences are worth stating.

SPAN_SWEEP_LIMIT counts trees, not spans. A span arrives far more often than a run, and this bound reads none of that rate: a tree is one read and then one paged delete, and the page is what bounds the rows. So it is the same arrival unit as RUN_SWEEP_LIMIT. The run bound's headroom argument does not transfer, though — that one is set against a table 13 months behind the writes it sweeps, and this sweep lags 90 days.

A tree goes whole, so a span inside the window goes with a tree whose root span is outside it. The wait sweep bounds a parked run at its own deadline, and approval_ttl_seconds carries no upper bound, so that deadline can sit past 90 days and a run parked on an approval reaches it. MAX_RUN_DURATION_S is 30 days and it is measured at a segment boundary, so it ends such a run only after the approval answers.

A live tree that loses its spans stops recording, and it can also stop running. A later span insert names a parent row that has gone, so spans_parent_same_tree_fk answers 23503; SpanRecorder reports that to Sentry and the work continues. RunExecutor raises when the run span does not open, and Inngest memoizes that step, so a resumed run of that tree fails. And a workflow_step child run INSERT meets agent.assert_parent_span_shape, which answers 23503, so the child never starts. The window outranks a run, which is what ENG-2166 states, but the cost is a failed run and not only a thinner trace.

Deletion on request

The windows above answer "when does this age out". They do not answer "delete this person now", and that request arrives from a customer, not from a schedule. We hold EU personal data, so the answer must exist before V1 ships.

A span is the hard case. It stores tool input and output, so a span already holds the email address it sent to and the message body it sent. Every other table holds a reference; a span holds the content.

One job answers both shapes of request.

Redaction rather than deletion, for the person case:

agent.spans            input and output -> {"redacted": "subject_erasure"}; kind, timing, status and usage_id stay
agent.runs             input and result summary redacted; status, timing and refs stay
agent.approvals        proposed arguments and proposed_content redacted; decision, actor and timing stay
agent.sessions         deleted; it is execution state and it is reconstructable from nothing
agent.memories         deleted, including superseded rows
agent.conversations    messages deleted
agent.knowledge_chunks deleted with their source
agent.idempotency_keys response redacted; the claim, the status and the hashes stay
ai_usage_log           kept; it carries a run reference and a price, and no personal data
agent.policy_decisions arguments_hash stays, principal user_id redacted

A pending approval is cancelled, not redacted. This is the one row where redaction would be worse than doing nothing. The resume path executes the approved call from the approval row, so redacting a pending proposal and then letting a person approve it sends a message whose recipient and body are the string redacted.

approval pending   -> cancel it, then redact.  The Run ends; nobody approves a hollow proposal
approval resolved  -> redact.  The decision, the actor and the timing stay

Erasure and an unanswered approval are the same request from two directions, and cancelling is the only answer that respects both.

Two of the redacted rows are easy to miss, and both hold content rather than a reference.

• agent.approvals stores the exact proposed arguments so a person can decide without opening the run. For email.send that is the recipient and the body.
• agent.idempotency_keys stores the tool response, because the claim is the replay journal. A crm.search response is a list of people.

Redacting the response does not break the journal for any live run: a run still inside its retention window is younger than the erasure request that reaches it, and a replay of a redacted claim returns a redacted result to a run whose subject asked to be erased. That is the correct answer.

Two reasons the timing survives. A deleted span would break the tree that explains what an agent did, and the cost record must still add up for the invoice already issued.

The job runs against one organization or one subject reference, reports what it would touch, and writes only under an explicit apply. It is a scheduled query like the reaper, not a new subsystem.

Every new table that stores a payload joins this list in the same change that creates it. A table that is easy to forget here is the one that leaks.

The test for whether a table belongs here is one question: does it store content, or a reference? agent.runs stores an input summary, so it is here. agent.run_control stores a reason and three IDs, so it is not.

Capability attribution

ENG-2298 owns Run attribution, frozen child execution and bounded operational metrics. ENG-2276 owns the stable start endpoint. ENG-2283 owns stable Front Door routing. ENG-2297 owns routing evaluation fixtures and expected outcomes.

Identity and frozen execution

Each new Run copies capability_id and contract_version from its resolved published binding. Both fields are null for an unbound definition. Old Runs keep null fields; no migration infers identity from current definitions. The two fields are immutable and appear in narrow Run reads, API responses and CLI output. A definition UUID start uses the same attribution path as a capability start. Inputs cannot override identity. Inactive bindings still identify retained executors. Registry activation controls new product selection, not identity.

A new root snapshot freezes all reachable workflow and agent definitions, including their rendered skills, contracts and scope ceilings. Each child entry has its executor UUID, kind, capability binding, execution snapshot and published scope sets. The snapshot format has an explicit version. Missing children in that format fail closed. A legacy snapshot keeps its existing child-start behavior and does not claim frozen capability metadata.

Child starts read the parent's stored snapshot in the same organization. They never use caller-supplied snapshots. Each child copies its frozen subtree. Later binding switches, configuration changes or disable operations cannot retarget it. Current actor rights, parent cancellation, policy and root budgets still apply. Frozen scope sets do not grant new rights. Cycle, missing reference, unsupported format, excessive depth and an oversized tree refuse the start before insertion. Input and result limits remain 32 KiB. A complete execution snapshot has a separate 256 KiB limit. The Signals Search fixture already exceeds 32 KiB when its four agent snapshots are included. Before rollout, a read-only preflight must freeze every active published tenant root and report failures and sizes. An oversized graph requires a smaller published graph before rollout. Do not trim it or silently use live children. A descendant publish can grow an indirect ancestor. Start always checks the complete size, even when publish checked only direct referrers. Publish measures the same complete snapshot. Repeated child references do not trigger repeated definition reads during one freeze.

From ac-python-api, run this command once for each tenant against the target environment:

python -m scripts.audit_capability_snapshots --organization-id <organization-uuid>

The command reads definitions only. It reports each snapshot size and exits with a failure if any tree cannot freeze.

Metric contract

Run rows, results and the canonical usage meter remain the durable facts. Metrics are operational samples, not billing records. Use the existing Sentry metrics transport. Add no metrics database, scheduler, dashboard or public metrics endpoint. Emit a start only after a new row is inserted. A duplicate start emits no second start. Emit completion only for a successful terminal state write. A retry, wait, resume or stale terminal write emits no completion. Metrics are best effort. A crash between a database commit and export can lose a sample; exporter failure cannot fail a Run. Do not claim exact delivery. Durable Run records provide the audit source when samples are incomplete.

Run failure or cancellation takes precedence over output. Runtime partial reasons and product outcome: partial both produce partial. An empty successful search is success with a zero-result sample. Total provider failure is failure. Signals keeps its existing output schema. Missing result counts, field states or usage remain unknown, not zero. Cost export matches the ambient Run organization and root against the usage record. It occurs after the write retry loop. Keep fractional amounts; rounding each call to integer cents loses small charges. Writes without a matching capability scope have unknown attribution and emit no capability cost sample. This includes asynchronous provider settlement without a Run scope. The canonical meter still records those charges. Internal Runs have no capability metrics. Child capabilities receive their own identity, not the parent's identity.

Metric attributes use only the five capability IDs, bounded source kinds, root/child level and fixed outcome enums. Contract versions remain exact on Runs, logs and spans. They are not metric attributes because upgrades can create unbounded versions. Do not include UUIDs, tenant IDs, entity IDs, executor names, error text, schema values or prompts in metric attributes. Structured diagnostic logs may carry Run correlation IDs and the exact version, but never copy input or output payloads. Run scope carries identity into both span APIs and resets it after concurrent or nested execution.

Metric names use the agentic.capability. prefix: starts, completions, zero_results, latency (seconds), cost (USD), enrichment_fields, enrichment_hits and routing. The final Sentry export filter removes inherited request and user attributes from these metrics.

Routing accuracy has correct, incorrect and unscored outcomes. Missing expected routes are unscored. Unknown product IDs and custom definition UUIDs use fixed buckets. Never export the unknown text as a label. A correct capability selection does not imply a successful Run. Routing and execution metrics stay separate.

Alerts

An alert is a scheduled query. V1 ships six, and adds none until scale proves the need.

The stuck run alert carries the same conditions as the reaper, deliberately. It reports what the reaper acted on, so a narrower query would alert on runs the reaper spared on purpose — a run merely queued behind its concurrency lane writes no heartbeat and is not stuck. See runtime execution.

There is no metrics database and no second scheduler. Inngest cron runs the queries, exactly as it runs the reaper.

Run explorer

Prospecting workflow
  |- ok       Find companies      12s
  |- ok       Find founders       35s
  |- error    Find emails         18s   upstream timeout
  \- waiting  Update CRM                approval

The tree comes from one query on root_run_id. The explorer shows the newest attempt by default, and can show earlier attempts and replayed calls on request.

Mission Control stays a separate internal fleet product. The customer facing run explorer stays in the product frontend.

Core code

governance/observability/
  models.py        RunSpan, SpanKind, SpanStatus, UsageRecord, UsageSummary
  tracing.py       run_scope(), current_run_id(), current_root_run_id(),
                   current_span_id()
  recorder.py      SpanRecorder, the scoped context manager.
                   LiveEventPublisher, the live seam it publishes through
  repository.py    RunSpanRepository
  usage.py         UsageMeter
  alerts.py        the five scheduled queries (not built yet)

class RunSpanRepository(Protocol):
    async def open(self, span: RunSpan) -> None: ...
    async def close(
        self,
        span_id: UUID,
        organization_id: UUID,
        *,
        status: Literal['ok', 'error'],
        output: dict | None = None,
        usage_id: UUID | None = None,
        error: dict | None = None,
        attributes: dict | None = None,
    ) -> None: ...
    async def tree(self, root_run_id: UUID, organization_id: UUID) -> list[RunSpan]:
        """Every span of one tree, in (started_at, span_id) order.

        It pages by keyset. max_rows is 1,000 and PostgREST truncates past it
        without saying so, and a discovery tree holds tens of thousands.
        """
    async def close_orphans(
        self, run_ids: Sequence[UUID], organization_id: UUID, error: dict
    ) -> int:
        """The open spans of the Runs the reaper just failed. One statement."""
    async def close_orphans_of_ended_runs(self, error: dict, limit: int) -> int:
        """Every span still running under a Run that has ended. Not scoped to one Run.

        A read with the join, then a write by span id. Never one UPDATE with an
        embedded filter; see the sweep section above for what that closes.
        """
    async def count_of_kind(
        self, run_id: UUID, organization_id: UUID, kind: SpanKind,
        *, exclude_replayed: bool = False,
    ) -> int:
        """The reader of (run_id, kind). It keys on the Run and never the tree.

        An exact count travels in Content-Range, so no row travels at all.

        exclude_replayed drops a span whose attributes carry replayed = true.
        The tool call ceiling needs it and the turn ceiling does not: a replayed
        call made no call, and a replayed model turn really spent the tokens.
        The key is absent on most spans, so the filter is a two member `or`
        rather than a `not.eq`, which would drop every span with no key.
        """

class UsageMeter(Protocol):
    async def record(self, usage: UsageRecord) -> UUID:
        """Writes one ai_usage_log row and returns its id. Not best effort."""
    async def run_usage(
        self, root_run_id: UUID, organization_id: UUID
    ) -> UsageSummary:
        """One call to agent.run_usage(). It reads no span.

        Raises MissingRunUsage on an empty answer or an odd column.
        """
    async def organization_day_cents(
        self, organization_id: UUID
    ) -> DayCost:
        """One call to agent.organization_day_cost(). Accrual reads it.

        Raises MissingDayCost on the same two shapes.
        """

class SpanRecorder:
    def span(
        self,
        *,
        kind: SpanKind,
        name: str,
        input: Payload | None = None,
        attributes: dict | None = None,
    ): ...

organization_day_cents is not here. Its only caller is AccrualChecker, and agent.cost_ceilings does not exist yet, so it ships with accrual rather than ahead of it.

LiveEventPublisher is declared beside SpanRecorder, in recorder.py. RedisRunEventPublisher is the one implementation, in runtime/events/publisher.py. PublishingAgentEventSink is the second caller, for text.delta and tool.updated, which no span produces. RunManager is the third, for the run lifecycle events.

Write cost

The numbers are small enough to keep the design boring, and worth stating so nobody guesses.

one agent run, 30 model turns, 20 tool calls
  spans      1 run + 4 segment + 30 llm + 20 tool  = 55 rows, 110 statements
  heartbeat  2 more statements per span open       = 110 statements
  usage      30 model rows + any metered vendor rows
                                                   = 250 round trips in total

Insert and update are two statements per span, on the same connection. Do not batch a span open across a step boundary. A crash must leave the open span visible, which is the whole point of writing it first.

Count the heartbeat, because it doubles the number. Every span open fires the two conditional UPDATEs above, and ac-python-api reaches Postgres through PostgREST, so each statement is an HTTP request. A conditional statement suppresses the write, not the request: a statement that matches zero rows still costs a round trip. A run that looked like 110 statements is 250.

So the recorder guards in the process. It already knows when it last wrote each row, so it keeps the last write time and skips the call entirely inside the window. One worker owns its Run's row, so that check is exact for it. Many workers touch the root row, so the local check is per process and still removes most of the traffic.

⚠️ There is no second guard in the statement, because PostgREST cannot write one. A filter value is a literal that Postgres casts; it is not SQL that Postgres evaluates. heartbeat_at=lt.now parses, because now is a timestamp input string, and every interval form is refused:

GET /runs?heartbeat_at=lt.now                      200
GET /runs?heartbeat_at=lt.now()-interval'10 seconds'
  -> 400  22007  invalid input syntax for type timestamp with time zone

An application clock cannot stand in for it. The write itself sends the string now, so heartbeat_at carries the database clock, and a dyno running one minute behind would compare a server timestamp against a cutoff a minute in the past. The filter would then match nothing, the heartbeat would never be written, and the reaper would fail a healthy Run at 120 seconds. A guard whose failure mode is a dead Run is worse than no guard.

Losing it costs writes and never costs correctness, which is the opposite of the fear. The process guard only ever skips a write this process already made; a fresh process has no memory and writes at once. So no heartbeat is missed. And the guard is per process, not per Run: one worker holds hundreds of Runs of one tree in one event loop, so five hundred children on ten dynos touch the root ten times a minute, not five hundred.

⚠️ The guard is a map keyed by run id, and never two timestamps. One worker process serves many runs at once, and an email sequence puts hundreds of them in one event loop. Two scalars let the newest run suppress the heartbeat of every other run in the process, and the reaper then fails healthy work. The map holds one entry per live run and one per root, and it evicts the oldest entry past a fixed size, because a worker lives for days. An evicted entry costs one extra round trip, which the SQL guard still answers correctly.

That takes the run above from 250 round trips to about 120.

Indexes:

agent.spans  (run_id, started_at)
agent.spans  (root_run_id, started_at)
agent.spans  (run_id, kind)                          -- the ceiling arithmetic
agent.spans  (run_id) where status = 'running'       -- close_orphans
agent.spans  (parent_span_id) where parent_span_id is not null
agent.runs   (id) where status in ('queued','running')
ai_usage_log (agent_root_run_id) where agent_root_run_id is not null
ai_usage_log (organization_id, created_at)

Three of those were wrong on this page until 2026-08-20, and each was wrong in the same way: a key that reads well and answers no query the platform makes.

Two of those are easy to leave out, because neither serves a screen. Both serve a loop that runs constantly, and both were measured on 200,000 runs and a 27,500 span tree.

• agent.spans (run_id, kind) serves the ceiling arithmetic. A segment computes ceiling - count of llm spans of this Run before it builds its request, and (run_id, started_at) cannot answer it, because the filter is on kind. Without the index that count is a sequential scan, at the top of every segment and every node: 509 buffers against 11, and 8.3 ms against 1.8 ms as an index only scan.
  The leading column is run_id, and it was root_run_id until the ceiling scope was corrected. Four of the five ceilings bound one Run, and only max_cost_cents sums the tree — which reads ai_usage_log, not a span. So no query counts spans by kind across a tree, and (root_run_id, kind) has no reader left. 20260819120200 shipped it, and a later migration drops it and creates the new key. A merged migration is not edited, so the drop is a statement a reader can find.
  The count itself is a HEAD with an exact count, not a read of the rows.Prefer: count=exact with limit=0 answers in Content-Range, so the ceiling costs one index only scan and carries no row back.
• agent.spans (run_id) where status = 'running' serves close_orphans. The key is run_id, not status: the predicate already fixes the status, so a fixed key carries no information and the index degenerates to one value.
• ai_usage_log (agent_root_run_id), and not run_id, which does not exist on that table. The live columns are workflow_run_id, a key into public.workflow_runs, and agent_run_id, a key into the legacy public.agent_runs.
  ⚠️ agent_run_id looks like the obvious column and it must not be used. It carries ai_usage_log_agent_run_uniq, a UNIQUE index, so it holds at most one usage row per Run. A model call writes one row per call, and one Run makes thirty. The second write answers 23505 and metering is terminal, so the Run fails. The new column is agent_root_run_id, it is nullable, it takes no unique index and no foreign key.
• agent.runs (id) where status in ('queued','running') serves the reaper, which runs every minute over a table kept for 13 months. Without it the cron is a parallel sequential scan: 3,461 buffers and 29.7 ms, against 8 buffers and 0.4 ms. The partial predicate keeps the index at tens of kilobytes however large the table grows, because only live Runs are in it, so the reaper reads a few hundred rows and filters heartbeat_at in the heap.
  The key is id, and it must never be heartbeat_at. SpanRecorder writes that column every ten seconds per Run, and an update of an indexed column cannot be a HOT update. A 2,000 row probe with page headroom measured 902 HOT updates with the column unindexed and 0 with it. Indexing the column the hot loop writes would trade the reaper's 29 ms a minute for a new index entry and a dead tuple on every heartbeat.

Evaluation

Production observability and offline evaluation share the span shape, not the run time path. An evaluation run writes ordinary spans and carries its source in the run row. Evaluation may sample production spans as fixtures. The eval_spans concept converges into agent.spans rather than living beside it.

Scenarios that shaped this design

Rules

• Every meaningful unit of work opens a span, owned by the code doing the work.
• A span carries organization_id and root_run_id.
• No durable event table beside spans.
• One run table, one span table, one usage meter.
• The durable write precedes the best effort live event.
• A tracing failure never fails a successful business action.
• Metering is not best effort, and a metering failure is terminal.
• An open span has an owner that closes it, even after a crash.
• A replayed call is recorded and marked, not hidden.
• SpanRecorder writes heartbeat_at between the claim and the finalize: at most once every 10 seconds for the run, and once every 60 for its root. The window is tested in the process, because a PostgREST filter holds a literal and not an expression. RunManager owns the insert and each lifecycle write.
• The process guard is a map keyed by run id, never two timestamps. One worker serves many runs at once.
• A filter value is a literal Postgres casts. It is never SQL Postgres evaluates.
• An index key is the column a query filters on. A key the predicate already fixes carries no information.
• A write never filters on an embedded resource. PostgREST drops that filter and keeps it in the response.
• A read that grows with a run tree pages by keyset. max_rows truncates in silence.
• A sum over ai_usage_log is a read only database function. PostgREST refuses an aggregate.
• ai_usage_log.agent_root_run_id is the agentic column. agent_run_id belongs to the legacy stack and is unique.
• No type in this package is named Context.
• Span payloads are bounded and redacted; prompts are not stored by default.
• Sentry is for our defects. Spans are for agent work.
• A defect on the path of one call reaches Sentry one time, and the log every time. A durable step bounds it per step execution. A component with no durable step holds the report itself. A static defect keeps its entry for the life of the component. A fault that recovers takes a time window, because a fault that alternates makes every failure follow a success.
• An alert is a query on a schedule, until scale proves otherwise.
• Health dashboards are linked, not rebuilt.
• A retention window is not a deletion path. Both exist.
• Erasure redacts a payload and keeps the timing, the tree and the cost.
• A table that stores content joins the erasure list in the change that creates it.
• A new table that stores a payload joins the deletion job in the same change.

Minimum contract tests

• The span context manager completes ok, and re-raises after writing error.
• A parent and child context produce the correct tree.
• A span carries the organization of its run, and RLS refuses a foreign read.
• One query on root_run_id returns the spans of a whole run tree.
• A pub/sub failure leaves the durable span intact.
• A span write failure does not fail the tool call it describes.
• A usage write failure fails the run, and does not trigger a segment retry.
• A replayed tool call writes a span marked replayed with no usage_id.
• The reaper closes open spans when it fails a run with a stale heartbeat.
• The second orphan sweep finds a running span under an ended run without being told which run.
• The second orphan sweep leaves a running span under a live run untouched.
• A tree of more than max_rows spans returns every span, in (started_at, span_id) order.
• A span open that fails leaves the parent span current, and the next child still writes.
• A truncated input keeps attributes.truncated true after the close writes the output.
• A span output that is a string is stored and bounded, and it does not raise.
• Two runs in one process each write their own heartbeat inside one guard window.
• Thirty model calls in one run write thirty usage rows.
• Thirty calls of a currency amount that rounds to zero cents each sum to a non zero total.
• The ceiling count for one Run reads only that Run's spans, not its tree's.
• An oversized tool result is stored truncated, the flag is set, and the stored value still parses.
• A run tree total equals the sum of its usage rows.
• The run explorer rebuilds current state after a disconnect, from the run and its spans alone.
• Erasure of a person leaves no payload in any span, and leaves the span tree walkable.
• Erasure of an organization leaves no row in any table carrying its organization_id.
• A run tree total still sums correctly after an erasure.