Runtime execution

Runtime definitions says what may run. Agentic runtime says which component owns what. This page says how one Run progresses, and how it survives a crash.

StartRunCommand
   -> RunManager           mint Principal, freeze snapshot, create Run
   -> Policy admission
   -> Inngest              function run.execute
   -> RunExecutor
        -> AgentExecutor    -> AgentRuntime -> Agno
        -> WorkflowExecutor -> WorkflowStepExecutor
   -> RunManager           outcome

The Run record

agent.runs is the one product Run table.

agent.runs
  id
  organization_id
  root_run_id          the top Run of the tree; a top level Run points at itself
  parent_run_id        null for a top level Run
  conversation_id      null for a machine Run
  definition_id
  idempotency_key      the start key; unique per organization
  kind                 agent | workflow
  source               front_door | trigger | api | workflow_step
  parent_span_id       the node span that started this Run; null unless workflow_step
  status               queued | running | waiting | succeeded | failed | cancelled
  waiting_on           approval | event | delay; null unless status is waiting
  waiting_ref_id       the approval or the wait node; null otherwise
  waiting_expires_at   when the wait dies; the wait sweep reads it
  snapshot   jsonb     frozen execution snapshot
  principal  jsonb     minted effective authority
  input      jsonb
  result     jsonb     RunResult; bounded
  error      jsonb     RunError
  execution_ref jsonb  Inngest correlation only: function run ID, attempt
  segment_index        the next agent segment to run; 0 for a workflow
  resumed_from_wait    true when the last transition was a wait resolving
  heartbeat_at         last progress write; the reaper reads it
  created_at / started_at / ended_at

parent_span_id is the other half of parent_run_id, and the span tree needs it. A child Run executes in another process, and its run span takes the node span as its parent so the tree keeps one root. The child worker cannot be handed that value at call time, because it reads it after a crash as well, so the Run row carries it. It takes no foreign key: the Run is written before any span of it exists.

root_run_id is set once, at creation. A child copies it from its parent. The live event publisher and the run explorer both read it, so a whole tree needs one subscription and one query. A top level Run points at itself in the same INSERT, which a self referencing foreign key accepts, so creation stays one statement.

Seven invariants are constraints, not conventions. Each one guards a reader that would otherwise be wrong, and none of them is business logic: they are the shape of a row.

status         one of the six product values
waiting shape  (status = 'waiting') = (waiting_on IS NOT NULL)
waiting ref    waiting_ref_id IS NULL OR status = 'waiting'
wait deadline  (status = 'waiting') = (waiting_expires_at IS NOT NULL)
ended shape    (ended_at IS NOT NULL) = (status IN ('succeeded','failed','cancelled'))
tree shape     (parent_run_id IS NULL) = (root_run_id = id)
start key      unique (organization_id, idempotency_key)

The wait deadline is an equivalence and the waiting reference is not, and the difference is deliberate. A waiting Run holding no deadline is a Run the wait sweep can never read, so the weaker rule would let one park for ever. The waiting reference stays weak because a wait node and the first admission mark both file no row.

The waiting reference and the ended shape are the two a review misses. A resume that clears waiting_on and forgets waiting_ref_id leaves the Run pointing at a resolved approval. And the orphan span closer keys on ended_at IS NOT NULL, so a Run with an ended_at and a live status either strands its open spans forever or closes the spans of work still running.

The tree shape costs nothing and catches the bug that is hardest to see later: a child whose root_run_id was copied from the wrong place. Every tree query, every live event channel and the cost ceiling all key on root_run_id, so one wrong copy splits a tree in three places at once.

Five columns are frozen once the Run starts. id, organization_id, definition_id, parent_run_id and root_run_id never change after the INSERT that proves them. A CHECK cannot express this, because each proof reads a second row, so a trigger holds it. The INSERT validates the Run against its definition and against its parent, and freezing the columns stops a later write undoing that proof. It also closes the parent side: a parent cannot move, so it cannot strand a child in another organization, and no descendant scan is needed. root_run_id is set once at creation for this reason.

The start key is the whole duplicate guard. The insert is the claim says why it lives on this row rather than in agent.idempotency_keys.

The tenancy column is tied to the Run, not merely copied from it. agent.spans and agent.sessions carry their own organization_id, because RLS reads it there. Nothing about a copy keeps it true, and a span holds tool arguments, so a wrong copy shows one tenant's data to another. A composite foreign key on (run_id, organization_id) against a matching unique key on the Run makes the mismatch impossible to write. It costs one extra unique index on agent.runs, which measured at roughly 50 MB per million Runs, and it removes a class of leak that no code review can catch reliably.

Two references on this row stay soft, and one restricts. conversation_id and waiting_ref_id point at tables that arrive in later phases, so they are plain UUIDs with no foreign key. agent.spans.usage_id keeps its soft reference permanently: a real key from agent.spans into public.ai_usage_log would tie the isolated schema back to a live table, which is the coupling the schema exists to prevent. definition_id is the opposite case and takes ON DELETE RESTRICT, because the design already gives an admin state: disabled and a Run's audit trail must not be deletable from under it. That key carries organization_id and kind as well, so a Run and its definition always agree on both. A definition with Runs behind it therefore cannot change its kind either.

agent.definitions carries a second unique key. The pair (id, organization_id) sits beside the triple above. agent.policies narrows a rule to one definition, and a policy holds no kind, so it cannot use the triple. The pair is what stops one organization naming another organization's definition.

execution_ref is telemetry, not lifecycle. It carries the Inngest function run ID and the current attempt, so an operator can jump from a Run to its Inngest trace. Nothing branches on it. There is no retrying status: a Run under retry stays running, and the failed attempt is already durable as a failed span.

One small companion table carries the control state.

agent.run_control
  run_id (pk)
  cancel_requested_at
  cancel_requested_by
  cancel_reason

Three columns on agent.runs would be simpler. Do not merge this table. The executor reads it before and after every tool call, about forty times in a normal Run. A Run that nobody cancelled has no row here, so the check is an index only scan that never touches the heap: two buffers, and zero heap fetches. The same check as a column on the Run row is a heap read of the widest table in the schema. The sparseness is the whole point, and it is invisible from the column list, which is why it is written down here.

The buffer count is not the cost this design pays, and reading it as one leads somewhere wrong. ac-python-api reaches Postgres through PostgREST, so each of those forty checks is an HTTP request: five to twenty milliseconds, against a fraction of a millisecond of buffer work either way. The table split is therefore free rather than fast, and it stays because free plus a cancel record that never touches the Run row is better than merged. The number to watch is forty round trips per Run, and the safe boundary rule is what bounds it: before a tool call, after a tool call, and between workflow nodes, and nowhere else. Observability makes the same correction for heartbeat_at: it suppresses the request, not only the write, and for the same reason.

There is no second table for the replay journal. agent.idempotency_keys already stores the claim, the argument hash and the stored response, which is exactly what a restarted segment reads back. The next section explains how.

Result and error

@dataclass(frozen=True)
class RunResult:
    summary: str
    output: dict                  # bounded; large payloads stay in product tables
    refs: list[ResourceRef]       # rows the Run produced
    truncated: bool = False
    partial_reason: str | None = None   # budget_exhausted | limit_reached
                                        #  | approval_expired
                                        #  | approval_not_actionable

@dataclass(frozen=True)
class RunError:
    code: str
    message: str
    retryable: bool
    span_id: UUID | None

A Run that stops on a budget or a ceiling succeeds with a partial_reason, when the Run produced output. It does not fail. Signals Search returns the companies it did qualify. The email sequence keeps the messages it did send.

One rule decides the status of every stop: nothing done is a failure, and something done is a partial success. A stop that produced nothing reports failed. The error carries the clock on RunError.partial_reason, so the record survives the failure and a parent node still reads it. A succeeded outcome with an empty result tells every surface the work was done. This rule prevents that report.

Each executor reads "nothing done" from the record it owns. WorkflowExecutor._to_outcome reads an empty outputs map, because a node that ran leaves a key in it. AgentExecutor._stop_short reads segment_index == 0, because a later segment means an earlier one already worked. Four ceilings reach that second branch: the money ceiling, the turn count, the tool call count and the wall clock. max_segments is the fifth, and it reaches neither. It bounds the loop above the segment, and the function refuses a frozen bound under one. So a Run that spends its segments already ran one, and that ending is always a partial success. An admission approval settles under the same rule from the other side: nothing ran before it, so it ends no Run as a partial success.

The rule is stated here for all of them, and no later section restates it. A section that needs the rule names this one.

partial_reason has four members, and each one names a different reason.budget_exhausted is the money ceiling or the organization day, read from the usage meter. limit_reached is one of the four ceilings a Run owns: turns, tool calls, wall clock or segments. approval_expired is an action approval nobody answered, and the wait writes it rather than the executor. approval_not_actionable is a decision the Run holds and cannot act on. It has two writers, and neither is the executor: the wait, when a person decided after the clock ran out, and the segment loop, when a segment parks again on the decision it was given. The second writes no wait step at all, so a Run that reports this reason may carry no wait.* span. A fifth member would have to answer one of those four questions again.

The last two are not one member, because they say opposite things about a person. approval_expired says nobody answered. approval_not_actionable says somebody did, and the Run could not use their answer. A person whose inbox card reads approved must never read that nobody answered.

A cancellation is not a partial_reason. A cancelled Run ends cancelled, and succeed() is a dash from a terminal status, so a RunResult carrying partial_reason='cancelled' can never be written. It read as a third member of the set and it was unreachable, which is worse than absent: it invites an executor to build a result nothing will store.

output never holds a large payload, and that is enforced rather than asked for. It passes bound(output, 32 KB), the one payload boundary, and sets truncated from what that returns. RunManager is the caller, not the executor, so it rebuilds the frozen RunResult with the bounded value and the flag it got back. The algorithm is defined once in the platform contract and restated nowhere.

Discovery writes intelligence rows and prospect rows, and refs points at them.

refs passes the same boundary, because it grows with the work exactly as output does. Bounding output alone bounds nothing: a five hundred person email sequence writes five hundred refs beside it, in the same result column, on the widest table in the schema. refs is a list and bound() takes a list, so it is the same call at the same 32 KB. The head survives, the tail becomes a Dropped marker, and RunResult.truncated is true when either half changed. summary needs no ceiling: it is one sentence the executor writes, not a set that grows.

The Run's input is refused rather than bounded, and that changed on 2026-08-21. It was a bound() caller at 32 KB, to keep a 5 MB trigger payload out of a row that is kept for thirteen months. Trimming it answers the storage question and creates a worse one: input is the caller's own instructions, so a drop makes the agent act on part of a request and report success, and no field on the row records the loss, because truncated describes the output. So RunManager refuses the start with input_too_large. The rule for a caller is unchanged and now enforced: a large payload passes a ResourceRef and leaves the body in the product table that owns it.

The complete snapshot has a separate 256 KiB limit. A check refuses it; bound() never trims it. The helper drops whole top level items, and a dropped item here is the frozen tool contracts or the rendered skill text. The Run would then execute an agent that lost a tool it was published with, with nothing raised and no way for a reader to tell. Validation refuses an oversized snapshot at publish, and SnapshotBuilder.freeze() fails the start with snapshot_too_large, which is a member of the closed StartRunResult set like every other refusal. Publish validation makes it rare rather than impossible: a definition published before the size rule existed still reaches freeze, and so does one whose rendered skill text grew after it was published. See runtime definitions.

No list query selects snapshot, input or result. The run explorer, the reaper and the tree query all read narrow column sets. One SELECT * on a list endpoint pulls three jsonb columns per row out of TOAST, and the page that was fast in testing is slow on a real tenant.

Run therefore has two read shapes, and StartRunResult.run is the narrow one. claim() hands its Run to the executor, which needs snapshot to rebuild the agent, so that read is wide by necessity. start() hands its Run to an API response, a trigger log and a workflow node, and none of the three opens snapshot. Return one shape for both and every start pulls three jsonb columns out of TOAST for a caller that reads six scalars. Every duplicate does too, and that is the common answer on a redelivering source. One model with an optional wide half, loaded only where it is named.

Shared start contract

The front door and the triggers use exactly one contract.

@dataclass(frozen=True)
class StartRunCommand:
    definition_id: UUID
    input: dict
    actor: ActorIdentity          # user or service identity, not a Principal
    source: RunSource
    idempotency_key: str
    conversation_id: UUID | None = None
    parent_run_id: UUID | None = None

input is a product mapping with two optional shared keys. text supplies the agent text. refs supplies a list of ResourceRef shaped objects that are already in scope. Other keys remain product input.

The executor copies valid refs entries into ContextRequest.entities. It drops a malformed entry from the context request and leaves the stored input unchanged. A source decides which registered ref kinds it supports.

The organization comes from the actor, not from the command. ActorIdentity carries organization_id, because the day cap gate and the start key both need it before a Principal exists, and the Principal is minted three steps later. Putting it on the command as well would give one fact two writers. See policy and governance.

start() answers with a closed set, never an exception, because four different callers branch on it: the front door, a trigger, a workflow node and the API route.

@dataclass(frozen=True)
class StartRunResult:
    outcome: Literal[
        'started',                        # a new Run; dispatched unless it is a child
        'duplicate',                      # this key already started a Run; run is that Run
        'parent_cancelled',
        'organization_budget_exhausted',
        'definition_not_found',           # missing, or another organization's
        'definition_not_published',       # a draft; a disabled definition with no
                                          # parent; or a skill, which never runs
        'snapshot_too_large',             # published before the size rule, or a skill grew
        'snapshot_unbuildable',           # the deploy moved out from under a published
                                          # definition: a tool the registry no longer holds
        'input_too_large',                # the caller's input is over 32 KB
        'idempotency_conflict',           # capability key reused with changed input
        'capability_unavailable',         # selected capability binding changed
        'capability_unauthorized',        # required capability scope is absent
        'policy_unavailable',             # the day cap gate could not decide: the meter did
                                          # not answer, or the rule set did not read
    ]
    run: Run | None = None                # set for 'started' and 'duplicate'
    reason: str | None = None             # set for every refusal

duplicate is a success for every caller. It carries the Run the first delivery created, so a trigger that fires twice reports one Run rather than an error.

started means the Run row exists. It does not mean the Run executes. Policy admission runs after the insert, so a started Run reads queued, or waiting on an admission approval, or failed on a policy denial. The set holds no policy_denied member for that reason: the denial has a Run row, and the Run carries the answer. Only the day cap refuses before a Run exists. The caller reads run.status, never the outcome alone.

An unavailable definition is two outcomes, because it needs two statuses. Another organization's definition must answer exactly as a missing one does, or the response tells the caller which identifiers exist. The caller's own draft or disabled definition is the opposite case: the Builder lists it on the next tab, so one answer that means "no such definition" reads as data loss. Surfaces maps the first to 404 and the second to 409.

A skill takes the second answer, and start() is what refuses it. agent.definitions.kind holds agent, workflow and skill; agent.runs.kind holds the first two. So a start against a published skill reaches the insert and raises 23514 from a method that promises a value and never an exception. start() refuses a resolved definition whose kind is not agent or workflow, and answers definition_not_published, with the reason naming the kind. 409 is right for the same reason a draft takes it: the Builder lists the skill, so 404 would read as data loss. The name is loose here and the status is not, and one more member in a set that four callers branch on costs more than the looseness.

The caller supplies an actor, not a Principal. RunManager mints the Principal, because the intersection needs the definition and the Run ID, and the caller has neither. See policy and governance.

PrincipalFactory is a seam of RunManager, exactly as the resolver and the snapshot builder are. It is named in the start flow and agent.runs.principal is NOT NULL, and the intersection itself belongs to the policy plane. So RunManager takes it as a protocol, and governance/policy/principals.py implements it. It is given the declared names and not the definition, because ResolvedDefinition lives in runtime and src.agentic.governance may not import src.agentic.runtime.

Four seams, five calls. PolicyGate is one protocol with two methods, because the policy plane is one owner and swaps in one commit. The other three are called once each.

PolicyGate.check_day_cap(actor)                        # before the resolve
DefinitionResolver.resolve(definition_id, actor)       # tenancy only
PrincipalFactory.for_run(definition_id, declared_scopes, actor, run_id)
SnapshotBuilder.freeze(definition)
PolicyGate.admit(run, principal, definition, arguments) # after the insert

start() calls them in that order, so no later phase reshapes the flow. Each is a protocol, and every one of the four has its real implementation. Definitions owns the resolver and the builder. governance/policy/principals.py owns the factory, and runtime/runs/admission.py owns the gate.

PolicyGate lives on the runtime side of the plane boundary, and its two arguments say why. admit() takes a Run and a ResolvedDefinition, and both live in runtime, so PolicyEngine cannot satisfy this protocol. AdmissionGate holds the engine, the accrual checker and the decision log, and it decides nothing of its own except the fatal scope set.

admit() takes the arguments, and the Run does not carry them. Run is the narrow read on the start path, so the row holds no input. Without this argument a rule reading the reserved arguments root saves and never fires. The gate writes the run fact root from the row it is given, because PolicyEngine writes principal and arguments alone.

The Run ID is minted in Python, before the insert. PrincipalFactory.for_run() needs it, and root_run_id points at the Run's own id for a top level Run. Both facts want the id before the statement runs, and runs_tree_shape refuses the row unless the two agree, so RunManager writes id and root_run_id in the same insert rather than reading a database default back.

parent_run_id is how a workflow node starts a child Run. There is no second start method and no private synchronous path.

parent_run_id also decides how a disabled definition is treated. Disable stops a new Run tree; it does not stop a tree that was admitted while the definition was still active. A workflow can wait three days and then start its next agent node, and failing that child would stop a Run nobody asked to stop.

parent_run_id IS NULL      draft -> refuse    disabled -> refuse
parent_run_id IS NOT NULL  draft -> refuse    disabled -> allow

assert_run_shape() already encodes this, and it raises for a disabled definition only when parent_run_id IS NULL. RunManager runs first, so whichever component refuses every disabled definition wins and the trigger's carve-out never fires. start() must carry the same asymmetry, or disable becomes a stop button for work already in flight. A draft is refused on both sides, because it holds no published_config and there is nothing to run.

The asymmetry lives in start(), and not in DefinitionResolver, because of what the seam returns. The resolver answers a definition or nothing, and nothing means definition_not_found. A draft, a disabled definition at the top level and a skill all answer definition_not_published, so a rule that lived in the resolver would turn a 409 into a 404 on a definition the Builder lists on its next tab, which is the answer this page already calls data loss. So the seam answers one question, does this actor's organization own a definition with this id, and it takes no parent_run_id. start() applies the three publishability rules to what comes back. A later resolver that needs to refuse for its own reasons needs a wider return type first, not a rule smuggled through None.

No caller outside the runtime supplies it. start() sends the dispatch event only when parent_run_id is null, so a Run created with one and no invoking parent is never claimed, and the reaper fails it two minutes later. The API start route therefore refuses the field rather than passing it through, and parent_cancelled never reaches an HTTP caller.

RunManager

RunManager is the only component allowed to create or control a Run.

class RunManager:
    async def start(self, command: StartRunCommand) -> StartRunResult: ...
    async def claim(self, run_id: UUID, execution_ref: dict) -> Run | None: ...
    async def mark_waiting(self, run_id: UUID, waiting_on: WaitingOn,
                           *, expires_at: datetime,
                           ref_id: UUID | None) -> None: ...
    async def resume(self, run_id: UUID) -> ResumeOutcome: ...
    async def advance_segment(self, run_id: UUID, from_index: int) -> None: ...
    async def succeed(self, run_id: UUID, result: RunResult) -> None: ...
    async def fail(self, run_id: UUID, error: RunError) -> None: ...
    async def cancel(self, run_id: UUID, reason: str, actor: ActorIdentity) -> None: ...

ref_id is optional because a delay wait points at no row. cancel() takes an actor, not a Principal: a Principal is scoped to one Run, and the person who stops a Run is often not the person who started it. agent.run_control.cancel_requested_by stores that actor.

One transition primitive, and one table of legal moves

Every method above is the same conditional UPDATE, and none of them is safe unconditionally. A worker that finishes in the same instant as a cancel would otherwise write succeeded over cancelled, and the person who stopped the Run would watch it succeed.

UPDATE agent.runs SET ... WHERE id = :run_id AND status IN (:legal_from_states)

Zero rows means the Run moved on. The caller stops, and it does not raise: another writer owns that Run now. This is the same fence idempotency puts on a lease token, for the same reason.

A dash is a no-op. A terminal Run stays terminal. succeed and advance_segment exclude queued. resume and admit accept only waiting.

admit releases the temporary capability admission hold. Its caller also requires waiting_on = approval and waiting_ref_id IS NULL. It clears the wait fields and dispatches only if the update returns a row. A cancellation or recovery that wins the race prevents dispatch. See Capability start identity.

NON_TERMINAL = frozenset({'queued', 'running', 'waiting'})

LEGAL_FROM = {
    'admit':           frozenset({'waiting'}),
    'claim':           NON_TERMINAL,
    'mark_waiting':    NON_TERMINAL,
    'advance_segment': NON_TERMINAL - {'queued'},
    'resume':          frozenset({'waiting'}),
    'succeed':         NON_TERMINAL - {'queued'},
    'fail':            NON_TERMINAL,
    'cancel':          NON_TERMINAL,
}

A table written cell by cell holds forty-eight answers, and a terminal cell filled in by hand is a Run that un-ends. Written as a subtraction from NON_TERMINAL, "a terminal Run stays terminal" is structural: no entry can name a terminal status, because none of them starts from the whole set.

The primitive clears the waiting three, so no method has to remember to. This page says three separate times that waiting_on = NULL, waiting_ref_id = NULL, waiting_expires_at = NULL is the line a review misses, which is a sign that five call sites is four too many. One rule replaces the repetition, and it takes the same shape the table above takes.

the write names a target status, and it is not `waiting`  -> clear the three
the write names `waiting`                                 -> mark_waiting sets the three
the write names no target status                          -> touch neither column

admit(), claim(), resume(), succeed() and fail() all take the first line, and none of them repeats it. mark_waiting() is the second. advance_segment() is the third and the only one: it moves segment_index, it leaves status alone, and a Run that is waiting must stay pointed at the row it waits on. One rule and one exception, rather than five statements each carrying two lines that must never be dropped.

advance_segment() is a lifecycle write like every other one, and it is guarded the same way. It drops queued for the same reason succeed() does. A segment runs only after claim(), so a queued Run has no segment to advance and the cell is unreachable.

mark_waiting from queued is the admission approval, which happens before the claim. mark_waiting from waiting is a second wait replacing the first, which a parallel node produces.

claim() is re-entrant, and it clears the waiting three.

UPDATE agent.runs
   SET status = 'running',
       waiting_on = NULL,               -- the waiting shape constraint rejects the row otherwise
       waiting_ref_id = NULL,
       resumed_from_wait = false,
       execution_ref = :execution_ref,
       started_at = COALESCE(started_at, now()),   -- a retry must not reset the wall clock
       heartbeat_at = now()
 WHERE id = :run_id AND status IN ('queued','waiting','running')

Two lines in that statement are the ones a review misses. claim() moves a Run out of waiting after an admission approval, so it must clear waiting_on; leave it and the waiting shape constraint rejects the update and the Run never starts. And started_at is set once: write now() and every Inngest retry restarts the wall clock, so max_run_duration bounds nothing on the Run that needs it most.

An Inngest retry claims the same Run again. A duplicate delivery finds the Run already running and continues from the memoized steps. Do not write claim() as a one way queued -> running transition. Every retry would fail.

A cancelled, succeeded or failed Run does not match the WHERE, so claim() returns None and the executor stops without an error. That is why the return type is nullable: a Run cancelled between the dispatch and the claim is an ordinary outcome, not a fault.

Start flow

StartRunCommand
  -> refuse when the parent Run reads cancelled   (the status, not the control row)
  -> accrual gate: the organization day cap
       deny -> refuse before any Run exists; agent.policy_decisions with run_id null
  -> resolve the active published definition
  -> PrincipalFactory.for_run(definition_id, declared_scopes, actor, run_id)
  -> SnapshotBuilder.freeze(definition)
  -> INSERT agent.runs, status=queued ... ON CONFLICT (organization_id,
                                                       idempotency_key) DO NOTHING
       inserted -> a new Run
       conflict -> read the row and return it as 'duplicate'
  -> Policy admission: grant, rules, and the run budget
       allow            -> send the Inngest dispatch event
       deny             -> fail the Run with the policy reason; never dispatch
       require_approval -> hold the Run, create the approval, aim the Run at it,
                           then send the same Inngest dispatch event

Capability starts add two steps to this generic flow. They first check for a stored request with the same caller, tenant, source and key. A matching digest returns the stored Run; a changed digest returns idempotency_conflict. A new capability Run is inserted in the temporary waiting admission hold. Policy allow must release that hold through admit before dispatch. A crash before release leaves an undispatched Run for the wait sweep to fail.

An admission approval is held before its row is written

require_approval makes four writes, and the order decides what a crash costs.

1  mark_waiting(run, 'approval', None, expires_at=now())
2  ApprovalService.create(proposal, ttl, run_deadline=None)
3  mark_waiting(run, 'approval', approval.id, expires_at=approval.expires_at)
4  the Inngest dispatch event

The hold comes first. Write the row first and a crash before the hold leaves a queued Run holding an orphan approval. The retry of that delivery reads the start key, answers duplicate and returns. The reaper then re-dispatches the queued Run, and it executes with nobody having approved it. That is a policy bypass, and nothing reports it: the person's inbox card resolves into no waiter.

Hold first and every crash leaves the Run waiting and undispatched. A waiting Run is a stall the wait sweep ends, and never a start nobody approved. runs_waiting_ref_shape holds waiting_ref_id to a waiting row and admits a null one, so step 1 is a legal row. mark_waiting() takes every non-terminal status, so step 3 is a re-aim and not a second statement.

A waiting Run whose waiting_ref_id is null is a real state. It means the approval was never written. The dispatch is the last write, so no function run ever sees it. The wait sweep is its only reader, and it ends that Run under wait_abandoned. Step 1 writes now() as the deadline, so that ending arrives one grace window later and not one approval TTL later.

The approval write can raise, and start() may not. Four callers branch on a closed answer, and one of them is an Inngest step. So the write is caught and the Run ends under approval_unwritable. The Run already reads waiting, and LEGAL_FROM['fail'] takes that status. The code is a platform stop: a rule asked for a person, the database refused the row, and nobody decided anything.

The day cap is checked before the expensive work

Admission has two halves, and they sit either side of the Run row on purpose.

Five hundred triggers firing against a spent day cap is the case that decides this. Check the cap after the freeze and each one resolves a definition, mints a principal, freezes a snapshot and writes a Run row, only to be refused. Check it first and each one costs one indexed query.

A refusal here creates no Run. There is nothing to attach it to, so the denial is recorded as a agent.policy_decisions row with run_id null, exactly as the front door records a usage row with run_id null for a turn that started no Run. RunManager returns organization_budget_exhausted, and triggers counts it as a budget skip.

The two calls sit either side of the insert, and each writes at most one row. The day cap half writes nothing when it allows: admission records one row for each run and the engine writes that row after the insert, so a row here would double the count for every start. A refusal writes the one row this log holds with a null run_id.

A fault is not a spent cap. AccrualChecker answers deny under metering_unavailable when the meter did not read, and PolicyEngine answers deny under policy_unavailable when the rule set or the live rights did not read. Read as organization_budget_exhausted a pooler outage refuses every start of that moment as a spent day cap, the API answers 429 with a Retry-After measured for a budget, and a workflow node tolerates it. So the day cap half answers policy_unavailable, which is a 503, and the admission half ends the run under the fault code itself.

That is the one denial with no Run behind it. Every other admission outcome has a Run row, because everything else needs the definition to decide.

The insert is the claim

Two writes cannot both happen here, and one of them must carry the duplicate guard.

resolve -> freeze -> INSERT agent.runs
                          ✗ the worker dies here
                     complete a claim in a second table   never runs

A separate claim table needs those two writes in one transaction, or the retry inserts a second Run. Both would be valid, and both would dispatch: one Slack message, two runs.

So the Run row carries its own start key, and the insert is the claim.

INSERT INTO agent.runs (organization_id, idempotency_key, ...) VALUES (...)
ON CONFLICT (organization_id, idempotency_key) DO NOTHING
RETURNING id

One statement is atomic on its own, so there is no transaction to lose. RETURNING an empty set means another delivery already created the Run; read that row and answer duplicate. A crash before the insert leaves nothing, and the retry starts exactly one Run.

Three things disappear with the second table, and each was carrying real weight.

• No processing state, so no start_in_progress. A claim in a second table has three answers, and the third one is a race: the claim exists, the Run does not, and the caller can be told nothing useful. Here the row and the guard are the same row, so a duplicate always reads a committed Run.
• No transaction, so no second database driver. ac-python-api reaches Postgres through PostgREST, which cannot span two statements. A start that needed a transaction would need an asyncpg pool beside it, and RLS would stop applying on that path.
• No complete() and no release() on the start path. A refusal after a claim would otherwise strand it in processing, and every refusal branch would need a release that is easy to forget. Here a refusal writes nothing at all.

agent.idempotency_keys protects tool.<name>, webhook.<provider> and the bounded surface.saved_search.start input freeze. Tools and webhooks protect an effect that is not a row we own. The saved-search scope freezes mutable product input before admission. None claims the Run. A Run start protects a row we own, and its unique index remains the admission guarantee. See idempotency.

A duplicate sends no dispatch event, and that is a rule rather than an omission. The stable event ID dedupes for 24 hours, so a redelivery on day two would reach Inngest as a new event and start a second worker on a Run that may still be running. claim() is re-entrant, so neither worker refuses. The recovery path for a send that never landed is the reaper, which re-dispatches a queued Run and fails it on the next pass, and it is the only recovery path. Reading started -> dispatch and adding duplicate -> dispatch too, in case the first send was lost is the natural mistake, and it is the one this paragraph exists to stop.

Two consequences follow from the insert carrying the guard, and both are accepted.

• A duplicate pays for the resolve and the freeze it then throws away. The window is milliseconds and the work is reads, so this is cheaper than the transaction it replaces.
• The start key is unique for the life of the Run row, not for 24 hours. Every source the design names is already globally unique: a Slack message ID, a provider delivery ID, a trigger:<id>:event:<id> pair, and a client request ID, which is the one the caller writes. A caller that reuses a key gets the first Run back, which is what the key asked for.

The key names the delivery, not the command. It must satisfy two rules at once, and they pull in opposite directions.

same intended start  ->  same key      a redelivered Slack message
next intended start  ->  new key       tomorrow's run of a daily trigger

A key that carries only the command satisfies the first rule and breaks the second: a schedule sends an identical command every day, so day two collides with day one and never runs. The row is kept for thirteen months, so that collision does not age out. A key that carries a timestamp or a random value satisfies the second and breaks the first, which is no guard at all.

Every source the design names already answers both, because each carries a delivery identity: a Slack message ID, a provider delivery ID, trigger:<id>:event:<id>, and a client request ID. The first three are minted by the platform. The fourth is caller text, and the next rule is what contains it.

RunManager hashes the source and what the caller supplies, and stores the hash. agent.runs.idempotency_key is CHECK (char_length BETWEEN 1 AND 255), because an oversized value fails the unique btree with index row size exceeds btree version 4 maximum. One of the four sources is caller text: an HTTP Idempotency-Key header is whatever the client sends. A hash gives every key one width, so no caller can reach the check, and the value stays a pure function of the source and the delivery identity, which is all the key promises.

The source is part of the key, and the caller text is why. The four sources share one unique index, (organization_id, idempotency_key), and only three of them are minted by the platform. Hash the delivery alone and a person sends Idempotency-Key: trigger:42:event:2026-08-22 from the API. That is tomorrow's key for trigger 42 in the same organization. Tomorrow the trigger inserts, conflicts, reads back a Run it did not start, and reports duplicate. The scheduled Run never executes and nothing raises. So the digest covers <source> and the delivery identity together, separated by a NUL, and the four namespaces cannot meet.

The guard is the source, not the separator. RunSource.kind is a closed set of four values and the platform chooses it, so the prefix is fixed before any caller text is appended and the join is injective. The NUL is a second line of defence: it cannot appear in an HTTP header, and Postgres refuses it in text. A source that ever took caller text would lose the guard, and the separator would not save it.

The hash is sha256, hex encoded. Python's built-in hash() is salted per process, so two dynos would produce two keys for one delivery and the guard would stop guarding with nothing raised. A stable, named digest is the only kind that works here, and 64 hex characters sit well inside the 255 the check allows.

The conflicting delivery waits for the first one to commit, so the read-back always finds the row. ON CONFLICT DO NOTHING blocks on an in-progress insert of the same key and returns only after that transaction ends. Measured on the local stack: the second statement returned an empty set after 4.0 seconds against a 4 second first transaction, and the following select read the committed Run. So the duplicate branch needs no retry, and twenty concurrent starts answer one started and nineteen duplicate. Each start is its own PostgREST request and its own short transaction, so the wait is the length of one insert.

The definition is never in the key. A duplicate Slack delivery re-runs the front door, and the model may select a different definition the second time. Fold definition_id in and the same message starts a second Run under a different definition, which is the exact thing the key exists to stop.

A cancelled parent refuses its next child

RunManager.cancel() writes a control row for the Run and every descendant that exists. A node that starts a child immediately after that write creates one with no row.

node N reads agent.run_control                  -> clear
        cancel lands, and enumerates the descendants it can see
node N calls start(parent_run_id=P)       -> the child would run to completion

Safe boundaries sit between nodes, so the window is small. It is not zero, and the cost is a real send from a Run a person stopped. start() therefore reads the parent Run status and refuses with parent_cancelled. It is one primary key read on a path that already does four.

That read returns three columns, not one. The child copies root_run_id from its parent, and runs_tree_shape plus assert_run_shape() both refuse the row when the two disagree. The parent's organization_id must match the child's for the same reason. So the one read selects status, root_run_id and organization_id together. Written as a status check alone it becomes two reads of one row, and the second one is added by whoever discovers the constraint.

Refuse on cancelled, and on nothing else. A parent that reads waiting is ordinary: a parallel node holds one branch on an approval while another branch starts its child. A rule of "refuse unless the parent is running" therefore breaks the container the design added waits for.

It reads the status, not the control row, because the cancel writes the status first. The two live one statement apart, so for that instant a cancelled parent already reads cancelled and still has no control row. A child admitted in that window would run to completion. The split also matches how often each caller reads: start() runs once per child and can afford the Run row, while the executor reads the sparse control row about forty times per Run and must never touch the heap.

The dispatch event carries a stable ID, so a retry inside RunManager cannot start a second worker on one Run.

await inngest.send(Event(
    name='agent/run.execute',
    id=f'run.execute:{run_id}',        # Inngest drops the duplicate send
    data={'run_id': str(run_id), 'organization_id': str(org_id), 'lane': lane},
))

claim() is re-entrant on purpose, so it would admit that second worker. The event ID is what stops it arriving.

The Run row exists before any wait. The product then has one durable unit to show, to cancel and to resume.

A child Run is invoked, never dispatched

start() dispatches a top level Run. It must not dispatch a child.

A workflow node starts its child with the same StartRunCommand, carrying parent_run_id, and then calls step.invoke so the parent step holds the child result. If start() also sent the dispatch event, the child would execute twice: once from the event, once from the invoke. Two workers would claim the same Run, and claim() is re-entrant, so neither would refuse.

result = await run_manager.start(command)     # parent_run_id set -> no dispatch event
outcome = await step.invoke('node.research', function=run_execute, data={
    'run_id': str(result.run.id),
    'organization_id': str(result.run.organization_id),
    'lane': 'batch',                          # every child is batch; see the lanes above
})

The invoke payload carries the same three fields the dispatch event does, and it is not decoration. run.execute keys its concurrency on event.data.organization_id + ":" + event.data.lane, and an invoked function reads event.data from exactly this dictionary. Send run_id alone and both fields are absent: every child Run in the platform falls into one bucket, and the batch lane stops existing for the runs that most need it. The Cancel expression reads event.data.run_id and would keep working, which is what makes this fail quietly rather than loudly.

A child is always batch. The lane follows the actor kind, and a workflow_step child is machine work whatever started the tree. A child that inherited an interactive lane would let one person's workflow fan out five hundred children into the budget other people are waiting in.

The rule is one line: start() sends the dispatch event only when parent_run_id is null. The caller does not choose. A caller-supplied flag is one refactor away from a child that dispatches.

An admission approval on a child still works. The invoked function waits before its first segment, exactly as a dispatched one does, and the parent step waits with it.

An admission approval dispatches too. It looks wasteful, and it is the only way the approval gets an expiry owner. The wait timeout is the writer that resolves an approval to expired, so a Run that waits without a waiter never ends. The dispatched function waits before its first segment, holds no worker while it waits, and costs nothing until a person answers.

The Inngest boundary

This is the part that decides what a crash costs.

One Inngest function runs one Run.

@inngest_client.create_function(
    fn_id='run.execute',
    trigger=TriggerEvent(event='agent/run.execute'),
    cancel=[Cancel(event='agent/run.cancelled', if_exp='async.data.run_id == event.data.run_id')],
    concurrency=[Concurrency(
        key='event.data.organization_id + ":" + event.data.lane',
        limit=settings.run_concurrency,
    )],
)
async def run_execute(ctx: Context) -> dict: ...

Two lanes, so a batch cannot starve a person.

lane = interactive     an interactive user actor: front_door, api
lane = batch           a machine actor, or any child: trigger, workflow_step

The lane follows the actor, not the source alone. api was interactive when the API meant a person pressing Run. It is not interactive when a script or an outside AI agent starts five hundred runs on a user seat, and those runs would then share the budget a person is waiting in.

ActorIdentity.kind decides it. Today that is user or trigger, so api under a user actor stays interactive, and a user's own bulk job contends only with itself. When machine identity exists, an api run under a machine actor takes the batch lane with no further change here. See agent access.

One key on organization_id alone lets a 500 person email sequence fill the whole budget, and a person waiting in chat then queues behind it. It is Inngest configuration, not a second limiter.

The lane is a column on agent.runs, and not only a field on the event. The reaper re-dispatches a queued Run whose event never landed, and it must rebuild that event. The row holds no actor — only source — and source gives the same answer as the actor only until machine identity exists. A reaper that derived the lane would then repair machine runs into the lane a person waits in, and no test would fail. RunManager writes the column at the insert, from the same value the dispatch event carries, so the two cannot disagree. The default is batch: a value nobody wrote must not read as interactive. It takes no authenticated grant, because queue state is infrastructure telemetry.

Inside it, every unit that costs money or touches the world is its own step.run. Inngest memoizes a completed step, so a retry replays the result instead of doing the work again.

⚠️ A step handler must return a JSON value, and the pseudocode on this page does not. The SDK memoizes what a handler answers, so no step may return a Run or an ExecutionOutcome. Each one returns a mapping and the loop reads it back. Measured while building ENG-2077: step.run's own signature says the handler "MUST return a JSON-serializable value". Two consequences are easy to miss.

• The claim cannot carry the run. A wide Run holds the frozen snapshot, so memoizing it puts the whole prompt surface into Inngest step state on every claim. The claim answers the few fields the loop branches on, and the segment re-reads the row — which it does anyway, because two columns move between segments.
• dataclasses.asdict is not a serializer. It leaves a UUID a UUID, and RunError.span_id is one. A segment that failed with a span attributed to it then fails to serialize after the work was done, and the retry redoes the segment for the same ending.

⚠️ A workflow walk may not sit inside a step of this function. The SDK refuses a nested step and answers STEP_NESTED, which is not retriable, so the first node step of the first workflow run would fail that run for ever. run.execute therefore reads kind off the memoized claim and branches above every step: an agent run enters the segment loop, and a workflow run walks between the claim and the finalize. Measured on the Python SDK: ReportedStep.__aenter__ raises NestedStepInterrupt and step.run converts it to the coded error.

The claim answers the kind for the same reason it answers max_segments — the branch must be decided before any step exists, and a value re-read between two passes could move.

The function answers three keys, and a parent Run reads them. A workflow agent node maps this mapping onto its own node outcome: result.output becomes steps.<node id>.output, which is what a later node resolves a reference against, and error becomes the node failure.

{"status": "succeeded", "result": {"summary": ..., "output": {...}}, "error": null}

Answering the status alone empties every steps.<id>.output in the tree and replaces every child failure with child_failed. The exit that claims nothing answers the same three keys, with an error code naming that the Run had already ended.

A child Run uses step.invoke. The parent step stays durable while the child runs, and the child result comes back to the parent. The parent does not poll.

This depends on one Inngest guarantee, so state it. Inngest concurrency limits active code execution, not function runs: a run suspended on step.invoke, step.wait_for_event or step.sleep holds no slot. A parent that invokes a child therefore releases its slot while the child executes.

If that were not true, a workflow tree would deadlock at the limit. Ten parents at a limit of ten would each hold a slot, waiting for ten children that could never get one, and a subworkflow nests three deep. The deadlock would appear under load and not in testing, so the guarantee is pinned by a contract test rather than trusted.

Concurrency and rate limits come from Inngest flow control, configured from the Policy limits. The runtime owns no second limiter.

⚠️ The Run's own span needs an open call and a close call in two steps. A recorder that offers one context manager cannot open it: the SDK re-executes the function body once per step, so a span opened outside a step is written again on every replay, and a block that closes where it opens cannot wrap the loop. The workflow coordinator solves this with wf.run and wf.end steps, and the agent function needs the same shape. run_scope() is entered regardless, because it is contextvars alone and costs nothing to re-enter.

Every Inngest function runs on the worker. run.execute is not special, and neither is a reaper that finishes in 200 milliseconds. The platform draws no line between a short function and a long one, because that line has to be judged for every new function and it eventually gets judged wrong. The web process sends events and serves the API, and it serves no Inngest function. See deployment topology.

The worker holds an Inngest Connect session, so no HTTP router timeout bounds a step. The platform must therefore set its own step limits. One session serves two Inngest apps: the live stack's functions move onto it in Phase 1 rather than waiting for cutover. See one session, two apps.

A deploy still bounds a step, and no router is involved. The session drains the in-flight step on SIGTERM, but the platform kills the dyno 30 seconds later, so any step still running is cut. The segment is the largest step we run and it is the one this hits. Completed steps stay memoized, and the run resumes at the first unfinished step. A completed journal entry stops the repeated effect of its matching tool call. An interrupted entry keeps the journal's lease behavior. The cost is latency, measured at roughly two minutes before the run is re-dispatched. Treat a long segment as restartable, not uninterruptible. The drain is what makes a restart happen; it is not the ceiling on a step. The step budget below is that ceiling, and it is set from this measured cost.

The step budget is a ceiling we choose, and it is not the drain. The drain is a platform fact of 30 seconds. The budget is the longest step this platform declares, and one rule sets it: never lose more work to a deploy kill than the restart costs. Re-dispatch is measured at roughly two minutes, so the budget is two minutes.

STEP_BUDGET_S = 120

src/agentic/shared/ceilings.py holds it. This page owns the number, and every other page links here rather than restating it. It is a Final constant and not an environment variable: publish validation reads it, so a definition that publishes on staging must publish on production.

Two limits sit inside the budget:

ToolSpec.timeout_s       one tool call                              enforced
MAX_SEGMENT_DURATION_S   one agent segment, the largest step we run  90 seconds

A tool whose timeout_s exceeds the step budget fails registry validation. See tools and integrations.

The segment wall clock leaves 30 seconds for cancellation, session cleanup, span cleanup, and the step response. The durable segment step wraps the existing executor in one asyncio.timeout(90). Segment.exhausted() still reads max_run_duration, which bounds the whole Run. The deploy clock is not a second Run ceiling, so RunCeilings does not carry it.

When this timer expires, the step answers an ExecutionOutcome whose next_status is running. It carries no result, error, wait, or partial reason. The loop advances segment_index in its own durable step and starts the next segment without a wait. A nested TimeoutError is still a fault: the step treats it as a yield only when its own timeout object reports expired().

Agent segments and the replay journal

Agno runs its whole loop in process. We cannot put each model turn in its own step.run without fighting the framework. So one segment is one step, and the segment can restart.

A restarted segment is a problem. The model is not deterministic, and it mints a new tool call ID on every attempt. A key built from the model tool call ID therefore fails to match, and an approved send happens twice.

The fix is a semantic key. The idempotency claim is the journal.

scope = tool.<tool_name>
key   = <run_id>:<step_path>:<args_hash>

step_path is the workflow node ID, or the literal agent for a call the model proposed. args_hash is the hash of the canonical tool arguments.

segment attempt 1
  model proposes crm.update(company=X, stage=qualified)
  ToolInvoker reads  run:agent:hash(...)  -> absent     -> decide
  ToolInvoker claims run:agent:hash(...)  -> claimed    -> execute -> complete
  model proposes email.send(...)
  worker dies

segment attempt 2
  model proposes crm.update(company=X, stage=qualified)
  ToolInvoker reads  run:agent:hash(...)  -> completed  -> return the stored response
  model proposes email.send(...)
  ToolInvoker reads                       -> absent     -> decide
  ToolInvoker claims                      -> claimed    -> execute -> complete

⚠️ The read and the claim are two steps, and the checkpoints sit between them. A single step puts the whole journal answer after the policy checkpoint. A turn that holds two gated calls then never finishes. The claim is the authority, because the read goes stale. Tools and integrations owns the rule.

A completed claim already returns its stored response, so the replay needs no table of its own. The model call in attempt 2 is paid for again. That is the accepted cost of one segment step, and the ceilings bound it.

A replayed model turn counts. The tokens were really spent and the meter really recorded them. A ceiling that ignored retries would not bound cost, which is the only thing it is for.

A replayed tool call is the opposite case and counts nothing: it made no call, it wrote no usage row, and its span carries replayed = true.

Neither count is stored. agent.runs grows no counter column, because the span tree already holds both facts and a second copy would be one more thing to keep true.

turns used       count of llm spans of THIS RUN
tool calls used  count of tool spans of THIS RUN, excluding replayed = true

Both counts filter on run_id, never on root_run_id, because both ceilings are per Run. The cost ceiling is the one that sums the tree. Read the tree here and a workflow of ten agent children shares one turn budget: the tree passes max_agent_turns part way down, every later child is handed a remainder of zero, and each one ends on ceiling without a model call. The workflow reports success and did a fraction of the work. See Ceilings for the split.

AgentExecutionResult.turns_used is the segment's count, which the executor needs before the segment ends. The Run total is the query above, and it follows the rule accrual follows: read the durable record, own no counter.

Two identical write calls inside one Run are treated as one effect on purpose. A workflow that needs the same tool twice uses two nodes, and the node ID separates them.

An approval row is keyed the same way, or a replay fills the inbox twice. A segment can persist an approval and then die before the wait starts. The replay reaches the same proposal, and a second INSERT would put two rows in front of a person for one decision, with only one of them wired to a wait. So the approval is claimed on the same semantic key as the effect it guards, and a replay that meets a pending row reuses it and stops on that id.

scope = approval.<tool_name>
key   = <run_id>:<step_path>:<args_hash>

A second approval proposed in the same turn is superseded. A model may propose several calls at once, and the first one that needs a person ends the segment. AgentExecutor closes the segment by cancelling every pending approval of the run other than the id the segment stopped on. A row nobody is waiting on must never sit in a person's inbox.

⚠️ The paused path files one row, and the read path is where the race is.AgnoAgentRuntime._resolve() decides the paused proposals one at a time, and the first ApprovalRequired leaves the loop, so no second row is written there. The framework runs the read calls of one turn concurrently, and it filters the external_execution calls out of that set. So two read tools can both file a row before either records the stop.

⚠️ A read tool approval is never the stop. A read call runs inside the framework loop, where there is no pause to carry a decision, so the runtime ends the run under read_tool_needs_approval. That segment stopped on no approval, and every row it left is cancelled.

⚠️ A segment that raises supersedes nothing, and that is the design. Inngest replays the whole step, and a paused approval carries the idempotency key the replay reuses. Cancel that row and the replay reads a cancelled row back, then raises on it, and the wait is never answered. The rows a raised segment left keep their own expiry clock.

The filter is the run, and never the root. root_run_id there would cancel the approvals of the parent and of every sibling, and a sibling branch waits on its own row. The statement is conditional on pending, so a supersede that races a person's answer touches no row and the decision stands.

Segments and approvals

A tool cannot wait for a person from inside a step.run. So the segment returns, and the function waits outside it.

state = step.run('admission.check', read_run_and_approval)   # one memoized step
if state is not proceed:
    if state is pending:
        answered = step.wait_for_event('admission.wait', 'agent/approval.resolved',
                        if_exp='async.data.run_id == event.data.run_id'
                               f' && async.data.approval_id == "{approval_id}"',
                        timeout=whole_seconds(expires_at - now))
        if answered:
            state = step.run('admission.read', read_row)   # which decision
        if not answered or state is pending:
            state = step.run('admission.expire', expire)   # the one writer of `expired`
    ending = admission_ending(state)            # the table below
    if ending is not proceed:
        return ending                           # no claim, and no work

run = step.run('claim', claim)          # None -> the function ends here
enter run_scope(run)                    # every step below writes spans through it

resumed_approval_id = None              # carried into the segment that follows a wait

loop while segment_index < max_segments:
    outcome = step.run(f'agent.segment.{n}', run_segment, resumed_approval_id)
    #   run_segment re-reads the Run row, and writes no lifecycle row of its own
    if outcome is running:                         # the segment wall clock fired
        if n + 1 >= max_segments:
            return succeeded(partial_reason='limit_reached')
        step.run(f'advance.{n}', advance_segment)  # no wait
        n += 1
        continue                                   # keep approval proof, if any
    handed, resumed_approval_id = resumed_approval_id, None
    if outcome is not waiting:
        break
    # Every refusal below comes before any write. A Run marked `waiting` and
    # then refused ends with a status it never needed.
    if handed is not None and outcome.waiting_ref_id == handed:
        return succeeded(partial_reason='approval_not_actionable')  # asked again
    if outcome.waiting_on is not approval:      # a delay needs step.sleep
        return failed('wait_kind_not_implemented')
    if n + 1 >= max_segments:                   # no segment left to act in
        return succeeded(partial_reason='limit_reached')
    step.run(f'wait.mark.{n}', mark_waiting)    # every surface reads `waiting`
    step.run(f'advance.{n}', advance_segment)   # clears resumed_from_wait
    state = step.run(f'wait.check.{n}', read_approval)   # closes the pre-wait gap
    if state is pending:
        answered = step.wait_for_event(f'wait.{n}', 'agent/approval.resolved',
                            if_exp='async.data.run_id == event.data.run_id'
                                   f' && async.data.approval_id == "{outcome.waiting_ref_id}"',
                            timeout=whole_seconds(outcome.waiting_expires_at - now))
        state = 'answered' if answered else step.run(f'wait.expire.{n}', expire)
    if state is gone:                   # absent, or another tenant's row
        return failed('approval_row_missing')
    if state is unusable:               # a person decided, and the clock beat them
        return succeeded(partial_reason='approval_not_actionable')
    if state is not answered:           # nobody decided, and nobody will
        return succeeded(partial_reason='approval_expired')
    woke = step.run(f'wait.resume.{n}', resume)   # conditional: see resume() below
    if woke is unowned:                 # another writer ended the Run
        return cancelled                # writes nothing
    if woke is not running:             # an approval nobody waits on holds it
        return failed('orphan_approval')
    resumed_approval_id = outcome.waiting_ref_id
step.run('finalize', finalize)

⚠️ Every exit above is the loop's answer, and finalize applies it. The return statements leave the loop, and never the function. finalize is the one writer of the terminal status, so a Run that skipped it would end with nothing to end it.

⚠️ The guard tests handed is not None, and it sits above the kind test.ExecutionOutcome accepts a delay wait that names no ref, so a first pass over one compares None against None. Drop the conjunct and that Run reports a decision a person made, in place of the wait this deployment cannot serve.

⚠️ The last test is negative, and that is not a style choice. Written as if state is over a sixth state matches no branch, falls through, and resumes a Run on an approval nobody granted. Written this way it reports an ending.

⚠️ A wait that expired is not a wait that was answered. wait_for_event answers the event, or None on the timeout. Resuming on both runs the very work the approval was gating, with nobody having agreed to it. A gate that opens by itself after an hour gates nothing. An action approval that expires ends the Run as a partial success, because some work already happened, and succeed() accepts waiting and clears the three itself, so no resume() runs on that path.

⚠️ A timeout is not proof that nobody answered. wait.expire decides that, and the row decides it, not the absent event. The step order below states why, and what the two reads around the wait cover.

⚠️ The function reads that a decision was made, and never which one. The answer lives in the approval row, and the runtime reads it there when the next segment hands it resumed_approval_id. A decision read in the function as well would be a second reader of one fact, and the two would disagree the first time an event was replayed. The publisher of agent/approval.resolved therefore owes both run_id and approval_id on the event: a publisher that omits either matches nothing, the wait times out, and the Run reports that nobody answered after a person answered.

The event path itself needs no read of the row. Inngest fires the timeout at the deadline, so an event that reaches the waiter arrived before it, and the inclusive boundary of decided_in_time() covers the same instant.

⚠️ The timeout is whole seconds, and at least one. The SDK rejects a duration under a second and one that is not a whole number of seconds, and a deadline minus now() carries microseconds. The raise lands outside every step, after the hold already moved the Run to waiting, so the function retries, fails the same way. The wait sweep is what ends the Run, one grace window past the deadline the hold wrote.

⚠️ The wait spends max_run_duration. That ceiling is wall clock from started_at and it counts the waits, so a person who answers an hour later can leave the segment after approval with nothing left. ApprovalService.create() caps expires_at at the run's own deadline for that reason, so the deadline this wait reads already sits inside the run's wall clock. The wait computes no second cap: one approval has one clock.

⚠️ A waiting Run has two owners, and the second one is a clock. The live function run is the first. resume() clears waiting_ref_id, so the approval id lives in the memoized step output alone, and losing that function run leaves the row with no writer at all. The Run therefore carries waiting_expires_at, and run.reaper runs a wait sweep over it. See the wait sweep.

The approval id reaches the next segment through the function, not the Run row. resume() clears waiting_ref_id on the branch that wakes the Run, and the other branch re-aims it at a different approval. The constraint runs_waiting_ref_shape forbids a running Run from holding one, so the column is null by the time the segment starts. The id lives in the memoized outcome of the segment that stopped, and the loop hands it to the next step. The alternative is a query over agent.approvals for the newest resolved row of this Run, which is a second source of truth for a value the caller already holds.

The run scope is entered after the claim and around the loop. It is a context manager, so a scope entered inside the claim step is gone when segment 1 starts. It also needs root_run_id and parent_span_id, which only the Run row carries, so it cannot be entered before the claim either. See observability and operations.

⚠️ mark_waiting() and advance_segment() are called by the Inngest function, and by nothing below it. No executor calls either. Each is its own step, and the order is the whole mechanism, because three writes in one step are not atomic: a failure after the hold re-runs the body, the executor re-reads a row that now says waiting, and it answers cancelled — which writes nothing, so the run sits at waiting with no wait pending and no writer.

step  segment.n      run the segment, and nothing else
step  wait.mark.n    mark_waiting(), so every surface reads `waiting` while it waits
step  advance.n      advance_segment(), which also clears resumed_from_wait
step  wait.check.n   read the row; a decision already made skips the wait
      wait.n         wait_for_event
step  wait.expire.n  on the timeout: resolve(expired), then read the row back
step  wait.resume.n  resume(), which sets resumed_from_wait

⚠️ One wait is not guaranteed delivery. A wait catches only what arrives after it registers, and segment.n commits the approval row three steps earlier. A person who answers in that gap sends into no waiter, and Inngest drops the event. That person is never told: the press answered 200 and the inbox reads resolved, so nobody presses again. A retry on wait.mark.n widens the gap to minutes.

Two reads close it, and neither is a poll loop. wait.check.n reads the row once before the wait registers, so an answer that arrived in the gap skips the wait outright. wait.expire.n reads it once after the wait ends, so an answer lost to any residual gap is still honoured.

A smaller gap survives between wait.check.n and the pause registering, because the step returns to the server in between. It costs latency and never an answer: the wait runs its whole timeout, and wait.expire.n then reads the row that carries the decision. The Run resumes late rather than never.

.claude/rules/12-inngest.md §4 names a bounded loop of short waits for this shape. It does not fit here: DEFAULT_APPROVAL_TTL is 24 hours, so 30-second slices are 5,760 steps against Inngest's cap of 1,000 for the whole run. The two reads above are constant at any TTL and lose no answer.

⚠️ The wait timeout is the one writer of expired. There is no sweeper job. wait.expire.n calls ApprovalService.resolve(expired), whose update is conditional on pending, so this call and a person who presses Approve in the same second produce one transition. The loser reads the winner's row back. Skip the write and the row stays pending for ever, because its expiry writer is a wait that has already returned.

Both reads answer whether a decision was made in time, and never which one.ApprovalService.authorizes refuses a row whose resolved_at is past its expires_at. A wait that resumed on such a row would meet a live policy decision and show that person a second card, on a Run whose own clock has already run out. unusable ends the Run instead. One predicate, decided_in_time(), serves both readers.

⚠️ One path still reaches that second card, and it is the clock-skew race below. expires_at comes from the worker and resolved_at from the database, so a press inside that skew arrives as an event, which proves a decision and skips the row read. The next segment decides the call again and files a fresh proposal, because ENG-2156 released the claim of a terminal row. _expiry gives that proposal min(now + approval_ttl, run_deadline), so it ends two ways. The Run deadline binds: the card is already dead, the wait times out at once, and the Run ends approval_expired rather than approval_not_actionable. The TTL binds: the card is live and the person answers once more. Both are bounded by one extra segment, on a window of milliseconds, and both cost less than a durable read on every answered approval.

One look at the row answers one of five states. Three of them end the Run, and no two of them end it the same way. approval_expired says a person did not answer in time, so every ending that reports it must be one where nobody answered.

answered is not an ending, and the row does not decide what follows it.resume() does. So the wait leaves this function six ways, and only one of them runs another segment.

A clock ending is a partial success. A defect is the other case, and both failed rows are defects.

unusable and gone each exist for one reason: reporting an expiry for them tells an operator the opposite of what happened. A person whose inbox card reads approved must never read that nobody answered.

over also takes the row that records no decision time. A resolved row whose resolved_at is null is a row approvals_resolution_time forbids, and this path knows nothing about when that person decided. unusable would report that they decided late, which is the one fact the row is missing, so it fails closed into the neutral ending instead.

gone is also the tenancy boundary of this path. ApprovalService.get() reads by id alone and the client holds the service role, so no policy filters the row. The read compares organization_id itself, and another tenant's row answers exactly as an absent one does. Any other answer would tell the holder of an approval id which rows exist in other tenants.

A segment never re-asks for the decision it was given. authorizes takes approved alone, so a rejected decision skips no checkpoint and the call is decided live. ToolInvoker answers it from the row: a rejection of this exact call returns a rejected result and files nothing, and every other refusal files a fresh proposal, which carries a new id. Both were one row before ENG-2156, because the idempotency key held every row of one claim.

The loop still carries the id it handed the segment, and a segment that parks on that same id ends the Run with approval_not_actionable. It is a guard on the two writers above rather than the ordinary path: without it a Run would spend every segment it has left on one call, and the person would never be asked a second time.

The advance comes before the resume, and reversing them is silent. advance_segment() is the only writer that clears resumed_from_wait and resume() is the one that sets it, so an advance placed after the resume clears the flag before any segment reads it, and a definition with refresh_on_resume never rebuilds its brief.

Every segment that continues ends with advance_segment(). It is the one writer of segment_index, and it is the only thing that clears resumed_from_wait. claim() clears the flag too, but claim() runs once per function run, so it cannot clear it between segment 3 and segment 4. Leave the flag set and every later segment rebuilds the context brief, on a Run where nothing moved.

It is conditional on the index it was handed and on the Run not having ended, WHERE segment_index = :from_index AND status IN ('running','waiting'). The status list is LEGAL_FROM['advance_segment'], so it drops queued like the table above. The index guard stops a step that commits and then crashes before Inngest memoizes it from skipping a segment on the retry. The status guard keeps it inside the rule every other lifecycle write follows: a Run cancelled mid segment must not be advanced afterwards.

Every wait ends with resume(). mark_waiting() moved the Run to waiting and set waiting_ref_id. Without the matching call the Run executes its next segment while every surface still reads waiting, pointing at an approval that was resolved. The call is not always a wake: the three answers below say what it did, and only one of them runs another segment. resume() is the inverse of mark_waiting(), and the two are always written as a pair.

UPDATE agent.runs
   SET status = 'running',
       waiting_on = NULL,               -- the waiting shape constraint rejects the row otherwise
       waiting_ref_id = NULL,
       resumed_from_wait = true,        -- the next segment rebuilds the brief; advance_segment clears it
       heartbeat_at = now()
 WHERE id = :run_id AND status = 'waiting'

resume() clears the waiting three on the branch that wakes the Run, and it is the write that most often forgets to. runs_waiting_shape is an equivalence, so status='running' beside a surviving waiting_on raises 23514 and the Run never wakes. cancel(), succeed() and fail() carry the same line for the same reason.

resume() is conditional, because a parallel node can hold two waits at once. Branch B resolving while branch C still waits must not report the Run as running: the next node of B would run against a Run every surface reads as awake, while a person still has C in their inbox. So resume() re-reads the outstanding waits of the Run.

The second branch is mark_waiting() again, so RunManager gains no new statement: that method takes every non-terminal status and targets waiting, which is exactly a re-aim. It is idempotent, so a duplicate resolve re-reads the same waits and writes the same row.

⚠️ resumed_from_wait is written on the waking branch alone. It tells the next segment to rebuild its context brief, and no segment runs next when the Run stays asleep. Set it on the re-aim and the brief is rebuilt on a Run where nothing moved.

⚠️ The clock is half the read. A row that reads pending past its expires_at is expired, exactly as the inbox queue treats it. Answer that row here and the Run holds at waiting for ever, pointed at an approval no wait will ever resolve.

⚠️ The agent loop reads the answer, and stops on anything but running. A segment holds one wait at a time, so it normally reads running and continues. A segment that left a second pending row points this at that row, and the loop must not run its next segment then: AgentExecutor re-reads a Run that is not running and answers cancelled, which writes nothing, so the Run would sit at waiting with no wait pending and no terminal writer.

The segment supersede is not the guard, because it is best effort. It catches and logs every fault of its own, on purpose: a row nobody waits on must not turn a finished segment into a failure. So a stale pending row surviving a segment is a state the platform accepts, and the answer below is what stops the loop.

resume() therefore answers three states, and it reads them off the Run and never off the read of its waits.

⚠️ A write that matched no row is three different facts. resume takes waiting alone, so a zero-row write says only that the Run is no longer waiting.

• This step's own committed attempt. A transport fault after the commit leaves the caller with no answer, Inngest retries the step, and the Run is already running with its index moved and its waiting three cleared. It is ready to continue. Answer unowned here and the loop ends cancelled, which writes nothing: the row then reads running with no function behind it until the reaper takes it, six hours later in V1.
• A person cancelling. cancel() writes the status first and clears the approvals last, so a cancel landing between the two reads leaves the read answering a pending row while the row already says cancelled. Report that as orphan_approval and one person's Stop becomes a platform stop, so a parent node stops honouring continue_on_error. cancelled is not a platform stop, and it writes nothing, so the real writer's status stands.

So resume() re-reads the Run row on a zero-row write. The count of written rows cannot tell the two apart, and this is the rule ResolutionOutcome already states for the approval writer: a retry reads its own write as another writer's.

The same pairing applies to a workflow wait node and a workflow approval node. The admission wait is the one exception: claim() follows it, and claim() already writes running.

The admission wait reads which decision was made, and every other wait does not. An action wait reads only that a decision was made, because the next segment reads the row and acts on it. The admission wait has no next segment: an approval runs the claim, and a rejection ends the Run.

So this wait has a third step. admission.check reads the row before the wait registers, and admission.expire reads it after the timeout, exactly as the segment loop does. admission.read is the one the loop has no use for: the event proves a decision and never which one, so the answered arm reads the row as well.

A row that still reads pending after the event runs admission.expire too. A redelivered or hand sent event reaches the waiter while the row is unresolved. This wait is the one writer of expired and it has already returned, so a row left pending there has no writer left and sits in a person's inbox for ever. The expiry update is conditional on pending, so a person who answers in the same second still wins and the read-back carries their decision.

⚠️ That write can end an approval before its own deadline, and the cost is accepted. ApprovalService publishes agent/approval.resolved only after a person decides, so an event that finds a pending row is out of band: a replay, an ops script, or a faulty integration. A wait that has matched cannot register again, so the two available endings are an approval closed early and a row that never closes at all. The early close ends the Run and frees the person to be asked again. The row that never closes ends nothing, and the person is told nothing.

One look at the row answers six endings, and one of them runs the Run.

A rejection needs no clock. Refusing to act is safe at any time, and a person who pressed Reject must read cancelled rather than a report that nobody answered. An approval is the opposite: acting on a decision the clock beat is the failure the one-clock rule exists to stop.

Nothing ran, so every ending here is failed and never a partial success. An action approval that expires is the other case: some work already happened, so that Run succeeds with partial_reason=approval_expired. This is the one status rule, and result and error states it. approval_not_actionable therefore reports failed at admission and succeeded inside the segment loop, from the same row state.

cancelled needs a writer of its own. apply_outcome() writes nothing for a cancelled outcome, because the ordinary cause of one is another writer that already ended the Run. RunManager.cancel() does not serve this either: it takes an ActorIdentity, walks the subtree, writes the control rows, and publishes agent/run.cancelled, which cancels the very function run that called it. So the rejection is one conditional transition to cancelled, in a step of its own, and the outcome that follows it writes nothing.

The pre-claim read is one memoized step, and that is not tidiness. The branch is chosen from run.waiting_on, and claim() clears that column. Read outside a step and the second execution of this function reads running, skips the branch the first pass took, and the step sequence stops matching. The same step reads the approval row, so it also closes the gap before the wait registers, exactly as wait.check.n does inside the loop.

Its answer is read with a default, as the claim's answer is. Inngest replays that mapping on every later invocation of the function run, so a Run held before a deploy re-enters with the previous shape. A bare subscript raises outside every step, and here that raise lands before the claim: the Run stays waiting, and only the wait sweep ends it, one grace window past its deadline. A shape this deploy cannot read answers approval_row_missing, which is an ending.

The read is narrow. Every Run start reaches this step, and nearly none of them holds an approval. A wide read would carry the frozen snapshot and the Run input over the wire for all of them. The common case costs one indexed read and one durable step.

The admission wait comes before the claim. The Run stays waiting, not running, for as long as a person takes to answer. That ordering is what keeps the heartbeat read away from it: it fails a stale running Run and a stuck queued Run, and it never takes a waiting one. The wait sweep does take one, and it reads the Run's own deadline rather than its silence. Claim first and a Run waiting overnight for a decision looks like a dead worker.

The next segment reconstructs the agent from the frozen snapshot, continues the same Agno session identity, and replays the journal. Never serialize an in memory Agno agent as the durable boundary.

Where the agent session lives

A segment resumes on a worker that never saw the first segment. It rebuilds the configuration from the snapshot, and it needs one more thing: the message history the loop already produced. That history is the Agno session, and it needs an owner.

Agno writes its session to one table in our database, and the runtime owns that table.

agent.sessions
  run_id (pk)          the Run this session belongs to; one session per Run
  organization_id      the tenancy boundary; RLS reads it
  agno_state jsonb     AgnoAgentRuntime owns the shape; nothing above it parses it
  segment_index        the segment that produced the stored history
  updated_at

The write cannot go backwards, and segment_index is why the column exists. Inngest delivers at least once, so two workers can hold one segment. A slow worker can finish segment 4 after a second worker finished it and a third finished segment 5. An unguarded write puts the older history back, the next segment rehydrates a conversation that lost its middle, and nothing reports the loss.

Two guards, and they answer two different writers.

They are not two mechanisms for one fact. The repository write is two statements that are each conditional on their own — an UPDATE filtered on segment_index < the new one, and an INSERT with ON CONFLICT DO NOTHING for the run that has no row yet — so an ordinary duplicate never reaches the trigger. The trigger exists because an invariant that lives in one WHERE clause in one method is one forgotten predicate away from silence.

The column is an envelope with two halves. Agno owns one of them.

agno_state = {
  'agno':         <AgentSession.to_dict()>,   # opaque, Agno's shape
  'instructions': '<the composed system block>',
}

The second half exists because Agno rebuilds the system message from instructions on every run and never replays the stored one. A later segment that arrives with context = None has nothing to keep unless the runtime kept it. See where the session lives for why that is the whole system block and not only the brief. Two halves in one column beat a second column: agent.sessions already shipped, and both halves have the same owner, the same lifetime and the same reader.

There is no session_id column. Agno takes the session identity from its caller, on Agent and on run(), so the runtime passes str(run_id) and the Agno session identity is the Run. A separate column would be a second unique key on a table that already has one, and a second identifier to keep in step with the first.

The column names its owner on purpose. state would read as platform state, and this is the one column in the platform that our own code may not open. A second runtime would add its own column, and AgentRuntime would not change.

Three rules make this a runtime table rather than a second store.

1. One session per Run, keyed on the Run. The Run is the primary key, so the one-to-one is a constraint rather than a convention. The session is execution state, so it lives and dies with the Run. It is not conversation state, and it is not memory. agent.conversations and agent.memories keep their own rows and their own meaning.
2. We never read agno_state. Agno writes it, Agno reads it. The platform treats the column as opaque. The moment our code parses it, the framework stops being replaceable, which is the whole point of AgentRuntime.
3. It is deleted with its Run. Retention follows agent.runs. A finished Run needs no session, and a session that outlives its Run is a leak with a customer's message history in it.

The runtime writes the session on every exit path that produces an AgentExecutionResult. Only the retryable-failure path writes nothing. The outer segment timer is not a runtime result.

completed          write the session, then return
ceiling reached    write the session, then return
needs_approval     write the session, then return the approval id
cancelled          write the session, then return
failed, terminal   write the session, then return the error
failed, retryable  WRITE NOTHING; Inngest replays this step

A wall-clock yield makes no promise about the session write. The outer timer can cancel the runtime before a write, while the database call is in flight, or after the database accepted it. The next segment therefore handles three states. It continues a saved paused run, starts one stable continuation turn from saved history with Continue the task from the saved conversation., or restarts the original Run input when no session exists. Only AgnoAgentRuntime inspects the opaque session to choose.

An approval proof survives repeated wall-clock yields. The runtime applies it only when it continues the saved paused run that still needs the decision. If the session holds completed history, or no session exists, the runtime starts a fresh turn and drops the proof. A new tool proposal must get its own decision.

The approval path is the one that costs the most when it is wrong. The loop is paused mid run, and the model has already read the results of every call before it. The session is what holds the paused run, so losing it there loses the requirement the continuation needs: acontinue_run looks the run up by id in the session and raises when it is absent. Even if it were rebuilt, the segment after approval would start from older history, and the model would pay for the same turns again and reason over a conversation that lost its middle.

The retryable failure is the opposite case, and writing it is the bug. Inngest replays the whole step, and a step is a whole segment. A session written from the failed attempt holds a half finished turn, and the replay rehydrates that wreckage as its starting point. Leaving the row alone leaves the clean pre-segment session in place, which is the state a replay is supposed to begin from.

Only AgnoAgentRuntime touches the shape of that column. The refreshed context brief arrives as AgentExecutionRequest.context, and the runtime replaces the system block inside the session before it continues the loop. Nothing above the runtime reads or writes agno_state, because replacing a system block means knowing Agno's message shape, and knowing it above this line is what would make the framework unreplaceable.

A second runtime would bring its own continuation shape. It would get its own table, and AgentRuntime would stay unchanged.

Agent execution

class AgentExecutor:
    async def execute(
        self, claimed: Run, *, resumed_approval_id: UUID | None = None
    ) -> ExecutionOutcome:
        run = await run_repository.get(claimed.id, claimed.organization_id)
        if run is None or run.status != 'running':      # a cancel, or a reaper
            return ExecutionOutcome.cancelled()

        accrual = await accrual_checker.check(
            run.organization_id, root_run_id=run.root_run_id,
            ceilings=run.snapshot.ceilings, scope='run_and_day',
        )
        if accrual.outcome != 'allow':
            return self.stop_short(run, 'budget_exhausted')

        ceilings = await self.remaining(run)
        if ceilings.exhausted:                          # nothing left to spend
            return self.stop_short(run, 'limit_reached')

        policy = run.snapshot.context_policy
        if run.segment_index == 0 or (policy.refresh_on_resume and run.resumed_from_wait):
            context = await context_builder.build(self.context_request(run), policy)
        else:
            context = None                       # reuse the instructions the runtime stored

        segment_input = self.segment_input(run, resumed_approval_id)
        tools = tool_factory.build(run.snapshot.tools, run.principal)
        request = AgentExecutionRequest.from_run(run, segment_input, context, tools, ceilings)
        result = await agent_runtime.execute(request, event_sink.for_run(run.id))
        return self.to_outcome(run, result)      # AgentExecutionResult -> ExecutionOutcome

The last line is a mapping, not a pass through. agent_runtime.execute() answers an AgentExecutionResult, which says why one segment stopped, and this method is typed -> ExecutionOutcome, which says what RunManager must do next. Returning the first where the second is declared collapses two of the three result types the design keeps apart, and the stop-to-status table stops having a place to live. See three result types.

The segment re-reads the Run row, and the claimed Run is not enough. claim() runs one time per function run and answers one Run. segment_index and resumed_from_wait move between segments, so the object the claim answered reports 0 and false for the whole run. Segment 4 reading it rebuilds no brief after a three day approval and advances from the wrong index. The read is one primary key query, and it carries a second fact for free: the status.

A status other than running ends the segment before anything else.cancel() writes the status first and the agent.run_control row last, so the Run row is the earlier signal and it needs no second repository. The control row stays the boundary inside the segment, where the tool adapter reads it about forty times per Run. Between segments, the row this step already reads answers the same question.

segment_index and resumed_from_wait are Run columns, because the worker that reads them is not the worker that wrote them. resume() sets resumed_from_wait, and advance_segment() clears it, so the flag describes the transition that woke this segment rather than the whole history of the Run.

Accrual is the first thing a segment checks, and it is not the remainder. The two answer different questions from different records. AccrualChecker reads the canonical usage meter for max_cost_cents over the tree, and the organization day. self.remaining() reads the span tree and the wall clock for the four ceilings this Run owns. A segment that checked only the second would pass the money ceiling by a whole segment. See policy and governance.

A stop before the runtime is a failure at segment 0 and a partial success after it. That is the one status rule, and result and error states it. segment_index is the record _stop_short reads it from: a Run that never reached a model call produced no output.

segment_index == 0    fail(RunError(code=<the reason>))
segment_index  > 0    succeed(RunResult(partial_reason=<the reason>))

The reason is budget_exhausted when the accrual check refused, and limit_reached when a remainder reached zero. approval_expired is the third member, and the wait writes it rather than the executor.

Two places name max_cost_cents, and RunCeilings.exhausted is neither. That property skips the money field on purpose: the meter answers it, not a remainder. The tool adapter records the field name inside the loop when ToolInvoker refuses a metered call. _reason_for then maps that one name to budget_exhausted, and every other field of RunCeilings to limit_reached, so a run stopped by money never reads as stopped by turns. AgentExecutor answers budget_exhausted directly at the segment boundary, from its own accrual check. The mark the adapter reads is ToolResult.stop, and tools and integrations owns it.

A retryable platform fault answers no ExecutionOutcome. It propagates out of execute(), and Inngest replays the whole step. A completed matching Tool claim returns its stored result. An interrupted claim keeps its lease and vendor recovery rules. stop = failed is the terminal case only. An executor that caught every exception and mapped it to next_status = 'failed' would turn a database outage into a failed Run that no retry can save.

The brief is not rebuilt after every successful segment. Segment 0 always builds one. A later segment builds one only when it resumed from a wait and the policy asks for it, which is the case where the world really moved. An Inngest replay of a completed segment uses its memoized result and builds nothing. A retry of a failed segment can build the brief again because the step committed no result.

context = None therefore means "reuse the instructions you stored", and it is the ordinary case. It does not mean "keep the block Agno has", because Agno keeps none: it rebuilds the system message from instructions on every run. The snapshot froze the ContextPolicy, not the content. Skills were rendered when the snapshot was frozen.

Every segment carries an input, because Agent.arun(input=...) has no default. Segment 0 carries the Run input. A segment that follows a resolved approval carries the outcome of that approval, and tools and integrations owns how the approved call reaches the loop.

A remainder of zero never reaches the runtime. The executor subtracts first, and a run with nothing left ends on ceiling without starting a segment. Handing a runtime a ceiling of zero asks it to guess whether zero means none or unlimited, and the two framework knobs that could carry it disagree.

Agno never calls a tool handler. AgnoToolAdapter declares the tool, and the call returns through ToolInvoker.

Workflow execution

node type    executes as
tool         ToolInvoker.invoke  + span
agent        child Run via step.invoke
subworkflow  child Run via step.invoke
branch       ConditionEvaluator selects one child + span
parallel     a fixed set of Inngest steps
wait         Inngest wait + span
approval     approval row + Inngest wait + span

There is no second workflow state machine beside the Run, step and span model.

The walk lives in WorkflowExecutor, not in the function

A workflow is a tree, and the Inngest function is a flat body. A walk written in the function would need a durable stack of its own, so the walk is ordinary recursion inside WorkflowExecutor. The durability comes from the steps it creates, and from nothing else.

The SDK re-executes the function body once per step. A step with no memoized value runs its body and then unwinds the function, and the server sends a new request that re-enters the body from the top. A twenty node workflow therefore runs its body about twenty times, and every memoized step returns at once.

Two consequences are load-bearing.

• The walk must be deterministic. The same definition and the same memoized outputs must produce the same node order and the same step IDs on every pass. A node ID is unique across the whole workflow, so it is the step ID.
• Every write a node makes sits inside a step. A write above the steps happens again on every pass. A span opened there is written twenty times.

WorkflowExecutor reaches Inngest through one seam, so the walk is testable with no dev server.

class WorkflowSteps(Protocol):
    """One durable step, one child invoke, and one parallel group."""

    async def run(self, step_id: str, body: Callable[[], Awaitable[T]]) -> T: ...
    async def invoke_run(self, step_id: str, run_id: UUID,
                         organization_id: UUID) -> dict: ...
    async def parallel(self, branches: tuple[Callable[[], Awaitable[T]], ...]
                       ) -> tuple[T, ...]: ...

The seam is built once per function run, so WorkflowExecutor.execute(run) still satisfies SegmentExecutor. That is also why the coordinator is not a field of the graph the worker caches: it closes over ctx.step, and ctx exists only inside one function call. The graph holds a factory instead, and executor_for() takes the seam of this call and passes it through.

The run is read wide again before the walk, outside every step. The walk rebuilds its node tree from the frozen snapshot on each pass, and a snapshot memoized into Inngest step state is the fault the claim already avoids. The columns the walk reads never move, so a later pass reads the same values, and the cost is one round trip per pass.

One node is one step, and a child node is three

A tool node and a branch node fit in one step.run: the span, the work and the result all sit inside the memoized body.

A child node cannot. The SDK refuses a nested step, so step.invoke may not sit inside step.run, and the node splits across two.

step.run     node:<id>          start(), and answer the child run ID
step.invoke  node:<id>:child    the child function; the parent holds no slot

Two steps and not three, because the node opens no span. The child's own run span is the node's span, so there is nothing to close after the invoke. Mapping the child's answer onto the node outcome is pure, so the walk rebuilds it on every pass rather than spending a step on it.

The first step is the one that must be idempotent, and it already is: the start key is deterministic, so a replay of that step meets duplicate and reads back the child its earlier attempt created.

The child's actor comes from the parent's Principal

RunManager.start() needs an ActorIdentity to mint the child's grant, and nothing at run time carries one. The Inngest function receives a run ID, a tenant and a lane, and agent.runs stores a Principal and no actor. So the node rebuilds an actor from the parent Run's stored Principal, which already declares organization_id, user_id, trigger_id and scopes. No column is added.

The parent's Principal is the right source, and the actor that started the tree is not. The grant is an intersection, so reading the original actor lets a child node intersect against the rights the tree started with, and a child can then hold a scope its own parent no longer has. Reading the parent's Principal narrows the grant at every level, which is what "a step cannot widen authority" means.

agent.runs.principal therefore stores the two identifiers beside the scopes. It stores no actor kind: Principal.kind derives that from the one identifier that is set, and a stored copy is a second thing to disagree. The run row carries the organization, the run and the definition as columns, so the grant repeats none of them.

A span that wraps more than one step needs an open and a close, not a block. The parallel span and a child node's span both do. SpanRecorder therefore offers open_span() and close_span() beside the scoped block. Each half sits in its own step, so a replay writes neither again. Application code that fits in one step keeps the block.

A parallel node runs in race mode

ctx.group.parallel is the only parallel primitive of the SDK. asyncio.gather does not work, because a step signals by raising a BaseException that gather propagates before the sibling steps are discovered. A plain loop over the branches is worse: it serialises the node and nothing reports it.

The mode is not a preference either.

A branch is a container, so almost every branch holds more than one step. Under WAIT a branch that parks holds its siblings at their next step boundary, which is the exact behaviour "an approval inside a parallel node" forbids. The node uses RACE.

⚠️ That was measured, not reasoned about. Against a real dev server, one branch parking on an event nobody sends and one branch of two steps:

ParallelMode.WAIT   sibling step 1 ran, sibling step 2 never ran
ParallelMode.RACE   both ran, while the first branch was still parked

The mocked runner in inngest.experimental.mocked cannot answer this. It pops one planned step per pass, so it serialises a parallel group by construction and reports the guarantee holding whether it holds or not.

A node writes no span of its own

The node table above already names every span a node produces, and three of the eight write none.

A parking node spends no step on its span. The parallel span needs two steps of its own, because the steps it covers are not its own. A wait node and an approval node already own a step on each side of the pause, so the open rides in the first and the close rides in the last. The workflow visualizer reads that span to show how long a Run held for a person.

A tool node opens nothing. ToolInvoker owns the tool span, and a second one around it would double every tool row in the tree.

A wide node keeps the same rule. Each ToolInvoker call opens one tool span. An item that stops at a ceiling before the call opens no span. The span count therefore follows started calls and not the input width.

An agent node and a subworkflow node open nothing either, and the child still joins the tree. parent_span_id is current_span_id() at the node, which is the enclosing parallel span, or the run's own run span. There is no ninth span kind for a node, and the eight are closed. The child's run span carries the node ID as its name, so a person still reads which node produced which child.

workflow run
 └─ span run "wf.outreach"                     <- current at a top level node
    ├─ child run  agent "classify"
    │   └─ span run "classify"                    parent = the run span
    └─ span parallel "fanout"                  <- current inside the container
        └─ child run  agent "draft"
            └─ span run "draft"                   parent = the parallel span

A tool node spends max_tool_calls

ToolInvoker counts nothing, and the frozen budget is read once per function run, so nothing above the node bounds a fan out. WorkflowStepExecutor therefore reads the Run's own tool spans before each tool node and compares the count against the frozen max_tool_calls. That is the rule the ceilings section already states: a per-Run ceiling counts spans of that Run. It is the same read an agent segment makes, so the two paths cannot disagree about what a tool call is, and a span marked replayed is dropped by both.

A spent ceiling settles the Run under the one status rule, and continue_on_error does not tolerate it. Tolerated, every node after it would be skipped and the run would report success having done a fraction of the work.

A wide tool node repeats the call-ceiling check for each item. Calls that are already in flight can pass the same count. The overshoot is bounded by fanout_concurrency, not by max_fanout. The node stops scheduling new items when the ceiling answers. It drains the current batch, keeps its ordered settled prefix through the first stopping input position, and marks the Run partial. A result after that position is discarded even if its call finished first. The call cannot be undone, but a later position cannot leave a hole in the ordered output. continue_on_error cannot turn a ceiling into a complete Run.

A wide tool settles items, not platform faults

The scheduler starts batches in input order. It keeps at most fanout_concurrency item workers in one batch. Each worker has one outer timeout_s guard around the complete call and result envelope. This is the clock that the publish formula measures. A second outer guard bounds the whole node at the step budget. Publish keeps 10 seconds of that guard outside the item batches.

A returned hard platform stop wins over a ceiling in the same batch. It fails the node at any input position. A partial result cannot hide an outage or a runtime defect. If the batch has several returned hard stops, the lowest input position chooses the error. If the batch has only ceiling stops, the lowest stopping input position chooses the prefix and its partial_reason.

An exception does not enter this selection. The first observed exception cancels the remaining workers and uses its action in the table. Input order decides only between returned results, even when calls finish in another order.

An empty list answers an empty list. An invalid list or item fails before the first call. A list over max_fanout also fails before the first call. The final serialized envelope is measured before the step returns. A value above Inngest's 4 MiB step-output limit fails with fanout_output_too_large. It is never cut and reported as complete.

The whole step is the retry unit. A raised fault discards every item result from that attempt. The next function attempt repeats the wide read. This can repeat a metered vendor read, but it cannot repeat an effect because Phase 4 permits only side_effects = read. A write or send fan out stays invalid until each item has a durable claim and partial effects have a recovery rule.

An approval is not one failed item. ApprovalRequired is a control signal and cannot sit in a result list. A read approval cannot resume inside a tool node, just as it cannot resume inside the Agno loop. The scheduler cancels its batch and fails the node with read_tool_needs_approval. It does not spend a function retry on a valid policy decision. This code is a non-tolerable platform stop for a wide node, so continue_on_error cannot leave the Run active. The normal terminal cleanup supersedes every pending approval row. Use a run.start admission rule when a wide search must wait for a person before it starts. Phase 4 does not park a wide tool node.

⚠️ A parallel fan out can overshoot, by the calls in flight. Each branch is its own step in its own request, so several branches read the same count before any of them opens a span. The span opens before the call, so the overshoot is bounded by how many steps the server runs at once, and never by the width of the fan out. An exact bound needs one atomic counter, and no product need asks for one.

A platform stop is never tolerated

continue_on_error says the author expects that step to fail sometimes. A node that declares it settles failed, and the walk runs the next node. Some failures are not that, and the flag must not decide them.

The rule: stop when the runtime could not read or could not measure, and tolerate when the graph, the data or the business answered. A tool that returned not_found answers. A meter that never replied did not answer. A snapshot this runtime cannot parse did not answer either, and that node never ran, so nothing knows what it would have produced. Tolerated, every later node runs. The Run then reports success during an outage, or on a workflow the runtime could not read.

The set spans three processes. A node reads the stored code of a child Run, so a code the agent runtime or the child's own walk produced decides this parent. WorkflowStepExecutor cannot import those modules, so it mirrors each literal and a test pins every one against the module that declares it. A rename on either side would break the match silently.

PLATFORM_STOP_CODES holds seventeen codes. Seven are raised by this node, and every one can arrive from a child Run. Eleven are declared only in another module, so WorkflowStepExecutor mirrors each literal. invalid_snapshot is in both groups.

The code carries the rule, and never a flag. start_child flattens an outcome into a ChildStart, and that record crosses an Inngest step as JSON. A boolean is dropped there, so the same fault reaches the parent tolerated. The code survives that trip. It also survives a child Run that ended under the same code in another process.

A spent ceiling is not a member, and a member is the wrong fix for one. A ceiling stop keeps the work the Run already did. The set turns a code into a hard failure, which would discard it. A ceiling carries a partial_reason instead: the accrual gate sets budget_exhausted, and the wall clock gate and the tool call ceiling both set limit_reached. The same check reads that mark as well, so a marked ceiling stops for its own reason and stays a partial success.

Five writers set the mark. The accrual gate, the wall clock gate and the tool call ceiling all run before the node calls anything. The other two arrive from outside the node.

• The tool call. ToolInvoker reads the run cost ceiling again at the call, so a parallel fan out passes the gate on several branches and the invoker refuses the later ones. Its accrual checkpoint writes the mark into ToolResultMeta, and InvokerToolNodeCaller copies it onto ToolAnswer. The mark travels in a field and never in the code: a handler answering the money code for a vendor quota would otherwise report a failed Run as a partial success. _bound is the only author of meta on the handler path, and it rebuilds the record rather than returning the handler's own, so a handler cannot write the field either.
• A child Run. A child that stops at its first segment reports a failure, because nothing was done, and it carries the clock it spent on RunError. child_outcome reads that field. Unread, the parent tolerates a spent ceiling, and a parallel of such children as the last node reports full success after a fraction of the work. limit_reached is the child's own frozen ceiling, so no parent gate ever re-catches it.

A child that succeeded partially still marks its parent. _stop_short reports a failure only at segment 0, where nothing was done. After that the child reports succeeded with a partial_reason, and an expired action approval always does. The node is ok and the walk goes on, because the child did succeed. The mark travels anyway: a sequence and a parallel each carry the first one an ok child produced, and the Run reports succeeded with that clock.

Dropped, the severity inverts: a child that did nothing stops its parent, and a child that did some of the work reports an unqualified success. The second is the common shape. approval_expired shows this most clearly, because the child ends with an empty output and a parent would report clean success for work nobody approved.

⚠️ A Run error code is not the runtime's alone. run_tool passes a tool's error_code through verbatim, and a failed node's code becomes the Run's terminal code. So a handler answering the money code for a vendor quota reaches a parent as budget_exhausted. The clock travels in a field at every boundary, and no reader derives it from a code:

ToolResultMeta -> ToolAnswer -> NodeOutcome -> RunError -> child payload
  -> parent NodeOutcome

⚠️ The start path is the one place a ceiling carries no mark, and that is sanctioned. start() refuses a child on the organization day cap, and the node answers start_refused with no partial_reason. A node that declares continue_on_error walks past a spent day cap. The Child Run rules below state that rule: the refusal names a cap the author can read, not a runtime that could not answer.

One refusal is not a refusal the author can read. snapshot_unbuildable says freeze could not assemble the snapshot, because the registry no longer holds a tool the definition names, or a skill no longer resolves. Publish checked both, so it names a deploy that changed after the definition was published. The node answers invalid_snapshot for it, and never start_refused.

That asymmetry is the reason. The same missing tool answers invalid_tool_contract when a later segment reads a snapshot that was already frozen, and that code stops the Run. Which branch fires depends only on whether the deploy rolled before or after the freeze, so both answer the same platform stop. Collapsed into start_refused, a declaring node skips the child and the Run reports success on a definition this deploy cannot build.

⚠️ A parallel container answers one outcome, so severity decides it. One branch may carry a mark and another may carry a bare platform stop. The container prefers the unmarked failure, whichever branch met it. Taking the first failure in declaration order instead, the same two faults report succeeded or failed depending on the order the author wrote the branches in.

These are not platform stops. The runtime read each one and answered it, so continue_on_error decides them as the author asked:

• start_refused — the day cap or a cancelled parent refused the child, and the Child Run rules below sanction tolerating it.
• workflow_cycle, subworkflow_unresolvable, depth_exceeded — the walk read the graph, and the graph is wrong.
• unresolved_reference — the reference is readable, and the step it names produced no value.
• tool_failed, child_failed, child_cancelled — the tool, the child or a person answered.
• tool_unavailable, timeout, denied, invalid_input — a tool answered. ⚠️ tool_unavailable and invalid_tool_contract name the same condition and answer differently. A snapshot naming a tool this deployment cannot serve stops an agent child, and is tolerated on a tool node. The agent case is the frozen snapshot the runtime could not read. The tool case is a name the registry read and did not find, so the author can fix it.
• model_provider_refused, read_tool_needs_approval — the vendor answered, and policy answered.
• run_already_ended — the claim found nothing to claim, and a person who cancelled the child is the ordinary cause. A cancelled child stays tolerable, so the reaper's lost worker is tolerated with it.

⚠️ A branch node declares no continue_on_error. The flag is on the three types that do work. So condition_unreadable reaches a node that declares the flag only across a Run boundary. A subworkflow node reads a child Run that failed on its own branch. The code must still be in the set. Without it that parent tolerates a snapshot nothing could read.

Cancellation, the wall clock and accrual are checked before every node

WorkflowStepExecutor checks all three, in this order, and each one answers a different question.

• The agent.run_control row. Safe boundaries sit between nodes. A workflow that meets a cancel stops at the next node and answers cancelled, which names no RunManager method because cancel() already wrote the status. It is first for two reasons. A person who pressed stop must not pay for an accrual read. A cancelled run must not be reported as out of budget.
• The wall clock, against run_deadline(run, ceilings). It is second, because it reads no record. A run past the deadline answers the code max_run_duration and the reason limit_reached. The run keeps the outputs of the nodes it did finish.
• Accrual, scope run_and_day. The run tree total and the organization day. RunExecutor claimed the run once and returned long before node 4.

All three sit inside the node's own step, so none repeats on a replay.

⚠️ RunCeilings.exhausted cannot answer the wall clock here. The walk hands every node the frozen budget, so max_run_duration holds the whole ceiling and never counts down. Only AgentExecutor._remaining subtracts the elapsed time, and it does so per segment.

⚠️ run_deadline reads the frozen ceilings, and never a remainder. One function answers the deadline for both readers: this gate, and the ToolInvocation that caps an approval expiry. A remainder subtracts the elapsed time a second time. A run at hour 3 of a 4 hour budget then files every approval already expired.

The deadline bounds the walk between nodes, and not inside one. A tool call clamps on the same instant. A child run does not: the parent waits for a child that holds its own ceiling, so the parent notices at the next node.

What a workflow run produces

A workflow node produces a value. By default, the Run result is the map of what the nodes produced.

RunResult(
    summary='4 of 5 nodes produced output',
    output={'classify': {...}, 'draft': {...}},   # one key per node with an output
    refs=[ResourceRef(kind='prospect', id='...')],
)

Only tool, agent and subworkflow produce a steps.<id>.output, so only those three appear. A node that did not run is absent rather than null, which is the same rule the reference check uses. RunManager bounds the map at 32 KB with bound(), exactly as it bounds an agent result.

An optional workflow-level result object selects the completed Run result from input and steps.<id>.output references. The executor resolves it after the full walk succeeds. This lets a product keep one small final projection without storing every intermediate output before it. If the selector cannot resolve, the Run fails with unresolved_reference.

A partial success keeps the default node-output map and its partial_reason. It does not resolve the selector, because the selected final node might not have run. This map is diagnostic only. A product consumer must reject a partial Run.

A successful write tool contributes resource refs when its live ToolSpec declares an item_kind. One returned row contributes one ref from its id. A list contributes refs in returned order. A read tool, a failed call, a tool with no item_kind and a row with no id contribute none.

The node stores its contributed refs in its memoized result. The workflow keeps declaration order and removes duplicate (kind, id) pairs. The first ref wins. A replay reads the memoized refs. It does not read a later live tool declaration or repeat the effect. Container nodes add no refs of their own.

The map is not a second state store. It is rebuilt on every pass of the body from the memoized step outputs, and no table holds it.

Child Run rules

• The child gets parent_run_id and inherits root_run_id.
• The child's start key is <parent_run_id>:<node_id>. Every other source of a start key is a delivery identity, and a workflow node has no delivery. It has a node identity, and that identity is already unique across the whole workflow and stable across a retry. This is the key that makes the next rule work: without a deterministic one, a retried node writes a new key and starts a second child, and duplicate is unreachable. Phase 4 keeps child Run nodes at width one.
• A refused start fails the node, and there is no child Run to show. start() can refuse before any Run exists, for the organization day cap or for a cancelled parent. The node treats it as a node failure with that reason, so continue_on_error decides the rest exactly as it does for a child that ran and failed. Do not retry: neither reason changes on a retry inside this Run. Two refusals are platform stops instead. snapshot_unbuildable answers invalid_snapshot, and policy_unavailable answers itself, because a gate that could not decide is a fault and never a business answer.
• A child ended at admission carries its own code. Admission runs after the insert, so a refused child answers started beside a Run that already reads failed, and the node never invokes it. Read as child_failed a node declaring continue_on_error walks past a policy outage. StartRunResult.error_code carries the code the Run ended under, because Run is the narrow read on this path and RunPayloads holds error. A rule that denied answers policy_denied and stays tolerable; a fault answers its fault code and stops the node.
• A duplicate answer is not a refusal. A retried node meets the child its earlier attempt created, and start() returns that Run. The node uses it and continues. It must never treat a duplicate as a failure, and it must never start a second child.
• The child Principal is the parent Principal. A step cannot widen authority.
• The child answers the three keys run.execute answers. The node reads result.output into steps.<node id>.output and error into its own failure. A child that ended before its claim answers a status of null beside an error naming that, so the node reports the reason rather than "the child run ended None".
• A failed child fails its parent node, unless the node declares continue_on_error. That flag is node configuration, not a new node type.
• A platform stop is never tolerated, whatever the node declares. The code carries that rule, so it survives the JSON of a ChildStart and a child Run that ended in another process.
• A cancelled parent cancels every descendant. RunManager.cancel() writes the control row for the whole subtree and publishes agent/run.cancelled for each Run ID.
• The depth cap from runtime definitions is checked again at run time, by the same function. A definition published after the parent was validated cannot deepen a running tree.
• The run time check needs the depth above this Run, and no column carries it. The definition side answers how deep the subworkflow goes below; the Run side answers how deep this node already sits. That is a walk up parent_run_id until it is null, which the cap bounds to a handful of primary key reads. Do not add a depth column: it would be a fourth frozen column to prove on every insert, and the walk is shorter than the tree it measures.
• The check is one inequality, and it must be written down.
  TEXTCopy

  a = the workflow Runs from the root to this Run, counted inclusive
  d = depth(the target workflow), from the definition side
  refuse the node when   a + d > 3
  

  
  Only a subworkflow node deepens the count. An agent Run starts no child of its own, so every Run above a workflow child is a workflow Run and the walk counts what it should. d reads the live published definitions and never the parent's frozen snapshot, which is the whole reason the check runs twice: a subworkflow published after the parent was validated is exactly the case publish could not see. So the shared function takes a definition loader, and the run time caller passes one that reads published_config.

Parallel node failure

A failed branch does not cancel its siblings. The parallel node waits for every branch to settle, then reports. This keeps the effects predictable, and it matches "one person fails in a batch, the others continue" in the email sequence.

⚠️ A business failure is a returned value, and it never raises. An exception out of a step is retried by Inngest and then fails the whole function, which ends every sibling branch. That is the opposite of the rule above, and it is the default behaviour of the obvious implementation. A node body therefore catches the failure it understands and answers a NodeOutcome naming it. Only a retryable platform fault leaves the body as an exception, exactly as it does on the agent side.

An approval inside a parallel node

A wait is not a settle, so the two rules above do not compose on their own. A branch that needs a person can hold the container for three days while its siblings are already done.

Each branch owns its own wait. A parallel node runs its children as independent Inngest steps, and a step that parks does not hold the others.

parallel
  ├─ branch A  ── tool ── done
  ├─ branch B  ── approval ── waiting 3 days ── resumed ── done
  └─ branch C  ── tool ── failed
                     the node settles when the last branch settles

Three rules make that safe.

1. A branch approval never raises out of the container. In an agent loop an approval stops the segment, because the model must not read a pending value. A workflow branch is not a model, so its approval step waits in place and the sibling steps keep running. This is an Inngest behaviour, so a contract test pins it, exactly as one pins the suspended parent holding no concurrency slot. tests/agentic/runtime/inngest/test_parallel_contract.py is that test, and it runs a real dev server: one branch parks and one branch of two steps must reach its second step. An approval wait is the same wait_for_event primitive, so the approval node extends that case rather than building a second harness. A parallel container is the one place the platform depends on a step parking without stalling the steps beside it, and reasoning about it is not the same as measuring it.
2. The Run is waiting while any branch waits. waiting_ref_id names the oldest unresolved approval. A person opening the Run sees one thing to do, and Human Review lists every branch approval separately.
3. The container inherits the longest timeout. The node settles at the last branch, so max_run_duration is the backstop for the whole container, exactly as it is for one wait.

Two branches may each raise their own approval. They are two rows in the inbox, they resolve in any order, and the container settles when both are answered. waiting_ref_id names the oldest of the two, so each raise reads oldest_pending() rather than writing its own id: two raises are two steps in either order, and the field would otherwise name whichever committed last.

⚠️ An approval never shares a container with a wait. Approvals are countable and a wait is not, so a container holding one of each wakes the Run on the answered approval while the other branch is still parked, and the reaper then fails it. Definitions owns the publish rule that refuses the shape.

A waiting Run can still be executing. Rule 2 says the Run reads waiting while any branch waits, so a sibling branch runs its next node against a Run every surface calls waiting. Nothing may read waiting as "no work in flight". The reaper is already correct here for its own reason: it reads queued and running and never waiting.

Waits and approvals

An approval may outlive the worker that proposed it. Everything needed to resume is durable before the wait starts.

Agno proposes a tool call
  -> ToolInvoker -> Policy: require_approval
  -> persist the approval row, ON CONFLICT (run_id, idempotency_key) DO NOTHING:
       run_id, tool name, exact arguments, args hash,
       idempotency key, Agno session identity, expires_at
  -> RunManager.mark_waiting(run_id, waiting_on=approval, ref_id=approval_id)
  -> the segment step returns needs_approval
  -> Inngest wait on agent/approval.resolved
  -> the worker may disappear
  -> a person resolves the approval
  -> ApprovalService publishes agent/approval.resolved
  -> a fresh worker rebuilds the agent from the snapshot and its session
  -> the runtime executes the approved call from the approval row
  -> it resolves the requirement, and acontinue_run fills that call's result slot

The approval row is claimed on the journal key, exactly as an executed call is. A segment that writes the row and then dies before its step returns is replayed by Inngest, and the replay rebuilds the agent, replays the journal, and reaches the same proposal again. An unconditional insert files a second inbox row, so a person sees one send twice and approves it twice.

The key is the same <run_id>:<step_path>:<args_hash> the journal uses, and unique(run_id, idempotency_key) makes the second write a no-op that reads the first row back. It has to be the journal key rather than a key of its own: a pending approval is a proposed effect, not an executed one, so agent.idempotency_keys does not hold it, and the two must agree on what "the same call" means.

That also closes an orphan. Without it the first row stays pending for ever: its expiry writer is the Inngest wait that its own segment never reached, so nothing terminal ever writes it. Human review says every terminal state has a real writer, and that row would be the exception.

The wait timeout is computed from approval.expires_at, and never from a second timer of its own. ApprovalService re-checks the proposal immediately before the effect runs, and a stale approval fails closed. Policy and governance owns both rules.

The resume does not wait for the model to ask again. The approval row holds the tool name and the exact arguments, so the runtime executes that call and hands the outcome back through the paused requirement. Relying on the model to re-propose it would make an approved send depend on a sampling outcome, and a person who approved would sometimes get nothing. A rejection resolves the requirement as rejected instead, and the agent adapts. ToolInvoker owns that answer, and it reads the rejection of the claim as well as the row it was handed, so a later segment of the same run does not ask the person again. Tools and integrations owns the invoke path this uses.

Cancellation

Cancellation is cooperative, durable and idempotent. It lives in agent.run_control.

RunManager.cancel()
  -> read the named Run     id, parent_run_id, root_run_id, organization_id
                            every filter below also carries organization_id
  -> STATUS FIRST
       the ROOT      -> one UPDATE, WHERE root_run_id = :id
       a MID-TREE Run-> per level: UPDATE this level, THEN read its children
       UPDATE agent.runs SET status='cancelled', ended_at=now(),
                            waiting_on=NULL, waiting_ref_id=NULL
              WHERE <filter> AND organization_id = :actor_org
                AND status IN ('queued','running','waiting')
       THEN read the answer, which is not the same set as the write
       SELECT id WHERE <filter> AND organization_id = :actor_org
                   AND status = 'cancelled'   -- this call's movers, and a crash's
  -> THEN publish agent/run.cancelled per returned run_id
       Inngest stops a queued, waiting or running function
  -> THEN the control rows
       INSERT agent.run_control (run_id, cancel_requested_at, by, reason)
              ON CONFLICT (run_id) DO NOTHING     -- the returned ids only
       a running step reads this at its next safe boundary
  -> LAST the pending approvals, whether or not any Run moved
       UPDATE agent.approvals SET status='cancelled'
              WHERE root_run_id = :id AND status = 'pending'      -- root cancel
       UPDATE agent.approvals SET status='cancelled'
              WHERE run_id IN (:movers) AND status = 'pending'    -- mid-tree

The walk descends from the level it attempted, never from the Runs that moved. A Run that did not move still has live children: an intermediate node a reaper failed, a branch that ended under continue_on_error, and every node of a subtree that a crashed earlier pass already cancelled. Descend from the RETURNING set and any one of those hides its whole subtree, permanently. Measured on a root -> M -> C -> G tree: a re-run of a mid-tree cancel answered an empty set and left C and G queued, which is the exact case the re-run advice below exists for, and a terminal middle node hid its grandchild with no crash at all. Keep the moved set for the control rows, and walk the attempted set.

A mid-tree cancel writes each level before it reads the next one, and the order is the whole reason the walk is safe. Collect the whole subtree first and a child born during the walk is missed twice over: it is not in the collected set, and its parent still reads running, so the parent status check does not refuse it either. That check was added for a window of one statement; a collect-then-update walk widens it to the whole walk, which is several round trips on a deep tree. Cancel level k first and every parent is already cancelled before its children are enumerated, so start() refuses a new child at level k+1 and no descendant can be born unseen. It costs no extra round trip. It is the same reads and the same writes, in the other order.

Every filter carries organization_id, because RLS is not on this path. Each write to the agent schema uses the service role, so a policy filters nothing and raises nothing. cancel() takes a run ID from a caller, and ActorIdentity carries the organization, so both filters take it: root_run_id = :id AND organization_id = :actor_org, and the same on each level of the walk. assert_run_shape() already keeps a tree inside one organization, so the predicate can never narrow a legitimate cancel, and it is the only thing standing between a caller's run ID and another tenant's Run.

The cancel answers what is cancelled, not what it moved, and only those get a control row. The status update already skips a terminal Run; the control insert must skip a Run that succeeded or failed, or cancelling a tree of five hundred children where four hundred and eighty succeeded writes four hundred and eighty rows nobody reads, into the table whose whole design is that it stays empty.

It must not skip a Run that is already cancelled, and that is why the answer is a read rather than a RETURNING. This page tells the operator to re-run cancel() to place the rows a crash lost. Answer with the Runs this call moved and that instruction is dead: the first pass already cancelled them, a cancelled Run is not in the from-states, so the re-run matches zero rows and repairs nothing. The worker inside a segment then reads agent.run_control, finds nothing, and runs to max_run_duration on a Run a person stopped. So the write is one conditional UPDATE and the answer is a second statement, WHERE <filter> AND organization_id = :actor_org AND status = 'cancelled'. It names the movers of this call and the movers of the crashed one together, and the control insert's ON CONFLICT DO NOTHING makes the overlap free.

The write needs no paging and the read does, which is the opposite of what it looks like. A returned representation does not obey PGRST_DB_MAX_ROWS, while a read does. Measured against postgrest 14.7 with the limit at 1000: a PATCH matching 1,501 Runs updated 1,501 and returned 1,501, and a GET over the same set returned 1,000. So the status write is one statement at any width, and the answering read is paged by the same keyset rule the walk uses. Do not add a page loop to the write to be safe: it costs a round trip on every wide cancel, and it would not help if the behaviour ever changed, because that statement has already moved those Runs out of the status filter and no re-run can name them again.

The control row insert is ON CONFLICT DO NOTHING, or the second cancel raises. run_id is the primary key of agent.run_control, so a plain INSERT answers 23505 on a Run that already carries a row. This page says the caller may re-run cancel() to place the rows a crash lost. Without the clause, that re-run fails on the first Run that already has one and never reaches the ones that are missing, which is the exact case the advice exists for.

cancel_requested_by is a public.profiles key, so only a person fills it. cancel() takes an ActorIdentity, and that actor is a user or a machine. A machine identity has no profiles row, so writing its id answers 23503 and the whole cancel tail is lost. The column is nullable for this reason: write the id for a user actor, and NULL for every other kind. cancel_reason still records who asked, in words.

The fourth write clears the approvals of the subtree. A root cancel filters root_run_id = :id, which is one indexed statement whatever the width. A mid-tree cancel names the Runs the walk reached, because root_run_id there would clear the approvals of the siblings and the parent, which the person did not ask to stop.

That write runs even when no Run moved, and it catches its own fault. A branch that already ended moves nothing and can still hold a pending approval: the invoker files the row, and the segment then fails inside the window the fail path clears it in. Guard the write with the other three and no later cancel() reaches it, because moved is empty every time. And a person who stopped a Run must not read a failure for an inbox row, so a fault there is logged rather than raised.

Every filter names pending. A person who resolves an approval in the same moment keeps that decision, and the freeze trigger would refuse the overwrite in any case.

The approvals filter takes the walk, and the other three writes take the movers. cancel_subtree walks every descendant and moves only the Runs that changed status, so the two sets differ by the descendants that already ended. Handed the movers, the fourth write skips exactly the Run whose approval is already an orphan, while a root cancel of the same tree clears it, and one intent gives two answers. A stopped function and a cooperative control row mean nothing on a Run that did not move, so those two keep the movers. cancel_subtree therefore answers both sets from one walk: a second read of the tree would race the first, and a child born between them would land in one answer and not the other. ENG-2157.

A terminal write that is not a cancel clears the approvals of its own Run. The same orphan arrives without any cancel: the invoker files a row and the segment then fails, or a resume leaves the Run waiting and the loop ends it on orphan_approval. resolve still accepts such a row and the approve route answers 200, so a person answers a question on a Run that ended minutes ago. fail() therefore clears the pending approvals of the Run it ended, when the write moved a row: zero rows matched means another writer owns the Run, and a cancel of it owns its approvals. succeed() does not, and an ordinary segment failure needs no help from it: AgentExecutor.execute() supersedes on every path that returns, so a segment that ended the Run either way already cleared its own rows. The write is for the endings that reach no segment supersede, which are the loop's own orphan_approval, the reaper, run_failed when Inngest gives up on the function run, an admission that could not write its hold, and a supersede that swallowed its own fault. A segment that raises reaches neither write: the raise leaves no terminal status, and the Inngest replay reads back the pending row that carries the idempotency key it reuses. run_failed clears that row only once Inngest stops replaying, and no replay then wants it. Every failed Run therefore pays one conditional statement and most match no row, and the alternative is an inbox that keeps a row for every such Run until its own clock expires it. ENG-2118.

The four writes are ordered by what a crash costs, because none of them shares a transaction with the next. The status is the product truth every surface reads, so it goes first. The Inngest publish is what actually stops a live function, so it goes second. The control rows are the cooperative backstop for a step already inside a segment, so they go third. Nothing waits on an approval of a stopped Run, so the approvals go last.

No filter carries a list of Run IDs, because a wide tree makes the URL too long to send. ac-python-api filters through PostgREST, so id IN (...) is a query string, and Kong refuses a request line over 8 KB. A UUID plus its comma is 37 bytes, so one in.() filter holds about 220 identifiers. Measured against the local stack: 200 ids is a 7,462 byte URL and answers; 400 ids is 14,862 bytes and answers 414. A five hundred person email sequence is the design's own named case, and it is twice over that line.

That is the same failure observability avoided by stamping root_run_id on every usage row rather than filtering the meter by twenty thousand span identifiers. Cancellation takes the same answer, and agent.approvals therefore carries root_run_id exactly as agent.spans does.

cancelling the root      the status and approvals writes filter on root_run_id = :id,
                         whatever the width; the control rows take the returned ids
cancelling a mid-tree    batch each in.() filter at 200 ids, and page the walk

PGRST_DB_MAX_ROWS is 1000, and it truncates a read silently. The level by level walk must therefore page, and never treat one response as the whole level.

Page the walk by keyset, and end on an empty page. Two things make the obvious form wrong. A Run has no ordering of its own, so PostgREST returns physical order, and any write that cannot be a HOT update relocates a row between two pages: measured, 1,500 children read in two offset pages with only heartbeat_at rewritten between them answered 1,000 and then 500, missing 244 children and repeating 244. SpanRecorder writes that column every ten seconds on every live Run, so the case is ordinary rather than rare. A Run is never deleted and its id never changes, so ORDER BY id with id > :last cannot skip one. And the loop must end on an empty page rather than a short one, because PostgREST applies the smaller of the client limit and db-max-rows, which is a per project setting the application cannot read: a project that sets it below the page size answers a short first page, and a loop that stops there truncates the level with no error.

The approvals write is cleanup, not correctness. A cancelled Run must not leave a row in a person's inbox, and human review names RunManager.cancel() as the writer of that cancelled approval status. It sits last because the inbox already defends itself: the queue reads the Run state and shows an approval on a cancelled Run as non-actionable, so a lost write costs a stale row rather than a wrong action.

Lose the tail and the Run still reads cancelled, which is the answer the person asked for. A worker inside a long segment then keeps running with no signal, and the reaper cannot help: it reads queued and running only, so a cancelled Run is invisible to it. The ceilings and max_run_duration bound that worker, and re-running cancel() places the missing rows. Order the writes the other way and the failure is worse in every case: the person is shown a failure the reaper wrote, on a Run they stopped themselves.

Any transition out of waiting clears the waiting three in the same statement. runs_waiting_shape is an equivalence, not an implication, so status='cancelled' beside a surviving waiting_on raises 23514 and the cancel fails. It is the easiest line to leave out here, because the queued and running cases work without it and the waiting case is the one a person actually cancels. The same applies to succeed() and fail(), which the transition table also allows from waiting.

Cancelling the root is one filter. Only a mid-tree cancel walks. Cancelling a Run cancels its subtree, not its siblings and not its parent, so root_run_id is the wrong key for a mid-tree cancel: it would stop the whole tree, which is not what the person asked for. But when the Run is the root, its subtree is exactly root_run_id = :id, and every row carries that column. A person stopping a Run from a surface is always that case.

That matters because the walk cannot be a recursive CTE. ac-python-api reaches Postgres through PostgREST, which cannot express one, exactly as it cannot express the start transaction or a publish transaction. A mid-tree cancel therefore reads parent_run_id IN (:level) once per level, which the nesting depth cap bounds to a handful of round trips. Do not add a write RPC for it, and do not reach for root_run_id to avoid the walk.

A cancelled child fails its parent node, and continue_on_error decides the rest exactly as it does for any failed child.

The status is written before the control row, and the order is the whole guarantee. They are two statements, and PostgREST gives them no transaction, so a crash lands between them.

control row first, then a crash
  control row set, status still `running`
  nothing writes `cancelled`; the reaper reaches it later and writes `failed`
  the person who pressed stop is shown a failure

status first, then a crash
  status `cancelled`, no control row
  the worker misses its safe boundary and runs one step longer
  its terminal write is refused, because the transition table stops at a terminal Run
  the record is right and the work is wasted

Wasted work is cheaper than a wrong status, so the status goes first. Cancel is idempotent, so the caller may simply run it again to place the missing control rows.

cancel() writes the status itself, and that is not redundant with the control row. Inngest stops the function, so no worker reaches a finalize, and nothing else would ever move the Run out of queued or running. The reaper does not cover it either: the reaper writes failed, and a Run a person stopped is cancelled. The two writes answer two questions. The status is what every surface reads. The control row is what a still running step reads at its next safe boundary, and it stays a separate table because that check happens about forty times per Run.

The whole subtree moves in one pass, and terminal Runs are skipped. A child that already succeeded keeps succeeded; cancelling a finished tree changes nothing and raises nothing. Cancel is therefore idempotent by construction: the second call matches zero rows.

Safe boundaries are: before a tool call, after a tool call, and between workflow nodes. Never stop an irreversible write or send in the middle. If an external call cannot be interrupted safely, let it settle, record its effect, then stop.

Inside an agent segment the tool adapter is that boundary, and it reports cancelled. Nothing else in the loop reaches a safe point: Agno owns the turns between the tool calls. So the adapter reads agent.run_control, halts the loop the same way an approval halts it, and the segment returns stop = cancelled. The status write already happened in cancel(), and RunManager refuses a second transition on a terminal Run, so the report is a record rather than a decision. Without the member the adapter would have to call a cancel a failure, and the surfaces would disagree with the person who pressed stop.

Steering is deferred to V2. A person who wants to correct a live Run cancels it, then starts it again with new input. Steering costs a control column, a read point at every safe boundary, and one more outcome branch on the front door, and cancel plus restart answers the same need in V1.

Ceilings

Every Run carries hard ceilings, from the definition budget and capped by the Policy limits.

max_segments        the loop bound of the durable function          per Run
max_agent_turns     model turns in one Run                          per Run
max_tool_calls      tool calls in one Run                           per Run
max_run_duration    wall clock from started_at, including waits     per Run
max_cost_cents      reads the canonical usage meter at accrual      per TREE

Four ceilings are per Run. The cost ceiling is per tree, and it comes from the root.

A workflow creates a child Run for every agent and subworkflow step, and every one of those definitions carries its own max_cost_cents. If each child enforced its own, a workflow of ten children could spend ten budgets, and the meter sums by root_run_id anyway, so the two would disagree.

a child Run inherits the root's max_cost_cents, and ignores its own
the other four ceilings stay the child's own

The split is not arbitrary. Money is one bill, and the tree spends it together. Turns, tool calls, duration and segments bound a loop, and each loop is its own, so a child that runs away must stop on its own count rather than on its siblings'.

This matches the two rules beside it. The child Principal is the parent Principal, and the child cost ceiling is the root's. Authority and money both come from the top; the loop bounds do not.

A ceiling settles the Run under the one status rule: a successful partial Run when the Run produced output, and a failed Run when it produced none. Without max_run_duration a Run that waits on an event nobody sends never ends.

A segment receives what is left, not the whole ceiling

max_agent_turns and max_tool_calls bound a Run, and a Run can be five segments. A segment handed the whole ceiling would spend it, and the next segment would be handed it again, so a Run with max_agent_turns=20 over five segments could spend a hundred turns.

So AgentExecutor subtracts before it builds the request, from the same durable counts accrual reads.

request.ceilings.max_agent_turns  =  ceiling  -  count of llm spans WHERE run_id = this Run
request.ceilings.max_tool_calls   =  ceiling  -  count of tool spans WHERE run_id = this Run,
                                                 attributes.replayed IS NOT true
request.ceilings.max_run_duration =  ceiling  -  (now - started_at)

Every filter here is run_id. Only max_cost_cents reads root_run_id. That is the same split as the section above, and it is the one line to get right: three of these bound a loop, and each loop is its own.

All three are remainders, including the duration. The Run owns a wall clock from started_at, and a segment cannot apply that as a timeout without knowing what an earlier segment already spent. Every subtraction floors at zero, and a remainder of zero ends the Run before a segment starts.

The segment duration is not one of them. MAX_SEGMENT_DURATION_S bounds the largest step the worker runs, and it comes from the deploy rather than from a definition. The durable segment step applies that wall clock outside the agent executor. RunCeilings does not carry it. See the Inngest boundary.

The accrual check before a segment belongs here, not to RunExecutor. RunExecutor claims the Run and dispatches it by kind, once per function run, so it is not running when segment 4 starts. AgentExecutor already subtracts the remainders at the top of every segment, and WorkflowStepExecutor is the equivalent code at every node. Those two are the per-segment and per-node accrual callers. See policy and governance.

AgentExecutionResult.turns_used stays the segment's count, which is what the loop needs before the segment ends. Neither number becomes a column: the span tree already holds both, and a counter would be one more thing to keep true. A remainder of zero ends the Run on ceiling, exactly as a segment that reaches it mid loop does.

max_segments looks redundant beside the other four, and it is not. The others bound cost and wall clock from outside the code. max_segments is the bound on the while loop itself, and a durable function must never contain an unbounded loop. Keep it whatever the other ceilings say.

The wall clock starts at started_at, not at created_at. A Run queued behind an organization concurrency limit is not spending its budget. Measure from creation and 500 saved searches firing at 09:00 kill each other: the last one in the queue passes its ceiling before a worker ever claims it, and the failure looks like a timeout rather than the queue it is.

A Run that never starts is the reaper's case, not the ceiling's.

Accrual checks max_cost_cents at the top of each segment and each node, and a metered tool call checks it too. Without the second check a fan out inside one step can pass the ceiling by the width of the fan out. A metered call reads the run tree total only; the organization day belongs to the node boundary. See policy and governance.

Failure, retries and orphans

Both backstops are needed. Without the run.failed handler a Run sits at running forever after Inngest gives up. Without the reaper a Run that never reached Inngest sits at queued forever.

⚠️ The id in that event may carry the app id. The SDK builds a function's fully qualified id as <app_id>-<fn_id>, so the failure event is expected to read agencycore-agentic-run.execute. Nothing measured yet says which form the event carries, and a filter on the wrong one matches nothing while raising nothing — the Run simply stays running until the reaper's window, hours later. The filter therefore accepts both exact strings until a live run narrows it. The filter also sits on the trigger and not in the body, so a function that must not act never starts.

inngest/function.failed fires for every function in the app, so the handler filters on the one it owns. The reaper emits it, and so does run.failed itself. Unfiltered, the handler tries to read a run_id from an event that has none, and its own failure re-triggers it. One condition closes both: act only when the failed function is run.execute, and read the Run ID from the original event data that the failure event carries.

A retry in progress needs no product state. The failed span is written before the retry starts, so the span tree shows the attempt that failed and the attempt that followed.

⚠️ execution_ref.attempt does not move, so a surface cannot count retries from it. claim() runs inside a memoized step, so a later attempt of the same function run replays the first attempt's answer and writes nothing. The column records the attempt that claimed, and function_run_id is the field that correlates every attempt. A surface that wants a retry count reads the span tree, which does grow one failed span per attempt.

The wait sweep

A waiting Run holds no worker, so it writes no heartbeat and the reaper's own read never sees it. iter_live_stale() takes queued and running and nothing else, and cross-document invariant 12 keeps it that way: a person answering an approval slowly must never be failed on a heartbeat.

The Run row therefore carries waiting_expires_at. mark_waiting() writes it, and every write that names a target other than waiting clears it, exactly as it clears waiting_on and waiting_ref_id. One rule in build_transition_values() owns all three, so claim(), resume(), succeed(), fail() and cancel() repeat none of them. runs_waiting_deadline_shape is an equivalence, not the weaker rule runs_waiting_ref_shape holds: a waiting Run with no deadline is a Run the sweep can never read, which is the fault the column exists to remove.

Three writers, and each one has a clock already.

an agent segment      the approval's own expires_at
a wait node           the hold plan_wait() clamped to the run deadline
the admission hold    now(), on the first of its two marks

The first admission mark writes now(), and that is not a placeholder. No approval row exists at that point, so no clock does either. The value states what the row means: this hold carries no clock, and if nothing replaces it inside the grace window then nothing ever will. The third step writes the approval's own expiry over it, milliseconds later. A Run stranded between the two is therefore recovered in one grace window rather than in one approval TTL.

The sweep reads a deadline plus a grace window, because no liveness lookup exists. SDK 0.5.18 exposes no method that asks whether a function run is alive, which is the same limit that gives the reaper its six hour abandon window. The live waiter fires its timeout at the deadline and then writes the ending in two more steps, which take seconds even under retry. INNGEST_AGENTIC_WAIT_GRACE_SECONDS is ten minutes, far above that and far below any approval TTL.

The ending is one code, and the sweep reads no approval row. A recovered Run ends failed under wait_abandoned. The sweep proves one fact — the Run passed its deadline and nothing wrote it — and that fact is the same for an approval wait, an event wait, a delay wait, and an admission hold whose approval row was never written. Two of those four hold no approval at all, so a code that named the approval would name nothing for them.

⚠️ It is never approval_expired. That is a partial success the live waiter writes after it ran the timeout itself. This sweep ran no timeout, and a person may have pressed Approve. failed states what happened: the platform lost the writer of a Run that was waiting. This is the recovery behaviour ENG-2120 chose for a Run whose approval was answered before the function run was lost — it takes the same ending as one nobody answered, because no cheaper reading of the row is honest.

The cost of that choice is written down here. A person approves, the function run is then lost, and the Run stays dead until the approval's own deadline — up to the TTL. No earlier detection exists while the liveness lookup does not.

The sweep writes no approval row either. Every pending read is already clocked: oldest_pending_for_run() filters expires_at > now, and the inbox lists a pending row past its expiry in the expired page. So a row left pending by a lost function run is inert, in the inbox and in resume() alike.

⚠️ The write is guarded by a filter, and never by a re-read. It is conditional on status = 'waiting' and waiting_expires_at < cutoff, the same cutoff the page was read under. A person who answers in the gap either wakes the Run or gets it re-aimed, and oldest_pending_for_run() reads a live row alone, so a re-aim always writes a deadline in the future. That Run then matches zero rows. A re-read would cost one round trip for each Run and answer the same thing.

One bounded read, and no keyset. iter_live_stale() pages because a Run it leaves queued keeps its place and would fill every later page. This read has no such head: the sweep ends every Run it reads, and a Run it cannot end has left waiting already. The answer is ascending on the deadline, so the oldest go first and the next pass takes the rest. idx_runs_waiting_expires_at is (waiting_expires_at) WHERE status = 'waiting', so the filter and the order are one index range.

It runs before the orphan span sweep, and the order carries the cause. close_orphans() stamps wait_abandoned on the spans of the Runs this sweep ended. Run it after, and the orphan sweep reads those same spans first and stamps the generic run_ended over the true code.

This sweep ends Runs. The two span sweeps close spans. They stay separate methods.

The heartbeat has one writer per moment

claim() stamps heartbeat_at once, and a segment may approach its 90 second wall clock. If nothing wrote it in between, the reaper would have no safe stale_after: set it under the longest segment and it kills healthy work, set it over and a dead worker sits for that long.

Two components write the column, and they cover moments that do not overlap. RunManager stamps it at the insert and at each lifecycle write, because no span covers those two moments. SpanRecorder owns everything between the claim and the finalize. Read the heading as one writer per moment, never as one statement in the code base.

SpanRecorder writes the heartbeat. It already fires on every span open, and it already carries the run ID, so it needs no new call site.

span opened -> the process wrote this Run more than 10s ago
               -> UPDATE agent.runs SET heartbeat_at = now() WHERE id = :run_id
            -> the process wrote this root more than 60s ago
               -> UPDATE agent.runs SET heartbeat_at = now() WHERE id = :root_run_id

⚠️ The window is tested in the process, and it cannot be tested in the statement. A PostgREST filter value is a literal Postgres casts, not SQL it evaluates, so heartbeat_at < now() - 10s has no filter form. Observability and operations carries the measurement and the reason an application clock cannot stand in for it.

⚠️ The reaper query below meets the same limit, and it answers it differently: it is a scheduled read, so it computes its own cutoff and a clock that drifts changes when a dead Run is noticed rather than whether a live one survives.

The window keeps the write rate flat whatever the span rate. An agent turn, a tool call and a workflow node each open a span, so a live Run cannot go quiet for longer than one model call.

Two statements with two different guards, not one statement with one. A Run's own row is touched by one worker, so 10 seconds is free. A root row is touched by every worker in its tree, and an email sequence has five hundred of them. At 10 seconds those five hundred contend on one row for a write that almost always matches nothing. At 60 seconds the root is touched at most once a minute however wide the tree grows, and stale_after is 120 seconds, so the reaper stays well clear. Observability and operations owns the recorder that writes both.

The write covers the root as well as the Run, and that is not decoration. A top level workflow suspended on step.invoke is running and writes no spans of its own, sometimes for ten minutes. Without the root row in the WHERE, the reaper would see a stale heartbeat on a healthy parent whose child is working normally. A tree with any live work is live at its root.

It covers the root and the leaf, and not the levels between. A subworkflow in the middle of a three deep tree is neither run_id nor root_run_id for any span its descendants write. Two writes cannot be three, and walking every ancestor would put an unbounded loop on the hottest path in the schema. The reaper's Inngest liveness clause is what covers those rows instead.

heartbeat_at is set at insert, to created_at. The reaper reads queued too, and NULL < now() - interval is NULL, so a Run that never reached a worker would never be reaped. That is the exact case the queued clause exists for.

stale_after  >  the longest model call, plus a margin
             >= 2 x the root heartbeat window
             =  120 seconds in V1

⚠️ The second line is a floor, not a preference. SpanRecorder refreshes a root row at most once per 60 seconds per process, so a healthy root is quiet for that whole window just before each refresh. A stale_after one second above it leaves one second of margin, and any clock skew between the dyno and the database then fails live work. Two windows is what "well clear" means, and the settings field refuses less.

A Run that is genuinely stuck inside one call for longer than that is a Run we want the reaper to take.

A stale heartbeat is not enough on its own. A Run held behind its own concurrency lane is queued, holds no worker, and writes no heartbeat — which is indistinguishable from a Run that never reached Inngest. 500 saved searches firing at 09:00 is the designed case, and the four hundredth of them waits far longer than any stale_after. So the reaper also asks Inngest whether a function run is alive for that Run, and it only fails the ones with none.

That question has two shapes, because a queued Run has no function run ID to ask about. execution_ref is written by claim(), so it is empty for exactly the Runs the clause was added to protect.

The re-send is the cheaper answer and it needs no liveness API. The event ID is run.execute:<run_id>, so Inngest drops it when the Run is already queued, and delivers it when the dispatch never landed. One send repairs the second case and costs nothing in the first.

⚠️ A queued Run is never failed on age, at any age. An earlier draft failed one that had sat past the dedupe window. It cannot: a Run parked behind a saturated lane holds a live function run, and its heartbeat_at is frozen at the insert, so it is indistinguishable from a Run whose event was lost. Failing on age kills the healthy one and drops its work, and the Run then reads a reason that never happened — claim() matches zero rows when the slot frees, and the function ends having done nothing.

The repair does not need that cutoff anyway. A send that never reached Inngest left no dedupe entry, so the first re-dispatch delivers, within a minute. Only an Inngest-side fault survives to the window, and no rule can tell it from a backlog.

Past the dedupe window the reaper stops sending, and still does not fail. A re-send there is a new event: it would deliver a second function run beside a first that may still be queued, and claim() is re-entrant so neither would refuse. The stuck-run alert reports what is left, and a person acts.

The event ID dedupes for 24 hours, and that bounds the repair. Past the window a re-send is a new event, so a Run that has sat queued for a day is failed rather than re-sent. That is the correct answer anyway: nothing is coming for it. The same window bounds the stable ID that stops RunManager starting two workers on one Run, and it is the reason the ID is not a permanent guarantee.

The middle of a deep tree is protected by this clause, and by nothing else. SpanRecorder refreshes two rows, the Run's own and its root. A workflow that invokes a subworkflow that invokes an agent has a middle parent that is neither: it writes no spans while it is suspended, and no descendant refreshes it. Its heartbeat goes stale on every pass. It survives only because it holds a live Inngest function run, so the liveness clause is not an optimisation and must not be dropped from the query.

A queued child is never re-dispatched, and its parent's status is the fact. A child is invoked and never dispatched, so an event beside a live step.invoke would execute it twice. A child under a parent that has ended is the opposite case: no invoke is coming and nothing else ends it, so the reaper fails it with parent_gone. One read answers it — of the parents of the queued children this pass found, which have an ended_at. It mirrors the orphan span sweep exactly: a span running under a Run that ended, and a Run queued under a parent that ended.

One pass is bounded, and one Run's fault never ends it. Every write is guarded and counted, because an unguarded raise on one tenant's Run aborts the whole pass — including the orphan sweep that follows the loop, in exactly the incident the reaper exists for. The pass also stops reading past a fixed number of Runs: a pass that runs past its own execution window closes no orphan span at all, and the next pass is a minute away.

The heartbeat read never takes a waiting Run. The wait sweep is the one part of this cron that ends one, and it reads waiting_expires_at rather than heartbeat_at.

WHERE status IN ('queued','running')
  AND heartbeat_at < now() - :stale_after
  AND NOT EXISTS (a live Inngest function run for this Run)

A waiting Run holds no worker, so it writes no heartbeat. A read that only tested the heartbeat would fail every approval that a person answers slowly. A waiting Run has three owners instead: the Inngest wait timeout, computed from approval.expires_at; max_run_duration, which caps that timeout; and the wait sweep, which takes the Run when the first two lose their function run.

WHERE status = 'waiting'
  AND waiting_expires_at < now() - :wait_grace
ORDER BY waiting_expires_at
LIMIT :wait_sweep_limit

The reaper is a scheduled query, in the style of every other alert in observability and operations. It is not a second scheduler.

The reaper also closes the spans that died with the worker. A span is written open, so a crash leaves it running with no end. In the same pass that fails the Run, the reaper closes its open spans with worker_lost. Nothing else owns them.

It closes the spans of a Run that already ended, too. Inngest cancels a Run by stopping its function, so the worker does not reach a finalize and its open spans stay running under a cancelled Run. The heartbeat clause would never find them, because the Run is no longer queued or running. This sweep stamps run_ended and never worker_lost: it reads every open span under an ended Run, so it cannot name a cause.

-- orphan spans, whatever ended the Run
WHERE run.ended_at IS NOT NULL
  AND span.status = 'running'

⚠️ That is a read, and the write follows it by span id. PostgREST ignores an embedded resource filter on an UPDATE and still applies it to the response, so the one statement form closes every running span in the schema and reports only the rows it was asked for. See observability and operations.

That second clause also covers a Run failed by inngest/function.failed. Every open span therefore has exactly one closer, and the orphan span alert stays a real alert rather than a permanent one.

Live events

Durable state is Runs, spans and results. Live events are transient, best effort and possibly out of order. The Run row wins any disagreement.

Executor / SpanRecorder
      -> LiveEventPublisher
      -> Redis Pub/Sub
      -> FastAPI SSE
      -> Web / Channel gateway

This is the one live event vocabulary. A surface page links here. It does not restate it.

run.updated        span.started       approval.requested
text.delta         span.completed     approval.resolved
tool.updated       span.failed        result.item_ready
run.completed      run.failed

Every event carries run_id and root_run_id, and a span event carries span_id. It carries no timestamp, because it needs none: every write to agent.spans is an insert, or an update filtered on status = 'running'. A span therefore makes exactly one transition, and a client that holds a span as ok or error ignores any later event about it.

result.item_ready carries one bounded product projection:

{
  "type": "result.item_ready",
  "run_id": "...",
  "root_run_id": "...",
  "item_kind": "prospect",
  "item": "<ProspectSummary>"
}

For item_kind = prospect, item has exactly the ProspectSummary fields in surfaces. It adds no detail fields, people, signals or provider data. The prospect write handler reads this projection from the durable row after its write completes. A read or publish fault changes no successful Tool result.

The item carries its own revision. A client upserts it by item.id and keeps the newer item.updated_at. A repeated frame therefore changes nothing, and an older frame cannot replace a newer card. Equal revisions with different bodies are ambiguous, so the client refetches the durable row.

One channel per Run, and the publisher writes two. Each event goes to the channel of the Run that produced it, and it is mirrored to the root Run channel. A workflow parent therefore streams the work its child Runs do, and one subscription follows a whole tree. A root Run's own channel is the root channel, so a top level event is written one time.

The mirror stops at width. A tree of five hundred children publishing every span event to one channel is a firehose no client can render, and the useful signal is the parent's progress, not five hundred simultaneous tool spans.

tree under 20 live children   every child event is mirrored to the root channel
20 or more                    the root channel carries child run.updated, run.completed
                              and run.failed only
                              the detail stays on each child's own channel

run.failed is in that set for the reason run.completed is. A child that failed and never reached the root would leave the parent showing a run that works for the life of the page, and no later event corrects it.

The count is of live children, and it is taken one time, when the run scope of a function run opens. A workflow that ran five hundred children one at a time floods no channel, and the gate costs one statement per function run rather than one per event. An unreadable count reads as wide: zero would open the root channel to the tree the gate exists to stop.

⚠️ The subscriber reads the channel of the Run it opened, and it never rewrites the id. A reader that resolved the root first would make the child channel unreachable, and the 20 child rule above would have no caller. The tree wide property is the publisher's mirror, not the reader's lookup. A client that opens one child therefore subscribes to it directly, which the Run Explorer does when a person expands a row.

The durable span tree is unchanged either way, so nothing is lost: the events are a live hint, and the record is the query.

The reaper publishes, and its spans do not. close_orphans writes agent.spans through the repository, so no span.failed reaches a channel. The reaper therefore publishes run.failed after its last orphan page, and a client refetches the spans of a Run on each terminal Run event. Without that event a client keeps a reaped span on the screen as running, because no later event tells it to look again.

One event per reaped Run, and it comes after the spans. RunManager.fail takes announce=False from the reaper. A client closes its stream on the first terminal event it reads, so an event published with the write would land while those spans still read running, and the refetch it triggers would read them that way. That publish also sat inside the page loop, one Run at a time, and one publish waits up to five seconds on a Redis that accepts a connection and answers nothing: a pass holding a thousand Runs would spend hours there, before the orphan sweep the pass bound exists to protect.

⚠️ A cancel closes no span, and the sweep is the only thing that may. Inngest cancels a Run by stopping its function at the next step boundary, so the worker is alive while cancel runs. A span cancel stamped would lose the race the worker is about to win: SpanRecorder.close filters on status = 'running', so the worker's own close then matches zero rows and its output and its usage_id are dropped. The row would read error for a step that succeeded, and the link to its ai_usage_log row would be gone. The sweep runs long enough after the Run ends for the worker to have finished, which is why it, and not cancel, owns those spans.

The cost falls on the reader: a client that refetches on the Run's terminal event reads a cancelled Run's spans as running, and no later event corrects it, because the client closed its stream on that event. A live event is a hint and the durable record is the answer, so a client re-reads a Run it holds as terminal while any of its spans still reads running. No write races a live worker to make that true.

On reconnect the client subscribes first, then refetches the Run and its spans, then applies what it buffered. The reverse order drops every event in the gap, and V1 has no event replay table. See surfaces.

Scenarios that shaped this design

Each row is a case the design was tested against. The right column names the mechanism that answers it.

Rules

• RunManager owns the product state. Inngest owns the durability.
• RunManager owns status, waiting_on, waiting_ref_id, ended_at, started_at, execution_ref, segment_index and resumed_from_wait. SpanRecorder owns heartbeat_at between the claim and the finalize; RunManager seeds it at the insert and stamps it on each lifecycle write, because those are the two moments no span covers. Nothing else writes a Run column.
• Every lifecycle write is one conditional UPDATE against the legal from-states. Zero rows is a no-op, never an error.
• The transition primitive clears the waiting three whenever it names a target status other than waiting. No method repeats the three lines, and advance_segment() names no status and touches none of the columns.
• Every write to the agent schema carries its own organization_id filter. The service role bypasses RLS, so a policy filters nothing and raises nothing.
• The Run row carries its own start key, and the insert is the claim. There is no second table and no transaction on the start path.
• The start key names the delivery: the same intended start repeats it, the next intended start changes it. It never carries the definition the model chose.
• A duplicate start returns the Run the first one created. It is a success for every caller, and it sends no dispatch event. The reaper is the only repair for a send that never landed.
• The start key is a sha256 hex digest of the source and the delivery identity. A per-process hash gives one delivery two keys, and a digest of the delivery alone lets an API caller claim a trigger's key.
• start() calls four seams in one order. PolicyGate is one seam with two methods, one either side of the insert.
• A skill definition never starts a Run. start() refuses it with definition_not_published, before the insert can raise. The resolver answers tenancy alone, because None from it means definition_not_found.
• start() refuses a child whose parent Run reads cancelled. It reads the status, never the control row.
• The organization day cap is checked before the resolve and the freeze, and its denial creates no Run.
• A rejected admission approval cancels the Run. An expired one fails it.
• The lane follows the actor kind, not the source alone.
• start() dispatches only a top level Run. A child is invoked by its parent, and no HTTP caller supplies parent_run_id.
• The agent session is one row per Run, opaque above AgnoAgentRuntime, deleted with the Run.
• The Agno session identity is the Run ID. There is no second session identifier.
• A Run row carries its shape as constraints: the waiting pair, the waiting reference, the wait deadline and the ended pair.
• Span and session tenancy is a composite foreign key to the Run, never a copied column.
• agent.run_control stays its own table. A Run with no cancel request has no row, so the safe boundary check never reads the heap.
• A definition with Runs behind it is disabled, never deleted. The foreign key restricts.
• The runtime writes the session for every AgentExecutionResult, including the approval stop. A retryable failure writes nothing. The outer timer produces an ExecutionOutcome and makes no session-write promise.
• The session row carries the composed instructions beside the Agno half, because Agno rebuilds the system block from them on every run.
• An approval is claimed on the same semantic key as the effect it guards, so a replayed segment reuses the pending row.
• An approval the segment did not stop on is superseded when the segment ends. A segment that raises supersedes nothing, because the replay reuses the row.
• A child Run inherits the root cost ceiling. The other four ceilings stay its own.
• A replayed model turn counts against the turn ceiling. A replayed tool call counts nothing.
• The reaper closes an open span on any Run that has ended, whatever ended it.
• The wall clock ceiling runs from started_at.
• The Run snapshot is the stable execution input. Policy and tool state are read live.
• RunResult.output and RunResult.refs are bounded at 32 KiB. The Run input is refused above 32 KiB. The complete snapshot is refused above 256 KiB. Neither is trimmed.
• A successful write tool with an item_kind contributes ordered, unique refs to the workflow Run. Reads, failures and rows without an id contribute none.
• Run has a narrow read shape and a wide one. Only claim() needs snapshot.
• claim() is a re-entrant conditional update, and it returns None when the Run is no longer claimable.
• claim() clears the waiting three, and it sets started_at only once.
• cancel() writes both the control row and the cancelled status, over the subtree it enumerates down parent_run_id.
• A mid-tree cancel writes each level before it reads the next, so no descendant is born unseen during the walk.
• Only the Runs a cancel actually moved get a control row. RETURNING names them.
• advance_segment() is the one writer of segment_index, and the one thing that clears resumed_from_wait between segments.
• SpanRecorder writes heartbeat_at: at most once every 10 seconds for the Run, and once every 60 for its root.
• mark_waiting() and resume() are always written as a pair.
• segment_index and resumed_from_wait live on the Run, because a fresh worker reads them.
• A segment receives each remaining Run ceiling, including max_run_duration.
• A remainder of zero ends the Run before a segment starts. The runtime is never called with a ceiling of zero.
• The segment duration limit is a 90 second deploy constant, not a definition ceiling.
• A timed segment leaves the Run running, advances once, and waits on nothing.
• A timed segment on the last allowed index ends with the existing limit_reached result.
• A segment after a wall-clock yield continues a saved pause, continues saved history, or restarts from the original input when no session was saved.
• An approval proof survives a wall-clock yield only while the saved run remains paused; a fresh turn drops it.
• Inside a segment the tool adapter is the cancellation boundary, and it reports cancelled.
• The reaper fails a Run only when its heartbeat is stale AND no Inngest run is alive for it.
• cancel() writes the status before the control row, and cancelling the root needs no walk.
• A node that meets a duplicate uses the Run it gets back. It never starts a second child.
• A branch approval waits in place. It never raises out of a parallel container.
• A suspended Inngest run holds no concurrency slot. A contract test pins it.
• One Inngest function per Run. One step per agent segment or workflow node.
• A wide tool runs ordered read calls inside one node step and one full-item timeout per call.
• A wide node reserves 10 seconds for work outside its item clocks and has one outer step-budget guard.
• Business failures are item values. A ceiling, approval or platform fault stops the node under its normal run rule.
• A wide node never returns more than Inngest's 4 MiB step-output limit.
• A tool idempotency key uses the run, the step path and the argument hash. It never uses the model tool call ID.
• An approval wait is durable data plus an Inngest wait. It is never an in memory object.
• One expiry clock per wait, and one waiter that owns it.
• An approval dispatches the function, whether it came from admission or from a tool call.
• A ceiling produces a successful partial Run when the Run produced output, and a failed Run when it produced none. A fault produces a failed Run.
• A retry is telemetry. A Run under retry stays running.
• The dispatch event carries a stable ID, so one Run gets one function run, for the 24 hours Inngest dedupes an event ID.
• An invoked child carries run_id, organization_id and lane in its payload, because the concurrency key reads all three.
• Every child Run takes the batch lane, whatever started its tree.
• A child Run's start key is its parent Run and its node ID. It is the one start key that is not a delivery identity.
• The run.failed handler acts on run.execute only. Every function in the app emits inngest/function.failed, including that handler.
• The reaper re-dispatches a queued Run before it fails one. It interrogates Inngest only for a running Run.
• SpanRecorder refreshes the Run and its root. The levels between are covered by the reaper's liveness clause, and by nothing else.
• Interactive work and batch work hold separate concurrency lanes.
• Every Inngest function runs on the Connect worker, never on a web request. There is no short-function exemption.
• The reaper reads queued and running only. A waiting Run has its own clock.
• Every Run has a wall clock ceiling.
• A durable span is written before the best effort live event.
• A web request handler never runs long agent work.

Minimum contract tests

• A front door command and a trigger command create the same Run shape.
• A start against another organization's definition returns definition_not_found, exactly as a missing one does.
• A start against the caller's own draft returns definition_not_published, whether or not it has a parent.
• A start against a disabled definition returns definition_not_published at the top level, and starts normally with a parent_run_id.
• The Run snapshot is never truncated; an oversized freeze fails the start.
• A start whose input is over 32 KB answers input_too_large and creates no Run row.
• A start against a published skill answers definition_not_published, and the insert is never reached.
• A duplicate StartRunCommand returns the same Run and the same result, and sends no dispatch event.
• A start reads its parent once, and the child carries the parent's root_run_id and organization_id from that read.
• A crash before the Run insert leaves no Run, and the retry starts exactly one.
• Twenty concurrent starts on one key produce one Run, and nineteen duplicate answers.
• A daily trigger starts a new Run every day, and its key does not collide with yesterday's.
• Cancelling a Run that is waiting clears the waiting three, and the shape constraint accepts it.
• A waiting Run past waiting_expires_at plus the grace window is ended wait_abandoned by the wait sweep.
• A waiting Run inside its deadline, or inside the grace window, is left alone.
• A Run re-aimed at a later wait between the sweep's read and its write matches zero rows, and the sweep leaves it.
• A duplicate delivery that resolves a different definition still returns the first Run.
• POST /api/v1/agentic/runs answers 200 with the existing Run for a repeated Idempotency-Key.
• Every legal cell of the transition table moves the Run, and every dash is a no-op.
• succeed() cannot overwrite a Run that was cancelled a moment earlier.
• claim() on a Run waiting on an admission approval clears waiting_on and does not break the shape constraint.
• A re-claim after an hour of running leaves started_at unchanged.
• cancel() stops the subtree below the named Run, and leaves its parent and its siblings alone.
• cancel() on a finished Run changes nothing and raises nothing.
• cancel() with another organization's run ID changes no row.
• A child started against a level the cancel walk has already written is refused, so a deep tree loses nobody.
• A cancel of a tree whose children already succeeded writes no control row for them.
• A child row cannot be written with a root_run_id that disagrees with its parent_run_id.
• advance_segment() clears resumed_from_wait, so segment 4 does not rebuild the brief that segment 3 rebuilt.
• advance_segment() run twice from the same index advances the Run once.
• A child started immediately after its parent is cancelled is refused.
• 500 trigger runs over the day cap are denied before the freeze, and none creates a Run row.
• A start denied for the organization day cap writes a decision with a null run_id.
• A rejected admission approval leaves the Run cancelled and sends nothing.
• An expired admission approval leaves the Run failed with approval_expired.
• An expired action approval leaves the Run succeeded with partial_reason.
• A batch actor starting through the API does not take the interactive lane.
• claim() succeeds on a retry of the same Run.
• A worker restart during an approval resumes and executes the tool once.
• A resume executes the approved call when the model proposes nothing.
• A child Run executes once, and no dispatch event is sent for it.
• A workflow node walk over the same definition and the same memoized outputs produces the same node order and the same step IDs.
• A node declaring continue_on_error still stops on reference_unreadable, on condition_unreadable and on no_parent_span.
• A tool refused for the run cost ceiling stops its node, and the Run stays a partial success when an earlier node produced output.
• A handler that answers the money code itself marks nothing, and the Run fails.
• A child Run that spent a ceiling stops its parent node, and names the clock in a field.
• A start refused for snapshot_unbuildable or policy_unavailable stops its parent node; every other refusal answers start_refused.
• A child ended by admission carries its code on StartRunResult.error_code, and the node reads it. A fault stops the node whatever continue_on_error says.
• A tool handler that answers the money code leaves the child Run error unmarked, and the parent fails.
• A tool handler cannot write partial_reason; the invoker rebuilds meta on the handler path.
• A child that succeeded partially leaves its parent Run succeeded with the same clock, in a sequence and in a parallel.
• A parallel whose branches carry a marked ceiling and a bare platform stop fails the Run, in either branch order.
• A child Run that ended under a platform stop code stops its parent node, and the parent Run reports failed.
• A retried child node meets duplicate, and the tree holds one child.
• A parallel node of two branches, each of two steps, advances one branch while the other parks.
• A failing branch of a parallel node settles its siblings and never raises out of the container.
• A width-one tool node produces one tool span and no run row.
• A wide tool preserves input order when calls finish out of order.
• An empty fan-out list succeeds without a call. An invalid or over-width list fails before a call.
• One business failure in a wide tool does not stop its siblings, whatever continue_on_error says.
• A call ceiling drains the current fan-out batch, keeps results through the first stopping input index and discards later results.
• A later item that finishes before an earlier ceiling still leaves no hole in the ordered output.
• A returned hard platform stop wins when its batch also contains a call or cost ceiling.
• The lowest input position selects between multiple returned hard stops.
• An exception observed with a ceiling uses the exception action and cancels the remaining workers.
• A raised platform fault cancels the current batch and retries the whole read-only step.
• A read approval cancels the batch and ends with non-tolerable read_tool_needs_approval; it spends no retry.
• A wide result above Inngest's 4 MiB step-output limit fails with fanout_output_too_large before the step returns.
• A wide item stopped before ToolInvoker starts writes no tool span; every started call writes one.
• An agent node opens no span of its own, and its child's run span names the node.
• A subworkflow node at the cap refuses the child with the depth error, and reads the frozen child definition from a versioned parent snapshot. Legacy snapshots retain their original live-definition behavior.
• A workflow run cancelled between two nodes stops at the next node and writes no further effect.
• The workflow run result holds one key per node that produced an output, and none for a node that did not run.
• Successful write tools add ordered, unique refs to the workflow result. Reads, failures and rows without an id add none.
• A Run that waits in the queue does not consume its wall clock ceiling.
• A finished Run leaves no agent.sessions row.
• A segment that ends on an approval leaves its session written.
• A segment that fails retryably leaves the earlier session untouched, so the replay starts clean.
• A second segment carries the brief with no rebuild, because the runtime stored what it composed.
• A segment that dies after persisting an approval reuses that row on the replay, and the inbox holds one item.
• A run cancelled mid segment ends cancelled, not failed.
• A segment after approval reads one system block, and it holds the refreshed brief.
• No component above AgnoAgentRuntime reads agno_state.
• A workflow of ten children cannot spend ten cost budgets.
• A cancelled Run leaves no span in running.
• A retried segment's model turns count against max_agent_turns.
• A RunResult above 32 KB is truncated without breaking its shape, and the result still parses.
• bound() replaces every item that is over the limit on its own, not the first one it meets.
• bound() reports truncated for a replacement as well as for a drop.
• bound() on a dict of ten thousand short keys returns inside the limit on the first confirmation, because the envelope is counted.
• A Dropped marker parses, and a payload that really holds reason and bytes is not read as one.
• A RunResult carrying five hundred refs is bounded, and the result column stays inside the limit.
• A completed matching journal entry returns its stored tool result.
• A restarted segment replays completed calls; interrupted and new calls keep the claim and lease rules.
• An expired approval releases the Run, and the Run does not send.
• An approval resolved in the same instant as its timeout still executes.
• Cancelling a parent cancels every descendant Run.
• Cancelling the root of a wide tree issues one status update, not one per child.
• A crash between the status write and the control row leaves the Run cancelled, never running.
• A child started against a parent cancelled one statement ago is refused.
• advance_segment() does not advance a Run that has ended.
• A failed parallel branch does not stop its siblings.
• A parallel branch waiting on an approval does not stop its siblings.
• One of two branch approvals resolving leaves the Run waiting on the other.
• A workflow parent suspended on a child for ten minutes is not reaped.
• A Run that never reaches a worker is reaped, because heartbeat_at was set at insert.
• Two branch approvals in one parallel node resolve in any order.
• A Run is running, not waiting, while it executes the segment after an approval.
• A four minute agent task spans timed segments and is not reaped, and a dead worker is reaped inside 120 seconds.
• A parent Run that invokes a child releases its concurrency slot.
• A tree three deep runs to completion at a concurrency limit of one.
• claim() returns None for a Run cancelled between dispatch and claim.
• A budget stop after work produces succeeded with partial_reason.
• A budget stop before any work produces failed, and the error carries the clock.
• A retried step leaves the Run at running and writes a failed span for the earlier attempt.
• inngest/function.failed moves the Run out of running.
• The reaper fails a Run with a stale heartbeat and no live Inngest run.
• The reaper closes the open spans of the Run it failed.
• A Run waiting three days on an approval survives every reaper pass.
• Two dispatch sends for one Run produce one Inngest function run.
• An invoked child lands in the batch concurrency bucket of its own organization, not in a shared one.
• A retried workflow node meets duplicate and the tree ends with one child, because the start key is the node ID.
• The reaper re-dispatches a queued Run whose event never landed, and fails it on the next pass if it did not move.
• A reaper failure does not trigger the run.failed handler.
• The middle Run of a three deep tree is not reaped while its descendant is working.
• A batch Run cannot consume the interactive concurrency budget.
• An admission approval that is never answered ends the Run at its timeout.
• A Run waiting on an admission approval stays waiting, and the reaper leaves it alone.
• An SSE failure does not corrupt the durable Run or span state.
• A child Run event reaches the root Run channel.
• A Run of five segments cannot spend max_agent_turns five times.
• A segment after approval rebuilds the brief. A completed segment replay does not, and a failed segment retry can.
• A Run queued behind its concurrency lane for ten minutes is not reaped.
• A node that re-runs its start meets duplicate, uses that Run, and the tree ends with one child.