Orchestration

L4. The layer that executes.

A decision that is not durable is a guess. This layer takes a plan and runs it, survives a crash, retries a failure, waits for a person, and records everything it did.

Components

The five nouns

Workflow          what should happen
   │
   ├── Run #001   what did happen, on Monday
   ├── Run #002   what did happen, on Tuesday
   └── Run #003   what is happening now

A workflow is a definition. A run is an instance. Confusing the two produces a system where you cannot answer "what did it do last week".

What a run holds

run
  workflow or agent, and the version pinned at start
  trigger           what started it, and who the principal is
  input             the brief
  state             one of the states below
  steps             child runs, each with its own state
  output            the result
  error             the failure, if any
  retries           the count, per step
  cost              tokens and money
  artifacts         what it produced
  events            the append-only log

The run row is the single source of truth for execution. There is one run table for agent runs and for workflow runs. A workflow run is the parent of its step runs.

Run states

queued → admitted → running ──→ succeeded
            │          │  │
            │          │  └──→ awaiting_approval ──→ running
            │          │
            │          ├──→ failed        (retries exhausted)
            │          └──→ cancelled     (a person intervened)
            │
            └──→ denied  (policy refused at admission; nothing ran)

denied and cancelled are separate states on purpose. One means the system refused. The other means a person stopped it. A single "stopped" state loses the difference, and that difference is the first question anyone asks.

Triggers

Five kinds start a run. The run does not know which one it was, beyond one field.

Policy admits every run, whatever the trigger. A scheduled run for an account over budget is stopped before it dispatches.

A webhook trigger always has a schedule beside it. External delivery is at-least-once and a message can be missed, so a slower schedule re-reads the same window. The steps must be idempotent, which makes the repeat harmless. A webhook on its own is data that is quietly incomplete.

The ingest workflow

Any pull from an external system uses one shape, and no model runs inside it.

fetch  →  normalize  →  upsert  →  emit

fetch calls a vendor tool. normalize discards the payload and keeps only what we model. upsert writes through an internal tool, keyed so a re-run overwrites rather than duplicates. emit writes the run event that a downstream trigger may match. The worked example is Metrics and connectors.

Rules

• A step is the unit of durability. A crash resumes at the last completed step. It does not restart the run.
• A step is idempotent, or it carries an idempotency key. A retry must not send a second email.
• A run pins its definition version at start. An edit to a workflow never changes a run in flight.
• A wait is a state, not a loop. A run that waits for an approval or for an external event consumes nothing.
• Every run has a deadline. The reaper closes a run that stopped reporting, and it records the reason.
• Events are append-only. A correction is a new event. Nothing in the log is edited.
• The engine sequences. It does not reason. A branch in a workflow is a condition over data, never a model call. A decision that needs a model is a step that runs an agent.

Events

One stream leaves this layer, and three consumers read it.

run event
   ├── Mission Control activity feed   L6
   ├── trigger matcher                 L4, an event can start another run
   └── the accrual checkpoint          policy plane, cost adds up here

An event carries the run, the step, the kind, the severity and the payload. It never carries a secret.

Data model

Risks

Open questions

1. Does a workflow support a parallel branch in version 1, or only a sequence? Parallel is where most of the complexity lives.
2. How long is the default run retention, and what is kept after it?
3. Can a person edit the input of a waiting run and resume it, or must they cancel and restart?
4. Does a step failure fail the whole run by default, or does the workflow declare the policy per step?

Diagram source. ac-docs/diagrams/mission-control-layers.drawio, page mission-control-orchestration. Edit that page, then run ./scripts/export-diagrams.sh mission-control-layers. Update this page and the diagram together.