High-level design

The Symphony daemon end to end — the standing agent workforce and its label-routed workflows, then the runtime that polls, dispatches, runs and writes back.

1 min read Updated Aug 12, 2026

High-level design

One canvas for the whole service: the workforce that files the work, and the runtime that turns it into merged code.

Symphony — high-level system design
A standing workforce of autonomous agents inspects the product on a cadence and files ordinary tickets; one Elixir/OTP daemon schedules them, and a coding agent does the writing.
Symphony — high-level system designA standing workforce of autonomous agents inspects the product on a cadence and files ordinary tickets; one Elixir/OTP daemon schedules them, and a coding agent does the writing.
1 · THE WORKFORCE
who files work, and how the label on a ticket turns it into a workflow
1 · THE WORKFORCEwho files work, and how the label on a ticket turns it into a workflow
Security review
report:security-agent
daily commit and system vulnerability review
Security reviewreport:security-agentdaily commit and systemvulnerability review
Performance review
report:performance-agent
daily production performance review
Performance reviewreport:performance-agentdaily production performancereview
Sentry triage
report:sentry-triage
daily triage and health summary
Sentry triagereport:sentry-triagedaily triage and health summary
Log analyzer
report:symphony-log-analyzer
daily VPS log and run-efficiency analysis
Log analyzerreport:symphony-log-analyzerdaily VPS log and run-efficiencyanalysis
API contract audit
report:api-contract-audit
weekly endpoint contract and REST audit
API contract auditreport:api-contract-auditweekly endpoint contract andREST audit
Staging regression QA
qa:staging-regression
nightly staging crawl with agent-browser
Staging regression QAqa:staging-regressionnightly staging crawl withagent-browser
Local staging E2E
qa:local-staging-e2e
nightly local Playwright run
Local staging E2Eqa:local-staging-e2enightly local Playwright run
Dogfood
qa:dogfood
nightly exploratory QA with the dogfood skill
Dogfoodqa:dogfoodnightly exploratory QA with thedogfood skill
Humans
any label
the router does not care who filed the ticket
Humansany labelthe router does not care who filedthe ticket
Every ticket carries a routing label
it is the only thing that selects a workflow
Every ticket carries a routing labelit is the only thing that selects a workflow
routes.md — always rendered
the label picks which partials load, so unrelated routes stay out of the prompt
routes.md — always renderedthe label picks which partials load, so unrelated routes stay out of the prompt
The route partial becomes the agent's instructions
what to inspect, what evidence to attach, what not to touch
The route partial becomes the agent's instructionswhat to inspect, what evidence to attach, what not to touch
All eight standing routes are report-only
findings are clustered and deduped, then filed as focused follow-up tickets
All eight standing routes are report-onlyfindings are clustered and deduped, then filed as focused follow-up tickets
2 · QUEUE
rented, not run
2 · QUEUErented, not run
3 · COORDINATE
sole writer of scheduling state
3 · COORDINATEsole writer of scheduling state
4 · EXECUTE
one run attempt per issue
4 · EXECUTEone run attempt per issue
5 · OPERATE
humans and department agents
5 · OPERATEhumans and department agents
Linear
one queue, whoever filed the ticket
Linearone queue, whoever filed the ticket
GitHub
PRs and merge state
GitHubPRs and merge state
Sentry
linked issues and cleanup
Sentrylinked issues and cleanup
Poll and reconcile
every 30s: refresh running, stop the ineligible
Poll and reconcileevery 30s: refresh running, stop the ineligible
Dispatch
claim up to the concurrency cap, retry with backoff
Dispatchclaim up to the concurrency cap, retry with backoff
Terminal guards
merged PR and Sentry cleanup, or it is not done
Terminal guardsmerged PR and Sentry cleanup, or it is not done
Per-issue workspace
one directory per issue, all file writes confined here
Per-issue workspaceone directory per issue, all file writes confined here
Coding agent session
Codex or Claude Code, one thread per issue
Coding agent sessionCodex or Claude Code, one thread per issue
Agent tools
git, gh, tracker CLI, tests, browser
Agent toolsgit, gh, tracker CLI, tests, browser
Web dashboard
Vue SPA over /api/v1
Web dashboardVue SPA over /api/v1
Operator
review and approvals over Slack
Operatorreview and approvals over Slack
DevOps and Accountant agents
in-process OTP controllers, dry-run by default
DevOps and Accountant agentsin-process OTP controllers, dry-run by default
CROSS-CUTTING LAYERS — each one touches every stage above, which is why no arrow points at it from the loop.
CROSS-CUTTING LAYERS — each one touches every stage above, which is why no arrow points at it from the loop.
POLICY
the rules a run obeys, versioned with the code they govern
POLICYthe rules a run obeys, versioned with the code they govern
OBSERVABILITY
read-only — orchestrator correctness never depends on it
OBSERVABILITYread-only — orchestrator correctness never depends on it
HOST HYGIENE
janitorial, not delegated — these timers file no ticket
HOST HYGIENEjanitorial, not delegated — these timers file no ticket
WORKFLOW.md — front matter for runtime settings, body for the prompt
WORKFLOW.md — front matter for runtime settings, body for the prompt
Workflow Store — watches the file, splits settings from the prompt body
Workflow Store — watches the file, splits settings from the prompt body
Config Layer — typed getters, defaults, $ENV, preflight validation
Config Layer — typed getters, defaults, $ENV, preflight validation
Orchestrator and Prompt Builder
Orchestrator and Prompt Builder
Extension controllers
keys read once at boot
Extension controllerskeys read once at boot
Turn events — tokens, rate limits, exit status
Turn events — tokens, rate limits, exit status
Structured logs
secrets redacted, bounded retention
Structured logssecrets redacted, bounded retention
Session metrics
tokens and runtime per issue
Session metricstokens and runtime per issue
journald
the systemd unit log
journaldthe systemd unit log
HTTP /api/v1
JSON snapshots, loopback-bound
HTTP /api/v1JSON snapshots, loopback-bound
Web dashboard
Web dashboard
Codex vacuum
compacts the local sqlite stores
Codex vacuumcompacts the local sqlite stores
Workspace prune
drops per-issue dirs idle over 7d
Workspace prunedrops per-issue dirs idle over 7d
Disk reclaim
stale /tmp, journal, apt cache
Disk reclaimstale /tmp, journal, apt cache
Disk
workspaces, caches, logs
Diskworkspaces, caches, logs
Shared failure path — one templated OnFailure= unit posts the failed unit name to Slack. Every timer on the host carries it, agent-filing or janitorial.
Shared failure path — one templated OnFailure= unit posts the failed unit name to Slack. Every timer on thehost carries it, agent-filing or janitorial.
LEGEND — a dashed border means rented, not run
LEGEND — a dashed border means rented, not run
the queue everything converges on
the queue everything converges on
an agent — costs an LLM call, and money
an agent — costs an LLM call, and money
deterministic work we run ourselves
deterministic work we run ourselves
plumbing — carries no LLM cost
plumbing — carries no LLM cost
a layer, a stage, or a cross-cutting concern
a layer, a stage, or a cross-cutting concern
files an ordinary ticket — its own, and every follow-up
files an ordinary ticket — its own, and every follow-up
polls candidates
polls candidates
dispatches a claim
dispatches a claim
streams live state
streams live state
the agent writes back — ticket state, comments, PR links
the agent writes back — ticket state, comments, PR links
hot-reloads in about 1s
hot-reloads in about 1s
needs a service restart
needs a service restart
Text is not SVG - cannot display
Symphony — high-level system design
A standing workforce of autonomous agents inspects the product on a cadence and files ordinary tickets; one Elixir/OTP daemon schedules them, and a coding agent does the writing.
Symphony — high-level system designA standing workforce of autonomous agents inspects the product on a cadence and files ordinary tickets; one Elixir/OTP daemon schedules them, and a coding agent does the writing.
1 · THE WORKFORCE
who files work, and how the label on a ticket turns it into a workflow
1 · THE WORKFORCEwho files work, and how the label on a ticket turns it into a workflow
Security review
report:security-agent
daily commit and system vulnerability review
Security reviewreport:security-agentdaily commit and systemvulnerability review
Performance review
report:performance-agent
daily production performance review
Performance reviewreport:performance-agentdaily production performancereview
Sentry triage
report:sentry-triage
daily triage and health summary
Sentry triagereport:sentry-triagedaily triage and health summary
Log analyzer
report:symphony-log-analyzer
daily VPS log and run-efficiency analysis
Log analyzerreport:symphony-log-analyzerdaily VPS log and run-efficiencyanalysis
API contract audit
report:api-contract-audit
weekly endpoint contract and REST audit
API contract auditreport:api-contract-auditweekly endpoint contract andREST audit
Staging regression QA
qa:staging-regression
nightly staging crawl with agent-browser
Staging regression QAqa:staging-regressionnightly staging crawl withagent-browser
Local staging E2E
qa:local-staging-e2e
nightly local Playwright run
Local staging E2Eqa:local-staging-e2enightly local Playwright run
Dogfood
qa:dogfood
nightly exploratory QA with the dogfood skill
Dogfoodqa:dogfoodnightly exploratory QA with thedogfood skill
Humans
any label
the router does not care who filed the ticket
Humansany labelthe router does not care who filedthe ticket
Every ticket carries a routing label
it is the only thing that selects a workflow
Every ticket carries a routing labelit is the only thing that selects a workflow
routes.md — always rendered
the label picks which partials load, so unrelated routes stay out of the prompt
routes.md — always renderedthe label picks which partials load, so unrelated routes stay out of the prompt
The route partial becomes the agent's instructions
what to inspect, what evidence to attach, what not to touch
The route partial becomes the agent's instructionswhat to inspect, what evidence to attach, what not to touch
All eight standing routes are report-only
findings are clustered and deduped, then filed as focused follow-up tickets
All eight standing routes are report-onlyfindings are clustered and deduped, then filed as focused follow-up tickets
2 · QUEUE
rented, not run
2 · QUEUErented, not run
3 · COORDINATE
sole writer of scheduling state
3 · COORDINATEsole writer of scheduling state
4 · EXECUTE
one run attempt per issue
4 · EXECUTEone run attempt per issue
5 · OPERATE
humans and department agents
5 · OPERATEhumans and department agents
Linear
one queue, whoever filed the ticket
Linearone queue, whoever filed the ticket
GitHub
PRs and merge state
GitHubPRs and merge state
Sentry
linked issues and cleanup
Sentrylinked issues and cleanup
Poll and reconcile
every 30s: refresh running, stop the ineligible
Poll and reconcileevery 30s: refresh running, stop the ineligible
Dispatch
claim up to the concurrency cap, retry with backoff
Dispatchclaim up to the concurrency cap, retry with backoff
Terminal guards
merged PR and Sentry cleanup, or it is not done
Terminal guardsmerged PR and Sentry cleanup, or it is not done
Per-issue workspace
one directory per issue, all file writes confined here
Per-issue workspaceone directory per issue, all file writes confined here
Coding agent session
Codex or Claude Code, one thread per issue
Coding agent sessionCodex or Claude Code, one thread per issue
Agent tools
git, gh, tracker CLI, tests, browser
Agent toolsgit, gh, tracker CLI, tests, browser
Web dashboard
Vue SPA over /api/v1
Web dashboardVue SPA over /api/v1
Operator
review and approvals over Slack
Operatorreview and approvals over Slack
DevOps and Accountant agents
in-process OTP controllers, dry-run by default
DevOps and Accountant agentsin-process OTP controllers, dry-run by default
CROSS-CUTTING LAYERS — each one touches every stage above, which is why no arrow points at it from the loop.
CROSS-CUTTING LAYERS — each one touches every stage above, which is why no arrow points at it from the loop.
POLICY
the rules a run obeys, versioned with the code they govern
POLICYthe rules a run obeys, versioned with the code they govern
OBSERVABILITY
read-only — orchestrator correctness never depends on it
OBSERVABILITYread-only — orchestrator correctness never depends on it
HOST HYGIENE
janitorial, not delegated — these timers file no ticket
HOST HYGIENEjanitorial, not delegated — these timers file no ticket
WORKFLOW.md — front matter for runtime settings, body for the prompt
WORKFLOW.md — front matter for runtime settings, body for the prompt
Workflow Store — watches the file, splits settings from the prompt body
Workflow Store — watches the file, splits settings from the prompt body
Config Layer — typed getters, defaults, $ENV, preflight validation
Config Layer — typed getters, defaults, $ENV, preflight validation
Orchestrator and Prompt Builder
Orchestrator and Prompt Builder
Extension controllers
keys read once at boot
Extension controllerskeys read once at boot
Turn events — tokens, rate limits, exit status
Turn events — tokens, rate limits, exit status
Structured logs
secrets redacted, bounded retention
Structured logssecrets redacted, bounded retention
Session metrics
tokens and runtime per issue
Session metricstokens and runtime per issue
journald
the systemd unit log
journaldthe systemd unit log
HTTP /api/v1
JSON snapshots, loopback-bound
HTTP /api/v1JSON snapshots, loopback-bound
Web dashboard
Web dashboard
Codex vacuum
compacts the local sqlite stores
Codex vacuumcompacts the local sqlite stores
Workspace prune
drops per-issue dirs idle over 7d
Workspace prunedrops per-issue dirs idle over 7d
Disk reclaim
stale /tmp, journal, apt cache
Disk reclaimstale /tmp, journal, apt cache
Disk
workspaces, caches, logs
Diskworkspaces, caches, logs
Shared failure path — one templated OnFailure= unit posts the failed unit name to Slack. Every timer on the host carries it, agent-filing or janitorial.
Shared failure path — one templated OnFailure= unit posts the failed unit name to Slack. Every timer on thehost carries it, agent-filing or janitorial.
LEGEND — a dashed border means rented, not run
LEGEND — a dashed border means rented, not run
the queue everything converges on
the queue everything converges on
an agent — costs an LLM call, and money
an agent — costs an LLM call, and money
deterministic work we run ourselves
deterministic work we run ourselves
plumbing — carries no LLM cost
plumbing — carries no LLM cost
a layer, a stage, or a cross-cutting concern
a layer, a stage, or a cross-cutting concern
files an ordinary ticket — its own, and every follow-up
files an ordinary ticket — its own, and every follow-up
polls candidates
polls candidates
dispatches a claim
dispatches a claim
streams live state
streams live state
the agent writes back — ticket state, comments, PR links
the agent writes back — ticket state, comments, PR links
hot-reloads in about 1s
hot-reloads in about 1s
needs a service restart
needs a service restart
Text is not SVG - cannot display
Symphony end to end. The workforce layer sits above the runtime: eight standing agents inspect the product on a cadence and file ordinary tickets, and the label on each ticket selects the workflow that runs it.

The workforce

The top layer is the part that actually automates things. Eight standing agents run on their own cadences, and the humans on the team sit in the same row because they enter the system the same way. The agents get no privileged path in — same queue, same labels, same workflow as anything a person raises.

AgentRouting labelWhat it inspects
Security reviewreport:security-agentdaily commit and system vulnerability review
Performance reviewreport:performance-agentdaily production performance review
Sentry triagereport:sentry-triagedaily triage and health summary
Log analyzerreport:symphony-log-analyzerdaily VPS log and run-efficiency analysis
API contract auditreport:api-contract-auditweekly endpoint contract and REST audit
Staging regression QAqa:staging-regressionnightly staging crawl with agent-browser
Local staging E2Eqa:local-staging-e2enightly local Playwright run
Dogfoodqa:dogfoodnightly exploratory QA with the repo-local dogfood skill

The label is the whole routing mechanism. routes.md is always rendered into the prompt and conditionally renders one route partial per matching label, so a run loads the instructions for its own route and nothing else. That partial is what tells the agent what to inspect, what evidence to attach, and what it must not touch. The manifest of every route lives in elixir/workflow_context/agencycore/README.md.

For each agent's cadence and its full brief, see the timed jobs pages; the agents section covers the two department controllers.

The part that makes this compound: all eight standing routes are report-only. They never change code. They investigate, cluster and dedupe what they find, and file focused follow-up tickets — which land back in the same queue as ordinary implementation work and get built by the normal workflow. Continuous inspection turns into routed, actionable, de-duplicated tickets without anyone triaging them by hand.

The runtime

  1. Queue — Linear holds every ticket regardless of who filed it. GitHub and Sentry sit beside it because the terminal guards read durable evidence from them. All three are rented, not run, which is what the dashed borders mean.
  2. Coordinate — the orchestrator polls for candidate tickets, reconciles what is already running, and claims new work up to the concurrency cap. It is the only writer of scheduling state.
  3. Execute — one run attempt per issue: a persistent workspace directory, a coding agent session, and the agent's own tools. The agent writes back to Linear itself, so ticket state, comments and PR links never pass back through the orchestrator.
  4. Operate — the dashboard, the human reviewing and approving over Slack, and the DevOps and Accountant controllers.

Purple marks an agent, and therefore a cost: the workforce, the coding agent session, and the two department controllers are the only things on the canvas that spend an LLM call.

The whole service is a single Elixir/OTP daemon. Optional extensions are in-process OTP controllers, and they are dry-run by default.

The cross-cutting layers

The three panels along the bottom touch every stage of the loop, which is why no arrow points at them from it. Drawing one arrow per stage would add fifteen lines and say nothing.

Policy is the target repo's WORKFLOW.md — front matter for runtime settings, body for the prompt. The store watches the file, and the config layer resolves types, defaults and $ENV before anything downstream sees a value. The reload behaviour is not uniform, and the difference bites: the orchestrator and the prompt builder pick up most keys about a second after a save, but the extension controllers read theirs once at boot, so flipping a devops.* or accountant.* key does nothing until the service is restarted.

Observability is strictly read-only. Turn events fan out into structured logs, redacted and retention-bounded, and into per-session metrics — token accounting, rate-limit state and runtime for one issue. Those surface through journald and the loopback-bound /api/v1 snapshot the dashboard reads. Nothing here feeds back into scheduling.

Host hygiene is the three timers that file no ticket. They act on the box directly, which is what separates them from the eight agents in the workforce layer: janitorial work, not delegated work. Their failure path is shared by every timer on the host, agent-filing or janitorial — one templated OnFailure= unit posts the failed unit's name to Slack, so a job that dies silently still surfaces.

For the workflow this design serves, and the day-to-day engineer playbook, see Symphony orchestration.