Symphony · Orchestration

Symphony orchestration

A long-running daemon that turns Linear tickets into isolated, autonomous Codex runs — reviewed by Claude, merged by humans. The high-level workflow, the system architecture, and the engineer playbook. No file paths, just the picture.

Control plane Linear statesWorker Codex app-serverReviewer Claude on every PRHost Hetzner VPS · ~EUR 4.79/mo
Linear ticketSymphony claims + dispatches·Codex in a sandboxPR you merge
00

At a glance

Symphony moves the team from supervising coding agents to managing work. You scope a ticket and drop it into a queue state; Symphony runs Codex in a clean per-issue sandbox, Claude reviews the PR, and the work comes back to you as a pull request ready for a merge decision. The practical interface is Linear — state transitions start, gate, and stop every run.

5
Roles
Linear · Symphony · Codex · Claude · GitHub
1
Sandbox per ticket
isolated, deterministic dir
Review on every push
Claude auto-reviews each commit
3
Nightly QA routes
staging crawl · E2E · Sentry
KEYLinear is the only control surface

Symphony is a scheduler and tracker reader. It mostly reads Linear; the ticket writes (state transitions, comments, PR links, the workpad) are performed by Codex through the workflow's linear_graphql tool. The human-controlled state gate stays in place — runtime improvements make the same Linear workflow more resilient and observable, they do not remove the human from the loop.

01

Mental model

Hold five roles in mind. Linear decides what, Symphony decides when, Codex does the work, Claude checks it, GitHub delivers it.

Linear · control plane

Humans choose what is eligible and review every handoff. State transitions are the API — moving a ticket between Symphony states starts, gates, and stops work.

Symphony · scheduler

Polls Linear, claims eligible issues, creates workspaces, dispatches Codex, retries failures with backoff, and stops when the Linear state changes.

Codex · worker

Runs in an isolated per-issue workspace. Inspects, edits, validates, updates the Linear workpad, and opens or updates a GitHub PR.

Claude · reviewer

Reviews every PR automatically on open and on each push. Codex fixes every blocking finding before it can hand off to a human.

GitHub · delivery

The merge decision still happens through the PR. A human approves; the merge-stage run lands it with a rebase merge and cleans up.

02

System architecture

The system splits into three layers — a control plane humans touch, the Symphony service daemon, and per-issue execution. Each has its own design below; the slate seam node in every diagram is the handoff to the adjacent layer. Click any node to trace its connections.

humans + policyWhat humans touch. Symphony never decides what is eligible — a person moves a ticket into a watched state, and policy lives in-repo so the prompt is versioned with the code.

Linear boardcandidate issues + stateOperatorqueue · review · approveWORKFLOW.mdpolicy · config · promptworkflow_context/rendered prompt partials↓ Symphony servicepolls Linear · loads policy
Workflow config flows down — Linear and WORKFLOW.md into the service — runtime state flows back up to observability, and the orchestrator dispatches each issue into an isolated workspace. Commands only ever run inside a per-issue workspace, the core safety invariant.
03

The Linear workflow

The states are Symphony-prefixed so normal team work is never picked up by accident. The work ping-pongs between you and the machine — colour marks who moves each state. Only four are watched (polled as active); the rest are handoff gates or terminal. Play the happy path, or click any state to inspect it.

You moveSymphony / Codex movewatched · polled
1 / 6
You movewatched · active

Ready for Symphony

You intentionally queue a small, well-scoped ticket for the VPS. Acceptance criteria and a validation command or narrow area to inspect are what make a ticket a good fit. Symphony claims it on the next poll.

!Keep it in a Symphony state

If an integration moves a ticket to a base state such as In Progress or Human Review, the VPS stops treating it as active. Move it back to the matching Symphony-prefixed state to resume. And do not hand Symphony broad epics or vague research — the best tickets have clear acceptance criteria and a narrow validation target.

04

Lifecycle of a ticket

The happy path for a single issue, from claim to complete. Each normal turn ends with a re-check of the Linear state; while the issue stays active, the worker continues on the same Codex thread up to max_turns.

1

Poll and claimorchestrator

Symphony polls the Linear project, finds an issue in Ready for Symphony, claims it to prevent duplicate dispatch, and moves it to Symphony In Progress.

2

Create the workspaceworkspace

A deterministic dir /var/lib/symphony/workspaces/<KEY> is created or reused. The after_create hook clones the AgencyCore parent repo plus submodules from SOURCE_REPO_SSH_URL.

3

Build prompt and implementcodex

The agent runner builds the prompt from the issue plus the workflow template, launches codex app-server, and streams turns. Codex records branch and starting commit, inspects, then makes focused changes for the ticket only.

4

Validate, routed by labelcodex

Repo labels such as repo:ac-frontend select which validation section renders, so the smallest relevant gate runs — not every check on every ticket. Per-repo validation freshness avoids re-running an unchanged repo.

5

Open the PR, link to Linearcodex → github

Codex pushes the branch and opens or updates a clean PR (no Linear key in the title or body), then links it back to Linear. For frontend work, Playwright visual evidence is captured and attached to Linear as a native attachment.

6

Claude review loopclaude

The Claude review runs automatically on PR open and on every push. Codex classifies findings, fixes every blocking one, reruns affected validation, and pushes — each push triggers a fresh review of the new commit.

7

Human review gatecodex

Before moving to Symphony Human Review, the gate re-checks the workpad, a completed clean Claude review of the latest commit, validation, and the frontend evidence gate. Any failure keeps the ticket in progress.

8

Merge and completehuman → codex

A human approves and moves to Symphony Merging. Codex lands the PR with a rebase merge, deletes the branch, resolves any attached Sentry issue, records merge evidence, and moves to Symphony Complete.

05

Claude review loop

Mandatory after the PR exists and before every transition to Symphony Human Review. The loop is automatic: opening a PR or pushing a commit triggers a fresh Claude review, so the last fix commit is always reviewed without a manual ping.

PR opened / pushedreview check: queued → in_progress → completedclassify findingsfix blocking + push↻ fresh review
Authoritative status

The review check run on the PR — not an eyes reaction, not a “Claude is working…” comment. Only a completed conclusion means the review finished.

Blocking vs not

Blocking: correctness, security, regression, validation, acceptance, or meaningful Medium+ maintainability. Low-severity polish, style, wording, and speculative follow-ups are nonblocking unless they expose a real failure.

No redundant pings

PR-open and push (synchronize) auto-trigger reviews. Do not post @claude to trigger those. Manual requests are capped at three per PR and reserved for when no automatic run fired.

In progress ≠ stuck

A running review usually takes 3–5 minutes. Poll the check with cheap one-shot reads up to ~12 minutes before judging. Only a terminal failed / absent state is a blocker.

06

Validation routing

Validation is progressive — a repo:* label selects which checks render, so the smallest relevant gate runs instead of every check on every ticket. If labels are missing, Codex infers the target repo from the title, description, stack trace, and changed files, then reads that repo's AGENTS.md. Pick a repo.

repo:ac-frontend
  • Lint
  • Type-check
  • Unit tests
  • Build / dependency checks when relevant
  • Playwright visual evidence for UI changes
07

Timed jobs and nightly QA

Beyond ticket execution, systemd timers run a second cadence: housekeeping that keeps the VPS healthy, and report-only QA routes that create their own Linear issues for Symphony to pick up. This is the self-improving half — a fast loop does the work, a slow loop watches how the work is going.

Each QA run also performs an agent-run health retrospective on Symphony itself — turn timeouts, redundant re-validation on an unchanged HEAD, handoff-guard false bounces, wasted budget — and files [Symphony run health] or [Staging QA improvement] follow-ups. The system is built to notice its own waste and queue the fix.
08

Observability

The dashboard is private to the VPS. Reach it over an SSH tunnel; the same data is on a JSON state API. A runtime snapshot reports running and retrying sessions, per-session turn counts, aggregate Codex token totals, runtime seconds, and the latest rate-limit payload.

running
Active sessions
with turn_count each
retrying
Retry queue
bounded backoff
tokens
codex_totals
input · output · total
runtime
seconds_running
live aggregate
# tunnel the private dashboard to localhost
ssh -L 4000:127.0.0.1:4000 symphony@138.201.190.183
# then open http://127.0.0.1:4000/

curl -fsS http://127.0.0.1:4000/api/v1/state | jq
curl -fsS http://127.0.0.1:4000/api/v1/ENG-123 | jq
curl -X POST http://127.0.0.1:4000/api/v1/refresh
Token accounting prefers absolute thread totals (thread/tokenUsage/updated, total_token_usage) and tracks deltas against the last reported total to avoid double-counting. Delta-style payloads like last_token_usage are ignored for dashboard totals — they are not cumulative.
09

Operations

The service runs as the unprivileged symphony user. Do not edit /opt/symphony directly — change code locally, push, then git pull --ff-only on the VPS. Treat any VPS or service change as operational work: capture the command, evidence, and a rollback note.

ItemValue
Hostagencycore-symphony-01 · Hetzner fsn1 · cx23
Public IPv4138.201.190.183
Servicesymphony.service (systemd) as user symphony
Dashboard127.0.0.1:4000 — SSH-only, never public
Workspaces/var/lib/symphony/workspaces
Workflow/etc/symphony/WORKFLOW.agencycore.md
Secrets/etc/symphony/symphony.env (root:symphony, 0640)
Cost~EUR 4.79 / month with backups
# service health + logs
ssh root@138.201.190.183 'systemctl is-active symphony'
ssh root@138.201.190.183 'journalctl -u symphony -f'
ssh root@138.201.190.183 'systemctl restart symphony'
!Codex auth is device-code, not API key

The service runs with CODEX_HOME=/home/symphony/.codex. On token_expired, refresh_token_reused, or a websocket 401, stop the service, re-run codex login --device-auth as the symphony user, smoke test, then restart. Do not switch to API-key auth unless the operator explicitly chooses that fallback.

10

Scaling and rollout

The highest operational risk is handing agents too many ambiguous tickets at once. Broaden deliberately.

  • Start at max_concurrent_agents: 1. One test ticket at a time.
  • After several clean issues, raise the cap to 2; add per-state limits if merge or rework should be stricter than implementation.
  • Keep the Symphony-prefixed Linear states until the team trusts the workflow.
  • Add SSH workers only when the single VPS is CPU, memory, or disk constrained.
  • Broaden eligible states only after the issue template and review culture are reliable.
11

Sources

AgencyCore · Symphony · OrchestrationElixir reference impl · 2026-06-09