Symphony orchestration
A long-running daemon that turns Linear tickets into isolated, autonomous Codex runs — reviewed by Claude, merged by humans. The high-level workflow, the system architecture, and the engineer playbook. No file paths, just the picture.
At a glance
tl;drSymphony moves the team from supervising coding agents to managing work. You scope a ticket and drop it into a queue state; Symphony runs Codex in a clean per-issue sandbox, Claude reviews the PR, and the work comes back to you as a pull request ready for a merge decision. The practical interface is Linear — state transitions start, gate, and stop every run.
Symphony is a scheduler and tracker reader. It mostly reads Linear; the ticket writes (state transitions, comments, PR links, the workpad) are performed by Codex through the workflow's linear_graphql tool. The human-controlled state gate stays in place — runtime improvements make the same Linear workflow more resilient and observable, they do not remove the human from the loop.
Mental model
five rolesHold five roles in mind. Linear decides what, Symphony decides when, Codex does the work, Claude checks it, GitHub delivers it.
Humans choose what is eligible and review every handoff. State transitions are the API — moving a ticket between Symphony states starts, gates, and stops work.
Polls Linear, claims eligible issues, creates workspaces, dispatches Codex, retries failures with backoff, and stops when the Linear state changes.
Runs in an isolated per-issue workspace. Inspects, edits, validates, updates the Linear workpad, and opens or updates a GitHub PR.
Reviews every PR automatically on open and on each push. Codex fixes every blocking finding before it can hand off to a human.
The merge decision still happens through the PR. A human approves; the merge-stage run lands it with a rebase merge and cleans up.
System architecture
three layersThe system splits into three layers — a control plane humans touch, the Symphony service daemon, and per-issue execution. Each has its own design below; the slate seam node in every diagram is the handoff to the adjacent layer. Click any node to trace its connections.
humans + policyWhat humans touch. Symphony never decides what is eligible — a person moves a ticket into a watched state, and policy lives in-repo so the prompt is versioned with the code.
WORKFLOW.md into the service — runtime state flows back up to observability, and the orchestrator dispatches each issue into an isolated workspace. Commands only ever run inside a per-issue workspace, the core safety invariant. The Linear workflow
interactiveThe states are Symphony-prefixed so normal team work is never picked up by accident. The work ping-pongs between you and the machine — colour marks who moves each state. Only four are watched (polled as active); the rest are handoff gates or terminal. Play the happy path, or click any state to inspect it.
Ready for Symphony
You intentionally queue a small, well-scoped ticket for the VPS. Acceptance criteria and a validation command or narrow area to inspect are what make a ticket a good fit. Symphony claims it on the next poll.
If an integration moves a ticket to a base state such as In Progress or Human Review, the VPS stops treating it as active. Move it back to the matching Symphony-prefixed state to resume. And do not hand Symphony broad epics or vague research — the best tickets have clear acceptance criteria and a narrow validation target.
Lifecycle of a ticket
one run, end to end The happy path for a single issue, from claim to complete. Each normal turn ends with a re-check of the Linear state; while the issue stays active, the worker continues on the same Codex thread up to max_turns.
Poll and claimorchestrator
Symphony polls the Linear project, finds an issue in Ready for Symphony, claims it to prevent duplicate dispatch, and moves it to Symphony In Progress.
Create the workspaceworkspace
A deterministic dir /var/lib/symphony/workspaces/<KEY> is created or reused. The after_create hook clones the AgencyCore parent repo plus submodules from SOURCE_REPO_SSH_URL.
Build prompt and implementcodex
The agent runner builds the prompt from the issue plus the workflow template, launches codex app-server, and streams turns. Codex records branch and starting commit, inspects, then makes focused changes for the ticket only.
Validate, routed by labelcodex
Repo labels such as repo:ac-frontend select which validation section renders, so the smallest relevant gate runs — not every check on every ticket. Per-repo validation freshness avoids re-running an unchanged repo.
Open the PR, link to Linearcodex → github
Codex pushes the branch and opens or updates a clean PR (no Linear key in the title or body), then links it back to Linear. For frontend work, Playwright visual evidence is captured and attached to Linear as a native attachment.
Claude review loopclaude
The Claude review runs automatically on PR open and on every push. Codex classifies findings, fixes every blocking one, reruns affected validation, and pushes — each push triggers a fresh review of the new commit.
Human review gatecodex
Before moving to Symphony Human Review, the gate re-checks the workpad, a completed clean Claude review of the latest commit, validation, and the frontend evidence gate. Any failure keeps the ticket in progress.
Merge and completehuman → codex
A human approves and moves to Symphony Merging. Codex lands the PR with a rebase merge, deletes the branch, resolves any attached Sentry issue, records merge evidence, and moves to Symphony Complete.
Claude review loop
mandatory gate Mandatory after the PR exists and before every transition to Symphony Human Review. The loop is automatic: opening a PR or pushing a commit triggers a fresh Claude review, so the last fix commit is always reviewed without a manual ping.
The review check run on the PR — not an eyes reaction, not a “Claude is working…” comment. Only a completed conclusion means the review finished.
Blocking: correctness, security, regression, validation, acceptance, or meaningful Medium+ maintainability. Low-severity polish, style, wording, and speculative follow-ups are nonblocking unless they expose a real failure.
PR-open and push (synchronize) auto-trigger reviews. Do not post @claude to trigger those. Manual requests are capped at three per PR and reserved for when no automatic run fired.
A running review usually takes 3–5 minutes. Poll the check with cheap one-shot reads up to ~12 minutes before judging. Only a terminal failed / absent state is a blocker.
Validation routing
interactive Validation is progressive — a repo:* label selects which checks render, so the smallest relevant gate runs instead of every check on every ticket. If labels are missing, Codex infers the target repo from the title, description, stack trace, and changed files, then reads that repo's AGENTS.md. Pick a repo.
- Lint
- Type-check
- Unit tests
- Build / dependency checks when relevant
- Playwright visual evidence for UI changes
Timed jobs and nightly QA
the second loopBeyond ticket execution, systemd timers run a second cadence: housekeeping that keeps the VPS healthy, and report-only QA routes that create their own Linear issues for Symphony to pick up. This is the self-improving half — a fast loop does the work, a slow loop watches how the work is going.
Creates a report-only Linear issue. Codex runs the ac-frontend Playwright E2E suite against the local staging stack, attaches evidence, and files grouped bug tickets.
Creates a report-only Linear issue. Codex runs a 60-minute agent-browser crawl of staging against a dedicated QA account, exercises a feature matrix, retests recent bugs, runs mutation probes, and files bug tickets.
Daily read-only Sentry triage that deduplicates tracked clusters and turns new actionable issues into ENG tickets.
Reviews the last 24 hours of commits, runs vulnerability posture checks, and creates security follow-up tickets only for actionable findings.
Reclaims space in Codex sqlite stores with a WAL checkpoint and VACUUM so long-lived sessions do not bloat the disk.
Prunes idle per-issue workspaces older than the configured TTL while keeping reusable caches out of scope.
Removes stale top-level /tmp debris, vacuums the systemd journal, and optionally cleans the apt package cache.
[Symphony run health] or [Staging QA improvement] follow-ups. The system is built to notice its own waste and queue the fix. Observability
dashboard + tokensThe dashboard is private to the VPS. Reach it over an SSH tunnel; the same data is on a JSON state API. A runtime snapshot reports running and retrying sessions, per-session turn counts, aggregate Codex token totals, runtime seconds, and the latest rate-limit payload.
# tunnel the private dashboard to localhost
ssh -L 4000:127.0.0.1:4000 symphony@138.201.190.183
# then open http://127.0.0.1:4000/
curl -fsS http://127.0.0.1:4000/api/v1/state | jq
curl -fsS http://127.0.0.1:4000/api/v1/ENG-123 | jq
curl -X POST http://127.0.0.1:4000/api/v1/refreshthread/tokenUsage/updated, total_token_usage) and tracks deltas against the last reported total to avoid double-counting. Delta-style payloads like last_token_usage are ignored for dashboard totals — they are not cumulative. Operations
runbook The service runs as the unprivileged symphony user. Do not edit /opt/symphony directly — change code locally, push, then git pull --ff-only on the VPS. Treat any VPS or service change as operational work: capture the command, evidence, and a rollback note.
| Item | Value |
|---|---|
| Host | agencycore-symphony-01 · Hetzner fsn1 · cx23 |
| Public IPv4 | 138.201.190.183 |
| Service | symphony.service (systemd) as user symphony |
| Dashboard | 127.0.0.1:4000 — SSH-only, never public |
| Workspaces | /var/lib/symphony/workspaces |
| Workflow | /etc/symphony/WORKFLOW.agencycore.md |
| Secrets | /etc/symphony/symphony.env (root:symphony, 0640) |
| Cost | ~EUR 4.79 / month with backups |
# service health + logs
ssh root@138.201.190.183 'systemctl is-active symphony'
ssh root@138.201.190.183 'journalctl -u symphony -f'
ssh root@138.201.190.183 'systemctl restart symphony' The service runs with CODEX_HOME=/home/symphony/.codex. On token_expired, refresh_token_reused, or a websocket 401, stop the service, re-run codex login --device-auth as the symphony user, smoke test, then restart. Do not switch to API-key auth unless the operator explicitly chooses that fallback.
Scaling and rollout
stay conservativeThe highest operational risk is handing agents too many ambiguous tickets at once. Broaden deliberately.
- Start at
max_concurrent_agents: 1. One test ticket at a time. - After several clean issues, raise the cap to 2; add per-state limits if merge or rework should be stricter than implementation.
- Keep the Symphony-prefixed Linear states until the team trusts the workflow.
- Add SSH workers only when the single VPS is CPU, memory, or disk constrained.
- Broaden eligible states only after the issue template and review culture are reliable.