Symphony / Sentry triage timed job

Daily Sentry triage

A read-only Sentry route that clusters unresolved errors, suppresses known tracked noise, opens only new actionable ENG bugs, and posts a compact daily summary.

Cadence 03:00 Europe/LondonMode read-only SentryScope ac-python-api + ac-frontendOutput new bugs + Slack
systemd timer->Sentry triage issue->cluster + dedupe->new ENG bugs
00

Operating model

timed job

The timer creates one report issue each morning. Symphony fetches unresolved Sentry issues with read-only credentials, groups effective clusters, checks Linear idempotency labels, and creates follow-up bugs only for genuinely new actionable clusters.

24h
Comparison windows
current and previous UTC windows
2
Projects
API and frontend
0
Sentry mutations
never resolve or annotate
1
Slack summary
compact and deduplicated
KEYRead-only means read-only

The route may create Linear issues and labels, but it never resolves, assigns, annotates, or changes Sentry state. Sentry mutations belong to merge-stage issue resolution, not daily triage.

01

System shape

click the nodes

The route intentionally avoids helper-path fragility. It can use an installed helper when healthy, but read-only CLI or API fallbacks are first-class paths.

read-only Sentry boundarysystemd timer03:00 dailyIssue seedercreate_sentry_triageLinear issuereport:sentry-triageSymphony routetriage policySentry APIread-only tokenCluster triagededupe + windowsLinear dedupesentry-id labelsHealth signalstransactionsReport outputsbugs + Slack
02

Cadence design

compare modes

Each timed job has a small set of modes that decide whether the run is healthy, degraded, or ready to create follow-up work. Use the controls to compare the operating contract.

Preflight

Prove access without mutating Sentry

The run validates Slack and Sentry access, forces env-token auth, and records helper fallback decisions early.

  • SENTRY_AUTH_TOKEN present
  • SENTRY_FORCE_ENV_TOKEN=1
  • Slack destination available or fallback planned
  • helper path not pinned
Triage confidence threshold82%

The run can close when Sentry, Linear dedupe, Slack or fallback, and run-health rows are complete.

03

Execution tracks

interactive

The tracks make each run inspectable. They separate inputs, outputs, and failure modes so reports do not collapse into vague status updates.

Unresolved issues

Error clusters

Group unresolved Sentry issues into effective clusters and decide whether each is new, tracked, canceled, duplicate, or noise.

Inputs
  • org issues endpoint
  • project filters
  • issue metadata
Outputs
  • effective clusters
  • new bug candidates
  • tracked cluster count
04

Triage risk map

priority map

These are the common ways a daily error triage can create noise or miss important work.

Read-only Sentry authinteractive login or mutable tokens can break the route contract
96
Linear idempotency labelsmissing label checks create duplicate bugs
92
Helper fallback behaviorpinned cache paths and hangs can consume the run
84
Current vs previous windowsmoving windows make trends inconsistent
80
Slack summary compactnesslong raw link dumps hide actionable items
70
Canceled and duplicate clustersknown noise should not re-enter the queue
62
If Sentry or Slack is unavailable, the route should finish degraded with the same summary in the report comment and stdout. It should not silently report a clean day.
05

Output rubric

ticket rules

The run should create follow-up work only when the output is concrete. Use the rubric to keep reports, bug tickets, and operational follow-ups separated.

New cluster

Use when the unresolved cluster is not already tracked and has a concrete user-facing or operational impact.

  • new backend exception
  • frontend crash with users
  • auth callback failure
06

Run contract

evidence rows

The daily Sentry workpad must separate raw counts, effective clusters, new tickets, tracked tickets, warning signals, and fallback paths.

Issue seed

Timed issue

Title format is Daily Sentry triage - YYYY-MM-DD with report:sentry-triage, sentry, and observability labels.

Allowed writes

Linear-only writes

Sentry stays read-only. Allowed writes are new Linear labels or bugs, one report comment, one Slack summary, and Symphony state changes.

Current window UTC
Previous window UTC
Slack preflight
Sentry preflight
Raw unresolved issue count
Effective cluster count
New ticket count
Tracked cluster count
Transaction warning count
Early-warning signal count
Run-health retrospective
Slack URL or stdout fallback
07

Command plan

operator path

The daily timer depends on read-only Sentry auth and the Linear CLI seeder.

Dry-run issuesudo -u symphony /opt/symphony/elixir/scripts/create_sentry_triage_issue.sh --dry-run --date 2026-05-21
Timer installsudo systemctl enable --now symphony-sentry-triage.timer
List timersystemctl list-timers | grep symphony-sentry-triage
Token modeSENTRY_FORCE_ENV_TOKEN=1 sentry-cli issues list --org agencycore --project ac-python-api
!Do not pin helper paths

Cache-keyed helper paths are brittle. Discover helpers dynamically and switch to equivalent read-only API or CLI calls after the first helper failure for an endpoint.

08

Rollout path

ship safely
1

Dry-run the seeder no writes

Confirm labels, title, state, active cycle behavior, report contract, and workpad rows.

2

Enable read-only access preflight

Install tokens and verify the Sentry CLI or API path works without interactive login.

3

Allow ticket creation idempotent

Create only untracked actionable clusters, then verify every label and state.

4

Tune noise filters daily review

Use repeated tracked or benign clusters to refine idempotency and summary rules.

AgencyCore / Symphony / Daily Sentry triageDesign draft / 2026-06-12