CIOps agent
Watches CI, deploy, Sentry, and release signals to keep the engineering loop fast and healthy.
Operating brief
design targetCIOps should be the second autonomous agent after CostOps. It has quick binary outcomes, mature runbook patterns, and mostly reversible actions. The design keeps incident-style evidence and CI-specific remediation separate so it can improve without surprising engineers.
Every agent uses the same loop: ingest, normalize, analyze, detect, decide, log. The useful design work is deciding what evidence enters the context, which actions exist, which actions stay dry-run, and how the ledger proves that the agent helped.
Agent design diagram
click nodesThe diagram is the high-level system boundary for this agent. Click a node below it to inspect what belongs in that layer and what should stay outside the agent.
CIOps loop
Select runbook, inspect diffs and recent incidents, propose retry, quarantine, pin, rollback draft, or owner escalation.
Signals, detectors, and outputs
design inventoryThis is the detector inventory to design first. Each detector needs a source signal, a user-visible output, and a measurement that can be written back to the ledger.
Flaky test cluster
- Signal
- Same test alternates pass/fail without relevant code changes.
- Output
- Retry, quarantine draft, owner task, or stability report.
- Ledger metric
- Flake rate and recurrence after action.
Autonomy gate
playgroundUse this as a policy-design sketch. The values are not final production thresholds; they show which classes of action should be eligible for auto-execution and which should escalate.
Auto-execute
autoThe action is reversible, confidence is above the floor, and impact stays inside the policy envelope.
- Class
- Reversible CI action
- Reversibility
- full
- Design note
- Good first auto-action when the failure matches known transient signatures.
Context and memory design
agent envelopeResearch converges on context engineering as the quality lever: keep a stable policy prefix, put fresh evidence near the task, retrieve only relevant lessons, and expose a small tool surface.
Stable prefix
CI policy, branch protection rules, runbook catalogue, ownership map, retry caps, and production-action boundaries.
Per-run evidence
Failed job logs, test report, commit range, changed files, deployment marker, Sentry release health, and linked incidents.
Memory retrieval
Past flakes, effective retries, bad quarantines, known dependency outages, and per-repo failure signatures.
Tool surface
Read CI run, read logs, retry job, draft PR, create owner task, read Sentry release, submit policy envelope.
Evals and rollout
ship safelyBehavior checks
- Failure classificationThe agent distinguishes transient, flaky, deterministic, dependency, and release-regression classes.
- Runbook fitEvery action cites the matching runbook and why alternative runbooks were rejected.
- Blast-radius guardBranch protection changes, production rollbacks, and multi-repo dependency pins always escalate.
- Outcome loopThe ledger records rerun result, duration delta, and recurrence within a fixed observation window.
Open design questions
- Which CI log format is canonical across repos?
- What maximum retry budget keeps the agent from hiding real failures?
- Where should quarantine expiry live so temporary skips do not become permanent?
CI event spine
Ingest GitHub Actions, test reports, deployments, and Sentry release health into one timeline.
Read-only triage
Classify failures and generate evidence packs without taking action.
Bounded auto-actions
Allow transient retries and internal labels through the policy engine with strict caps.
Draft remediation
Generate quarantine or rollback drafts, but keep merge and production changes behind approval.
Research inputs
sourcesThese are the sources used to shape the page. The resulting design keeps the vendor-specific advice at the architecture level: tool surfaces, guardrails, context, orchestration, outcome logging, and domain metrics.