Stacks/SRE / incident response
Cleric's ~40-agent AI SRE
Cleric split its single Claude investigation agent into about 40 role-based agents under a central investigator, mixing the Claude Agent SDK and Google ADK with Gemini, OpenAI, and other models per role. The change made evals 100x cheaper and 5x faster, so the team now runs about 5,000 evals a week and picks each role's model with a focused eval run.
- Use case
- An AI SRE that takes a system alert, plans an investigation, queries observability systems and Kubernetes (logs, metrics, traces, pod status), and iterates on hypotheses to find the root cause
- Category
- SRE / incident response
- Team size
- not stated
- Orchestrator / framework
- A central investigator agent orchestrates a team of about 40 loosely coupled agents referred to by role (analyst, judge, auditor, and so on), across two harnesses and several model families. The new architecture ran in shadow mode beside the old single agent before taking all traffic. Swapping a model or harness for one role is a config change plus a focused eval run.
- Memory
- not stated
- Evals
- End-to-end evals simulate a failure with a known root cause and check the agent finds it; focused per-role evals now take about 30 seconds and about 1 cent each. They run about 5,000 evals a week (up from about 180), with model swaps decided per role on recall, latency, and cost, and still run end-to-end evals, though not for every change.
- Observability
- In-house tracing, cost tracking, and profiling, extended for multi-agent runs where one investigation spans several correlated traces.
- Cost (stated in writeup)
- Eval runs: about $1.50 per end-to-end investigation run (full suite $200) on the old single agent; over $12,000 on evals in one month; now about 8 cents per eval run on average and $1.50 for the whole suite · per run
- Cost per output (stated)
- Stated: $2.03 per investigationNew multi-agent version in production shadow mode, vs $3.30 for the main agent (47 paired investigations)
“In production, over the first ~50 paired investigations, the shadow ran at comparable quality with the main agent, but significantly cheaper and faster. [Chart: Production shadow, 47 investigations: cost per investigation $2.03 shadow, $3.30 main agent.]”
Stated: ~8¢ per eval runEvalDown from about $1.50 per end-to-end eval run“We now run 5,000 evals a week (30x more), at about 8 cents each on average.”
As stated in the writeup, checked 2026-11-13. Separate from the WhatWeRun estimate below.
- WhatWeRun estimate
WhatWeRun estimate
Not enough info to estimate
The post states eval-run costs (shown above) but not the production cost per investigation, and the per-role model mix varies. The stated costs stand.
Reviewed Nov 13, 2026
- Models
- Observability tools
- not stated
- Other tools
- PrometheusKubernetes
- Source
- https://cleric.ai/blog/how-we-made-evals-100x-cheaper-with-multi-agent-architecture
Dan Siwiec · Published Nov 3, 2026 · Last verified Nov 13, 2026
Related
More like this
blog.duolingo.comDuolingo AI STR Fix bug agent
Investigating and fixing Shake-to-Report bug tickets: gathering context (relevant code, tracking events, experiment states, recent PRs, repro steps) and proposing a fix for engineers to verify
developer.nvidia.comNVIDIA AI Factory Operations Agent
Detecting failures and correlating root causes across compute, network, hardware, and scheduling for NVIDIA DGX SuperPOD operations, with recommendations in a chat interface for SREs
getstream.ioStream's production investigation agent
First-line investigation of production issues and support questions for Stream's chat API backend; posts a cited root cause in Slack and waits for a human to approve any fix