Stacks/SRE / incident response

Stream's production investigation agent

Stream's investigation agent triages Slack messages and alerts with Claude Haiku, investigates with Claude Opus in a locked-down sandbox, and has a fresh session try to disprove the answer before posting. Every claim must cite a query result or a line of code, the agent is read-only by construction, and only engineers can approve fixes. A 22-case eval set comes mostly from real incidents.

Original writeup

getstream.io

getstream.io

How We Built an AI Agent to Investigate Production Issues

Our AI agent reads logs, databases, metrics and code, then posts a root cause with evidence in Slack. This is how it works, how we kept it safe, and the four design choices behind it.

Read the writeup

Aditya NugrahaPublished Oct 22, 2026Last verified Nov 13, 2026
Custom pipeline service
Claude Haiku (triage)Claude Opus (investigation)
Use case
First-line investigation of production issues and support questions for Stream's chat API backend; posts a cited root cause in Slack and waits for a human to approve any fix
Category
SRE / incident response
Team size
not stated
Orchestrator / framework
A service reads Slack support and incident channels and starts when tagged or when a production alert fires. Pipeline: Haiku triages whether to investigate; Opus investigates in a sandbox; a fresh-context session tries to prove the result wrong; the service posts the answer, or numbered fix options only Stream's chat engineering team can approve. Approved code changes become draft PRs; operational fixes become runbooks an engineer carries out.
Memory
not stated
Evals
22 cases, most from real support threads and incidents, several from the agent's own past mistakes, including a fabrication trap and cases that should be dropped at triage. Each runs through the real pipeline (without Slack) and is scored against the known outcome; answers making 'wrong-answer' claims fail. Run by hand, not on every commit, because it uses real model calls.
Observability
not stated
Cost (stated in writeup)
Not stated
Cost per output (stated)
Not stated in the writeup
WhatWeRun estimate

WhatWeRun estimate

Low confidence

$2–$7 per investigation

List prices as of Nov 2026 · estimated Nov 13, 2026 · list prices, no enterprise discounts · not a writeup-stated cost

How we estimated

Stated in the writeup

  • “Triage. A small, fast model (Claude Haiku) decides whether the message needs an investigation at all.” source ↗
  • “Investigate. A larger model (Claude Opus) works in a sandbox and produces either an answer or a proposed fix.” source ↗
  • “Review. A fresh session tries to prove the result wrong.” source ↗

List prices used (Nov 2026)

  • Claude Opus 5.5 (input / output), investigation and review: $4 / $20 per million tokens docs.claude.com ↗
  • Claude Haiku 5.5 (input / output), triage: $0.10 / $0.50 per million tokens docs.claude.com ↗

Our assumptions (not from the writeup)

  • Assumed:Model versions aren't stated; priced at the current Claude Opus 5.5 and Haiku 5.5 list prices.
  • Assumed:The review session is assumed to use the same Opus model as the investigation.
  • Assumed:Low: investigation 300,000 input + 15,000 output tokens; review 100,000 input + 5,000 output.
  • Assumed:High: investigation 1.2M input + 40,000 output tokens; review 300,000 input + 10,000 output.
  • Assumed:Triage is a few thousand Haiku tokens (well under a cent). No prompt-cache discount; sandbox compute excluded.

Arithmetic

  • Low: 300k x $4/M = $1.20 + 15k x $20/M = $0.30 + review 100k x $4/M = $0.40 + 5k x $20/M = $0.10 -> $2.00
  • High: 1.2M x $4/M = $4.80 + 40k x $20/M = $0.80 + review 300k x $4/M = $1.20 + 10k x $20/M = $0.20 -> $7.00

What would tighten this

Tokens per investigation (or which Opus version runs it).

Models
Claude Haiku (triage)Claude Opus (investigation)
Observability tools
not stated
Other tools
SlackIsolated sandbox container with read-only code copyRead-only tools for logs, databases, metrics, storage, and codeHost-side read-query service for production DB credentialsOutput filter for secrets and customer content
Source
https://getstream.io/blog/ai-agent-production-investigation/

Aditya Nugraha · Published Oct 22, 2026 · Last verified Nov 13, 2026

Related

More like this

developer.nvidia.com

SRE / incident response

Nov 4, 2026

NVIDIA AI Factory Operations Agent

Detecting failures and correlating root causes across compute, network, hardware, and scheduling for NVIDIA DGX SuperPOD operations, with recommendations in a chat interface for SREs

NVIDIA NemoClaw blueprintOpenClaw (gateway and plugins)+5
cleric.ai

SRE / incident response

Nov 3, 2026

Cleric's ~40-agent AI SRE

An AI SRE that takes a system alert, plans an investigation, queries observability systems and Kubernetes (logs, metrics, traces, pod status), and iterates on hypotheses to find the root cause

Stated: $2.03 per investigation+1

Claude Agent SDKGoogle ADK+7
traversal.com

SRE / incident response

Oct 16, 2026

Traversal AI SRE investigation agent

Autonomous production-incident investigation: agents reason over a live 'Production World Model' of a customer's systems, narrow candidate signals, and produce a root-cause report while someone is watching an outage

TemporalCustom agent harness+4