Stacks/SRE / incident response

Traversal AI SRE investigation agent

Traversal moved its AI SRE investigator off an API-server process and onto Temporal, first by wrapping a whole investigation in one activity, then by rebuilding the harness so each model and tool call is durable. The write-up is a detailed account of granular retry, resumable runs, child-workflow sub-agents, a durable state ledger, and native stop-and-steer controls during live incidents.

Use case

Autonomous production-incident investigation: agents reason over a live 'Production World Model' of a customer's systems, narrow candidate signals, and produce a root-cause report while someone is watching an outage

Category

SRE / incident response

Team size

not stated

Orchestrator / framework

A custom investigation harness rebuilt around Temporal primitives. The core agent loop is a Temporal workflow that decides the next step and dispatches work; every model call and tool call is an activity with its own deadline and retry policy; sub-agents and long-running analysis are child workflows. A related Traversal Worker is a long-running incident-channel agent whose workflow sleeps until signals, schedules, or timers wake it. Humans, Workers, and parent agents steer or stop in-flight investigations via Temporal Updates.

Memory

State lives in a durable ledger rather than in process memory: model messages, evidence, analysis results, and sandbox assignment are written so any activity can reconstruct what it needs. A retry clears partial rows before rebuilding so a turn is not duplicated in the transcript.

Evals

not stated

Observability

Temporal event history and a per-step timeline of what the agent is working on. The agent itself analyzes customer metrics, events, logs, traces, code, and deploys, and can open a scoped shell against the customer's cluster.

Monthly cost

not stated

Models

not stated

Observability tools

Temporal

Other tools

Production World ModelCausal Search EngineScoped customer-cluster shellSlack and UI steering

Source

https://www.traversal.com/blog/rebuilding-the-traversal-agent-runtime-on-temporal

Lyndon Vickrey