Stacks/Agent platform / infrastructure
Datadog's Insights agent, tuned by its own traces
Datadog points its Agent Observability Insights at the agent that powers Insights. Traces showed oversized search results re-sent to the model, a prompt layout that defeated caching (projected at about $2,734 a month in savings, not yet measured), and failing tool calls. Fixes go to a coding agent that opens a tested PR, then get validated by replaying investigations.
- Use case
- The agent behind Datadog Agent Observability Insights, which analyzes agent traces to find cost and reliability problems; Datadog uses Insights on that agent's own traces to cut tokens and fix failing tool calls
- Category
- Agent platform / infrastructure
- Team size
- not stated
- Orchestrator / framework
- The agent investigates recorded traces with search tools and a skill loader that returns exact resource names. Findings can be passed via 'Fix with Bits' to a coding agent, which changes the code, adds tests, and opens a PR.
- Memory
- not stated
- Evals
- Fixes are validated by replaying affected investigations and comparing tool output size, model tokens, and answer quality; resource-name fixes are checked by replaying the affected turns.
- Observability
- Datadog Agent Observability traces every run; Insights flagged ~80,000 tokens of search results re-sent across two runs, poor prompt-cache reuse (task details placed before tool definitions), a tool call retried four times over ~18 minutes, and guessed resource names across 193 sessions.
- Cost (stated in writeup)
- Not stated
- Cost per output (stated)
- Not stated in the writeup
- WhatWeRun estimate
WhatWeRun estimate
Not enough info to estimate
The post doesn't name the model or give run volume; the $2,734/month figure is a projected saving, not the agent's cost.
Reviewed Nov 13, 2026
- Models
- Observability tools
- Other tools
- Bits (coding agent, 'Fix with Bits')
- Source
- https://www.datadoghq.com/blog/improving-insights-agent-observability/
Alex Barksdale, Miguel Tulla Lizardi, and Zhengda Lu · Published Oct 30, 2026 · Last verified Nov 13, 2026
Related
More like this
developer.nvidia.comNVIDIA Agent Substrate
A security-hardened, zero-trust Kubernetes runtime for running sandboxed agents and tools at high density — coding assistants, RL post-training environments, and short-lived GPU sandboxes — now donated to the CNCF Sandbox
langchain.comWorkday's internal Agent Platform
An internal developer platform for building, deploying, and running AI agents for HR, finance, and IT on Workday's infrastructure, with auth, security, compliance, and audit built in
Cloudflare Project Think / Agents SDK
Cloudflare's platform and Agents SDK for building long-running, durable, serverless agents, described as already powering thousands of production agents, with Project Think adding the next set of primitives