Stacks/SRE / incident response

NVIDIA AI Factory Operations Agent

NVIDIA runs a multi-agent operations assistant fully on-premises across its DGX H100, B200, and GB300 production clusters: a NemoClaw orchestrator routes SRE questions to six domain agents powered by locally served Nemotron models, with MCP tools, per-session OpenShell sandboxes, and read-only Kubernetes access. Lessons include read-only investigation to build trust and precise MCP tool definitions.

Original writeup

developer.nvidia.com

NVIDIA Technical Blog · developer.nvidia.com

How NVIDIA Uses AI Agents to Streamline AI Factory Operations | NVIDIA Technical Blog

As GPU clusters scale, operations span compute, network, hardware, and scheduling. Site reliability engineers (SREs) often have to examine multiple dashboards to identify the cause of training…

Read the writeup

Kavya G Y and Rohan AdwankarPublished Nov 4, 2026Last verified Nov 13, 2026
NVIDIA NemoClaw blueprintOpenClaw (gateway and plugins)NVIDIA OpenShell runtime
NVIDIA Nemotron 3 (locally served open models)
Use case
Detecting failures and correlating root causes across compute, network, hardware, and scheduling for NVIDIA DGX SuperPOD operations, with recommendations in a chat interface for SREs
Category
SRE / incident response
Team size
not stated
Orchestrator / framework
A NemoClaw agent receives SRE queries and routes them to domain agents: Job Recovery, Hardware, Observability, Research, Kubernetes, and Cluster Management. Agents reach infrastructure through OpenClaw plugins and MCP servers that each expose a focused set of approved tools. It runs as a Kubernetes control-plane app deployed with Helm; OpenShell creates a per-session Kubernetes sandbox for shell execution, and Kubernetes inspection uses read-only credentials and RBAC.
Memory
not stated
Evals
NVIDIA measures root-cause-analysis accuracy and tracks SRE agreement.
Observability
Grafana dashboards are exposed as a cross-cutting MCP tool.
Cost (stated in writeup)
Not stated
Cost per output (stated)
Not stated in the writeup
WhatWeRun estimate

WhatWeRun estimate

Not enough info to estimate

Models are self-hosted open models on NVIDIA's own GPUs, so there's no list price per token, and no tokens or volume are given.

Reviewed Nov 13, 2026

Models
NVIDIA Nemotron 3 (locally served open models)
Observability tools
Grafana
Other tools
MCPKubernetes and HelmWeb search and code execution tools
Source
https://developer.nvidia.com/blog/how-nvidia-uses-ai-agents-to-streamline-ai-factory-operations/

Kavya G Y and Rohan Adwankar · Published Nov 4, 2026 · Last verified Nov 13, 2026

Related

More like this

cleric.ai

SRE / incident response

Nov 3, 2026

Cleric's ~40-agent AI SRE

An AI SRE that takes a system alert, plans an investigation, queries observability systems and Kubernetes (logs, metrics, traces, pod status), and iterates on hypotheses to find the root cause

Stated: $2.03 per investigation+1

Claude Agent SDKGoogle ADK+7
getstream.io

SRE / incident response

Oct 22, 2026

Stream's production investigation agent

First-line investigation of production issues and support questions for Stream's chat API backend; posts a cited root cause in Slack and waits for a human to approve any fix

Custom pipeline serviceClaude Haiku (triage)+6Est. $2–$7 per investigation
traversal.com

SRE / incident response

Oct 16, 2026

Traversal AI SRE investigation agent

Autonomous production-incident investigation: agents reason over a live 'Production World Model' of a customer's systems, narrow candidate signals, and produce a root-cause report while someone is watching an outage

TemporalCustom agent harness+4