Stacks/Data / ML platform
Netflix Pensive AI failure diagnosis
Netflix extended its rules-based Pensive system with Pensive AI: always-on LLM error extraction that now covers 100% of failures, plus cached, on-demand debugging guidance through UI, API, and MCP. Legacy rules double as eval labels, PRs are gated by eval experiments in preview environments, and Braintrust-based tracing ties every response to user feedback.
- Use case
- AI error extraction and on-demand root-cause debugging for failed data and ML workloads on Netflix's data platform (Maestro, Spark, Trino, Druid, Titus), where roughly 100,000 workloads fail on peak days
- Category
- Data / ML platform
- Team size
- not stated
- Orchestrator / framework
- Two paths share log sources, an LLM gateway, tracing, and a cache: always-on AI error extraction for every failure, and on-demand Pensive AI Debugging (root cause plus remediation) via API, MCP, and the Data Portal UI. Debugging is cached per failed step attempt with a conditional-write claim so duplicate requests become key-value reads; generation streams and persists even if the client disconnects.
- Memory
- Debugging responses are cached per step attempt. Fetched logs are archived to S3 asynchronously so eval inputs stay stable after source logs expire.
- Evals
- Legacy Pensive rules serve as reference labels; four signals for extraction (an LLM judge for known-error recall, a second judge for actionability, a dataset-level script for recurring boilerplate, and latency). Debugging ground truth is mined from engineering-support Slack threads. Every PR deploys to a preview environment where a CI eval experiment gates the merge on precision and inclusion thresholds; evals also run after promotion and on a schedule against production.
- Observability
- Distributed tracing with child spans for each log fetch and extraction tool call, on a Braintrust-based tracing backend. Thumbs up/down feedback joins the exact trace, and each thumbs-down sends a business-hours Slack alert.
- Cost (stated in writeup)
- Not stated
- Cost per output (stated)
- Not stated in the writeup
- WhatWeRun estimate
WhatWeRun estimate
Not enough info to estimate
The post doesn't name the model or give tokens per extraction or debugging response.
Reviewed Nov 13, 2026
- Observability tools
- Other tools
- MCPS3LLM gatewayData Portal
- Source
- https://netflixtechblog.com/making-failure-diagnosis-reliable-with-pensive-ai-6b0811a1c015
Praneeth Yenugutala, Victor Dusa, Anastasia Samoilova, Megan Ren, Justin Wei, and Marcelo Mayworm · Published Oct 26, 2026 · Last verified Nov 13, 2026
Related
More like this
Pinterest's Analytics Agent and its tuning loop
An internal Analytics Agent that searches Pinterest's data ecosystem (tables, dashboards, metrics) to answer questions, plus an automated 'loop engineering' process that improves how reliably it follows required steps
databricks.comDatabricks MemCurrent memory for Genie
Persistent memory for production data agents; powers Agent Bricks Memory and Genie, where users ask questions about their enterprise data
engineering.fb.comMeta's thin orchestrator for data pipeline agents
Automating data column onboarding across Meta recommendation-system pipelines, spanning three code repositories, three programming languages, and up to six pipeline stages