Stacks/Data / ML platform
Pinterest's Analytics Agent and its tuning loop
Pinterest's Analytics Agent answers data questions by searching tables, dashboards, and metrics. To make it reliably follow required steps, the team built a 'loop engineering' process that proposes small edits to prompts, skills, context, or tools, races them against paired required and skip suites, and accepts a change only if it passes regression and anti-overfitting checks.
- Use case
- An internal Analytics Agent that searches Pinterest's data ecosystem (tables, dashboards, metrics) to answer questions, plus an automated 'loop engineering' process that improves how reliably it follows required steps
- Category
- Data / ML platform
- Team size
- not stated
- Orchestrator / framework
- The agent uses skills (such as data discovery), table-metadata inspection, and search across source types. The improvement loop starts each iteration in a fresh session with a structured handoff (scores, failing cases, best change so far, failed approaches), drafts up to four edits to the system prompt, skills, context layer, or tools, races them on a small sample, and scores the winner on full suites. Edits are capped at a few files and lines, and the cap shrinks over the run.
- Memory
- The tuning loop's memory is a compact structured handoff between fresh sessions rather than conversation history.
- Evals
- Per-rule paired 'required' and 'skip' suites labeled by a focused LLM labeler and reviewed by humans, a larger regression suite, tuning/validation splits, and a final holdout. Edits must pass three checks (net fixes, no hard-coding, no regression). Runs found 10.3% of cases flip verdict between identical runs, so the loop tracks case-level changes, not just the score.
- Observability
- not stated
- Cost (stated in writeup)
- Not stated
- Cost per output (stated)
- Not stated in the writeup
- WhatWeRun estimate
WhatWeRun estimate
Not enough info to estimate
The post doesn't name the model or give tokens or volume per question.
Reviewed Nov 13, 2026
- Models
- Observability tools
- not stated
- Other tools
- Chat logs (parsed for compliance)
- Source
- https://medium.com/pinterest-engineering/how-pinterest-improved-analytics-agent-compliance-with-loop-engineering-d3c52d0abaa2
Skyler Zheng, Chris Moradi, Lu Liu, Aman Gairola, and Michael Chau · Published Nov 12, 2026 · Last verified Nov 13, 2026
Related
More like this
Netflix Pensive AI failure diagnosis
AI error extraction and on-demand root-cause debugging for failed data and ML workloads on Netflix's data platform (Maestro, Spark, Trino, Druid, Titus), where roughly 100,000 workloads fail on peak days
databricks.comDatabricks MemCurrent memory for Genie
Persistent memory for production data agents; powers Agent Bricks Memory and Genie, where users ask questions about their enterprise data
engineering.fb.comMeta's thin orchestrator for data pipeline agents
Automating data column onboarding across Meta recommendation-system pipelines, spanning three code repositories, three programming languages, and up to six pipeline stages