Stacks/E-commerce assistant
Shopify Sidekick
Shopify's Sidekick evolved from a simple tool-calling assistant into a production agentic platform for merchants. The main architectural fix for tool sprawl was just-in-time instructions, which kept the system prompt small and cache-friendly. The team is unusually explicit about evaluation: real-conversation ground truth, judges correlated with humans, a user simulator, and a documented fight with reward hacking during GRPO training.
Andrew McNamara (based on a talk with Ben Lafferty and Michael Garner at ICML 2025)Aug 25, 2025Original writeup ↗
- Use case
- An AI assistant that helps merchants manage stores through natural language — querying customer segments, writing SEO descriptions, filling product forms, and navigating admin interfaces
- Category
- E-commerce assistant
- Team size
- not stated
- Orchestrator / framework
- Sidekick is built around an agentic loop (human input, LLM decision, tool execution, feedback). As the tool inventory grew past dozens of capabilities, the team moved guidance out of a bloated system prompt and into just-in-time instructions returned alongside tool data. The authors advise staying with a simple single-agent system early rather than jumping to multi-agent architectures.
- Memory
- not stated
- Evals
- Evaluation uses Ground Truth Sets sampled from real merchant conversations, labeled by at least three product experts. Specialized LLM judges were calibrated against human agreement (improved from Cohen's Kappa 0.02 to 0.61, versus a 0.69 human baseline). An LLM-powered merchant simulator replays real conversation goals against candidate systems. For fine-tuning, the team used Group Relative Policy Optimization (GRPO) with N-Stage Gated Rewards (procedural validators plus LLM judges) and had to iterate after the model learned reward hacks such as opting out of hard tasks.
- Observability
- not stated
- Monthly cost
- not stated
- Models
- not stated
- Observability tools
- not stated
- Other tools
- LLM-as-judge evaluatorsMerchant user simulatorGRPO training
- Source
- https://shopify.engineering/building-production-ready-agentic-systems
Andrew McNamara (based on a talk with Ben Lafferty and Michael Garner at ICML 2025) · Aug 25, 2025