Stacks/AI product infrastructure
An anonymous startup's production AI infrastructure (HN)
On an Ask HN thread, k9294 shares a full production stack from a small startup: an OpenAI-compatible gateway with fallbacks and cost tracking, a homegrown job queue they already want to replace, OTEL plus SigNoz (no LLM-output logging), Postgres and S3 for memory, LlamaIndex and Qdrant for RAG, ClickHouse via Tinybird for usage analytics, and Stripe meters for billing. The company name is withheld.
- Use case
- Production AI infrastructure at a relatively small startup with a few thousand monthly active users, covering chat, transcription, image generation, embeddings, RAG, and user-facing tool integrations
- Category
- AI product infrastructure
- Team size
- A relatively small startup (headcount not stated); a few thousand monthly active users
- Orchestrator / framework
- The team started with LangChain and moved off it. They now use the OpenAI SDK plus a custom API gateway that handles cost tracking, metered billing, model fallbacks, aliases, and feature flags (structured output, reasoning, modality). Workflow orchestration is a simple task executor and background job queue for retries and durable execution; the author says this has become a limitation and they are considering Temporal, trigger.dev, and Conductor, with no chosen winner yet.
- Memory
- Postgres for agent memory, chat history, and persistence (a chat-app-style schema of threads, messages, and attachments), plus S3 for files and images. RAG is a custom LlamaIndex ingestion service with managed Qdrant for vector search.
- Evals
- not stated
- Observability
- OpenTelemetry and SigNoz. The team does not track LLM outputs, for security reasons, because the product is a consumer app with a lot of private data.
- Monthly cost
- not stated
- Observability tools
- Other tools
- ClickHouse via Tinybird (usage and cost analytics)Stripe meters (evaluating Orb as a replacement)LlamaIndexQdrantLLM tool wrappers and OAuth/API-key credential management
- Source
- https://news.ycombinator.com/item?id=43388679
k9294 · Mar 16, 2025