LLM Observability: Monitor AI in Production

You ship an AI feature. Users can use it. It responds quickly in staging. You push to production.

Three weeks later, a customer support ticket arrives: “The AI gave me completely wrong information.” You check your logs. You see the HTTP request. You see the 200 OK. You have no idea what the model actually received, what it produced, or why it went wrong.

That gap — between “the API call succeeded” and “the AI feature worked” — is where most teams are flying blind.

What Standard Monitoring Misses

Your APM tool tracks latency, error rates, and p99 response times. It’s good at telling you when your server crashed. It tells you almost nothing useful about AI behavior.

LLM calls don’t fail the way APIs fail. A 200 OK response with a plausible-sounding hallucination looks identical to a 200 OK with a genuinely useful answer. Your current dashboards cannot tell the difference. The request succeeded — the model just gave the user bad information.

The four signals you actually need

Beyond uptime, production AI features require monitoring across four dimensions:

• Cost per call — LLM APIs are priced per token. A prompt that works at 1,000 users might cost 10× what you budgeted at 100,000. You need per-request cost visibility before the bill arrives.
• Output quality — Did the model answer the question? Did it follow your format instructions? Did it stay in scope? This requires some form of automated scoring, not just a “request succeeded” boolean.
• Latency breakdown — Total response time is one number. Time-to-first-token, time-per-token, and model processing time separately are what tell you where the bottleneck actually is.
• Full input/output traces — When something goes wrong, you need to see the exact prompt the model received and the exact response it returned. Not a sanitized summary — the full trace.

Tools Worth Using

Adding observability to an LLM feature doesn’t require rearchitecting your app. Three tools cover most production needs.

Langfuse

Langfuse is open-source and self-hostable, which matters when you’re working with sensitive data and can’t route production traffic through a third-party proxy. It wraps around your LLM calls and captures traces, scores, and user-level cost aggregations.

The integration is minimal. For Python with the OpenAI SDK:

from langfuse.openai import openai

response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": prompt}],
    name="summarize-document",
)

Langfuse starts recording metadata — model, tokens in, tokens out, latency, and cost estimate — from that point. You can drill into individual traces from the dashboard, filter by user, and track metric trends over time.

Helicone

Helicone takes a proxy approach. You change one line:

client = OpenAI(
    base_url="https://oai.helicone.ai/v1",
    default_headers={"Helicone-Auth": f"Bearer {HELICONE_API_KEY}"}
)

All your OpenAI calls now flow through Helicone’s logging layer. No SDK changes, no decorator wrapping. The tradeoff: your data touches their infrastructure. Fine for most SaaS products, a non-starter for enterprise or healthcare.

Arize Phoenix

If you’re running multi-step RAG pipelines or agent chains, Phoenix by Arize is built for that. It handles evaluation workflows — you can run LLM-as-judge scoring against logged traces and surface quality regressions automatically. Think of it as your LLM evals running continuously against production traffic, not just in CI.

What to Actually Instrument

The tool is the easy part. The harder part is deciding what granularity to trace and what to do with the data.

Trace each pipeline step independently

If you have a multi-step pipeline — retrieve context, rerank, summarize, extract structured output — trace each step independently, not just the full request. When a user gets a bad answer, you need to know whether it was a retrieval failure or a generation failure. Tracing the whole pipeline as one call makes that impossible to diagnose.

This is especially true if you’re building on agentic architectures. A 10-step agent chain that fails on step 7 is undebuggable without step-level traces.

Track cost by user or session

LLMs can be surprisingly expensive when a small number of power users run long sessions. Tracking cost by user_id surfaces these outliers before your cloud bill does. Set a soft threshold and log a warning when a session exceeds it — not to block users, but to identify candidates for prompt optimization or response caching. This pairs directly with AI API cost control techniques you can apply once you know where the spend is concentrated.

Build a quality signal early

Waiting for user complaints to surface bad outputs is too slow. Even a simple automated check — does the response match the expected JSON schema? is it at least 50 words? does it reference the user’s input correctly? — creates a quality signal you can monitor over time.

Pair that with periodic eval runs against logged traces, and you have a feedback loop that catches regressions before they accumulate.

The Feedback Loop That Matters

Observability isn’t just a debugging tool.

Every trace you log is a potential training example for your next prompt iteration. Every failed output you capture is a test case for your eval suite. When you treat production logs as a dataset — not just an audit trail — your AI features compound in quality over time instead of silently drifting.

Teams that ship AI features and then ignore them are operating on hope. Teams that instrument properly know exactly what’s working, what’s degrading, and where to spend engineering time next.

This is the infrastructure we build into every AI engagement at quickdev. If you’re starting an AI feature and want to get the monitoring layer right from day one rather than retrofitting it later, our AI development service covers the full stack — from initial integration through production observability.

---

Yaniv Amrami is founder of quickdev. He has helped Israeli startups ship and operate production AI features since the early days of the LLM API ecosystem.