Back to Blog
Technical

AI agent monitoring: how to run AgentOps in production

AI agent monitoring explained: tracing, evaluations, cost and latency tracking, drift alerts and human review queues, with the tools teams actually use.

K

Klevere AI Team

Technical guide

22 November 202612 min read

An AI agent that worked perfectly in a demo can quietly start giving worse answers, calling the wrong tool or spending three times its budget a month after launch, and nobody notices until a customer complains. That gap between a working prototype and a dependable production system is where many agent projects stall. Gartner expects that by the end of 2027 over 40% of agentic AI projects will be cancelled, citing escalating costs, unclear business value and inadequate risk controls, as reported by SD Times.

Good AI agent monitoring addresses all three of those causes directly. It shows you what each run costs, whether the agent is actually doing useful work, and when its behaviour drifts outside the limits you set. This guide explains what to measure, which tools teams use, and how to build a monitoring setup that a small or mid-sized business can run without a dedicated platform team.

Quick answer

AI agent monitoring, often called AgentOps or LLM observability, means recording every step an agent takes (prompts, model calls, tool calls and outputs), then tracking quality, cost, latency and errors over time. Teams combine tracing tools such as LangSmith or Langfuse with automated evaluations, alerts and a human review queue for risky or low-confidence runs.

What AI agent monitoring actually covers

Traditional application monitoring tells you whether a server is up and how fast it responds. That is still necessary for an agent, but it misses the part that matters most: whether the agent made a sensible decision. An agent can return a fast, error-free response that is completely wrong, and a standard uptime dashboard will show green.

Monitoring an agent therefore has two layers. The first is operational: latency, error rates, token usage, cost per run and the health of every API the agent depends on. The second is behavioural: did the agent pick the right tool, follow its instructions, stay within policy and produce an answer a human would accept. The behavioural layer is what separates AgentOps from ordinary DevOps.

It also helps to agree on vocabulary. In [LangSmith's observability concepts]({LSC}), a run is a single unit of work executed by an agent, a trace is a collection of runs for a single operation, and a thread is a sequence of traces representing one multi-turn session. Most other tools use similar ideas, even if the names differ slightly, so these three terms are a useful shared language for your team.

The failures you are trying to catch are covered in depth in our guide to AI agent failure modes. Monitoring is the system that tells you, in production, which of those failures is happening and how often.

The five signals every production agent needs

You do not need hundreds of metrics. In practice, five groups of signals give a clear picture of an agent's health, and every one of them can be captured from traces.

  • Traces of every run: the full sequence of prompts, model responses, tool calls, retrieved documents and final output, so you can replay exactly what happened.
  • Quality scores: automated evaluations and human feedback attached to runs, showing whether answers were correct, grounded and on policy.
  • Cost per run and per outcome: token usage and API spend, ideally tied to a business result such as a resolved ticket or a booked meeting.
  • Latency at each step: total response time plus the time spent in each model call and tool, which shows where the slow parts are.
  • Errors and escalations: failed tool calls, timeouts, refusals, retries and handovers to a human.
  • The reason to capture all five together is that they trade off against each other. A change that cuts cost by switching to a cheaper model may reduce quality. A retry policy that lowers error rates may double latency. Seeing the signals side by side is how you make those trade-offs deliberately rather than by accident.

    Tracing: the foundation of LLM observability

    Tracing is the starting point because every other signal is derived from it. Without a trace, a bad answer is just a bad answer. With one, you can see that the retrieval step returned an outdated policy document, or that the agent called a CRM lookup with the wrong customer ID.

    Most modern tools record traces in roughly the same way. Langfuse's documentation describes traces as structured logs of every request that capture the exact prompt sent, along with the model response, token usage, latency and any tool or retrieval steps. Langfuse is also open source and can be self-hosted, which matters for teams with strict data residency needs.

    An industry standard is emerging underneath these tools. The OpenTelemetry project, which already defines how most software emits traces, has published semantic conventions for GenAI agent spans. They describe standard operation names such as create_agent and invoke_agent, so that different frameworks label agent activity consistently. The conventions are still marked as in development, so expect changes, but building on OpenTelemetry reduces the risk of being locked into one vendor's format.

    A practical tip: decide early what you will not log. Traces often contain customer names, email addresses and account details. Mask or redact personal data at the point of capture, set retention periods that match your data protection obligations, and restrict who can view raw traces.

    Evaluations: measuring whether the agent is any good

    Tracing shows what happened. Evaluations tell you whether it was right. There are three common approaches, and mature teams use all of them.

    Offline evaluations before release

    Build a test set of realistic inputs with known good outcomes, drawn from real conversations wherever possible. Every time you change a prompt, model or tool, run the agent against the test set and compare scores with the previous version. This is the agent equivalent of regression testing, and it stops a well-meant prompt tweak from breaking something that used to work.

    Online evaluations in production

    Once the agent is live, score a sample of real runs automatically. Checks can be simple rules (did the reply include a required disclaimer, did the agent call the approval tool before issuing a refund) or a second model acting as a judge of relevance and accuracy. Arize Phoenix is one open-source option here: it accepts traces over OpenTelemetry and provides evaluations that help you measure the output quality of your application.

    Human feedback

    Automated scores are only as good as the criteria behind them, so keep humans in the loop. Thumbs up or down from end users, corrections from staff and scheduled expert reviews all become labelled data. In LangSmith, feedback lets you score an individual run against set criteria, which turns those human judgements into a metric you can trend over time.

    One signal deserves its own attention: factual accuracy. Model error rates vary a lot by task and by how well the agent is grounded in your own data, which we cover in our AI hallucination rate benchmarks. Your evaluations should test grounding specifically, for example by checking that every claim in an answer can be traced back to a retrieved source.

    Cost and latency monitoring

    Agents are more expensive to run than single-prompt chatbots because one user request can trigger many model calls, tool calls and retries. A loop bug or an unexpectedly long document can multiply the cost of a single run, and without monitoring the first sign is often the monthly invoice.

    Gateway-style tools address this by sitting in the request path. Helicone's platform overview explains that every request is logged with costs, latency and errors tracked, and that session tracking lets you view the full conversation history across multiple model calls. Routing traffic through a gateway also gives you one place to add caching, rate limits and fallback providers.

    Whichever tool you choose, three practices keep spending under control:

  • Set a hard ceiling on steps, tokens or spend per run, and alert when runs approach it.
  • Track cost per successful outcome, not just cost per call, so a cheaper model that fails more often does not look like a saving.
  • Review the most expensive one per cent of runs each week, because that is where loops, oversized contexts and wasted retries hide.
  • Latency works the same way. Break the total response time into steps, find the slowest one and fix that first. Very often the bottleneck is a slow internal API or an oversized retrieval step rather than the model itself.

    Drift, alerts and human review queues

    An agent that passes every test on launch day will not stay the same. Model providers update their models, your own data changes, and users start asking questions nobody anticipated. Drift is the gradual change in inputs or outputs that slowly pulls quality away from where it started.

    Catching drift needs a baseline. Record your quality scores, escalation rate, average cost and common input topics during the first weeks of production, then alert when any of them move beyond an agreed threshold. LangSmith's observability features include dashboards and alerts for exactly this purpose, so you can track quality and catch issues early. It works with many frameworks and providers, and can run in the cloud, in a hybrid set-up or self-hosted.

    Alerts should lead somewhere useful. A human review queue is the simplest and most valuable control most businesses can add. Route runs into the queue when the agent's confidence is low, when an automated evaluation fails, when a run involves money or personal data above a set level, or when a customer gives negative feedback. Reviewers correct the outcome, label the cause, and those labels feed back into your test set.

    Security belongs in the same loop. Unusual tool calls, attempts to override instructions and requests for data outside the agent's scope are all signals of possible manipulation. Log and alert on these the same way you would on quality failures, because a manipulated agent often looks healthy on an uptime dashboard.

    Regulation and governance across major markets

    Monitoring is no longer only good engineering practice. In the European Union, Article 12 of the AI Act requires high-risk AI systems to technically allow for the automatic recording of events (logs) over the lifetime of the system, partly to support post-market monitoring. Article 19 adds that providers must keep those automatically generated logs, to the extent they control them, for a period appropriate to the system's purpose of at least six months unless other Union or national law says otherwise.

    Most business agents, such as a support assistant or an internal research tool, will not fall into the high-risk category. Even so, the logs and records described above are exactly what you would need if a regulator, auditor or customer asked how a decision was made. Our overview of EU AI Act obligations for SMBs explains which categories apply to smaller firms.

    Outside the EU, the guidance is voluntary but points the same way. In the United States, NIST published the Generative Artificial Intelligence Profile of its AI Risk Management Framework in July 2024 as a cross-sectoral companion resource intended for voluntary use. Organisations in the UK, Asia-Pacific and the Middle East frequently use the NIST framework as a reference too, so a monitoring set-up designed around clear logs, measured quality and documented human oversight will meet most expectations wherever you operate.

    How to choose AI agent monitoring tools

    The tools mentioned in this guide overlap, but each has a different centre of gravity. Use these questions to narrow the choice:

  • Where must the data live? If traces cannot leave your environment, favour tools that can be self-hosted, such as Langfuse or Arize Phoenix.
  • What framework are you using? LangSmith fits naturally with LangChain and LangGraph, though it also supports other frameworks through manual instrumentation.
  • Is cost control the main worry? A gateway such as Helicone gives you cost tracking, caching and provider fallback in one place.
  • Do you want portability? Prefer tools that accept OpenTelemetry traces, so you can switch vendors later without re-instrumenting the agent.
  • Who will look at the dashboards? A tool your operations team will actually open each week beats a more powerful one only an engineer understands.
  • Start small. One tracing tool, a test set of fifty to a hundred real examples, a weekly review of failed and expensive runs, and three or four alerts will catch most problems. Add complexity only once you know which signals matter for your agent.

    How Klevere approaches agent monitoring

    At Klevere we treat monitoring as part of the build, not an afterthought. Every custom AI agent development project includes tracing from the first prototype, a test set built from the client's real cases, and agreed quality thresholds before anything goes live. That way the first week in production has a baseline to compare against.

    We have deployed 500+ AI agents across 12 industries, and we design monitoring on the assumption that many production problems come from the surrounding systems rather than the model: a changed field in a CRM, a slow API, a policy document that was updated without the agent knowing. Monitoring that covers tool calls and data sources, not just model outputs, is what catches those issues early.

    We also design the human review queue with the client's team, so reviewers are the people who already own the process the agent supports. Their corrections improve the agent each month, and they keep clear ownership of the outcomes. It is one reason our clients stay with us, with 98% client retention across more than 50 projects.

    Frequently asked questions

    What is AgentOps?

    AgentOps is the practice of operating AI agents reliably in production. It covers tracing every run, evaluating output quality, tracking cost and latency, detecting drift, and routing risky cases to humans. It extends DevOps and MLOps with checks on the agent's decisions, not only on whether the software is running.

    What is the difference between LLM observability and AI agent monitoring?

    LLM observability usually refers to visibility into individual model calls: prompts, responses, tokens and latency. AI agent monitoring goes further, following multi-step runs where the agent plans, calls tools and makes decisions. In practice the same tools handle both, but agent monitoring needs traces that link every step of a run together.

    How do you evaluate an AI agent in production?

    Score a sample of live runs with automated checks, such as rule-based tests and a second model acting as judge, and combine those scores with user feedback and periodic expert review. Compare results against a baseline from launch, and feed failed cases back into your offline test set so they are checked on every future change.

    Which tools are used for AI agent monitoring?

    Common choices include LangSmith, Langfuse, Arize Phoenix and Helicone. LangSmith and Langfuse focus on tracing and evaluation, Phoenix is an open-source tracing and evaluation tool built on OpenTelemetry, and Helicone offers a gateway with cost tracking and caching. Many teams pair one tracing tool with their existing alerting system.

    How much logging does the EU AI Act require?

    For high-risk AI systems, the AI Act requires automatic logging of events over the system's lifetime and, for providers, keeping those logs for at least six months unless other law says otherwise. Most everyday business agents are not high-risk, but keeping similar logs is still sensible for audits, disputes and debugging.

    How often should you review an AI agent's performance?

    Check alerts continuously and review a sample of runs, including failures and the most expensive cases, at least weekly in the first months. Once the agent is stable, a monthly review of quality trends, cost per outcome and escalation rates is usually enough, with an extra review after any model or prompt change.

    If you are running agents today and are not sure what they cost, how often they get things wrong or where they are drifting, a short review can show you where to start. Book a free AI audit and we will map your current agents, the signals worth tracking and the quickest monitoring wins for your team.

    Ready to implement AI in your business?

    Let's discuss how AI agents can transform your operations and reduce costs.