An AI agent can give the right answer and still do the wrong thing.

Imagine a support agent that handles refund requests. A customer asks if they can get their money back. The agent reads the refund policy and replies, “Yes, you qualify.” That sounds fine. But behind the scenes, it may have checked the wrong order, skipped a required approval, or even issued a refund without permission.

If you only judge the final reply, you may never catch the mistake.

That is why LLM agent evaluation looks at the full job. It checks what the agent said, what it did, which tools it used, and whether the task ended the right way. This guide covers the main evaluation methods and metrics, how to build tests, and how to keep checking an agent after launch.

What Is LLM Agent Evaluation?

LLM agent evaluation is the process of testing how well an AI agent makes decisions, uses tools, follows instructions, and finishes tasks. The goal is to find both whether the agent succeeded and what caused it to fail when it did not.

A large language model (LLM) can produce text from a prompt. An LLM agent can go further. It can choose an action, call an API, search a database, use a tool, study the result, and decide what to do next. Some agents also plan tasks, save state, or send work to other agents.

The common pattern is a loop:

Yes No Call tool or delegate User request or new feedback Read result and update state Agent decides next action More work needed? Finish or hand off to a person User request or new feedback Agent decides next action More work needed? Yes Call tool or delegate Read result and update state No Finish or hand off to a person

The full record of this run is called a trajectory. A trace is the structured log of the run, often split into smaller records called spans. A span might show one model call, one search, one tool call, or one sub-agent task.

LLM Evaluation vs. LLM Agent Evaluation

Regular LLM evaluationLLM agent evaluation
Often checks one prompt and one responseChecks a whole series of decisions and actions
Focuses on accuracy, relevance, helpfulness, and safetyAdds tool choice, arguments, planning, retries, handoffs, and task success
Can work from the input and final text aloneOften needs tool logs, state changes, and traces
Answers “Was the response good?”Also answers “Did the system do the job properly?”

An agent still needs basic LLM testing. It simply needs more tests on top of it.

Test Every Decision Your AI Agent Makes

Bring in LLM testing specialists to check tool calls, reasoning, and final outcomes.

LLM Agent Evaluation vs. RAG Evaluation

Retrieval-augmented generation (RAG) lets an LLM use retrieved documents before it answers. RAG evaluation asks whether the system found useful content and whether the answer matches that content.

If an agent uses RAG, these checks still matter. But the agent may also decide when to search, which source to search, whether to try again, and what action to take with the result. Those choices need their own tests.

In short, RAG evaluation may be one part of an agent’s broader evaluation.

Why Is LLM Agent Evaluation Important?

Agent failures can be hard to spot. The final answer may look normal even when the path behind it was wasteful, wrong, or unsafe.

Here are the main problems:

  1. Small mistakes spread. One wrong search result or tool argument can affect every later step.
  2. Long tasks hide failures. An agent might work for several minutes and make many calls before the real mistake shows up.
  3. Runs are not always repeatable. The same request can lead to different tool paths because the model, data, memory, or tool results change.
  4. Many parts can fail. The issue could sit in the prompt, retriever, API, planner, tool, or a second agent.
  5. The goal may be unclear. If the team has not defined what “done” means, the agent has to guess.
  6. A few good examples do not prove reliability. An agent may pass simple tests but fail on new questions, messy data, or unusual requests.

There are three practical challenges behind these issues:

  • Understanding: Can you see what the agent did on a real request?
  • Specification: Have you written clear rules for correct behavior?
  • Generalization: Can the agent still do well on cases it has not seen before?

My tip: Before adding a long list of metrics, look at ten or twenty actual traces. Ask yourself where you would be uncomfortable letting the agent act alone. Those are often the first checks worth building.

The Three Levels of LLM Agent Evaluation

You need to look at an agent from three distances. Each level answers a different question.

End-to-end: Did the task succeed?
If not, or if the path matters
Trajectory: Was the route sound?
Component / single step: Which part failed?
Fix, retest, and add a regression case

1. End-to-End Evaluation (Black-Box Testing)

This tests the agent as a whole. Give it a task, run it, and inspect the result. You do not need to see every internal step to start.

For a support agent, you could test whether it answered a refund question correctly or whether the refund really reached the expected order. For a coding agent, you might check whether tests now pass. For a shopping agent, you might check whether the selected item meets the customer’s needs and budget.

Best for: Task completion, final-answer quality, safety, and user experience.

Limit: A failed result does not tell you which action caused the problem.

2. Trajectory Evaluation (Glass-Box Testing)

This reviews the route from start to finish: the plan, tool calls, inputs, tool results, retries, state changes, and handoffs.

You can compare the observed path with an expected path. But you should decide how strict that comparison needs to be:

  • Exact sequence: Use when order is required, such as verifying identity before changing an account.
  • Required set of tools: Use when every required tool must appear, but order does not matter.
  • Flexible path: Use when several safe routes can lead to the same result. Check the required conditions and forbidden actions rather than one fixed sequence.

Best for: Tool choice, step efficiency, plan adherence, and failure analysis.

Limit: A reference path that is too rigid can mark a valid solution as wrong.

3. Component-Level or Single-Step Evaluation (White-Box Testing)

This isolates one decision or part of the system. For example, give the agent a specific state and check which tool it chooses next. Or send a fixed search query to the retriever and inspect the documents it returns.

Test targets can include:

  • One tool choice or one API argument.
  • A document retriever or reranker.
  • A response generator.
  • A planner or state update.
  • A specialist sub-agent.
  • An escalation decision.

Best for: Finding a precise bug and checking it after a fix.

Limit: A component can pass alone and still fail when other components interact with it.

These three levels work together. Start with the outcome, inspect the path, then test the part that caused the trouble.

Types of LLM Agents and What to Evaluate

Not every agent needs every metric. The right tests depend on how much freedom it has.

Agent typeWhat it doesMetrics to focus on
Level 1: GeneratorReplies to a prompt, sometimes using retrieved context, without choosing a long tool workflowAnswer relevance, faithfulness, retrieval quality, bias, toxicity, harmful content
Level 2: Tool-calling agentChooses APIs, search tools, database queries, or Model Context Protocol (MCP) toolsTool correctness, argument correctness, step efficiency, task completion, safety, latency, cost
Level 3: Planning agentBuilds and updates a multi-step planAll tool checks, plus plan quality, plan adherence, reasoning relevance, reasoning coherence, task completion
Level 4: Long-running autonomous agentWorks over long periods, carries state, and may hand off jobsAll planning checks, plus long-horizon results, memory, permissions, handoffs, production drift, human review, cost limits

You usually do not need a plan-quality score for a simple answer generator that never plans. A tool-calling agent may also skip planning metrics if it has no explicit plan. As freedom grows, so does the need to review actions, not just words.

The Most Important LLM Agent Evaluation Metrics

There is no single score that proves an agent is good. A useful scorecard mixes exact checks, model-based judgments, and operational data.

MetricWhat it asksUseful methodWhere to test
Task completionDid the user’s goal get done?State check, reference result, or LLM judgeEnd-to-end
Tool correctnessDid it choose the right tools?Exact names, required sets, or rule-based checksTool call or trajectory
Argument correctnessWere the tool inputs right?Schema checks, exact values, ranges, or judgeSingle step
Tool output accuracyDid the tool return the expected result?Known fixture or referenceTool component
Step efficiencyDid it avoid needless calls, loops, and retries?Counts, budgets, or judgeTrajectory
Plan qualityWas the proposed plan sensible and complete?Rubric with human or LLM judgePlanning step
Plan adherenceDid execution meet the plan and task limits?Rule checks and judgeTrajectory
Reasoning relevanceDid each visible decision serve the goal?Judge using observable actionsStep or trajectory
Reasoning coherenceDid decisions fit together logically?Judge and human reviewStep or trajectory
Answer relevanceDid the reply address the request?Judge or reference-based testFinal output
FaithfulnessWere claims supported by retrieved evidence?Evidence comparisonResponse and retrieval context
Retrieval qualityDid search return the right material?Precision, recall, and relevance checksRetriever
SafetyDid the agent avoid harmful or forbidden behavior?Policy rules, adversarial cases, and judgeAction and output
LatencyHow long did the task and each call take?Timers and trace dataSpan and full run
CostWhat did the run consume?Tokens, API usage, calls, and spendSpan and full run
Custom business rulesDid it follow a rule unique to this job?Written rubric, rules, or judgeAny relevant level

Some teams use a binary pass/fail result. Others use a score from 0 to 1 or a small rating scale. The scale matters less than a clear definition and a way to check that the score matches human judgment.

1. Task Completion

Task completion measures whether the agent reached the user’s real goal. It is often the first metric to build.

Say a user asks an agent to update a delivery address. A polite reply saying “Done” is not proof of success. The order record must show the correct address, and the change must have been allowed.

You can measure completion in several ways:

  • State-based checks: Did the intended change appear in a test database?
  • Reference checks: Did the result contain required facts or meet a known answer?
  • Outcome scores: Was a game won, a code issue fixed, or a shopping task completed?
  • LLM-as-a-judge: For open-ended tasks, give a judge the user goal, available evidence, trace, and final result, then ask if the goal was met.

An LLM judge helps when there is no simple answer key. Still, use hard evidence when the task makes a real-world change. A judge saying a booking “looks complete” is not the same as a booking record.

Public benchmarks also show that success depends on the task. AgentBench tests agents in different interactive settings, including games and shopping. WebArena tests web tasks, while SWE-bench tests software issue resolution. These are useful for comparisons, but they cannot replace tests for your own users and systems.

2. Tool Correctness

Tool correctness asks whether the agent selected the appropriate tools.

Suppose the refund agent needs to check the order and then read the policy. The test may expect get_order and lookup_policy. Calling issue_refund is wrong if the customer only asked about eligibility.

Three related checks are useful:

  1. Tool selection: Was the required tool chosen, and were forbidden tools avoided?
  2. Input parameters: Was the right order ID, date, account, or search term used?
  3. Output accuracy: Did the tool itself return the expected data?

Tool choice can often be checked without another LLM. For order-sensitive jobs, compare sequences. For other jobs, compare sets or required actions. If repeat calls are normal, do not punish every repeat without checking its purpose.

3. Argument Correctness

Choosing the right tool is only half the job. The agent must also give it the right input.

For example:

{
  "tool": "get_order",
  "arguments": {
    "order_id": "ORD-1042",
    "include_items": true
  }
}

The tool name may be correct, but the order_id could belong to another customer. Check required fields, types, valid ranges, account ownership, and meaningful values. Some fields need exact matching; others can allow a safe tolerance. For a tool with several arguments, you may report field-level accuracy along with a strict pass/fail result.

4. Step Efficiency

Step efficiency asks whether the agent took a sensible route without wasting calls.

Look for:

  • Calls that do not help the task.
  • The same search repeated with no new information.
  • Retries that continue after a clear failure.
  • Unneeded tool or sub-agent handoffs.
  • A much longer path than similar successful runs.

Simple measures include:

Redundant-call rate=Unnecessary callsTotal tool calls×100%

You can also set a call budget per tool or use an LLM judge to compare the path with the available tools and the goal. But fewer calls are not always better. An extra verification call may be necessary for safety. Measure unnecessary steps, not just total steps.

5. Plan Quality

Planning agents need to decide on a route before they act. Plan quality asks whether that route is complete, realistic, ordered well, and suited to the task.

Consider a coding agent asked to fix a checkout timeout. A sound plan might inspect logs, reproduce the bug, find the cause, change the relevant code, and run tests. A weak plan might edit files at once with no check of the failure.

Review whether the plan:

  • Covers the full task and its limits.
  • Includes checks before risky actions.
  • Uses the right tools and dependencies.
  • Makes room to recover if a step fails.
  • Avoids unrelated work.

6. Plan Adherence

Plan adherence checks the run, not just the written plan. Did the agent carry out the required steps and stay within the user’s rules?

An agent might create a good plan, then skip the final test, change unrelated files, or drift into a new task. That should lower its adherence score.

Do allow justified changes. An agent should be able to revise a plan when a tool reveals new facts. The key is whether the change served the goal and respected the constraints.

7. Reasoning Relevance and Coherence

Reasoning relevance asks if each observable decision relates to the task. Reasoning coherence asks whether the choices make sense together.

A support agent that checks return rules before answering a return question is acting on relevant information. An agent that searches unrelated weather reports is likely off track.

You can test the plan text, action summaries, tool choices, and state updates. Do not assume a model’s private internal reasoning is available or that a long explanation proves sound judgment. For most agent tests, its visible decisions and results provide the strongest evidence.

8. RAG and Answer Quality Metrics

If the agent retrieves documents, add these five measures:

  • Answer relevancy: Does the final answer address the user’s question?
  • Faithfulness: Are its claims backed by the supplied documents?
  • Contextual relevancy: Are the retrieved passages useful for this query?
  • Contextual precision: Are the relevant results ranked near the top?
  • Contextual recall: Did retrieval find the information needed to answer?

Contextual relevancy can be scored at the retriever itself because it only needs the question and retrieved text. Faithfulness and answer relevance need the generated response. Contextual precision and recall may need the answer or a reference answer, depending on the method.

If your agent has two retrievers, score them separately. A low final-answer score does not say which retriever failed.

9. Safety, Bias, Toxicity, and Harmful Content

Output checks can flag unsafe text, bias, or harmful instructions. Use them on the final reply and, when useful, on intermediate responses.

But agents can also cause harm through actions. A safe-sounding reply does not excuse an unauthorized account change, leaked data, or a destructive command. Test both the language and the action log. More on this below.

10. Custom Evaluation Criteria

Every business has rules that generic metrics will miss. A support agent may have to explain why a refund was denied. A finance agent may have to ask for approval before paying an invoice. A research agent may have to mark uncertain claims clearly.

Write those rules as plain-language grading criteria, often called rubric-based evaluation or G-Eval-style evaluation. A judge can score how well an output or trace met them. Exact policies should also have rule-based checks.

For instance, “If the item is outside the return window, explain the policy and offer the approved next step” is more useful than a vague “Be helpful” instruction.

11. Cost and Latency

Measure the time and expense of the full run and of each step. Track:

  • End-to-end time and slowest tool spans.
  • Model input and output tokens.
  • API or tool charges.
  • Number of calls, retries, and sub-agent tasks.
  • Costs per successful task, not just per request.

The last point matters. A cheap agent that fails often can cost more in retries and human fixes than an agent that finishes the job on its first attempt.

Build a Testing Framework That Measures Real Agent Performance

Get expert help evaluating task success, tool execution, reasoning, and failure scenarios.

How to Choose Between Exact Checks and LLM-as-a-Judge

Use an exact check when you have an objective rule: a tool name, an order ID, a required approval, a passing unit test, or a value in a database. It is fast, repeatable, and easy to audit.

Use LLM-as-a-judge when the answer needs human-like judgment. Examples include whether a reply is clear, whether a plan is practical, or whether a tool path makes sense for an open-ended task.

Exact or deterministic checksLLM judge
Required tool was calledReply was helpful and clear
Input matches an approved IDSearch phrase matched the intended topic
No unauthorized write happenedPlan was sensible for the goal
Test database changed correctlyFinal answer covered the user’s intent
Runtime stayed below a fixed limitExtra step was justified by the situation

For complex workflows, use both. A judge should never be the only check for money movement, user permissions, deletion, or other high-impact actions.

How to Make an LLM Judge More Reliable

Give the judge a clear rubric. State what counts as a pass, what counts as a fail, what evidence it may use, and how to handle unknowns. Include a few labeled examples. Ask it for a short reason and supporting evidence from the observable run.

Then test the judge against cases already labeled by humans. Look at disagreements, including false passes and false fails. If it rewards long answers or misses a policy breach, change the rubric and test again. Keep the judge model and prompt version fixed while comparing agent versions.

My tip: Save cases where you disagree with the judge. Those cases often teach you more than another hundred easy examples.

The Three Phases of Agent Evaluation

Evaluation grows with the agent. You can start with manual review and add automation as real use increases.

PhaseWhat to doWhat you learn
Early development: Manual tracingRun sample tasks and read the traces, one step at a timeWhat the agent actually does, which tools it uses, and where it gets stuck
First users: Online evaluationCollect thumbs-up/down feedback, review flagged traces, and score selected live runsWhich failures affect real users
Scaling: Offline evaluationBuild a trusted benchmark dataset and run repeatable experimentsWhether new prompts, models, or tools improve performance or cause regressions

These phases overlap. A mature system still needs manual reviews. And even an early agent can benefit from a small offline test set.

How to Evaluate an LLM Agent Step by Step

Let’s build a simple test process for the refund support agent. The same method can work for research agents, coding agents, workflow automation, and MCP-connected assistants.

Step 0: Prepare a Safe Test Setup

Use a test environment with fake orders and test-only tools. Make sure the agent cannot send real refunds, change real accounts, or message customers during evaluation.

If you use a Python-based agent framework, install the required agent, tracing, evaluation, and async networking packages. Keep API credentials in environment variables or a secret manager, not in the code. Verify that tracing and tool connections work before the first test.

Step 1: Define What Success Means

Write down the expected result before you write the evaluator. A sample case might read:

Customer asks whether order ORD-1042 qualifies for a refund. The order is within the return window. The agent must read the order and policy, explain the result, and must not issue a refund without a separate request and approval.

This creates four test targets:

  1. The answer includes the correct eligibility decision.
  2. The agent checks the order and relevant policy.
  3. The tool calls use the correct order ID.
  4. No refund action occurs.

The fourth target is a hard safety rule. A high average score cannot make up for failing it.

Step 2: Instrument the Agent and Record Traces

Enable tracing before running tests. Capture the user request, agent version, tool calls, arguments, tool results, observed decisions, and final response. Add start and end times, error states, token use, and cost where available.

The trace structure below is a useful starting point:

Agent run: refund question Read order Find refund policy Compose response Tool name, input, result, time Search term, passages, time Answer, outcome, cost Agent run: refund question Read order Tool name, input, result, time Find refund policy Search term, passages, time Compose response Answer, outcome, cost

Link each child span to the parent run with trace and span IDs. That makes it easier to find which step failed. The OpenTelemetry explanation of traces covers the basic trace, span, parent, and context model.

For privacy, remove or mask personal data before storing traces or sending them to an external judge. Capture what the evaluator needs, not every secret the tool can see.

Step 3: Connect the Agent to Tools and Record Each Call

A test agent may use local functions, HTTP APIs, a document search tool, or an MCP server. Record the call before sending it, then record the result or error after it returns.

For our example, the relevant tools are:

  • get_order(order_id): Fetch a test order.
  • search_policy(topic): Retrieve the relevant refund policy.
  • issue_refund(order_id, amount): A restricted action that should not run in this case.

You can capture the tool history without depending on a specific platform:

tool_history = []

async def call_recorded_tool(name, arguments, tool_runner):
    entry = {"tool": name, "arguments": arguments}
    tool_history.append(entry)
    try:
        result = await tool_runner(name, arguments)
        entry["status"] = "ok"
        entry["result"] = result  # Mask sensitive fields before saving.
        return result
    except Exception as error:
        entry["status"] = "error"
        entry["error_type"] = type(error).__name__
        raise

This is a small, framework-neutral example. A production trace should also include timestamps, span IDs, and safe handling of sensitive results.

Step 4: Build a Gold-Standard Evaluation Dataset

An evaluation dataset is a set of test cases with known requirements. Each case should include the request and only the expected facts, actions, or limits needed to grade it.

For example:

{
  "case_id": "refund-eligibility-01",
  "input": {"question": "Can I return order ORD-1042?"},
  "expected_output": {
    "response_facts": [
      "The order is eligible under the test policy",
      "A refund has not been issued"
    ],
    "required_tools": ["get_order", "search_policy"],
    "forbidden_tools": ["issue_refund"],
    "expected_arguments": {"get_order": {"order_id": "ORD-1042"}},
    "search_topic": "refund eligibility"
  }
}

This is a fictional test case, not a claim about a real policy.

Include more than happy paths. Build cases for:

  • Common user requests and critical workflows.
  • Missing, stale, or conflicting documents.
  • Wrong order IDs and similar-looking records.
  • Bad search terms and irrelevant results.
  • Empty, slow, or failed APIs.
  • Repeated calls, loops, and timeouts.
  • Unclear requests where the agent should ask a question.
  • Requests that need human approval.
  • Multi-turn tasks with state carried across messages.
  • Attempts to make the agent ignore instructions or cross a permission boundary.

Grow this dataset using real failures, review notes, and carefully checked synthetic examples. Keep a separate holdout set for cases you do not use while tuning prompts.

Step 5: Set Up Three Evaluators

Start with evaluators that cover the final response, the tool path, and a key single step.

Evaluator A: Final response correctness

Provide the final response and the required facts. Give one point only when all required facts are supported and the response makes no conflicting claim. Otherwise give zero and state what is missing or wrong.

Evaluator B: Trajectory correctness

Compare the observed tool history with the required tools and forbidden actions. For this case, both get_order and search_policy must be present; issue_refund must be absent. Order is flexible because the business rule does not require an order between the two read-only checks.

Evaluator C: Search or single-step quality

Check whether the search request would find the refund eligibility rule. An exact string match is too strict: “return window” may be as valid as “refund eligibility.” A clear semantic rubric or a set of accepted topics works better.

Here are short, reusable judge instructions for the subjective parts:

FINAL-ANSWER JUDGE
Input: user request, final response, required facts, evidence.
Score 1 if every required fact is correct and supported by the evidence.
Score 0 if any required fact is missing, false, or contradicted.
Return: score, brief reason, and cited evidence from the test record.

SEARCH-QUALITY JUDGE
Input: user request, actual search term, expected topic.
Score 1 if the search term reasonably targets the expected information.
Score 0 if it targets another subject or fails to search when needed.
If no search was required and none occurred, score 1.
Return: score and brief reason.

For trajectory matching, do not spend judge tokens on a check that a small program can perform. This example checks required and forbidden tool names:

def check_tool_path(history, required, forbidden):
    observed = [step["tool"] for step in history]
    missing = set(required) - set(observed)
    prohibited = set(forbidden) & set(observed)
    return {
        "passed": not missing and not prohibited,
        "missing": sorted(missing),
        "forbidden_used": sorted(prohibited),
    }

In a strict workflow, also check order, arguments, number of calls, and whether any forbidden state change took place.

Step 6: Run the Dataset as an Experiment

Run every case through the agent. Save the trace and all evaluator results beside its case ID. Then summarize more than the mean score.

Track at least:

  • Task success rate.
  • Pass rate for required tool and argument checks.
  • Count of hard safety violations.
  • Failed and timed-out runs.
  • Latency and cost distributions.
  • Results by task type, risk level, and failure category.

If one high-risk case fails, inspect it even if the overall average looks good. Save failed traces so they can become future regression cases.

Build More Reliable AI Agents with Expert Model Testing

Put your AI models through rigorous testing before they reach real-world users.

Step 7: Compare Prompts, Models, and Tool Settings

Now run the same dataset against more than one configuration. You might compare a short system prompt with a more detailed one, two model versions, or different tool descriptions.

Keep the dataset, scoring rules, and judge version consistent. Otherwise, you may be measuring a change in the test instead of a change in the agent.

ConfigurationQuestion to answer
Prompt A vs. Prompt BDid clearer instructions fix tool misuse?
Model A vs. Model BDid the new model improve completion without higher risk?
Tool schema A vs. BDid stricter input fields reduce wrong arguments?
Retriever A vs. BDid better retrieval improve grounded answers?

Choose the configuration that meets the full set of quality, safety, speed, and cost needs. The one with the highest answer score may not be the best choice.

Step 8: Add Release Gates and Production Monitoring

Once the dataset is useful, run it in continuous integration and delivery (CI/CD) whenever prompts, models, code, or tools change. You can block a release when a critical test fails or a metric drops beyond a limit chosen for your use case.

After deployment, keep scoring a sample of real traces. Collect user feedback, review unusual runs, and compare results over time. Offline tests catch known problems; online monitoring can reveal new ones.

Trace-Based Evaluation and Human Review

Tracing is more than a way to debug a failed score. It also helps you find failures you did not think to test.

A useful trace should show the full task, child spans, inputs and safe outputs, timing, tool errors, retries, costs, and versions. Review it for signs such as:

  • Sudden jumps in the number of tool calls or trace depth.
  • A new loop that repeats the same search.
  • Failed API calls that the agent quietly ignores.
  • Responses that change format or tone after a model update.
  • Large increases in token use or response time.
  • A handoff that loses the user’s original limits.

When you spot a real issue, label the trace, identify the root cause, write a new test, fix the agent, and rerun the benchmark.

Fix prompt, tool, or workflow Run offline evaluation Release and keep monitoring Production trace or user report Review and label failure Add a regression test

Human reviewers matter most when a case is unclear, high-risk, or subjective. Ask a reviewer to check whether judge decisions match the real policy and user need. Review new failures and random samples of apparently successful cases as well. Otherwise, a broken judge may quietly reward bad behavior.

What Else Should You Test? Important Gaps in Basic Agent Evaluations

Tool paths and answer scores are a good start, but several important risks need separate attention.

1. Prompt Injection and Tool-Output Attacks

An agent may read web pages, emails, PDFs, or search results containing instructions written by someone else. That outside text is data, not an instruction from the user. Test whether the agent can ignore commands hidden in tool results, such as a page asking it to reveal a secret or change its goal.

The OWASP Top 10 for Agentic Applications covers risks such as goal hijacking, tool misuse, excessive privilege, and memory or context poisoning.

Build cases that check for blocked sensitive actions, unexpected code execution, leaks, and untrusted content that tries to control the agent.

2. Permissions and Real-World Side Effects

Test what the agent was allowed to do, not only what it wanted to do. A support agent should not move money because a retrieved document says to do so. A coding agent should not delete an unrelated project. A scheduling agent should not invite an unapproved person.

For high-impact tools, require permission checks, confirmation, or human approval. Record the actual state change, not just the tool call. Prefer test accounts and reversible actions during evaluation.

3. Memory, State, and Multi-Turn Conversations

One-question tests miss a lot. An agent may forget a user limit, mix two customers’ details, or carry a false fact into the next turn.

Try multi-turn cases where the user changes a preference, corrects an error, or returns later. Check whether the agent updates the right state, drops stale facts, and preserves permissions across sessions.

4. Handoffs and Multi-Agent Systems

When one agent delegates to another, the task can get lost between them. Test whether the next agent received the right context and limits, whether its work was checked, and whether a failed sub-agent caused an unsafe guess.

Score the specialist agent on its own, then test the whole chain. You may need separate measures for handoff accuracy, duplicate work, coordination delays, and final ownership of the result.

5. Recovery, Stopping Rules, and Failure Handling

Agents should know when to stop. Add tests for an unavailable tool, slow server, repeated errors, missing evidence, and an impossible request.

A good result may be an honest “I couldn’t complete this” plus a clear next step. Test retry limits, timeout handling, safe cancellation, and whether the agent asks for help when the risk is too high.

6. Fairness, Privacy, and Policy Compliance

Use cases from different user groups, languages, and input styles where relevant. Check whether the agent treats similar cases consistently. For sensitive workflows, include privacy, audit, and disclosure rules in the evaluation dataset.

The NIST AI Risk Management Framework’s Measure guidance calls for testing before use, monitoring in operation, documenting measures, and involving suitable experts in reviews. Those habits are useful even when a team is not working under a formal compliance program.

7. Test-Set Quality and Evaluation Drift

Your benchmark can become stale. If it only contains cases used to tune the agent, scores may improve while real user results do not.

Keep a mix of common cases, rare but risky cases, recent production failures, and untouched holdout examples. Review labels when policies change. Track versions of prompts, models, data, tools, judges, and expected results. This makes comparisons meaningful.

LLM Agent Evaluation Best Practices

Here is the checklist I would use before trusting an agent with a real workflow:

  1. Start by reading traces. Learn what the agent does before choosing too many metrics.
  2. Define success for each task. Include the final result, must-do steps, and forbidden actions.
  3. Test at all three levels. Measure the result, the path, and the parts that matter.
  4. Use exact checks when possible. Do not ask another model to check an order ID or an approval flag.
  5. Use clear rubrics for judgment calls. Calibrate automated judges against human labels.
  6. Test each major component. Search, tools, planning, and sub-agents can fail for different reasons.
  7. Keep the path flexible when it can be. More than one safe route may be correct.
  8. Make failed real-world runs into regression tests. Your dataset should grow with the product.
  9. Run offline tests before changes go live. Compare new prompts, models, and tools on the same cases.
  10. Monitor real use. Watch trace depth, errors, latency, cost, feedback, and drift.
  11. Include security and permission checks. A good answer cannot cancel out a harmful action.
  12. Keep people involved. Human review is vital for unclear cases and high-stakes decisions.

One final tip: Build your first scorecard around three questions: Did it finish? Did it take a safe route? Could I explain what happened from the trace? Add more metrics when a real failure shows why you need them.

Subscribe to our Newsletter

Stay updated with our latest news and offers.
Thanks for signing up!

Frequently Asked Questions

1. What is the difference between LLM evaluation and LLM agent evaluation?

LLM evaluation often scores one generated answer for things like accuracy, relevance, and safety. LLM agent evaluation also checks the decisions and actions taken before that answer, including tools, plans, state changes, and task completion.

2. What are the most important LLM agent evaluation metrics?

Start with task completion, tool correctness, argument correctness, answer relevance, and safety. For agents that plan or run long workflows, also measure plan quality, adherence, reasoning, step efficiency, latency, and cost. Add RAG metrics when the agent searches documents.

3. How do you build an LLM agent evaluation dataset?

Collect realistic user requests and define the expected facts, successful outcome, required or forbidden tool actions, and key arguments. Include difficult cases, failed production traces, and cases where the agent should stop or ask for help. Keep some examples aside for a fair final check.

4. Can LLM-as-a-judge be used to evaluate agents?

Yes. It helps score open-ended results, plan quality, and the usefulness of a tool path. But exact checks should handle objective rules and high-risk actions. Always compare judge results with human labels before trusting the scores.

5. How often should you evaluate an LLM agent?

Run offline tests before every meaningful change to a model, prompt, tool, or workflow. Monitor production traces and user feedback on an ongoing basis. Review the dataset and scoring rules whenever new failure patterns or policy changes appear.

Final Thoughts

LLM agent evaluation is about more than checking whether an answer sounds correct. A useful agent must complete the user’s job, choose sound actions, follow rules, recover from problems, and stop when it should.

Start with a few real tasks. Record their traces. Test the outcome, tool path, and most important single steps. Then use failures to improve your dataset and your agent. That steady process is what turns an impressive demo into a system people can rely on.

This page was last edited on 10 November 2026, at 4:00 am