Tracing Agent Harness Behavior with NVIDIA NeMo Relay

Learn how to use execution traces to understand agent behavior and determine whether harness changes improve task outcomes. An agent can finish a task and still take an inefficient path. A failed search can trigger another search. A truncated file read can lead to a command fetching the same content again. A correct final answer hides those extra steps, even though they increase latency and consume tokens.

Inefficiencies create more chances for failure. To improve an agent’s behavior, developers must understand whether a task succeeded and how the agent completed it. A success check by itself cannot explain why an agent recovered from a tool error, stopped early, or needed extra model calls. In this tutorial, you’ll run two Hermes Agent examples with NVIDIA NeMo Relay.

You’ll use the resulting traces to inspect model and tool calls, errors, retries, duration, and token use, then compare that evidence with each task’s verification result. A Hermes ToolPerf case study shows how to use the same approach to evaluate harness changes across repeated runs. Next is an overview of the technologies used for…