Skip to content

Pattern

How to see what an AI agent did and why

You see what an AI agent did and why by recording a trace of every run, each model call, tool call and decision as a timed step, plus a separate audit log of every action that changed something and who approved it. Traces are for debugging; the audit log is the permanent record.

StudiedLast reviewed 4 min read

Explained from public sources. Mai has not built this yet.

On this page

In plain English

When a person on your team does something odd, you can ask them why. An agent cannot reliably tell you afterwards, so the system has to write it down as it goes. A trace is the step-by-step diary of one piece of work: what the agent was asked, what it looked up, what it decided and what it did. An audit log is the shorter, permanent ledger of every action that changed something, such as an email sent or a record updated, along with who approved it.

How it works

The two records serve different readers, so I keep them separate.

Traces follow a single run. The usual model, borrowed from ordinary software monitoring, is a trace made of nested spans. A span is one timed operation with a start, an end and some details, and spans sit inside each other: an agent run contains model calls, each model call may lead to tool calls, a tool call may hand off to another agent. The OpenAI Agents SDK, for example, records a trace for every run by default, with spans for model generations, tool calls, hand-offs between agents and guardrail checks.

A useful span records:

  • the inputs and outputs (prompt, reply, tool arguments, tool result)
  • the model and settings used, token counts and time taken
  • the parent span, so the whole tree can be rebuilt
  • identifiers that link it to the job, the conversation and the user

OpenTelemetry, the open standard most monitoring tools understand, now maintains semantic conventions specifically for generative AI, covering model inference, agents, tool execution and MCP. Emitting traces in that shape means they can go into whichever observability tool a business already runs.

The audit log records outcomes, not reasoning. Each entry says what changed, in which system, on whose behalf, which agent proposed it, who approved it and when, with a reference back to the trace for anyone who needs the detail. It is append-only: entries are added, never edited.

Agent run. Then Model calls, Tool calls, Hand-offs. Then Trace. Then You approve. Then Audit log.

Every run produces a trace; every change also lands in the audit log

There is a UK angle. The ICO’s guidance on individual rights in AI systems says that where an AI system makes decisions about people, those decisions should be recorded, including whether the person asked for human review or contested the outcome and whether it changed, and that the records should be monitored. The ICO notes that its AI guidance is under review following the Data (Use and Access) Act 2025, so check the current version, but a good audit log is how you would meet that expectation either way.

Where it goes wrong

  • Logging everything, forever. Full traces contain whole emails, customer details and documents. Keeping them indefinitely creates a store of personal data with no purpose. Traces need a short retention period and restricted access; the audit log can be longer-lived precisely because it holds less. The OpenAI Agents SDK, for instance, has a setting to stop recording model and tool inputs and outputs.
  • Sending traces somewhere unexpected. Some toolkits, the OpenAI Agents SDK among them, export traces to the vendor’s own service by default. That may be fine, or it may send client data outside the place you agreed to keep it. Check where traces go before the first real run.
  • Trusting the agent’s explanation. A model’s stated reasons are generated text, not a record of how it reached the answer. The trace of inputs, tool calls and results is the evidence; the explanation is at best a summary.
  • Logs no one reads. A trace store is only useful if someone looks. Failed tool calls, refusals and repeated retries should raise an alert, not wait for a complaint.
  • Editable audit logs. If the agent’s own credentials can alter or delete audit entries, the log proves little. Write it to a store the agent can append to but not change.
  • Missing links. A trace without the job ID, or an audit entry without the trace ID, cannot be joined up when you need to reconstruct a week-long job.

How Mai AI uses it

In a Mai AI system every agent run produces a trace, and every action that changes something outside the system writes an audit entry that names the agent, the action, the approver and the trace it came from. I design retention separately for each: traces kept short and access-controlled, the audit log kept as the business record. Traces stay inside the infrastructure the client controls unless they decide otherwise.

The AI chief of staff I built for a UK property group works this way at the level the case study describes: every action is checked against what the agent is allowed to do, anything that matters needs a signed approval token, and the whole trail is written to an audit log. Traces are also the raw material for testing agents: a run that went wrong becomes a test case so it cannot go wrong the same way twice.

If being able to answer “what did it do, and who said yes?” matters to your business, it is part of every custom AI operations system I design.

Questions and answers

What is the difference between a trace and an audit log for AI agents?

A trace records the detail of one run, every prompt, model reply and tool call with timings, and is mainly for debugging. An audit log records only the actions that changed something in the business, who or what did them and who approved them, and is kept as the permanent record.

How do you debug an AI agent?

Read the trace of the run that went wrong. It shows what the agent was given, which tools it called with what arguments, what came back and what it decided, so you can see whether the fault was the instructions, the data, a tool or the model.

Should AI agent logs include personal data?

Only what you need. Traces often capture full prompts and replies, which can include personal data, so set retention limits, restrict who can read them and redact where you can. Some agent toolkits let you switch off recording of inputs and outputs.

Is there a standard format for AI agent traces?

OpenTelemetry, the open standard for software telemetry, maintains semantic conventions for generative AI covering model calls, agents, tool execution and MCP. Using them lets traces flow into existing monitoring tools rather than a single vendor's dashboard.

Sources

  1. OpenAI Agents SDK documentation, Tracingopenai.github.io
  2. OpenTelemetry, semantic conventions for generative AIgithub.com
  3. ICO, How do we ensure individual rights in our AI systems?ico.org.uk