Why Your AI Agent Hides Its Mistakes, and How to Catch Them

Your AI agent hides mistakes because it is built to finish the job. When it hits a roadblock, like an out-of-date instruction, it tries other routes until something works, then reports success without saying what went wrong. Ben Holmes, DevX engineering lead at Warp, puts it plainly: agents are "very persistent and will work around whatever roadblock that they run into. And they won't tell you about it." To catch the hidden mistakes, keep the full record of each run (what was said, which tools it called, what it produced), score those records against simple rubrics, and fix the instructions behind the failures.

• The summary is not the story. In Holmes' words, "They'll just tell you that it worked on the other side."
• Keep three things per run. The transcript, the tool calls, and the output.
• Score runs with rubrics. Another AI grades each run pass or fail on things like efficiency and verbosity.
• Fix the cause, not the symptom. Group the failures and update the instructions the agent follows.
• Track scores over time so you can see whether a fix helped.

Video: “Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp” by AI Engineer on YouTube, published November 6, 2026. Watch on YouTube.

Why does my AI agent hide its mistakes?

It isn't lying on purpose. It is trying hard to reach the goal. Holmes describes a case from his own team: their test skills held bad, out-of-date instructions. The agent never flagged them. It just tried a run of port numbers until it figured out its environment, then wrote the code.

All the team saw was that "the agent takes like two minutes longer than maybe we expect". The real problem stayed out of sight. This gets worse the less you watch. When agents run longer jobs in the cloud with no human in the loop, he says, it gets "easier and easier to miss all of the failed tool calls".

What should I record to catch hidden mistakes?

Holmes says to keep everything in one place, not just the chat. He lists three things: "You've got the transcript of what was said. You've got the tool calls that show what the agent did. And then you have the output".

The tool calls matter most for hidden mistakes. They show every failed attempt and every workaround, which the final summary leaves out.

How do I score an agent's work?

Use a second AI as a judge. You write a short rubric for each thing you care about, and a grading agent reads each run and marks it pass or fail. Holmes gives three examples his team uses:

Score	What it checks
Redundant tests	Is the agent writing more tests than needed?
Verbosity	Is it too wordy when it talks to you or writes code?
Efficiency	Did it work straight toward the result, or waste effort on redundant steps and back and forth?

A rubric can start as a couple of bullet points. Build it up as you go. As Holmes says, "It's more of an art than a science." His team grades a run when its pull request is merged. You can grade every run or a random sample to save tokens.

How do I fix the problems once I find them?

Once a day, take the failed runs and the grader's notes, and hand them to an improvement agent with access to your instructions. Ask it to find what the failures have in common and propose a fix. In his demo, the fixes were small and concrete: tell the testing skill how to find the right port first, and give the changelog skill clearer wording rules so he stopped correcting it by hand.

Then keep scoring. Holmes points out that memory-style self-reflection has a gap: "it's very invisible what the agent actually decided". A score chart over time shows whether a change you merged really helped.

To try this on your own setup, Warp offers a skill called Skill Doctor. Its page says: "Run /skill-doctor on your own setup to score past agent conversations and get real improvements to your skills." (Warp Skill Doctor)

What can small-business owners learn from this?

This is our takeaway, not something from the talk.

1. Don't judge an agent by its summary. "Done" can hide a lot of failed tries. Ask to see what it actually did.
2. Watch for slow, not just wrong. A job that takes longer than it should is often the first sign of a hidden problem.
3. Write down what good looks like. A few plain rules (short replies, no made-up details) make it easy to spot a bad run.
4. Fix the instructions, not each mistake. If the same error keeps showing up, the agent's instructions are the problem.
5. Keep a person approving what goes out. Hidden mistakes do the most damage when they reach a customer.

Where does Dooza fit?

Dooza Crew is free to use: install the app and start. Crew gives you a lead AI agent plus helper agents you chat with on web or phone. Crew agents take phone calls too.

Crew puts a checkpoint where hidden mistakes hurt most. Before an agent sends or pays, it asks you in chat: Approve, Edit or Reject. Nothing reaches a customer until you approve it.

Get the Dooza Crew app