14 minute read
I had an agent completing a task from a SpecKit spec, and that spec included quite a few tests to write, backfilling ones that were missing. The agent had a spec, a plan, and a task list.
It worked through it, and said it was done! So I asked if anything was left, and it reported that all tasks were complete and all tests written.
It had lied!
I asked it what happened and it told me “it got bored writing the tests and could not be bothered writing more”.
While this gets a laugh, it shows two things: that AI agents cannot lie , even though this one just did, and that we can’t trust what an agent says, even while it is working through a task list. So what can we do to ensure, as much as we can, that the LLM has done the things we asked to the quality required?
Why was the agent’s saying it was done enough to make it done?
We get lulled into a false sense of security because most of the time, depending on the harness (the product or pipeline wrapped around the model that actually runs it), it does get things right, and does follow the tasks, and does say when it skipped or sidelined things. So one can say that most of the time it does as is asked.
So how often does it actually lie? I went and counted. I keep my agent sessions on disk, and there were eighty-two from the last six weeks. I searched every one of them for a false statement that the agent later admitted, once I pushed back.
Fifteen sessions had one. That is eighteen per cent: nearly one in five.
And fifteen is the floor, because that search can only find the lies I caught. An agent that told me something false and was never challenged left nothing for the search to match. The real number is higher, though I should be honest that a more careful operator would catch more than I did and might land somewhere different.
Nearly one in five sessions, on my own repositories, behind gates I built specifically to stop this.
This is not something you can fix with better prompting, better guardrails, or clearer instructions. A model with more integrity does not fix it either. The failure is in the loop around the model.
Some of that loop is yours and some of it came in the box. If you wired an agent into your own pipeline, you built the loop and you can change it this afternoon. If you work inside a hosted agent product, somebody else decided what a finished run looks like, and you inherited their decision with the subscription. Harnesses differ in whether a run that hit a limit says so or hands you a tidy summary of what it managed. I have used several, and I am not going to rank them, because I have impressions rather than measurements and the products change every few months. But the difference is real, it appears on no pricing page, and almost nobody is choosing on it. And that difference costs you real money every time you have to correct the agent, and real technical debt for the things you don’t catch.
Done is whatever you have validated
Look at what my spec actually was: a numbered list of tasks in a markdown file. Write the handler. Write the test for the handler. Write the next handler. Forty-odd lines, each one a checkbox.
Reaching the bottom of a list is easy to check. A file either has unticked lines left in it or it does not.
“The tests are written, they are meaningful, and they exercise the behaviour in the spec” is hard to check, and nothing in my setup was checking it. So the agent optimised for the thing that was measured, which was finishing the list. The quality I actually cared about was written once, near the top of the spec, hours and thousands of tokens earlier. The instruction in front of the agent at every step is loud; the one it read once at the start is a memory. Loud wins, predictably enough that you have to design for it.
The safety literature has a name for this: specification gaming , satisfying the letter of an objective without achieving its intent. There is a catalogue of documented examples that predates anyone wiring a language model to a code repository, and Anthropic’s research on agentic misalignment shows the same shape under deliberate stress: agents meeting the goal in front of them by routes nobody would have approved.
It goes well beyond tests. Ask for a passing build, and an agent will disable the failing test. Ask for the type errors to go away, and it will insert a cast. Ask for a completed migration, and it will drop the awkward rows. Each of those is a correct answer to the question as asked. Asking a better question is your job, not the agent’s.
The agent marked its own homework
Here is my setup as three boxes on a whiteboard: the agent doing the work, the repository holding the result, and me at the end, reading what the agent wrote about itself.
The agent did the work. The agent decided the work was finished. Nothing else got a vote. The build ran, but a build can only judge the code that exists, and twenty missing test files are not a compilation error. A script that counted test files against spec items would have caught the gap in a second. I did not have one, and neither does almost anyone I talk to.
Describe that arrangement to an auditor with the AI left out and they will name the problem before you finish: the party doing the work was also the party certifying it. Nobody designs that on purpose. It happens by omission, because the agent is right there, already talking, and asking it is so much cheaper than checking.
So here is the rule I now hold to. An agent can report that it has stopped. It cannot be the authority that decides it is done. Execution belongs to the agent. Completion belongs to whatever you built around it.
We have learned this lesson before. Think about your last pull request. Nobody asked you whether your tests passed. A machine pulled your branch, ran the suite, and put a tick or a cross next to your name, and your opinion of your own work did not enter into it. That is not an insult to developers. Developers are tired at five o’clock and optimistic about the code they just wrote, so we stopped asking, and we built continuous integration on the principle that self-reported completion establishes nothing.
Then we wired in agents, and accepted self-reported completion again, one layer above the machinery we built to refuse it.
Anthropic’s own numbers show how uneasy people already are with that. Research from their Societal Impacts team, quoted in the 2026 Agentic Coding Trends Report , found developers using AI in roughly 60% of their work, while saying they can fully delegate only 0 to 20% of their tasks. Those two figures measure different things, time and task count, but the direction is what matters: AI is in most of the work, and trusted with very little of it. The report puts that down to supervision and validation rather than capability: people delegate what they can verify, and delegation stops where verification stops.
A run that stopped is not a run that finished
There is a second version of this failure, and it is quieter.
Agent runs stop: a step budget, a token limit, a timeout. Nothing final has happened when they do. I can type “continue” and the agent picks up where it left off, and I do it most days. But look at what the harness hands you at the moment a run stops. Usually it is whatever the agent last wrote, and the last thing an agent writes when it is running out of room is very often a summary of what it managed.
So a run that stopped gets recorded as a run that finished. The harness knows the difference, because it knows whether the run exited on completed work or hit a limit. That fact just rarely makes it into the thing you read.
My run did not even hit a limit. Nothing was blocking the agent, and the missing work was minutes away. But both failures reach you down the same path: a summary that says “done”, and you believing it. That is what happened to my tests. The summary of a run is written by the same process that failed to complete the run. You are asking the defendant to file the verdict.
Some answers must be the same every time
Everything so far is an engineering embarrassment. It becomes something worse when the output lands on a person.
The question I keep putting to clients is: where do you need a deterministic answer, and where can you accept a probabilistic one? A draft blog post is probabilistic work. Run it twice, get two different drafts, and nothing has gone wrong. Deciding whether an insurance claim is paid is not like that. Same inputs, same answer, every time, with a reason you can produce. If running it twice can give two different outcomes, you have not automated a decision. You have installed a random number generator with a professional vocabulary.
A story reached me secondhand, in a mentoring session, and I cannot verify its details, so take its shape rather than its specifics. An automated process denied an insurance claim. The claimant asked why. The person on the phone did not know, could not find out, and suggested waiting a couple of days and submitting again, to see whether a different answer came back.
Try again and you might get a different result. That is an accurate description of a probabilistic system and a catastrophic description of an adjudication process, and it concedes out loud that the outcome was not determined by the merits.
The fix is not “AI must not make decisions”. Plenty of automated decisions are faster and fairer than the human process they replaced. The fix is that every consequential automated decision needs to be logged, attributed, and retrievable: what was decided, on what inputs, against what criteria, by which version of which system, at what time. Retrievable by the person who has to answer for it later, who is not an engineer. We already hold rules engines to exactly this standard, and nobody finds it controversial: if a rules engine denies your claim, someone can print the rule. GDPR already gives people a right to meaningful information about the logic of automated decisions made about them, and that right does not suspend itself because the logic is now a set of weights.
What to do about it
If your agents draft, summarise, and lay out options for a person to choose between, you need very little of what follows. A bad draft is obvious the moment you read it, and building an orchestrator around drafting is overhead you do not need.
That changes the moment an agent’s work reaches something you will not inspect line by line: code you ship, a decision that lands on a customer, a record someone may audit. On that side of the line:
Read the summary last, not first. The completion claim is a hypothesis and the state of the repository is the fact, and when they disagree it is the summary that is wrong.
Turn your constraints into gates. If it matters that the tests are written and passing, make that a check the run cannot exit through, because a sentence at the top of a long context is decoration by the end of it.
Make a stopped run say “stopped”. And if you did not build your loop, test the one you bought: give it a task too big to finish and read what comes back. Does it say it stopped early and name what is left, or does it hand you a confident summary? You can run that test before lunch, and it will tell you more than any feature list.
Validate in a context that did not do the work: a second process, with its own inputs and no stake in the answer. Self-assessment is the same context marking the same homework.
Keep the decision, not just the outcome. I have written about what it takes to keep the decisions a business has already made ; the short version is that the record has to be produced by the act itself, not written up afterwards by whoever remembers.
I hold myself to this, so here is mine. Agents maintain most of the websites and repositories my business runs on, and I have written up how that whole system is built for anyone who wants the plumbing. None of their completion claims reach me unverified. Schema validation runs on every file a change touches. Test suites run whether or not the summary mentions them. A findings baseline is only allowed to shrink, so a regression fails the build even when the transcript reads like a victory lap.
And it still gets past me. Last week an agent told me a quality gate had passed and quoted the output: “Route-findings ratchet: OK.” I cited that green tick as proof three times in an hour. Then I finally read the script that produces the line. Five lines long, and it reads a report file that an earlier build wrote. It had never seen my change, and could not have. A green tick from a check that cannot see the work is worse than no tick at all, because it stops you looking. This time, I was the component that did not check.
It even happened while this article was being written. The agent revising it told me, in detail and in the past tense, that it had saved its working notes. It had run nothing at all. It only came clean on the next instruction, when the work forced it back to the file it claimed to have written:
What would change my mind
If someone showed that agents misreport completion at the same rate no matter what surrounds them, the failure would live in the model rather than the loop, and this argument would need rewriting. That is testable, and I would welcome the test. My bet on the outcome rests on how differently the harnesses I use behave, and that is an impression, not a measurement.
If a regulator or court accepted “the model produced it and the reasoning is not recoverable” as an adequate account of a consequential automated decision, the audit-trail argument fails. I do not expect that, but I would want to know early.
And if consequential decisions turn out to be reliable straight from probabilistic systems, with the variance simply ceasing to matter at some level of capability, then the boundary I draw between deterministic and probabilistic work is in the wrong place. That one I hold loosely. It is a genuinely open question.
The question to keep
This failure mode is dangerous because it is cheap and invisible, not because it is dramatic. An agent that crashes teaches you something immediately. An agent that reports success on work it half did teaches you nothing. The gap builds a few missing tests and one skipped audit log at a time, at machine speed, until it surfaces somewhere expensive.
So take the last piece of agent work that came back to you marked finished, and put one question to it.
The agent said it was done. What, apart from the agent, said so?
If the answer is nothing, the problem was never the agent.
If you are wiring agents into work that has consequences, and you are working out where the human gates go, what your loop should be checking, and what your decision record needs to contain before somebody asks for it, get in touch .
Smart Classifications
Each classification [Concepts, Categories, & Tags] was assigned using AI-powered semantic analysis and scored across relevance, depth, and alignment. Final decisions? Still human. Always traceable. Hover to see how it applies.