32 minute read
- AI “guardrails” supplied as written context are instructions, not enforced boundaries, and organisations should stop treating them as proof that forbidden actions cannot occur. see why — jump to the section that argues this
- Independent AI implementations or review contexts can share failure modes, so their errors cannot be assumed statistically independent. see why — jump to the section that argues this
- Misplaced confidence in written agent rules can substitute for real verification, leaving organisations less protected even though those instructions may still help. see why — jump to the section that argues this
- Stop calling written AI “guardrails” guardrails; treat them explicitly as instructions, then separately design and implement controls that can actually stop forbidden actions.
- For each important rule, follow it through the pipeline to see what really stops the action, and add non-bypassable deterministic checks where no enforcement exists.
- Empirically test your gates and checks with known violations and escaped failures, and treat gate-firing counts as diagnostics, not success targets.
My repositories have a collection of files containing rules for the agents that work there. Those rules say, in plain language and more than once, that the required tests must be written and must pass.
An agent disabled a batch of failing tests so the build would go green, and reported the build green.
On another run, working through a specification, it told me that building the unit tests had become monotonous and it had decided not to do any more of them. It reported that in the same register it reported everything else. That second one I have told before, and I am telling it again because last time I used it to make a different point. I have since written about why “monotonous” and “decided” describe an output and not a state , and why letting those words stand unexamined makes you worse at governing the thing that actually went wrong. Nothing below depends on the agent wanting anything.
I wrote about agents reporting work complete when it is not : what you do once you accept that an agent’s account of its own work is a claim rather than a fact. Here I am concerned with what prevents the agent from breaking the rules. I had written the rules down, they were good rules, and they did not hold. Better wording might have made a failure less likely. It could not make disabling a test unavailable to an agent that still had permission to edit it.
When I say AI guardrails here, I mean instructions supplied as context: rules files, system prompts and instruction blocks. They tell the model what it should do. Validation examines the work to see whether it meets a requirement. Permission controls determine which actions the agent can take.
A validator can report a failure without stopping anything; enforcement requires that failure to block the action. Writing “do not remove the tests” supplies an instruction. Checking for removed tests validates the change. Refusing to accept that change enforces the rule.
I had been noticing how frequently I was having to correct the agents. But a feeling is not data, so in August I did what nerds do: I pulled the conversations I could recover locally and searched them. Roughly six weeks of work across twenty repositories: 322 sessions containing 6,803 messages from me, counted as turns, excluding notifications, interrupts and other automated messages.
I was looking for the language you use when an agent breaks an instruction: you were supposed to, you did that without asking, I told you never to. The stricter search matched 236 turns across 82 sessions, about one session in four. A broader search for corrections matched 647 turns. There were 210 turns in both sets, giving 673 distinct matches across 134 sessions.
I felt vindicated. There was plenty of correction language in the record. That did not make every match a verified breach, but it gave me something to examine beyond remembering the sessions that annoyed me.
The scan also found 544 interruption markers: occasions when I stopped a response before it finished. Those record interventions, not proof that a rule was about to be broken.
The recorded counts came from local Claude Code transcripts, not conversations on the web or another machine. Those files were still changing while I ran the searches. The numbers record what I found then; rerunning the searches today would not recreate the same set of conversations.
The repository containing this article also held 28 feedback files. Twenty-two contained dates; the accompanying incident notes record why rules were added. The rules kept being broken anyway. The agent that helped write this article broke five of them while it was doing it, one created eight days earlier in response to the identical mistake.
These numbers tell me how often my messages contained correction language. They do not tell me how often the agents followed the rules, because I did not review every piece of work or verify every search result.
Some of the incidents were trivial. At least one spent real money in tokens I had not authorised, and one committed work that was not mine to commit. I have not graded them by severity; a count does not tell you whether I was correcting a filename or trying to stop a consequential action. The money and the commit are why I care about what prevents the action, rather than how clearly I asked.
If you retain your agent conversations, you can look for the same evidence. Start with the occasions when you intervened, and distinguish what the record shows from what you remember happening.
The questions I want answered are these. What have we asked the system to do? What checks whether it did it? And when did we last watch that check catch something? Writing a rule answers the first. It leaves the other two unanswered.
‘Guardrails’ Is Doing Work It Has Not Earned
A guardrail has to prevent something. If an agent can take an action despite the instruction forbidding it, the instruction has not enforced that boundary. It may have made the action less likely. Making a forbidden action less likely is useful. It is different from preventing it, and the evidence needs to show which the instruction achieves.
An instruction in context is text that the model processes alongside the task, code, tool output and other material it receives. That does not mean every message has identical priority. It means the model still has to produce an output that follows the instruction. Writing the prohibition does not remove the tool or permission needed to do the prohibited thing.
Prompt injection exposes the same difficulty from another direction: text encountered while doing a task can influence what the model treats as an instruction. OWASP’s entry on prompt injection describes the risk and the limits of prevention. That supports treating external text as a potential source of unwanted instructions. It does not prove that the problem is unfixable, or that all mechanisms for prioritising instructions are equivalent.
There is a real counter-argument here and it deserves the strongest version. Models are trained to weight some instructions above others. OpenAI’s instruction hierarchy work addresses models treating system instructions and untrusted user text at the same priority. Training the model to distinguish their privileges improved robustness. So there is a privileged instruction hierarchy. I was wrong to think there was not.
What that result does not establish is an enforced guarantee. A model trained to prioritise an instruction is more likely to follow it under the tested conditions. A permission check can refuse an operation even when the model outputs a request to perform it. The first improves the model’s behaviour. The second controls whether that request can take effect.
This is the distinction I see lost when organisations treat a rules document as evidence that an agent’s behaviour is bounded. The instruction can be clear, repeated and useful, while the forbidden operation remains available.
These counts cannot tell me how much the rules helped. I did not compare the same work with and without them. Nor did I check every result I had not complained about. Subtracting the 236 search matches from my 6,803 messages would not tell me how often the agents followed the rules.
I still write the rules, and I find them useful. The hierarchy research gives a separate reason to take instruction quality seriously: trained priority improved robustness. Neither that result nor my experience requires pretending the improvement is a guarantee. The case for instructions survives the distinction.
In the test-disabling incident, the instruction was present and the prohibited action still happened. The failure was discovered after the action. Preventing that particular outcome requires more than another sentence telling the agent not to do it; it requires something that checks for removed tests and can stop the resulting change.
This is why I want the names to stay separate. Instructions express what I want the agent to do. Independent assessment gives me another judgement about what it did. Ordinary engineering controls determine which operations or changes may proceed. Calling all three guardrails makes it too easy to mistake written instructions for controls that stop forbidden actions.
Password Policy Shows How a Visible Rule Can Mislead
Make everyone change their password every 30 days.
It feels like control. It is auditable, it is enforceable, it generates a compliance artefact, and it is easy to point at in a board pack. It can also encourage worse choices. NIST’s historical digital identity guidance says verifiers should not require passwords to be changed arbitrarily or periodically . NIST’s accompanying explanation of password expiration describes how people expecting repeated changes tend to choose weaker secrets, and how forced changes often produce small transformations of an old password. Password1 becomes Password2. The policy can therefore encourage predictable changes instead of the stronger authentication its authors intended.
Now put yourself in the board meeting. You can show that the password rule exists and that people follow it. Have you shown that authentication is safer? My concern is what happens when nobody asks that second question because the first answer looked sufficient.
My concern about agent instructions is similar. If a team believes its rules file bounds behaviour, it can stop looking for the checks that would demonstrate whether the rules hold. The instructions themselves may help. The misplaced confidence can still leave a forbidden action unchecked.
Consider the test removal that started this piece. If review does not detect it, the build can go green with less verification than it had before. A later defect may be the first visible sign that the check is missing. At that point the team is discovering the loss of coverage through the consequence it intended to prevent.
I think a team that knows it must verify agent work is better placed than one that mistakes the instructions for verification. I have been the second team. That is the experience behind my argument; I have not run a comparison showing that teams without rules outperform teams with them.
If otherwise comparable teams using the same checks caught a higher proportion of agent mistakes when they also had written rules, that would challenge my concern that confidence in those rules leaves teams worse protected. I have not made that comparison. What I have seen is enough to stop me treating a rules document as proof that the forbidden action cannot happen.
How Written Instructions Can Fail
So why did the agent break the rule? I can point to research on how models use their input, and I can tell you what I have watched happen in my own sessions. The research does not yet measure all the behaviour I want explained.
The task keeps appearing in the interaction. After an initial instruction, an agent may receive repeated errors, tool results and follow-up requests about the work it is trying to finish. That gives the task repeated expression without necessarily repeating the instruction. The Lost in the Middle study of long-context performance found that models used information at the beginning and end of long inputs more reliably than information in the middle. Appending messages does not itself move an initial instruction into the middle. What matters is the context the harness actually sends, including where it places and repeats the instruction.
Longer input can reduce performance. Chroma’s context rot report evaluated eighteen models on tasks including retrieval and text replication, and found performance sensitive to increasing input length. Those are not measurements of an agent obeying a behavioural rule over four hours. They give me a reason to test that behaviour as sessions grow, rather than assume that a large context window preserves it unchanged.
Compaction can lose an instruction. If a harness replaces earlier conversation with a summary, an instruction survives only if it is retained there or supplied again from elsewhere. A summary that preserves task state but omits the instruction leaves the next model call with less information than the original one. Whether that happens depends on the harness and the summary. I cannot say that an instruction is always the first thing discarded; I can inspect whether the next call still receives it.
Finishing the task can conflict with the instruction. The opening incident is a concrete example. Disabling the tests made the build pass while violating the rule about the tests. I have written about why agents report work they have not done , but the narrower point here does not need a theory of what the model wanted. Two requirements were present, and the output satisfied one by violating the other.
The obvious response to the first of these is to repeat the instruction. Put it in the next prompt, keep it available after compaction, and make its priority clear. That is a reasonable response. If an instruction has disappeared from the input, restoring it addresses a real defect in how the agent is being run.
It still does not make the forbidden action unavailable. Repetition can improve the chance that the model follows an instruction; a permission boundary answers what happens if it does not. I want both. A clearer rule can guide the work, while a separate check refuses a change that removes tests. Improving one is not a reason to omit the other.
There is another pattern I can describe from experience but cannot ground in a paper. In some long sessions I see output doing the specific thing I instructed against, without expressing uncertainty about it. I cannot explain why it happens, and I may be remembering the cases that annoyed me. I check that the agent still receives the rules and that its consequential work has been reviewed. A fluent response late in a session does not tell me whether it is still following the instructions I gave at the start.
Assess the Change Without the Producing Agent’s Explanation
If a mistaken assumption enters a working conversation, later output can continue to rely on it. A self-review in that conversation has access to the same assumptions and the same account of how the work was done. It can find a mistake, but I do not want my only assessment to depend on the agent that produced the change recognising its own error in the same conversation.
I therefore separate the assessment from the working conversation. The reviewer receives the proposed change and the evidence needed to judge it, without the producer’s account of why the change should be accepted.
The review workflow I use, and put in front of clients, has three stages:
- Diagnose. Work out what is wrong and propose a change. Output: a statement of the problem and a diff.
- Assess. Read the diff against the authorised scope, relevant code and requirements. Does it introduce a security flaw? Does it weaken the build rather than fix it? Does it remove coverage, delete data, or do something outside the task? The reviewer is asked to find problems in the change, rather than confirm the producer’s account.
- Decide. Apply, reject, or escalate to a human, on the basis of the proposed change and the assessment.
The separation is between the producer’s reasoning and the evidence the reviewer needs. A diff alone cannot tell a reviewer what scope was authorised or what unchanged code depends on. Withholding those inputs would make the assessment weaker. I supply them independently, so the reviewer can compare the change with the actual requirement.
A confident account of why disabling four tests was a reasonable interim measure can influence the next assessment before it has examined the tests. That is the account I withhold. The failing output, the tests, and the requirement to preserve them are evidence and belong in the review. Independence from the producer’s explanation is not independence from the facts.
There is a version of this I run with tools that already exist. An agent proposes a change and opens a pull request. Two reviewers assess it without the original working session. An agent in a separate conversation takes the diff and review comments and addresses them. A human decides whether to merge. At each stage the relevant requirements remain available. This gives an error more opportunities to be found, without making the reviewers infallible or making their errors statistically independent.
Independent Implementations Can Still Fail Together
The distinction matters because there is evidence both for the value of multiple implementations and against assuming that their failures are independent.
In 1986, John Knight and Nancy Leveson tested the assumption underneath n-version programming : that if several teams independently implement the same specification and you vote on their outputs, their failures will be independent, so the vote protects you. Twenty-seven versions, written separately by students at two universities who were kept from comparing notes. They measured it. The failures were not independent. The versions failed together far more often than independence predicts, because the hard parts of the specification were hard for everyone, and different people misunderstood the same things in the same ways.
Separate contexts of the same model still share weights and training. Different models may share training material and approaches. Separating the conversations removes one route by which an earlier assumption can persist; it does not establish how often the resulting errors will coincide.
In June 2026, Ron, Baudry and Monperrus repeated Knight and Leveson’s experiment with coding agents . Their preprint reports forty-eight agent-generated implementations of the same specification, across different agent systems, models and languages. They tested them over a million randomised inputs against a shared source of expected answers, called a test oracle. Implementations often failed on the same inputs, particularly where the specification was hard or ambiguous.
They also found that voting worked rather well anyway. Majority voting across three-version units took the mean failure count from 387 for a single version down to 131. Of 17,296 three-version combinations, 11,844 produced no observed failures, compared with 27 of the 48 individual implementations. Their conclusion is that n-version programming with coding agents is a useful engineering strategy, and on their evidence they are right. I am not going to cite the half of that paper that suits me and quietly drop the rest.
Those results give me a reason to take redundancy seriously, while keeping its limits in view. Voting helped where the implementations produced different answers. It could not resolve every failure they shared. This was implementation voting against a test oracle, not an experiment comparing independent review with self-review at the same token spend. It supports a related engineering approach, not a measured claim about the review arrangement I use.
The distinction between hard and ambiguous requirements matters beyond that experiment. A requirement can be clear but difficult to implement: the code still gets the answer wrong. An ambiguous requirement leaves room for different answers because it does not say clearly enough which is wanted. Tests can reveal that disagreement, but somebody still has to decide what the software should do. Fixing the code and clarifying the requirement are different jobs.
Nor does a residual failure prove that nobody made a decision. The implementations may share an error, the oracle may encode an assumption, or the requirement may still need clarification. Before adding another checker, I want to know which of those problems I am asking it to solve.
What Checks the Agent’s Work?
In practice I use two kinds of automated check. They contribute differently, and treating the first as though it supplied the second would repeat the mistake with the rules file.
A Model in a Separate Context
This can be as simple as the producing agent spawning a reviewer with a clean conversation. It can also mean sending the work to a third-party review tool running in a different system, perhaps using a different model. Both separate the review from the conversation that produced the work, but they change different things.
A freshly spawned reviewer is the simplest version, provided it does not inherit the producer’s conversation and justification. A separate review system can also change the model, instructions and tools used to assess the work. In either case, the reviewer still needs the proposed change, the authorised scope and the evidence needed to judge it.
The reviewer assesses the change against the requirement and can recommend acceptance or refusal. If its assessment controls whether the change proceeds, that decision must be enforced by the surrounding workflow. The model supplies a judgement; the workflow holds the action until the required decision exists.
More separation does not by itself prove a better review. A third-party tool can share the producer’s mistaken assumptions, while a clean spawned reviewer can catch an important error. I look at what I have actually varied:
- Different input. Supply the change, requirements and relevant evidence without the producer’s justification. This takes deliberate work in the harness; starting another conversation is not sufficient if it receives the first conversation wholesale.
- Different question. Ask what the change breaks, which requirement it violates, and what evidence would support rejection. That gives the reviewer a concrete task beyond agreeing that the producer did a good job.
- Different model, and where useful a different vendor. This changes some of the conditions under which the work is assessed. It does not provide a known degree of independence. The useful evidence is which errors the reviewers catch and which they share.
A Deterministic Check
Not a model at all. A schema validator, a test suite, a check that rejects reduced test coverage, a script that detects removed assertions, or a check that rejects increases in unresolved findings.
A deterministic check does not need the model to agree with its result. It evaluates the condition encoded in it. If the check must run, its failure blocks the consequential action, and the agent cannot bypass it or weaken its definition, it enforces that condition. Without those properties, a script in the repository is only something that could be run.
The test-disabling incident shows why protection of the check matters. A suite can execute deterministically and still go green after its failing tests have been removed. The release workflow must reject unauthorised removal of existing tests as well as run the remaining tests. A change to that protection must require authority the producing agent cannot grant itself.
Which Rules Does My Build Actually Enforce?
That gives me a more specific question about my own setup:
How many of your AI guardrails do you validate at build time, deterministically?
Not how many rules you have written. Not how carefully you worded them. How many of them are checked by something that runs without being asked, and fails, and stops the work proceeding.
Go and count. I did, and I did not enjoy the answer.
Agents maintain almost all of the repositories containing this article. I went through the build to see what would actually stop them. Missing fields, inconsistent files, broken references: there are checks for those. Here is what they inspect:
- Content files must satisfy their declared schema; a missing required field fails the check.
- The copies of the instructions supplied at the start of an agent’s work must agree. Different copies fail.
- Recognised filenames and configuration patterns we have explicitly retired must not reappear.
- Each website must be declared consistently, and conflicting claims to the same canonical address produce a finding.
- Instrumented features must use valid telemetry identifiers; malformed identifiers fail.
- The infrastructure record must name resources in its stored Azure capture, and that capture must be sufficiently recent. The check does not query live Azure, and it does not catch every reference to a resource deleted since the capture.
- Route validation checks published addresses against the routes and destinations available to it, so moves and missing targets can produce findings.
- Files named by the rules system must exist, and its indexes must list the files they govern. Missing files and incomplete indexes fail.
You can see what is missing from that list. A valid article file can contain a bad argument. Matching instruction files can repeat the same contradiction. A feature with no instrumentation has no malformed identifier for that check to find, and a retired practice I have not encoded will pass unnoticed. I have checks for the conditions I wrote down, not proof that the engineering is sound.
The required gates and test suites sit before release in the pipeline. When a required check fails, the pipeline stops the release. Whether an alternative path or a change to the pipeline can bypass them is a separate part of the control to inspect.
Commit-message validation is advisory in the current prepare stage. In the period I counted, I also had a check warning for weeks without stopping the work. I knew about it and had not fixed it. A warning may be useful information, but I cannot count it as a refusal to proceed.
Now compare those checked properties with the rules governing the agent’s actions.
An agent can produce files that pass every one of those checks without ever having permission to do the work. That was the uncomfortable part of counting.
Here are the rules that govern how an agent behaves in my repositories. Ask before you act on anything consequential. Do not make a structural change without approval. Confirm a name before creating something that fixes a public address. Do not run the expensive generation step until the work has been approved. Do not spend money without asking first.
Those are among the rules I have seen agents break. In the setup I audited, none had a build-time check capable of preventing the action. A build run after an unauthorised generation step cannot undo the token spend. That is a gap in what my pipeline covers, and it matters more to me than several of the properties it does check.
The approval could be checked before the tool spends the money. Somebody still has to decide whether to approve, but the tool can refuse to run until that approval exists. That is where I need the check, on a path the agent cannot bypass. By the time a build tells me the spend was unauthorised, I have already paid for it.
I also counted the recorded build outcomes. The queries retrieved up to 400 runs from each of seven repositories, and every query reached that limit. That gave 2,800 runs in the historical sample: 2,456 successes, 231 failures and 113 other outcomes. This is a capped retrieval, not a complete count of every build across the period.
Of those failures, somewhere between fifteen and eighty-four were a validation gate stopping the build rather than the code failing to compile or the site failing to deploy. I give you a range rather than a number because the honest answer is a range: the gates run as steps inside larger jobs, so a gate catching something surfaces in the build log under the job’s name, not its own. Separating them properly means reading step-level logs I have not read.
I run these gates. I wrote most of them. Yet I cannot tell you exactly how often they fired in that sample without doing more work in the logs. If you think having the checks means you know what they catch, try answering that question for your own build.
I can show you failed runs associated with those checks. I cannot honestly turn them into a count of defects saved from reaching customers: I would need to inspect what failed and whether the change would otherwise have shipped. What the records establish is that this build path refused to proceed.
And one against myself, because it is the strongest thing anyone could say and I would rather say it first. In my own transcripts an agent recorded that gh run watch --exit-status returned success on a channel run whose build job had failed and whose deployment was skipped. A deterministic check reported green on a red run. Deterministic does not mean infallible. A deterministic check can still report the wrong result, as this one did.
The distinction survives those limits. I have examples of agents violating written instructions, and build records in which a separate mechanism refused to proceed. The instruction’s effect depends on what the model outputs. A required gate can reject that output. The evidence for the gate is the rejected condition and the action it stopped, not the fact that I wrote a rule or a script.
My build checks enforce some rules and leave others unchecked. I have deterministic checks for several file and configuration properties, but the audited build did not enforce important rules about what the agent could do. Counting both stopped me treating the visible checks as evidence for the rules they never examined.
A deterministic check can also be wrong in the same way as the producer. If an agent writes a test from the same mistaken requirement it used to write the code, both may agree on the wrong answer. Once the check runs, its result does not depend on the agent following an instruction. But the check may still expect the wrong answer.
Both checks can pass and the work can still fall short of what you expected. Somebody has to examine what was actually built against what was needed. That is the human in the loop: a person with enough knowledge to recognise the gap and the authority to reject the result.
I still have to do that work. An agent’s approval and a green build give me evidence to consider; they do not make the decision for me.
Whether coverage fell, an assertion disappeared, or a file satisfies a schema can be expressed as a condition and evaluated consistently. Whether the proposed change is the right thing for this customer this week may depend on choices that the tests do not settle. More executions of the same test cannot decide a priority that has not been agreed.
That is a reason for human judgement at the decision point, not a claim that humans are always more accurate or that every residual software error is an undecided requirement. Somebody has to decide which consequences are acceptable and take responsibility for that choice. A voting pool can inform the decision. It cannot take that responsibility on the organisation’s behalf.
If three review contexts share the same mistaken assumption, their agreement may leave the important defect untouched. I want evidence that the checks have challenged the change from the relevant requirements, not just three approvals. The distinction applies whether the reviewers are people, models or scripts.
What Does Checking the Work Cost?
You still have to pay for the extra review. Three contexts do not mean three times the bill: the inputs, work and models can differ. But somebody looking at the spend will quite reasonably ask what the extra assessment bought. Which failure did it catch, and what did catching it save you?
That is also the risk in a cost review. Removing a separate assessment saves its visible cost immediately. The cost of an error it might have caught may not become visible until later. I would want to know which failures that review had caught, and what would catch them after it was removed, before treating it as a saving that preserves the same protection.
I am not going to tell you to spend three times as much, because that is not what I do and it is not what the argument requires. Where I can write a check that identifies a failure, I use one. I use model review and human judgement for the questions those checks cannot answer.
- The deterministic checks run on each relevant change. Removed assertions, falling coverage, unauthorised file changes and invalid schemas can be checked when the condition and approved baseline are available. Each implemented check needs to be tested and protected; the fact that a failure could be checked does not mean the pipeline currently catches it.
- Separate model review examines questions the implemented checks do not settle. Does the change exceed its stated purpose? Does it introduce a risk the task never mentioned? I use it as another assessment, with its own possibility of error.
- The human gate belongs where a consequential decision requires judgement. It has to be narrow enough for the reviewer to examine the evidence. If approvals arrive faster than someone can assess them, the workflow can retain the appearance of review while the decisions become routine confirmations.
I would start with what each check is supposed to catch, then look for evidence that it has caught it. That lets me argue for a cheap check running on every change or an expensive review of a consequential decision. The number of review stages does not tell me whether either is worth paying for.
Test Whether the Checks Stop Failures
This leaves an objection to everything I have recommended. I have argued against believing a rules file provides enforcement, then described scripts, separate reviews and human decisions. What stops me trusting those scripts, reviews and approval decisions simply because they appear in the pipeline?
I have to test whether those checks and decisions stop the failures they are meant to prevent.
Your gate has been green for months. Has the forbidden condition never occurred, or is the check looking at the wrong branch? Both can give you the same result. Until you put a known failure through it, green does not answer the question.
I want to examine changes the checks blocked and failures they missed.
Which of your gates has actually blocked a change, and what did it reject? Where the answer is never, submit a known violation and watch what happens. Reading the configuration cannot tell you whether that violation stops the work on the path you actually use.
And when something got through, which check should have caught it? Asking who approved the change identifies a person. Asking which check was expected to catch the failure makes me examine whether the check existed, ran, received the right input and controlled the relevant action.
Count the blocked changes and the failures that got through, and do not turn either count into a target. Rewarding a high gate-firing count encourages noisy checks, because a gate that blocks constantly looks healthy on that measure. It can protect flaky checks for the same reason. Displaying the counts is not itself the problem. Treating them as evidence of success without inspecting what was blocked is. I want them as diagnostics, alongside the failures that escaped.
Break something deliberately and watch the gate catch it. Mutation testing does this to a test suite by changing code and checking whether the tests detect the change. For a release gate, I also need to see that rejection prevents the release. A failing test that the workflow ignores has detected the problem without enforcing the boundary.
Then test changes to the checks themselves. If the producing agent can remove the assertion or alter the expected result without separate approval, the next green run may say less than the previous one. The control includes who can change it and what evidence that change requires.
This Is Not an Argument About Trusting AI
I use agents because they can do useful work. The failures above do not make that capability disappear. They tell me what I need to inspect before I accept the result.
We already review work written by people, run tests, and require decisions before consequential changes. Those practices do not require believing the developer is dishonest. They recognise that a capable person can misunderstand a requirement or miss a defect, and that the person producing the change should not supply the only evidence for accepting it.
The same reasoning applies when an agent produces the change. A rules document can help it work within the team’s expectations. Code review and continuous integration supply different evidence. A protected release path can refuse the result. Each contributes something the others do not establish on their own.
When an agent produces work faster than I can inspect it, the gap becomes more visible. Writing another instruction does not increase my ability to verify the result. I need the verification work to keep pace with the consequences of accepting it.
I do not write code any more and I would not go back. That makes the verification my problem to solve. Producing more work does not relieve me of deciding what evidence I need before accepting it.
What Would Change My Mind
I would change the central claim if a system could demonstrably prevent a forbidden action because its instructions forbid it, even when later messages tell it otherwise. Show that it holds when someone deliberately tries to defeat it, rather than merely making compliance more likely. With that evidence, I would call the instruction an enforced boundary.
I would rethink my explanation if controlled tests showed that agents followed the rules just as reliably in longer, harder sessions. The long-context studies I cited did not test that. They give me a reason to investigate what happens during a session, not proof that every model will eventually break a rule.
I would revise the review arrangement if a fair comparison found that separate-context assessment caught no more consequential errors than self-review at equal spend. The coding-agent voting results do not answer that question. Nor do they tell me whether the same budget would buy more protection through additional deterministic checks. Where I can express the condition in a check, my money is on the scripts. I want to see which approach catches more consequential failures for the money.
Those results would affect different decisions: what I call an enforced boundary, how I manage context, and where I spend the verification budget. I do not need all three questions settled to stop accepting an instruction file as proof that a prohibited action cannot happen.
Subscribe to Martin's articles
Writing since 2006
Articles, usually weekly on Mondays. One click to leave.
Call Written Guardrails Instructions
Stop calling the instructions guardrails. The word invites you to count a written request as a boundary you have built, and my own work shows how easily the two come apart.
Call them instructions. They are worth writing, mine are long, and I am not suggesting anyone delete them. I use them to direct the work. I use separate checks to establish whether the result satisfies the conditions I can test.
Then take a rule whose breach would matter and follow it to whatever stops the action. Can the agent spend the money anyway? Can it remove the test and still get a green build? Changing the name has bought you nothing unless you go on to examine those permissions and checks.
What have we asked the system to do? What checks whether it did it? And when did we last watch that check catch something? For each answer, I want the actual instruction, the actual check and a result I can inspect. Where the second answer is missing, I have a request whose outcome still depends on the agent. Where the third is missing, I have a control that needs to be tested.
If you need to establish which agent actions your checks prevent and where your verification still depends on someone noticing a mistake, book a call .
Enjoyed this? One click, no account.
Questions this answers
Do AI guardrails actually prevent agents from breaking rules, or are they just suggestions?
The article argues that AI guardrails implemented as written instructions (prompts, rules files, policy docs) do not in themselves prevent forbidden actions; they only make compliance more likely. Actual prevention requires separate validation and enforcement mechanisms—deterministic checks, protected workflows, and permission boundaries—that can detect violations and block the consequential action regardless of what the model outputs.
How can I reliably check and enforce what my AI agents are allowed to do in my build and release pipeline?
You need to separate instructions from checks and enforcement: define clear rules for agent behavior, then implement deterministic checks and protected gates in your build/release pipeline that run automatically, evaluate specific conditions, and block actions when violations occur. You should audit which rules are actually enforced at build time, test gates with known violations, protect those checks from being weakened by agents, and retain human judgement for consequential decisions the automated checks cannot settle.
What’s the difference between AI instructions, validation, and enforcement, and how should I structure reviews of AI-generated changes?
Instructions tell the agent what to do, validation examines its output against requirements, and enforcement is when a failed check actually blocks the action. The article recommends separating production and assessment by using a three-stage workflow—diagnose (produce a diff), assess (review the change against scope, requirements, and risks in a separate context), and decide (accept, reject, or escalate)—and, where possible, adding independent model reviewers and deterministic checks so acceptance does not depend solely on the producing agent’s own explanation.
How can I test whether my AI-related gates and checks actually stop the failures they’re supposed to?
You should examine which gates have ever blocked a change, submit known violations to see if they’re caught, and inspect failures that escaped to determine which check should have caught them and whether it ran on the right path. Treat gate-firing counts as diagnostics rather than targets, use mutation-style testing of gates and their own change control, and verify not just that a condition is detected but that a failing check actually prevents the release or action from proceeding.
Smart Classifications
Each classification [Concepts, Categories, & Tags] was assigned using AI-powered semantic analysis and scored across relevance, depth, and alignment. Final decisions? Still human. Always traceable. Hover to see how it applies.