When AI Starts Breaking the Systems Meant to Control It: Rethinking Responsibility
For years, the conversation around artificial intelligence has been relatively straightforward. Build more capable systems. Deploy them responsibly. Introduce governance to keep risks in check. However, something has started to shift.
Recent events suggest that AI is no longer just a tool being shaped by systems of control. It is beginning to interact with, influence and in some cases, undermine those very systems.
That is a very different problem.
A Subtle but Important Turning Point
Two developments, recently reported in the news, illustrate this shift clearly. First, as reported by Reuters, a national AI policy in South Africa had to be withdrawn after it was discovered to contain AI-generated references that did not exist. A framework designed to guide responsible AI adoption had itself been compromised by unverified AI output. Second, the Guardian, highlighted concerns around advanced AI models, including systems like Claude Mythos, a topic we also covered, that are capable of identifying and exploiting software vulnerabilities at scale, raising serious questions about cybersecurity risks if such capabilities become widely accessible.
On their own, these might appear as isolated issues. Together, they point to something more structural. AI is no longer just producing outputs. It is influencing decisions, systems and the frameworks designed to govern it.
The Illusion of Control
Much of AI governance today is built on an implicit assumption that humans remain firmly in control. Policies are written, safeguards are defined, oversight is applied. The idea is that, with the right checks in place, AI can be guided safely. However, what happens when those checks themselves become dependent on AI?
When policy drafts are assisted by systems that can hallucinate sources;
When risk assessments are informed by tools that are not fully understood;
When cybersecurity defences are tested by models that can outpace human analysis.
The boundary between controller and controlled begins to blur.
This is not a failure of intention. It is a failure of assumption.
From Errors to Systemic Risk
AI mistakes are not new. Hallucinations, bias, incorrect outputs have been well documented. What is new is where those errors are appearing, not just in chat interfaces or experimental tools, but in:
Policy documents;
Governance frameworks;
Critical infrastructure decisions.
This changes the nature of the risk.
An incorrect answer in a chat is inconvenient.
An incorrect assumption in a national policy is consequential. And, when AI begins to influence systems at scale, small errors do not remain small. They propagate.
The Fragility Beneath the Surface
At the same time, AI is exposing weaknesses in the systems we rely on. The concerns raised around models like Claude Mythos are not just about capability. They highlight how many critical systems still contain deep, undiscovered vulnerabilities, even after decades of scrutiny.
This is not a new problem, but AI is accelerating our visibility into it.
What was once hidden can now be surfaced. What required years of expertise can now be explored systematically. This is powerful, but it is also destabilising because it reveals that many of the systems we trust are more fragile than we assumed.
The Governance Gap Widens
All of this points to a growing gap. On one side: rapidly advancing capability. On the other: slower-moving governance, standards and professional frameworks.
The South Africa policy incident is a clear example of this gap in practice. AI was used within a governance process, but the safeguards around verification did not keep pace. At the same time, advanced model capabilities are evolving faster than the frameworks designed to manage their risks. This gap is not theoretical. It is operational.
Decisions about AI use are being made daily, across organisations and governments, often without:
Clear standards of competence;
Formal recognition of expertise;
Consistent accountability mechanisms.
When Systems Depend on Systems
We are entering a phase where systems increasingly depend on other systems. AI tools assist in writing policy. AI systems monitor infrastructure. AI models are used to test and secure other systems. This creates a form of recursive dependency.
Each layer adds capability.
Each layer also adds complexity and potential failure points.
The risk is not that any single system fails. It is that failures become harder to detect, trace and correct as systems become more interconnected.
Rethinking Responsibility
In this environment, responsibility becomes harder to define. If an AI-assisted policy contains errors, who is accountable? If a vulnerability is exploited using AI-generated insight, where does responsibility sit? If decisions are influenced by systems that are not fully understood, how do we assign ownership?
These are not purely technical questions. They are professional and institutional ones, and they highlight a growing need: the need for clear standards, recognised expertise, and accountable practice in AI.
A Different Kind of Maturity
There is a tendency to equate progress with capability. More powerful models. Faster systems. Greater reach, but maturity is something different. Maturity is about understanding limits.
It is about knowing when to question outputs. It is about building systems that are not just powerful, but reliable and trustworthy. The current moment suggests that AI is reaching a point where capability alone is no longer enough.
Final Thought
AI has always been seen as something to be controlled. What we are now beginning to see is something more complex. A technology that is not just shaped by systems of control, but one that is starting to interact with and influence those systems in return. That does not mean control is lost, but it does mean it must be rethought because when AI starts shaping the systems meant to control it, the margin for error becomes very small, and the cost of getting it wrong becomes very real.
