The real issue is no longer whether firms are adopting AI. It is whether governance maturity is keeping pace with how quickly AI is becoming embedded in operational delivery.
On this page
Why this matters now
At events like Accountex, it is increasingly common to hear software vendors frame their AI as safe because it sits inside a “closed system”. The claim usually comes bundled with other reassurances too: approved datasets, restricted internet access, validated outputs, and a human reviewing the result before anything important happens.
Those controls may all be useful. But they do not, on their own, amount to governance.
That matters because AI is no longer sitting at the edges of the accounting profession as a future possibility. It is already appearing across bookkeeping, summarisation, forecasting, anomaly detection, document extraction, workflow automation and client-facing support. Professional bodies are now discussing AI as a growing part of the operating landscape for accounting and finance, not a distant theory. ACCA and AICPA & CIMA both reflect that shift.
Adoption is accelerating
AI is moving from isolated experimentation into normal workflow support across accounting and finance teams.
Trust is compounding
As systems become more useful, staff naturally begin to rely on them more heavily and challenge them less often.
Governance now matters more
Once firms depend on AI operationally, the question becomes whether oversight is strong enough to keep pace.
Technical safety is not governance
A closed AI environment can reduce certain technical risks. It may help limit exposure to untrusted sources, narrow data boundaries, reduce some prompt injection opportunities and create a more controlled operating environment.
That is important. But governance is broader than technical containment.
Technical safety
- Controlled datasets
- Restricted access
- Secure deployment and operation
- Output validation
- Reduced exposure to untrusted inputs
Governance
- Clear accountability
- Meaningful oversight
- Auditability and evidence
- Escalation and monitoring
- Staff competence and defensibility
The UK government’s AI framework makes this distinction clearly. It separates “safety, security and robustness” from “accountability and governance”, alongside transparency, fairness and contestability. In other words, even a technically robust system still requires governance structures around it. See the UK Government white paper.
A system can be technically closed and still create serious professional, operational and regulatory risk.
Why “human in the loop” is not enough
Another familiar reassurance is that “the human reviews the output”. Again, that sounds sensible. And sometimes it is. But governance requires firms to ask sharper questions.
The ICO is explicit that meaningful human review requires more than nominal involvement. Reviewers should have appropriate knowledge, authority, independence, training and sufficient resource. Organisations should document review methodology, define acceptable tolerances, maintain override logs and report outcomes to senior management. See the ICO guidance on human review.
This is also where automation bias becomes critical. Once AI is consistently useful, challenge becomes lighter and approval becomes faster. Over time, the control can remain on paper while weakening in practice.
ICAEW makes the same point in professional terms, warning accountants against overreliance on AI outputs and emphasising the need for professional judgement, competence and due care. See ICAEW’s guidance.
AI risk is cumulative, not isolated
One of the biggest weaknesses in current AI discussions is that firms still assess risk feature by feature. An AI email assistant appears low risk. An AI bookkeeping suggestion tool appears low risk. An AI meeting summary tool appears low risk. An AI forecasting assistant appears manageable.
Individually, many of these judgements may seem reasonable. But that is not how risk accumulates inside an organisation. The real shift is that firms are embedding many AI-assisted decisions at once across bookkeeping, tax, audit support, client communication, reporting, forecasting, compliance checking and operational decision-making.
The governance challenge is therefore rarely one dramatic failure. More often, it is the accumulation of small dependencies, assumptions, weak reviews and under-documented decision points spread across the business.
What good visibility looks like
Firms need to know
- Which AI systems are in use
- Where they sit in workflows
- What decisions they influence
- Who owns each use case
- What controls are expected
Firms need to evidence
- How human review works in practice
- What staff training exists
- How issues are escalated
- What monitoring is performed
- How the control environment is reviewed
Without that visibility, governance becomes reactive rather than deliberate. You cannot govern what you cannot see.
The UK regulatory context
The UK has chosen a relatively flexible, pro-innovation and regulator-led approach to AI regulation. That flexibility has obvious benefits. It avoids premature rigidity, supports experimentation and allows existing regulators to apply principles in context. See the UK white paper.
But regulatory flexibility should not be mistaken for a low governance burden.
In practice, UK guidance is already quite clear on what responsible AI use requires. The ICO expects senior management accountability, proportionate governance structures, documented trade-offs, ongoing review and the ability to demonstrate compliance on an auditable basis. It is explicit that these responsibilities cannot simply be delegated to technical teams. See the ICO guidance.
The NCSC, meanwhile, emphasises secure design, development, deployment and operation across the AI system lifecycle. That supports the point that security is essential, but it is only one component of a broader governance picture. See the NCSC guidance.
Governance should enable adoption, not obstruct it
This is where the discussion often becomes unnecessarily polarised. Good governance is sometimes framed as if it were the anti-innovation position: slower, more cautious, more bureaucratic, less commercial.
That is the wrong lens.
Good governance is what makes sustainable AI adoption possible. It clarifies accountability, strengthens supplier challenge, improves review quality, creates evidence and allows leaders to move with more confidence because they understand where the controls are and where the gaps remain.
What governance is not
It is not about slowing every project down, creating theatre around risk, or treating AI as something firms should avoid.
What governance is
It is about making AI use visible, accountable, reviewable and defensible as adoption scales across the organisation.
What this means in practice for firms
Not every firm needs a large enterprise AI governance programme. But every firm does need some basic foundations: visibility over use cases, accountability for decisions, proportionate policies, meaningful review, escalation routes, supplier challenge and evidence that controls work in practice.
That is the point at which governance stops looking like a compliance exercise and starts acting like a business enabler. For firms in accounting and finance, this is exactly where the conversation is heading.
A closed system may reduce some technical risks. It may even be part of a well-governed AI environment. But it is not, by itself, a substitute for governance.
Light-touch references
Final thought
For accounting firms, the defining question is no longer whether AI is entering the profession. It already has.
The more important question is whether firms are building the governance maturity needed to use AI confidently, responsibly and at scale.
That is the real gap many firms are now starting to recognise — and exactly the problem AIGAS is intended to help solve.
— Sal Nasser
Founder, Prime PC Services (PPCS)
Frequently Asked Questions
Does a closed AI system solve the governance problem?
No. A closed system may reduce some technical risks, but governance also includes accountability, oversight, documentation, escalation, monitoring and evidence that controls work in practice.
What does “human in the loop” need to look like in practice?
It needs to be meaningful rather than nominal. Reviewers need the knowledge, authority, independence, training and time to challenge outputs properly, with override logging and documented review processes.
Why is this especially relevant to accounting firms?
Because accounting firms operate in environments built on trust, judgement, professional defensibility and accountability. As AI becomes more embedded in delivery, weak governance becomes a professional risk as well as an operational one.
Is UK regulation already strict enough to force action?
The UK has taken a flexible, pro-innovation approach, but that should not be mistaken for a low governance burden. Existing UK guidance already expects proportionate accountability, documentation and auditable oversight.
What is the minimum a firm should have in place?
At minimum: visibility over AI use cases, clear ownership, documented policies, meaningful human review, escalation routes, staff training and evidence that controls are actually working.