AI Governance — Supporting Reading
What we see in practice
These are not theoretical risks. They are patterns we have observed in practice, where systems produced outputs that were relied on — and where the consequences followed.
In many cases, the issue was not the technology itself, but how it was used, trusted, and controlled. This reading supports our AI governance work, which applies the same evidence-based approach as our flagship fraud and AML rule defensibility service.
Pattern 01
Decisions applied widely before being fully understood
Detection rules and models are applied across every transaction before their behaviour is fully understood. A single flawed rule can misfire at volume before anyone notices.
Seen in
Banking — Interest Calculation
Interest calculation errors persisted across customer balances before the scale of the problem was recognised — creating financial and regulatory exposure at volume.
Pattern 02
Outputs relied on without clear explanation
When a decision is challenged, firms often cannot show why a rule fired or how the outcome was reached. That gap becomes critical in disputes or regulatory review.
Seen in
Financial Crime — Fraud Model Review
Fraud decisions were made at scale, but there was too little traceability when individual outcomes were challenged by customers or compliance teams.
Pattern 03
Teams rely on outputs more than they realise
What begins as support — alerts, scores, suggested categorisations — quietly becomes the basis for decisions, often before anyone recognises the shift.
Seen in
Education — AI in Teaching
AI-generated feedback and marking support became relied upon without structured controls around consistency or review before material reached students.
Pattern 04
Ownership is unclear when something goes wrong
Systems are used across teams, but accountability is not always defined. When issues arise, responsibility becomes difficult to establish.
Seen in
Financial Crime — Fraud Detection
Fraud detection ran across banking and payments at scale, but no one clearly owned the decisions or the review.
Pattern 05
Review exists, but not consistently
Outputs may be checked in some cases but not others. This inconsistency allows errors to pass through unnoticed.
Seen in
Accounting — AI in Bookkeeping
AI-automated bookkeeping classification was used without consistent validation before submission stages, allowing small errors to flow through into final outputs.
Pattern 06
Financial and customer impact emerges later
The impact is rarely immediate. It surfaces over time — through complaints, discrepancies, and redress.
Seen in
Retail Finance — BNPL Calculation Risk
Incorrect interest on Buy-Now-Pay-Later balances went undetected, building widespread customer harm and regulatory exposure over time.
Pattern 07
Problems surface when challenged, not before
In many cases, risk remains invisible until a complaint, audit, or external question forces the business to explain what happened.
Seen in
Transactions — Investor Scrutiny
Data quality and explainability issues only became apparent when investors began scrutinising underlying information closely during a sale process.
These patterns recur across environments — from financial systems to operational tooling and AI-driven applications. They are reflected in the cases shown throughout this site.
Most firms do not recognise these risks while their systems appear to be working as expected. The same approach underpins our flagship work for UK FCA-regulated firms — fraud and AML rule defensibility.
Understanding your exposure