AI & Automation — supporting reading
What we see in practice
These are patterns we have met in practice, where systems produced outputs that were relied on — and where the consequences followed.
In many cases, the issue was not the technology itself, but how it was used, trusted, and controlled. This reading supports our AI & Automation work, which applies the same evidence-based approach as our Financial Crime work.
Pattern 01
Decisions applied widely before being fully understood
Detection rules and models are applied across every transaction before their behaviour is fully understood. A single flawed rule can misfire at volume before anyone notices.
Illustrated in
Banking: interest calculationAn interest calculation error runs across customer balances for months before anyone measures how far it reaches, and the exposure is financial and regulatory at once.
Pattern 02
Outputs relied on without clear explanation
When a decision is challenged, firms often cannot show why a rule fired or how the outcome was reached. That gap becomes critical in disputes or regulatory review.
Illustrated in
Financial crime: fraud rule reviewFraud rules decide on every transaction, but when a customer or the compliance team challenges one outcome there is too little trace to say why it fired.
Pattern 03
Teams rely on outputs more than they realise
What begins as support — alerts, scores, suggested categorisations — quietly becomes the basis for decisions, often before anyone recognises the shift.
Illustrated in
Education: AI in teachingAI-generated feedback and marking support become relied upon without structured controls around consistency or review before material reaches students.
Pattern 04
Ownership is unclear when something goes wrong
Systems are used across teams, but accountability is not always defined. When issues arise, responsibility becomes difficult to establish.
Illustrated in
Financial crime: fraud detectionFraud detection runs across banking and payments every day, but nobody clearly owns the decisions or the review of them.
Pattern 05
Review exists, but not consistently
Outputs may be checked in some cases but not others. This inconsistency allows errors to pass through unnoticed.
Illustrated in
Accounting: AI in bookkeepingAI-automated bookkeeping classification is used without consistent validation before submission, so small errors flow through into final outputs.
Pattern 06
Financial and customer impact emerges later
The impact is rarely immediate. It surfaces over time — through complaints, discrepancies, and redress.
Illustrated in
Consumer credit: Buy Now Pay Later calculationIncorrect interest on Buy Now Pay Later balances goes undetected, and customer harm and regulatory exposure build over time.
Pattern 07
Problems surface when challenged, not before
In many cases, risk remains invisible until a complaint, audit, or external question forces the business to explain what happened.
Illustrated in
Transactions: investor scrutinyData quality and explainability issues only become apparent when investors scrutinise the underlying information closely during a sale process.
These patterns recur across environments — from financial systems to operational tooling and AI-driven applications. They are reflected in the cases shown throughout this site.
These risks often go unrecognised while systems appear to be working as expected. The same approach underpins our work for UK FCA-regulated firms on fraud and AML rules.
See the method run
This one can be shown to you, running
Every pattern above ends the same way: an output was relied on because nothing in the process could tell the difference between a checked answer and an unchecked one. This automation is built so that difference cannot be lost — and on the case below it declines to conclude from a green, healthy-looking 91.2%, because the evidence underneath it will not carry the claim.
PRIN 2A.9
Consumer Duty Outcomes Monitoring
Monitoring must determine whether customers receive good outcomes — not report that monitoring happened.
The automation behind it is a financial-regulation check — what it demonstrates is the mechanism above, running on a real rulebook. It is shown by invitation and walked through with you, because the cases inside it are constructed for the demonstration. Illustrative cases are here.
Understanding the exposure