AI Risks
What we see in practice
These are not theoretical risks. They are patterns observed in real situations where systems produced outputs that were relied on — and where problems followed.
In many cases, the issue was not the technology itself, but how it was used, trusted, and controlled.
Pattern 01
Decisions applied widely before being fully understood
Systems produce outputs that are applied across customers or operations. Small issues can scale quickly before they are identified.
Seen in
Banking — Interest Calculation
Interest calculation errors persisted across customer balances before the scale of the problem was recognised — creating financial and regulatory exposure at volume.
Pattern 02
Outputs relied on without clear explanation
When decisions are questioned, it is often difficult to explain how an output was produced or why it was used. This becomes critical in disputes or regulatory review.
Seen in
Financial Crime — Fraud Model Review
Fraud decisions were being made at scale, but traceability around outputs was insufficient when individual decisions were challenged by customers or compliance teams.
Pattern 03
Teams rely on outputs more than they realise
What begins as support — summaries, recommendations, categorisation — gradually becomes something decisions depend on. This shift often happens without being recognised.
Seen in
Education — AI in Teaching
AI-generated feedback and marking support became relied upon without structured controls around consistency or review before material reached students.
Pattern 04
Ownership is unclear when something goes wrong
Systems are used across teams, but accountability is not always defined. When issues arise, responsibility becomes difficult to establish.
Seen in
Financial Crime — Fraud Detection
Fraud detection operated across banking and payment environments at scale, but decision ownership and review accountability were not clearly established across the process.
Pattern 05
Review exists, but not consistently
Outputs may be checked in some cases but not others. This inconsistency allows errors to pass through unnoticed.
Seen in
Accounting — AI in Bookkeeping
AI-automated bookkeeping classification was used without consistent validation before submission stages, allowing small errors to flow through into final outputs.
Pattern 06
Financial and customer impact emerges later
Issues are not always immediate. They often appear over time — through complaints, discrepancies, or unexpected outcomes.
Seen in
Retail Finance — BNPL Calculation Risk
Incorrect interest calculations on Buy Now Pay Later balances persisted unnoticed, creating widespread customer impact and regulatory risk over time.
Pattern 07
Problems surface when challenged, not before
In many cases, risk remains invisible until a complaint, audit, or external question forces the business to explain what happened.
Seen in
Transactions — Investor Scrutiny
Data quality and explainability issues only became apparent when investors began scrutinising underlying information closely during a sale process.
These patterns appear across different environments — from financial systems to operational tools and AI-driven applications. They are reflected in the cases shown throughout this site.
Most businesses do not recognise these risks while systems are working as expected.
Understanding your exposure