BCG X - Forward Deployed AI Scientist / Engineer - AI-Enabled Fraud Detection for a Digital Bank

i

Supervised binary classification at transaction level: the model estimates the probability that a transaction is fraudulent. It should be a calibrated probability rather than a yes/no label, so the decision engine can apply different actions at different risk levels and weigh the risk against the amount at stake.

Main data issues:

1. Class imbalance: fraud is a tiny share of transactions, so accuracy is meaningless. Evaluate on precision-recall and, above all, on cost.
2. Delayed labels: fraud is confirmed up to 2 months later, so the most recent weeks of data are incomplete, both for training and for monitoring.
3. No outcome for blocked transactions: when the rules decline a transaction we never learn whether it was fraud. A model trained only on approved transactions inherits the blind spots of the old rules. Options: use transactions that went through additional authentication, or let a small share of low-risk declines through to observe the outcome.
4. Fraud patterns change: fraudsters adapt to controls, so the model needs regular retraining and drift monitoring.
5. Leakage: only information available at the moment of the transaction can be used; a chargeback recorded weeks later cannot be an input.

Features, grouped. A few examples per group are enough:

• Transaction: amount, merchant category, online vs in-store, domestic vs international, time of day.
• Customer behaviour: how far the transaction deviates from the customer's usual amounts, merchants, locations and times.
• Velocity: number of transactions and spend in the last minutes or hours, failed attempts, two payments too far apart for the time between them.
• Device and network: new or known device, IP location vs transaction location, number of cards linked to the same device.
• Merchant: historical fraud and dispute rate.

Models:

• Baselines: the current rules and a logistic regression.
• Main model: gradient-boosted trees (XGBoost or LightGBM). Strong on tabular data, capture interactions between features, and score comfortably within 100 ms.
• Two add-ons: graph features linking cards, devices, accounts and merchants to catch organised fraud rings, and an anomaly score for patterns not yet in the labels.
• Keep a small set of hard rules as guardrails, for example known compromised cards, instead of switching all rules off on day one.

• Average loss per missed fraud: €180
• Average cost per false decline (lost revenue, churn, service calls): €6

Follow-up, once the candidate has the annual value: implementation costs €6M one-off and €4M per year to run. Is it worth it?

Step 1: Monthly cost of each option

• Current rules: 8,000 × €180 = €1.44M; 180,000 × €6 = €1.08M; total €2.52M
• ML model: 4,000 × €180 = €0.72M; 80,000 × €6 = €0.48M; total €1.20M

Step 2: Annual gross benefit

€2.52M − €1.20M = €1.32M per month × 12 ≈ €15.8M per year

Step 3: Net impact (follow-up)

Payback: €6M / (€11.8M / 12) ≈ 6 months. The investment is clearly worth it.

Decisions: risk bands rather than a single cut-off.

• Low risk: approve.
• Medium risk: step-up authentication, which stops most fraud without losing the customer.
• High risk, uncertain, high value: manual review, within analyst capacity.
• Very high risk with strong signals, for example a known compromised card: decline.

The bands should also depend on the amount: the same probability on a €5 and on a €5,000 payment does not justify the same action.

Rollout in 3 steps:

1. Offline validation on a time-based split, training on older months and testing on the latest ones. A random split overstates performance because fraud patterns change over time.
2. Shadow mode: the model scores live traffic without acting, so we can compare it with the rules and check it stays under 100 ms.
3. Controlled pilot on one segment, for example online transactions in one country, with the model as challenger to the current rules. Then scale step by step.

Monitoring: fraud losses, false declines and approval rate on the business side; score distribution, input data quality and latency on the technical side. Alerts and automatic fallback to the rules if the model or the data pipeline fails. Since labels arrive late, early drift has to be read from score and feature distributions, not from fraud outcomes.

"NovaBank should move ahead with a controlled pilot of a machine-learning fraud model.

Today's rules cost about €2.5M a month between missed fraud and good customers blocked. The model brings this down to about €1.2M, roughly €15.8M a year. After €6M of implementation and €4M a year of running costs, the net benefit is about €5.8M in the first year and €11.8M a year after that, with a payback of around 6 months.

The model should produce a risk score feeding a decision engine that approves, challenges, reviews or declines, rather than blocking everything it flags.

Next steps: audit the data and the fraud labels, validate offline, run in shadow mode, then pilot on online transactions in one country before scaling. We keep a few hard rules as a safety net and retrain the model regularly as fraud patterns change."