Machine learning project · neuefische Data Science
A fraud priority list for Tunisia's electricity and gas company, built on real billing data from 2005 to 2019.
Tim Pfeiffer · Irem Baykal · Dhanya Rajendran
The problem
Tunisian dinars lost because consumers tamper with their meters.
STEG can only inspect a limited number of homes. Every wasted visit costs money.
The question: whom should STEG visit first?
Our task
of every client
learns what fraud looks like
highest risk first
The list tells STEG where to send its limited inspections.
The data we looked at
clients with a known fraud result (training data)
bills, about 30 per client
facts: 6 per client (region, client category, start date, …) and 16 per bill (date, price plan, meter, consumption, …)
more clients (1.9 M bills) without fraud result, for the final test
The model needs one row per client, so the bills are summarised per client.
First finding
clients is fraud (5.6%).
right if we said "no fraud" for everyone. And we would find nobody.
So we do not ask "how often are we right?", but "how well do we rank the clients?"
First look at the data
Each bar: share of fraud among clients with this striking trait. Dashed line: average of all clients (5.6%). Links in the data, not proof of cause. Hover or tap a bar for details.
Three signals up close
Like a car whose odometer was swapped several times. Something may be going on.
One household heats steadily all winter, another one's gas use jumps up and down.
An odometer that runs backward. Rare: only 1,816 clients.
Share of clients with fraud. Dashed line: average of 5.6%.
Gas and electricity
of gas clients are fraud, against 5.6% of all clients.
result on both meters for region and client category.
of electricity clients whose readings do not add up to the billed consumption are fraud (5.5% for the others). On gas: 7.9% vs 7.1%.
Some signals are mainly an electricity signal. Many gas groups are small, so we read them with care.
Our approach
same rules for training and test data
summarise about 30 bills each
a simple model as the score to beat
four models, same fair test
search for better settings
which fraud cases do we miss, and why?
keep only what really helps
trained on all labelled clients, one submission
An early split and fair tests keep the result honest.
Result: ranking quality
Pick one fraud client and one honest client at random: the better the score, the more often the model puts the fraud client higher on the list. Bars start at 0.5 (coin flip). Hover or tap for details.
Key lesson
from new signals (+0.084 and +0.022)
from tuning the settings (+0.007 and +0.001)
The biggest gains came from learning from the model's mistakes.
What it means for STEG
of all fraud cases found, about 150 of 1,513
of all fraud cases found: 864 of 1,513
visits is a hit (5.6%)
visits is a hit (32%)
ranking quality (ROC-AUC): a coin flip
ranking quality (ROC-AUC); Zindi test: 0.874
Same 2,710 visits (top 10% of 27,099 held-out clients, 1,513 fraud). Random checks: expected values from the 5.6% fraud rate, not a measured test.
Honest limits
Our own check (0.876) was close to the leaderboard (0.874), so the validation was honest. Comparison is approximate: different competition rounds use different parts of the test set.
Number of meters and time since the last bill may partly show what STEG did after it found fraud: replace a meter, stop visiting closed accounts.
Our data has no date of detection. Next step: ask STEG how these columns come about.
Tim Pfeiffer · Irem Baykal · Dhanya Rajendran