Keeping Predictive Fraud Models Accurate In Exceptional Times

Shelves were still empty as I went for my groceries last week. Sound familiar? We’re constantly assured that supply chains are being restored, yet according to an Institute for Supply Management survey, as early as March 11, three quarters of companies were already experiencing supply chain disruptions.

There still seems to be a shortage of many items as families are staying home because of COVID-19. Those who used to eat out often are now cooking most of their meals at home. Many are hoarding pasta, toilet paper and things the average person never bought, like yeast. Times are strange.

Technology failed to deliver

It’s easy to understand why these shelves are bare. People are trying to feed their families as inexpensively as possible. They’re learning to bake bread. And, toilet paper has a certain necessity that goes without saying.

There’s another reason the shelves stay bare: failed data science models.

Retail supply chains are highly dependent on forecasting models that predict how much is needed and when. We know that more ice cream is needed in the summer than during winter months, for instance. Chips and salsa are stocked up before major sporting events. Sunscreen is swapped for sweatshirts as the summer disappears.

Deviations cause disruptions

These predictive models use data and trends from past years to make a good estimate for what needs to be available. These models normally incorporate generic trends, national holidays, days of the week etc. And they typically do a good job.

However, COVID-19 proved that these models have their shortcomings and can be less than reliable… or fail us completely. This is where many retailers are struggling with:

• The data before COVID-19 does not say anything about this weird jump in demand during the crisis
• The data during COVID-19 is too limited to make accurate predictions
• The data After COVID-19 won’t represent the future, and can thus be discarded

Coping with exceptional times

Models are always affected and the quality of models degrades over time. Novarica’s Eric Weinsberg suggests predictive models can be counterproductive during this time. These are of course exceptional times, and our team has spent a great deal of time considering how our fraud-fighting models could be affected.

In times of crisis, economic uncertainty, or any other disruption, the FRISS team takes some important actions to ensure our models are reaching optimal performance:

• Continuous monitoring of model performance, including precision and hit rate.
• Periodic retraining of models, incorporating new data, in search of new features and trends
• Ensuring customers provide the feedback necessary to keep our models strong

Staying in control

We’ve recently seen a drop in claims due to COVID-19, with some lines of business experiencing a 50% decline. At the same time, we see that our models are performing at the same high level as they were before COVID-19 struck. The virus did not break our predictive models.

As the Head of Data Science for FRISS it makes me proud to remain in control during these exceptional times. It certainly helps that I’m surrounded by a great team of knowledgeable Data Scientists.

If you’d like to learn more about what we do with predictive insurance fraud models, I recommend looking at these vlogs by my colleagues Daan Bakker and Gian Luigi Chiesa.

Daan Bakker: How insurers benefit from real time risk scoring
Gian Luigi Chiesa: The need for explainable AI