“All models are wrong, but some are useful.” This is a famous quote from 20th century statistical thinker George EP Box.
This may seem like a strange message – shouldn’t all the models we build be as correct as possible? However, as a data scientist, I see great wisdom in this statement. After all, companies don’t buy AI for model accuracy, they buy it to add business value. Many companies are investing in AI today without realizing the full potential for business impact. It’s time to postpone the conversation.
The problem? Most groups start building their AI solution by discussing what they want to predict and quickly move on to discussing model accuracy. This strategy often leads data scientists into the doldrums of model metrics unrelated to business KPIs. Instead, we need to focus on the business outcomes we want and what actions the AI can dictate to achieve those goals.
Let’s use an example from a software company to illustrate this. This company’s accounts receivable team may use AI to predict whether an invoice will be paid on time. On its own, that forecast has limited business value – an accurate prediction that every customer will pay on time doesn’t quite meet the goal of shortening the cash-to-sales cycle. Instead, this team should think holistically about the AI solution: how can they align their prediction with the most important recommendations and actions that will help the user focus on their time?
How do we do that? We need to break the silos between business leaders and data scientists. Let’s get executives and data scientists to work together on four key pillars to realign businesses on a smarter core approach:
MEASURE UP the KPI. What is the business bottom line that we track that is used as a measure of tracking the impact on your model?
INTERVENTION based on what the AI dictates. What organizational levers and restrictions are there and how can your AI provide guidance?
EXPERIMENT Measure impact. Build models and use them in controlled experiments to attribute effects to the use of AI.
ITERATE through constant monitoring, optimization and experimentation. Data changes, opportunities arise, no model is permanent.
These four pillars will help data scientists ask their business partners more valuable questions and give business leaders a deeper understanding of the power of AI within the organization. Too often it is difficult or time consuming for technologists to inform their business partners about AI or to ask them why a particular predictive model is proposed. AI can be more than the data sets that drive it. Adopting these four pillars early and having honest conversations can often lead to greater agility and resilience – crucial, as local and global events change the business landscape around us from temporary anomalies to events involving black swans.
Let’s go back to the bottom line we talked about – receiving payment on bills.
Typically, companies create a predictive model to identify which customers are at risk of not paying on time. However, when we focus on a better way to measure the impact, we turn this forecast flag into a prescriptive solution and train the model to increase expected sales within 30 days of submitting the invoice.
Today, accounts receivable workers may have several tools at their disposal to ensure that payment is collected within 30 days. Everyone has their own effectiveness, from phone calls to email nudges to automatic payment proposals or texts to suspend service. Employees can choose from any number of these actions in an attempt to hit a target. However, they can be limited to where to spend their time. A model that predicts outcomes alone will not help employees choose what action to take. Instead, try to build models that predict the outcomes of these interventions, influencing actions that will lead to optimal results.
We have now turned our Predictive Flag program into mandatory interventions. However, models are not intended to be static. Therefore, running tests, tracking real-time interactions, accessing temporal data (one at a time), and monitoring your KPI is a critical step in ensuring that your models don’t collapse in the event of unforeseen events. Models will not live forever. So be agile and know how to deploy new models. Iteration isn’t just about fixing problems. It’s also about opportunities. Yes, you can react quickly to problems like data drift, but you can also experiment and keep developing your business.
This shift in mind from predictive to prescriptive is a natural evolution in how we understand and use AI in their business. And it’s more important in today’s unpredictable economic and competitive business climate, where the ability to make real-time decisions and deliver value quickly can separate the winners from the losers.
Published on February 25, 2021 – 23:03 UTC