Explaining Why Data & Models Aren’t Always Right & Getting Leaders To Act On Them
I was reminded by a subscriber that I hadn’t run a sale on my courses in a very long time. I took that feedback to heart, and now my most popular self-paced courses are almost half off for a limited time, but this post isn’t about the sale. It’s about why I rarely run sales and how I have convinced even die-hard discounting strategy believers to stop relying on sales to drive revenue.
I’m about to explain how to solve a problem that has set data teams up for failure for years. I have spent the last 9 years solving it and improving my approach. Data and models can’t gain traction until we confront and resolve this.
Considering discounting as dangerous, not an overall positive for the business, disagrees with conventional wisdom and strongly held beliefs. Data and models reveal the long-term impacts of sales and discounting strategies. However, most business leaders are thinking quarterly. By the time discounting risks turn into a sales slow-down and declining margins, the decisions that led to them are too far in the rearview mirror to be seen as the cause.
Getting leaders to change their actions based on data and models is one of our most significant strategic challenges. Nothing can be done at a tactical level to drive that kind of change. Data teams paddle in circles unless someone at the strategy level takes on the challenge.
I teach AI strategy and product management because if you trace the root causes of many execution challenges, you’ll find the symptoms originate from gaps in one or both. Data must improve outcomes to deliver quantifiable value. Outcomes won’t improve unless data can drive changes.
Solving this challenge takes us beyond anything change management frameworks are capable of. In change management, a leader decides to make a change, and the frameworks support efficient implementation. With this challenge, the data or model indicates that changing will improve outcomes. We need frameworks to get leaders to listen to data like they listen to experts and their own intuitions.
The Loop Of Madness And Managed Decline
The definition of insanity is doing the same thing repeatedly and expecting better results. Doing the same things with data doesn’t improve those activities’ value either. Data must inform change, so data teams focus on delivering data that indicates high-value changes. If leaders won’t act on data-driven recommendations or follow data that disagrees with them, value never materializes.
We must prepare the business to use data or it never will. Most data teams are in an infinite loop.
Business leaders demand more value.
The highest-value data reveals opportunities for improvement and how that value can be realized.
Those often require changing how things are done now, and leaders must act to make it happen.
Leaders ignore data that doesn’t agree with them, and nothing changes.
Data doesn’t deliver any value. GOTO 1.
Business leaders talk about building an agile company but still can’t escape this loop. The inability to change stems from a lack of effective change management frameworks. It’s the root cause of many different business problems. Data teams are just the most recent victims.
I have heard more top-notch startup pitches in the last 2 years than in the decade before. All of them are growing up in a tight funding environment. They don’t get funding unless they bootstrap their way to early success. The last decade belonged to companies that took years to achieve profitability. The next decade belongs to startups that become profitable in months.
Bigger businesses have too much inertia to compete with them. A startup can discover an opportunity on September 1st and deliver it by September 30th. In a large corporate environment, the idea may never get any visibility. Even if it does, opportunities spend months working through the approval process. Agility is just a word for businesses that won’t change.
People make a very comfortable living solving change management problems, and data gives us one of the best paths in. Businesses know they must change to succeed with data and AI. Business leaders have been forced to confront change management. This is one of those rare convergences that creates a massive opportunity.
A Symptom Of The Loop
The Bureau of Labor Statistics got into trouble by improving a model. What sounds like an excellent thing to data scientists can be perceived as a negative by users and stakeholders. Improving the model and increasing the frequency of model output reviews led to a dramatic downward revision in the total number of jobs created in the last 12 months. The BLS framed it as part of a standard annual comparison between its estimates and state unemployment insurance numbers, which is considered a more accurate benchmark.
The public’s response wasn’t, ‘Good job. Thanks for correcting that mistake!’ It was, ‘The government is covering up the truth. The economy is terrible. We’re on the verge of a recession!’ The BLS didn’t explain what happened, so several analysts had to work it out on their own. It’s still speculation, but it makes sense. There hasn’t been a downward revision this big since 2009…The Great Recession. That was also a time when model assumptions failed to hold, and estimates were way off as a result.
Business leaders expect data and models to work the same way software does. Enter 3 + 3 into a calculator app; the answer is always 6. Software is stable and deterministic. Data and models are stochastic. Deterministic systems are well-defined, and their dynamics or rules are well-understood. Outcomes are accurately predicted.
Stochastic systems are partially defined and understood. Outcomes seem to have an inherent randomness that comes from the parts we don’t understand. Data changes over time. If I measure tidal surges for 6 months, I’ll get a good picture of how tides work. If there wasn’t a storm surge during that time, my understanding of tides won’t work for storm conditions.
However, my dataset will be more complete after a storm hits, and my understanding of tides will improve. Models work until they don’t. We learn from model failures, and that leads to better models. In software, failures are unexpected defects. In data and models, failures are invaluable.
Data and models answer much more complex questions than software, but model outputs are the most probable answer based on our data. As time goes on, we always get more data. When it’s the same as our current data, nothing improves. When data contradicts our models, we can retrain them to be more accurate.
Businesses are also stochastic. People who do deterministic, well-defined work are considered low-skill. People who handle stochastic work are called knowledge workers and decision-makers. Software and digital systems have automated many of the business’s deterministic tasks. Now, we’re entering an era where data and models are trying to tackle stochastic tasks. Models and knowledge workers aren’t always right. Both learn from their mistakes and improve based on experience.
If no one explains this to business leaders and implements frameworks to support stochastic systems, data and AI are set up to fail, along with the data team. Legacy processes and strategy frameworks are designed to support a deterministic business, but, in reality, the business has never been fully deterministic.
Why I Don’t Run Sales
Sales and discounting strategies are dangerous. The first time I saw this was when I worked in retail. Macy’s positioned itself as a luxury retailer that targeted upper-middle-class customers. The brand’s perception of value enabled it to charge more and maintain healthy margins.
Retailers run sales to cycle out inventory and make space for new products or to diversify into a more price-sensitive customer segment. Inventory takes up space. If a product isn’t selling, it’s preventing the company from bringing in products that should sell better. There’s also money (the amount the company paid for the products) locked up in poorly selling inventory.
The sale lets more price-sensitive customers access products they usually couldn’t afford. The new customer segment unlocks the capital invested in the products and frees up space for new products. Everyone wins, right?
Macy’s growth was slowing, so it decided to increase revenue by holding more sales. Customer traffic surged during sales, and the company sold a lot more products. Growth restarted, and things were working well. Why change anything?
After a couple of years, I saw fewer high-end shoppers in the store. Many talked about spending more at Nordstrom now than in the past. While sales had given Macy’s a short-term boost, they had an unintended consequence. Sales train customers.
Macy’s core customer base, upper-middle-class buyers, saw constant clearance racks and markdowns. They were reminded of lower-end stores like Sears, Target, and JC Penny’s. Nordstrom maintained its perception of prestige and began to take market share from Macy’s.
Sales trained other Macy’s customers to wait for a sale. Why buy at regular price when products would eventually go on sale? The counter-argument is that the store may run out of the product before the sale starts. However, with high-end shoppers buying less, Macy’s ran out of products less often.
If you look at long-term trend data, sales and discounting strategies can create a death spiral. From a short-term perspective, sales look like an easy fix when growth stalls. Getting the benefits of sales and discounting without falling into the death spiral requires data from across the business and complex pricing models.
I have worked with several retail clients, and the models are never enough to prevent the discount death spiral. Leaders must be willing to change their minds and act on data that indicates a new discounting strategy. That’s a strategic decision-making problem that can’t be solved during the meetings when decisions are made. After explaining the difference between software and AI, we must implement frameworks to support the stochastic business.
Breaking The Loop With Pricing And Why I’m Running A Sale Now
You’ve already seen the first step in solving this problem. The last section was a quick case study. Instead of providing data, I tell a story that the data supports. Industry analysis or surveys can be good supporting material, too. “According to Accenture’s {some study}, retailers with aggressive discounting strategies saw a 12% increase in year 1 but an 18% decrease every year after.”
If you know business leaders don’t trust or use data to make decisions, that can’t be our starting point or primary supporting evidence. Storytelling is a powerful persuasion tool we can use to introduce everything else. We must Meet the Business Where It Is, not where we think it should be.
I explain that one use case and a study don’t guarantee we will see the same outcome. Data and models are similar. Based on what we know, this is the most likely outcome, but it’s not a guaranteed outcome. Models tell us that, based on the data we have, this is the most likely outcome. It’s also not a guarantee.
The comparison is much easier to digest than defining the differences between deterministic and stochastic. C-level leaders are used to making decisions that have risks associated with them. They run a stochastic business even if they aren’t explicitly aware of it.
I need to use data and models to prescribe a new discounting strategy. I put my own CEO hat on to explain this part of the process. Someone from the data team explained the risk associated with aggressive discounting strategies. But I also know we’re leaving money on the table and letting competitors take customers from us if we don’t discount our products. No discounts aren’t an option I can choose.
We want to train CEOs to ask the right questions because we can answer them with data and models. A stochastic question doesn’t have a single answer. It has a most likely answer based on what we know, but there are also less likely alternatives. Most models aren’t built to present all options in response to a question or all the risks associated with an option.
Models must be explainable to be trustworthy. Again, we must tell a story to answer decision-makers’ questions. The right questions to ask are, “What risks and gains come with discounting strategies? Which discounting strategy has the highest gain while respecting our risk tolerance?”
Stochastic questions are built with the understanding that the model providing the answer is also stochastic. In this case, there are discounting strategies that mitigate the most significant risks.
Risk 1: Undermining the courses’ perceived value. Let’s face it. Low prices raise questions about quality. We call inexpensive products and poorly built products cheap for a reason. If products are always on sale, the perception is that they aren’t good enough to command a premium price. The discounting strategy must make customers feel like they got a bargain vs. buying a cheap product.
Risk 2: Training customers to wait until the product goes on sale again. I worked at a casino slot machine manufacturer that set aggressive sales targets for itself. Its sales team would begin to panic near the end of each quarter because they weren’t on pace to hit their targets. The head of sales would authorize discounts to get customers who were on the fence to buy.
You can probably guess what happened after 2 years of this. Customers held off on ordering until the end of the quarter to get the discount. Margins dropped even though the company kept making its sales targets. Discounting strategy must make customers feel like they should act fast because they don’t know when they’ll get another shot.
Obviously, there are more factors, but this article is getting lengthy, and these two illustrate the point sufficiently. I can run the sale now because it aligns with a low-risk discounting strategy. That’s how we must present options to decision-makers.
Part one explains that software and data or models are different.
Part two explains the differences in a way non-data scientists will understand.
Part three uses storytelling and technical metaphors to help decision-makers synthesize the differences more broadly.
Part four teaches decision-makers to ask questions data teams can answer with data and models.
Part five explains why the model or data came up with the answer it did and where the risks are.
Act fast to take advantage of this special offer. It ends September 6th. The self-paced versions of my Data and AI Strategy, Data & AI Product Management, and Executive Data Leadership courses are heavily discounted, but they won’t be for long.