From Business Problem To Data Science Experiment
Over the past month, I have been conducting a few simple experiments for my business. It’s a good use case for how I structure experiments and solve problems with data. When I add data to strategy planning, even simple data like I’ll use in this post, outcomes improve.
Data strategy must be an extension of business strategy. I reevaluate my business strategy quarterly and look for improvements or new opportunities. My starting question for this post is, are there opportunities to improve revenue based on my existing product lines?
V Squared (us) has identified a high-value opportunity to improve revenue. My social media presence is under-monetized. Why do I think that?
Discovering And Assessing The Opportunity
I estimated the opportunity by interviewing other creators and pulling data from third-party sources. I discovered that most content creators in the data and broader technology space were pulling in more revenue than I do. Some make much of their income from content and marketing.
Part of my problem was pricing. I wasn’t charging enough, so I repriced to match current market rates.
I was also missing a key, high-end service offering. I added whitepapers which are now my most expensive content marketing product. I wonder how many opportunities have sailed by because I didn’t list those sooner.
In my strategy certification course, I explain how dangerous opaque parts of the business are. Transparent aspects of the company generate data and can be easily monitored. Opaque parts of the business generate no data, and the implications for V Squared were significant.
Like a rookie, I wasn’t focused on my content marketing product line, which led to a lack of data gathering. I had an opaque section of my business model. I brought transparency with simple data gathering and made immediate improvements.
The data gathering initiative cost 14 hours of my time ($5600) and $75 to buy third-party reports. The initiative will be profitable with 1 whitepaper sale. In September, increased pricing alone has gotten me almost halfway to breaking even.
During my client assessment phase, I look for opaque parts of the business and operating models. My earliest data gathering initiatives focus on removing opacity, and there are immediate gains. Those are my early wins, and they support more significant initiatives, including experiments.
Moving To Analytics
I analyzed my social media community composition to assess my optimal target market. By optimal, I mean the customers I can provide the highest value for. There are two sides to value: what I can deliver and the customer segment’s perception of what I provide.
The community analytics described a surface-level view of what I can deliver.
C-level and Founders: 26,000
Directors and VPs: 18,000
Senior Technical ICs and Managers: 62,000
Total Data Professionals (ICs): 26,000
Combined IT, Engineering, and Research: 68,000
I am a small business, so optimization is critical. The strategic ramifications of my current operating model limit the number of customers I can acquire and service. Again, the connection between data science initiatives and business reality is critical.
My highest value segments based on audience and perceived value are:
B2B technology products servicing data organizations
Data R&D services vendors
Educational products servicing technical ICs and leaders
Technology events
I will not get specific about how I determined audience value and customers’ perceived value. That’s the secret sauce. One basic formula the industry uses is the cost per 1000 viewers from a specific segment multiplied by the average number of those viewers per marketing post for social media marketing.
I have my own model that is a reliable descriptor of value for each segment. One critical insight for the B2B technology products companies is their sales process. My specific community composition is valuable to them, and my marketing reflects that. Even descriptive models produce insights that are valuable across multiple use cases. I know who to target and how to explain my value proposition to trigger a buying response.
Understanding decision chains is a critical success factor. My customers’, customers make buying decisions in chains. The people closest to the problem and need are senior technical ICs. They escalate their needs to managers and directors. Mid-level leaders then escalate to C-level leaders during the budgeting process. C-level leaders approve and sign the checks.
My community on LinkedIn has a strong representation in all three parts of the decision chain. An IC who sees the post and connects with the pain point hits the like button. That makes it more likely that one of their mid-level managers sees it and continues the like chain. This can happen at any level of the decision chain and cascade to the other parts.
These statements result from aggregating multiple descriptive models into a dashboard. I cannot predict the performance of my approach to selling, but the data gives me ideas to test out. My experience and the data combine to improve outcomes.
Moving From Analytics To Experiments
In my strategy certification, I talk about how introducing analytics to a workflow creates data about changes to that workflow. Those datasets make predictive models possible. It starts as basic A/B testing, but more advanced experiments are usually possible.
An A/B test I’m running now is with my LinkedIn headline. My baseline was:
“Technical Strategy & Strategic Leadership Certification Courses | AI & Strategy Influencer Since 2014 | Business & Executive Coach”
I A/B tested 2 versions against each other.
“Unlocking Your Data Organization’s Value | Teaching Data Professionals & Leaders To Fasttrack Your Career”
“Unlocking Your Data Organization’s Value | Teaching Data Professionals & Leaders To Fast Track Your Career”
Both are incorrect uses of fast-track. I will survey 200 randomly selected people from all who viewed my profile while each version and the baseline were live. My questions are:
“Did you know I offer courses?”
“If you do, what is your impression of my courses’ impact on students’ careers?”
I am looking for a higher percentage of responses, including some synonym of accelerate with the fast track variations than baseline. If that result is present, I will see if there’s a significant difference between the two variants. This is a very low level of effort and potentially interesting experiment. I will need to use more rigorous methods for any follow-ups.
In a more complex experiment, I am looking for a solution to a marketing challenge. LinkedIn’s recommender doesn’t reward posts that have links, and most marketing posts contain links. Those posts’ reach is very limited, so my customers don’t get the highest possible value.
If I can find a way to post links and not get penalized by the recommender, my marketing posts will have a competitive advantage. I would also be able to support a premium price point.
How do I advertise while avoiding the penalty? Hypothesis: putting an advertisement in an image will effectively avoid the penalty. Experiment: put a website link in images and measure website visits for significant changes.
I already have a baseline of data from my website over a time when I did not put my website in images. My first round of experiments put my website DataScience.vin at the bottom right corner of informational shares. I used single-image infographics and multi-page PDFs.
After two weeks of measurement, I saw a 50% increase in website traffic. The number of pages didn’t seem to matter. The results were significant enough for me to run a second round of experiments. I put my website into the same place on 4 memes I ran over 3 weeks.
The change in content type was my focus. Memes get higher views and engagement (800K-2M views) than informational shares (25K-50K views), but would a less informative post get the same response? Website traffic increased by 2.5X-3X over those 3 weeks vs. any 3 weeks over the 2 months I ran informational shares.
Volume is a good thing. It isn’t a 1:1 improvement which could indicate informational shares are higher impact. However, the increase in views appears to have made up for the decrease in content quality.
Confounders And Conclusions
The hardest part about experiments is confounding. I must look at what else could have impacted the outcomes. With online experiments, this is a nightmare.
Controlling for confounders meant keeping the variables that I controlled stable. Before I started, I had to stop introducing other sources that could increase traffic to my website. LinkedIn had to be my only promotional change for three months.
I could not attribute all new traffic to the post image. I took out any traffic that came directly from LinkedIn and other sources like Google searches. The image was not a clickable link, so the only new traffic that mattered was direct. I measured people who typed the link in and came to my site.
My most significant uncontrolled source of confounding is the increase in profile views. I don’t know how many people typed in the link after seeing a post image vs. finding it in my LinkedIn bio. Before starting the experiment, I should have removed it, but I did not realize the website was displayed as plain text, not a link.
That’s a problem because views increased by 50% with the memes. I got thousands more profile views and have no way of sorting those out. These kinds of experimental flaws are common. No matter how many eyes review the plan, assumptions always creep in.
The problem is many data science organizations don’t go to this level. Advertising the 2.5X-3X improvement goes straight to senior leaders, who make decisions based on what they believe to be complete information. Even worse, in most cases, the assumption will hold, and the data science team will believe their methods are sound. Ask Zillow what happens when the luck runs out, and assumptions fail in a critical use case.
I am balancing two different parts: certainty and the risks of my assumptions failing to hold. I am selling content marketing services. Is a single social media post doing half as well as advertised going to have a significant business impact on my customers? No.
It will have a value impact, and that’s the risk I must mitigate. It would be irresponsible to price my services based on the maximum measured impact, knowing confounders are involved. I will use the numbers for advertising, but I will price the product to include the potential confounding.
That is a responsible balance. I have enough confidence in my results to put my name behind them, but not enough to put my clients’ money behind them.