Many businesses hired data scientists just to say they were doing data science. It sounds crazy, but this is a very common situation. I coach 3-4 people a week who are in the position of a data scientist in a company that doesn't care about data.
Most are relegated to doing basic reporting and analytics. It's tough because these data scientists know self-service tools could replace the data team. They are worried about their jobs and demoralized by the low-end work.
Many data scientists stuck in this situation feel their skills are degrading. That's valid. When they interview for new roles, it's obvious, based on their answers to interview questions, that they've spent one or more years not delivering data science projects. Companies are more focused than ever on hiring people who can deliver models to production.
A Bottom To Top Leadership Problem
They ask their manager to advocate for them, but their manager doesn't understand how. I'm on both sides of this fence. I also mentor leaders from other technical domains and help them upskill to improve their effectiveness. Technical management at most companies is filled with former cloud architects and software engineers. They get pressed into service leading data teams or solo data scientists.
It takes months to upskill these leaders. Most have no goals or KPIs connected with data initiatives. Every minute they spend on the data team is a minute away from initiatives they are accountable for. For these managers, sticking their necks out for a data initiative has no rewards, and they'll be blamed if the initiative fails.
Asking for more money to build out data infrastructure or increase headcount doesn't make sense to them. Again, they'd rather spend that money on the teams delivering products they are incentivized for. Eventually, data scientists must go around their managers and advocate for themselves.
They pitch projects to senior leaders, but those don't go anywhere. The lack of incentivization runs to the top of the org chart. On the other end, leaders tell them to be more strategic and business-value focused. Leadership openly questions the data team's ROI and asks data scientists to deliver higher-value projects. It's a catch-22.
A New Kind Of Technical Debt
There's no easy way out of this situation. It takes a whole lot of non-data science work to succeed in a company with this mentality and culture. First, this isn't your fault and fixing it's not your job. What you're dealing with are strategy and cultural debt. We've all heard of technical debt, and you're probably managing a significant amount of that too.
However, there are other types of debt, and the fact that you're in this situation means you're probably dealing with one or both. Cultural debt is a huge challenge to overcome. Most businesses are set up to cover up mistakes. When data contradicts firmly held beliefs, people refuse to use it.
I hear this story often. Senior leaders are willing to accept anything data that agrees with them. They'll endlessly question and eventually reject data that disagree with their thoughts. For these types of senior leaders, the only accurate data is the data that reinforces their biases. Business culture has everything to do with this.
In most companies, you're punished for bad decisions. In this culture, any data contradicting a leader's decision gets them in trouble. There are negative consequences for them in simply revealing the problem. What's worse, there's no incentivization to fix it.
Addressing the problem means they get blamed twice. They get dinged for making a bad decision when they expose the problem. When they deliver the solution, they are hit again for spending money due to bad decision-making. This leads to a cover-up culture. Senior leadership will often spend more resources covering up problems than it would cost to fix them.
Strategic debt is a lack of structure to support monetizing data and AI products. Even if the business decides to commit to a data science initiative, it isn't built to monetize the outcome. Internal teams can't integrate data and AI products into their workflows. Customer-facing products aren't designed with space for data and AI features.
The data product manager and technical strategist roles don't exist. There's no data and AI strategy or product strategy to support monetization. The team has no link to business value and no C-level leader advocating for them. I don't know how businesses think their data teams will succeed without support at any level. But that's the pervasive perception.
What Should You Do?
If you find yourself in this situation, there's a lot of work in front of you. The first question you must answer is, do you really want to do it? Stepping forward and taking the lead in the strategic side of data has significant career upsides. Establishing a track record of success in this environment is a setup for a high-end career path. You'll be in high demand with an easy runway to product management, strategy, and leadership roles.
In the near term, you face high effort and low return. The current business will take months to come around to see the value that you can deliver. Establishing a track record of success typically takes three to six months. I don't blame people for deciding not to put the cape on. A valid response to this scenario is to find a role in a company that'll be far more supportive.
If you decide to stay, the first piece of the puzzle is getting beyond your immediate manager. You'll spend the first few weeks building relationships with people above your leader and in other organizations. Treat the data team like a startup and leadership like your VCs.