Mondays Are Variations On A Theme

This has been an amazing week for feedback and likes. Thank you to everyone who took the time to help me improve and benefit the community. I appreciate every subscriber and all the feedback you have provided.

Two posts are in the works.

Democratizing Experimentation: Processes And Platform

Getting Buy-In For The Data And Analytics Organization To Own Their Projects End To End

These are great topics. Based on your feedback, I will continue the leadership and causal machine learning series. Case studies and use cases will also become a larger part of my posts. Those have been popular and sparked discussions.

In music theory, variations on a theme from two foundations that complex music can be built on. The theme is the core innovation, and the composition begins by articulating the theme. The piece would be short and not that interesting if the innovation stood alone.

Enter the variations. The theme is changed noticeably while retaining an obvious connection to the original innovation. Sometimes two themes alternate, and variations extend their duel.

Research spawns variations on the original theme. There is a lineage in the citations. The origins of the new research are laid out, and I think that’s one of the most valuable insights from reading research. I get a glimpse into the researchers’ discovery and research process.

Machine learning research’s critics say the field has too many themes and not enough variation. Papers are often unsupported. Researchers claim their data and code are available upon request, but few respond to requests. In other fields, research that incorporates models faces a reproducibility crisis.

In music, variations test the theme’s quality. A shoddy theme produces even worse variations, but variations will also produce a better theme. Extending the theme is the musician’s tool for discovery and extrapolation. The parallels between music and science extend in many directions.

When researchers do not explore variations on their theme, publications are highly theoretical. How would this work in the real world? What applications would this be most successfully applied to? More importantly, will this work reliably in a production environment?

Applications, counterfactuals, gaps, bias, negative impacts, and reproducibility are often overlooked. Machine learning research has a wealth of themes and a poverty of variation. It is a symptom of our disconnect from the audience.

Composers are feet away from their most vocal critics. If their composition stinks, it is immediately revealed in a humiliating way. Some concerts were so long that people would bring their lunches. The audience would voice their discontent by throwing their food at the performers.

The broader scientific community is throwing their lunches at machine learning research, but the disconnect means it doesn’t ever get on the researchers. Our field rewards innovative publications and conference presentations with high-paying jobs. The research may never be implemented, but researchers are not held to that metric.

Incremental improvements or applications of existing research are not as well rewarded. The data scientist who figures out how to apply DALL-E2 to a real-world business problem will go unnoticed. Even when they provide significant business value, the field couldn’t care less. If you don’t believe me, name the data scientists at Microsoft who implemented GPT-3 into Copilot. Turning GPT-3 into a product required a significant level of effort, and the major conferences don’t have a track for that.

Before then, GPT-3 was a theme in search of variation. The research community applauds the theme and leaves the concert hall too soon for variation to be necessary. The researchers leave the stage, and the performers are left to take the rotten tomatoes thrown by an audience expecting more substance.

The field rewards research creation, and businesses reward value creation. Customers pay for products, and users adopt solutions. McDonald’s CEO summed it up nicely by saying the economics of robotic workforce automation did not work out. Others have said the field is headline-heavy and application-light. I used to spend time debunking the claims for senior leaders, but they’ve stopped reading the grandiose headlines.

The broader research community is questioning machine learning research’s validity. The business has growing doubts about data science’s value proposition. Users don’t trust, and customers don’t adopt. Applied data scientists and machine learning engineers have begun to ignore it. Applied research has no resemblance to what is presented at the major conferences.

The machine learning research community is at war with everyone and risking irrelevance. They need to open the door for variations and slow the pace of pumping out themes. In the process, many themes will be exposed and discarded. Progress stands on the foundations of failed research.

Even failed themes will end up in the citations section for variations that become validated themes. Something must spark progress, and failures work just as well as successes. “Well, that didn’t work, but what if they had tried…”

Our opportunity is to keep the door open so the research community can start to reconnect. Everyone is piling on, and they need someplace welcoming. Our field’s applied side can make room for them and help them transition. I want to see what happens when theoretical and applied research finds ways to amplify each other. That’s a concert worth listening to.

Vin