The Inversion Of The Yield Curve And The Fracture In Data Science
These are confusing times, both economically and technically. As data science's power progresses and proves out, so do our critics. Data science is having a quantum uncertainty moment. Data Science methods are both effective and ineffective.
Between 2015 and 2018, I was brought in by an avalanche of clients who needed a model turnaround. That term is a play on the term business turnaround when consultants are brought in to right the ship at a troubled business. I was brought in to right the ship at a troubled data science team.
Clients had models in production that the business depended on. They had worked well for a period and then abruptly failed. By the time the failure happened, the original data scientist was at another company. One of the problems with the rapid job-hopping our field enables is we rarely stay long enough to see our approaches' shortcomings. We improve most by fixing and understanding the root causes of failure.
In 3 years, I had 7 model turnaround projects. My experience cleaning up messes came from cleaning up after myself. My only advantages were making the mistakes sooner than everyone else and following my projects for longer.
I was pushing to production in 2012 and 2013. Those were basic models for limited, well-defined use cases. By 2015, I had refined my approach to start with expert systems and improve functionality with models. I kept the human in the loop for continuous feedback. The dataset created by their interventions was effective for model retraining and improvement. Changes in the frequency of interventions were my validation method.
I used the same paradigm for processes I could not control in the same way. For these problems spaces, there was no opportunity to have a human in the loop who would put up with the continuous improvement cycles. I replaced the expert systems with software automation that gathered data about the workflow. I improved using observational studies on that dataset and validated using more reliable experimental methods.
We did not have MLOps then, so I made a version of it myself. My clients' requirements drove my design. The models need to work and not stop working randomly. Tradeoffs were required to achieve that. The biggest was the model was always partial. It covered a set of classes with high reliability, and that was it. Continuous improvement expanded the number of classes in that set.
That process is expensive and time-consuming. Model reliability was very high, but the ceiling for diminishing returns came up quickly. This approach is only feasible for very high-value use cases.
In 2015, that was fine because machine learning was the domain of very large companies. They have problems of complexity at a scale where improving response time, reducing their reliance on large workforces, and improving outcome quality have significant business impacts. The cost savings and revenue generated at scale more than supported a more rigorous approach. They had hard problems that could not be solved using traditional software engineering approaches.