Greatly inspired by this paper, I have been thinking about business benchmarks for Machine Learning model performance. When a new model is built, how do we explain the impacts to the rest of the business?

‘AI and the Everything in the Whole Wide World Benchmark’ talks honestly about the flaws in the academic model benchmarks. GLUE, ImageNet, and many others are widely used to support claims of improved performance. State of the art performance is thrown around a lot. The paper explains several gaps in the current suite of benchmarks.

Limited Task Design
Arbitrarily Selected Tasks and Collections
Critical Misunderstandings of Domain Knowledge and Applications Problem Space
De-contextualized Data and Performance Reporting
Limited Scope
Benchmark Subjectivity
Inappropriate Community Use
Limits of Competitive Testing
Redirection of Focus for the Field
Justification for Practical out of Context or Unsafe Applications

Most of these have bullet points have parallel shortcomings in the way we benchmark models for the business.

High ROI Data Science

Benchmarking And Evaluating The Business…

Comments on this post are for paid subscribers