High ROI Data Science

High ROI Data Science

Share this post

High ROI Data Science
High ROI Data Science
Benchmarking And Evaluating The Business Value Of Machine Learning Models

Benchmarking And Evaluating The Business Value Of Machine Learning Models

I talk about connecting business metrics to model metrics. Benchmarks are a good framework for achieving that. However, you have to avoid the gaps that academic benchmarks suffer from.

Vin Vashishta's avatar
Vin Vashishta
Dec 06, 2021
∙ Paid
3

Share this post

High ROI Data Science
High ROI Data Science
Benchmarking And Evaluating The Business Value Of Machine Learning Models
Share

Greatly inspired by this paper, I have been thinking about business benchmarks for Machine Learning model performance. When a new model is built, how do we explain the impacts to the rest of the business?

‘AI and the Everything in the Whole Wide World Benchmark’ talks honestly about the flaws in the academic model benchmarks. GLUE, ImageNet, and many others are widely used to support claims of improved performance. State of the art performance is thrown around a lot. The paper explains several gaps in the current suite of benchmarks.

  • Limited Task Design

  • Arbitrarily Selected Tasks and Collections

  • Critical Misunderstandings of Domain Knowledge and Applications Problem Space

  • De-contextualized Data and Performance Reporting

  • Limited Scope

  • Benchmark Subjectivity

  • Inappropriate Community Use

  • Limits of Competitive Testing

  • Redirection of Focus for the Field

  • Justification for Practical out of Context or Unsafe Applications

Most of these have bullet points have parallel shortcomings in the way we benchmark models for the business.

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2025 Vin Vashishta
Privacy ∙ Terms ∙ Collection notice
Start writingGet the app
Substack is the home for great culture

Share