I\u2019ve They rarely have a clear idea of the project objective.<\/strong><\/p>\n This is one of the main obstacles data scientists face, especially now that Gen AI is taking over every domain.<\/p>\n But let\u2019s suppose that after some back and forth, the objective becomes clear. We managed to pin down a specific question to answer. For example:<\/p>\n I want to classify my customers into two groups according to their probability to churn: \u201chigh likelihood to churn\u201d and \u201clow likelihood to churn\u201d<\/strong><\/em><\/p>\n<\/blockquote>\n Well, now what? Easy, let\u2019s start building some models!<\/p>\n Wrong!<\/strong><\/p>\n If having a clear objective is rare, having a reliable\u00a0benchmark<\/strong>\u00a0is even rarer. <\/p>\n In my opinion, one of the most important steps in delivering a data science project is defining and agreeing on a\u00a0set of benchmarks<\/strong>\u00a0with the client.<\/p>\n In this blog post, I\u2019ll explain:<\/p>\n A\u00a0benchmark<\/strong>\u00a0is a standardized way to evaluate the performance of a model. It provides a reference point against which new models can be compared.<\/p>\n A benchmark needs two key components to be considered complete:<\/p>\n The concept at its core is simple: every time I develop a new model I compare it against both previous versions and the baseline models. This ensures improvements are real and tracked.<\/p>\n It is essential to understand that this baseline shouldn\u2019t be model or dataset-specific, but rather business-case-specific. It should be a general benchmark for a given business case.<\/p>\n If I encounter a new dataset, with the same business objective, this benchmark should be a reliable reference point.<\/p>\n Now that we\u2019ve defined what a benchmark is, let\u2019s dive into why I believe it\u2019s worth spending an extra project week on the development of a strong benchmark.<\/p>\n With a clear benchmark, every step in the model development will provide\u00a0immediate feedback<\/strong>, making the whole process more\u00a0intentional and data-driven<\/strong>.<\/p>\n I hope I\u2019ve convinced you of the importance of having a benchmark. Now, let\u2019s actually build one.<\/p>\n Let\u2019s start from the business question we presented at the very beginning of this blog post:<\/p>\n I want to classify my customers into two groups according to their probability to churn: \u201chigh likelihood to churn\u201d and \u201clow likelihood to churn\u201d<\/em><\/strong><\/p>\n<\/blockquote>\n For simplicity, I\u2019ll assume\u00a0no additional business constraints,<\/strong>\u00a0but in real-world scenarios, constraints often exist.<\/p>\n For this example, I am using\u00a0this dataset<\/em><\/strong><\/a>\u00a0(CC0: Public Domain<\/a>). The data contains some attributes from a company\u2019s customer base (e.g., age, sex, number of products, \u2026) along with their churn status.<\/p>\n Now that we have something to work on let\u2019s build the benchmark:<\/p>\n We are dealing with a churn use case, in particular, this is a\u00a0binary classification problem<\/strong>. Thus the main metrics that we could use are:<\/p>\n These are some of the \u201csimple\u201d metrics that could be used to evaluate the output of a model.<\/p>\n However<\/strong>, it is not an exhaustive list, standard metrics aren\u2019t always enough. In many use cases, it might be useful to\u00a0build custom metrics<\/strong>.<\/p>\n Let\u2019s assume that in our business case the\u00a0customers labeled as \u201chigh likelihood to churn\u201d are offered a discount.<\/strong>\u00a0This creates:<\/p>\n Following on this definition we can build a custom metric that will be crucial in our scenario:<\/p>\n When you are building\u00a0business-driven metrics<\/strong>\u00a0these are usually the most relevant. Such metrics could take any shape or form: Financial goals, minimum requirements, percentage of coverage and more.<\/p>\n Now that we\u2019ve defined our metrics, we can define a set of baseline models to be used as a reference.<\/p>\n In this phase, you should define a list of simple-to-implement model in their simplest possible setup. There is no reason at this state to spend time and resources on the optimization of these models, my mindset is:<\/p>\n If I had 15 minutes, how would I implement this model?<\/strong><\/p>\n<\/blockquote>\n In later phases of the model, you can add mode baseline models as the project proceeds.<\/p>\n In this case, I will use the following models:<\/p>\n Again, as in the case of the metrics, we can build\u00a0custom benchmarks<\/strong>.<\/p>\n Let\u2019s assume that in our business case the\u00a0the marketing team contacts every client who\u2019s<\/strong>:<\/p>\n Following this rule we can build this model:<\/p>\n To run the benchmark I will use the following class. The entry point is the method\u00a0 Now all we need is a prediction. For this example, I made a quick feature engineering and some hyperparameter tuning.<\/p>\n The last step is just to run the benchmark:<\/p>\n This generates a\u00a0comparison table<\/strong>\u00a0of all models across all metrics. Using this table, it is possible to draw concrete conclusions on the model\u2019s predictions and make informed decisions on the following steps of the process.<\/p>\n As we\u2019ve seen there are plenty of reasons why it is useful to have a benchmark. However, even though benchmarks are incredibly useful, there are some\u00a0pitfalls<\/strong>\u00a0to watch out for:<\/p>\n Benchmarks provide clarity, ensure improvements are measurable, and create a\u00a0shared reference point<\/strong>\u00a0between data scientists and clients. They help avoid the trap of assuming a model is performing well without proof and ensure that every iteration brings real value.<\/p>\n They also act as a\u00a0communication tool<\/strong>, making it easier to explain progress to clients. Instead of just presenting numbers, you can show clear comparisons that highlight improvements.<\/p>\n Here you can find a notebook with a full implementation from this blog post<\/em><\/a>.<\/em><\/p>\n The post How To Build a Benchmark for Your Models<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
\n
\nWhat is a benchmark?<\/h2>\n
\n
\nWhy building a benchmark is important<\/h2>\n
\n
\nHow I would build a benchmark<\/h2>\n
\n
1. Defining the metrics<\/h3>\n
\n
\n
# Defining the business case-specific reference metric\ndef financial_gain(y_true, y_pred): \n loss_from_fp = np.sum(np.logical_and(y_pred == 1, y_true == 0)) * 250 \n gain_from_tp = np.sum(np.logical_and(y_pred == 1, y_true == 1)) * 1000 \n return gain_from_tp - loss_from_fp<\/code><\/pre>\n2. Defining the benchmarks<\/h3>\n
\n
\n
import numpy as np \nimport xgboost as xgb \nfrom sklearn.neighbors import KNeighborsClassifier \n \nclass BinaryMean(): \n @staticmethod \n def run_benchmark(df_train, df_test): \n np.random.seed(21) \n return np.random.choice(a=[1, 0], size=len(df_test), p=[df_train['y'].mean(), 1 - df_train['y'].mean()]) \n \nclass SimpleXbg(): \n @staticmethod \n def run_benchmark(df_train, df_test): \n model = xgb.XGBClassifier() \n model.fit(df_train.select_dtypes(include=np.number).drop(columns='y'), df_train['y']) \n return model.predict(df_test.select_dtypes(include=np.number).drop(columns='y')) \n \nclass MajorityClass(): \n @staticmethod \n def run_benchmark(df_train, df_test): \n majority_class = df_train['y'].mode()[0] \n return np.full(len(df_test), majority_class) \n \nclass SimpleKNN(): \n @staticmethod \n def run_benchmark(df_train, df_test): \n model = KNeighborsClassifier() \n model.fit(df_train.select_dtypes(include=np.number).drop(columns='y'), df_train['y']) \n return model.predict(df_test.select_dtypes(include=np.number).drop(columns='y'))<\/code><\/pre>\n\n
# Defining the business case-specific benchmark\nclass BusinessBenchmark(): \n @staticmethod \n def run_benchmark(df_train, df_test): \n df = df_test.copy() \n df.loc[:,'y_hat'] = 0 \n df.loc[(df['IsActiveMember'] == 0) & (df['Age'] >= 50), 'y_hat'] = 1 \n return df['y_hat']<\/code><\/pre>\nRunning the benchmark<\/h3>\n
compare_with_benchmark()<\/code>\u00a0that, given a prediction, runs all the models and calculates all the metrics.<\/p>\nimport numpy as np \n \nclass ChurnBinaryBenchmark(): \n def __init__( \n\t self, \n metrics = [], \n benchmark_models = [], \n ): \n self.metrics = metrics \n self.benchmark_models = benchmark_models \n \n def compare_pred_with_benchmark( \n\t self, \n df_train, \n df_test, \n my_predictions, \n ): \n \n output_metrics = { \n 'Prediction': self._calculate_metrics(df_test['y'], my_predictions) \n } \n dct_benchmarks = {} \n \n for model in self.benchmark_models: \n dct_benchmarks[model.__name__] = model.run_benchmark(df_train = df_train, df_test = df_test) \n output_metrics[f'Benchmark - {model.__name__}'] = self._calculate_metrics(df_test['y'], dct_benchmarks[model.__name__]) \n \n return output_metrics \n \n def _calculate_metrics(self, y_true, y_pred): \n return {getattr(func, '__name__', 'Unknown') : func(y_true = y_true, y_pred = y_pred) for func in self.metrics}<\/code><\/pre>\nbinary_benchmark = ChurnBinaryBenchmark( \n metrics=[f1_score, precision_score, recall_score, tp, tn, fp, fn, financial_gain], \n benchmark_models=[BinaryMean, SimpleXbg, MajorityClass, SimpleKNN, BusinessBenchmark] \n ) \n \nres = binary_benchmark.compare_pred_with_benchmark( \n df_train=df_train, \n df_test=df_test, \n my_predictions=preds, \n) \n \npd.DataFrame(res)<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-134-1024x289.png?resize=1024%2C289&ssl=1\")
\nSome drawbacks<\/h2>\n
\n
\nFinal thoughts<\/h2>\n