In Machine Learning<\/a>, ensembling <\/strong>is a broad term that refers to any technique that creates predictions by combining the predictions from multiple models. If there is more than one model involved in making a prediction, the technique is using ensembling!<\/p>\n Ensembling approaches can often improve the performance of a single model. Ensembling can help reduce:<\/p>\n Bagging and boosting are both ensemble methods that can perform much better than their single-model counterparts. Let\u2019s get into the details of these now!<\/p>\n Bagging is a specific ensembling technique that is used to reduce the variance of a predictive model. Here, I\u2019m talking about variance in the machine learning sense \u2013 i.e., how much a model varies with changes to the training dataset \u2013 not variance in the statistical sense which measures the spread of a distribution. Because bagging helps reduce an ML model\u2019s variance, it will often improve models that are high variance (e.g., decision trees and KNN) but won\u2019t do much good for models that are low variance (e.g., linear regression).<\/p>\n Now that we understand when<\/strong><\/em> bagging helps (high variance models), let\u2019s get into the details of the inner workings to understand how <\/strong><\/em>it helps! The bagging algorithm is iterative in nature \u2013 it builds multiple models by repeating the following three steps:<\/p>\n The collection of models created in this process is called an ensemble. When it is time to make a prediction, each model in the ensemble makes its own prediction \u2013 the final bagged prediction is the average (for regression) or majority vote (for classification) of all of the ensemble\u2019s predictions.<\/p>\n Now that we understand how bagging works, let\u2019s take a few minutes to build an intuition for why it works. We\u2019ll borrow a familiar idea from traditional statistics: sampling to estimate a population mean.<\/p>\n In statistics, each sample drawn from a distribution is a random variable. Small sample sizes tend to have high variance and may provide poor estimates of the true mean. But as we collect more samples, the average of those samples becomes a much better approximation of the population mean.<\/p>\n Similarly, we can think of each of our individual decision trees as a random variable \u2014 after all, each tree is trained on a different random sample of the data! By averaging predictions from many trees, bagging reduces variance and produces an ensemble model that better captures the true relationships in the data.<\/p>\n Bagging Example<\/strong><\/p>\n We will be using the load_diabetes1<\/sup> dataset from the scikit-learn Python<\/a> package to illustrate a simple bagging example. The dataset has 10 input variables \u2013 Age, Sex, BMI, Blood Pressure and 6 blood serum levels (S1-S6). And a single output variable that is a measurement of disease progression. The code below pulls in our data and does some very simple cleaning. With our dataset established, let\u2019s start modeling!<\/p>\n For our example, we will use basic decision trees as our base models for bagging. Let\u2019s first verify that our decision trees are indeed high variance. We will do this by training three decision trees on different bootstrapped datasets and observing the variance of the predictions for a test dataset. The graph below shows the predictions of three different decision trees on the same test dataset. Each dotted vertical line is an individual observation from the test dataset. The three dots on each line are the predictions from the three different decision trees.<\/p>\n In the chart above, we see that individual trees can give very different predictions (spread of the three dots on each vertical line) when trained on bootstrapped datasets. This is the variance we have been talking about!<\/p>\n Now that we see that our trees aren\u2019t very robust to training samples \u2013 let\u2019s average the predictions to see how bagging can help! The chart below shows the average of the three trees. The diagonal line represents perfect predictions. As you can see, with bagging, our points are tighter and more centered around the diagonal.<\/p>\n We\u2019ve already seen significant improvement in our model with the average of just three trees. Let\u2019s beef up our bagging algorithm with more trees!<\/p>\n Here is the code to bag as many trees as we want:<\/p>\n The functions above are very simple, the first trains the bagging ensemble model, the second takes the ensemble (simply a list of trained trees) and makes predictions given a dataset.<\/p>\n With our code established, let\u2019s run multiple ensemble models and see how our out-of-bag predictions change as we increase the number of trees.<\/p>\n Admittedly, this chart looks a little crazy. Don\u2019t get too bogged down with all of the individual data points, the lines dashed tell the main story! Here we have 1 basic decision tree model and 3 bagged decision tree models \u2013 with 3, 50 and 150 trees. The color-coded dotted lines mark the upper and lower ranges for each model\u2019s residuals. There are two main takeaways here: (1) as we add more trees, the range of the residuals shrinks and (2) there is diminishing returns to adding more trees \u2013 when we go from 1 to 3 trees, we see the range shrink a lot, when we go from 50 to 150 trees, the range tightens just a little.<\/p>\n Now that we\u2019ve successfully gone through a full bagging example, we are about ready to move onto boosting! Let\u2019s do a quick overview of what we covered in this section:<\/p>\n Okay, let\u2019s move on to boosting!<\/p>\n With bagging, we create multiple independent models \u2013 the independence of the models helps average out the noise of individual models. Boosting is also an ensembling technique; similar to bagging, we will be training multiple models\u2026. But very different from bagging, the models we train will be dependent<\/strong>. Boosting is a modeling technique that trains an initial model and then sequentially trains additional models to improve the predictions of prior models. The primary target of boosting is to reduce bias \u2013 though it can also help reduce variance.<\/p>\n We\u2019ve established that boosting iteratively improves predictions \u2013 let\u2019s go deeper into how. Boosting algorithms can iteratively improve model predictions in two ways:<\/p>\n Because boosting\u2019s main goal is to reduce bias, it works well with base models that typically have more bias (e.g., shallow decision trees). For our examples, we are going to use shallow decision trees as our base model \u2013 we will only cover the residual prediction approach in this article for brevity. Let\u2019s jump into the boosting example!<\/p>\n Predicting prior residuals<\/strong><\/p>\n The residuals prediction approach starts off with an initial model (some algorithms provide a constant, others use one iteration of the base model) and we calculate the residuals of that initial prediction. The second model in the ensemble predicts the residuals of the first model. With our residual predictions in-hand, we add the residual predictions to our initial prediction (this gives us residual corrected predictions) and recalculate the updated residuals\u2026. we continue this process until we have created the number of base models we specified. This process is pretty simple, but is a little hard to explain with just words \u2013 the flowchart below shows a simple, 4-model boosting algorithm.<\/p>\n When boosting, we need to set three main parameters: (1) the number of trees, (2) the tree depth and (3) the learning rate. I\u2019ll spend a little time discussing these inputs now.<\/p>\n Number of Trees<\/strong><\/p>\n For boosting, the number of trees means the same thing as in bagging \u2013 i.e., the total number of trees that will be trained for the ensemble. But, unlike boosting, we should not<\/strong> err on the side of more trees! The chart below shows the test RMSE against the number of trees for the diabetes dataset.<\/p>\n This shows that the test RMSE drops quickly with the number of trees up until about 200 trees, then it starts to creep back up. It looks like a classic \u2018overfitting\u2019 chart \u2013 we reach a point where more trees becomes worse for the model. This is a key difference between bagging and boosting \u2013 with bagging, more trees eventually stop helping<\/em>, with boosting more trees eventually start hurting<\/em>!<\/p>\n With bagging, more trees eventually stops helping<\/strong><\/em>, with boosting more trees eventually starts hurting<\/strong><\/em>!<\/p>\n<\/blockquote>\n We now know that too many trees are bad, and too few trees are bad as well. We will use hyperparameter tuning to select the number of trees. Note \u2013 hyperparameter tuning is a huge subject and way outside of the scope of this article. I\u2019ll demonstrate a simple grid search with a train and test dataset for our example a little later.<\/p>\n Tree Depth<\/strong><\/p>\n This is the maximum depth for each tree in the ensemble. With bagging, trees are often allowed to go as deep they want because we are looking for low bias, high variance models. With boosting however, we use sequential models to address the bias in the base learners \u2013 so we aren\u2019t as concerned about generating low-bias trees. How do we decide how the maximum depth? The same technique that we\u2019ll use with the number of trees, hyperparameter tuning.<\/p>\n Learning Rate<\/strong><\/p>\n The number of trees and the tree depth are familiar parameters from bagging (although in bagging we often didn\u2019t put a limit on the tree depth) \u2013 but this \u2018learning rate\u2019 character is a new face! Let\u2019s take a moment to get familiar. The learning rate is a number between 0 and 1 that is multiplied by the current model\u2019s residual predictions before it is added to the overall predictions.<\/p>\n Here\u2019s a simple example of the prediction calculations with a learning rate of 0.5. Once we understand the mechanics of how the learning rate works, we will discuss the why the learning rate is important.<\/p>\n So, why would we want to \u2018discount\u2019 our residual predictions, wouldn\u2019t that make our predictions worse? Well, yes and no. For a single iteration, it will likely make our predictions worse \u2013 but, we are doing multiple iterations. For multiple iterations, the learning rate keeps the model from overreacting to a single tree\u2019s predictions. It will probably make our current predictions worse, but don\u2019t worry, we will go through this process multiple times! Ultimately, the learning rate helps mitigate overfitting in our boosting model by lowering the influence of any single tree in the ensemble. You can think of it as slowly turning the steering wheel to correct your driving rather than jerking it. In practice, the number of trees and the learning rate have an opposite relationship, i.e., as the learning rate goes down, the number of trees goes up. This is intuitive, because if we only allow a small amount of each tree\u2019s residual prediction to be added to the overall prediction, we are going to need a lot more trees before our overall prediction will start looking good.<\/p>\n Ultimately, the learning rate helps mitigate overfitting in our boosting model by lowering the influence of any single tree in the ensemble.<\/p>\n<\/blockquote>\n Alright, now that we\u2019ve covered the main inputs in boosting, let\u2019s get into the Python coding! We need a couple of functions to create our boosting algorithm:<\/p>\n Here are the functions written in Python:<\/p>\n Sweet, let\u2019s make a model on the same diabetes dataset that we used in the bagging section. We\u2019ll do a quick grid search (again, not doing anything fancy with the tuning here) to tune our three parameters and then we\u2019ll train the final model using the And our winner is While our boosting ensemble model seems to capture the trend reasonably well, we can see off the bat that it isn\u2019t predicting as well as the bagging model. We could probably spend more time tuning \u2013 but it could also be the case that the bagging approach fits this specific data better. With that said, we\u2019ve now earned an understanding of bagging and boosting \u2013 let\u2019s compare them in the next section!<\/p>\n We\u2019ve covered bagging and boosting separately, the table below brings all the information we\u2019ve covered to concisely compare the approaches:<\/p>\n Note: In this article, we wrote our own bagging and boosting code for educational purposes. In practice you will just use the excellent code that is available in Python packages or other software. Also, people rarely use \u2018pure\u2019 bagging or boosting \u2013 it is much more common to use more advanced algorithms that modify the plain vanilla bagging and boosting to improve performance. <\/p>\n Bagging and boosting are powerful and practical ways to improve weak learners like the humble but flexible decision tree. Both approaches use the power of ensembling to address different problems \u2013 bagging for variance, boosting for bias. In practice, pre-packaged code is almost always used to train more advanced machine learning models that use the main ideas of bagging and boosting but, expand on them with multiple improvements.<\/p>\n I hope that this has been helpful and interesting \u2013 happy modeling!<\/p>\n The post Strength in Numbers: Ensembling Models with Bagging and Boosting<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
Creating powerful models with ensembling<\/h2>\n
\n
Bagging: Adding stability to ML models<\/h2>\n
\n
# pull in and format data\nfrom sklearn.datasets import load_diabetes\n\ndiabetes = load_diabetes(as_frame=True)\ndf = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)\ndf.loc[:, 'target'] = diabetes.target\ndf = df.dropna()<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/tree_variance-1-1024x768.jpg?resize=1024%2C768&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/3_tree_model-1024x768.jpg?resize=1024%2C768&ssl=1\")
def train_bagging_trees(df, target_col, pred_cols, n_trees):\n\n '''\n Creates a decision tree bagging model by training multiple \n decision trees on bootstrapped data.\n\n inputs\n df (pandas DataFrame) : training data with both target and input columns\n target_col (str) : name of target column\n pred_cols (list) : list of predictor column names\n n_trees (int) : number of trees to be trained in the ensemble\n\n output:\n train_trees (list) : list of trained trees\n \n '''\n\n train_trees = []\n \n for i in range(n_trees):\n \n # bootstrap training data\n temp_boot = bootstrap(train_df)\n\n #train tree\n temp_tree = plain_vanilla_tree(temp_boot, target_col, pred_cols)\n\n # save trained tree in list\n train_trees.append(temp_tree)\n\n return train_trees\n\ndef bagging_trees_pred(df, train_trees, target_col, pred_cols):\n\n '''\n Takes a list of bagged trees and creates predictions by averaging \n the predictions of each individual tree.\n \n inputs\n df (pandas DataFrame) : training data with both target and input columns\n train_trees (list) : ensemble model - which is a list of trained decision trees\n target_col (str) : name of target column\n pred_cols (list) : list of predictor column names\n\n output:\n avg_preds (list) : list of predictions from the ensembled trees \n \n '''\n\n x = df[pred_cols]\n y = df[target_col]\n\n preds = []\n # make predictions on data with each decision tree\n for tree in train_trees:\n temp_pred = tree.predict(x)\n preds.append(temp_pred)\n\n # get average of the trees' predictions\n sum_preds = [sum(x) for x in zip(*preds)]\n avg_preds = [x \/ len(train_trees) for x in sum_preds]\n \n return avg_preds <\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/preds_by_tree_count_v2-1024x768.jpg?resize=1024%2C768&ssl=1\")
\n
Boosting: Reducing bias in weak learners<\/h2>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/Boosting-Error-Predictions-Page-2-2-1024x664.jpeg?resize=1024%2C664&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/boosting_error_by_trees-1024x768.jpg?resize=1024%2C768&ssl=1\")
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/learning_reate-1-1024x243.jpeg?resize=1024%2C243&ssl=1\")
\n
\n
# same base tree function as in prior section\ndef plain_vanilla_tree(df_train, \n target_col,\n pred_cols,\n max_depth = 3,\n weights=[]):\n\n X_train = df_train[pred_cols]\n y_train = df_train[target_col]\n\n tree = DecisionTreeRegressor(max_depth = max_depth, random_state=42)\n if weights:\n tree.fit(X_train, y_train, sample_weights=weights)\n else:\n tree.fit(X_train, y_train)\n\n return tree\n\n# residual predictions\ndef boost_resid_correction(df_train,\n target_col,\n pred_cols,\n num_models,\n learning_rate=1,\n max_depth=3):\n '''\n Creates boosted decision tree ensemble model.\n Inputs:\n df_train (pd.DataFrame) : contains training data\n target_col (str) : name of target column\n pred_col (list) : target column names\n num_models (int) : number of models to use in boosting\n learning_rate (float, def = 1) : discount given to residual predictions\n takes values between (0, 1]\n max_depth (int, def = 3) : max depth of each tree model\n\n Outputs:\n boosting_model (dict) : contains everything needed to use model\n to make predictions - includes list of all\n trees in the ensemble \n '''\n\n\n\n # create initial predictions\n model1 = plain_vanilla_tree(df_train, target_col, pred_cols, max_depth = max_depth)\n initial_preds = model1.predict(df_train[pred_cols])\n df_train['resids'] = df_train[target_col] - initial_preds\n \n # create multiple models, each predicting the updated residuals\n models = []\n for i in range(num_models):\n temp_model = plain_vanilla_tree(df_train, 'resids', pred_cols)\n models.append(temp_model)\n temp_pred_resids = temp_model.predict(df_train[pred_cols])\n df_train['resids'] = df_train['resids'] - (learning_rate*temp_pred_resids)\n \n boosting_model = {'initial_model' : model1,\n 'models' : models,\n 'learning_rate' : learning_rate,\n 'pred_cols' : pred_cols}\n \n return boosting_model\n\n# This function takes the residual boosted model and scores data\ndef boost_resid_correction_predict(df,\n boosting_models,\n chart = False):\n\n '''\n Creates predictions on a dataset given a boosted model.\n \n Inputs:\n df (pd.DataFrame) : data to make predictions\n boosting_models (dict) : dictionary containing all pertinent\n boosted model data\n chart (bool, def = False) : indicates if performance chart should\n be created\n Outputs:\n pred (np.array) : predictions from boosted model\n rmse (float) : RMSE of predictions\n '''\n\n # get initial predictions\n initial_model = boosting_models['initial_model']\n pred_cols = boosting_models['pred_cols']\n pred = initial_model.predict(df[pred_cols])\n\n # calculate residual predictions from each model and add\n models = boosting_models['models']\n learning_rate = boosting_models['learning_rate']\n for model in models:\n temp_resid_preds = model.predict(df[pred_cols])\n pred += learning_rate*temp_resid_preds\n\n if chart:\n plt.scatter(df['target'], \n pred)\n plt.show()\n\n rmse = np.sqrt(mean_squared_error(df['target'], pred))\n\n return pred, rmse\n <\/code><\/pre>\nboost_resid_correction<\/code> function.<\/p>\n# tune parameters with grid search\nn_trees = [5,10,30,50,100,125,150,200,250,300]\nlearning_rates = [0.001, 0.01, 0.1, 0.25, 0.50, 0.75, 0.95, 1]\nmax_depths = my_list = list(range(1, 16))\n\n# Create a dictionary to hold test RMSE for each 'square' in grid\nperf_dict = {}\nfor tree in n_trees:\n for learning_rate in learning_rates:\n for max_depth in max_depths:\n temp_boosted_model = boost_resid_correction(train_df, \n 'target',\n pred_cols, \n tree, \n learning_rate=learning_rate, \n max_depth=max_depth)\n temp_boosted_model['target_col'] = 'target'\n preds, rmse = boost_resid_correction_predict(test_df, temp_boosted_model)\n dict_key = '_'.join(str(x) for x in [tree, learning_rate, max_depth])\n perf_dict[dict_key] = rmse\n \nmin_key = min(perf_dict, key=perf_dict.get)\nprint(perf_dict[min_key])<\/code><\/pre>\n![\"\ud83e\udd41\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f941.png?ssl=1\")
\u2014 50 trees, a learning rate of 0.1 and a max depth of 1! Let\u2019s take a look and see how our predictions did.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/boosting_perf.jpg?ssl=1\")
Bagging vs. Boosting \u2013 understanding the differences<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/boost_vs_bag-1.jpg?ssl=1\")
Wrapping it up<\/h2>\n
\n