As always, the code used in this tutorial is available on my GitHub<\/a>. A video version<\/a> of this tutorial is also available on my YouTube channel for those who prefer to follow along visually. With that, let\u2019s get started!<\/p>\n Random forests can be categorized as bagging algorithms (b<\/strong>ootstrap agg<\/strong>regating)<\/strong>. Bagging consists of two steps:<\/p>\n 1.) Bootstrap sampling: Create multiple training sets by randomly drawing samples with replacement from the original dataset. These new training sets, called bootstrapped datasets, typically contain the same number of rows as the original dataset, but individual rows may appear multiple times or not at all. On average, each bootstrapped dataset contains about 63.2% of the unique rows from the original data. The remaining ~36.8% of rows are left out and can be used for out-of-bag (OOB) evaluation. For more on this concept, see my sampling with and without replacement blog post<\/a>.<\/p>\n 2.) Aggregating predictions: Each bootstrapped dataset is used to train a different decision tree model. The final prediction is made by combining the outputs of all individual trees. For classification, this is typically done through majority voting. For regression, predictions are averaged.<\/p>\n Training each tree on a different bootstrapped sample introduces variation across trees. While this doesn\u2019t fully eliminate correlation\u2014especially when certain features dominate\u2014it helps reduce overfitting when combined with aggregation. Averaging the predictions of many such trees reduces the overall variance<\/strong> of the ensemble, improving generalization.<\/p>\n Suppose there\u2019s a single strong feature in your dataset. In bagged trees<\/a>, each tree may repeatedly split on that feature, leading to correlated trees and less benefit from aggregation. Random Forests reduce this issue by introducing further randomness. Specifically, they change how splits are selected during training:<\/p>\n 1). Create N bootstrapped datasets. Note that while bootstrapping is commonly used in Random Forests, it is not strictly necessary because step 2 (random feature selection) introduces sufficient diversity among the trees.<\/p>\n 2). For each tree, at each node, a random subset of features is selected as candidates, and the best split is chosen from that subset. In scikit-learn, this is controlled by the 3). Aggregating predictions: vote for classification and average for regression.<\/p>\n Note: Random Forests use sampling with replacement for bootstrapped datasets and sampling without replacement<\/a> for selecting a subset of features.\u00a0<\/p>\n Because ~36.8% of training data is excluded from any given tree, you can use this holdout portion to evaluate that tree\u2019s predictions. Scikit-learn allows this via the oob_score=True parameter, providing an efficient way to estimate generalization error. You\u2019ll see this parameter used in the training example later in the tutorial.<\/p>\n Random Forests remain a strong baseline for tabular data thanks to their simplicity, interpretability, and ability to parallelize<\/a> since each tree is trained independently. This section demonstrates how to load data, perform a train test split<\/a>, train a baseline model, tune hyperparameters using grid search, and evaluate the final model on the test set.<\/p>\n Before tuning, it\u2019s good practice to train a baseline model using reasonable defaults. This gives you an initial sense of performance and lets you validate generalization using the out-of-bag (OOB) score, which is built into bagging-based models like Random Forests. This example uses the House Sales in King County dataset (CCO 1.0 Universal License), which contains property sales from the Seattle area between May 2014 and May 2015. This approach allows us to reserve the test set for final evaluation after tuning.<\/p>\n While the baseline model gives a strong starting point, performance can often be improved by tuning key hyperparameters. Grid search cross-validation, as implemented by Now that we\u2019ve selected the best-performing model based on cross-validation, we can evaluate it on the held-out test set to estimate its generalization performance.<\/p>\n One of the key advantages of Random Forests is their interpretability \u2014 something that large language models (LLMs) often lack. While LLMs are powerful, they typically function as black boxes and can exhibit biases that are difficult to identify<\/a>. In contrast, scikit-learn supports two main methods for measuring feature importance in Random Forests: Mean Decrease in Impurity and Permutation Importance.<\/p>\n 1). Mean Decrease in Impurity (MDI): Also known as Gini importance, this method calculates the total reduction in impurity brought by each feature across all trees. This is fast and built into the model via 2). Permutation Importance: This method assesses the decrease in model performance when a single feature\u2019s values are randomly shuffled. Unlike MDI, it accounts for feature interactions and correlation. It is more reliable but also more computationally expensive.<\/p>\n It is important to note that our geographic features lat and long are also useful for visualization as the plot below shows. It\u2019s likely that companies like Zillow leverage location information extensively in their valuation models.<\/p>\n A Random Forest consists of multiple decision trees\u2014one for each estimator specified via the You can now view all the individual trees from the fitted model.\u00a0<\/p>\n You can now visualize individual trees. The code below visualizes the first decision tree.<\/p>\n Although plotting many trees can be difficult to interpret, you may wish to explore the variety across estimators. The following example shows how to visualize the first five decision trees in the forest:<\/p>\n Random forests consist of multiple decision trees trained on bootstrapped data in order to achieve better predictive performance than could be obtained from any of the individual decision trees. If you have questions or thoughts on the tutorial, feel free to reach out through YouTube<\/a> or X<\/a>.<\/p>\n The post Understanding Random Forest using Python (scikit-learn)<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
What is Bagging (Bootstrap Aggregating)<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/Bagging-1-1024x467.png?resize=1024%2C467&ssl=1\")
What Makes Random Forests Different<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXdxSzrJ8EbM32JIq4zBNQK2SAIpsB3p_hv6sJxyq6XaCKt6B7pz_4XpiPWiwsTzQh4iwUQZZSeU6b4svfHSZ2IwjgwPCdQ0pfqS4ojSGUnRuiD8TIV3W4Lk2dHMyVFTxh4QCjRztA.png?ssl=1\")
max_features<\/code> parameter, which defaults to 'sqrt'<\/code> for classifiers and 1<\/code> for regressors (equivalent to bagged trees).<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXeJs7z-C1yKyMmpFLDOoA3O8Ac0CnoKVDq0WqRy0db2vMGKGSXhcRhXnFoW33Obn3e_lsaO_moUGCmlZpkNPqsLi5k_ffjkfifB026LEavIoP60kci8rzaeIh6n7Su5zvMXZ-qO2-2zdpBnoFjSGugRob1r.png?ssl=1\")
Out-of-Bag (OOB) Score<\/h3>\n
Training and Tuning a Random Forest in Scikit-Learn<\/h2>\n
Step 1: Train a Baseline Model<\/h3>\n
Python\"># Import libraries\n\n# Some imports are only used later in the tutorial\nimport matplotlib.pyplot as plt\n\nimport numpy as np\n\nimport pandas as pd\n\n# Dataset: Breast Cancer Wisconsin (Diagnostic)\n# Source: UCI Machine Learning Repository\n# License: CC BY 4.0\nfrom sklearn.datasets import load_breast_cancer\n\nfrom sklearn.ensemble import RandomForestClassifier\n\nfrom sklearn.ensemble import RandomForestRegressor\n\nfrom sklearn.inspection import permutation_importance\n\nfrom sklearn.model_selection import GridSearchCV, train_test_split\n\nfrom sklearn import tree\n\n# Load dataset\n# Dataset: House Sales in King County (May 2014\u2013May 2015)\n# License CC0 1.0 Universal\nurl = 'https:\/\/raw.githubusercontent.com\/mGalarnyk\/Tutorial_Data\/master\/King_County\/kingCountyHouseData.csv'\n\ndf = pd.read_csv(url)\n\ncolumns = ['bedrooms',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'bathrooms',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'sqft_living',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'sqft_lot',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'floors',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'waterfront',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'view',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'condition',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'grade',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'sqft_above',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'sqft_basement',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'yr_built',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'yr_renovated',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'lat',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'long',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'sqft_living15',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'sqft_lot15',\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0'price']\n\ndf = df[columns]\n\n# Define features and target\n\nX = df.drop(columns='price')\n\ny = df['price']\n\n# Train\/test split\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)\n\n# Train baseline Random Forest<\/a>\n\nreg = RandomForestRegressor(\n\n\u00a0\u00a0\u00a0\u00a0n_estimators=100,\u00a0 \u00a0 \u00a0 \u00a0 # number of trees\n\n\u00a0\u00a0\u00a0\u00a0max_features=1\/3,\u00a0 \u00a0 \u00a0 \u00a0 # fraction of features considered at each split\n\n\u00a0\u00a0\u00a0\u00a0oob_score=True,\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # enables out-of-bag evaluation\n\n\u00a0\u00a0\u00a0\u00a0random_state=0\n\n)\n\nreg.fit(X_train, y_train)\n\n# Evaluate baseline performance using OOB score\n\nprint(f\"Baseline OOB score: {reg.oob_score_:.3f}\")<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXeu5QApkjaUIPs5Y2N-vC2-rdPTNz0zLSNamD3MW39ZnlIuChnGgp2_JAOiTIgaIf2-zeZAmtFSZjy-mQCkuGFOuj5wGCOQWdr2PEo-UH9pXztZPMSObGrnqtOrtCdzaClTIEvWgA.png?ssl=1\")
<\/figure>\nStep 2: Tune Hyperparameters with Grid Search<\/h3>\n
GridSearchCV<\/code>, systematically explores combinations of hyperparameters and uses cross-validation to evaluate each one, selecting the configuration with the highest validation performance.The most commonly tuned hyperparameters include:<\/p>\n\n
n_estimators<\/code>: The number of decision trees in the forest. More trees can improve accuracy but increase training time.<\/li>\nmax_features<\/code>: The number of features to consider when looking for the best split. Lower values reduce correlation between trees.<\/li>\nmax_depth<\/code>: The maximum depth of each tree. Shallower trees are faster but may underfit.<\/li>\nmin_samples_split<\/code>: The minimum number of samples required to split an internal node. Higher values can reduce overfitting.<\/li>\nmin_samples_leaf<\/code>: The minimum number of samples required to be at a leaf node. Helps control tree size.<\/li>\nbootstrap<\/code>: Whether bootstrap samples are used when building trees. If False, the whole dataset is used.<\/li>\n<\/ul>\nparam_grid = {\n\n\u00a0\u00a0\u00a0\u00a0'n_estimators': [100],\n\n\u00a0\u00a0\u00a0\u00a0'max_features': ['sqrt', 'log2', None],\n\n\u00a0\u00a0\u00a0\u00a0'max_depth': [None, 5, 10, 20],\n\n\u00a0\u00a0\u00a0\u00a0'min_samples_split': [2, 5],\n\n\u00a0\u00a0\u00a0\u00a0'min_samples_leaf': [1, 2]\n\n}\n\n# Initialize model\n\nrf = RandomForestRegressor(random_state=0, oob_score=True)\n\ngrid_search = GridSearchCV(\n\n\u00a0\u00a0\u00a0\u00a0estimator=rf,\n\n\u00a0\u00a0\u00a0\u00a0param_grid=param_grid,\n\n\u00a0\u00a0\u00a0\u00a0cv=5, \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 # 5-fold cross-validation\n\n\u00a0\u00a0\u00a0\u00a0scoring='r2', \u00a0 \u00a0 # evaluation metric\n\n\u00a0\u00a0\u00a0\u00a0n_jobs=-1 \u00a0 \u00a0 \u00a0 \u00a0 # use all available CPU cores\n\n)\n\ngrid_search.fit(X_train, y_train)\n\nprint(f\"Best parameters: {grid_search.best_params_}\")\n\nprint(f\"Best R^2 score: {grid_search.best_score_:.3f}\")<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXeT2kvJd2Q7J-uuTQVChf2dfS0XbfScKqsDbx2RGmcSpIZUGpqdFhyh3MOWh_ymGALMo8VJs0vjlLvKz54S9EQOlYAb6HHBYg7b7qoyv3WFEcMkutWmkZcJl3DKClEzZdeFmHCXlw.png?ssl=1\")
<\/figure>\nStep 3: Evaluate Final Model on Test Set<\/h3>\n
# Evaluate final model on test set\n\nbest_model = grid_search.best_estimator_\n\nprint(f\"Test R^2 score (final model): {best_model.score(X_test, y_test):.3f}\")<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXdk4jzZD6S1_6CN05h6ZZF3zTbFqbsf4p2cU0eJRojCsWdZyAREvMyy9nReNOBWO-y04fmWNr0pn__1GAIAwVMA-5rUnBrFrR4j2U_n-a1aWrobhYTNoXiqHSECDUd5-JW1ODQxvw.png?ssl=1\")
<\/figure>\nCalculating Random Forest Feature Importance<\/h2>\n
reg.feature_importances_<\/code>. However, impurity-based feature importances can be misleading, especially for features with high cardinality (many unique values), as these features are more likely to be chosen simply because they provide more potential split points.<\/p>\nimportances = reg.feature_importances_\n\nfeature_names = X.columns\n\nsorted_idx = np.argsort(importances)[::-1]\n\nfor i in sorted_idx:\n\n\u00a0\u00a0\u00a0\u00a0print(f\"{feature_names[i]}: {importances[i]:.3f}\")<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXcBgUsFONPKRmPEk7yvNJYTC7NAo-UbcMKBdONU1gXN3prQ02toUp-v5oujpWNtg-x_LRfkZ9wFR-TlP8spyOBlZ6N6mL-YnPe6R4aoOH8fHaGrai9WHVkrbWBU62kpwpSJhG0jkg.png?ssl=1\")
<\/figure>\n# Perform permutation importance on the test set\n\nperm_importance = permutation_importance(reg, X_test, y_test, n_repeats=10, random_state=0)\n\nsorted_idx = perm_importance.importances_mean.argsort()[::-1]\n\nfor i in sorted_idx:\n\n\u00a0\u00a0\u00a0\u00a0print(f\"{X.columns[i]}: {perm_importance.importances_mean[i]:.3f}\")<\/code><\/pre>\n![](\"https:\/\/lh7-rt.googleusercontent.com\/docsz\/AD_4nXcgRkkAgBLSiSTRFuX8q0TlMe1I5SsCWSCfRMkXiAA_zlwqT6Okf5wRCknY1M4ps4v-kNDXl0_FgkVd4egBc93nec58VBEN1s0cCFVG5D0FYiIFbEFgVu7xOnqaBNdML7sHiyD0?key=MJNz7w8xYykgqioGAsk1Pg\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXc-_FsaMfuDfXO9-KqJM6Pfm-zpZOOz_fw79YsgPDb10gwG8b3mYYVieN-6c4agITiaNkaatp1TUrQIPsyOyuR5kf9KHULeAZAD0pgg8Ae2oJ4K16Vw79F9CscBTDcqF7r7Rmr.png?ssl=1\")
Visualizing Individual Decision Trees in a Random Forest<\/h2>\n
n_estimators <\/code>parameter. After training the model, you can access these individual trees through the .estimators_ attribute. Visualizing a few of these trees can help illustrate how differently each one splits the data due to bootstrapped training samples and random feature selection at each split. While the earlier example used a RandomForestRegressor, here we demonstrate this visualization using a RandomForestClassifier trained on the Breast Cancer Wisconsin dataset (CC BY 4.0 license) to highlight Random Forests\u2019 versatility for both regression and classification tasks. This short video<\/a> demonstrates what 100 trained estimators from this dataset look like.<\/p>\nFit a Random Forest Model using Scikit-Learn<\/h3>\n
# Load the Breast Cancer (Diagnostic) Dataset\n\ndata = load_breast_cancer()\n\ndf = pd.DataFrame(data.data, columns=data.feature_names)\n\ndf['target'] = data.target\n\n# Arrange Data into Features Matrix and Target Vector\n\nX = df.loc[:, df.columns != 'target']\n\ny = df.loc[:, 'target'].values\n\n# Split the data into training and testing sets\n\nX_train, X_test, Y_train, Y_test = train_test_split(X, y, random_state=0)\n\n# Random Forests in `scikit-learn` (with N = 100)\n\nrf = RandomForestClassifier(n_estimators=100,\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0random_state=0)\n\nrf.fit(X_train, Y_train)<\/code><\/pre>\nPlotting Individual Estimators (decision trees) from a Random Forest using Matplotlib<\/h3>\n
rf.estimators_<\/code><\/p>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXe_h1wnJBIjZjFmZ6kRzXsjF24Pn38jzEq2DzUZakIZeAUJpoWhiHzI10rf6dWYCBIDQCPG6qbaQXxD9RyQOy_z4i9djyr7KeCQug0fvCeQADgHDxh5quO3cAeNu0iPPLPncd3srQ.png?ssl=1\")
<\/figure>\nfn=data.feature_names\n\ncn=data.target_names\n\nfig, axes = plt.subplots(nrows = 1,ncols = 1,figsize = (4,4), dpi=800)\n\ntree.plot_tree(rf.estimators_[0],\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0feature_names = fn,\u00a0\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0class_names=cn,\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0filled = True);\n\nfig.savefig('rf_individualtree.png')<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXfw0lZAueYuvVSRYfB4PnIF8zrOM4lYuNOfV5JyCVABwerUNrbJReCEfg_QQfDfQO1YQgUnDPwKKHFt0YRkbrT3LxH4eZi5LnPtSYHJAw-_NW1E4ovQH4WZsADs8-f6KzWQlmg9ng.png?ssl=1\")
<\/figure>\n# This may not the best way to view each estimator as it is small\n\nfig, axes = plt.subplots(nrows=1, ncols=5, figsize=(10, 2), dpi=3000)\n\nfor index in range(5):\n\n\u00a0\u00a0\u00a0\u00a0tree.plot_tree(rf.estimators_[index],\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0feature_names=fn,\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0class_names=cn,\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0filled=True,\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0ax=axes[index])\n\n\u00a0\u00a0\u00a0\u00a0axes[index].set_title(f'Estimator: {index}', fontsize=11)\n\nfig.savefig('rf_5trees.png')<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/AD_4nXfVbyCIL1rejZJJCN9Wt_I697LyTTCRHqIJ102GLq0hnQlL7B-QEprQPTP4-jvfWtMYMvnjsiMFIRdxyHWjjxlPEyyM90FW19OyFSZGjwfq1pScUbaPzaAn-NRm9onGWfI6QHeHoA.png?ssl=1\")
<\/figure>\nConclusion<\/h2>\n