Building a Regression Model to Predict Delivery Durations: A Practical Guide<\/h3>\nE2E walkthrough for approaching a regression modeling\u00a0task<\/h4>\n
In this article, we\u2019re going to walk through the process of building a regression model\u200a\u2014\u200afrom dataset cleaning & preparation, to model training & evaluation. The specific regression task we will model for is predicting the expected delivery time for a food delivery\u00a0service.<\/p>\n
Comprehensive information about the project & the dataset involved can be found\u00a0here<\/a>.<\/p>\n If you want to follow along with the analysis, you can find the original notebooks in the GitHub\u00a0here<\/a>.<\/p>\n To provide some motivation for this task, let\u2019s assume we are working for some food delivery service, such as DoorDash.<\/p>\n When a customer places an order on DoorDash, DoorDash must display the estimated amount of time it will take until the customer receives their order. Predicting this arrival time accurately is crucial for DoorDash\u2019s customer satsifaction:<\/p>\n Building a regression model from scratch is an extensive process that requires some trial & error. Essentially, the general approach we will take looks something like\u00a0this:<\/p>\n Additionally, data preparation & exploration can be broken down further into the following steps:<\/p>\n In practice, there may be some back & forth between these steps (features highlighted as important during model training may motivate additional feature engineering related to those features, or poor model performance may motivate the need to gather more data). For now, we\u2019ll walk through this task following the process\u00a0above.<\/p>\n Now that we\u2019ve outlined our approach, let\u2019s take a look at our data and what kind of features we\u2019re working\u00a0with.<\/p>\n From the above, we see our data contains ~197,000 deliveries, with a variety of numeric & non-numeric features. None of the features are missing a large percentage of values (lowest non-null count ~181,000), so we likely won\u2019t have to worry about dropping any features entirely.<\/p>\n Let\u2019s check if our data contains any duplicated deliveries, and if there are any observations that we cannot compute delivery time\u00a0for.<\/p>\n We see that all the deliveries are unique. However, there are 7 deliveries that are missing a value for actual_delivery_time, which means we won\u2019t be able to compute the delivery duration for these orders. Since there\u2019s only a handful of these, we\u2019ll remove these observations from our\u00a0data.<\/p>\n Now, let\u2019s create our prediction target. We want to predict the delivery duration (in seconds), which is the elapsed time between when the customer placed the order (\u2018created_at\u2019) and when they recieved the order (\u2018actual_delivery_time\u2019).<\/p>\n The last thing we\u2019ll do before splitting our data into train\/test is check for missing values. We already viewed the non-null counts for each feature above, but let\u2019s view the proportions to get a better\u00a0picture.<\/p>\n We see that the market features (\u2018onshift_dashers\u2019, \u2018busy_dashers\u2019, \u2018outstanding_orders\u2019) have the highest percentage of missing values (~8% missing). The feature with the second-highest missing data rate is \u2018store_primary_category\u2019 (~2%). All other features have < 1%\u00a0missing.<\/p>\n Since none of the features have a high missing count, we won\u2019t remove any of them. Later on, we will look at the feature distributions to help us decide how to appropriately deal with missing observations for each\u00a0feature.<\/p>\n But first, let\u2019s split our data into train\/test. We will proceed with an 80\/20 split, and we\u2019ll write this test data to a separate file which we won\u2019t touch until evaluating our final\u00a0model.<\/p>\n Now, let\u2019s dive into the specifics of our train data. We\u2019ll establish our numeric & categorical features, to make it clear which columns are being referenced in later exploratory steps.<\/p>\n Let\u2019s revisit the categorical features with missing values (\u2018market_id\u2019, \u2018store_primary_category\u2019, \u2018order_protocol\u2019). Since there was little missing data among those features (< 3%), we will simply impute those missing values with an \u201cunknown\u201d category.<\/p>\n Let\u2019s look at our categorical features.<\/p>\n Since \u2018market_id\u2019 & \u2018order_protocol\u2019 have low cardinality, we can visualize their distributions easily. On the other hand, \u2018store_id\u2019 & \u2018store_primary_category\u2019 are high cardinality features. We\u2019ll take a deeper look at those\u00a0later.<\/p>\n Some key things to\u00a0note:<\/p>\n Unfortunately, we don\u2019t have any additional information about these variables, such as which \u2018market_id\u2019 values are associated with which cities\/locations, and what each \u2018order_protocol\u2019 number represents. At this point, asking for additional data concerning this information may be a good idea, as it may help for investigating trends in delivery duration across broader region\/location categorizations.<\/p>\n Let\u2019s look at our higher cardinality categorical features. Perhaps each \u2018store_primary_category\u2019 has an associated \u2018store_id\u2019 range? If so, we may not need \u2018store_id\u2019, as \u2018store_primary_category\u2019 would already encapsulate a lot of the information about the store being ordered\u00a0from.<\/p>\n Clearly not the case: we see that \u2018store_id\u2019 ranges overlap across levels of \u2018store_primary_category\u2019.<\/p>\n A quick look at the distinct values and associated frequencies for \u2018store_id\u2019 & \u2018store_primary_category\u2019 shows that these features have high cardinality<\/a> and are sparsely distributed. In general, high cardinality categorical features may be problematic in regression tasks, particularly for regression algorithms that require solely numeric data. When these high cardinality features are encoded, they may enlarge the feature space drastically, making the available data sparse and decreasing the model\u2019s ability to generalize to new observations in that feature space. For a better & more professional explanation of the phenomena, you can read more about it\u00a0here<\/a>.<\/p>\n Let\u2019s get a sense of how sparsely distributed these features\u00a0are.<\/p>\n We see that there are a handful of stores that have hundreds of orders, but the majority of them have much less than\u00a0100.<\/p>\n To handle the high cardinality of \u2018store_id\u2019, we\u2019ll create another feature, \u2018store_id_freq\u2019, that groups the \u2018store_id\u2019 values by frequency.<\/p>\n Our encoding shows us that ~60,000 deliveries were ordered from stores catgorized in the 90\u201399th percentile in terms of popularity, whereas ~12,000 deliveries were ordered from stores that were in the 0\u201350th percentile in popularity.<\/p>\n Now that we\u2019ve (attempted) to capture relevant \u2018store_id\u2019 information in a lower dimension, let\u2019s try to do something similar with \u2018store_primary_category\u2019.<\/p>\n Let\u2019s look at the most popular \u2018store_primary_category\u2019 levels.<\/p>\n A quick look shows us that many of these \u2018store_primary_category\u2019 levels are not exclusive to each other (ex: \u2018american\u2019 & \u2018burger\u2019). Further investigation shows many more examples of this kind of\u00a0overlap.<\/p>\n So, let\u2019s try to map these distinct store categories into a few basic, all-encompassing groups.<\/p>\n This grouping is probably brutally simple, and there may very well be a better way to group these store categories. Let\u2019s proceed with it for now for simplicity.<\/p>\n We\u2019ve done a good deal of investigation into our categorical features. Let\u2019s look at the distributions for our numeric features.<\/p>\n Many of the distributions appear to be more right skewed then they are due to the presence of outliers.<\/p>\n In particular, there seems to be an order with 400+ items. This seems strange as the next largest order is less than 100\u00a0items.<\/p>\n Let\u2019s look more into that 400+ item\u00a0order.<\/p>\n That order was a fast-food order that consisted of 411 total items, but only 5 distinct items. It was also placed late at night (midnight-1 am).<\/p>\n While it\u2019s not impossible for somebody to place an order like that (McDonalds late night run??<\/a>), we\u2019ll proceed to remove this observation from our data since it\u2019s highly unlikely that our model will have to make delivery duration predictions for orders with more than 100\u00a0items.<\/p>\n A couple of additional feature engineering steps:<\/p>\n Let\u2019s revisit our numeric features with missing\u00a0data.<\/p>\n From the boxplots above, it seemed like most of our features tended to skew right. Thus, we\u2019ll impute missing values for our numeric features with their median, as that is likely more representative of the center of their distributions compared to the\u00a0mean.<\/p>\n There are certainly other imputation strategies<\/a> to consider here. Other approaches include KNN imputation (imputing missing values with the average of the K most \u201csimilar\u201d observations), or building another regression model to predict those missing values<\/a> from features thought to be potentially relevant. For now, we\u2019ll proceed with median imputation for simplicity.<\/p>\n The last thing we\u2019ll do before moving on to modeling is check correlations among our numeric features. Some of the regression algorithms we will experiment with assume that the features are independent i.e. no collinearity.<\/p>\n There is some high collinearity present:<\/p>\n Among sets of highly collinear features, we will only keep one from each set for the final feature set for the model. We will add this step as part of our preprocessing pipeline for our\u00a0models.<\/p>\n Interestingly, each of our numeric features show fairly weak correlation (r < 0.2) with our prediction target (\u2018seconds_to_delivery\u2019). This may give us a clue that a linear model will not effectively capture the relationship between our feature set & delivery duration (if there is any relationship).<\/p>\n Now that we\u2019ve prepped & explored our data, let\u2019s build some\u00a0models.<\/p>\n The first things to consider for modeling are what algorithms we want to build models with, and how to measure model performance<\/a>.<\/p>\n We will experiment with the following regression algorithms, as they are algorithms commonly used for regression tasks. They include a mix of algorithms that can capture linear & non-linear relationships between the feature space & prediction target.<\/p>\n We\u2019ll carry out hyperparameter tuning<\/a> for each of the algorithms above to maximize the performance of each algorithm.<\/p>\n We\u2019ll evaluate our models using Root Mean Squared Error (RMSE)<\/a>, as that will allow us to discuss the errors of each model in seconds, which makes sense in the context of delivery duration.<\/p>\n Although we prepped\/cleaned our data, there are still some data transformations<\/a> that need to be done to get the data compatible for each of the regression algorithms above. Specifically, the regression algorithms will require some or all of the following preprocessing steps prior to prediction:<\/p>\n These transformations may need to be applied to different subsets of our feature space, and some of these transformations may need to be applied in sequential order. Fortunately, we can use scikit-learn\u2019s ColumnTransformer<\/a> to specify which features to apply specific data transformations to, and scikit-learn\u2019s Pipeline<\/a> to define a series of data transformations to apply in sequence.<\/p>\n Note that we already took care of some of these preprocessing steps (such as imputing missing values) manually in our data preparation. However, defining this pipeline will come in handy when we want to apply this preprocessing workflow to new data, as these steps will already be saved as part of our model artifact.<\/p>\n Additionally, Lasso Regression & Support Vector Regression require features to be on consistent scales<\/a> to maximize their effectiveness, since they perform distance-based computations. In contrast, Random Forest & Gradient Boosted Trees are built on top of Decision Trees<\/a>, which involve threshold based splitting. Therefore, we\u2019ll define two preprocessing pipelines, one with scaling and one\u00a0without.<\/p>\n Let\u2019s build our first model using <\/strong>Lasso regression<\/strong><\/a>.<\/strong><\/p>\n Lasso regression models a linear relationship between a scalar target and one or more explanatory variables. It includes a penalty term to the linear regression objective function that may zero out coefficients for features with little to no relationship with the prediction target.<\/p>\n Preprocessing requirements include:<\/p>\n Lasso Regression Results:<\/p>\n Let\u2019s move to <\/strong>Support Vector Regression<\/strong><\/a>, a popular regression algorithm derived from <\/strong>Support Vector Machines<\/strong><\/a>.<\/strong><\/p>\n Preprocessing requirements include:<\/p>\n SVR results:<\/p>\n Our next algorithm is <\/strong>Random Forest Regression<\/strong><\/a>.<\/strong><\/p>\n Random Forest Regression is an ensemble learning method<\/a> that combines the predictions of multiple decision trees, where each tree is built from a bootstrap sample drawn from the training\u00a0set.<\/p>\n Random Forest\u00a0results:<\/p>\n Our last algorithm is <\/strong>Gradient Boosted Regression<\/strong><\/a>.<\/strong><\/p>\n Gradient Boosted Regression is also an ensemble method like Random Forest Regression, but unlike the former, the learners in Gradient Boosted Regression are not independent\u200a\u2014\u200aeach base learner attempts to correct the errors of the previous.<\/p>\n Gradient Boosted Regression results:<\/p>\n Let\u2019s compare the RMSE scores from the best models we achieved with each regression algorithm.<\/strong><\/p>\n Our Gradient Boosted Regression model had the lowest RMSE, but outside of Support Vector Regression, all models had comparable performance. The RMSEs between Lasso, RFR, & GBR were all within 40 seconds of each other, and considering the prediction errors were already > 17 minutes off on average, this doesn\u2019t seem like\u00a0much.<\/p>\n That being said, we\u2019ll proceed to evaluate our trained GBR model on the test\u00a0set.<\/strong><\/p>\n We\u2019ll write out the hyperparameter-tuned GBR model artifact. This model serialization<\/a> will contain the necessary data transformations prior to prediction, as well as the model hyperparameters that led to optimal GBR performance.<\/p>\n Now, we\u2019ll evaluate our chosen model on our holdout test data. Before we generate predictions on the test data, we\u2019ll have to repeat some feature engineering steps that we did in our initial data prep. Specifically, we\u2019ll need to create the following features in our test\u00a0data:<\/p>\n The rest of the data cleaning\/preprocessing steps (imputing missing values, feature scaling, dropping correlated features) will be taken care of in the pipeline defined in the model artifact.<\/p>\n Let\u2019s evaluate our model on the test\u00a0data.<\/p>\n Our final test RMSE comes out to be ~1080\u00a0seconds.<\/p>\n Thus, on average, our predictions for delivery duration are ~18 minutes off from the true delivery duration. So overall, pretty bad. I\u2019m not sure if DoorDash models delivery duration from these features only, but I certainly would not recommend using this model as an endpoint for returning delivery duration estimates.<\/p>\n Some ideas for further work\/investigation could include the following:<\/p>\n If you made it this far, thanks! There were a lot of data investigation & modeling decisions that we went through, many of which had no clear cut answer. However, I did my best to lay out all the relevant decisions made during the investigation process, as well as the reasoning behind those decisions. If there\u2019s any part of the process that you would\u2019ve done differently, please let me know in the comments, I\u2019d love to hear\u00a0it!<\/p>\n I read through many great resources while doing this investigation. I did my best to include most of them below\u200a\u2014\u200aI highly recommend checking at least some of them\u00a0out!<\/p>\n Missing data:<\/p>\n Handling high cardinality features:<\/p>\n Pipelines & Data Transformers:<\/p>\n Scikit-learn Regression APIs:<\/p>\n Hyperparameter tuning:<\/p>\n Model selection:<\/p>\n Most of all, the entirety of the scikit-learn user guide<\/a> & examples<\/a>.<\/p>\n The author has created all the images in this\u00a0article.<\/em><\/p>\n Building a Regression Model: Delivery Duration Prediction<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\nContents<\/h3>\n
\n
Background<\/h4>\n
\n
Dataset Info<\/h4>\n
\n
High-Level Approach<\/h4>\n
\n
\n
Data Preparation & Exploratory Analysis<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/735\/1%2AXHCI8UtBNOcDv5aYbdJN_A.png?ssl=1\")
<\/figure>\nprint(f\"Number of duplicates: {df.duplicated().sum()} n\")
print(pd.DataFrame({'Missing Count': df[['created_at', 'actual_delivery_time']].isna().sum()}))<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/347\/1%2Al9ZyXToDNvLkiwDBhkJCJg.png?ssl=1\")
<\/figure>\n# convert columns to datetime
df['created_at'] = pd.to_datetime(df['created_at'], utc=True)
df['actual_delivery_time'] = pd.to_datetime(df['actual_delivery_time'], utc=True)
# create prediction target
df['seconds_to_delivery'] = (df['actual_delivery_time'] - df['created_at']).dt.total_seconds()<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/553\/1%2A16zJU1CivJuGK-L05MBlcQ.png?ssl=1\")
<\/figure>\nfrom sklearn.model_selection import train_test_split
import os
# shuffle
df = df.sample(frac=1, random_state=42)
df = df.reset_index(drop=True)
# split
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
# write test data to separate file
directory = 'datasets'
file_name = 'test_data.csv'
file_path = os.path.join(directory, file_name)
os.makedirs(directory, exist_ok=True)
test_df.to_csv(file_path, index=False)<\/pre>\ncategorical_feats = [
'market_id',
'store_id',
'store_primary_category',
'order_protocol'
]
numeric_feats = [
'total_items',
'subtotal',
'num_distinct_items',
'min_item_price',
'max_item_price',
'total_onshift_dashers',
'total_busy_dashers',
'total_outstanding_orders',
'estimated_order_place_duration',
'estimated_store_to_consumer_driving_duration'
]<\/pre>\n\n
missing_cols_categorical = ['market_id', 'store_primary_category', 'order_protocol']
train_df[missing_cols_categorical] = train_df[missing_cols_categorical].fillna(\"unknown\")<\/pre>\npd.DataFrame({'Cardinality': train_df[categorical_feats].nunique()}).rename_axis('Feature')<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/305\/1%2A7fq8oo9RydsQxzLWNoR8BA.png?ssl=1\")
<\/figure>\nimport seaborn as sns
import matplotlib.pyplot as plt
categorical_feats_subset = [
'market_id',
'order_protocol'
]
# Set up the grid
fig, axes = plt.subplots(1, len(categorical_feats_subset), figsize=(13, 5), sharey=True)
# Create barplots for each variable
for i, col in enumerate(categorical_feats_subset):
sns.countplot(x=col, data=train_df, ax=axes[i])
axes[i].set_title(f\"Frequencies: {col}\")
# Adjust layout
plt.tight_layout()
plt.show()<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2Aq2jJDB87lUKza9or_H_bjQ.png?ssl=1\")
<\/figure>\n\n
store_info = train_df[['store_id', 'store_primary_category']]
store_info.groupby('store_primary_category')['store_id'].agg(['min', 'max'])<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/322\/1%2A1kO3-C3aQCW-gA5O8kLCLQ.png?ssl=1\")
<\/figure>\nstore_id_values = train_df['store_id'].value_counts()
# Plot the histogram
plt.figure(figsize=(8, 5))
plt.bar(store_id_values.index, store_id_values.values, color='skyblue')
# Add titles and labels
plt.title('Value Counts: store_id', fontsize=14)
plt.xlabel('store_id', fontsize=12)
plt.ylabel('Frequency', fontsize=12)
plt.xticks(rotation=45) # Rotate x-axis labels for better readability
plt.tight_layout()
plt.show()<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/733\/1%2Ask7RT5uyobxkRhijZY8ESw.png?ssl=1\")
<\/figure>\n\n
def encode_frequency(freq, percentiles) -> str:
if freq < percentiles[0]:
return '[0-50)'
elif freq < percentiles[1]:
return '[50-75)'
elif freq < percentiles[2]:
return '[75-90)'
elif freq < percentiles[3]:
return '[90-99)'
else:
return '99+'
value_counts = train_df['store_id'].value_counts()
percentiles = np.percentile(value_counts, [50, 75, 90, 99])
# apply encode_frequency to each store_id based on their number of orders
train_df['store_id_freq'] = train_df['store_id'].apply(lambda x: encode_frequency(value_counts[x], percentiles))
pd.DataFrame({'Count':train_df['store_id_freq'].value_counts()}).rename_axis('Frequency Bin')<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/164\/1%2A9gW0KsNrNjIin5j9VSYBww.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/261\/1%2ATaUSijHy72EEP8nkkP0Glw.png?ssl=1\")
<\/figure>\nstore_category_map = {
'american': ['american', 'burger', 'sandwich', 'barbeque'],
'asian': ['asian', 'chinese', 'japanese', 'indian', 'thai', 'vietnamese', 'dim-sum', 'korean',
'sushi', 'bubble-tea', 'malaysian', 'singaporean', 'indonesian', 'russian'],
'mexican': ['mexican'],
'italian': ['italian', 'pizza'],
}
def map_to_category_type(category: str) -> str:
for category_type, categories in store_category_map.items():
if category in categories:
return category_type
return \"other\"
train_df['store_category_type'] = train_df['store_primary_category'].apply(lambda x: map_to_category_type(x))
value_counts = train_df['store_category_type'].value_counts()
# Plot pie chart
plt.figure(figsize=(6, 6))
value_counts.plot.pie(autopct='%1.1f%%', startangle=90, cmap='viridis', labels=value_counts.index)
plt.title('Category Distribution')
plt.ylabel('') # Hide y-axis label for aesthetics
plt.show()<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/452\/1%2Ati8R3V11c8gGewWsDODUrg.png?ssl=1\")
<\/figure>\n# Create grid for boxplots
fig, axes = plt.subplots(nrows=5, ncols=2, figsize=(12, 15)) # Adjust figure size
axes = axes.flatten() # Flatten the 5x2 axes into a 1D array for easier iteration
# Generate boxplots for each numeric feature
for i, column in enumerate(numeric_feats):
sns.boxplot(y=train_df[column], ax=axes[i])
axes[i].set_title(f\"Boxplot for {column}\")
axes[i].set_ylabel(column)
# Remove any unused subplots (if any)
for i in range(len(numeric_feats), len(axes)):
fig.delaxes(axes[i])
# Adjust layout for better spacing
plt.tight_layout()
plt.show()<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AOfzdBdsT1n7EUBVGkN6Rtg.png?ssl=1\")
train_df[train_df['total_items']==train_df['total_items'].max()]<\/pre>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AEBnOKPbtcEoCoad96_SVbg.png?ssl=1\")
\n
# extract price range
train_df['item_price_range'] = train_df['max_item_price'] - train_df['min_item_price']
# extract hour of day
time_info = train_df['created_at'].astype(str).str.split().str[1]
train_df['hour_of_day'] = time_info.str.split(\":\").str[0]<\/pre>\nvalues = {
'total_onshift_dashers': train_df['total_onshift_dashers'].median(),
'total_busy_dashers': train_df['total_busy_dashers'].median(),
'total_outstanding_orders': train_df['total_outstanding_orders'].median(),
'estimated_store_to_consumer_driving_duration': train_df['estimated_store_to_consumer_driving_duration'].median()
}
train_df[col].fillna(value=values, inplace=True)<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AEKp2RmplnzczOTqozgsucA.png?ssl=1\")
<\/figure>\n\n
Building Models<\/h4>\n
\n
\n
# sequence of transformations to apply to categorical features
categorical_pipeline = Pipeline(
steps=[
(\"categorical_imputer\", SimpleImputer(strategy=\"constant\", fill_value=\"unknown\")),
(\"encoder\", OneHotEncoder(handle_unknown=\"ignore\", drop='first', sparse_output=False)),
]
)
# sequence of transformations to apply to numeric features
numeric_pipeline_scaled = Pipeline(
steps=[
(\"numeric_imputer\", SimpleImputer(strategy=\"median\")),
(\"scaler\", StandardScaler()),
(\"drop_correlated_feats\", DropHighlyCorrelatedFeatures()),
]
)
numeric_pipeline = Pipeline(
steps=[
(\"numeric_imputer\", SimpleImputer(strategy=\"median\")),
(\"drop_correlated_feats\", DropHighlyCorrelatedFeatures()),
]
)
# combine numeric & categorical feature transformations into single ColumnTransformer() object
preprocessor_w_scaling = ColumnTransformer(
transformers=[
('numeric', numeric_pipeline_scaled, numeric_feats),
('cat', categorical_pipeline, categorical_feats),
],
)
preprocessor_wo_scaling = ColumnTransformer(
transformers=[
('numeric', numeric_pipeline, numeric_feats),
('cat', categorical_pipeline, categorical_feats),
]
)<\/pre>\n\n
from sklearn.linear_model import Lasso
from sklearn.model_selection import GridSearchCV
# lasso regression preprocessing pipeline
lasso_reg = Pipeline(
steps=[
(\"preprocessor\", preprocessor_w_scaling),
(\"regression\", Lasso(random_state=13)),
]
)
# lasso regression hyperparameter space
lasso_param_grid = {
\"regression__alpha\": np.logspace(-3, -2, 10),
}
lasso_search_cv = GridSearchCV(lasso_reg, lasso_param_grid, scoring='neg_root_mean_squared_error', verbose=4)
# fit model, performing exhaustive search over hyperparameter space
lasso_search_cv.fit(df_X, df_y)
# retrieve results of hyperparameter tuning
lasso_cv_results_df = pd.DataFrame(lasso_search_cv.cv_results_)
lasso_cv_results_df.sort_values(by='mean_test_score', ascending=False).drop(columns=['params','split0_test_score','split1_test_score','split2_test_score','split3_test_score','split4_test_score'])<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/527\/1%2Ad9kuOftTZW4iBbm-I24jzQ.png?ssl=1\")
\n
\n
from sklearn.svm import SVR
from sklearn.model_selection import GridSearchCV
svr_reg = Pipeline(
steps=[
(\"preprocessor\", preprocessor_w_scaling),
(\"regression\", SVR(kernel='poly', max_iter=1000)), # proceeding with polynomial kernel based on sub-par linear regression performance
]
)
svr_param_grid = {
\"regression__C\": np.logspace(-3, 1, 4),
'regression__epsilon': np.logspace(-3, 1, 4),
# 'regression__kernel': ['linear', 'poly', 'rbf']
}
svr_search_cv = GridSearchCV(svr_reg, svr_param_grid, verbose=4, scoring='neg_root_mean_squared_error')
svr_search_cv.fit(df_X, df_y)
svr_cv_results_df = pd.DataFrame(svr_search_cv.cv_results_)
svr_cv_results_df.sort_values(\"mean_test_score\", ascending=False).drop(columns=['params','split0_test_score','split1_test_score','split2_test_score','split3_test_score','split4_test_score'])<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/687\/1%2AA-CKZNIJ9O2UDAy2Ra0A9Q.png?ssl=1\")
\n
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
rfr_clf = Pipeline(
steps=[
(\"preprocessor\", preprocessor_wo_scaling),
(\"regressor\", RandomForestRegressor(random_state=13))
]
)
rfr_param_grid = {
\"regressor__n_estimators\": [10, 100, 200],
# 'regressor__max_depth': [None, 50],
'regressor__max_features': ['sqrt', 'log2', None],
}
rfr_search_cv = GridSearchCV(rfr_clf, rfr_param_grid, verbose=4, scoring='neg_root_mean_squared_error')
rfr_search_cv.fit(df_X, df_y)
rfr_cv_results = pd.DataFrame(rfr_search_cv.cv_results_)
rfr_cv_results = rfr_cv_results.sort_values(\"mean_test_score\", ascending=False).drop(columns=['params','split0_test_score','split1_test_score','split2_test_score','split3_test_score','split4_test_score'])<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/790\/1%2AeHNvA67Ro9H7DEoZxImAAg.png?ssl=1\")
\n
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
gbr_clf = Pipeline(
steps=[
(\"preprocessor\", preprocessor_wo_scaling),
(\"regressor\", GradientBoostingRegressor(random_state=13))
]
)
gbr_param_grid = {
\"regressor__n_estimators\": [10, 100, 200],
'regressor__learning_rate': np.logspace(-3, 0, 4),
}
gbr_search_cv = GridSearchCV(gbr_clf, gbr_param_grid, verbose=4, scoring='neg_root_mean_squared_error')
gbr_search_cv.fit(df_X, df_y)
gbr_cv_results = pd.DataFrame(gbr_search_cv.cv_results_)
gbr_cv_results = gbr_cv_results.sort_values(\"mean_test_score\", ascending=False).drop(columns=['params','split0_test_score','split1_test_score','split2_test_score','split3_test_score','split4_test_score'])<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/787\/1%2A6IPLuUxjTwpc6pFChYfueg.png?ssl=1\")
\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/742\/1%2AIz085yoVDOL3DLQErA28aw.png?ssl=1\")
<\/figure>\n\n
# Use joblib for efficient model serialization
import joblib
# Save trained GBR model
joblib.dump(gbr_search_cv.best_estimator_, '..\/models\/best_model.pkl')
print(\"Best model saved to 'best_model.pkl'\")<\/pre>\nFinal Model Evaluation<\/h4>\n
\n
from feature_eng_utils import encode_frequency, map_to_category_type
# create store_id_freq
value_counts = test_df['store_id'].value_counts()
percentiles = np.percentile(value_counts, [50, 75, 90, 99])
test_df['store_id_freq'] = test_df['store_id'].apply(lambda x: encode_frequency(value_counts[x], percentiles))
# create store_category_type
test_df['store_category_type'] = test_df['store_primary_category'].apply(lambda x: map_to_category_type(x))
# create item_price_range
test_df['item_price_range'] = test_df['max_item_price'] - test_df['min_item_price']
# create hour_of_day
time_info = test_df['created_at'].astype(str).str.split().str[1]
test_df['hour_of_day'] = time_info.str.split(\":\").str[0]<\/pre>\nimport joblib
from custom_transformers import DropHighlyCorrelatedFeatures
from sklearn.metrics import root_mean_squared_error
# Load the saved model
loaded_model = joblib.load('..\/models\/best_model.pkl')
# Make predictions
y_pred = loaded_model.predict(test_df_X)
# Compute root mean squared error
test_rmse = root_mean_squared_error(test_df_y, y_pred)
print(\"Test RMSE:\", test_rmse)<\/pre>\n\n
Conclusion<\/h4>\n
Sources<\/h4>\n
\n
\n
\n
\n
\n
\n
<\/p>\n
\n