MODEL EVALUATION & OPTIMIZATION<\/h4>\nWhen all models have similar accuracy, now\u00a0what?<\/h4>\n
You\u2019ve trained several classification models, and they all seem to be performing well with high accuracy scores. Congratulations!<\/p>\n
But hold on\u200a\u2014\u200ais one model truly better than the others? Accuracy alone doesn\u2019t tell the whole story. What if one model consistently overestimates its confidence, while another underestimates it? This is where model calibration<\/strong> comes\u00a0in.<\/p>\n Here, we\u2019ll see what model calibration is and explore how to assess the reliability of your models\u2019 predictions\u200a\u2014\u200ausing visuals and practical code examples to show you how to identify calibration issues. Get ready to go beyond accuracy and light up the true potential of your machine learning\u00a0models!<\/p>\n Model calibration measures how well a model\u2019s prediction probabilities<\/a> match its actual performance. A model that gives a 70% probability score should be correct 70% of the time for similar predictions. This means its probability scores should reflect the true likelihood of its predictions being\u00a0correct.<\/p>\n While accuracy tells us how often a model is correct overall, calibration tells us whether we can trust its probability scores<\/strong>. Two models might both have 90% accuracy, but one might give realistic probability scores while the other gives overly confident predictions. In many real applications, having reliable probability scores is just as important as having correct predictions.<\/p>\n A perfectly calibrated model would show a direct match between its prediction probabilities and actual success rates: When it predicts with 90% probability, it should be correct 90% of the time. The same applies to all probability levels.<\/p>\n However, most models aren\u2019t perfectly calibrated. They can\u00a0be:<\/p>\n This mismatch between predicted probabilities and actual correctness can lead to poor decision-making when using these models in real applications. This is why understanding and improving model calibration is necessary for building reliable machine learning\u00a0systems.<\/p>\n To explore model calibration, we\u2019ll continue with the same dataset used in my previous articles on Classification Algorithms<\/a>: predicting whether someone will play golf or not based on weather conditions.<\/p>\n Before training our models, we normalized numerical weather measurements through standard scaling<\/a> and transformed categorical features with one-hot encoding<\/a>. These preprocessing steps ensure all models can effectively use the data while maintaining fair comparisons between\u00a0them.<\/p>\n For this exploration, we trained four classification models to similar accuracy\u00a0scores:<\/p>\n For those who are curious with how those algorithms make prediction and their probability, you can refer to this\u00a0article:<\/p>\n Predicted Probability, Explained: A Visual Guide with Code Examples for Beginners<\/a><\/p>\n While these models achieved the same accuracy in this simple problem, they calculate their prediction probabilities differently.<\/p>\n Through these differences, we\u2019ll explore why we need to look beyond accuracy.<\/p>\n To assess how well a model\u2019s prediction probabilities match its actual performance, we use several methods and metrics. These measurements help us understand whether our model\u2019s confidence levels are reliable.<\/p>\n The Brier Score<\/strong> measures the mean squared difference<\/strong> between predicted probabilities and actual outcomes. It ranges from 0 to 1, where lower scores indicate better calibration. This score is particularly useful because it considers both calibration and accuracy together.<\/p>\n Log Loss<\/strong> calculates the negative log probability of correct predictions. This metric is especially sensitive to confident but wrong predictions\u200a\u2014\u200awhen a model says it\u2019s 90% sure but is wrong, it receives a much larger penalty than when it\u2019s 60% sure and wrong. Lower values indicate better calibration.<\/p>\n ECE<\/strong> measures the average difference between predicted and actual probabilities (taken as average of the label), weighted by how many predictions fall into each probability group. This metric helps us understand if our model has systematic biases in its probability estimates.<\/p>\n Similar to ECE, a reliability diagram (or calibration curve) visualizes model calibration by binning predictions and comparing them to actual outcomes. While ECE gives us a single number measuring calibration error, the reliability diagram shows us the same information graphically<\/strong>. We use the same binning approach and calculate the actual frequency of positive outcomes in each bin. When plotted, these points show us exactly where our model\u2019s predictions deviate from perfect calibration, which would appear as a diagonal\u00a0line.<\/p>\n Each of these metrics shows different aspects of calibration problems:<\/p>\n Together, these metrics give us a complete picture of how well our model\u2019s probability scores reflect its true performance.<\/p>\n For our models, let\u2019s calculate the calibration metrics and draw their calibration curves:<\/p>\n Now, let\u2019s analyze the calibration performance of each model based on those\u00a0metrics:<\/p>\n The k-Nearest Neighbors (KNN) model performs well at estimating how certain it should be about its predictions. Its graph line stays close to the dotted line, which shows good performance. It has solid scores\u200a\u2014\u200aa Brier score of 0.148 and the best ECE score of 0.090. While it sometimes shows too much confidence in the middle range, it generally makes reliable estimates<\/strong> about its certainty.<\/p>\n The Bernoulli Naive Bayes model shows an unusual stair-step pattern in its line. This means it jumps between different levels of certainty instead of changing smoothly. While it has the same Brier score as KNN (0.148), its higher ECE of 0.150 shows it\u2019s less accurate at estimating its certainty. The model switches between being too confident and not confident enough<\/strong>.<\/p>\n The Logistic Regression model shows clear issues with its predictions. Its line moves far away from the dotted line, meaning it often misjudges how certain it should be. It has the worst ECE score (0.181) and a poor Brier score (0.164). The model consistently shows too much confidence in its predictions<\/strong>, making it unreliable.<\/p>\n The Multilayer Perceptron shows a distinct problem. Despite having the best Brier score (0.129), its line reveals that it mostly makes extreme predictions\u200a<\/strong>\u2014\u200aeither very certain or very uncertain, with little in between. Its high ECE (0.167) and flat line in the middle ranges show it struggles to make balanced certainty estimates.<\/p>\n After examining all four models, the k-Nearest Neighbors clearly performs best<\/strong> at estimating its prediction certainty. It maintains consistent performance across different levels of certainty and shows the most reliable pattern in its predictions. While other models might score well in certain measures (like the Multilayer Perceptron\u2019s Brier score), their graphs reveal they aren\u2019t as reliable when we need to trust their certainty estimates.<\/p>\n When choosing between different models, we need to consider both their accuracy and calibration quality. A model with slightly lower accuracy but better calibration might be more valuable than a highly accurate model with poor probability estimates.<\/p>\n By understanding calibration and its importance, we can build more reliable machine learning systems that users can trust not just for their predictions, but also for their confidence in those predictions.<\/p>\n This article uses Python 3.7 and scikit-learn 1.5. While the concepts discussed are generally applicable, specific code implementations may vary slightly with different versions.<\/p>\n Unless otherwise noted, all images are created by the author, incorporating licensed design elements from Canva\u00a0Pro.<\/p>\n \ud835\ude4e\ud835\ude5a\ud835\ude5a \ud835\ude62\ud835\ude64\ud835\ude67\ud835\ude5a \ud835\ude48\ud835\ude64\ud835\ude59\ud835\ude5a\ud835\ude61 \ud835\ude40\ud835\ude6b\ud835\ude56\ud835\ude61\ud835\ude6a\ud835\ude56\ud835\ude69\ud835\ude5e\ud835\ude64\ud835\ude63 & \ud835\ude4a\ud835\ude65\ud835\ude69\ud835\ude5e\ud835\ude62\ud835\ude5e\ud835\ude6f\ud835\ude56\ud835\ude69\ud835\ude5e\ud835\ude64\ud835\ude63 \ud835\ude62\ud835\ude5a\ud835\ude69\ud835\ude5d\ud835\ude64\ud835\ude59\ud835\ude68 \ud835\ude5d\ud835\ude5a\ud835\ude67\ud835\ude5a:<\/p>\n Model Evaluation & Optimization<\/a><\/p>\n \ud835\ude54\ud835\ude64\ud835\ude6a \ud835\ude62\ud835\ude5e\ud835\ude5c\ud835\ude5d\ud835\ude69 \ud835\ude56\ud835\ude61\ud835\ude68\ud835\ude64 \ud835\ude61\ud835\ude5e\ud835\ude60\ud835\ude5a:<\/p>\n Model Calibration, Explained: A Visual Guide with Code Examples for Beginners<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AD8F9yFBSGWaKIzGQm8NYiQ.png?ssl=1\")
Understanding Calibration<\/strong><\/h3>\n
Why Calibration Matters<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2APrRS_sYDYy1VEcUUDFLpuw.png?ssl=1\")
Perfect Calibration vs.\u00a0Reality<\/h4>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2An8jSXAr3jlGjd3tV3dgQHQ.png?ssl=1\")
\ud83d\udcca Dataset\u00a0Used<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AEiN9aTP0wioYtNVMpZwAHA.png?ssl=1\")
import pandas as pd
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
# Create and prepare dataset
dataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast',
'sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy',
'sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast',
'rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0,
72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0,
88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0,
90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0,
65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True,
True, False, True, True, False, False, True, False, True, True, False,
True, False, False, True, False, False],
'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes',
'Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes',
'Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
# Prepare data
df = pd.DataFrame(dataset_dict)<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AT-JB7XdN5wVyFqL6mkOIpg.png?ssl=1\")
<\/figure>\nfrom sklearn.preprocessing import StandardScaler
df = pd.get_dummies(df, columns=['Outlook'], prefix='', prefix_sep='', dtype=int)
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Yes').astype(int)
# Rearrange columns
column_order = ['sunny', 'overcast', 'rainy', 'Temperature', 'Humidity', 'Wind', 'Play']
df = df[column_order]
# Prepare features and target
X,y = df.drop('Play', axis=1), df['Play']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)
# Scale numerical features
scaler = StandardScaler()
X_train[['Temperature', 'Humidity']] = scaler.fit_transform(X_train[['Temperature', 'Humidity']])
X_test[['Temperature', 'Humidity']] = scaler.transform(X_test[['Temperature', 'Humidity']])<\/pre>\nModels and\u00a0Training<\/h4>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AfkNi_MNiU1E0BoS22pWDow.png?ssl=1\")
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score
from sklearn.naive_bayes import BernoulliNB
# Initialize the models with the found parameters
knn = KNeighborsClassifier(n_neighbors=4, weights='distance')
bnb = BernoulliNB()
lr = LogisticRegression(C=1, random_state=42)
mlp = MLPClassifier(hidden_layer_sizes=(4, 2),random_state=42, max_iter=2000)
# Train all models
models = {
'KNN': knn,
'BNB': bnb,
'LR': lr,
'MLP': mlp
}
for name, model in models.items():
model.fit(X_train, y_train)
# Create predictions and probabilities for each model
results_dict = {
'True Labels': y_test
}
for name, model in models.items():
# results_dict[f'{name} Pred'] = model.predict(X_test)
results_dict[f'{name} Prob'] = model.predict_proba(X_test)[:, 1]
# Create results dataframe
results_df = pd.DataFrame(results_dict)
# Print predictions and probabilities
print(\"nPredictions and Probabilities:\")
print(results_df)
# Print accuracies
print(\"nAccuracies:\")
for name, model in models.items():
accuracy = accuracy_score(y_test, model.predict(X_test))
print(f\"{name}: {accuracy:.3f}\")<\/pre>\nMeasuring Calibration<\/h3>\n
Brier Score<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A2X9aJEpKRCKClV8HY7xqCg.png?ssl=1\")
Log Loss<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A6nvB6FGlluSDPd0DSlKRVA.png?ssl=1\")
Expected Calibration Error\u00a0(ECE)<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AmUnGPaLSmDhwgXOfQWCBNQ.png?ssl=1\")
Reliability Diagrams<\/h4>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2A1ZE1JT_WUnn3YqN9Uy9iZA.png?ssl=1\")
Comparing Calibration Metrics<\/h4>\n
\n
Our Models<\/h4>\n
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt
# Initialize models
models = {
'k-Nearest Neighbors': KNeighborsClassifier(n_neighbors=4, weights='distance'),
'Bernoulli Naive Bayes': BernoulliNB(),
'Logistic Regression': LogisticRegression(C=1.5, random_state=42),
'Multilayer Perceptron': MLPClassifier(hidden_layer_sizes=(4, 2), random_state=42, max_iter=2000)
}
# Get predictions and calculate metrics
metrics_dict = {}
for name, model in models.items():
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]
metrics_dict[name] = {
'Brier Score': brier_score_loss(y_test, y_prob),
'Log Loss': log_loss(y_test, y_prob),
'ECE': calculate_ece(y_test, y_prob),
'Probabilities': y_prob
}
# Plot calibration curves
fig, axes = plt.subplots(2, 2, figsize=(8, 8), dpi=300)
colors = ['orangered', 'slategrey', 'gold', 'mediumorchid']
for idx, (name, metrics) in enumerate(metrics_dict.items()):
ax = axes.ravel()[idx]
prob_true, prob_pred = calibration_curve(y_test, metrics['Probabilities'],
n_bins=5, strategy='uniform')
ax.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
ax.plot(prob_pred, prob_true, color=colors[idx], marker='o',
label='Calibration curve', linewidth=2, markersize=8)
title = f'{name}nBrier: {metrics[\"Brier Score\"]:.3f} | Log Loss: {metrics[\"Log Loss\"]:.3f} | ECE: {metrics[\"ECE\"]:.3f}'
ax.set_title(title, fontsize=11, pad=10)
ax.grid(True, alpha=0.7)
ax.set_xlim([-0.05, 1.05])
ax.set_ylim([-0.05, 1.05])
ax.spines[['top', 'right', 'left', 'bottom']].set_visible(False)
ax.legend(fontsize=10, loc='upper left')
plt.tight_layout()
plt.show()<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AK04o6vy013s-jb4uR91V1Q.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AUHL3W15Z1qHAdi4EjG7Qxg.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AUWbk8Q6F_CfTl-pyJOKh-Q.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AeDNNG7Hl1MJm6QER6kCOHQ.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1%2AaBVwcq7ZeVUhlKVbhWugwg.png?ssl=1\")
<\/figure>\nFinal Remark<\/h3>\n
\ud83c\udf1f Model Calibration Code Summarized (1\u00a0Model)<\/h3>\n
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import BernoulliNB
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt
# Define ECE
def calculate_ece(y_true, y_prob, n_bins=5):
bins = np.linspace(0, 1, n_bins + 1)
ece = 0
for bin_lower, bin_upper in zip(bins[:-1], bins[1:]):
mask = (y_prob >= bin_lower) & (y_prob < bin_upper)
if np.sum(mask) > 0:
bin_conf = np.mean(y_prob[mask])
bin_acc = np.mean(y_true[mask])
ece += np.abs(bin_conf - bin_acc) * np.sum(mask)
return ece \/ len(y_true)
# Create dataset and prepare data
dataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast','sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy','sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast','rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0,72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0,88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0,90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0,65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True,True, False, True, True, False, False, True, False, True, True, False,True, False, False, True, False, False],
'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes','Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes','Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
# Prepare and encode data
df = pd.DataFrame(dataset_dict)
df = pd.get_dummies(df, columns=['Outlook'], prefix='', prefix_sep='', dtype=int)
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Yes').astype(int)
df = df[['sunny', 'overcast', 'rainy', 'Temperature', 'Humidity', 'Wind', 'Play']]
# Split and scale data
X, y = df.drop('Play', axis=1), df['Play']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)
scaler = StandardScaler()
X_train[['Temperature', 'Humidity']] = scaler.fit_transform(X_train[['Temperature', 'Humidity']])
X_test[['Temperature', 'Humidity']] = scaler.transform(X_test[['Temperature', 'Humidity']])
# Train model and get predictions
model = BernoulliNB()
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]
# Calculate metrics
metrics = {
'Brier Score': brier_score_loss(y_test, y_prob),
'Log Loss': log_loss(y_test, y_prob),
'ECE': calculate_ece(y_test, y_prob)
}
# Plot calibration curve
plt.figure(figsize=(6, 6), dpi=300)
prob_true, prob_pred = calibration_curve(y_test, y_prob, n_bins=5, strategy='uniform')
plt.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
plt.plot(prob_pred, prob_true, color='slategrey', marker='o',
label='Calibration curve', linewidth=2, markersize=8)
title = f'Bernoulli Naive BayesnBrier: {metrics[\"Brier Score\"]:.3f} | Log Loss: {metrics[\"Log Loss\"]:.3f} | ECE: {metrics[\"ECE\"]:.3f}'
plt.title(title, fontsize=11, pad=10)
plt.grid(True, alpha=0.7)
plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.gca().spines[['top', 'right', 'left', 'bottom']].set_visible(False)
plt.legend(fontsize=10, loc='lower right')
plt.tight_layout()
plt.show()<\/pre>\n\ud83c\udf1f Model Calibration Code Summarized (4\u00a0Models)<\/h3>\n
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import BernoulliNB
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import brier_score_loss, log_loss
from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt
# Define ECE
def calculate_ece(y_true, y_prob, n_bins=5):
bins = np.linspace(0, 1, n_bins + 1)
ece = 0
for bin_lower, bin_upper in zip(bins[:-1], bins[1:]):
mask = (y_prob >= bin_lower) & (y_prob < bin_upper)
if np.sum(mask) > 0:
bin_conf = np.mean(y_prob[mask])
bin_acc = np.mean(y_true[mask])
ece += np.abs(bin_conf - bin_acc) * np.sum(mask)
return ece \/ len(y_true)
# Create dataset and prepare data
dataset_dict = {
'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast','sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy','sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast','rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0,72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0,88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0,90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0,65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
'Wind': [False, True, False, False, False, True, True, False, False, False, True,True, False, True, True, False, False, True, False, True, True, False,True, False, False, True, False, False],
'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes','Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes','Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
}
# Prepare and encode data
df = pd.DataFrame(dataset_dict)
df = pd.get_dummies(df, columns=['Outlook'], prefix='', prefix_sep='', dtype=int)
df['Wind'] = df['Wind'].astype(int)
df['Play'] = (df['Play'] == 'Yes').astype(int)
df = df[['sunny', 'overcast', 'rainy', 'Temperature', 'Humidity', 'Wind', 'Play']]
# Split and scale data
X, y = df.drop('Play', axis=1), df['Play']
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.5, shuffle=False)
scaler = StandardScaler()
X_train[['Temperature', 'Humidity']] = scaler.fit_transform(X_train[['Temperature', 'Humidity']])
X_test[['Temperature', 'Humidity']] = scaler.transform(X_test[['Temperature', 'Humidity']])
# Initialize models
models = {
'k-Nearest Neighbors': KNeighborsClassifier(n_neighbors=4, weights='distance'),
'Bernoulli Naive Bayes': BernoulliNB(),
'Logistic Regression': LogisticRegression(C=1.5, random_state=42),
'Multilayer Perceptron': MLPClassifier(hidden_layer_sizes=(4, 2), random_state=42, max_iter=2000)
}
# Get predictions and calculate metrics
metrics_dict = {}
for name, model in models.items():
model.fit(X_train, y_train)
y_prob = model.predict_proba(X_test)[:, 1]
metrics_dict[name] = {
'Brier Score': brier_score_loss(y_test, y_prob),
'Log Loss': log_loss(y_test, y_prob),
'ECE': calculate_ece(y_test, y_prob),
'Probabilities': y_prob
}
# Plot calibration curves
fig, axes = plt.subplots(2, 2, figsize=(8, 8), dpi=300)
colors = ['orangered', 'slategrey', 'gold', 'mediumorchid']
for idx, (name, metrics) in enumerate(metrics_dict.items()):
ax = axes.ravel()[idx]
prob_true, prob_pred = calibration_curve(y_test, metrics['Probabilities'],
n_bins=5, strategy='uniform')
ax.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
ax.plot(prob_pred, prob_true, color=colors[idx], marker='o',
label='Calibration curve', linewidth=2, markersize=8)
title = f'{name}nBrier: {metrics[\"Brier Score\"]:.3f} | Log Loss: {metrics[\"Log Loss\"]:.3f} | ECE: {metrics[\"ECE\"]:.3f}'
ax.set_title(title, fontsize=11, pad=10)
ax.grid(True, alpha=0.7)
ax.set_xlim([-0.05, 1.05])
ax.set_ylim([-0.05, 1.05])
ax.spines[['top', 'right', 'left', 'bottom']].set_visible(False)
ax.legend(fontsize=10, loc='upper left')
plt.tight_layout()
plt.show()<\/pre>\nTechnical Environment<\/h4>\n
About the Illustrations<\/h4>\n
\n
<\/p>\n
\n