{"id":292,"date":"2024-12-01T07:02:40","date_gmt":"2024-12-01T07:02:40","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2024\/12\/01\/model-validation-techniques-explained-a-visual-guide-with-code-examples-eb13bbdc8f88\/"},"modified":"2024-12-01T07:02:40","modified_gmt":"2024-12-01T07:02:40","slug":"model-validation-techniques-explained-a-visual-guide-with-code-examples-eb13bbdc8f88","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2024\/12\/01\/model-validation-techniques-explained-a-visual-guide-with-code-examples-eb13bbdc8f88\/","title":{"rendered":"Model Validation Techniques, Explained: A Visual Guide with Code Examples"},"content":{"rendered":"

Model Validation Techniques, Explained: A Visual Guide with Code Examples
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

MODEL EVALUATION & OPTIMIZATION<\/h4>\n

12 must-know methods to validate your machine\u00a0learning<\/strong>
\n<\/h4>\n

Every day, machines make millions of predictions\u200a\u2014\u200afrom detecting objects in photos to helping doctors find diseases. But before trusting these predictions, we need to know if they\u2019re any good. After all, no one would want to use a machine that\u2019s wrong most of the\u00a0time!<\/p>\n

This is where validation comes in. Validation methods test machine predictions to measure their reliability. While this might sound simple, different validation approaches exist, each designed to handle specific challenges in machine learning.<\/p>\n

Here, I\u2019ve organized these validation techniques\u200a\u2014\u200aall 12 of them\u200a\u2014\u200ain a tree structure, showing how they evolved from basic concepts into more specialized ones. And of course, we will use clear visuals and a consistent dataset to show what each method does differently and why method selection matters.<\/p>\n

\"\"
All visuals: Author-created using Canva Pro. Optimized for mobile; may appear oversized on\u00a0desktop.<\/figcaption><\/figure>\n

What is Model Validation?<\/h3>\n

Model validation is the process of testing how well a machine learning model works with data it hasn\u2019t seen or used during training. Basically, we use existing data to check the model\u2019s performance instead of using new data. This helps us identify problems before deploying the model for real\u00a0use.<\/p>\n

There are several validation methods, and each method has specific strengths and addresses different validation challenges:<\/p>\n

    \n
  1. Different validation methods can produce different results, so choosing the right method\u00a0matters.<\/li>\n
  2. Some validation techniques work better with specific types of data and\u00a0models.<\/li>\n
  3. Using incorrect validation methods can give misleading results about the model\u2019s true performance.<\/li>\n<\/ol>\n

    Here is a tree diagram showing how these validation methods relate to each\u00a0other:<\/p>\n

    \"\"
    The tree diagram shows which validation methods are connected to each\u00a0other.<\/figcaption><\/figure>\n

    Next, we\u2019ll look at each validation method more closely by showing exactly how they work. To make everything easier to understand, we\u2019ll walk through clear examples that show how these methods work with real\u00a0data.<\/p>\n

    \ud83d\udcca \ud83d\udcc8 Our Running\u00a0Example<\/h3>\n

    We will use the same example throughout to help you understand each testing method. While this dataset may not be appropriate for some validation methods, for education purpose, using this one example makes it easier to compare different methods and see how each one\u00a0works.<\/p>\n

    \ud83d\udcca The Golf Playing\u00a0Dataset<\/h4>\n

    We\u2019ll work with this dataset that predicts whether someone will play golf based on weather conditions.<\/p>\n

    \"\"
    Columns: \u2018Overcast (one-hot-encoded into 3 columns)\u2019, \u2019Temperature\u2019 (in Fahrenheit), \u2018Humidity\u2019 (in %), \u2018Windy\u2019 (Yes\/No) and \u2018Play\u2019 (Yes\/No, target\u00a0feature)<\/figcaption><\/figure>\n
    import pandas as pd
    import numpy as np

    # Load the dataset
    dataset_dict = {
        'Outlook': ['sunny', 'sunny', 'overcast', 'rainy', 'rainy', 'rainy', 'overcast', 
                    'sunny', 'sunny', 'rainy', 'sunny', 'overcast', 'overcast', 'rainy',
                    'sunny', 'overcast', 'rainy', 'sunny', 'sunny', 'rainy', 'overcast',
                    'rainy', 'sunny', 'overcast', 'sunny', 'overcast', 'rainy', 'overcast'],
        'Temperature': [85.0, 80.0, 83.0, 70.0, 68.0, 65.0, 64.0, 72.0, 69.0, 75.0, 75.0,
                       72.0, 81.0, 71.0, 81.0, 74.0, 76.0, 78.0, 82.0, 67.0, 85.0, 73.0,
                       88.0, 77.0, 79.0, 80.0, 66.0, 84.0],
        'Humidity': [85.0, 90.0, 78.0, 96.0, 80.0, 70.0, 65.0, 95.0, 70.0, 80.0, 70.0,
                     90.0, 75.0, 80.0, 88.0, 92.0, 85.0, 75.0, 92.0, 90.0, 85.0, 88.0,
                     65.0, 70.0, 60.0, 95.0, 70.0, 78.0],
        'Wind': [False, True, False, False, False, True, True, False, False, False, True,
                 True, False, True, True, False, False, True, False, True, True, False,
                 True, False, False, True, False, False],
        'Play': ['No', 'No', 'Yes', 'Yes', 'Yes', 'No', 'Yes', 'No', 'Yes', 'Yes', 'Yes',
                 'Yes', 'Yes', 'No', 'No', 'Yes', 'Yes', 'No', 'No', 'No', 'Yes', 'Yes',
                 'Yes', 'Yes', 'Yes', 'Yes', 'No', 'Yes']
    }

    df = pd.DataFrame(dataset_dict)

    # Data preprocessing
    df = pd.DataFrame(dataset_dict)
    df = pd.get_dummies(df, columns=['Outlook'], prefix='', prefix_sep='', dtype=int)
    df['Wind'] = df['Wind'].astype(int)

    # Set the label
    X, y = df.drop('Play', axis=1), df['Play']<\/pre>\n
    \ud83d\udcc8 Our Model\u00a0Choice<\/h4>\n
    We will use a decision tree classifier<\/a> for all our tests. We picked this model because we can easily draw the resulting model as a tree structure, with each branch showing different decisions. To keep things simple and focus on how we test the model, we will use the default scikit-learn parameter with a fixed random_state.<\/p>\n
    Let\u2019s be clear about these two terms we\u2019ll use: The decision tree classifier is our learning algorithm<\/strong>\u200a\u2014\u200ait\u2019s the method that finds patterns in our data. When we feed data into this algorithm, it creates a model<\/strong> (in this case, a tree with clear branches showing different decisions). This model is what we\u2019ll actually use to make predictions.<\/p>\n
    \"\"<\/figure>\n
    from sklearn.tree import DecisionTreeClassifier, plot_tree
    import matplotlib.pyplot as plt

    dt = DecisionTreeClassifier(random_state=42)<\/pre>\n
    Each time we split our data differently for validation, we\u2019ll get different models with different decision rules. Once our validation shows that our algorithm works reliably, we\u2019ll create one final model using all our data. This final model is the one we\u2019ll actually use to predict if someone will play golf or\u00a0not.<\/p>\n
    With this setup ready, we can now focus on understanding how each validation method works and how it helps us make better predictions about golf playing based on weather conditions. Let\u2019s examine each validation method one at a\u00a0time.<\/p>\n
    Hold-out Methods<\/h3>\n
    Hold-out methods are the most basic way to check how well our model works. In these methods, we basically save some of our data just for\u00a0testing.<\/p>\n
    Train-Test Split<\/h4>\n
    This method is simple: we split our data into two parts. We use one part to train our model and the other part to test it. Before we split the data, we mix it up randomly so the order of our original data doesn\u2019t affect our\u00a0results.<\/p>\n
    Both the training and test dataset size depends on our total dataset size, usually denoted by their ratio. To determine their size, you can follow this guideline:<\/p>\n