{"id":3308,"date":"2025-04-24T07:03:16","date_gmt":"2025-04-24T07:03:16","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/04\/24\/data-science-from-school-to-work-part-iv\/"},"modified":"2025-04-24T07:03:16","modified_gmt":"2025-04-24T07:03:16","slug":"data-science-from-school-to-work-part-iv","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/04\/24\/data-science-from-school-to-work-part-iv\/","title":{"rendered":"Data Science: From School to Work, Part IV"},"content":{"rendered":"

Data Science: From School to Work, Part IV
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

Introduction<\/mdspan><\/h2>\n

Let\u2019s start with a simple example that will appeal to most of us. If you want to check if the blinkers of your car are working properly, you sit in the car, turn on the ignition and test a turn signal to see if the front and rear lights work. But if the lights don\u2019t work, it\u2019s hard to tell why. The bulbs may be dead, the battery may be dead, the turn signal switch may be faulty. In short, there\u2019s a lot to check. This is exactly what the tests are for. Every part of a function such as the blinker must be tested to find out what is going wrong. A test of the bulbs, a test of the battery, a test of the communication between the control unit and the indicators, and so on.<\/p>\n

To test all this, there are different types of tests, often presented in the form of a pyramid, from the fastest to the slowest and from the most isolating to the most integrated. This test pyramid can vary depending on the specifics of the project (database connection test, authentication test, etc.).<\/p>\n

\"\"
Tests pyramid | Image from author.<\/figcaption><\/figure>\n

The Base of the Pyramid: Unit Tests<\/h3>\n

Unit tests form the basis of the test pyramid, regardless of the type of project (and language). Their purpose is to test a unit of code, e.g. a method or a function. For a unit test to be truly considered as such, it must adhere to a basic rule: A unit test must not depend on functionalities outside the unit under test. They have the advantage of being fast and automatable.<\/p>\n

Example:<\/strong> Consider a function that extracts even numbers from an iterable. To test this function, we\u2019d need to create several types of iterable with integers and check the output. But we\u2019d also need to check the behavior in the case of empty iterables, element types other than int, and so on.<\/p>\n

Intermediate Level: Integration and Functional Tests<\/h3>\n

Just above the unit tests are the integration tests. Their purpose is to detect errors that cannot be detected by unit tests. These tests check that the addition of a new feature does not cause problems when it is integrated into the application. The functional tests are similar but aim at testing one precise fonctionality (e.g an authentification process).<\/p>\n

In a project, especially in a team environment, many functions are developed by different developers. Integration\/functional tests ensure that all these features work well together. They are also run automatically, making them fast and reliable.<\/p>\n

Example<\/strong>: Consider an application that displays a bank balance. When a withdrawal operation is carried out, the balance is modified. An integration test would be to check that with a balance initialized at 1000 euros, then a withdrawal of 500 euros, the balance changes to 500 euros.<\/p>\n

The Top of the Pyramid: End-to-End Tests<\/h3>\n

End-to-end (E2E) tests are tests at the top of the pyramid. They verify that the application functions as expected from end to end, i.e. from the user interface to the database or external services. They are generally long and complicated to set up, but there\u2019s no need for a lot of tests.<\/p>\n

Example<\/strong>: Consider a forecasting application based on new data. This can be very complex, involving data retrieval, variable transformations, learning and so on. The aim of the End-to-End test is to check that, given the new data selected, the forecasts correspond to expectations.<\/p>\n

\n

The Unit Tests with Doctest<\/h2>\n

A fast and simple way of making unit tests is to use docstring<\/code>. Let\u2019s take the example of a script calculate_stats.py<\/code> with two functions: calculate_mean()<\/code> with a complete docstring, which was presented in Python best practices<\/a>, and the function calculate_std()<\/code> <\/strong>with a usual one.<\/p>\n

import math\nfrom typing import List\n\ndef calculate_mean(numbers: List[float]) -> float:\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 Calculate the mean of a list of numbers.\n\n\u00a0 \u00a0 Parameters\n\u00a0 \u00a0 ----------\n\u00a0 \u00a0 numbers : list of float\n\u00a0 \u00a0 \u00a0 \u00a0 A list of numerical values for which the mean is to be calculated.\n\n\u00a0 \u00a0 Returns\n\u00a0 \u00a0 -------\n\u00a0 \u00a0 float\n\u00a0 \u00a0 \u00a0 \u00a0 The mean of the input numbers.\n\n\u00a0 \u00a0 Raises\n\u00a0 \u00a0 ------\n\u00a0 \u00a0 ValueError\n\u00a0 \u00a0 \u00a0 \u00a0 If the input list is empty.\n\n\u00a0 \u00a0 Notes\n\u00a0 \u00a0 -----\n\u00a0 \u00a0 The mean is calculated as the sum of all elements divided by the number of elements.\n\n\u00a0 \u00a0 Examples\n\u00a0 \u00a0 --------\n\u00a0 \u00a0 >>> calculate_mean([1.0, 2.0, 3.0, 4.0])\n\u00a0 \u00a0 2.5\n\u00a0 \u00a0 >>> calculate_mean([])\n\u00a0 \u00a0 0\n\u00a0 \u00a0 \"\"\"\n\n\u00a0 \u00a0 if len(numbers) > 0:\n\u00a0 \u00a0 \u00a0 \u00a0 return sum(numbers) \/ len(numbers)\n\u00a0 \u00a0 else:\n\u00a0 \u00a0 \u00a0 \u00a0 return 0\n\ndef calculate_std(numbers: List[float]) -> float:\n\u00a0 \u00a0 \"\"\"\n\u00a0 \u00a0 Calculate the standard deviation of a list of numbers.\n\n\u00a0 \u00a0 Parameters\n\u00a0 \u00a0 ----------\n\u00a0 \u00a0 numbers : list of float\n\u00a0 \u00a0 \u00a0 \u00a0 A list of numerical values for which the mean is to be calculated.\n\n\u00a0 \u00a0 Returns\n\u00a0 \u00a0 -------\n\u00a0 \u00a0 float\n\u00a0 \u00a0 \u00a0 \u00a0 The std of the input numbers.\n\u00a0 \u00a0 \"\"\"\n\n\u00a0 \u00a0 if len(numbers) > 0:\n\u00a0 \u00a0 \u00a0 \u00a0 m = calculate_mean(numbers)\n\u00a0 \u00a0 \u00a0 \u00a0 gap = [abs(x - m)**2 for x in numbers]\n\u00a0 \u00a0 \u00a0 \u00a0 return math.sqrt(sum(gap) \/ (len(numbers) - 1))\n\u00a0 \u00a0 else:\n\u00a0 \u00a0 \u00a0 \u00a0 return 0<\/code><\/pre>\n
The test is included in the \u201cExamples\u201d section at the end of the docstring of the function calculate_mean()<\/code>. A doctest follows the layout of a terminal: three chevrons at the beginning of a line with the command to be executed and the expected result just below. To run the tests, simply type the command<\/p>\n
  python -m doctests calculate_stats.py -v<\/code><\/pre>\n
or if you use uv<\/a> (what I encourage)<\/p>\n
uv run python -m doctest calculate_stats.py -v<\/code><\/pre>\n
The -v<\/code> argument permits to display the following output:<\/p>\n
\"\"<\/figure>\n
As you can see that there have been two tests and no failures, and doctest <\/code>has the intelligence to point out all the methods that don\u2019t have a test (as with calculate_std()<\/code>).<\/p>\n
\n
The Unit Tests with Pytest<\/h2>\n
Using doctest <\/code>is interesting, but quickly becomes limited. For a truly comprehensive testing process, we use a specific framework. There are two main frameworks for testing: unittest <\/code>and Pytest<\/a><\/code>. The latter is generally considered simpler and more intuitive.<\/p>\n
To install the package, simply type:<\/p>\n
pip install pytest (in your virtual environment)<\/code><\/pre>\n
or<\/p>\n
uv add pytest<\/code><\/pre>\n
1 \u2013 Write your first test<\/h3>\n
Let\u2019s take the calculate_stats.py<\/code> <\/strong>script and write a test for the calculate_mean() <\/code>function. To do this, we create a script test_calculate_stats.py<\/code> containing the following lines:<\/p>\n
from calculate_stats import calculate_mean\n\ndef test_calculate_mean():\n    assert calculate_mean([1, 2, 3, 4, 5, 6]) == 3.5<\/code><\/pre>\n
Tests are based on the assert command. This instruction is used with the following syntax:<\/p>\n
\n
assert expression1 [, expression2]<\/p>\n<\/blockquote>\n
The expression1 is the condition to be tested, and the optional expression2 is the error message if the condition is not verified.<\/p>\n
The Python<\/a> interpreter transforms each assert statement into:<\/p>\n
if __debug__:\n    if not expression1:\n        raise AssertionError(expression2)<\/code><\/pre>\n
2 \u2013 Run a test<\/h3>\n
To run the test, we use the following command:<\/p>\n
pytest (in your virtual environment)<\/code><\/pre>\n
or<\/p>\n
uv run pytest<\/code><\/pre>\n
The result is as follows:<\/p>\n
\"\"<\/figure>\n
3 \u2013 Analyse the output<\/h3>\n
One of the great advantages of pytest <\/code>is the quality of its feedback. For each test, you get:<\/p>\n