mailitics

Category: Data Validation

  • Pydantic Performance: 4 Tips on How to Validate Large Amounts of Data Efficiently

    Pydantic Performance: 4 Tips on How to Validate Large Amounts of Data Efficiently The real value lies in writing clearer code and using your tools right The post Pydantic Performance: 4 Tips on How to Validate Large Amounts of Data Efficiently appeared first on Towards Data Science. Mike Huls Go to original source

  • What Advent of Code Has Taught Me About Data Science

    What Advent of Code Has Taught Me About Data Science Five key learnings that I discovered during a programming challenge and how they apply to data science The post What Advent of Code Has Taught Me About Data Science appeared first on Towards Data Science. Jasper Schroeder Go to original source

  • Stop Writing Spaghetti if-else Chains: Parsing JSON with Python’s match-case

    Stop Writing Spaghetti if-else Chains: Parsing JSON with Python’s match-case Introduction If you work in data science, data engineering, or as as a frontend/backend developer, you deal with JSON. For professionals, its basically only death, taxes, and JSON-parsing that is inevitable. The issue is that parsing JSON is often a serious pain. Whether you are…

  • Change-Aware Data Validation with Column-Level Lineage

    Change-Aware Data Validation with Column-Level Lineage Data transformation tools like dbt make constructing SQL data pipelines easy and systematic. But even with the added structure and clearly defined data models, pipelines can still become complex, which makes debugging issues and validating changes to data models difficult. The post Change-Aware Data Validation with Column-Level Lineage appeared…

  • How to Write Queries for Tabular Models with DAX

    How to Write Queries for Tabular Models with DAX Introduction EVALUATE is the statement to query tabular models. Unfortunately, knowing SQL or any other query language doesn’t help as EVALUATE follows a different concept. EVALUATE has only two “Parameters”: A table to show A sort order (ORDER BY) You can pass a third parameter (START…

  • An LLM-Based Workflow for Automated Tabular Data Validation 

    An LLM-Based Workflow for Automated Tabular Data Validation  This article is part of a series of articles on automating data cleaning for any tabular dataset: Effortless Spreadsheet Normalisation With LLM You can test the feature described in this article on your own dataset using the CleanMyExcel.io service, which is free and requires no registration. What…

  • Manage Environment Variables with Pydantic

    Manage Environment Variables with Pydantic Introduction Developers work on applications that are supposed to be deployed on some server in order to allow anyone to use those. Typically in the machine where these apps live, developers set up environment variables that allow the app to run. These variables can be API keys of external services,…