Another challenge was to somehow stick with a traditional approach to have data stored in a structured way, but without the necessity to design complex and time-consuming ETL workloads to move this data into the enterprise data warehouse. Additionally, what if half of the data professionals in your organization are proficient with, let\u2019s say, Python (data scientists, data engineers), and the other half (data engineers, data analysts) with SQL? Would you insist that \u201cPythonists\u201d learn SQL? Or, vice-versa?<\/p>\n Or, would you prefer a storage option that can play to the strengths of your entire data team? I have good news for you \u2013 something like this has already existed since 2013, and it\u2019s called Apache Parquet<\/a>!<\/p>\n Before I show you the ins and outs of the Parquet file format, there are (at least) five main reasons why Parquet is considered a de facto standard for storing data nowadays:<\/p>\n We\u2019ve already mentioned that Parquet is a column-based storage format. However, to understand the benefits of using the Parquet file format, we first need to draw the line between the row-based and column-based ways of storing the data.<\/p>\n In traditional, row-based storage, the data is stored as a sequence of rows. Something like this:<\/p>\n Now, when we are talking about\u00a0OLAP<\/a>\u00a0scenarios, some of the common questions that your users may ask are:<\/p>\n To be able to answer any of these questions, the engine must scan each and every row from the beginning to the very end! So, to answer the question: how many users from the USA bought T-shirt, the engine has to do something like this:<\/p>\n Essentially, we just need the information from two columns: Product (T-Shirts) and Country (USA), but the engine will scan all five columns! This is not the most efficient solution \u2013 I think we can agree on that\u2026<\/p>\n Let\u2019s now examine how the column store works. As you may assume, the approach is 180 degrees different:<\/p>\n In this case, each column is a separate entity \u2013 meaning, each column is physically separated from other columns! Going back to our previous business question: the engine can now scan only those columns that are needed by the query (Product and country), while\u00a0skipping scanning<\/em>\u00a0the unnecessary columns. And, in most cases, this should improve the performance of the analytical queries.<\/p>\n Ok, that\u2019s nice, but the column store existed before Parquet and it still exists outside of Parquet as well. So, what is so special about the Parquet format?<\/p>\n Wait, what?! Wasn\u2019t it complicated enough even before this? Don\u2019t worry, it\u2019s much easier than it sounds Let\u2019s go back to our previous example and depict how Parquet will store this same chunk of data:<\/p>\n Let\u2019s stop for a moment and explain the illustration above, as this is exactly the structure of the Parquet file (some additional things were intentionally omitted, but we will come soon to explain that as well). Columns are still stored as separate units, but Parquet introduces additional structures, called Row group.<\/p>\n Why is this additional structure super important?<\/p>\n You\u2019ll need to wait for an answer for a bit :). In OLAP scenarios, we are mainly concerned with two concepts:\u00a0projection<\/em><\/strong>\u00a0and\u00a0predicate(s)<\/em><\/strong>. Projection refers to a\u00a0SELECT<\/strong>\u00a0statement in SQL language \u2013 which columns are needed by the query. Back to our previous example, we need only the Product and Country columns, so the engine can skip scanning the remaining ones.<\/p>\n Predicate(s) refer to the\u00a0WHERE<\/strong>\u00a0clause in SQL language \u2013 which rows satisfy criteria defined in the query. In our case, we are interested in T-Shirts only, so the engine can completely skip scanning Row group 2, where all the values in the Product column equal socks!<\/p>\n Let\u2019s quickly stop here, as I want you to realize the difference between various types of storage in terms of the work that needs to be performed by the engine:<\/p>\n Obviously, this is an oversimplified example, with only 6 rows and 5 columns, where you will definitely not see any difference in performance between these three storage options. However, in real life, when you\u2019re dealing with much larger amounts of data, the difference becomes more evident.<\/p>\n Now, the fair question would be: how does Parquet \u201cknow\u201d which row group to skip\/scan?<\/p>\n This means that every Parquet file contains \u201cdata about data\u201d \u2013 information such as minimum and maximum values in a specific column within a certain row group. Furthermore, every Parquet file contains a footer, which keeps the information about the format version, schema information, column metadata, and so on. You can find more details about Parquet metadata types\u00a0here<\/a>.<\/p>\n Important:<\/strong>\u00a0In order to optimize the performance and eliminate unnecessary data structures (row groups and columns), the engine first needs to \u201cget familiar\u201d with the data, so it first reads the metadata. It\u2019s not a slow operation, but it still requires a certain amount of time. Therefore, if you\u2019re querying the data from multiple small Parquet files, query performance can degrade, because the engine will have to read metadata from each file. So, you should be better off merging multiple smaller files into one bigger file (but still not too big :)\u2026<\/p>\n I hear you, I hear you: Nikola, what is \u201csmall\u201d and what is \u201cbig\u201d? Unfortunately, there is no single \u201cgolden\u201d number here, but for example,\u00a0Microsoft Azure Synapse Analytics recommends that the individual Parquet file should be at least a few hundred MBs in size<\/a>.<\/p>\n Here is a simplified, high-level illustration of the Parquet file format:<\/p>\n Ok, we\u2019ve explained how skipping the scan of the unnecessary data structures (row groups and columns) may benefit your queries and increase the overall performance. But, it\u2019s not only about that \u2013 remember when I told you at the very beginning that one of the main advantages of the Parquet format is the reduced memory footprint of the file? This is achieved by applying various compression algorithms.<\/p>\n I\u2019ve already written about various data compression types in Power BI (and the Tabular model in general)\u00a0here<\/a>, so maybe it\u2019s a good idea to start by reading this article.<\/p>\n There are two main encoding types that enable Parquet to compress the data and achieve astonishing savings in space:<\/p>\n You might think: why this overhead, when product names are quite short, right? Ok, but now imagine that you store the detailed description of the product, such as: \u201cLong arm T-Shirt with application on the neck\u201d. And, now imagine that you have this product sold million times\u2026Yeah, instead of having million times repeating value \u201cLong arm\u2026bla bla\u201d, the Parquet will store only the Index value (integer instead of text).<\/p>\n Ok, what the heck is now a Delta Lake format?! This is the article about Parquet, right?<\/p>\n So, to put it in plain English: Delta Lake is nothing else but the Parquet format \u201con steroids\u201d.\u00a0<\/em><\/strong>When I say \u201csteroids\u201d, the main one is the versioning of Parquet files. It also stores a transaction log to enable tracking all changes applied to the Parquet file. This is also known as\u00a0ACID-compliant transactions<\/a>.<\/p>\n Since it supports not only ACID transactions, but also supports time travel (rollbacks, audit trails, etc.) and DML (Data Manipulation Language) statements, such as INSERT, UPDATE and DELETE, you won\u2019t be wrong if you think of the Delta Lake as a \u201cdata warehouse on the data lake\u201d (who said: Lakehouse We evolve! Same as we, the data is also evolving. So, new flavors of data required new ways of storing it. The Parquet file format is one of the most efficient storage options in the current data landscape, since it provides multiple benefits \u2013 both in terms of memory consumption, by leveraging various compression algorithms, and fast query processing by enabling the engine to skip scanning unnecessary data.<\/p>\n Thanks for reading!<\/p>\n The post Parquet File Format \u2013 Everything You Need to Know!<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nParquet file format in a nutshell<\/h3>\n
\n
Row-store vs Column-store<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-106.png?ssl=1\")
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-107.png?ssl=1\")
Column store<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-108.png?ssl=1\")
Parquet is a columnar format that stores the data in row groups<\/h2>\n
![\"\ud83d\ude42\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f642.png?ssl=1\")
<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-109-1024x355.png?resize=1024%2C355&ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-110-1024x390.png?resize=1024%2C390&ssl=1\")
\n
Parquet file contains metadata<\/h3>\n
What else is in there?<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/111.png?ssl=1\")
Can it be better than this? Yes, with data compression<\/h2>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/image-111.png?ssl=1\")
\n
Can it be better than\u00a0THIS?! Yes, with the Delta Lake file format<\/h2>\n
![\"\ud83d\ude09\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f609.png?ssl=1\")
). Examining the pros and cons of the Lakehouse concept is out of the scope of this article, but if you\u2019re curious to go deeper into this, I suggest you read\u00a0this article<\/a>\u00a0from Databricks.<\/p>\nConclusion<\/h3>\n