In recent years, Parquet has become a standard format for data storage in Big Data<\/a> ecosystems. Its column-oriented format offers several advantages:<\/p>\n When combined with storage frameworks like Delta Lake or Apache Iceberg, it seamlessly integrates with query engines (e.g., Trino) and data warehouse compute clusters (e.g., Snowflake, BigQuery). In this article, the content of a Parquet file is dissected using mainly standard Python tools to better understand its structure and how it contributes to such performances.<\/p>\n To produce Parquet files, we use PyArrow, a Python binding for Apache Arrow that stores dataframes in memory in columnar format. PyArrow allows fine-grained parameter tuning when writing the file. This makes PyArrow ideal for Parquet manipulation (one can also simply use Pandas<\/a>).<\/p>\n The output clearly reflects a columns-oriented storage, unlike Pandas, which usually displays a traditional \u201crow-wise\u201d table.<\/p>\n Parquet files are generally stored in cheap object storage databases like S3 (AWS) or GCS (GCP) to be easily accessible by data processing pipelines. These files are usually organized with a partitioning strategy by leveraging directory structures:<\/p>\n If The strategy enables partition pruning: when a query filters on these columns, the engine can use folder names to read only the necessary files. This is why the partitioning strategy is crucial for limiting delay, I\/O, and compute resources when handling large volumes of data (as has been the case for decades with traditional relational databases).<\/p>\n The pruning effect can be easily verified by counting the files opened by a Python script that filters the birth year:<\/p>\n Only 23 files are read out of 100.<\/p>\n Let\u2019s decode a raw Parquet file without specialized libraries. For simplicity, the dataset is dumped into a single file without compression or encoding.<\/p>\n The first thing to know is that the binary file is framed by 4 bytes whose ASCII representation is \u201cPAR1\u201d. The file is corrupted if this is not the case.<\/p>\n As indicated in the documentation<\/a>, the file is divided into two parts: the \u201crow groups\u201d containing actual data, and the footer containing metadata (schema below).<\/p>\n The size of the footer is indicated in the 4 bytes preceding the end marker as an unsigned integer written in \u201clittle endian\u201d format (noted \u201c<I\u201d for the The footer information is encoded in a cross-language serialization format called Apache Thrift<\/a>. Using a human-readable but verbose format like JSON and then translating it into binary would be less efficient in terms of memory usage. With Thrift, one can declare data structures as follows:<\/p>\n On the basis of this declaration, Thrift can generate Python code to decode byte strings with such data structure (it also generates code to perform the encoding part). The thrift file containing all the data structures implemented in a Parquet file can be downloaded here<\/a>. After having installed the thrift binary, let\u2019s run:<\/p>\n The generated Python code is placed in the \u201cgen-py\u201d folder. The footer\u2019s data structure is represented by the FileMetaData class \u2013 a Python class automatically generated from the Thrift schema. Using Thrift\u2019s Python utilities, binary data is parsed and populated into an instance of this FileMetaData class.<\/p>\n The footer contains extensive information about the file\u2019s structure and content. For instance, it accurately tracks the number of rows in the generated dataframe. These rows are all contained within a single \u201crow group.\u201d But what is a \u201crow group?\u201d<\/em><\/p>\n Unlike purely column-oriented formats, Parquet employs a hybrid approach. Before writing column blocks, the dataframe is first partitioned vertically into row groups (the parquet file we generated is too small to be split in multiple row groups).<\/p>\n This hybrid structure offers several advantages:<\/p>\n Parquet calculates statistics (such as min\/max values) for each column within each row group. These statistics are crucial for query optimization, allowing query engines to skip entire row groups that don\u2019t match filtering criteria. For example, if a query filters for The smallest substructure of a Parquet file is the page. It contains a sequence of values from the same column and, therefore, of the same type. The choice of page size is the result of a trade-off:<\/p>\n Now let\u2019s decode the contents of the first page of the column dedicated to addresses whose location can be found in the footer (given by the The generated values finally appear, in plain text and not encoded (as specified when writing the Parquet file). However, to optimize the columnar format, it is recommended to use one of the following encoding algorithms: dictionary encoding, run length encoding (RLE), or delta encoding (the latter being reserved for int32 and int64 types), followed by compression using gzip or snappy (available codecs are listed here<\/a>). Since encoded pages contain similar values (all addresses, all decimal numbers, etc.), compression ratios can be particularly advantageous.<\/p>\n As documented in the specification<\/a>, when character strings (BYTE_ARRAY) are not encoded, each value is preceded by its size represented as a 4-byte integer. This can be observed in the previous output:<\/p>\n To read all the values (for example, the first 10), the loop is rather simple:<\/p>\n And there we have it! We have successfully recreated, in a very simple way, how a specialized library would read a Parquet file. By understanding its building blocks including headers, footers, row groups, and data pages, we can better appreciate how features like predicate pushdown and partition pruning deliver such impressive performance benefits in data-intensive environments. I am convinced knowing how Parquet works under the hood helps making better decisions about storage strategies, compression choices, and performance optimization.<\/p>\n All the code used in this article is available on my GitHub repository at https:\/\/github.com\/kili-mandjaro\/anatomy-parquet<\/a>, where you can explore more examples and experiment with different Parquet file configurations.<\/p>\n Whether you are building data pipelines, optimizing query performance, or simply curious about data storage formats, I hope this deep dive into Parquet\u2019s inner structures has provided valuable insights for your Data Engineering<\/a> journey.<\/p>\n All images are by the author.<\/em><\/p>\n The post Anatomy of a Parquet File<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
Writing Parquet file(s)<\/h2>\n
# generator.py\n\nimport pyarrow as pa\nimport pyarrow.parquet as pq\nfrom faker import Faker\n\nfake = Faker()\nFaker.seed(12345)\nnum_records = 100\n\n# Generate fake data\nnames = [fake.name() for _ in range(num_records)]\naddresses = [fake.address().replace(\"n\", \", \") for _ in range(num_records)]\nbirth_dates = [\n fake.date_of_birth(minimum_age=67, maximum_age=75) for _ in range(num_records)\n]\ncities = [addr.split(\", \")[1] for addr in addresses]\nbirth_years = [date.year for date in birth_dates]\n\n# Cast the data to the Arrow format\nname_array = pa.array(names, type=pa.string())\naddress_array = pa.array(addresses, type=pa.string())\nbirth_date_array = pa.array(birth_dates, type=pa.date32())\ncity_array = pa.array(cities, type=pa.string())\nbirth_year_array = pa.array(birth_years, type=pa.int32())\n\n# Create schema with non-nullable fields\nschema = pa.schema(\n [\n pa.field(\"name\", pa.string(), nullable=False),\n pa.field(\"address\", pa.string(), nullable=False),\n pa.field(\"date_of_birth\", pa.date32(), nullable=False),\n pa.field(\"city\", pa.string(), nullable=False),\n pa.field(\"birth_year\", pa.int32(), nullable=False),\n ]\n)\n\ntable = pa.Table.from_arrays(\n [name_array, address_array, birth_date_array, city_array, birth_year_array],\n schema=schema,\n)\n\nprint(table)<\/code><\/pre>\npyarrow.Table\nname: string not null\naddress: string not null\ndate_of_birth: date32[day] not null\ncity: string not null\nbirth_year: int32 not null\n----\nname: [[\"Adam Bryan\",\"Jacob Lee\",\"Candice Martinez\",\"Justin Thompson\",\"Heather Rubio\"]]\naddress: [[\"822 Jennifer Field Suite 507, Anthonyhaven, UT 98088\",\"292 Garcia Mall, Lake Belindafurt, IN 69129\",\"31738 Jonathan Mews Apt. 024, East Tammiestad, ND 45323\",\"00716 Kristina Trail Suite 381, Howelltown, SC 64961\",\"351 Christopher Expressway Suite 332, West Edward, CO 68607\"]]\ndate_of_birth: [[1955-06-03,1950-06-24,1955-01-29,1957-02-18,1956-09-04]]\ncity: [[\"Anthonyhaven\",\"Lake Belindafurt\",\"East Tammiestad\",\"Howelltown\",\"West Edward\"]]\nbirth_year: [[1955,1950,1955,1957,1956]]<\/code><\/pre>\nHow is a Parquet file stored?<\/h2>\n
# generator.py\n\nnum_records = 100\n\n# ...\n\n# Writing the parquet files to disk\npq.write_to_dataset(\n table,\n root_path='dataset',\n partition_cols=['birth_year', 'city']\n)<\/code><\/pre>\nbirth_year<\/code> and city columns<\/code> are defined as partitioning keys, PyArrow creates such a tree structure in the directory dataset:<\/p>\ndataset\/\n\u251c\u2500 birth_year=1949\/\n\u251c\u2500 birth_year=1950\/\n\u2502 \u251c\u2500 city=Aaronbury\/\n\u2502 \u2502 \u251c\u2500 828d313a915a43559f3111ee8d8e6c1a-0.parquet\n\u2502 \u2502 \u251c\u2500 828d313a915a43559f3111ee8d8e6c1a-0.parquet\n\u2502 \u2502 \u251c\u2500 \u2026\n\u2502 \u251c\u2500 city=Alicialand\/\n\u2502 \u251c\u2500 \u2026\n\u251c\u2500 birth_year=1951 \u251c\u2500 ...\n<\/code><\/pre>\n# query.py\nimport duckdb\n\nduckdb.sql(\n \"\"\"\n SELECT * \n FROM read_parquet('dataset\/*\/*\/*.parquet', hive_partitioning = true)\n where birth_year = 1949\n \"\"\"\n).show()<\/code><\/pre>\n> strace -e trace=open,openat,read -f python query.py 2>&1 | grep \"dataset\/.*.parquet\"\n\n[pid 37] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Box%201306\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 37] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Box%201306\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Box%201306\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Box%203487\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Box%203487\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Clarkemouth\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Clarkemouth\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=DPO%20AP%2020198\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=DPO%20AP%2020198\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=East%20Morgan\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=East%20Morgan\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=FPO%20AA%2006122\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=FPO%20AA%2006122\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=New%20Michelleport\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=New%20Michelleport\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=North%20Danielchester\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=North%20Danielchester\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Port%20Chase\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Port%20Chase\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Richardmouth\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Richardmouth\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 4\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Robbinsshire\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 5\n[pid 39] openat(AT_FDCWD, \"dataset\/birth_year=1949\/city=Robbinsshire\/e1ad1666a2144fbc94892d4ac1234c64-0.parquet\", O_RDONLY) = 3<\/code><\/pre>\nReading a raw Parquet file<\/h2>\n
# generator.py\n\n# ...\n\npq.write_table(\n table,\n \"dataset.parquet\",\n use_dictionary=False,\n compression=\"NONE\",\n write_statistics=True,\n column_encoding=None,\n)<\/code><\/pre>\n# reader.py\n\nwith open(\"dataset.parquet\", \"rb\") as file:\n parquet_data = file.read()\n\nassert parquet_data[:4] == b\"PAR1\", \"Not a valid parquet file\"\nassert parquet_data[-4:] == b\"PAR1\", \"File footer is corrupted\"<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/file.png?ssl=1\")
<\/figure>\nThe footer<\/h3>\n
unpack<\/code> function).<\/p>\n# reader.py\n\nimport struct\n\n# ...\n\nfooter_length = struct.unpack(\"<I\", parquet_data[-8:-4])[0]\nprint(f\"Footer size in bytes: {footer_length}\")\n\nfooter_start = len(parquet_data) - footer_length - 8\nfooter_data = parquet_data[footer_start:-8]<\/code><\/pre>\nFooter size in bytes: 1088<\/code><\/pre>\nstruct Customer {\n\t1: required string name,\n\t2: optional i16 birthYear,\n\t3: optional list<string> interests\n}<\/code><\/pre>\nthrift -r --gen py parquet.thrift<\/code><\/pre>\n# reader.py\n\nimport sys\n\n# ...\n\n# Add the generated classes to the python path\nsys.path.append(\"gen-py\")\nfrom parquet.ttypes import FileMetaData, PageHeader\nfrom thrift.transport import TTransport\nfrom thrift.protocol import TCompactProtocol\n\ndef read_thrift(data, thrift_instance):\n \"\"\"\n Read a Thrift object from a binary buffer.\n Returns the Thrift object and the number of bytes read.\n \"\"\"\n transport = TTransport.TMemoryBuffer(data)\n protocol = TCompactProtocol.TCompactProtocol(transport)\n thrift_instance.read(protocol)\n return thrift_instance, transport._buffer.tell()\n\n# The number of bytes read is not used for now\nfile_metadata_thrift, _ = read_thrift(footer_data, FileMetaData())\n\nprint(f\"Number of rows in the whole file: {file_metadata_thrift.num_rows}\")\nprint(f\"Number of row groups: {len(file_metadata_thrift.row_groups)}\")\n\nNumber of rows in the whole file: 100\nNumber of row groups: 1<\/code><\/pre>\nRow groups<\/strong><\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/Parquet-2-1024x612.png?resize=1024%2C612&ssl=1\")
<\/figure>\nbirth_year > 1955<\/code> and a row group\u2019s maximum birth year is 1954, the engine can efficiently skip that entire data section. This optimisation is called \u201cpredicate pushdown\u201d. Parquet also stores other useful statistics like distinct value counts and null counts.<\/p>\n# reader.py\n# ...\n\nfirst_row_group = file_metadata_thrift.row_groups[0]\nbirth_year_column = first_row_group.columns[4]\n\nmin_stat_bytes = birth_year_column.meta_data.statistics.min\nmax_stat_bytes = birth_year_column.meta_data.statistics.max\n\nmin_year = struct.unpack(\"<I\", min_stat_bytes)[0]\nmax_year = struct.unpack(\"<I\", max_stat_bytes)[0]\n\nprint(f\"The birth year range is between {min_year} and {max_year}\")<\/code><\/pre>\nThe birth year range is between 1949 and 1958<\/code><\/pre>\n\n
row_group_size<\/code> property in function write_table<\/code> when using PyArrow).\n<\/li>\n<\/ul>\n# generator.py\n\n# ...\n\npq.write_table(\n table,\n \"dataset.parquet\",\n row_group_size=100,\n)\n\n# \/! Keep the default value of \"row_group_size\" for the next parts<\/code><\/pre>\n\n
Data Pages<\/strong><\/h3>\n
\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/Parquet-3.png?ssl=1\")
<\/figure>\ndata_page_offset<\/code> attribute of the right ColumnMetaData<\/code>) . Each page is preceded by a Thrift PageHeader<\/code> object containing some metadata. The offset actually points to a Thrift binary representation of the page metadata that precedes the page itself. The Thrift class is called a PageHeader<\/code> and can also be found in the gen-py<\/code> directory.<\/p>\n\n
![\"\ud83d\udca1\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/1f4a1.png?ssl=1\")
Between the PageHeader and the actual values contained within the page, there may be a few bytes dedicated to implementing the <\/em>Dremel<\/em><\/a> format, which allows encoding <\/em>nested data structures<\/em><\/a>. Since our data has a regular tabular format and the values are not nullable, these bytes are skipped when writing the file (<\/em>https:\/\/parquet.apache.org\/docs\/file-format\/data-pages\/<\/em><\/a>).<\/em><\/p>\n<\/blockquote>\n# reader.py\n# ...\n\naddress_column = first_row_group.columns[1]\ncolumn_start = address_column.meta_data.data_page_offset\ncolumn_end = column_start + address_column.meta_data.total_compressed_size\ncolumn_content = parquet_data[column_start:column_end]\n\npage_thrift, page_header_size = read_thrift(column_content, PageHeader())\npage_content = column_content[\n page_header_size : (page_header_size + page_thrift.compressed_page_size)\n]\nprint(column_content[:100])<\/code><\/pre>\nb'6x00x00x00481 Mata Squares Suite 260, Lake Rachelville, KY 874642x00x00x00671 Barker Crossing Suite 390, Mooreto'<\/code><\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/Screenshot-2025-03-13-at-5.52.46%25E2%2580%25AFPM-1024x395.png?resize=1024%2C395&ssl=1\")
<\/figure>\nidx = 0\nfor _ in range(10):\n str_size = struct.unpack(\"<I\", page_content[idx : (idx + 4)])[0]\n print(page_content[(idx + 4) : (idx + 4 + str_size)].decode())\n idx += 4 + str_size<\/code><\/pre>\n481 Mata Squares Suite 260, Lake Rachelville, KY 87464\n671 Barker Crossing Suite 390, Mooretown, MI 21488\n62459 Jordan Knoll Apt. 970, Emilyfort, DC 80068\n948 Victor Square Apt. 753, Braybury, RI 67113\n365 Edward Place Apt. 162, Calebborough, AL 13037\n894 Reed Lock, New Davidmouth, NV 84612\n24082 Allison Squares Suite 345, North Sharonberg, WY 97642\n00266 Johnson Drives, South Lori, MI 98513\n15255 Kelly Plains, Richardmouth, GA 33438\n260 Thomas Glens, Port Gabriela, OH 96758<\/code><\/pre>\n