Awesome06 commented on issue #55272: URL: https://github.com/apache/arrow/issues/55272#issuecomment-6719708638
I ran benchmarks comparing Parquet Data Page V1 vs V2 across **TPC-H**, **SSB**, and **TPC-DS** at SF 1.0 (PyArrow 26.0). **TL;DR:** The file size penalty of V2 ranges from **0.0%** on clean flat schemas to **~1.2%** on wide tables with scattered nulls, while V2 reads **12% to 35% faster**. | Benchmark (SF 1.0) | Rows | Codec | V1 Size | V2 Size | Delta (%) | | :--- | :--- | :--- | :--- | :--- | :--- | | **TPC-H `lineitem`** (0 nulls) | 6.0M | Snappy / ZSTD | 208 MB / 162 MB | 208 MB / 162 MB | **+0.00% / -0.02% (V2 smaller)** | | **SSB `lineorder`** (clean) | 6.0M | Snappy / ZSTD | 179 MB / 143 MB | 179 MB / 143 MB | **+0.00% / -0.01% (V2 smaller)** | | **SSB `lineorder`** (2% nulls) | 6.0M | Snappy / ZSTD | 180 MB / 144 MB | 180 MB / 145 MB | **+0.24% / +0.33%** | | **TPC-DS `store_sales`** (sparse nulls) | 2.88M | Snappy / ZSTD | 121 MB / 105 MB | 123 MB / 107 MB | **+1.22% / +1.24%** | | **TPC-DS `catalog_sales`** | 1.44M | Snappy / ZSTD | 101 MB / 86 MB | 101 MB / 86 MB | **+0.16% / +0.13%** | **Why the small difference exists:** * Per the Parquet specification, V2 stores definition/repetition levels **uncompressed** (only values are compressed). * In V1, definition levels are compressed together with values. For sparse/random nulls, bit-packed levels have low entropy, allowing Snappy/ZSTD to compress them by 1.2x to 50x in V1. For clean non-null schemas, RLE levels take only 2-3 bytes anyway, so V1 and V2 are practically identical. **Performance trade-off:** * **Read Throughput:** In `store_sales`, V2 was **~12.1% faster** on full-table scans and **~35.9% faster** on single nullable column reads (`ss_promo_sk`) because uncompressed levels bypass decompression via `memcpy`. * Readers also benefit from explicit `num_nulls` in `DataPageHeaderV2` to short-circuit null handling. Defaulting to V2 seems well worth the minor <= 1.2% size trade-off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
