GitHub user cshuo edited a discussion: Use ZSTD as the default compression codec for Parquet files
### Summary Hudi MOR tables now use native Parquet log files by default, regardless of whether the LSM file-group layout is enabled. Native log files currently use GZIP compression by default. GZIP provides a good compression ratio, but its relatively low compression throughput adds considerable CPU overhead to the MOR write path. This discussion proposes using ZSTD as the default compression codec for native Parquet log files. ### Motivation For MOR tables, updates are first written to native log files. Readers merge base files and log files, and compaction eventually rewrites the file slice into a new base file. Native log files are therefore intermediate files in the MOR write lifecycle. Compression throughput and CPU efficiency are particularly important because the data will later be rewritten during compaction. The main codec choices are: | Codec | Compression speed | Compression ratio | |---|---|---| | GZIP | Low | High | | Snappy | High | Relatively low | | ZSTD | High | Higher than Snappy and often close to GZIP | ZSTD appears to provide a better balance between write throughput, storage size, and read I/O ([Benchmark](https://github.com/facebook/zstd#benchmarks)) Related Hudi discussions include: - [Evaluate rebasing Hudi's default compression from Gzip to Zstd](https://github.com/apache/hudi/issues/14938) - [Should we change default compression codec for Parquet files to snappy?](https://github.com/apache/hudi/issues/13568) ### Adoption in other table formats Apache Iceberg changed the default Parquet compression codec for new tables from GZIP to ZSTD in Iceberg 1.4.0. Existing tables retained their previous behavior to minimize compatibility impact ([Iceberg PR #8593](https://github.com/apache/iceberg/pull/8593)). Delta Lake has also moved its protocol-level recommendation toward ZSTD. The Delta protocol now defines `delta.parquet.compression.codec` and specifies that writers should default to ZSTD when the property is absent ([Delta PR #6324](https://github.com/delta-io/delta/pull/6324)). Its Spark implementation is still transitioning: Spark-based Delta writes currently retain their existing default, and support for honoring the table property is being developed in [Delta PR #6802](https://github.com/delta-io/delta/pull/6802). ### Known ZSTD off-heap memory issue Older parquet-java versions have a known ZSTD off-heap memory issue: - [PARQUET-2160](https://issues.apache.org/jira/browse/PARQUET-2160) - [parquet-java PR #982](https://github.com/apache/parquet-java/pull/982) The old Parquet decompression path did not explicitly close `ZstdDecompressorStream`. Because the stream holds native resources allocated through `zstd-jni`, long-running jobs could experience off-heap memory growth or fragmentation and eventually run out of memory. The issue was fixed in parquet-java 1.13.0. Spark also introduced a workaround in [SPARK-41952](https://issues.apache.org/jira/browse/SPARK-41952), released in Spark 3.2.4, 3.3.3, and 3.4.0. However, the Spark workaround only covers the vectorized Parquet reader. Hudi's file-group reader uses the non-vectorized reader for row-based record merging, so profiles using parquet-java 1.12.x may still be affected. This is currently relevant to the Spark 3.3 and Spark 3.4 Hudi profiles. ### Proposal Change the default compression codec for native Parquet log files from GZIP to ZSTD. For dependency profiles using parquet-java 1.12.x, Hudi should add a scoped workaround equivalent to PARQUET-2160 in its non-vectorized file-group read path. Profiles using parquet-java 1.13.0 or newer should continue using the upstream implementation. The workaround would only protect Hudi's own file-group reader. It would not attempt to modify arbitrary non-vectorized Parquet reads outside Hudi. ### Runtime dependency Parquet's `ZstandardCodec` depends on `zstd-jni`. Before making ZSTD the default, Hudi should verify that supported runtime environments provide a compatible `zstd-jni` dependency. Where the runtime does not guarantee this dependency, the corresponding Hudi bundle should package it explicitly. GitHub link: https://github.com/apache/hudi/discussions/19615 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
