GitHub user rangareddy added a comment to the discussion: Use ZSTD as the
default compression codec for Parquet files
+1 to this proposal. Switching native Parquet log files from GZIP to ZSTD makes
complete sense for MOR workloads.
Since log files are intermediate representations that undergo frequent write
and compaction cycles, prioritizing compression throughput and CPU efficiency
without a significant penalty on compression ratio is a huge win for MOR write
latency.
A few thoughts on the implementation & safeguard details:
- **Off-Heap Safeguard (PARQUET-2160):** Porting a scoped fix for Hudi's
non-vectorized file-group reader on older parquet-java 1.12.x profiles (Spark
3.3/3.4) is critical to avoid off-heap memory leaks in long-running
streaming/compaction jobs.
- **zstd-jni Bundle Verification:** Ensuring zstd-jni is shaded/packaged
cleanly in hudi-spark-bundle across supported Spark versions will prevent
missing dependency runtime errors.
- **Benchmarking:** It would be great to run a quick benchmark on standard MOR
workloads (e.g., 100GB update-heavy ingestion) comparing GZIP vs ZSTD to
measure:
1. MOR write throughput / CPU cycles saved.
2. Log file size delta.
3. Compaction read/write performance.
Thanks for putting together a detailed proposal!
GitHub link:
https://github.com/apache/hudi/discussions/19615#discussioncomment-17996833
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
