linliu-code opened a new pull request, #20149: URL: https://github.com/apache/hudi/pull/20149
### Describe the issue this Pull Request addresses Closes #14369; part of #14367. The native HFile compressor factory does not support ZSTD, and metadata writes do not inherit the main table's HFile compression setting. This is a tested successor to Ethan's #14378. It depends on #19930: the branch currently includes that PR's three original commits, preserving their authorship. Keep this PR as a draft until #19930 lands, then rebase to remove the prerequisite commits from this diff. #14378 remains open for its author to retire when this successor is accepted. ### Summary and Changelog - Register Ethan's Aircompressor ZSTD stream implementation in the native compressor factory, using bounded `readFully` for decompression. - Propagate `hoodie.hfile.compression.algorithm` into the metadata write configuration. - Calculate checksum length from the on-disk data size excluding checksum bytes, fixing exact checksum-chunk boundaries. - Cover heap/direct read-only buffers, empty and bounded decompression, embedded blocks with checksum boundaries, multi-block ZSTD scans, and MDT compression-setting propagation. - Preserve the existing 50 MB file-cache threshold; drop #14378's unrelated 1000 MB default change. ### Impact Users can opt into ZSTD for MDT HFile base files through `hoodie.hfile.compression.algorithm=ZSTD`. GZ remains the default. Existing NONE/GZ files remain readable. Deploy readers with ZSTD support before enabling ZSTD writes; older native readers do not support this codec. ### Risk Level medium. Adds an opt-in storage codec and propagates its setting into metadata writes. Validation on JDK 11: - Full `hudi-io` suite: 138 tests passed, zero failures/errors/skips; Checkstyle and RAT passed. - `TestHoodieMetadataWriteUtils` with its reactor dependencies: 31 tests passed, zero failures/errors/skips; Checkstyle and RAT passed. - Manual cross-reader validation: native Hudi and HBase 2.4.13 each wrote a multi-block ZSTD HFile with 2,000 records; native full scans and HBase point lookups verified every key/value for both files. The local Hadoop 2 test classpath lacks Hadoop's ZSTD codec, so this check registered an independent zstd-jni 1.5.7-6 stream adapter with HBase. This validates the framing and HFile interoperability but does not exercise Hadoop's built-in JNI codec. ### Documentation Update The existing option and its GZ default are unchanged. The configuration description documents propagation to MDT and the native reader's supported codecs. A website configuration reference update should accompany release publication. ### Contributor's checklist - [ ] Read through contributor's guide - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
