1fanwang opened a new pull request, #2921: URL: https://github.com/apache/iceberg-rust/pull/2921
## Which issue does this PR close? - Closes #2419 Supersedes https://github.com/apache/iceberg-rust/pull/2438, which the stale bot closed after it sat in the review queue. Same change, rebased onto current `main`, with @xanderbailey's review feedback from that PR already folded in. GitHub refuses to reopen the old PR, hence the new one. ## What changes are included in this PR? The [Puffin spec](https://iceberg.apache.org/puffin-spec/#footer-payload) allows a footer payload to be uncompressed or LZ4-compressed. `PuffinWriter` already exposes a `compress_footer` flag that records LZ4 as the footer codec, and `validate_puffin_compression` already accepts `Lz4` as a valid Puffin codec — but the codec itself was never implemented, so `close()` with `compress_footer = true` failed: ``` FeatureUnsupported => LZ4 compression is not supported currently ``` The reader had the symmetric gap: a Puffin file carrying an LZ4-compressed footer could not be read back. This implements the codec with `lz4_flex` in the frame format — a single frame with content size present, as the spec requires. `lz4_flex` was already in the lock file as a transitive `parquet` dependency, so no new crate enters the tree. ## Are these changes tested? Yes. Unit tests cover the codec and the Puffin writer/reader: - Codec round-trips (empty, highly compressible, less compressible) across LZ4, Zstd and Gzip, plus an LZ4 frame magic-number assertion. - The issue reproducer — `close()` with `compress_footer = true` — now succeeds, and footer metadata plus blobs round-trip through an LZ4 footer. - The reader honors the `FooterPayloadCompressed` flag and decodes an LZ4-compressed footer. The tests drive the real write/read path, not the codec in isolation: reverting the two `Lz4` arms in `compression.rs` makes all four fail at `close()` with the error from the issue. <details><summary>Before the fix — <code>Lz4</code> arms reverted</summary> ``` running 4 tests test compression::tests::test_lz4_frame_magic_number ... FAILED test puffin::metadata::tests::test_lz4_compressed_footer_is_decoded ... FAILED test puffin::writer::tests::test_compress_footer_lz4_round_trips ... FAILED test puffin::writer::tests::test_compress_empty_footer_lz4_succeeds ... FAILED ---- puffin::writer::tests::test_compress_empty_footer_lz4_succeeds stdout ---- thread 'puffin::writer::tests::test_compress_empty_footer_lz4_succeeds' panicked at crates/iceberg/src/puffin/writer.rs:327:30: called `Result::unwrap()` on an `Err` value: FeatureUnsupported => LZ4 compression is not supported currently test result: FAILED. 0 passed; 4 failed ``` </details> <details><summary>After the fix — <code>cargo test -p iceberg --lib -- compression:: puffin::</code></summary> ``` running 47 tests test compression::tests::test_compression_codec_compress ... ok test compression::tests::test_lz4_frame_magic_number ... ok test compression::tests::test_snappy_compression_is_unsupported ... ok test puffin::metadata::tests::test_lz4_compressed_footer_is_decoded ... ok test puffin::writer::tests::test_compress_empty_footer_lz4_succeeds ... ok test puffin::writer::tests::test_compress_footer_lz4_round_trips ... ok test puffin::writer::tests::test_write_lz4_compressed_metric_data ... ok test puffin::writer::tests::test_uncompressed_empty_puffin_file_is_bit_identical_to_java_generated_file ... ok test puffin::writer::tests::test_zstd_compressed_metric_data_is_bit_identical_to_java_generated_file ... ok ... test result: ok. 47 passed; 0 failed; 0 ignored; 0 measured; 1426 filtered out ``` </details> `cargo clippy -p iceberg --all-targets --all-features -- -D warnings`, `cargo fmt --check` and `cargo machete` are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
