yihua commented on issue #713: URL: https://github.com/apache/hudi-rs/issues/713#issuecomment-5548633590
# Changelog All notable changes to this project will be documented in this file. ## [unreleased] ### ๐ Features - Add `get_file_slices_splits_between` API (#411) by @yunchipang - Expose datafusion custom table provider to python (#412) by @brayanjuls - Remove table version 5 support (#409) by @yunchipang - Support picking up storage options from `HOODIE_ENV_` env vars (#410) by @yunchipang - Add table v8+ timeline support for loading old and new table versions (#395) by @codope - Add hfile reader (#490) by @xushiyan - Support reading log files from `files` partition in metadata table (#492) by @xushiyan - Add API to read file slice from base file and list of log files (#446) by @yunchipang - Support reading metadata table (files) (#499) by @xushiyan - Implement table v8 file slicing for read path (#501) by @xushiyan - Add streaming table and file group APIs (#508) by @xushiyan - Support column project and row filter pushdown (#510) by @xushiyan - Support column stats pruning (#516) by @xushiyan - Modernize Table API with async-only design (#537) by @xushiyan - Support IN and NOT IN operators in partition filter (#566) by @yunchipang - Export `HudiTableConfig` and `HudiReadConfig` as python enums (#433) by @yunchipang - Unify Hudi read APIs around ReadOptions and add Python bindings (#587) by @xushiyan - *(core)* Enrich FileSlice with stats and streamline config flow (#592) by @xushiyan - *(core)* Add general config alias support with ordering fields as primary config (#598) by @xushiyan - *(core)* Add lance base file format support (#596) by @xushiyan - *(datafusion)* Support Lance scan and broaden MOR snapshot e2e coverage (#608) by @xushiyan - *(core)* Select the file group reader by version, defaulting to 2 (#677) by @linliu-code - *(core)* Serve merge-on-read reads with file group reader version 2 (#660) by @linliu-code - *(core)* Read the metadata table as Arrow, and expose hudi-rs over a C ABI for JVM callers (#700) by @linliu-code - *(reader)* Merge-aware pushdown gate, row-group pruning, and read-volume counters (#708) by @linliu-code - *(core)* Serve Hudi metadata tables through reader v2, sharded and memory-bounded (#706) by @linliu-code - *(deps)* Upgrade to arrow 58, DataFusion 54, object_store 0.13, and Lance 11 (#711) by @yihua - *(benchmark)* Add AWS EC2 setup for the TPC-H harness (#718) by @yihua ### ๐ Bug Fixes - Handle non partition in MDT files partition (#505) by @xushiyan - Avoid dashmap deadlock for correlated queries (#539) by @xushiyan - Make timestamp keygen work for partition pruning (#462) by @yunchipang - Order partition schema by config and error on missing fields (#581) by @sushiljacksparrow - Stop .gitignore from swallowing test fixtures (#676) by @linliu-code - *(core)* Gate the log-block scan on instant state (#678) by @linliu-code - *(core)* Fold partial updates in both ordering directions (#679) by @linliu-code - *(core)* Keep the custom merger's own columns in the required schema (#704) by @linliu-code - *(python)* Compile the datafusion feature into every python build (#714) by @yihua - *(release)* Align the release guide and scripts with their actual behavior (#717) by @yihua - *(release)* Set the macOS deployment target for the x86_64 wheel build (#722) by @yihua - *(benchmark)* Stop the parquet run overwriting the Hudi benchmark results (#747) by @yihua - *(core)* Decline base-read pushdown when a predicate reads a repaired column (#748) by @linliu-code ### ๐ Refactor - Rename variable names in splitting table APIs (#424) by @yunchipang - Replace custom `.to::()` with `into()` for `HudiConfig` conversions (#432) by @yunchipang - Move .crc filtering logic from table to storage layer (#458) by @yunchipang - Define avro model for parsing commit metadata (#477) by @xushiyan - Add targeted MDT lookup APIs for partition pruning (#502) by @xushiyan - Consolidate file metadata population for snapshot and incremental queries (#586) by @xushiyan - *(core)* Move parquet io into base file reader (#599) by @xushiyan ### ๐ Documentation - Update release guide (#287) by @xushiyan - Add AGENTS.md as canonical AI coding agent guide (#591) by @xushiyan - Deduplicate AI agent guidance with sub-directory AGENTS.md (#594) by @xushiyan - Correct read config scoping and MOR streaming, document reader v2 knobs (#709) by @yihua - Fix stale API claims in README, reader spec, and agent guides (#716) by @yihua ### โก Performance - *(core)* Skip decoding out-of-range log blocks and add unit test (#420) by @codope - Setup tpch benchmark infra (#538) by @xushiyan - Improve datafusion integration (#548) by @xushiyan - *(core)* Copy an HFile record rather than its whole block per record (#697) by @linliu-code - *(core)* Bound the merge chunk rather than moving the merge off the executor (#689) by @linliu-code ### ๐ ๏ธ Build - Use `uv` for Python package management (#391) by @codephage2020 - *(deps)* Bump pytest from 8.3.5 to 8.4.1 in /python (#355) by @dependabot[bot] - *(deps)* Bump astral-sh/setup-uv from 5 to 6 (#399) by @dependabot[bot] - *(deps)* Bump actions/checkout from 4 to 5 (#416) by @dependabot[bot] - *(deps)* Bump actions/download-artifact from 4 to 5 (#417) by @dependabot[bot] - *(deps)* Bump mypy from 1.15 to 1.17.1 in /python (#413) by @dependabot[bot] - *(deps)* Bump actions/labeler from 5 to 6 (#443) by @dependabot[bot] - *(deps)* Bump actions/setup-python from 5 to 6 (#444) by @dependabot[bot] - *(deps)* Bump actions/setup-node from 4 to 5 (#442) by @dependabot[bot] - *(deps)* Bump pytest from 8.4.1 to 8.4.2 in /python (#441) by @dependabot[bot] - *(deps)* Fix integration test setup issue with pyarrow and rust versions (#467) by @xushiyan - *(deps)* Bump astral-sh/setup-uv from 6 to 7 (#465) by @dependabot[bot] - *(deps)* Bump coverage from 7.8 to 7.10.7 in /python (#456) by @dependabot[bot] - *(deps)* Bump mypy from 1.17.1 to 1.18.2 in /python (#454) by @dependabot[bot] - *(deps)* Bump actions/setup-node from 5 to 6 (#469) by @dependabot[bot] - *(deps)* Bump ruff from 0.12 to 0.14.1 in /python (#472) by @dependabot[bot] - Use build wrapper to handle macos 26 SDK loading (#479) by @xushiyan - *(deps)* Bump apache/skywalking-eyes from 0.7.0 to 0.8.0 (#468) by @dependabot[bot] - *(deps)* Bump actions/checkout from 5 to 6 (#485) by @dependabot[bot] - *(deps)* Bump actions/download-artifact from 5 to 7 (#496) by @dependabot[bot] - *(deps)* Bump actions/upload-artifact from 4 to 6 (#495) by @dependabot[bot] - *(deps)* Bump actions/cache from 4 to 5 (#494) by @dependabot[bot] - *(deps-dev)* Bump mypy from 1.18.2 to 1.19.1 in /python (#497) by @dependabot[bot] - *(deps)* Update tokio requirement from ~1.45 to ~1.48 (#470) by @dependabot[bot] - *(deps)* Upgrade apache-avro to v0.21.0 (#503) by @xushiyan - *(deps)* Upgrade datafusion, arrow, and other deps (#504) by @xushiyan - *(deps)* Upgrade rust and make deps verisons more compatible (#506) by @xushiyan - *(deps-dev)* Bump ruff from 0.14.1 to 0.14.13 in /python (#526) by @dependabot[bot] - *(deps)* Update strum requirement from 0.27 to 0.28 (#533) by @dependabot[bot] - *(deps)* Bump actions/upload-artifact from 6 to 7 (#531) by @dependabot[bot] - *(deps)* Upgrade datafusion to 52, require python 3.10 (#535) by @xushiyan - *(deps)* Bump actions/download-artifact from 7 to 8 (#532) by @dependabot[bot] - *(deps)* Bump docker/setup-buildx-action from 3 to 4 (#540) by @dependabot[bot] - *(deps)* Update strum_macros requirement from 0.27 to 0.28 (#541) by @dependabot[bot] - *(deps)* Update zip requirement from 4 to 8 (#545) by @dependabot[bot] - *(deps-dev)* Bump datafusion from 52.0.0 to 52.3.0 in /python (#544) by @dependabot[bot] - *(deps)* Bump actions/checkout from 5 to 6 (#568) by @dependabot[bot] - *(deps)* Bump codecov/codecov-action from 5 to 6 (#567) by @dependabot[bot] - *(deps-dev)* Bump pytest from 8.4.2 to 9.0.3 in /python (#577) by @dependabot[bot] - *(deps-dev)* Bump mypy from 1.19.1 to 1.20.2 in /python (#584) by @dependabot[bot] - *(deps-dev)* Bump ruff from 0.14.13 to 0.15.12 in /python (#585) by @dependabot[bot] - Upgrade rust dev toolchain to 1.94 (#595) by @xushiyan - *(deps)* Bump actions/cache from 5 to 6 (#635) by @dependabot[bot] - *(deps)* Bump actions/checkout from 6 to 7 (#632) by @dependabot[bot] - *(deps-dev)* Bump pytest from 9.0.3 to 9.1.1 in /python (#634) by @dependabot[bot] - *(deps)* Bump codecov/codecov-action from 6 to 7 (#628) by @dependabot[bot] - *(deps-dev)* Bump ruff from 0.15.12 to 0.15.18 in /python (#633) by @dependabot[bot] - *(deps)* Bump astral-sh/setup-uv from 8.3.2 to 10.0.1 (#626) by @dependabot[bot] - *(deps-dev)* Bump coverage from 7.10.7 to 7.16.0 in /python (#625) by @dependabot[bot] - *(deps)* Bump docker/setup-buildx-action from 4.2.0 to 4.3.0 (#617) by @dependabot[bot] - *(deps-dev)* Bump mypy from 1.20.2 to 2.3.1 in /python (#615) by @dependabot[bot] - Raise MSRV to 1.94.1 to match what the dependency graph requires (#715) by @yihua - Adopt the MSRV-aware resolver and check the MSRV in CI (#721) by @yihua - *(python)* Bound the datafusion extra to the 54.x major (#720) by @yihua - Mark hudi-jvm-ffi as not published (#719) by @yihua ### ๐งช Testing - Add basic v8 test tables (#394) by @xushiyan - Add v9 test tables and test cases (#536) by @xushiyan - *(core)* Expand v9 sample-table coverage in core tests (#578) by @xushiyan - *(core)* Cover event-time ordering losing in the gold corpus (#680) by @linliu-code - *(core)* Cover every metadata partition's log blocks through reader v2 (#692) by @linliu-code - *(core)* Add hfile compression codec coverage. (#622) by @slfan1989 ### โ๏ธ Miscellaneous Tasks - Update src verify script (#387) by @xushiyan - Update changlog for 0.4.0 (#390) by @xushiyan - Update asf.yaml for PR requirements (#393) by @xushiyan - Bump script should use -dev suffix (#408) by @yunchipang - Enable discussions (#431) by @xushiyan - *(ci)* Update rust toolchain to 1.85 (#439) by @codope - Reduce CI run time (#478) by @xushiyan - Support generating local coverage report (#491) by @xushiyan - Add copilot review instructions (#509) by @xushiyan - Pin third-party actions to Apache-approved SHAs (#550) by @xushiyan - Add pre-commit hooks for code formatting and linting (#103) by @muyihao - Run rust-tests on windows (#92) by @crrow - Remove write perm and fix vulnerability (#572) by @xushiyan - Use action refs the ASF allowlist accepts (#674) by @linliu-code - Let consumers opt out of the merge map's on-disk tier (#681) by @linliu-code ## New Contributors * @linliu-code made their first contribution in #748 * @slfan1989 made their first contribution in #622 * @yunchipang made their first contribution in #433 * @sushiljacksparrow made their first contribution in #581 * @crrow made their first contribution in #92 * @brayanjuls made their first contribution in #412 * @codephage2020 made their first contribution in #391 <!-- generated by git-cliff --> --- *Note: #692 was added manually. Its squash commit subject lost the conventional-commit prefix (`test(core):`), so `git-cliff` skipped it.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
