yihua commented on issue #713: URL: https://github.com/apache/hudi-rs/issues/713#issuecomment-5534962102
## Pre-cut validation summary Validated `main` @ 922a62cd (with #714, #715, #716, #717 merged) as the base for the 0.5.0 release branch. ### Test suites - Rust workspace, `--all-targets --all-features`: ~1,680 tests pass (includes spill-rocksdb, jvm-ffi, and the cpp bridge) - `hudi-core --no-default-features`: clippy with `-D warnings` and unit tests pass - Python: 73 tests pass ### Interop against Spark-written tables Tables written by Spark 3.5.5 with Hudi 1.1.1 (table version 9, metadata table on): COW and MOR with insert, update, and delete commits (the MOR delete landing as a log delete block), plus a MOR table with inline compaction triggered. - Row-for-row equality with Spark's own reads (all columns, including record keys) across snapshot, read-optimized, compacted, and time-travel queries - MOR incremental across the update and delete deltacommits returns exactly the changed record - Every README example runs as written in both Rust and Python: snapshot with filters, read-optimized mode, time travel, incremental (bounded, unbounded, filtered), streaming with projection and batch size, file group reader from paths, DataFusion SQL - Query engine integration path: `get_file_slices`, then `create_file_group_reader_with_options`, then per-slice reads, summing exactly to the snapshot row count; `compute_table_stats` returns metadata-table-based estimates - Storage: local filesystem and S3 (MinIO) for both the v6/v8 fixture tables and the Spark-written tv9 tables; all four demo apps pass (Rust table API, DataFusion SQL, Python, C++ over the Arrow C ABI) ### Release pipeline - A release wheel built exactly as the publish workflow builds it (no CLI feature flags) imports cleanly in a fresh venv and exposes `HudiDataFusionDataSource`, confirming #714; before that fix the same build failed `import hudi` - MSRV: the workspace checks clean on rustc 1.94.1 and the full suite ran on it; 1.91.1 fails on the aws crates, confirming #715 - Wheel size heads-up: the macOS arm64 release wheel is roughly 39 MB with datafusion compiled in (about 23 MB without; 0.4.0 was about 8 MB, with the rest of the growth coming from RocksDB and Lance). Well under PyPI limits, and in line with comparable projects (deltalake, datafusion-python) ### CI All four PRs were fully green before merging. On `main`, the runs for the three intermediate merge commits were cancelled by the workflow's own concurrency rule (each push cancels the previous run); on the surviving run for 922a62cd, every test job completed green (rust-tests, no-spill leg, python-tests on all three platforms, and the four demo-app integration tests), with only the coverage upload still finishing at the time of writing. ### Open items before the cut - Decide whether `hudi-jvm-ffi` publishes to crates.io for 0.5.0: it has no `publish = false` and is not in the release publish matrix (hudi-core, hudi-datafusion, hudi) - No MSRV check in CI yet; a sound one needs either a committed `Cargo.lock` or the MSRV-aware resolver first (see the note in #715) - The `hudi[datafusion]` extra pins `datafusion==54.0.0` while the Rust side uses DataFusion 54.1; tested fine together, flagging as a conscious call - The release wheels will compile DataFusion for the first time on targets the CI test matrix does not cover (Windows, cross x86_64 macOS); a dry-run build of those two targets before tagging would be prudent -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
