GayathriSrividya opened a new pull request, #1022:
URL: https://github.com/apache/hudi-rs/pull/1022
Closes #859
## Description
Parquet omits NaN values from float min/max statistics, which can make Spark
incorrectly skip files for matching predicates. Mark Float32/Float64 column
ranges as unknown when they contain NaN, so those incomplete stats are not used
to prune files. Writer paths inspect original batches rather than decoding
newly written Parquet again.
<!--- Describe your changes in detail -->
<!--- If it fixes an open issue, please link to the issue here. -->
<!--- Please link any related issues and PRs as well. -->
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
Regression test: `./build-wrapper.sh cargo test -p hudi-core
metadata::table::column_stats::tests::test_column_ranges_from_parquet_nan_values_are_not_prunable
--lib` — passed (1 passed, 1708 filtered out).
Spark read check not run: `SPARK_HOME` is unset and `spark-submit` is
unavailable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]