ump45nose opened a new pull request, #853:
URL: https://github.com/apache/hudi-rs/pull/853
## Description
When `ColumnStatistics::merge` aggregates row-group Parquet footer stats up
to file level, a missing min or max previously kept the other side's bound.
That is wrong when the missing bound means "unknown" (no column-chunk
statistics, or stats without min/max) rather than "all-null": the file-level
range then ignores row groups that can hold any value, and `FilePruner` can
skip a file that still has matching rows.
This change treats a missing bound as unknown and clears the merged min/max
whenever either side lacks it. Distinguishing all-null row groups would need a
null count on `ColumnStatistics`; until that exists, this is the conservative
fix described in #852.
Closes #852.
## How are the changes test-covered
- [x] Automated tests (unit and/or integration tests)
- Updated `test_column_statistics_merge` for the unknown-bound case.
- Added
`row_group_without_stats_keeps_file_when_filter_outside_known_bounds` (the #852
repro) under `file_pruner` tests.
- [ ] N/A
- [ ] Manual tests
- [ ] Details are described below
## AI assistance
Assisted by an AI coding agent (Cursor / Grok Bot) for locating the merge
path, implementing the conservative fix, and drafting tests/PR text. I reviewed
the diff and ran the targeted `hudi-core` unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
