waterWang opened a new pull request, #17682:
URL: https://github.com/apache/iceberg/pull/17682
Fixes #17655
## Problem
`ParquetDictionaryRowGroupFilter.notStartsWith` can skip row groups that
contain null values, silently dropping rows from scan results.
The filter decides whether a row group can be skipped by inspecting only the
column's dictionary. A Parquet dictionary contains non-null values only, so
when every dictionary entry starts with the prefix the filter returns
`ROWS_CANNOT_MATCH` — even when the column also contains nulls.
In Iceberg a null value matches `notStartsWith`: `Evaluator` implements it
as `!startsWith(...)`, and `startsWith` evaluates to false for null. A row
group holding nulls therefore does contain matching rows and must not be
skipped.
## Fix
Add a `mayContainNulls` check to `notStartsWith`, consistent with how
`notEq`, `notIn`, and `notNaN` already handle null-containing row groups.
## Changes
1. **`ParquetDictionaryRowGroupFilter.java`**: Added `if
(mayContainNulls.get(id)) return ROWS_MIGHT_MATCH;` in `notStartsWith`,
matching the pattern used by `notNaN`.
2. **`TestDictionaryRowGroupFilter.java`**: Updated the assertion for
`notStartsWith("some_nulls", "some")` from `isFalse()` to `isTrue()` — this row
group contains nulls, so it must be read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
