SEPURI-SAI-KRISHNA opened a new pull request, #17656:
URL: https://github.com/apache/iceberg/pull/17656
Closes #17655
`ParquetDictionaryRowGroupFilter.notStartsWith` decides whether a row group
can be
skipped by checking only the column's dictionary. A Parquet dictionary
contains
non-null values only, so when every dictionary entry starts with the prefix
the
filter returns `ROWS_CANNOT_MATCH` — even if the column also contains nulls.
In Iceberg, a null value matches `notStartsWith`: `Evaluator` implements it
as
`!startsWith(...)` and `startsWith` is false for null. So a row group holding
nulls does contain matching rows and must not be skipped, which silently
drops
those rows from the scan result.
Every other filter already handles this:
- `InclusiveEvalVisitor#notStartsWith` returns `ROWS_MIGHT_MATCH` when
`mayContainNull(id)`
- `ParquetMetricsRowGroupFilter#notStartsWith` returns `ROWS_MIGHT_MATCH`
when `mayContainNull(colStats)`
- `ParquetBloomRowGroupFilter` and ORC's `ExpressionToSearchArgument` do not
push the predicate down at all
Within `ParquetDictionaryRowGroupFilter` itself, `notEq`, `notIn` and
`notNaN`
all consult `mayContainNulls` for the same reason. `notStartsWith` was the
only
negated predicate missing the check.
This adds the missing `mayContainNulls` check, placed before the dictionary
page
is read so a null-containing column short-circuits without the extra I/O,
matching
`notNaN`.
The existing test asserted the previous behavior:
```java
shouldRead =
new ParquetDictionaryRowGroupFilter(SCHEMA, notStartsWith("some_nulls",
"some"))
.shouldRead(parquetSchema, rowGroupMetadata, dictionaryStore);
assertThat(shouldRead).as("Should skip: no match in dictionary").isFalse();
```
`some_nulls` is written as `(i % 10 == 0) ? null : "some"`, so the row group
holds
both nulls and `"some"`, and the null rows match
`notStartsWith("some_nulls", "some")`.
The assertion is corrected to expect a read. `data`'s
`TestMetricsRowGroupFilter`
already expects `notStartsWith("some_nulls", "som")` to read, so this brings
the
dictionary filter in line with the metrics filter.
A case for an optional column with no nulls (`no_nulls`) is added to confirm
row
groups are still skipped when the column is nullable but actually holds no
nulls.
This is not a regression — `notStartsWith` is unchanged in `1.11.0`, so
released
versions are affected the same way.
Both assertions fail without the fix and pass with it, for `PARQUET_1_0` and
`PARQUET_2_0`. `:iceberg-parquet:test` and `:iceberg-data:test` pass in full.
---
Generated-by: Claude Code (Claude Opus 5) — fully reviewed by the author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
