mattfaltyn opened a new issue, #17301:
URL: https://github.com/apache/iceberg/issues/17301
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Other
### Please describe the bug 🐞
## Summary
A Parquet read using a `notNaN` filter crashes when the referenced float or
double column exists in the projected Iceberg schema but is absent from an
older Parquet file.
This is a normal schema-evolution scenario:
1. Write a Parquet file with schema `{ required long id }`.
2. Add an optional float column such as `added_float`.
3. Read the old file using the evolved schema and filter with
`notNaN("added_float")`.
The added column should read as null for old rows. Iceberg defines null as
not NaN, so the row group must remain readable. Instead, reader initialization
throws a `NullPointerException`.
I reproduced this on `main` at commit:
```text
1ec15051dc09ef6a0d61fa332eb8f2503f3170f8
```
The same faulty code is present in the `apache-iceberg-1.11.0` tag.
## Minimal reproduction
In `TestDictionaryRowGroupFilter#testColumnNotInFile`, add
`notNaN("not_in_file")` to the existing `exprs` array:
```java
Expression[] exprs =
new Expression[] {
lessThan("not_in_file", 1.0f),
lessThanOrEqual("not_in_file", 1.0f),
equal("not_in_file", 1.0f),
greaterThan("not_in_file", 1.0f),
greaterThanOrEqual("not_in_file", 1.0f),
notNull("not_in_file"),
isNull("not_in_file"),
notEqual("not_in_file", 1.0f),
notNaN("not_in_file")
};
```
Then run:
```shell
./gradlew --no-build-cache \
:iceberg-parquet:test \
--tests org.apache.iceberg.parquet.TestDictionaryRowGroupFilter \
--no-daemon
```
I also reproduced the failure twice through the production
`Parquet.read(...)` path by writing an old-schema file and reading it with the
evolved schema.
## Expected behavior
The reader should retain the row group. Because the column is absent from
the file, its values are null, and null values satisfy Iceberg's `notNaN`
expression semantics.
## Actual behavior
The read fails with:
```text
Cannot invoke "java.lang.Boolean.booleanValue()" because
the return value of "java.util.Map.get(Object)" is null
```
The exception originates from `ParquetDictionaryRowGroupFilter.java:178` and
is reached through `ReadConf.java:109`.
## Root cause
`ParquetDictionaryRowGroupFilter.EvalVisitor#notNaN` evaluates:
```java
if (mayContainNulls.get(id)) {
return ROWS_MIGHT_MATCH;
}
```
before checking whether the field ID exists in the file metadata:
```java
Boolean hasNonDictPage = isFallback.get(id);
if (hasNonDictPage == null || hasNonDictPage) {
return ROWS_MIGHT_MATCH;
}
```
For a column absent from the file, neither map has an entry for the field
ID. The first lookup returns null and is unboxed as a boolean, causing the
exception.
Moving the existing absent/fallback-column guard before the nullability
lookup should preserve behavior for present columns while conservatively
retaining row groups for absent columns.
## Related history
- #6431 added the nullability check for physically present columns
containing nulls, but did not cover columns absent from the file.
- #16692 addresses an initial-default problem in
`ParquetMetricsRowGroupFilter` and does not change this dictionary-filter path.
I searched existing open and closed issues and pull requests using
combinations of `notNaN`, `ParquetDictionaryRowGroupFilter`, `missing column`,
`column not in file`, and `schema evolution`, and did not find an active fix
for this crash.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
