hubgeter opened a new pull request, #68128:
URL: https://github.com/apache/doris/pull/68128
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Doris orders NaN above every other floating-point value, so `d > 0` and `d
>= 0` are true for a NaN row. Iceberg's metrics and manifest evaluators assume
the opposite: NaN is excluded from the lower/upper bounds and an all-NaN column
is pruned for any range predicate. Pushing a bare `greaterThan` therefore
prunes files that do hold matching rows, BE never sees the split, and the rows
are silently lost. This is not limited to all-NaN files: a file holding `{1.0,
NaN}` has upper bound `1.0`, so `d > 5` drops it too.
The fix ORs an `isNaN` arm into the GT/GE pushed on a float/double column. A
file whose `nan_value_count` is 0 is still pruned by the range arm, so pruning
is kept where it is correct. LT/LE are unchanged, since Doris also evaluates
`NaN < v` as false.
Two related cases in the same converter:
- `NOT (d < v)` is now negated in Doris semantics before conversion, because
Iceberg's RewriteNot turns `not(d < v)` back into `d >= v` without the isNaN
arm.
- A NaN literal is mapped to `isNaN` / `notNaN` instead of reaching
`Expressions.*`, which rejects it with `Cannot create expression literal from
NaN`.
Conflict detection and `rewrite_data_files` share the leaf conversion and
get the same treatment.
### Release note
Fix Iceberg queries silently losing NaN rows when a FLOAT/DOUBLE range
filter is pushed down.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Unit test: `IcebergPredicateConverterNaNTest`, 16 cases. They assert what
`InclusiveMetricsEvaluator` does to a file's metrics rather than the
rendered expression string,
over three shapes: an all-NaN file, a `{1.0, NaN}` file whose bounds omit
the NaN, and a NaN-free
file that must still be pruned.
Regression test:
`external_table_p0/iceberg/test_iceberg_nan_filter.groovy`, with the tables
built
by `create_preinstalled_scripts/iceberg/run32.sql` (one file per INSERT,
so each metrics shape is
separable). It asserts the returned rows for GT/GE/NOT/LT/LE, NaN literals
and IN, plus
`inputSplitNum` — which pins that a NaN-free file below the bound is still
pruned.
- Behavior changed:
- [x] Yes. An Iceberg FLOAT/DOUBLE range filter now returns the NaN rows
it should have returned,
and fewer files are pruned for such a filter when the data files carry
no `nan_value_counts`.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
