JeonDaehong opened a new pull request, #1145:
URL: https://github.com/apache/paimon-rust/pull/1145
### Purpose
Linked issue: close #1143
`Predicate::negate` wrapped leaves in `Not`, and `Not` was evaluated as
`!inner`, so a NULL input made `NOT(p = 'a')` true. Reads returned those rows
and `delete_by_predicate` deleted them. Java flips the leaf instead
(`Equal.negate()` is `NotEqual`), which is false for NULL.
### Brief change log
- `Predicate::negate` now does what Java `Predicate.negate` does: flip the
leaf operator (`=`/`!=`, `<`/`>=`, `<=`/`>`, `IS NULL`/`IS NOT NULL`, `IN`/`NOT
IN`, `BETWEEN`/`NOT BETWEEN`) and push through AND/OR with De Morgan.
- DataFusion: the NOT BETWEEN coverage check in `scan.rs` now matches the
new shape of the negated filter (`x < lo OR x > hi` instead of `NOT(x >= lo AND
x <= hi)`).
- Leaves with no negated operator here (`STARTS_WITH`, `ENDS_WITH`,
`CONTAINS`, `LIKE`, array ops) stay as `Not(leaf)`. `eval_row` and the residual
row filter now return false for those when the input is NULL, same as Java
`NotStartsWith`/`NotLike`/... .
### Tests
- `negated_predicates_keep_null_rows` and
`negated_data_predicate_keeps_null_rows` cover the cases from the issue on the
partition and DV delete paths and on reads. Both fail on main.
- Unit tests for `negate` and for NULL evaluation in `eval_row` and the
residual filter.
- Changed existing tests:
- `test_file_index_nested_not_with_added_column_falls_back` now expects
`[1]` instead of `[1, 2]`. `added` is NULL in the old file, so for id 2 the
predicate is `NOT(true AND unknown)`. Java drops that row too.
- `test_not` / `test_display_not`, three DataFusion pushdown tests, the
global index exactness test and the stats fail-open test asserted the old `Not`
shape. The last two now build `Predicate::Not` directly so they keep testing
the `Not` branch.
fmt and clippy (1.98.0, `-D warnings`) pass. `cargo test --no-fail-fast -p
paimon -p paimon-datafusion -p paimon-c`: 5804 passed. The 39 failures are all
`TableNotExist` / table not found for the fixture tables CI provisions, plus 2
that need the lumina library. The ones I checked fail the same way on main.
Not covered here: a `Predicate::Not` around AND/OR built by hand (not
through `negate`) is still evaluated as `!inner`. Nothing in core builds that
today, but I can add it if you'd like.
### API and Format
No API change. `Predicate::negate` returns a different but equivalent tree
for non-NULL inputs.
### Documentation
None.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
