JeonDaehong opened a new issue, #1143:
URL: https://github.com/apache/paimon-rust/issues/1143
### Search before asking
- [x] I searched in the issues and found nothing similar.
### Please describe the bug 🐞
`Predicate::negate` wraps most predicates in `Predicate::Not`, and `Not` is
evaluated as plain `!inner`. So if the column is NULL, `NOT(p = 'a')` comes out
true. In Java, `equal(p, 'a').negate()` becomes `NotEqual(p, a)`, which is
false for NULL.
So reads return the NULL rows, and `TableUpdate::delete_by_predicate`
deletes them. Using `not_equal` directly works fine.
Repro on main (9568f39) with the fixture from
`table_predicate_delete_test.rs` (id 3 has `p = NULL`):
```rust
let pred = Predicate::negate(pb.equal("p", Datum::String("a".into()))?);
let msgs =
table.new_write_builder().new_update()?.delete_by_predicate(Some(pred)).await?;
// after commit: remaining ids [0, 1], expected [0, 1, 3]
```
| predicate | op | got | expected |
|---|---|---|---|
| `NOT(p = 'a')` | delete | [0, 1] | [0, 1, 3] |
| `NOT(q = 2 AND p = 'b')` | delete | [2] | [2, 3] |
| `NOT(p = 'a')` | read | [2, 3] | [2] |
| `NOT(v = 10)`, v is NULL for id 3 | read | [1, 2, 3] | [1, 2] |
| `NOT(v = 10)`, v is NULL for id 3 | delete | [0] | [0, 3] |
The deletes fail the same way on the partition-only path and on the DV path.
I checked Java with paimon-bundle 2.1.0: `NOT(p='a')`, `NOT(v=10 AND p='b')`
and `NOT(v=10)` all return false for the NULL row.
DataFusion SQL isn't affected because `NOT` is pushed down with
`requires_residual`, and the Python binding rejects `not`. Go `Predicate.Not()`
and the C API both go through `negate`, though.
### Solution
Do what Java does in `negate`: flip the leaf op (`=`/`<>`, `<`/`>=`, `IS
NULL`/`IS NOT NULL`, `IN`/`NOT IN`, ...) and push through AND/OR with De
Morgan. For ops Java can't negate (`starts_with` etc.), `Not` would need
NULL-aware evaluation, or deletes should reject it.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
