jackylee-ch opened a new pull request, #3918:
URL: https://github.com/apache/iceberg-python/pull/3918
# Rationale for this change
Pushing `NotEqualTo` down to Arrow drops every row where the column is null:
an Arrow comparison yields null for a null input, and `filter` treats that as
no match.
```python
table = pa.table({"foo": [None, "hello", "world"]})
table.filter(pc.field("foo") != "hello") # ["world"] -- the null is gone
```
Every other evaluator treats a null as satisfying `NotEqualTo`.
`_ExpressionEvaluationVisitor.visit_not_equal` returns `None != value`;
`_StrictMetricsEvaluationVisitor.visit_not_equal` reports `ROWS_MUST_MATCH` for
a column proven to hold only nulls; and the Arrow `NOT IN` path already keeps
them, because `~isin(...)` is false for a null. The reference implementation
agrees: `Evaluator.notEq` is `!eq`, and `eq` compares with a nulls-first
comparator, so a null never equals the literal.
Keep the null explicitly. `visit_not_in` needs no change.
## Are these changes tested?
Yes, `test_expr_not_equal_to_pyarrow_keeps_nulls` asserts that the
pushed-down filter and the in-memory evaluator return the same rows. It and the
existing `test_expr_not_equal_to_pyarrow` both fail without the change.
## Are there any user-facing changes?
A `row_filter` using `!=`, or a `NOT IN` that reduces to a single literal,
now returns rows where the column is null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
