JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4365633028
##########
crates/paimon/src/arrow/residual.rs:
##########
@@ -166,9 +166,16 @@ pub(crate) fn evaluate_predicate_mask(
else {
return Ok(None);
};
- Ok(Some(boolean_mask_from_predicate(mask.len(), |row_index| {
- !mask.value(row_index)
- })))
+ let negated =
+ boolean_mask_from_predicate(mask.len(), |row_index|
!mask.value(row_index));
+ if let Some(guard) = inner.not_null_guard() {
Review Comment:
[P2] Preserve NULL semantics when negations are composed under AND
The inner NOT now maps NULL to false, but an outer NOT around a compound
expression receives no guard and negates that collapsed false to true. A public
raw predicate NOT(NOT(value = 1) AND id >= 0) on value=[NULL,1,2] now returns
ids [0,1] after supported INT-to-BIGINT schema evolution; the unchanged probe
on base fae8acff returns only [1]. Although general raw NOT(AND/OR) already has
a documented limitation, this particular input was correctly excluded before
this change and newly admits NULL, violating with_filter's exact row-filter
contract. Normalize compound NOT through De Morgan before losing UNKNOWN, or
preserve NULL state through the outer evaluation, consistently in residual and
BinaryRow paths; cover a nested-NOT-under-compound regression.
##########
crates/paimon/src/table/global_index_scanner/evaluator.rs:
##########
@@ -386,6 +387,28 @@ impl GlobalIndexScanner {
&self,
entries: &[&GlobalIndexEntry],
predicates: &[(PredicateOperator, &[Datum], &DataType)],
+ ) -> Result<Option<(Vec<RowRange>, Vec<RowRange>, bool)>> {
+ // A bitmap answers floating ranges and negations with every non-null
+ // row (see `bitmap_meta_may_match`), so those results are candidates
+ // only. Sorted indexes answer them exactly.
+ // An all-NULL bitmap shard is pruned before it answers anything.
+ let bitmap_superset = entries.iter().any(|entry| {
Review Comment:
[P2] Derive Bitmap candidate status after all conjunct pruning
Excluding only all-NULL entries still downgrades exact results when another
conjunct prunes a finite Bitmap. A public probe builds Bitmap(f) over f=[0,1],
appends f=[2,3], then builds BTree(f) and IVF-flat. For f > 1.5 AND f = 2, the
equality metadata check completely prunes the old Bitmap, and the surviving
BTree answers both predicates exactly. This wrapper nevertheless sees the
non-NULL Bitmap and marks the result inexact: Top-1 with refine-from-data=false
returns [] instead of row 2. The identical test passes on base fae8acff;
refinement=true and no-refinement equality-only controls also return row 2 on
this head. Compute inexactness from retained Bitmap query plans/candidates
after every conjunct is considered, rather than from all selected entries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
