JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4345189684
##########
crates/paimon/src/arrow/residual.rs:
##########
@@ -166,9 +166,16 @@ pub(crate) fn evaluate_predicate_mask(
else {
return Ok(None);
};
- Ok(Some(boolean_mask_from_predicate(mask.len(), |row_index| {
- !mask.value(row_index)
- })))
+ let negated =
+ boolean_mask_from_predicate(mask.len(), |row_index|
!mask.value(row_index));
+ if let Some(guard) = inner.not_null_guard() {
Review Comment:
[P2] Keep raw double negation from selecting NULL rows
The new guard makes an inner Not(leaf) false for NULL, but an outer Not has
no guard when its child is another Not, so it turns that false back into true.
Public Predicate::Not is a logical-negation variant and existing code accepts
raw nested Not trees. A public read probe writes value=[NULL,1,2], persists a
supported INT-to-BIGINT schema evolution, and reads with Not(Not(value = 1)):
this head returns ids [0,1], while exact base cea0caaf returns only [1]. This
is a newly introduced double-NOT regression, separate from the pre-existing raw
Not(AND/OR) limitation. Normalize/eliminate double negation before evaluation,
or preserve NULL semantics through nested Not consistently in both residual and
BinaryRow evaluation.
##########
crates/paimon/src/table/global_index_scanner/evaluator.rs:
##########
@@ -386,6 +387,27 @@ impl GlobalIndexScanner {
&self,
entries: &[&GlobalIndexEntry],
predicates: &[(PredicateOperator, &[Datum], &DataType)],
+ ) -> Result<Option<(Vec<RowRange>, Vec<RowRange>, bool)>> {
+ // A bitmap answers floating ranges and negations with every non-null
+ // row (see `bitmap_meta_may_match`), so those results are candidates
+ // only. Sorted indexes answer them exactly.
+ let bitmap_superset = entries
Review Comment:
[P2] Preserve exactness when every Bitmap entry was pruned
This flag considers every selected entry before file-level pruning. I
reproduced a public write/index/search scenario that builds Bitmap(f) over two
NULL rows, appends finite f=[2,3], then builds BTree(f) and an IVF-flat vector
index. For f > 1.5, the Bitmap shard is pruned by its all-NULL metadata and the
remaining BTree answer is exact. Nevertheless bitmap_superset downgrades it, so
Top-1 with global-index.filter.refine-from-data=false returns [] instead of row
2. The same test passes unchanged on base cea0caaf; refinement=true also
returns row 2 on this head. Derive candidate inexactness from Bitmap entries
that actually contribute a superset, retaining exactness when they are all
pruned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
