JingsongLi commented on code in PR #1145:
URL: https://github.com/apache/paimon-rust/pull/1145#discussion_r4313636772
##########
crates/paimon/src/table/global_index_scanner.rs:
##########
@@ -385,11 +385,19 @@ impl GlobalIndexScanner {
predicate: &Predicate,
evaluated_fields: &std::collections::HashSet<i32>,
) -> bool {
- use crate::spec::PredicateOperator;
+ use crate::spec::{DataType, PredicateOperator};
match predicate {
Predicate::NestedLeaf { .. } => false,
- Predicate::Leaf { column, op, .. } => {
+ Predicate::Leaf {
+ column,
+ op,
+ data_type,
+ ..
+ } => {
predicates::is_sorted_global_index_supported_op(*op)
+ // Bitmap answers these with every non-null row.
+ && !(matches!(data_type, DataType::Float(_) |
DataType::Double(_))
+ &&
super::bitmap_global_index_format::is_bitmap_floating_residual_sensitive_op(*op))
Review Comment:
[P2] Keep exact floating BTree answers exact
This bitmap-specific downgrade currently applies to BTree results as well.
BTree already implements floating `!=` and ranges using the same bit-preserving
total ordering as the residual filter, so those answers are exact. With the
supported `global-index.filter.refine-from-data=false` option,
`scalar_matched_row_ids` excludes all candidates that this method marks
inexact. I reproduced the regression using actual Parquet data with finite
f=[1.0, 2.0], a scalar BTree on f, an ivf-flat vector index, and filtered Top-1
with `f != 1.0`: baseline 46e68710 returns id 1, while this head returns no
rows. No NaN or predicate negation is involved. Please derive exactness from
the contributing index entries/readers and downgrade the floating bitmap
fallback specifically, retaining exact BTree results; add a regression with
refinement disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
