xudong963 opened a new pull request, #27321: URL: https://github.com/apache/datafusion/pull/27321
## Which issue does this PR close? Follow-up to #27119, specifically the [scalar FILTER review discussion](https://github.com/apache/datafusion/pull/27119#discussion_r4520242815). ## Rationale for this change An aggregate FILTER that evaluates to a scalar boolean currently expands that value into a batch-sized BooleanArray before deciding which rows to evaluate. Runtime scalar predicates, such as scalar-returning UDFs, can avoid this allocation and bitmap scanning. ## What changes are included in this PR? - Retain scalar boolean filter results and treat scalar NULL like false. Evaluate the predicate for every batch, since a runtime scalar can change between batches. - For ungrouped and compacting grouped aggregation, borrow the input for scalar true and use empty typed arguments for false/NULL without evaluating rejected arguments. Preserve groups even when all their rows are rejected. - For skip-partial conversion, scalar true uses the unfiltered argument path. False/NULL skips argument evaluation but still supplies a rejection bitmap: the existing `GroupsAccumulator::convert_to_state` interface requires one to preserve rejected rows and the correct empty states, including zero counts. - Add an end-to-end SQL/UDF regression and extend the existing `aggregate_filter` benchmark with scalar predicates and controls. ## What is the testing strategy for this PR? - `cargo fmt --all` and the complete `./dev/rust_lint.sh` suite passed, including Clippy with all targets/features and `-D warnings`, dependency checks, Rust documentation, and the documentation website build. - The extended workspace test command from `AGENTS.md` passed: 13,630 Rust tests, zero failures, eight ignored; all 548 SQLLogicTest files passed. - The new SQL test covers changing scalar true/false/NULL results across differently sized batches, nullable arguments, literal arguments, rejected division-by-zero expressions, retained empty groups, multiple partitions, and skip-partial conversion. It checks `skipped_aggregation_rows` to verify that conversion was exercised. Compared with main `34eb6dc5b`, using the same benchmark source and `release-nonlto` builds (65,536 rows, batches of 8,192): | Ungrouped FILTER | main median | PR median | Change | | --- | ---: | ---: | ---: | | Scalar true | 13.54 us | 12.28 us | -9.3% | | Scalar false | 5.75 us | 3.93 us | -31.7% | | Scalar NULL | 7.21 us | 4.06 us | -43.7% | | Scalar varying between batches | 7.45 us | 5.74 us | -23.0% | These are medians of three runs per build in alternating order, with 30 Criterion samples, 0.5 s warm-up, and 1 s measurement per case. The suite also covers grouped and skip-partial execution, shuffled 50%-selective array filters, and unfiltered controls. An initial short run suggested a 6.2% slowdown in the skip-partial array-filter control. Three longer runs per build (60 samples, 1 s warm-up, 3 s measurement, alternating order) did not reproduce it: grouped/skip-partial array-control medians changed by -0.3%/-1.2%, and unfiltered controls by -0.5%/-0.2%. Grouped and skip-partial scalar-true timings were essentially unchanged. These local microbenchmarks ran on a shared macOS machine; they are not end-to-end query speedup claims. ## Are there any user-facing changes? Lower overhead for aggregate FILTER predicates that evaluate to a scalar. SQL results and public APIs are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
