sunchao commented on code in PR #24526:
URL: https://github.com/apache/datafusion/pull/24526#discussion_r3858438804
##########
datafusion/pruning/src/pruning_predicate.rs:
##########
@@ -1582,6 +1629,13 @@ fn build_predicate_expression(
}
}
if let Some(in_list) = expr.downcast_ref::<phys_expr::InListExpr>() {
+ if in_list.list().len() > MAX_IN_LIST_SIZE
Review Comment:
The lower bound preserves the existing pruning-expression representation for
lists of at most 20 values and keeps this PR focused on the cost of large lists
when callers explicitly raise the cap. The value 20 comes from the existing
default; it is not an established performance crossover.
Using the compact form for smaller eligible string lists is worth
considering separately. The current benchmark keeps the 20-value case on the
legacy path, so it does not establish that compact construction/normalization
is cheaper across the small-list cases. A focused comparison could justify
broadening the optimization later. Agreed that the scope/default-behavior
rationale should be stated next to this condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
