zhuqi-lucas commented on code in PR #24074:
URL: https://github.com/apache/datafusion/pull/24074#discussion_r3719923644
##########
datafusion/pruning/src/pruning_predicate.rs:
##########
@@ -388,7 +388,29 @@ pub fn build_pruning_predicate(
file_schema: &SchemaRef,
predicate_creation_errors: &Count,
) -> Option<Arc<PruningPredicate>> {
- match PruningPredicate::try_new(predicate, Arc::clone(file_schema)) {
+ build_pruning_predicate_with_max_in_list_size(
+ predicate,
+ file_schema,
+ predicate_creation_errors,
+ MAX_LIST_VALUE_SIZE_REWRITE,
Review Comment:
Great suggestion — implemented as `PruningPredicateBuilder` with
`.with_file_schema(...)`, `.with_error_counter(...)`,
`.with_max_in_list_size(...)`, and `.build(predicate)` returning
`Option<Arc<PruningPredicate>>` for the parquet scan path, plus
`.try_build(predicate)` returning `Result<PruningPredicate>` for callers that
want to surface errors themselves. The standalone
`build_pruning_predicate_with_max_in_list_size` and
`PruningPredicate::try_new_with_max_in_list_size` are gone.
##########
datafusion/pruning/src/pruning_predicate.rs:
##########
@@ -461,7 +483,19 @@ impl PruningPredicate {
/// returns a new expression.
/// It is recommended that you pass the expressions through
[`PhysicalExprSimplifier`]
/// before calling this method to make sure the expressions can be used
for pruning.
- pub fn try_new(mut expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) ->
Result<Self> {
+ pub fn try_new(expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) ->
Result<Self> {
+ Self::try_new_with_max_in_list_size(expr, schema,
MAX_LIST_VALUE_SIZE_REWRITE)
Review Comment:
Removed the `try_new_with_max_in_list_size` variant.
`PruningPredicate::try_new` keeps its historical signature; the new
`PruningPredicateBuilder` is the entry point for callers that want to override
`max_in_list_size` (or supply an error counter).
##########
datafusion/common/src/config.rs:
##########
@@ -1189,6 +1189,22 @@ config_namespace! {
/// parquet reader setting. 0 means no caching.
pub max_predicate_cache_size: Option<usize>, default = None
+ /// Maximum number of values in an `IN (...)` list for which the
+ /// pruning predicate will rewrite the list into a chain of per-value
+ /// statistics checks. Lists longer than this fall back to the
+ /// unhandled-predicate hook (defaulting to "keep the container"),
+ /// which effectively skips container-level pruning for large IN
+ /// lists.
+ ///
+ /// Higher values keep row-group / file-range statistics pruning
+ /// effective for larger IN lists (for example, REST endpoints that
+ /// filter by a batch of ~25-100 identifiers), at the cost of a
+ /// larger rewritten predicate expression evaluated for every
+ /// container. Set to 0 to disable the rewrite path entirely.
+ ///
+ /// The default of 20 preserves the previous hardcoded behaviour.
Review Comment:
Accepted the suggestion verbatim (already committed via GitHub's
Add-suggestion). Also fixed a small accidental "of / of" duplication that snuck
in the merge.
##########
datafusion/common/src/config.rs:
##########
@@ -1189,6 +1189,22 @@ config_namespace! {
/// parquet reader setting. 0 means no caching.
pub max_predicate_cache_size: Option<usize>, default = None
+ /// Maximum number of values in an `IN (...)` list for which the
+ /// pruning predicate will rewrite the list into a chain of per-value
+ /// statistics checks. Lists longer than this fall back to the
+ /// unhandled-predicate hook (defaulting to "keep the container"),
+ /// which effectively skips container-level pruning for large IN
+ /// lists.
+ ///
+ /// Higher values keep row-group / file-range statistics pruning
+ /// effective for larger IN lists (for example, REST endpoints that
+ /// filter by a batch of ~25-100 identifiers), at the cost of a
+ /// larger rewritten predicate expression evaluated for every
+ /// container. Set to 0 to disable the rewrite path entirely.
+ ///
+ /// The default of 20 preserves the previous hardcoded behaviour.
+ pub pruning_max_in_list_size: usize, default = 20
Review Comment:
Renamed to `max_in_list_size` (matches the `max_predicate_cache_size`
neighbour). Section is already `execution.parquet.*` so the pruning context is
inferable from the key path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
