adriangb commented on issue #20324: URL: https://github.com/apache/datafusion/issues/20324#issuecomment-3894747355
> do we need to update the filters for each batch? could we skip the bounds don't change? > > [datafusion/datafusion/physical-plan/src/aggregates/no_grouping.rs](https://github.com/apache/datafusion/blob/f48dc72220c64420172a90c7aed7dfc3916b311a/datafusion/physical-plan/src/aggregates/no_grouping.rs#L192) > > Line 192 in [f48dc72](/apache/datafusion/commit/f48dc72220c64420172a90c7aed7dfc3916b311a) > > filter_state.filter.update(predicate)?; > [@adriangb](https://github.com/adriangb) i think it could be this. we rebuild the pruning predicates for each record batch. > > [datafusion/datafusion/pruning/src/file_pruner.rs](https://github.com/apache/datafusion/blob/f48dc72220c64420172a90c7aed7dfc3916b311a/datafusion/pruning/src/file_pruner.rs#L98-L99) > > Lines 98 to 99 in [f48dc72](/apache/datafusion/commit/f48dc72220c64420172a90c7aed7dfc3916b311a) > > // If the generation hasn't changed, we can skip rebuilding the pruning > // predicate, which is an expensive operation involving expression analysis. This certainly seems like it could be a smoking gun, good find! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
