adriangb commented on PR #22450: URL: https://github.com/apache/datafusion/pull/22450#issuecomment-4518898867
I like the idea! I had to do something similar in https://github.com/apache/datafusion/pull/22144 (re-evaluate filter decisions every row group). We should look at the APIs and check how they match up. I am also interested in how this intersects with @Dandandan 's work on row group = morsel (morsels would be a good natural boundary to re-evaluate pruning at). > A new force_per_row_group flag (set by is_dynamic_physical_expr) disables coalescing for dynamic predicates only, so static WHERE queries pay nothing. Do we pay this price as well for hash join dynamic filters (where the filter won't be changing)? Maybe we should check `!filter.is_complete()` or something... I am traveling the next couple days but will try to review more in depth soon 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
