adriangb commented on PR #20664: URL: https://github.com/apache/datafusion/pull/20664#issuecomment-4010497321
> @adriangb FYI > > Can we reconsider creating a large crazy large expression for the dynamic filters? > > now the size of the dynamic expression is something like > > * `number_of_join_keys` * `number_of_partitions` which creates _extremely large expressions_ on large core machines. > > > Perhaps create a `EvaluateByIdExpr` `PhysicalExpr` or something that has `Vec<PhysicalExpr>` that evaluates them by id, or disabling dynamic filters when having partitioned joins for the moment. I'm open to suggestions. We should find a solution that keeps the performance wins for small number of join keys / CPUs without degrading for large combinations of those. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
