LiaCastaneda opened a new issue, #17486: URL: https://github.com/apache/datafusion/issues/17486
### Describe the bug Hello! We are very interested in using the Dynamic Filtering on hash joins optim feature, so I brought the dynamic filtering work https://github.com/DataDog/datafusion/pull/43 to test it on a production environment. The API works as expected (we do receive dynamic filters). However the change put all our resources at maximum CPU and profiling showed that almost 2/3 of the overall CPU usage was spent in `min_batch` and `max_batch` when comparing **lists**. Specifically the profiler showed the issue was in min_batch -> min_max_batch_generic -> ScalarValue::partial_cmp for lists (via [partial_cmp_list](https://github.com/apache/datafusion/blob/c83466c20ea5ec2c2c98e61917a3324cd71211ee/datafusion/common/src/scalar/mod.rs?utm_source=chatgpt.com#L528)). It looks like comparing list values during bounds computation is expensive, is this expected? Also I noticed `compute_bounds` is called for every query that has Hash Join by default. ### To Reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
