adriangb commented on PR #19761: URL: https://github.com/apache/datafusion/pull/19761#issuecomment-3842318682
> Looking a bit more in the history of changes in join filter pushdown and checking some benchmark results, I am a bit concerned that we might have introduced a few regressions here and there without really noticing, that changes like these might "hide" > > For example, #17452 introduced waiting for all partitions to finish before starting right side evaluation, which it seems was not benchmarked on tpch/tpcds I guess the point is that this change may essentially be undoing #17452 and that the improvements really come from that? @LiaCastaneda and I have been talking about reworking #17452 so that the dynamic filter is updated as partitions complete, making it less necessary to wait. But yeah I think maybe we can make a branch that removes that synchronization point and see what the benchmark numbers look like? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
