Dandandan commented on issue #20324: URL: https://github.com/apache/datafusion/issues/20324#issuecomment-3924971015
> I am on branch-52 and just benchmarked Q40 and see a speed-up of 35% (20 ms without VS 13 ms with) > > **SET datafusion.execution.parquet.pushdown_filters = false;** > > ``` > vinayjain@Vinays-MacBook-Pro datafusion % ./target/profiling/datafusion-cli -f ./bench_q40_wo.sql | grep Elapsed > Elapsed 0.000 seconds. > Elapsed 0.000 seconds. > Elapsed 0.038 seconds. > Elapsed 0.000 seconds. > Elapsed 0.026 seconds. > Elapsed 0.022 seconds. > Elapsed 0.020 seconds. > Elapsed 0.020 seconds. > Elapsed 0.019 seconds. > Elapsed 0.020 seconds. > Elapsed 0.019 seconds. > Elapsed 0.020 seconds. > Elapsed 0.019 seconds. > Elapsed 0.019 seconds. > Elapsed 0.020 seconds. > Elapsed 0.020 seconds. > Elapsed 0.020 seconds. > Elapsed 0.020 seconds. > Elapsed 0.019 seconds. > Elapsed 0.020 seconds. > Elapsed 0.020 seconds. > Elapsed 0.020 seconds. > Elapsed 0.019 seconds. > Elapsed 0.019 seconds. > ``` > > **SET datafusion.execution.parquet.pushdown_filters = true;** > > ``` > vinayjain@Vinays-MacBook-Pro datafusion % ./target/profiling/datafusion-cli -f ./bench_q40_w.sql | grep Elapsed > Elapsed 0.000 seconds. > Elapsed 0.000 seconds. > Elapsed 0.037 seconds. > Elapsed 0.000 seconds. > Elapsed 0.017 seconds. > Elapsed 0.015 seconds. > Elapsed 0.014 seconds. > Elapsed 0.014 seconds. > Elapsed 0.014 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > Elapsed 0.013 seconds. > ``` Could very well be different between machines. Fewer cores / less parellism can mean some dynamic filters are more effective (more opportunity to skip). Also IO / CPU is different on machines (on osx IO operations can be slower), so the CPU/IO tradeoff might be different. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
