alamb opened a new issue, #26169: URL: https://github.com/apache/datafusion/issues/26169
### Is your feature request related to a problem or challenge? @ritchie46 posted a blog https://pola.rs/posts/release-polars-2/ recently that provides benchmark numbers for Polars 2.0 that make Datafusion look quite bad: <img width="1323" height="833" alt="Image" src="https://github.com/user-attachments/assets/a05b95f0-b651-4f09-848b-b1b6902808f8"; /> I suspect it is telling that ### Describe the solution you'd like I would like the performance of DataFusion to be better (obviously) ### Describe alternatives you've considered My strong suspicion is that the difference is largely caused by the fact that TPC-H and TPC-DS require good join ordering for high performance, something that DataFusion has been historically weak at I also think this problem is also likely due to the same thing: > Polars and both DuckDB versions completed all queries. DataFusion timed out on TPC-DS q72 (and once on q67) and ran out of memory on TPC-H q18 on c7a.4xlarge; those queries are excluded from the results above for all engines. Another corroborating piece of evidence is that the post does not mention ClickBench https://benchmark.clickhouse.com/ where we have focused a huge amount of effort on benchmarking and the difference is much less pronounced. See for example the [results](https://benchmark.clickhouse.com/#system=+ti%20rud|olrP&type=-&machine=-6t|ca2|ae-|6ax|g4e|6ale|3al&cluster_size=-&opensource=-&hardware=+c&tuned=+n&storage=-&metric=combined&queries=-) here: <img width="1141" height="224" alt="Image" src="https://github.com/user-attachments/assets/f27f9e10-c2e4-40ad-ae2b-c0122b3a59b1"; /> ### Additional context _No response_ ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
