parthchandra commented on issue #622: URL: https://github.com/apache/datafusion-comet/issues/622#issuecomment-2206748349
Spark produces the worst possible query plan for q72 which amplifies the difference in performance. The C2R overhead for comet is amplified because the conversion happens on a dataset that is larger than the source data. To get a reasonable query plan for q72 we need to have `spark.sql.cbo.enabled` and `spark.sql.cbo.joinReorder.enabled` set to true. This also requires stats. Also perhaps we can try with a larger broadcast threshold (`spark.sql.autoBroadcastJoinThreshold`)? Irrespective of the plan though, given the same number of input rows are the Comet operators also slower than the corresponding Spark operators? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
