andygrove opened a new issue, #2892: URL: https://github.com/apache/datafusion-comet/issues/2892
### Describe the bug The epic is intended to outline the work needed to address various limitations in Comet's hash aggregate planning. The current situation: - Comet converts Spark nodes to Comet starting from the leaf node and walking up the tree. When considering whether a partial aggregate should be converted to Comet there is no check to see if the parent would also be supported. - When Comet is considering converting a final aggregate to Comet, there is a check to see if the partial ran in Comet, and the final will fall back to Spark if this is not the case. This is overly aggressive and we miss out on performance in the case where mixed Comet/Spark aggregates are actually compatible. The proposed solution: - Improve the planning so that the final and partial are considered together and we have a way of defining which aggregates are safe to be mixed and which ones must either run fully in Spark or Comet. ### Steps to reproduce _No response_ ### Expected behavior _No response_ ### Additional context Here are some known issues when stop falling back for the final aggregate: ### Sum/Avg in ANSI mode When allowing Spark partial with Comet final aggregate, there are test failures in Spark SQL: ``` [info] - SPARK-28224: Aggregate sum big decimal overflow *** FAILED *** (140 milliseconds) [info] org.apache.comet.CometNativeException: [ARITHMETIC_OVERFLOW] decimal overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. was not instance of java.lang.ArithmeticException (DataFrameSuite.scala:214) [info] - SPARK-28067: Aggregate sum should not return wrong results for decimal overflow *** FAILED *** (270 milliseconds) [info] Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (DataFrameSuite.scala:211) [info] - SPARK-35955: Aggregate avg should not return wrong results for decimal overflow *** FAILED *** (977 milliseconds) [info] Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (DataFrameSuite.scala:211) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
