[ https://issues.apache.org/jira/browse/SPARK-47287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Chester Jenks updated SPARK-47287: -------------------------------------- Description: The below snippet is confirmed working with Spark 3.2.1 and broken Spark 3.4.1. i believe this is a bug. {code:java} Dataset<Row> ds = dummyDataset .withColumn("flag", functions.not(functions.coalesce(functions.col("bool1"), functions.lit(false)).equalTo(true))) .groupBy("code") .agg(functions.max(functions.col("flag")).alias("flag")); ds.show(); {code} It fails with: {code:java} Caused by: java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:208) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpression$7(V2ExpressionBuilder.scala:185) at scala.Option.map(Option.scala:230) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:184) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33) at org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateAggregateFunc(V2ExpressionBuilder.scala:293) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33) at org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803) at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.translate$1(DataSourceStrategy.scala:700){code} was: The below snippet is confirmed working with Spark 3.2.1 and broken Spark 3.4.1. i believe this is a bug. {code:java} Dataset<Row> ds = dummyDataset .withColumn("flag", functions.not(functions.coalesce(functions.col("bool1"), functions.lit(false)).equalTo(true))) .groupBy("code") .agg(functions.max(functions.col("flag")).alias("flag")); ds.show(); {code} It fails with: {code:java} Caused by: java.lang.AssertionError: assertion failed at scala.Predef$.assert(Predef.scala:208) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpression$7(V2ExpressionBuilder.scala:185) at scala.Option.map(Option.scala:230) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:184) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33) at org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateAggregateFunc(V2ExpressionBuilder.scala:293) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98) at org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33) at org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803) at org.apache.spark.sql.execution.datasources.DataSourceStrategy$.translate$1(DataSourceStrategy.scala:700) {code} > Aggregate in not causes > ------------------------ > > Key: SPARK-47287 > URL: https://issues.apache.org/jira/browse/SPARK-47287 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.4.1 > Reporter: Ted Chester Jenks > Priority: Major > > > The below snippet is confirmed working with Spark 3.2.1 and broken Spark > 3.4.1. i believe this is a bug. > {code:java} > Dataset<Row> ds = dummyDataset > .withColumn("flag", > functions.not(functions.coalesce(functions.col("bool1"), > functions.lit(false)).equalTo(true))) > .groupBy("code") > .agg(functions.max(functions.col("flag")).alias("flag")); > ds.show(); {code} > It fails with: > {code:java} > Caused by: java.lang.AssertionError: assertion failed > at scala.Predef$.assert(Predef.scala:208) > at > org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpression$7(V2ExpressionBuilder.scala:185) > at scala.Option.map(Option.scala:230) > at > org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:184) > at > org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33) > at > org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803) > at > org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateAggregateFunc(V2ExpressionBuilder.scala:293) > at > org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98) > at > org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:33) > at > org.apache.spark.sql.execution.datasources.PushableExpression$.unapply(DataSourceStrategy.scala:803) > at > org.apache.spark.sql.execution.datasources.DataSourceStrategy$.translate$1(DataSourceStrategy.scala:700){code} > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org