Takuya Ueshin created SPARK-39293: ------------------------------------- Summary: The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map Key: SPARK-39293 URL: https://issues.apache.org/jira/browse/SPARK-39293 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.2.1, 3.1.2, 3.0.3, 3.3.0 Reporter: Takuya Ueshin
The accumulator of ArrayAggregate should copy the intermediate result if string, struct, array, or map. {code:scala} import org.apache.spark.sql.functions._ val reverse = udf((s: String) => s.reverse) val df = Seq(Array("abc", "def")).toDF("array") val testArray = df.withColumn( "agg", aggregate( col("array"), array().cast("array<string>"), (acc, s) => concat(acc, array(reverse(s))))) aggArray.show(truncate=false) {code} should be: {code} +----------+----------+ |array |agg | +----------+----------+ |[abc, def]|[cba, fed]| +----------+----------+ {code} but: {code} +----------+----------+ |array |agg | +----------+----------+ |[abc, def]|[fed, fed]| +----------+----------+ {code} -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org