sridhar09 opened a new issue, #13522:
URL: https://github.com/apache/gluten/issues/13522
### Backend
VL (Velox)
### Bug description
> **1.6.0 users: do not turn on `spark.gluten.sql.native.union` without
reading this issue.** It silently returns wrong results on 1.6.0.
**[Expected behavior]** A wide transformation on a unioned DataFrame returns
the same rows as vanilla Spark.
**[Actual behavior]** On Gluten 1.6.0 (Velox backend) with
`spark.gluten.sql.native.union=true`, every column of a given type gets the
value of the first column of that type. Nothing errors and nothing is logged.
**When it happens**
1. Two partitioned DataFrames are unioned, so Gluten runs the union natively
in Velox (`UnionExecTransformer`).
2. A wide transformation (shuffle, aggregate or join) runs on the unioned
DataFrame.
**Repro** (spark-shell, Spark 3.3.0, Gluten 1.6.0; correct on 1.7.0)
```scala
val df1 = (1 to 10).map(i => (s"a$i", s"b$i", s"c$i")).toDF("a", "b", "c")
val df2 = (11 to 20).map(i => (s"a$i", s"b$i", s"c$i")).toDF("a", "b", "c")
df1.repartition($"a").union(df2.repartition($"a")).repartition($"b").show()
```
```
expected actual
|a3|b3|c3| |a3|a3|a3|
```
**Cause:** already fixed on main by #11832 (released in 1.7.0), but not on
`branch-1.6`. There, `SubstraitToVeloxPlan.cc` still names the union's output
columns after their type, so every same-typed column resolves to the first one.
**Request**
1. Backport #11832 (commit `985b19ae4`) to `branch-1.6`. It applies cleanly
to `v1.6.0`, test included.
2. Until then, add a warning to the `spark.gluten.sql.native.union` config
doc: do not enable on 1.6.0 without checking this issue.
**Workaround:** keep `spark.gluten.sql.native.union=false` (the default), or
upgrade to 1.7.0.
### Gluten version
_No response_
### Spark version
Spark-3.4.x
### Spark configurations
```
spark.plugins org.apache.gluten.GlutenPlugin
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 2g
spark.shuffle.manager
org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.gluten.sql.native.union true
```
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
