[ 
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

XiDuo You updated SPARK-61211:
------------------------------
    Description: 
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
the union's output by zipping child.output with union.output into an 
AttributeMap. When a child attribute does not map correctly, the union can 
advertise a layout its rows do not have. EnsureRequirements then omits a 
necessary exchange, and window, aggregate or join results are wrong. No error 
is raised.

*Case 1 (default configs): a partitioning references an attribute the child 
does not output.* An unmapped attribute was left unchanged and can equal 
another child's real output attribute. A column-pruned InMemoryTableScanExec 
reports the cached plan's partitioning over columns it no longer outputs, and 
MapGroupsExec and SerializeFromObjectExec pass the grouping key's partitioning 
through while outputting only the mapped value.

{code:scala}
val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
val c = kv.keys.toDF().repartition($"_1").cache()
c.count()
c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
// expected 10, actual 38 (AQE on and off)
{code}

{code:scala}
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
  Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
  .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
{code}

With AQE on and partition coalescing on, the second query's few rows are 
coalesced into one partition, which hides the wrong result.

*Case 2 (non-default configs): duplicate ExprIds.* Runtime binding 
(AttributeSeq.indexOf) picks the first column carrying an ExprId, while 
AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach a 
union only with 
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.

{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
 "false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled", 
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
"false")

val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
  row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
{code}

Case 2 is fixed by SPARK-61356.

*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with 
default true, and every release since then has the same AttributeMap mapping. 
The mapGroups and duplicate-ExprId reproductions give the same wrong results on 
branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition coalescing is 
off. 4.0 does not have this feature.

Disabling spark.sql.unionOutputPartitioning avoids both cases.

Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
https://github.com/apache/spark/pull/60271#discussion_r4522188035

  was:
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
the union's output by zipping child.output with union.output into an 
AttributeMap. When a child attribute does not map correctly, the union can 
advertise a layout its rows do not have. EnsureRequirements then omits a 
necessary exchange, and window, aggregate or join results are wrong. No error 
is raised.

*Case 1 (default configs): a partitioning references an attribute the child 
does not output.* An unmapped attribute was left unchanged and can equal 
another child's real output attribute. A column-pruned InMemoryTableScanExec 
reports the cached plan's partitioning over columns it no longer outputs, and 
MapGroupsExec and SerializeFromObjectExec pass the grouping key's partitioning 
through while outputting only the mapped value.

{code:scala}
val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
val c = kv.keys.toDF().repartition($"_1").cache()
c.count()
c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
// expected 10, actual 38 (AQE on and off)
{code}

{code:scala}
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
  Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
  .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
{code}

With AQE on and partition coalescing on, the second query's few rows are 
coalesced into one partition, which hides the wrong result.

*Case 2 (non-default configs): duplicate ExprIds.* Runtime binding 
(AttributeSeq.indexOf) picks the first column carrying an ExprId, while 
AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach a 
union only with 
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.

{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
 "false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled", 
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
"false")

val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
  row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
{code}

*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with 
default true, and every release since then has the same AttributeMap mapping. 
The mapGroups and duplicate-ExprId reproductions give the same wrong results on 
branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition coalescing is 
off. 4.0 does not have this feature.

Disabling spark.sql.unionOutputPartitioning avoids both cases.

Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
https://github.com/apache/spark/pull/60271#discussion_r4522188035


> UnionExec reports a wrong output partitioning when a child's partitioning 
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61211
>                 URL: https://issues.apache.org/jira/browse/SPARK-61211
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.4.0
>            Reporter: XiDuo You
>            Priority: Major
>              Labels: pull-request-available
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
> the union's output by zipping child.output with union.output into an 
> AttributeMap. When a child attribute does not map correctly, the union can 
> advertise a layout its rows do not have. EnsureRequirements then omits a 
> necessary exchange, and window, aggregate or join results are wrong. No error 
> is raised.
> *Case 1 (default configs): a partitioning references an attribute the child 
> does not output.* An unmapped attribute was left unchanged and can equal 
> another child's real output attribute. A column-pruned InMemoryTableScanExec 
> reports the cached plan's partitioning over columns it no longer outputs, and 
> MapGroupsExec and SerializeFromObjectExec pass the grouping key's 
> partitioning through while outputting only the mapped value.
> {code:scala}
> val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
> val c = kv.keys.toDF().repartition($"_1").cache()
> c.count()
> c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
> // expected 10, actual 38 (AQE on and off)
> {code}
> {code:scala}
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
>   Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
>   .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
> {code}
> With AQE on and partition coalescing on, the second query's few rows are 
> coalesced into one partition, which hides the wrong result.
> *Case 2 (non-default configs): duplicate ExprIds.* Runtime binding 
> (AttributeSeq.indexOf) picks the first column carrying an ExprId, while 
> AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach 
> a union only with 
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
>  "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
>  "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
> "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
>   row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
> {code}
> Case 2 is fixed by SPARK-61356.
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 
> with default true, and every release since then has the same AttributeMap 
> mapping. The mapGroups and duplicate-ExprId reproductions give the same wrong 
> results on branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition 
> coalescing is off. 4.0 does not have this feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741
> https://github.com/apache/spark/pull/60271#discussion_r4522188035



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to