[ 
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

XiDuo You updated SPARK-61211:
------------------------------
    Description: 
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
the union's output by zipping child.output with union.output into an 
AttributeMap. When a child attribute does not map correctly, the union can 
advertise a layout its rows do not have. EnsureRequirements then omits a 
necessary exchange, and window, aggregate or join results are wrong. No error 
is raised.

*Case 1 (default configs): a partitioning references an attribute the child 
does not output.* An unmapped attribute was left unchanged and can equal 
another child's real output attribute. A column-pruned InMemoryTableScanExec 
reports the cached plan's partitioning over columns it no longer outputs, and 
MapGroupsExec and SerializeFromObjectExec pass the grouping key's partitioning 
through while outputting only the mapped value.

{code:scala}
val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
val c = kv.keys.toDF().repartition($"_1").cache()
c.count()
c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
// expected 10, actual 38 (AQE on and off)
{code}

{code:scala}
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
  Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
  .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
{code}

With AQE on and partition coalescing on, the second query's few rows are 
coalesced into one partition, which hides the wrong result.

*Case 2 (non-default configs): duplicate ExprIds.* Runtime binding 
(AttributeSeq.indexOf) picks the first column carrying an ExprId, while 
AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach a 
union only with 
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.

{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
 "false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled", 
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
"false")

val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
  row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
{code}

*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with 
default true, and every release since then has the same AttributeMap mapping. 
The mapGroups and duplicate-ExprId reproductions give the same wrong results on 
branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition coalescing is 
off. 4.0 does not have this feature.

Disabling spark.sql.unionOutputPartitioning avoids both cases.

Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
https://github.com/apache/spark/pull/60271#discussion_r4522188035

  was:
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
the union's output by zipping child.output with union.output into an 
AttributeMap. Two cases leave a child attribute that does not map correctly, 
and the union can then advertise a layout its rows do not have. 
EnsureRequirements omits a necessary exchange, and window, aggregate or join 
results are wrong. No error is raised.

*Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the 
first column carrying an ExprId, while AttributeMap keeps the last pair for a 
repeated key. If the child output repeats an ExprId, the child's partitioning 
is bound to its first occurrence but mapped from its last. If the union output 
repeats an ExprId, a child attribute can map to a position a parent never 
binds. Duplicate ExprIds reach a union only with 
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.

{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
 "false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled", 
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
"false")
spark.conf.set("spark.sql.codegen.wholeStage", "false")

val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
  row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
{code}

*Case 2: an attribute the child does not output.* An unmapped attribute is left 
unchanged and can equal another child's real output attribute. Object operators 
such as MapGroupsExec and SerializeFromObjectExec pass their child's 
partitioning through while outputting only the object column. This needs no 
non-default conf, since spark.sql.unionOutputPartitioning has defaulted to true 
since 4.1.0.

{code:scala}
spark.conf.set("spark.sql.adaptive.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
  Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
  .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
{code}

The union reports hash(value#K) over a child whose only column is value#V, so 
the window's exchange is omitted.

*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with 
default true, and every release since then has the same AttributeMap mapping. 
All three reproductions above give the same wrong results on branch-4.1 
(4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with AQE on there; 
case 1 is wrong with AQE on and off. 4.0 does not have this feature.

Disabling spark.sql.unionOutputPartitioning avoids both cases.

Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741


> UnionExec reports a wrong output partitioning when a child's partitioning 
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61211
>                 URL: https://issues.apache.org/jira/browse/SPARK-61211
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.4.0
>            Reporter: XiDuo You
>            Priority: Major
>              Labels: pull-request-available
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
> the union's output by zipping child.output with union.output into an 
> AttributeMap. When a child attribute does not map correctly, the union can 
> advertise a layout its rows do not have. EnsureRequirements then omits a 
> necessary exchange, and window, aggregate or join results are wrong. No error 
> is raised.
> *Case 1 (default configs): a partitioning references an attribute the child 
> does not output.* An unmapped attribute was left unchanged and can equal 
> another child's real output attribute. A column-pruned InMemoryTableScanExec 
> reports the cached plan's partitioning over columns it no longer outputs, and 
> MapGroupsExec and SerializeFromObjectExec pass the grouping key's 
> partitioning through while outputting only the mapped value.
> {code:scala}
> val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
> val c = kv.keys.toDF().repartition($"_1").cache()
> c.count()
> c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
> // expected 10, actual 38 (AQE on and off)
> {code}
> {code:scala}
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
>   Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
>   .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
> {code}
> With AQE on and partition coalescing on, the second query's few rows are 
> coalesced into one partition, which hides the wrong result.
> *Case 2 (non-default configs): duplicate ExprIds.* Runtime binding 
> (AttributeSeq.indexOf) picks the first column carrying an ExprId, while 
> AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach 
> a union only with 
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
>  "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
>  "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
> "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
>   row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
> {code}
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 
> with default true, and every release since then has the same AttributeMap 
> mapping. The mapGroups and duplicate-ExprId reproductions give the same wrong 
> results on branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition 
> coalescing is off. 4.0 does not have this feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741
> https://github.com/apache/spark/pull/60271#discussion_r4522188035



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to