[ 
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

XiDuo You updated SPARK-61211:
------------------------------
    Affects Version/s:     (was: 4.4.0)
          Description: 
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
the union's output by zipping child.output with union.output into an 
AttributeMap. Two cases leave a child attribute that does not map correctly, 
and the union can then advertise a layout its rows do not have. 
EnsureRequirements omits a necessary exchange, and window, aggregate or join 
results are wrong. No error is raised.

*Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the 
first column carrying an ExprId, while AttributeMap keeps the last pair for a 
repeated key. If the child output repeats an ExprId, the child's partitioning 
is bound to its first occurrence but mapped from its last. If the union output 
repeats an ExprId, a child attribute can map to a position a parent never 
binds. Duplicate ExprIds reach a union only with 
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.

{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
 "false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled", 
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
"false")
spark.conf.set("spark.sql.codegen.wholeStage", "false")

val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
  row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
{code}

*Case 2: an attribute the child does not output.* An unmapped attribute is left 
unchanged and can equal another child's real output attribute. Object operators 
such as MapGroupsExec and SerializeFromObjectExec pass their child's 
partitioning through while outputting only the object column. This needs no 
non-default conf, since spark.sql.unionOutputPartitioning has defaulted to true 
since 4.1.0.

{code:scala}
spark.conf.set("spark.sql.adaptive.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
  Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
  .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
{code}

The union reports hash(value#K) over a child whose only column is value#V, so 
the window's exchange is omitted.

*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with 
default true, and every release since then has the same AttributeMap mapping. 
All three reproductions above give the same wrong results on branch-4.1 
(4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with AQE on there; 
case 1 is wrong with AQE on and off. 4.0 does not have this feature.

Disabling spark.sql.unionOutputPartitioning avoids both cases.

Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741

  was:
UnionExec.prepareOutputPartitioning can report an incorrect hash layout when a 
child output repeats an exprId. Runtime BindReferences selects the first 
occurrence, while AttributeMap(child.output.zip(union.output)) retains the last 
occurrence. A union can consequently advertise clustering on a column whose 
rows were actually shuffled by another column, causing EnsureRequirements to 
omit a necessary exchange.

This is independent of SPARK-60379's shuffle push-down. Reviewers reproduced 
the existing bug on merge base 352ac9bf228 with push-down off, and with AQE 
both on and off. Disabling spark.sql.unionOutputPartitioning restores the 
expected result. The union output partitioning mapping dates back to 
SPARK-52921 and shipped in 4.1.0; the two analyzer confs this shape needs were 
also added in 4.1.0, so 4.1.0 is the earliest affected release.

Reported reproduction (Scala):
{code:scala}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window
import spark.implicits._

spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
 "false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled", 
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.unionOutputPartitioning", "true")
// Test harness controls; avoid resolver dual-run validation and codegen 
compilation of duplicates.
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
"false")
spark.conf.set("spark.sql.codegen.wholeStage", "false")

val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
  row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1)
{code}

Expected: 1000. Reported actual: 4582. Collect all columns before counting to 
retain the triggering plan shape.

The duplicate second child reports hash(p), physically bound to its first 
column. Its mapping into the outer union reports hash(c), corresponding to the 
second column. Both outer children then appear to report hash(c), although 
their rows are not co-located by c.

Investigate a conservative refusal to propagate partitioning from ambiguous 
child outputs, or a mapping aligned with first-occurrence runtime binding, with 
coverage for both mapping directions and all supported partitioning types. Keep 
this general UnionExec fix separate from the narrower push-down eligibility 
guards in SPARK-60379.

Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741


              Summary: UnionExec reports a wrong output partitioning when a 
child's partitioning references an attribute that does not map to the union 
output  (was: Avoid incorrect union partitioning with duplicate child 
expression IDs)

> UnionExec reports a wrong output partitioning when a child's partitioning 
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61211
>                 URL: https://issues.apache.org/jira/browse/SPARK-61211
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0
>            Reporter: XiDuo You
>            Priority: Major
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
> the union's output by zipping child.output with union.output into an 
> AttributeMap. Two cases leave a child attribute that does not map correctly, 
> and the union can then advertise a layout its rows do not have. 
> EnsureRequirements omits a necessary exchange, and window, aggregate or join 
> results are wrong. No error is raised.
> *Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the 
> first column carrying an ExprId, while AttributeMap keeps the last pair for a 
> repeated key. If the child output repeats an ExprId, the child's partitioning 
> is bound to its first occurrence but mapped from its last. If the union 
> output repeats an ExprId, a child attribute can map to a position a parent 
> never binds. Duplicate ExprIds reach a union only with 
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
>  "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
>  "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
> "false")
> spark.conf.set("spark.sql.codegen.wholeStage", "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
>   row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
> {code}
> *Case 2: an attribute the child does not output.* An unmapped attribute is 
> left unchanged and can equal another child's real output attribute. Object 
> operators such as MapGroupsExec and SerializeFromObjectExec pass their 
> child's partitioning through while outputting only the object column. This 
> needs no non-default conf, since spark.sql.unionOutputPartitioning has 
> defaulted to true since 4.1.0.
> {code:scala}
> spark.conf.set("spark.sql.adaptive.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
>   Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
>   .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
> {code}
> The union reports hash(value#K) over a child whose only column is value#V, so 
> the window's exchange is omitted.
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 
> with default true, and every release since then has the same AttributeMap 
> mapping. All three reproductions above give the same wrong results on 
> branch-4.1 (4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with 
> AQE on there; case 1 is wrong with AQE on and off. 4.0 does not have this 
> feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to