[ 
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61211:
-----------------------------------
    Labels: pull-request-available  (was: )

> UnionExec reports a wrong output partitioning when a child's partitioning 
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61211
>                 URL: https://issues.apache.org/jira/browse/SPARK-61211
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.4.0
>            Reporter: XiDuo You
>            Priority: Major
>              Labels: pull-request-available
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto 
> the union's output by zipping child.output with union.output into an 
> AttributeMap. Two cases leave a child attribute that does not map correctly, 
> and the union can then advertise a layout its rows do not have. 
> EnsureRequirements omits a necessary exchange, and window, aggregate or join 
> results are wrong. No error is raised.
> *Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the 
> first column carrying an ExprId, while AttributeMap keeps the last pair for a 
> repeated key. If the child output repeats an ExprId, the child's partitioning 
> is bound to its first occurrence but mapped from its last. If the union 
> output repeats an ExprId, a child attribute can map to a position a parent 
> never binds. Duplicate ExprIds reach a union only with 
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and 
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
>  "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
>  "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", 
> "false")
> spark.conf.set("spark.sql.codegen.wholeStage", "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
>   row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1)  // expected 1000, actual 4582
> {code}
> *Case 2: an attribute the child does not output.* An unmapped attribute is 
> left unchanged and can equal another child's real output attribute. Object 
> operators such as MapGroupsExec and SerializeFromObjectExec pass their 
> child's partitioning through while outputting only the object column. This 
> needs no non-default conf, since spark.sql.unionOutputPartitioning has 
> defaulted to true since 4.1.0.
> {code:scala}
> spark.conf.set("spark.sql.adaptive.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
>   Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
>   .collect().count(_.getInt(1) == 1)  // expected 11, actual 19
> {code}
> The union reports hash(value#K) over a child whose only column is value#V, so 
> the window's exchange is omitted.
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 
> with default true, and every release since then has the same AttributeMap 
> mapping. All three reproductions above give the same wrong results on 
> branch-4.1 (4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with 
> AQE on there; case 1 is wrong with AQE on and off. 4.0 does not have this 
> feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to