[
https://issues.apache.org/jira/browse/SPARK-61211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-61211:
------------------------------
Description:
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
the union's output by zipping child.output with union.output into an
AttributeMap. When a child attribute does not map correctly, the union can
advertise a layout its rows do not have. EnsureRequirements then omits a
necessary exchange, and window, aggregate or join results are wrong. No error
is raised.
*Case 1 (default configs): a partitioning references an attribute the child
does not output.* An unmapped attribute was left unchanged and can equal
another child's real output attribute. A column-pruned InMemoryTableScanExec
reports the cached plan's partitioning over columns it no longer outputs, and
MapGroupsExec and SerializeFromObjectExec pass the grouping key's partitioning
through while outputting only the mapped value.
{code:scala}
val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
val c = kv.keys.toDF().repartition($"_1").cache()
c.count()
c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
// expected 10, actual 38 (AQE on and off)
{code}
{code:scala}
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
.collect().count(_.getInt(1) == 1) // expected 11, actual 19
{code}
With AQE on and partition coalescing on, the second query's few rows are
coalesced into one partition, which hides the wrong result.
*Case 2 (non-default configs): duplicate ExprIds.* Runtime binding
(AttributeSeq.indexOf) picks the first column carrying an ExprId, while
AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach a
union only with
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
"false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
"false")
val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
{code}
*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with
default true, and every release since then has the same AttributeMap mapping.
The mapGroups and duplicate-ExprId reproductions give the same wrong results on
branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition coalescing is
off. 4.0 does not have this feature.
Disabling spark.sql.unionOutputPartitioning avoids both cases.
Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
https://github.com/apache/spark/pull/60271#discussion_r4522188035
was:
UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
the union's output by zipping child.output with union.output into an
AttributeMap. Two cases leave a child attribute that does not map correctly,
and the union can then advertise a layout its rows do not have.
EnsureRequirements omits a necessary exchange, and window, aggregate or join
results are wrong. No error is raised.
*Case 1: duplicate ExprIds.* Runtime binding (AttributeSeq.indexOf) picks the
first column carrying an ExprId, while AttributeMap keeps the last pair for a
repeated key. If the child output repeats an ExprId, the child's partitioning
is bound to its first occurrence but mapped from its last. If the union output
repeats an ExprId, a child attribute can map to a position a parent never
binds. Duplicate ExprIds reach a union only with
spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
{code:scala}
spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
"false")
spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
"false")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
spark.conf.set("spark.sql.shuffle.partitions", "5")
spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
"false")
spark.conf.set("spark.sql.codegen.wholeStage", "false")
val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
val r = p1.union(p2).repartition(p1.col("p"))
val result = x.union(r).withColumn("rn",
row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
{code}
*Case 2: an attribute the child does not output.* An unmapped attribute is left
unchanged and can equal another child's real output attribute. Object operators
such as MapGroupsExec and SerializeFromObjectExec pass their child's
partitioning through while outputting only the object column. This needs no
non-default conf, since spark.sql.unionOutputPartitioning has defaulted to true
since 4.1.0.
{code:scala}
spark.conf.set("spark.sql.adaptive.enabled", "false")
val kv = spark.range(100).as[Long].groupByKey(_ % 10)
val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
u.withColumn("rn", row_number().over(
Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
.collect().count(_.getInt(1) == 1) // expected 11, actual 19
{code}
The union reports hash(value#K) over a child whose only column is value#V, so
the window's exchange is omitted.
*Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0 with
default true, and every release since then has the same AttributeMap mapping.
All three reproductions above give the same wrong results on branch-4.1
(4.1.5-SNAPSHOT). Case 2 is wrong with AQE off and correct with AQE on there;
case 1 is wrong with AQE on and off. 4.0 does not have this feature.
Disabling spark.sql.unionOutputPartitioning avoids both cases.
Reviewer evidence:
https://github.com/apache/spark/pull/59630#discussion_r4495483202
https://github.com/apache/spark/pull/59630#discussion_r4493424741
> UnionExec reports a wrong output partitioning when a child's partitioning
> references an attribute that does not map to the union output
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61211
> URL: https://issues.apache.org/jira/browse/SPARK-61211
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0, 4.2.0, 4.3.0, 5.0.0, 4.4.0
> Reporter: XiDuo You
> Priority: Major
> Labels: pull-request-available
>
> UnionExec.prepareOutputPartitioning maps each child's outputPartitioning onto
> the union's output by zipping child.output with union.output into an
> AttributeMap. When a child attribute does not map correctly, the union can
> advertise a layout its rows do not have. EnsureRequirements then omits a
> necessary exchange, and window, aggregate or join results are wrong. No error
> is raised.
> *Case 1 (default configs): a partitioning references an attribute the child
> does not output.* An unmapped attribute was left unchanged and can equal
> another child's real output attribute. A column-pruned InMemoryTableScanExec
> reports the cached plan's partitioning over columns it no longer outputs, and
> MapGroupsExec and SerializeFromObjectExec pass the grouping key's
> partitioning through while outputting only the mapped value.
> {code:scala}
> val kv = spark.range(1000).as[Long].groupByKey(x => (x % 10, x % 7))
> val c = kv.keys.toDF().repartition($"_1").cache()
> c.count()
> c.select("_1").union(c.select("_2")).groupBy("_1").count().count()
> // expected 10, actual 38 (AQE on and off)
> {code}
> {code:scala}
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> val kv = spark.range(100).as[Long].groupByKey(_ % 10)
> val u = kv.keys.toDF().union(kv.mapGroups((k, _) => k + 1).toDF())
> u.withColumn("rn", row_number().over(
> Window.partitionBy(u.columns(0)).orderBy(u.columns(0))))
> .collect().count(_.getInt(1) == 1) // expected 11, actual 19
> {code}
> With AQE on and partition coalescing on, the second query's few rows are
> coalesced into one partition, which hides the wrong result.
> *Case 2 (non-default configs): duplicate ExprIds.* Runtime binding
> (AttributeSeq.indexOf) picks the first column carrying an ExprId, while
> AttributeMap keeps the last pair for a repeated key. Duplicate ExprIds reach
> a union only with
> spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved and
> spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled turned off.
> {code:scala}
> spark.conf.set("spark.sql.analyzer.unionIsResolvedWhenDuplicatesPerChildResolved",
> "false")
> spark.conf.set("spark.sql.enforceTypeCoercionBeforeUnionDeduplication.enabled",
> "false")
> spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "false")
> spark.conf.set("spark.sql.shuffle.partitions", "5")
> spark.conf.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy",
> "false")
> val p1 = spark.range(1000).selectExpr("id p", "id w").select($"p", $"p", $"w")
> val p2 = spark.range(1000, 11000).selectExpr("id q", "id % 1000 r", "id w2")
> val x = spark.range(1000).selectExpr("id x", "id c", "id u").repartition($"c")
> val r = p1.union(p2).repartition(p1.col("p"))
> val result = x.union(r).withColumn("rn",
> row_number().over(Window.partitionBy(x.col("c")).orderBy(x.col("u"))))
> result.collect().count(_.getInt(3) == 1) // expected 1000, actual 4582
> {code}
> *Affected versions.* spark.sql.unionOutputPartitioning was added in 4.1.0
> with default true, and every release since then has the same AttributeMap
> mapping. The mapGroups and duplicate-ExprId reproductions give the same wrong
> results on branch-4.1 (4.1.5-SNAPSHOT), with AQE on as well once partition
> coalescing is off. 4.0 does not have this feature.
> Disabling spark.sql.unionOutputPartitioning avoids both cases.
> Reviewer evidence:
> https://github.com/apache/spark/pull/59630#discussion_r4495483202
> https://github.com/apache/spark/pull/59630#discussion_r4493424741
> https://github.com/apache/spark/pull/60271#discussion_r4522188035
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]