[
https://issues.apache.org/jira/browse/SPARK-60750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60750:
-----------------------------------
Labels: Correctness pull-request-available (was: Correctness)
> Dataset.cube() and rollup() no longer treat integer literals as GROUP BY
> ordinals, changing results since 4.1.0
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60750
> URL: https://issues.apache.org/jira/browse/SPARK-60750
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.1, 4.1.4
> Reporter: Muhammad Ali Gulzar
> Priority: Major
> Labels: Correctness, pull-request-available
>
> {code:python}
> from pyspark.sql import functions as F
> df = spark.range(5).toDF("id").withColumn("g", F.col("id") % 2)
> df.cube(F.col("g"), F.lit(1)).count().collect()
> # expected (4.0.5, and 4.2.1 through Spark Connect): 7 rows, lit(1) is
> ordinal 1, i.e. cube(g, g)
> # [(0, 1, 3), (0, 1, 3), (0, 1, 3), (1, 1, 2), (1, 1, 2), (1, 1, 2), (None,
> 1, 5)]
> # actual (4.2.1, Spark Classic): 6 rows, lit(1) is a constant
> # [(0, 1, 3), (0, None, 3), (1, 1, 2), (1, None, 2), (None, 1, 5), (None,
> None, 5)]{code}
>
> The issue is in Spark Classic. Spark Connect returns the expected result. The
> same with {{{}rollup{}}}, which returns 5 rows either way, but different
> rows. With a literal that is not a valid position, 4.0.5 and Spark Connect
> fail with GROUP_BY_POS_OUT_OF_RANGE or GROUP_BY_POS_AGGREGATE, while Spark
> Classic in 4.2.1 accepts it. For example, {{df.cube(F.lit(-28)).count()}}
> returns {{{}[(-28, 5), (None, 5)]{}}}. {{groupBy}} is not affected, and all
> versions agree when {{{}spark.sql.groupByOrdinal=false{}}}.
> Cause: SPARK-51820 ([https://github.com/apache/spark/pull/50606]) moved the
> conversion of integer literals to UnresolvedOrdinal out of the analyzer and
> into the API layer. In Spark Classic, only {{Dataset.groupBy}} does the
> conversion (Dataset.scala, {{{}groupingExpressionsWithOrdinals{}}}), while
> {{{}rollup{}}}, {{cube,}} and {{groupingSets}} pass {{cols.map(_.expr)}}
> unchanged. SparkConnectPlanner converts the grouping expressions for every
> group type. Applying the same conversion in {{{}Dataset.rollup{}}}, {{cube}}
> and {{groupingSets}} would fix it.
> Reproduces on 4.1.4, 4.2.0, 4.2.1 and master (5.0.0-SNAPSHOT, 2026-10-29).
> Spark 3.5.9 and 4.0.5 read the literal as an ordinal, as Spark Connect does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]