[ 
https://issues.apache.org/jira/browse/SPARK-60750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-60750:
-----------------------------------
    Labels: Correctness pull-request-available  (was: Correctness)

> Dataset.cube() and rollup() no longer treat integer literals as GROUP BY 
> ordinals, changing results since 4.1.0
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60750
>                 URL: https://issues.apache.org/jira/browse/SPARK-60750
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.2.1, 4.1.4
>            Reporter: Muhammad Ali Gulzar
>            Priority: Major
>              Labels: Correctness, pull-request-available
>
> {code:python}
> from pyspark.sql import functions as F
> df = spark.range(5).toDF("id").withColumn("g", F.col("id") % 2)
> df.cube(F.col("g"), F.lit(1)).count().collect()
> # expected (4.0.5, and 4.2.1 through Spark Connect): 7 rows, lit(1) is 
> ordinal 1, i.e. cube(g, g)
> #   [(0, 1, 3), (0, 1, 3), (0, 1, 3), (1, 1, 2), (1, 1, 2), (1, 1, 2), (None, 
> 1, 5)]
> # actual (4.2.1, Spark Classic): 6 rows, lit(1) is a constant
> #   [(0, 1, 3), (0, None, 3), (1, 1, 2), (1, None, 2), (None, 1, 5), (None, 
> None, 5)]{code}
>  
> The issue is in Spark Classic. Spark Connect returns the expected result. The 
> same with {{{}rollup{}}}, which returns 5 rows either way, but different 
> rows. With a literal that is not a valid position, 4.0.5 and Spark Connect 
> fail with GROUP_BY_POS_OUT_OF_RANGE or GROUP_BY_POS_AGGREGATE, while Spark 
> Classic in 4.2.1 accepts it. For example, {{df.cube(F.lit(-28)).count()}} 
> returns {{{}[(-28, 5), (None, 5)]{}}}. {{groupBy}} is not affected, and all 
> versions agree when {{{}spark.sql.groupByOrdinal=false{}}}.
> Cause: SPARK-51820 ([https://github.com/apache/spark/pull/50606]) moved the 
> conversion of integer literals to UnresolvedOrdinal out of the analyzer and 
> into the API layer. In Spark Classic, only {{Dataset.groupBy}} does the 
> conversion (Dataset.scala, {{{}groupingExpressionsWithOrdinals{}}}), while 
> {{{}rollup{}}}, {{cube,}} and {{groupingSets}} pass {{cols.map(_.expr)}} 
> unchanged. SparkConnectPlanner converts the grouping expressions for every 
> group type. Applying the same conversion in {{{}Dataset.rollup{}}}, {{cube}} 
> and {{groupingSets}} would fix it.
> Reproduces on 4.1.4, 4.2.0, 4.2.1 and master (5.0.0-SNAPSHOT, 2026-10-29). 
> Spark 3.5.9 and 4.0.5 read the literal as an ordinal, as Spark Connect does.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to