(spark) branch master updated: [SPARK-48265][SQL] Infer window group limit batch should do constant folding

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 7974811218c9 [SPARK-48265][SQL] Infer window group limit batch should 
do constant folding
7974811218c9 is described below

commit 7974811218c9fb52ac9d07f8983475a885ada81b
Author: Angerszhuuuu <angers....@gmail.com>
AuthorDate: Tue May 14 13:44:47 2024 +0800

    [SPARK-48265][SQL] Infer window group limit batch should do constant folding
    
    ### What changes were proposed in this pull request?
    Plan after PropagateEmptyRelation may generate double local limit
    ```
     GlobalLimit 21
     +- LocalLimit 21
    !   +- Union false, false
    !      :- LocalLimit 21
    !      :  +- Project [item_id#647L]
    !      :     +- Filter (xxxx)
    !      :        +- Relation db.table[,... 91 more fields] parquet
    !      +- LocalLimit 21
    !         +- Project [item_id#738L]
    !            +- LocalRelation <empty>, [, ... 91 more fields]
    ```
    to
    ```
     GlobalLimit 21
        +- LocalLimit 21
           - LocalLimit 21
              +- Project [item_id#647L]
                +- Filter (xxxx)
                   +- Relation db.table[,... 91 more fields] parquet
    ```
    after `Infer window group limit batch` batch's `EliminateLimits`
    will be
    ```
     GlobalLimit 21
        +- LocalLimit least(21, 21)
              +- Project [item_id#647L]
                +- Filter (xxxx)
                   +- Relation db.table[,... 91 more fields] parquet
    ```
    It can't work, here miss a `ConstantFolding`
    
    ### Why are the changes needed?
    Fix bug
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #46568 from AngersZhuuuu/SPARK-48265.
    
    Authored-by: Angerszhuuuu <angers....@gmail.com>
    Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
 .../src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
index 70a35ea91153..6173703ef3cd 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala
@@ -89,7 +89,8 @@ class SparkOptimizer(
       InferWindowGroupLimit,
       LimitPushDown,
       LimitPushDownThroughWindow,
-      EliminateLimits) :+
+      EliminateLimits,
+      ConstantFolding) :+
     Batch("User Provided Optimizers", fixedPoint, 
experimentalMethods.extraOptimizations: _*) :+
     Batch("Replace CTE with Repartition", Once, ReplaceCTERefWithRepartition)
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org