[ https://issues.apache.org/jira/browse/SPARK-32542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
karl wang updated SPARK-32542: ------------------------------ Description: Split an expand into several smallExpand,which contains the Specified number of projections. For instance,like this sql.select a,b,c,d,count(1) from table1 group by a,b,c,d with cube. It can expand 2^4 times of original data size. Now we specify the spark.sql.optimizer.projections.size=4.The Expand will be split into 2^4/4 smallExpand.It can reduce the shuffle pressure and improve performance in multidimensional analysis when data is huge. > add a batch for optimizing logicalPlan > -------------------------------------- > > Key: SPARK-32542 > URL: https://issues.apache.org/jira/browse/SPARK-32542 > Project: Spark > Issue Type: Improvement > Components: Optimizer > Affects Versions: 3.0.0 > Reporter: karl wang > Priority: Minor > Fix For: 3.0.0 > > > Split an expand into several smallExpand,which contains the Specified number > of projections. > For instance,like this sql.select a,b,c,d,count(1) from table1 group by > a,b,c,d with cube. It can expand 2^4 times of original data size. > Now we specify the spark.sql.optimizer.projections.size=4.The Expand will be > split into 2^4/4 smallExpand.It can reduce the shuffle pressure and improve > performance in multidimensional analysis when data is huge. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org