GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/9419
[SPARK-11275][SQL][WIP] Rollup and Cube Generates the Incorrect Results when Aggregation Functions Use Group By Columns In the current implementation, Rollup and Cube are unable to generate the correct results for the following cases: When the aggregation functions use the group by key columns: sql("select b, a, sum(a), min(a), min(b+b) from mytable group by a, b with rollup").collect() sql("select a, b, sum(a), min(a), min(b+b) from mytable group by b, a with cube").collect() The problem becomes more complex if the group by clauses have the functions whose inputs are also appear in the group by. sql("select a + b, b, sum(a - b) from mytable group by a + b, b with rollup").collect() sql("select a + b, b, sum(a - b) from mytable group by a + b, b with cube").collect() The basic solutions are adding extra Projection when the query are part of the above situations. The projection will add duplicate values for these affected columns with alias names so that the column values will not be removed when expand is evaluated during the runtime. Working on the test cases. Will add more cases into Hive golden buckets. Welcome any comment and suggestion! Thank you! You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark rollupCube Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9419.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9419 ---- commit b10418e161d5809f3b1de92cf4a33b2f362cd2b4 Author: Xiao Li <xiaoli@xiaos-imac.local> Date: 2015-11-02T09:05:31Z Spark-11275 commit 7721442cdf65924af204d39e3b3b7bda6c41dfc6 Author: Xiao Li <xiaoli@xiaos-imac.local> Date: 2015-11-02T09:51:16Z syntax cleaning ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org