[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 Thanks! Merging to master/2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 Really appreciate your patience and your work! This is how we work in Spark SQL. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80277/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80277/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 I learned a lot from you, thanks all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80277/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 LGTM except a few minor comments. The fix looks great now! Thanks everyone! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80249/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80249/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80249 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80249/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80246/ Test FAILed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80246/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80244/ Test FAILed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80244/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 Thanks for working this! Sorry I've confused you in previous comments. Current changes looks good to me. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 LGTM. cc @gatorsmile for final check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80246/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80237/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80237/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80244/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80237/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80232/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80232/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80232/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 ok,thanks @viirya --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 As using `resolveOperators` can solve the whole bug, let's do it and simplify the whole change. Sorry for confusing. --- If your project is set up for it, you can reply to this email and

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 We need to backport this issue to branch-2.2? I think the opinion depends on the backport decision. If no, I'm with your suggestion (keep this issue as a blocker for branch-2.3). --- If your

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 Well, maybe we should revisit this after #17770 gets merged. Because after that, we won't go through analyzed plans anymore. At that time, we can simply solve all the issues by making

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 I deleted the previous comment because I find I forgot that `resolveOperators` will be removed by #17770 in near future. --- If your project is set up for it, you can

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 Sorry @10110346 would you mind to try and see if https://github.com/apache/spark/pull/18779#discussion_r131287607 works to solve all the issues. If it could, that maybe more simple.

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 It might help to document this in the Dataset `groupBy` comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 Unfortunately, our Dataset APIs support it. We have to keep the support. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 Aha, I see... I'm not sure this is an expected behaviour though, yea, we might have the compatibility issue as you said. I feel, if we explicitly support `group-by-ordinal` in dataset apis, the

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80206/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80206/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @maropu Because we already support it in Dataset API as the example https://github.com/apache/spark/pull/18779#issuecomment-319880753 shows, I'm afraid that there are users using this feature. If we

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 @viirya btw, we still need to support group/sort-by-ordinal in Dataset? If this pr merged, the behaviour seems to change. I feel the syntax in Dataset is a little confusing.. --- If your project

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 @viirya great! thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80206/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @maropu The PR is at #17770. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80197/ Test FAILed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80197/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 ok, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @maropu I have a PR before to solve that. Due to some reasons it will be merged on 2.3. I am out of laptop, will refer it once I can access laptop. On Aug 3, 2017 5:07 PM,

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 @gatorsmile @viirya I looked into why it applied some analyzer rules into already-analyzed plans and I noticed that some rules used `transform/transformUp` instead of `resolveOperators` in `apply`.

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80197/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @gatorsmile scala> df.groupBy(lit(2)).agg(col("a")).queryExecution.logical res6: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan = 'Aggregate [2], [2 AS 2#51,

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80191/ Test FAILed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80191/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 When using Dataset groupBy API, if you use int literals as grouping expressions, do we filter this case out for substituting `UnresolvedOrdinals`? Seems there is no related logic to prevent it.

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-03 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 Our Dataset APIs support `conf.groupByOrdinal`? If so, this might surprise me. `conf.groupByOrdinal` was introduced for SQL APIs only. --- If your project is set up for it, you can reply to

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 Let's remove the rule `SubstituteUnresolvedOrdinals` and move all the `UnresolvedOrdinal` stuffs (order by and group by) into parser. Btw, also don't forget Dataset API, please see

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 Thank you for finding the root cause. @maropu Moving them to the parser sounds reasonable to me, but we also should avoid analyzing the analyzed plan again. Thanks for your works!

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18779 I think this will fail the test case in `SubstituteUnresolvedOrdinalsSuite`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80191/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 I think it is a perfect solution,thank you very much. @viirya @maropu --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 oh, yea. I feel it's ok to do so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 `AstBuilder` can access conf. So it can only replace int literals with `UnresolvedOrdinal` when the config is enabled. Otherwise, leave it as it's. In Analyzer, we remove the rule

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 yea, it is like; it adds `UnresolvedOrdinal` in `AstBuilder` and, if `conf.groupByOrdinal`=false, analyzer drops `UnresolvedOrdinal`. Since `conf.groupByOrdinal`=true by defualt, dropping

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 Well, I think we can not avoid the various cases bringing int literals into grouping expressions. To fix it, I think we should not replace any int literals with `UnresolvedOrdinal` in

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 Thanks, I looked into this failure and I could easily fix this like: https://github.com/apache/spark/compare/master...maropu:SPARK-21580#diff-57b3d87be744b7d79a9beacf8e5e5eb2R2287. But, I

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 Yeah, thanks. Please confirm it. I think running `SubstituteUnresolvedOrdinals` with `Once` should work. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80156/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80156/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 @maropu `select 3 as c, 4 as d, sum(b) from data group by c, d` This test case still has exeception using your modification:GROUP BY position 4 is not in select list (valid range is [1, 3]);

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 ok, fixed: https://github.com/apache/spark/compare/master...maropu:SPARK-21580#diff-a0f2e45a5da747e9ec483f3557aa1b8bR210 --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 I have ran `SubstituteUnresolvedOrdinals` rule with `Once `, it still looks like some problems, i will confirm it --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 oh, I missed. I'll re-check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 @maropu Could you not put the test cases in the end of `group-by-ordinal.sql `? because it has set `spark.sql.groupByOrdinal=false; ` in the end --- If your project is set up for it, you

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 @viirya yea, it does: https://github.com/apache/spark/compare/master...maropu:SPARK-21580#diff-a0f2e45a5da747e9ec483f3557aa1b8bR210. I also think `SubstituteUnresolvedOrdinals` should be applied

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 After rethinking this, do we have any chance to have a group by ordinal or order by ordinal which is not coming from user input and produced by analysis? If the answer is no, we can simply

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @maropu does it solve @10110346's issue at https://github.com/apache/spark/pull/18779#issuecomment-319601312? --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18779 Another way to fix this is something like https://github.com/apache/spark/compare/master...maropu:SPARK-21580. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 Actually I don't think `SubstituteUnresolvedOrdinals` rule should run with `fixedPoint`. It should be run with `Once`. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 `k` is resolved to `4` in `ResolveAggAliasInGroupBy`,and then `4` is resolved to `ResolvedOrdinal(4)` --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 Why? In the query `select a, 4 AS k, count(b) from data group by k, 1`, `1` is resolved to `ResolvedOrdinal(1)` first. Then at the beginning of optimization, `ResolvedOrdinal(1)`

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 I have updated this PR, please help to review it again. @viirya also cc @cloud-fan @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 @viirya Maybe adding `ResolvedOrdinal` is not very well. I have another problem: `select a, **4 AS k**, count(b) from data group by k, 1;` This test case has the same exception:

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80156/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-01 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18779 @10110346 Can't we also do the same on order by ordinal? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-08-01 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/18779 @viirya Only to `group-by ordinal`, i think this is a good idea. but this will also result in inconsistent processing between `order-by ordinal` and `group-by ordinal`. and i feel

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80105/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80105/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-07-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18779 **[Test build #80105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80105/testReport)** for PR 18779 at commit

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80070/ Test PASSed. ---

[GitHub] spark issue #18779: [SPARK-21580][SQL]Integers in aggregation expressions ar...

2017-07-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18779 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   >