[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9123 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-149980991 Thank you! Merging to master and 1.5 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-149790801 ping @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148519127 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148519131 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43802/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148518898 [Test build #43802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43802/console) for PR 9123 at commit [`4f162ed`](https://github.com/apache/spark/commit/4f162edae1201179817067a2fef1d469e45283a6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148512630 cc @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148512143 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43796/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148512142 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148511879 [Test build #43796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43796/console) for PR 9123 at commit [`9ed26d1`](https://github.com/apache/spark/commit/9ed26d16ee19c514f3d5281a1e34cc72661f111e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148487314 [Test build #43802 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43802/consoleFull) for PR 9123 at commit [`4f162ed`](https://github.com/apache/spark/commit/4f162edae1201179817067a2fef1d469e45283a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148487000 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148486845 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148486974 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148484237 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43798/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148484236 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148484150 [Test build #43798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43798/console) for PR 9123 at commit [`4f162ed`](https://github.com/apache/spark/commit/4f162edae1201179817067a2fef1d469e45283a6). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148484211 LGTM pending test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148480193 [Test build #43798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43798/consoleFull) for PR 9123 at commit [`4f162ed`](https://github.com/apache/spark/commit/4f162edae1201179817067a2fef1d469e45283a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148479594 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148479569 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148476915 [Test build #43796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43796/consoleFull) for PR 9123 at commit [`9ed26d1`](https://github.com/apache/spark/commit/9ed26d16ee19c514f3d5281a1e34cc72661f111e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148476166 hi @dilipbiswal , can you update the PR title to make it a complete sentence(i.e. no ellipsis)? and can you also put my explaination of this bug in PR description? so that if someone look into the commit log in the future, they can easily understand what's going on here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148475647 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148475604 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9123#discussion_r42158784
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
---
@@ -135,4 +135,14 @@ class AnalysisSuite extends AnalysisTest {
plan = testRelation.select(CreateStructUnsafe(Seq(a, (a +
1).as("a+1"))).as("col"))
checkAnalysis(plan, plan)
}
+
+ test("SPARK-10534: resolve attribute references in order by clause") {
+val a = testRelation2.output(0)
+val c = testRelation2.output(2)
+
+val plan = testRelation2.select(c).orderBy(Floor(a).asc)
--- End diff --
use `'c` and `'a` instead of `c` and `a`. The `plan` will get analyzed in
the method `checkAnalysis`, so we should give unresolved attributes, to
simulate normal query plans. (FYI, in spark SQL DSL, the symbol `'a` will be
automatically turned into `UnresolvedAttribute`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/9123#discussion_r42158115 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -482,7 +482,9 @@ class Analyzer( val newOrdering = resolveSortOrders(ordering, grandchild, throws = true) // Construct a set that contains all of the attributes that we need to evaluate the // ordering. - val requiredAttributes = AttributeSet(newOrdering.filter(_.resolved)) + + val requiredAttributes = AttributeSet(newOrdering).filter(_.resolved) + --- End diff -- remove these 2 extra blank lines please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148342508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43778/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148342506 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148342359 [Test build #43778 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43778/console) for PR 9123 at commit [`6f97fb7`](https://github.com/apache/spark/commit/6f97fb7c9fce2920641e5b9dc63b2830e8037702). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148335402 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43776/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148335401 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148335260 [Test build #43776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43776/console) for PR 9123 at commit [`ac78af1`](https://github.com/apache/spark/commit/ac78af14508ac02a74202de564253b927f06c237). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148314896 [Test build #43778 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43778/consoleFull) for PR 9123 at commit [`6f97fb7`](https://github.com/apache/spark/commit/6f97fb7c9fce2920641e5b9dc63b2830e8037702). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148314571 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148314559 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148314200 @cloud-fan Hi Wenchen, i have fixed the code based on your comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/9123#discussion_r42092689
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
---
@@ -135,4 +135,21 @@ class AnalysisSuite extends AnalysisTest {
plan = testRelation.select(CreateStructUnsafe(Seq(a, (a +
1).as("a+1"))).as("col"))
checkAnalysis(plan, plan)
}
+
+ test("SPARK-10534: resolve attribute references in order by clause") {
+
+val a = testRelation2.output.head
+val c = testRelation2.output.toArray.apply(2)
+
+val sortProjected = Floor(Cast(Floor(c), DoubleType))
+val projected = Alias(a, "a")()
+val plan = testRelation2.select(a).orderBy(SortOrder(Floor(Floor(c)),
Ascending))
+
+val expected =
+ Project(Seq(a),
+Sort(Seq(SortOrder(sortProjected, Ascending)), true,
+ Project(Seq(a, c), testRelation2)))
+checkAnalysis(plan, expected)
+
+ }
--- End diff --
Thanks a LOT. I will make the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user dilipbiswal commented on a diff in the pull request:
https://github.com/apache/spark/pull/9123#discussion_r42092649
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -482,7 +482,12 @@ class Analyzer(
val newOrdering = resolveSortOrders(ordering, grandchild, throws =
true)
// Construct a set that contains all of the attributes that we need
to evaluate the
// ordering.
- val requiredAttributes = AttributeSet(newOrdering.filter(_.resolved))
+
+ val resolvedAttributes =
+newOrdering.flatMap(_.collect {case a : AttributeReference if
a.resolved => a})
+
+ val requiredAttributes = AttributeSet(resolvedAttributes)
--- End diff --
Thanks. Much simpler :-). Will make the change and test..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148307180 [Test build #43776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43776/consoleFull) for PR 9123 at commit [`ac78af1`](https://github.com/apache/spark/commit/ac78af14508ac02a74202de564253b927f06c237). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9123#discussion_r42091740
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -482,7 +482,12 @@ class Analyzer(
val newOrdering = resolveSortOrders(ordering, grandchild, throws =
true)
// Construct a set that contains all of the attributes that we need
to evaluate the
// ordering.
- val requiredAttributes = AttributeSet(newOrdering.filter(_.resolved))
+
+ val resolvedAttributes =
+newOrdering.flatMap(_.collect {case a : AttributeReference if
a.resolved => a})
+
+ val requiredAttributes = AttributeSet(resolvedAttributes)
--- End diff --
how about
`val requiredAttributes = AttributeSet(newOrdering).filter(_.resolved)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148306682 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148306706 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/9123#issuecomment-148306333
I wanna explain a bit more about this bug.
When we resolve sort ordering, we will use a [special
method](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L429-L447),
which only resolves `UnresolvedAttributes` and `UnresolvedExtractValue`.
However, for something like `Floor('a)`, even the `'a` is resolved, the floor
expression may still being unresolved as data type mismatch(for example, `'a`
is string type and `Floor` need double type), thus can't pass [this
filter](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L485),
and we can't push down this missing attribute `'a`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148306360 ok to test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9123#discussion_r42089770
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
---
@@ -135,4 +135,21 @@ class AnalysisSuite extends AnalysisTest {
plan = testRelation.select(CreateStructUnsafe(Seq(a, (a +
1).as("a+1"))).as("col"))
checkAnalysis(plan, plan)
}
+
+ test("SPARK-10534: resolve attribute references in order by clause") {
+
+val a = testRelation2.output.head
+val c = testRelation2.output.toArray.apply(2)
+
+val sortProjected = Floor(Cast(Floor(c), DoubleType))
+val projected = Alias(a, "a")()
+val plan = testRelation2.select(a).orderBy(SortOrder(Floor(Floor(c)),
Ascending))
+
+val expected =
+ Project(Seq(a),
+Sort(Seq(SortOrder(sortProjected, Ascending)), true,
+ Project(Seq(a, c), testRelation2)))
+checkAnalysis(plan, expected)
+
+ }
--- End diff --
the test case can be simplified to:
```
val a = testRelation2.output(0)
val c = testRelation2.output(2)
val plan = testRelation2.select(c).orderBy(Floor(a).asc)
val expected = testRelation2.select(c,
a).orderBy(Floor(a.cast(DoubleType)).asc).select(c)
checkAnalysis(plan, expected)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148220968 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
Github user dilipbiswal commented on the pull request: https://github.com/apache/spark/pull/9123#issuecomment-148220723 @cloud-fan Can you please take a look. Thanks .. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10534][SQL] ORDER BY clause allows only...
GitHub user dilipbiswal opened a pull request: https://github.com/apache/spark/pull/9123 [SPARK-10534][SQL] ORDER BY clause allows only columns that are present in S⦠Find out the missing attributes by recursively looking at the sort order expression and rest of the code takes care of projecting them out. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dilipbiswal/spark SPARK-10534 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9123.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9123 commit ac78af14508ac02a74202de564253b927f06c237 Author: Dilip Biswal Date: 2015-10-14T21:44:10Z [SPARK-10534] ORDER BY clause allows only columns that are present in SELECT statement --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
