[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9011#discussion_r41773979
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -833,6 +833,33 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3)))
}
+ test("window function: refer column in inner select block") {
+val data = Seq(
+ WindowData(1, "a", 5),
+ WindowData(2, "a", 6),
+ WindowData(3, "b", 7),
+ WindowData(4, "b", 8),
+ WindowData(5, "c", 9),
+ WindowData(6, "c", 10)
+)
+sparkContext.parallelize(data).toDF().registerTempTable("windowData")
--- End diff --
Ah, thanks for suggesting it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9011 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9011#discussion_r41773288
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -833,6 +833,33 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3)))
}
+ test("window function: refer column in inner select block") {
+val data = Seq(
+ WindowData(1, "a", 5),
+ WindowData(2, "a", 6),
+ WindowData(3, "b", 7),
+ WindowData(4, "b", 8),
+ WindowData(5, "c", 9),
+ WindowData(6, "c", 10)
+)
+sparkContext.parallelize(data).toDF().registerTempTable("windowData")
--- End diff --
btw, you can use `withTempTable` in future to let our test infra
automatically drop the temp table created at here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147449256 Merging to master and 1.5 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147449167 LGTM. @viirya Thank you for fixing this bug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147293397 cc @yhuai to take another look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147212401 ping @cloud-fan Any other comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147028680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147028681 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43507/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147028653 [Test build #43507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43507/console) for PR 9011 at commit [`23f106d`](https://github.com/apache/spark/commit/23f106d25fdf3cd01117c8d5fc69cd6d2b8496d1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147020686 [Test build #43507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43507/consoleFull) for PR 9011 at commit [`23f106d`](https://github.com/apache/spark/commit/23f106d25fdf3cd01117c8d5fc69cd6d2b8496d1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147020290 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147020283 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9011#discussion_r41691082
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -831,6 +838,13 @@ class Analyzer(
val withName = Alias(agg, s"_w${extractedExprBuffer.length}")()
extractedExprBuffer += withName
withName.toAttribute
+
+ case ne: Alias if hasWindowFunction(ne) && !hasAggregation(ne) =>
+ne.children.map(_.transform {
+ case e: NamedExpression => extractExpr(e)
+})
+ne
+
--- End diff --
Thanks. I've updated it according to your suggestion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/9011#discussion_r41666551
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -831,6 +838,13 @@ class Analyzer(
val withName = Alias(agg, s"_w${extractedExprBuffer.length}")()
extractedExprBuffer += withName
withName.toAttribute
+
+ case ne: Alias if hasWindowFunction(ne) && !hasAggregation(ne) =>
+ne.children.map(_.transform {
+ case e: NamedExpression => extractExpr(e)
+})
+ne
+
--- End diff --
A one-line change can fix this and pass your test: `case attr: Attribute =>
extractExpr(attr)`
I think the missing case is `Attribute`, not `Alias`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146932670 @cloud-fan Thanks for suggesting. I've updated it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146677593 Hi @viirya , can you add more description about the reason of this bug? e.g. why does this bug exist and how do you handle it, so that other people can understand your fix easily. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146438339 cc @cloud-fan can you take a look at this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146313115 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146312905 [Test build #43331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43331/console) for PR 9011 at commit [`15b40ee`](https://github.com/apache/spark/commit/15b40eea444f69a08114edd3c82fb0998e4edab3). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class AFTSurvivalRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol,` * `class AFTSurvivalRegressionModel(JavaModel):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146313121 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43331/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146283702 [Test build #43331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43331/consoleFull) for PR 9011 at commit [`15b40ee`](https://github.com/apache/spark/commit/15b40eea444f69a08114edd3c82fb0998e4edab3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146283043 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-146283006 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/9011 [SPARK-10960][SQL] SQL with windowing function should be able to refer column in inner select JIRA: https://issues.apache.org/jira/browse/SPARK-10960 Currently, when accessing a column in inner select from a select with window function, `AnalysisException` will be thrown. This patch fixes it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 fix-window-inner-column Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9011.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9011 commit 15b40eea444f69a08114edd3c82fb0998e4edab3 Author: Liang-Chi Hsieh Date: 2015-10-07T18:11:27Z SQL with windowing function should be able to refer column in inner select block. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
