[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11198 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186687592 LGTM, merging into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186660111 @davies Could you review it again? I start working on another PR for pushing down AggregateExpression into Aggregate. That might introduce conflicts with this one. If possible, could we complete this at first? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186122845 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51530/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186122844 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186122497 **[Test build #51530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51530/consoleFull)** for PR 11198 at commit [`63ba356`](https://github.com/apache/spark/commit/63ba3565ddf96c186a13a04de83fca5cfa816f54). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186102602 LGTM pending test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-186096228 **[Test build #51530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51530/consoleFull)** for PR 11198 at commit [`63ba356`](https://github.com/apache/spark/commit/63ba3565ddf96c186a13a04de83fca5cfa816f54). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r53427826
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -927,6 +927,33 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3, i._4)))
}
+ test("Sorting columns are not in Generate") {
+withTempTable("data") {
+ val rdd = sparkContext.makeRDD((1 to 5)
+.map(i => s"""{"a":[$i, ${i + 1}], "b":"$i", "c":"${10 - i}",
"d":[$i, ${i + 5}]}"""))
--- End diff --
Thanks! Let me do the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r53425790
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -927,6 +927,33 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3, i._4)))
}
+ test("Sorting columns are not in Generate") {
+withTempTable("data") {
+ val rdd = sparkContext.makeRDD((1 to 5)
+.map(i => s"""{"a":[$i, ${i + 1}], "b":"$i", "c":"${10 - i}",
"d":[$i, ${i + 5}]}"""))
--- End diff --
This is hard to read, how about
```
df = sqlContext.range(5).select(array($"id", $"id" + 1).as("a"),
$"id".as("b"), ...)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184900495 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51382/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184900493 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184900143 **[Test build #51382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51382/consoleFull)** for PR 11198 at commit [`24ba670`](https://github.com/apache/spark/commit/24ba67076b4977cdfa7c63a42cf0715d5f5074bd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r53072823
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -638,6 +645,17 @@ class Analyzer(
}
val newAggregateExpressions = a.aggregateExpressions ++
missingAttrs
a.copy(aggregateExpressions = newAggregateExpressions)
+case g: Generate =>
+ if (g.outer && !g.join) {
--- End diff --
Sure, let me remove it and revert the correction back. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184867406 **[Test build #51382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51382/consoleFull)** for PR 11198 at commit [`24ba670`](https://github.com/apache/spark/commit/24ba67076b4977cdfa7c63a42cf0715d5f5074bd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r53072307
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -638,6 +645,17 @@ class Analyzer(
}
val newAggregateExpressions = a.aggregateExpressions ++
missingAttrs
a.copy(aggregateExpressions = newAggregateExpressions)
+case g: Generate =>
+ if (g.outer && !g.join) {
--- End diff --
If `join` is false, `outer` is ignored, so we don't need to correct it,
just remove this branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184851646 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51372/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184851642 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184850918 **[Test build #51372 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51372/consoleFull)** for PR 11198 at commit [`07de4bc`](https://github.com/apache/spark/commit/07de4bcaafdad13fa5528ad280781247aa40f63e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184802672 **[Test build #51372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51372/consoleFull)** for PR 11198 at commit [`07de4bc`](https://github.com/apache/spark/commit/07de4bcaafdad13fa5528ad280781247aa40f63e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184799192 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184570191 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51344/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184570186 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184569973 **[Test build #51344 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51344/consoleFull)** for PR 11198 at commit [`07de4bc`](https://github.com/apache/spark/commit/07de4bcaafdad13fa5528ad280781247aa40f63e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184549831 **[Test build #51344 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51344/consoleFull)** for PR 11198 at commit [`07de4bc`](https://github.com/apache/spark/commit/07de4bcaafdad13fa5528ad280781247aa40f63e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184501312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51332/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184501309 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184501104 **[Test build #51332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51332/consoleFull)** for PR 11198 at commit [`49a2d6e`](https://github.com/apache/spark/commit/49a2d6e8c153609901ed79035cd1abe236f1d39c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r52960865
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -638,6 +645,17 @@ class Analyzer(
}
val newAggregateExpressions = a.aggregateExpressions ++
missingAttrs
a.copy(aggregateExpressions = newAggregateExpressions)
+case g: Generate =>
+ if (g.outer && !g.join) {
--- End diff --
```
case p @ Project(_, g: Generate) if g.join &&
p.references.subsetOf(g.generatedSet) =>
p.copy(child = g.copy(join = false))
```
The optimizer changes the `join` value but did not change the `outer`
value. uh... Let me correct it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184471941 **[Test build #51332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51332/consoleFull)** for PR 11198 at commit [`49a2d6e`](https://github.com/apache/spark/commit/49a2d6e8c153609901ed79035cd1abe236f1d39c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r52960702
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -638,6 +645,17 @@ class Analyzer(
}
val newAggregateExpressions = a.aggregateExpressions ++
missingAttrs
a.copy(aggregateExpressions = newAggregateExpressions)
+case g: Generate =>
+ if (g.outer && !g.join) {
--- End diff --
Yeah, true. However, if that happens, we will see the errors. Thus, I put
it for ensuring it will never happen. Or should I just put `assert`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r52960600
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -638,6 +645,17 @@ class Analyzer(
}
val newAggregateExpressions = a.aggregateExpressions ++
missingAttrs
a.copy(aggregateExpressions = newAggregateExpressions)
+case g: Generate =>
+ if (g.outer && !g.join) {
--- End diff --
I think `outer` can't be `true` if `join` is false.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184468332 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184466102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51325/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184466101 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184465977 **[Test build #51325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51325/consoleFull)** for PR 11198 at commit [`49a2d6e`](https://github.com/apache/spark/commit/49a2d6e8c153609901ed79035cd1abe236f1d39c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-184445898 **[Test build #51325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51325/consoleFull)** for PR 11198 at commit [`49a2d6e`](https://github.com/apache/spark/commit/49a2d6e8c153609901ed79035cd1abe236f1d39c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-18684 @davies Besides the issue we discussed above, the latest commit also resolves another issue. So far, if we could issue an error message like ```Can't add $missingAttrs to $other``` when we are unable to resolve the missing sort-by columns. However, this is misleading. Users might not be able to understand how to correct the issue. Thus, after the fix, now, we can get a better message like ```cannot resolve 'b' in given input columns [a, c, a3] ```. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r52944115
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -655,6 +655,8 @@ class Analyzer(
resolved
} else {
plan match {
+ case g: Generate =>
+if (g.join) resolveExpressionRecursively(resolved, g.child)
else resolved
--- End diff --
Sure, will do it. Thanks!
Will exclude the cases when `g.outer==false`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11198#discussion_r52861656
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -655,6 +655,8 @@ class Analyzer(
resolved
} else {
plan match {
+ case g: Generate =>
+if (g.join) resolveExpressionRecursively(resolved, g.child)
else resolved
--- End diff --
The `join` is not specified by user explicitly. If we allow `sort` refer to
the column that is not in project list, I think we should also allow it even
`join` is false, it means we need to change `join` from `false` to `true` to
make it works.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-183808318 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51251/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-183808315 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-183807835 **[Test build #51251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51251/consoleFull)** for PR 11198 at commit [`c74e7b9`](https://github.com/apache/spark/commit/c74e7b911abc1e420f9d2bd775c7b02fb6cf3079). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11198#issuecomment-183782726 **[Test build #51251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51251/consoleFull)** for PR 11198 at commit [`c74e7b9`](https://github.com/apache/spark/commit/c74e7b911abc1e420f9d2bd775c7b02fb6cf3079). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13310] [SQL] Resolve Missing Sorting Co...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/11198
[SPARK-13310] [SQL] Resolve Missing Sorting Columns in Generate
```scala
// case 1: missing sort columns are resolvable if join is true
sql("SELECT explode(a) AS val, b FROM data WHERE b < 2 order by val, c")
// case 2: missing sort columns are not resolvable if join is false. Thus,
issue an error message in this case
sql("SELECT explode(a) AS val FROM data order by val, c")
```
When sort columns are not in `Generate`, we can resolve them when `join` is
equal to `true`. Still trying to add more test cases for the other `UnaryNode`
types.
Could you review the changes? @davies @cloud-fan Thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark missingInSort
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11198.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11198
commit 0f4cbdced54349e5c3af8aa8b3dc509561700fe3
Author: gatorsmile
Date: 2016-02-14T00:37:31Z
resolve missing sorting columns in Generate
commit c74e7b911abc1e420f9d2bd775c7b02fb6cf3079
Author: gatorsmile
Date: 2016-02-14T00:48:41Z
style fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
