[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13418#discussion_r65302021
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
---
@@ -84,6 +84,13 @@ object ScalarSubquery {
case _ => false
}.isDefined
}
+
+ def hasScalarSubquery(e: Expression): Boolean = {
+e.find {
--- End diff --
```
e.find(_.isInstanceOf[ScalarSubquery]).isDefined
```
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user ioana-delaney commented on a diff in the pull request:
https://github.com/apache/spark/pull/13418#discussion_r65294821
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1468,7 +1468,8 @@ object DecimalAggregates extends Rule[LogicalPlan] {
*/
object ConvertToLocalRelation extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
-case Project(projectList, LocalRelation(output, data)) =>
+case p @ Project(projectList, LocalRelation(output, data))
+if !p.expressions.exists(ScalarSubquery.hasScalarSubquery) =>
--- End diff --
@davies Thank you for the comment. I am looking into generalizing the
condition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13418 **[Test build #3033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3033/consoleFull)** for PR 13418 at commit [`faded1d`](https://github.com/apache/spark/commit/faded1df9de00185e02adcffe09a473fc46cbf14). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/13418#discussion_r65273025
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -1468,7 +1468,8 @@ object DecimalAggregates extends Rule[LogicalPlan] {
*/
object ConvertToLocalRelation extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
-case Project(projectList, LocalRelation(output, data)) =>
+case p @ Project(projectList, LocalRelation(output, data))
+if !p.expressions.exists(ScalarSubquery.hasScalarSubquery) =>
--- End diff --
Is it more general to check unvaluable expressions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13418 **[Test build #3033 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3033/consoleFull)** for PR 13418 at commit [`faded1d`](https://github.com/apache/spark/commit/faded1df9de00185e02adcffe09a473fc46cbf14). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/13418 LGTM CC @hvanhovell @davies @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13418 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15677][SQL] Query with scalar sub-query in the SE...
GitHub user ioana-delaney opened a pull request:
https://github.com/apache/spark/pull/13418
[SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws
UnsupportedOperationException
## What changes were proposed in this pull request?
Queries with scalar sub-query in the SELECT list run against a local,
in-memory relation throw
UnsupportedOperationException exception.
Problem repro:
```SQL
scala> Seq((1, 1), (2, 2)).toDF("c1", "c2").createOrReplaceTempView("t1")
scala> Seq((1, 1), (2, 2)).toDF("c1", "c2").createOrReplaceTempView("t2")
scala> sql("select (select min(c1) from t2) from t1").show()
java.lang.UnsupportedOperationException: Cannot evaluate expression:
scalar-subquery#62 []
at
org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:215)
at
org.apache.spark.sql.catalyst.expressions.ScalarSubquery.eval(subquery.scala:62)
at
org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:142)
at
org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:45)
at
org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:29)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at
org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$37.applyOrElse(Optimizer.scala:1473)
```
The problem is specific to local, in memory relations. It is caused by rule
ConvertToLocalRelation, which attempts to push down
a scalar-subquery expression to the local tables.
The solution prevents the rule to apply if Project references scalar
subqueries.
## How was this patch tested?
Added regression tests to SubquerySuite.scala
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ioana-delaney/spark scalarSubV2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13418
commit faded1df9de00185e02adcffe09a473fc46cbf14
Author: Ioana Delaney
Date: 2016-05-31T18:53:48Z
[SPARK-15677] Query with scalar sub-query in the SELECT list throws
UnsupportedOperationException.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
