GitHub user ioana-delaney opened a pull request:
https://github.com/apache/spark/pull/13418
[SPARK-15677][SQL] Query with scalar sub-query in the SELECT list throws
UnsupportedOperationException
## What changes were proposed in this pull request?
Queries with scalar sub-query in the SELECT list run against a local,
in-memory relation throw
UnsupportedOperationException exception.
Problem repro:
```SQL
scala> Seq((1, 1), (2, 2)).toDF("c1", "c2").createOrReplaceTempView("t1")
scala> Seq((1, 1), (2, 2)).toDF("c1", "c2").createOrReplaceTempView("t2")
scala> sql("select (select min(c1) from t2) from t1").show()
java.lang.UnsupportedOperationException: Cannot evaluate expression:
scalar-subquery#62 []
at
org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:215)
at
org.apache.spark.sql.catalyst.expressions.ScalarSubquery.eval(subquery.scala:62)
at
org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:142)
at
org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:45)
at
org.apache.spark.sql.catalyst.expressions.InterpretedProjection.apply(Projection.scala:29)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at
org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation$$anonfun$apply$37.applyOrElse(Optimizer.scala:1473)
```
The problem is specific to local, in memory relations. It is caused by rule
ConvertToLocalRelation, which attempts to push down
a scalar-subquery expression to the local tables.
The solution prevents the rule to apply if Project references scalar
subqueries.
## How was this patch tested?
Added regression tests to SubquerySuite.scala
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ioana-delaney/spark scalarSubV2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13418.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13418
----
commit faded1df9de00185e02adcffe09a473fc46cbf14
Author: Ioana Delaney <ioanamdela...@gmail.com>
Date: 2016-05-31T18:53:48Z
[SPARK-15677] Query with scalar sub-query in the SELECT list throws
UnsupportedOperationException.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
