GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/21964
[SPARK-24788][SQL] RelationalGroupedDataset.toString with unresolved exprs
should not fail
## What changes were proposed in this pull request?
In the current master, `toString` throws an exception when
`RelationalGroupedDataset` has unresolved expressions;
```
scala> spark.range(0, 10).groupBy("id")
res4: org.apache.spark.sql.RelationalGroupedDataset =
RelationalGroupedDataset: [grouping expressions: [id: bigint], value: [id:
bigint], type: GroupBy]
scala> spark.range(0, 10).groupBy('id)
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to
dataType on unresolved object, tree: 'id
at
org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:105)
at
org.apache.spark.sql.RelationalGroupedDataset$$anonfun$12.apply(RelationalGroupedDataset.scala:474)
at
org.apache.spark.sql.RelationalGroupedDataset$$anonfun$12.apply(RelationalGroupedDataset.scala:473)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.spark.sql.RelationalGroupedDataset.toString(RelationalGroupedDataset.scala:473)
at
scala.runtime.ScalaRunTime$.scala$runtime$ScalaRunTime$$inner$1(ScalaRunTime.scala:332)
at scala.runtime.ScalaRunTime$.stringOf(ScalaRunTime.scala:337)
at scala.runtime.ScalaRunTime$.replStringOf(ScalaRunTime.scala:345)
```
Closes #21752
## How was this patch tested?
Added tests in `DataFrameAggregateSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-24788
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21964.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21964
----
commit 465e7624073016d01ae6d3c5df501bf9b2c6410b
Author: Chris Horn <chorn4033@...>
Date: 2018-07-11T21:25:26Z
SPARK-24788 failing test case
commit e995b0bf2824593532056ff0048e65e8a33e5aad
Author: Chris Horn <chorn4033@...>
Date: 2018-07-11T21:25:54Z
SPARK-24788 fixed UnresolvedException when toString an unresolved grouping
expression
commit 5213635d595f76261a8387e5a5135ebd9bcfa8d9
Author: Chris Horn <chorn4033@...>
Date: 2018-07-13T19:09:35Z
simplify test description; remove whitespace
commit 2e48604ff9aadebc4f7f3f8edeee252722967da9
Author: Chris Horn <chorn4033@...>
Date: 2018-07-13T19:22:07Z
do not use Matchers
commit c4e7490f1762aff5ae5b7126adb9ddd8d987a77d
Author: Takeshi Yamamuro <yamamuro@...>
Date: 2018-08-02T06:20:34Z
Fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
