[
https://issues.apache.org/jira/browse/SPARK-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16832314#comment-16832314
]
Terry Moschou commented on SPARK-20193:
---------------------------------------
I am in agreement with [~a.ionescu], and have a need for empty structs, built
dynamically. It is quite confusing when spark can (or seems to) support empty
structs constructed in other (non dynamic) ways.
{code:java}
case class Empty()
val emptyStruct = udf(() => Empty())
val nullEmptyStruct = udf(() => null.asInstanceOf[Empty])
val df1 = spark.range(10).select(emptyStruct() as "a", nullEmptyStruct() as "b")
case class X(a: Empty, b: Empty)
val df2 = Seq(X(Empty(), null)).toDF
val df3 = spark.createDataFrame(
spark.sparkContext.parallelize(Seq(
Row(Row(), null)
)),
StructType(StructField("a", StructType(Nil)) :: StructField("b",
StructType(Nil)) :: Nil)
){code}
> Selecting empty struct causes ExpressionEncoder error.
> ------------------------------------------------------
>
> Key: SPARK-20193
> URL: https://issues.apache.org/jira/browse/SPARK-20193
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, SQL
> Affects Versions: 2.1.0
> Reporter: Adrian Ionescu
> Priority: Minor
>
> {{def struct(cols: Column*): Column}}
> Given the above signature and the lack of any note in the docs saying that a
> struct with no columns is not supported, I would expect the following to work:
> {{spark.range(3).select(col("id"), struct().as("empty_struct")).collect}}
> However, this results in:
> {quote}
> java.lang.AssertionError: assertion failed: each serializer expression should
> contains at least one `BoundReference`
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$$anonfun$11.apply(ExpressionEncoder.scala:240)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$$anonfun$11.apply(ExpressionEncoder.scala:238)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
> at scala.collection.immutable.List.flatMap(List.scala:344)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.<init>(ExpressionEncoder.scala:238)
> at
> org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:63)
> at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
> at
> org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$withPlan(Dataset.scala:2837)
> at org.apache.spark.sql.Dataset.select(Dataset.scala:1131)
> ... 39 elided
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
