https://issues.apache.org/jira/browse/SPARK-17668
On Mon, Sep 26, 2016 at 3:40 PM, Koert Kuipers <ko...@tresata.com> wrote:
> ok will create jira
>
> On Mon, Sep 26, 2016 at 3:27 PM, Michael Armbrust <mich...@databricks.com>
> wrote:
>
>> I agree this should work. We just haven't finished killing the old
>> reflection based conversion logic now that we have more powerful/efficient
>> encoders. Please open a JIRA.
>>
>> On Sun, Sep 25, 2016 at 2:41 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> after having gotten used to have case classes represent complex
>>> structures in Datasets, i am surprised to find out that when i work in
>>> DataFrames with udfs no such magic exists, and i have to fall back to
>>> manipulating Row objects, which is error prone and somewhat ugly.
>>>
>>> for example:
>>> case class Person(name: String, age: Int)
>>>
>>> val df = Seq((Person("john", 33), 5), (Person("mike", 30),
>>> 6)).toDF("person", "id")
>>> val df1 = df.withColumn("person", udf({ (p: Person) => p.copy(age =
>>> p.age + 1) }).apply(col("person")))
>>> df1.printSchema
>>> df1.show
>>>
>>> leads to:
>>> java.lang.ClassCastException: org.apache.spark.sql.catalyst.
>>> expressions.GenericRowWithSchema cannot be cast to Person
>>>
>>
>>
>
