[ https://issues.apache.org/jira/browse/SPARK-19716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan updated SPARK-19716: -------------------------------- Description: if we have a DataFrame with schema {{a: int, b: int, c: int}}, and convert it to Dataset with {{case class Data(a: Int, c: Int)}}, it works and we will extract the `a` and `c` columns to build the Data. However, if the struct is inside array, e.g. schema is {{arr: array<struct<a: int, b: int, c: int>>}}, and we wanna convert it to Dataset with {{case class ComplexData(arr: Seq[Data])}}, we will fail. The reason is, to allow compatible types, e.g. convert {{a: int}} to {{case class A(a: Long)}}, we will add cast for each field, except struct type field, because struct type is flexible, the number of columns can mismatch. We should probably also skip cast for array and map type. was: if we have a DataFrame with schema {{a: int, b: int, c: int}}, and convert it to Dataset with {{case class Data(a: Int, c: Int)}}, it works and we will extract the `a` and `c` columns to build the Data. However, if the struct is inside array, e.g. schema is {{arr: array<struct<a: int, b: int, c: int>>}}, and we wanna convert it to Dataset with {{case class ComplexData(arr: Seq[Data])}}, we will fail. The reason is, we will add cast for each field, except struct type field, because struct type is flexible, the number of columns can mismatch. We should probably also skip cast for array and map type. > Dataset should allow by-name resolution for struct type elements in array > ------------------------------------------------------------------------- > > Key: SPARK-19716 > URL: https://issues.apache.org/jira/browse/SPARK-19716 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 2.2.0 > Reporter: Wenchen Fan > > if we have a DataFrame with schema {{a: int, b: int, c: int}}, and convert it > to Dataset with {{case class Data(a: Int, c: Int)}}, it works and we will > extract the `a` and `c` columns to build the Data. > However, if the struct is inside array, e.g. schema is {{arr: array<struct<a: > int, b: int, c: int>>}}, and we wanna convert it to Dataset with {{case class > ComplexData(arr: Seq[Data])}}, we will fail. The reason is, to allow > compatible types, e.g. convert {{a: int}} to {{case class A(a: Long)}}, we > will add cast for each field, except struct type field, because struct type > is flexible, the number of columns can mismatch. We should probably also skip > cast for array and map type. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org