[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia commented on SPARK-14927: - [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: ``` scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ ``` I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: ``` create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') ``` Now we insert data into table: ``` scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) ``` I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: ``` model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc") ``` Or even if I persist it as a ORC file: ``` scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc_asTable") ``` I get the ClassCastException: ``` Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more ``` If I persist the DataFrame as a table with Parquet: ``` scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows ``` >From Hive I can see the table though it looks empty (I know that Spark told me >it was stored in a Spark SQL specific format): ``
[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia edited comment on SPARK-14927 at 10/27/16 6:46 AM: --- [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: {code:xml} scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ {code} I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: ``` create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') ``` Now we insert data into table: ``` scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) ``` I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: ``` model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc") ``` Or even if I persist it as a ORC file: ``` scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc_asTable") ``` I get the ClassCastException: ``` Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more ``` If I persist the DataFrame as a table with Parquet: ``` scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows ``` >From Hive I can see the table though it looks empty (I know that Spar
[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia edited comment on SPARK-14927 at 10/27/16 6:46 AM: --- [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: {code:scala} scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ {code} I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: ``` create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') ``` Now we insert data into table: ``` scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) ``` I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: ``` model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc") ``` Or even if I persist it as a ORC file: ``` scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc_asTable") ``` I get the ClassCastException: ``` Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more ``` If I persist the DataFrame as a table with Parquet: ``` scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows ``` >From Hive I can see the table though it looks empty (I know that Sp
[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia edited comment on SPARK-14927 at 10/27/16 6:50 AM: --- [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: {code:scala} scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ {code} I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: {code:sql} create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') {code} Now we insert data into table: {code:scala} scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) {code} I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: {code:scala} model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc") {code} Or even if I persist it as a ORC file: {code:scala} scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc_asTable") {code} I get the ClassCastException: {code:scala} Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more {code} If I persist the DataFrame as a table with Parquet: {code:scala} scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows {code}
[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia edited comment on SPARK-14927 at 10/27/16 6:51 AM: --- [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: {code:scala} scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ {code} I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: {code:sql} create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') {code} Now we insert data into table: {code:scala} scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) {code} I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: {code:scala} model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc_asTable") {code} Or even if I persist it as a ORC file: {code:scala} scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc") {code} I get the ClassCastException: {code:scala} Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more {code} If I persist the DataFrame as a table with Parquet: {code:scala} scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows {code}
[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia edited comment on SPARK-14927 at 10/27/16 6:55 AM: --- [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: {code:scala} scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ {code} I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: {code:sql} create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') {code} Now we insert data into table: {code:scala} scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) {code} I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: {code:scala} model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc_asTable") {code} Or even if I persist it as a ORC file: {code:scala} scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc") {code} I get the ClassCastException: {code:scala} Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more {code} If I persist the DataFrame as a table with Parquet: {code:scala} scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows {code}
[jira] [Comment Edited] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610900#comment-15610900 ] Raul Saez Tapia edited comment on SPARK-14927 at 10/27/16 7:36 AM: --- [~xwu0226] for me is working fine your example with Spark 1.6.1. However it is not working when we use UDT. My DataFrame shows: {code} scala> model_date.toDF.show +++ |date| model| +++ |20160610|[aa.bb.spark.types.PersonWrapper@8542...| |20160610|[aa.bb.spark.types.PersonWrapper@8831..| ... ... +++ {code} I have created the table with some specific properties so I can say how is defined the table and how to parse from PersonType UDT to table schema: {code:sql} create table model_orc (`model` struct>) PARTITIONED BY (`date` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ('path'='hdfs:///user/raulsaez/model_orc') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs:///user/raulsaez/model_orc' TBLPROPERTIES('spark.sql.sources.schema.numParts'='1','spark.sql.sources.schema.part.0'='{ \"type\":\"struct\",\"fields\":[{ \"name\":\"personWrapper\",\"type\":{ \"type\":\"udt\",\"class\":\"aa.bb.spark.types.PersonType\",\"pyClass\":null,\"sqlType\":{ \"type\":\"struct\",\"fields\":[{ \"name\":\"id\",\"type\": \"integer\",\"nullable\":true,\"metadata\":{} } ,{ \"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{} }] } },\"nullable\":true,\"metadata\":{} }] }') {code} Now we insert data into table: {code} scala> hiveContext.sql("insert into model_orc partition(date=20160610) select model,date from dfJune") org.apache.spark.sql.AnalysisException: cannot resolve 'cast(model as struct>>)' due to data type mismatch: cannot cast StructType(StructField(personWrapper,,true)),true) to StructType(StructField(person,StructType(StructField(id,IntegerType,true),StructField(name,StringType,true),),true) {code} I have the same issue with both Parquet and ORC. And if I persist the DataFrame as a table with ORC: {code} model_date.toDF.write.format("orc").partitionBy("date").saveAsTable("model_orc_asTable") {code} Or even if I persist it as a ORC file: {code} scala> model_date.toDF.write.mode(SaveMode.Append).format("orc").partitionBy("date").save("model_orc") {code} I get the ClassCastException: {code} Caused by: java.lang.ClassCastException: aa.bb.spark.types.PersonType cannot be cast to org.apache.spark.sql.types.StructType at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:557) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:590) at org.apache.spark.sql.hive.HiveInspectors$$anonfun$wrap$1.apply(HiveInspectors.scala:589) at org.apache.spark.sql.catalyst.util.ArrayData.foreach(ArrayData.scala:135) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:589) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.HiveInspectors$class.wrap(HiveInspectors.scala:568) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrap(OrcRelation.scala:66) at org.apache.spark.sql.hive.orc.OrcOutputWriter.wrapOrcStruct(OrcRelation.scala:128) at org.apache.spark.sql.hive.orc.OrcOutputWriter.writeInternal(OrcRelation.scala:139) at org.apache.spark.sql.execution.datasources.DynamicPartitionWriterContainer.writeRows(WriterContainer.scala:358) ... 8 more {code} If I persist the DataFrame as a table with Parquet: {code} scala> model_date.toDF.write.mode(SaveMode.Append).format("parquet").partitionBy("date").saveAsTable("model_parquet_asTable") 16/10/27 09:39:24 WARN HiveContext$$anon$2: Persisting partitioned data source relation `model_parquet_asTable` into Hive metastore in Spark SQL specific format, which is NOT compatible with Hive. Input path(s): hdfs://dev-nameservice/apps/hive/warehouse/model_parquet_astable ... ... ... scala> hiveContext.sql("select * from model_parquet_asTable where date=20160610").show +++ | model|date| +++ |[aa.bb.spark.types.PersonWrapper@8542...|20160610| |[aa.bb.spark.types.PersonWrapper@8831...|20160610| |[aa.bb.spark.types.PersonWrapper@3661...|20160610| ... ... ... +++ only showing top 20 rows {code} >From Hive I can see the table th
[jira] [Commented] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106058#comment-16106058 ] Raul Saez Tapia commented on SPARK-14927: - *SOMASUNDARAM SUDALAIMUTHU* I am sorry but I am out of project so I can't check it. > DataFrame. saveAsTable creates RDD partitions but not Hive partitions > - > > Key: SPARK-14927 > URL: https://issues.apache.org/jira/browse/SPARK-14927 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.2, 1.6.1 > Environment: Mac OS X 10.11.4 local >Reporter: Sasha Ovsankin > > This is a followup to > http://stackoverflow.com/questions/31341498/save-spark-dataframe-as-dynamic-partitioned-table-in-hive > . I tried to use suggestions in the answers but couldn't make it to work in > Spark 1.6.1 > I am trying to create partitions programmatically from `DataFrame. Here is > the relevant code (adapted from a Spark test): > hc.setConf("hive.metastore.warehouse.dir", "tmp/tests") > //hc.setConf("hive.exec.dynamic.partition", "true") > //hc.setConf("hive.exec.dynamic.partition.mode", "nonstrict") > hc.sql("create database if not exists tmp") > hc.sql("drop table if exists tmp.partitiontest1") > Seq(2012 -> "a").toDF("year", "val") > .write > .partitionBy("year") > .mode(SaveMode.Append) > .saveAsTable("tmp.partitiontest1") > hc.sql("show partitions tmp.partitiontest1").show > Full file is here: > https://gist.github.com/SashaOv/7c65f03a51c7e8f9c9e018cd42aa4c4a > I get the error that the table is not partitioned: > == > HIVE FAILURE OUTPUT > == > SET hive.support.sql11.reserved.keywords=false > SET hive.metastore.warehouse.dir=tmp/tests > OK > OK > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Table tmp.partitiontest1 is not a > partitioned table > == > It looks like the root cause is that > `org.apache.spark.sql.hive.HiveMetastoreCatalog.newSparkSQLSpecificMetastoreTable` > always creates table with empty partitions. > Any help to move this forward is appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-14927) DataFrame. saveAsTable creates RDD partitions but not Hive partitions
[ https://issues.apache.org/jira/browse/SPARK-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Saez Tapia updated SPARK-14927: Comment: was deleted (was: *SOMASUNDARAM SUDALAIMUTHU* I am sorry but I am out of project so I can't check it.) > DataFrame. saveAsTable creates RDD partitions but not Hive partitions > - > > Key: SPARK-14927 > URL: https://issues.apache.org/jira/browse/SPARK-14927 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.5.2, 1.6.1 > Environment: Mac OS X 10.11.4 local >Reporter: Sasha Ovsankin > > This is a followup to > http://stackoverflow.com/questions/31341498/save-spark-dataframe-as-dynamic-partitioned-table-in-hive > . I tried to use suggestions in the answers but couldn't make it to work in > Spark 1.6.1 > I am trying to create partitions programmatically from `DataFrame. Here is > the relevant code (adapted from a Spark test): > hc.setConf("hive.metastore.warehouse.dir", "tmp/tests") > //hc.setConf("hive.exec.dynamic.partition", "true") > //hc.setConf("hive.exec.dynamic.partition.mode", "nonstrict") > hc.sql("create database if not exists tmp") > hc.sql("drop table if exists tmp.partitiontest1") > Seq(2012 -> "a").toDF("year", "val") > .write > .partitionBy("year") > .mode(SaveMode.Append) > .saveAsTable("tmp.partitiontest1") > hc.sql("show partitions tmp.partitiontest1").show > Full file is here: > https://gist.github.com/SashaOv/7c65f03a51c7e8f9c9e018cd42aa4c4a > I get the error that the table is not partitioned: > == > HIVE FAILURE OUTPUT > == > SET hive.support.sql11.reserved.keywords=false > SET hive.metastore.warehouse.dir=tmp/tests > OK > OK > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Table tmp.partitiontest1 is not a > partitioned table > == > It looks like the root cause is that > `org.apache.spark.sql.hive.HiveMetastoreCatalog.newSparkSQLSpecificMetastoreTable` > always creates table with empty partitions. > Any help to move this forward is appreciated. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org