I think this is a bug in our date handling that should be fixed in Spark 1.4.
On Wed, May 13, 2015 at 8:23 AM, ayan guha <[email protected]> wrote: > Your stack trace says it can't convert date to integer. You sure about > column positions? > On 13 May 2015 21:32, "Ishwardeep Singh" <[email protected]> > wrote: > >> Hi , >> >> I am using Spark SQL 1.3.1. >> >> I have created a dataFrame using jdbc data source and am using >> saveAsTable() >> method but got the following 2 exceptions: >> >> java.lang.RuntimeException: Unsupported datatype DecimalType() >> at scala.sys.package$.error(package.scala:27) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$fromDataType$2.apply(ParquetTypes.scala:372) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$fromDataType$2.apply(ParquetTypes.scala:316) >> at scala.Option.getOrElse(Option.scala:120) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:315) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$4.apply(ParquetTypes.scala:395) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$4.apply(ParquetTypes.scala:394) >> at >> >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at >> >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at scala.collection.immutable.List.foreach(List.scala:318) >> at >> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetTypes.scala:393) >> at >> >> org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:440) >> at >> >> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.prepareMetadata(newParquet.scala:260) >> at >> >> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:276) >> at >> >> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:269) >> at >> >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at >> >> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >> at scala.collection.immutable.List.foreach(List.scala:318) >> at >> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >> at scala.collection.AbstractTraversable.map(Traversable.scala:105) >> at >> >> org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.refresh(newParquet.scala:269) >> at >> org.apache.spark.sql.parquet.ParquetRelation2.<init>(newParquet.scala:391) >> at >> >> org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:98) >> at >> >> org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:128) >> at >> org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:240) >> at >> >> org.apache.spark.sql.hive.execution.CreateMetastoreDataSourceAsSelect.run(commands.scala:218) >> at >> >> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54) >> at >> >> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54) >> at >> org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64) >> at >> >> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099) >> at >> >> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099) >> at >> org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1121) >> at >> org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1071) >> at >> org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1037) >> at >> org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1015) >> >> java.lang.ClassCastException: java.sql.Date cannot be cast to >> java.lang.Integer >> at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:106) >> at >> >> org.apache.spark.sql.parquet.RowWriteSupport.writePrimitive(ParquetTableSupport.scala:215) >> at >> >> org.apache.spark.sql.parquet.RowWriteSupport.writeValue(ParquetTableSupport.scala:192) >> at >> >> org.apache.spark.sql.parquet.RowWriteSupport.write(ParquetTableSupport.scala:171) >> at >> >> org.apache.spark.sql.parquet.RowWriteSupport.write(ParquetTableSupport.scala:134) >> at >> >> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:120) >> at >> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:81) >> at >> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:37) >> at >> org.apache.spark.sql.parquet.ParquetRelation2.org >> $apache$spark$sql$parquet$ParquetRelation2$$writeShard$1(newParquet.scala:671) >> at >> >> org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689) >> at >> >> org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$insert$2.apply(newParquet.scala:689) >> at >> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >> at org.apache.spark.scheduler.Task.run(Task.scala:64) >> at >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >> at >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:722) >> >> Earlier I was using Spark SQL 1.3.0 and was getting some other exception >> so >> I upgraded to 1.3.1 but got a different exception. >> >> Any help would be appreciated >> Regards, >> Ishwardeep >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-1-3-1-data-frame-saveAsTable-returns-exception-tp22867.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >>
