[ https://issues.apache.org/jira/browse/SPARK-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yin Huai updated SPARK-4768: ---------------------------- Fix Version/s: 1.3.0 > Add Support For Impala Encoded Timestamp (INT96) > ------------------------------------------------ > > Key: SPARK-4768 > URL: https://issues.apache.org/jira/browse/SPARK-4768 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Pat McDonough > Assignee: Yin Huai > Priority: Blocker > Fix For: 1.3.0 > > Attachments: 5e4481a02f951e29-651ee94ed14560bf_922627129_data.0.parq, > string_timestamp.gz > > > Impala is using INT96 for timestamps. Spark SQL should be able to read this > data despite the fact that it is not part of the spec. > Perhaps adding a flag to act like impala when reading parquet (like we do for > strings already) would be useful. > Here's an example of the error you might see: > {code} > Caused by: java.lang.RuntimeException: Potential loss of precision: cannot > convert INT96 > at scala.sys.package$.error(package.scala:27) > at > org.apache.spark.sql.parquet.ParquetTypesConverter$.toPrimitiveDataType(ParquetTypes.scala:61) > at > org.apache.spark.sql.parquet.ParquetTypesConverter$.toDataType(ParquetTypes.scala:113) > at > org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:314) > at > org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:311) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at > org.apache.spark.sql.parquet.ParquetTypesConverter$.convertToAttributes(ParquetTypes.scala:310) > at > org.apache.spark.sql.parquet.ParquetTypesConverter$.readSchemaFromFile(ParquetTypes.scala:441) > at > org.apache.spark.sql.parquet.ParquetRelation.<init>(ParquetRelation.scala:66) > at org.apache.spark.sql.SQLContext.parquetFile(SQLContext.scala:141) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org