[
https://issues.apache.org/jira/browse/SPARK-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-4768:
------------------------------------
Assignee: Yin Huai
> Add Support For Impala Encoded Timestamp (INT96)
> ------------------------------------------------
>
> Key: SPARK-4768
> URL: https://issues.apache.org/jira/browse/SPARK-4768
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Pat McDonough
> Assignee: Yin Huai
> Priority: Blocker
> Attachments: 5e4481a02f951e29-651ee94ed14560bf_922627129_data.0.parq,
> string_timestamp.gz
>
>
> Impala is using INT96 for timestamps. Spark SQL should be able to read this
> data despite the fact that it is not part of the spec.
> Perhaps adding a flag to act like impala when reading parquet (like we do for
> strings already) would be useful.
> Here's an example of the error you might see:
> {code}
> Caused by: java.lang.RuntimeException: Potential loss of precision: cannot
> convert INT96
> at scala.sys.package$.error(package.scala:27)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.toPrimitiveDataType(ParquetTypes.scala:61)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.toDataType(ParquetTypes.scala:113)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:314)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:311)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.convertToAttributes(ParquetTypes.scala:310)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.readSchemaFromFile(ParquetTypes.scala:441)
> at
> org.apache.spark.sql.parquet.ParquetRelation.<init>(ParquetRelation.scala:66)
> at org.apache.spark.sql.SQLContext.parquetFile(SQLContext.scala:141)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
