aokolnychyi commented on issue #1138:
URL: https://github.com/apache/iceberg/issues/1138#issuecomment-675095986
@gustavoatt @rdblue quick question on int96 representation in different
query engines.
Here is what I see in Spark:
```
val PARQUET_INT96_AS_TIMESTAMP =
buildConf("spark.sql.parquet.int96AsTimestamp")
.doc("Some Parquet-producing systems, in particular Impala, store
Timestamp into INT96. " +
"Spark would also store Timestamp as INT96 because we need to avoid
precision lost of the " +
"nanoseconds field. This flag tells Spark SQL to interpret INT96 data
as a timestamp to " +
"provide compatibility with these systems.")
.booleanConf
.createWithDefault(true)
val PARQUET_INT96_TIMESTAMP_CONVERSION =
buildConf("spark.sql.parquet.int96TimestampConversion")
.doc("This controls whether timestamp adjustments should be applied to
INT96 data when " +
"converting to timestamps, for data written by Impala. This is
necessary because Impala " +
"stores INT96 data with a different timezone offset than Hive &
Spark.")
.booleanConf
.createWithDefault(false)
```
I remember there were changes/plans on Impala side. Does anybody know the
current state?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
