beliefer opened a new pull request #34647: URL: https://github.com/apache/spark/pull/34647
### What changes were proposed in this pull request? Spark not supports TimestampNTZ type in Hive. ``` [info] Caused by: java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'timestamp_ntz:timestamp' but 'timestamp_ntz' is found.[info] Caused by: java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'timestamp_ntz:timestamp' but 'timestamp_ntz' is found. [info] at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:372) [info] at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:355) [info] at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:416) [info] at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:329) [info] at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:814) [info] at org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters.extractColumnInfo(LazySerDeParameters.java:162)[info] at org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters.<init>(LazySerDeParameters.java:91) [info] at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initialize(LazySimpleSerDe.java:116) [info] at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) [info] at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533) [info] at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:453) [info] at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:440) [info] at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281) [info] at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:199) [info] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:842) ... ``` Hive only providers the timestamp type, so Spark should write both timestamp_ltz and timestamp_ntz as Hive' timestamp. When Spark read schema form Hive, We should restore the timestamp_ntz type. ### Why are the changes needed? The hive 2.3.9 does not have 2 timestamp or a type named timestamp_ntz. FYI, In hive 3.0, the will be a timestamp with local timezone added. ### Does this PR introduce _any_ user-facing change? 'No'. timestamp_ntz is new and not public yet ### How was this patch tested? New test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
