Mathew Wicks created SPARK-28032:
------------------------------------
Summary: DataFrame.saveAsTable( in AVRO format with Timestamps
create bad Hive tables
Key: SPARK-28032
URL: https://issues.apache.org/jira/browse/SPARK-28032
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.3
Environment: Spark 2.4.3
Hive 1.1.0
Reporter: Mathew Wicks
I am not sure if it's my very old version of Hive (1.1.0), but when I use the
following code, I end up with a table which Spark can read, but Hive cannot.
That is to say, when writing AVRO format tables, they cannot be read in Hive if
they contain timestamp types.
*Hive error:*
{code:java}
Error while compiling statement: FAILED: UnsupportedOperationException
timestamp is not supported.
{code}
*Spark Code:*
{code:java}
import java.sql.Timestamp
import spark.implicits._
val currentTime = new Timestamp(System.currentTimeMillis())
val df = Seq(
(currentTime)
).toDF()
df.write.mode("overwrite").format("avro").saveAsTable("database.table_name")
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
