[
https://issues.apache.org/jira/browse/SPARK-19713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895991#comment-15895991
]
Eric Maynard commented on SPARK-19713:
--------------------------------------
In general instead of using `DataFrameWriter.saveAsTable`, I find it's better
to create the table in advance and then insert data into it
`DataFrameWriter.insertInto`. If you use choose to use
`DataFrameWriter.saveAsTable` there is a chance of the folder being created and
the Hive table not being updated, but as a developer you can handle these
errors with `HiveContext.refreshTable` or by using `FileSystem.delete`. I think
this is not an issue.
> saveAsTable
> -----------
>
> Key: SPARK-19713
> URL: https://issues.apache.org/jira/browse/SPARK-19713
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.1
> Reporter: Balaram R Gadiraju
>
> Hi,
> I just observed that when we use dataframe.saveAsTable("table") -- In
> oldversions
> and dataframe.write.saveAsTable("table") -- in the newer versions
> When using the method “df3.saveAsTable("brokentable")” in
> scale code. This creates a folder in hdfs and doesn’t update hive-metastore
> that it plans to create the table. So if anything goes wrong in between the
> folder still exists and hive is not aware of the folder creation. This will
> block the users from creating the table “brokentable” as the folder already
> exists, we can remove the folder using “hadoop fs –rmr
> /data/hive/databases/testdb.db/brokentable”. So below is the workaround
> which will enable to you to continue the development work.
> Current Code:
> val df3 = sqlContext.sql("select * fromtesttable")
> df3.saveAsTable("brokentable")
> THE WORKAROUND:
> By registering the DataFrame as table and then using sql command to load the
> data will resolve the issue. EX:
> val df3 = sqlContext.sql("select * from testtable").registerTempTable("df3")
> sqlContext.sql("CREATE TABLE brokentable AS SELECT * FROM df3")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
