[
https://issues.apache.org/jira/browse/SPARK-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15024369#comment-15024369
]
Siva Gudavalli commented on SPARK-11623:
----------------------------------------
Hi Sean,
At the first place, we should not perform the check at all when we specify
"SaveMode.Append".
SaveMode.Append means we know there is an existing table in the database. And
we want to insert records from the dataframe into an existing table.
In Spark 1.3.1 we have 2 methods i.e.. CreateJdbcTable and InsertIntoJdbc.
They are replaced with write.jdbc() in Spark 1.4.1
When we specify SaveMode.Append we are letting application know that there is a
table in the database which means "tableExists = true". And we do not need to
perform "JdbcUtils.tableExists(conn, table)".
Please let me know if you think differently.
Regards
Shiv
def jdbc(url: String, table: String, connectionProperties: Properties): Unit = {
val conn = JdbcUtils.createConnection(url, connectionProperties)
try {
var tableExists = JdbcUtils.tableExists(conn, table)
if (mode == SaveMode.Ignore && tableExists) {
return
}
if (mode == SaveMode.ErrorIfExists && tableExists) {
sys.error(s"Table $table already exists.")
}
if (mode == SaveMode.Overwrite && tableExists) {
JdbcUtils.dropTable(conn, table)
tableExists = false
}
// Create the table if the table didn't exist.
if (!tableExists) {
val schema = JDBCWriteDetails.schemaString(df, url)
val sql = s"CREATE TABLE $table ($schema)"
conn.prepareStatement(sql).executeUpdate()
}
} finally {
conn.close()
}
JDBCWriteDetails.saveTable(df, url, table, connectionProperties)
}
> Sparksql-1.4.1 DataFrameWrite.jdbc() bug
> ----------------------------------------
>
> Key: SPARK-11623
> URL: https://issues.apache.org/jira/browse/SPARK-11623
> Project: Spark
> Issue Type: Bug
> Components: Java API, Spark Submit, SQL
> Affects Versions: 1.4.1, 1.5.1
> Environment: Spark stand alone cluster
> Reporter: Nguyen Van Nghia
>
> I am running spark-submit in window 8.1 with spark standalone cluster (01
> worker and 01 master), the job throw Exception in DataFrameWrite.jdbc(..)
> scala function.
> We found that the following test:
> var tableExists = JdbcUtils.tableExists(conn, table)
> always return false event if we already created a table.
> That drive the function to do creating table from specified DataFrame and
> the SQL Syntax error for creating table, we locate the SQL execution
> statement hereafter:
> if (!tableExists) {
> val schema = JDBCWriteDetails.schemaString(df, url)
> val sql = s"CREATE TABLE $table ($schema)"
> conn.prepareStatement(sql).executeUpdate() // This execution cause
> sql syntax error
> }
> This happened with spark-1.4.1 and Spark-1.5.1 (our dev environment).
> Please help!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
