Re: Dataframe fails to save to MySQL table in spark app, but succeeds in spark shell

2017-01-26 Thread Suresh Thalamati
I notice columns are quoted wit double quotes in the error message ('"user","age","state”)) . By chance did you override the MySQL JDBC dialect, default MySQL identifiers are quoted with ` override def quoteIdentifier(colName: String): String = { s"`$colName`" } Just wondering if the error

Re: Dataframe fails to save to MySQL table in spark app, but succeeds in spark shell

2017-01-26 Thread Didac Gil
Are you sure that “age” is a numeric field? Even numeric, you could pass the “44” between quotes: INSERT into your_table ("user","age","state") VALUES ('user3’,’44','CT’) Are you sure there are no more fields that are specified as NOT NULL, and that you did not provide a value (besides user,

Re: Dataframe fails to save to MySQL table in spark app, but succeeds in spark shell

2017-01-25 Thread Takeshi Yamamuro
Hi, I'm not familiar with MyQL though, your MySQL got the same query in the two patterns? Have you checked MySQL logs? // maropu On Thu, Jan 26, 2017 at 12:42 PM, Xuan Dzung Doan < doanxuand...@yahoo.com.invalid> wrote: > Hi, > > Spark version 2.1.0 > MySQL community server version 5.7.17 >

Dataframe fails to save to MySQL table in spark app, but succeeds in spark shell

2017-01-25 Thread Xuan Dzung Doan
Hi, Spark version 2.1.0 MySQL community server version 5.7.17 MySQL Connector Java 5.1.40 I need to save a dataframe to a MySQL table. In spark shell, the following statement succeeds: scala> df.write.mode(SaveMode.Append).format("jdbc").option("url",