Suresh Thalamati created SPARK-10170: ----------------------------------------
Summary: Writing from data frame into db2 database using jdbc data source api fails with error for string, and boolean column types. Key: SPARK-10170 URL: https://issues.apache.org/jira/browse/SPARK-10170 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.4.1, 1.5.0 Reporter: Suresh Thalamati Repro : -- start spark shell with classpath set to the db2 jdbc driver. SPARK_CLASSPATH=~/myjars/db2jcc.jar ./spark-shell // set connetion properties val properties = new java.util.Properties() properties.setProperty("user" , "user") properties.setProperty("password" , "password") // load the driver. Class.forName("com.ibm.db2.jcc.DB2Driver").newInstance // create data frame with a String type val empdf = sc.parallelize( Array((1,"John"), (2,"Mike"))).toDF("id", "name" ) // write the data frame. this will fail with error. empdf.write.jdbc("jdbc:db2://bdvs150.svl.ibm.com:60000/SAMPLE:retrieveMessagesFromServerOnGetMessage=true;", "emp_data", properties) Error : com.ibm.db2.jcc.am.SqlSyntaxErrorException: TEXT at com.ibm.db2.jcc.am.fd.a(fd.java:679) at com.ibm.db2.jcc.am.fd.a(fd.java:60) .. // create data frame with String , and Boolean types val empdf = sc.parallelize( Array((1,"true".toBoolean ), (2, "false".toBoolean ))).toDF("id", "isManager") // write the data frame. this will fail with error. empdf.write.jdbc("jdbc:db2://<server>:<port> /SAMPLE:retrieveMessagesFromServerOnGetMessage=true;", "emp_data", properties) Error : com.ibm.db2.jcc.am.SqlSyntaxErrorException: TEXT at com.ibm.db2.jcc.am.fd.a(fd.java:679) at com.ibm.db2.jcc.am.fd.a(fd.java:60) Write is failing because by default JDBC data source implementation generating table schema with unsupported data types TEXT for String, and BIT1(1) for Boolean. I think String type should get mapped to CLOB/VARCHAR, and boolean type should be mapped to CHAR(1) for DB2 database. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org