GitHub user sureshthalamati opened a pull request: https://github.com/apache/spark/pull/16208
[WIP][SPARK-10849][SQL] Adds a new column metadata property to the jdbc data source for users to specify database column type using the metadata ## What changes were proposed in this pull request? Currently JDBC data source creates tables in the target database using the default type mapping, and the JDBC dialect mechanism. Â If users want to specify different database data type for only some of columns, there is no option available. In scenarios where default mapping does not work, users are forced to create tables on the target database before writing. This workaround is probably not acceptable from a usability point of view. This PR is to provide a user-defined type mapping for specific columns. Â The solution is based on the existing Redshift connector (https://github.com/databricks/spark-redshift#setting-a-custom-column-type). We add a new column metadata property to the jdbc data source for users to specify database column type using the metadata. Â Example : ```Scala val nvarcharMd = new MetadataBuilder().putString(âcreateTableColumnType", "NVARCHAR(123)").build() val newDf = df.withColumn("name", col("name"), nvarcharMd) newDf.write.mode(SaveMode.Overwrite).jdbc(url, "TEST.USERDBTYPETEST", properties) ``` One restriction with this approach metadata modification is unsupported in the Python, SQL, and R language APIs. Users have to create a new data frame to specify the metadata with the _createTableColumnType_ property. Â Alternative approach is to add JDBC data source option for users to specify database column types information as JSON String. TODO: Documentation for specifying the database column type ## How was this patch tested? Added new test case to the JDBCWriteSuite You can merge this pull request into a Git repository by running: $ git pull https://github.com/sureshthalamati/spark jdbc_custom_dbtype-spark-10849 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16208.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16208 ---- commit 38349033a306a733e83975ca09b6cf8a8d69d397 Author: sureshthalamati <suresh.thalam...@gmail.com> Date: 2016-12-02T23:22:17Z [SPARK-10849][SQL} Add new jdbc datasource metadata property to allow users to specify database column type when creating table on write. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org