Hi, I think there is a sort of inconsistency in how DataFrameReader.jdbc deals with a user-defined schema as it makes sure that there's no user-specified schema [1][2] yet allows for setting one using customSchema option [3]. Why is so? Has this been merely overlooked or similar?
I think assertNoSpecifiedSchema should be removed from DataFrameReader.jdbc and support for DataFrameReader.schema for jdbc should be added (with the customSchema option marked as deprecated to be removed in 2.4 or 3.0). Should I file an issue in Spark JIRA and do the changes? WDYT? [1] https://github.com/apache/spark/blob/v2.3.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala?utf8=%E2%9C%93#L249 [2] https://github.com/apache/spark/blob/v2.3.1/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala?utf8=%E2%9C%93#L320 [3] https://github.com/apache/spark/blob/v2.3.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala#L167 Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski
