GitHub user MaxGekk opened a pull request: https://github.com/apache/spark/pull/22489
[SPARK-25425][SQL][BACKPORT-2.3] Extra options should override session options in DataSource V2 ## What changes were proposed in this pull request? In the PR, I propose overriding session options by extra options in DataSource V2. Extra options are more specific and set via `.option()`, and should overwrite more generic session options. ## How was this patch tested? Added tests for read and write paths. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MaxGekk/spark-1 session-options-2.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22489.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22489 ---- commit 0cc174160bf04bc97de62c438d571b27224c53a0 Author: Maxim Gekk <maxim.gekk@...> Date: 2018-09-16T00:24:11Z [SPARK-25425][SQL] Extra options should override session options in DataSource V2 In the PR, I propose overriding session options by extra options in DataSource V2. Extra options are more specific and set via `.option()`, and should overwrite more generic session options. Entries from seconds map overwrites entries with the same key from the first map, for example: ```Scala scala> Map("option" -> false) ++ Map("option" -> true) res0: scala.collection.immutable.Map[String,Boolean] = Map(option -> true) ``` Added a test for checking which option is propagated to a data source in `load()`. Closes #22413 from MaxGekk/session-options. Lead-authored-by: Maxim Gekk <maxim.g...@databricks.com> Co-authored-by: Dongjoon Hyun <dongj...@apache.org> Co-authored-by: Maxim Gekk <max.g...@gmail.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> commit 744c5b8b0c8fbc7caa6d03902ed323df5c73986b Author: Maxim Gekk <maxim.gekk@...> Date: 2018-09-19T19:40:25Z Adding missing import after merge to 2.4 commit c87a6de4a1de82514fad590c68f35fbfaf54d825 Author: Maxim Gekk <maxim.gekk@...> Date: 2018-09-19T20:20:21Z Fix merge commit f8b5aa6b3d1fc9e953bcfef8f23aeb6f3ab7818d Author: Maxim Gekk <maxim.gekk@...> Date: 2018-09-20T09:47:57Z Adding options to Reader ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org