Repository: spark Updated Branches: refs/heads/master 2b1ac6cea -> 594a1bf20
[SPARK-15520][SQL] Also set sparkContext confs when using SparkSession builder in pyspark ## What changes were proposed in this pull request? Also sets confs in the underlying sc when using SparkSession.builder.getOrCreate(). This is a bug-fix from a post-merge comment in https://github.com/apache/spark/pull/13289 ## How was this patch tested? Python doc-tests. Author: Eric Liang <e...@databricks.com> Closes #13309 from ericl/spark-15520-1. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/594a1bf2 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/594a1bf2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/594a1bf2 Branch: refs/heads/master Commit: 594a1bf200fea8d6bcf25839a49186f66f922bc8 Parents: 2b1ac6c Author: Eric Liang <e...@databricks.com> Authored: Thu May 26 12:05:47 2016 -0700 Committer: Andrew Or <and...@databricks.com> Committed: Thu May 26 12:05:47 2016 -0700 ---------------------------------------------------------------------- python/pyspark/sql/session.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/594a1bf2/python/pyspark/sql/session.py ---------------------------------------------------------------------- diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py index 52e7f3d..8f7dcb5 100644 --- a/python/pyspark/sql/session.py +++ b/python/pyspark/sql/session.py @@ -144,7 +144,7 @@ class SparkSession(object): default. >>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate() - >>> s1.conf.get("k1") == "v1" + >>> s1.conf.get("k1") == s1.sparkContext.getConf().get("k1") == "v1" True In case an existing SparkSession is returned, the config options specified @@ -168,6 +168,8 @@ class SparkSession(object): session = SparkSession(sc) for key, value in self._options.items(): session.conf.set(key, value) + for key, value in self._options.items(): + session.sparkContext._conf.set(key, value) return session builder = Builder() --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
