Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13289#discussion_r64620005
--- Diff: python/pyspark/sql/session.py ---
@@ -138,24 +138,37 @@ def getOrCreate(self):
"""Gets an existing :class:`SparkSession` or, if there is no
existing one, creates a
new one based on the options set in this builder.
- This method first checks whether there is a valid thread-local
SparkSession,
- and if yes, return that one. It then checks whether there is a
valid global
- default SparkSession, and if yes, return that one. If no valid
global default
- SparkSession exists, the method creates a new SparkSession and
assigns the
- newly created SparkSession as the global default.
+ This method first checks whether there is a valid global
default SparkSession, and if
+ yes, return that one. If no valid global default SparkSession
exists, the method
+ creates a new SparkSession and assigns the newly created
SparkSession as the global
+ default.
+
+ >>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
+ >>> s1.conf.get("k1") == "v1"
+ True
In case an existing SparkSession is returned, the config
options specified
in this builder will be applied to the existing SparkSession.
+
+ >>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
+ >>> s1.conf.get("k1") == s2.conf.get("k1")
+ True
+ >>> s1.conf.get("k2") == s2.conf.get("k2")
+ True
"""
with self._lock:
- from pyspark.conf import SparkConf
from pyspark.context import SparkContext
- from pyspark.sql.context import SQLContext
- sparkConf = SparkConf()
+ from pyspark.conf import SparkConf
+ session = SparkSession._instantiatedContext
+ if session is None:
+ sparkConf = SparkConf()
+ for key, value in self._options.items():
+ sparkConf.set(key, value)
+ sc = SparkContext.getOrCreate(sparkConf)
+ session = SparkSession(sc)
--- End diff --
actually before this line we might have to explicitly set the confs through
`sc.conf.set`, since the `SparkContext` may be an existing one. There was a
patch that did this for scala recently: 01e7b9c85bb84924e279021f9748774dce9702c8
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
