[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13289 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13289#discussion_r64620005
--- Diff: python/pyspark/sql/session.py ---
@@ -138,24 +138,37 @@ def getOrCreate(self):
"""Gets an existing :class:`SparkSession` or, if there is no
existing one, creates a
new one based on the options set in this builder.
-This method first checks whether there is a valid thread-local
SparkSession,
-and if yes, return that one. It then checks whether there is a
valid global
-default SparkSession, and if yes, return that one. If no valid
global default
-SparkSession exists, the method creates a new SparkSession and
assigns the
-newly created SparkSession as the global default.
+This method first checks whether there is a valid global
default SparkSession, and if
+yes, return that one. If no valid global default SparkSession
exists, the method
+creates a new SparkSession and assigns the newly created
SparkSession as the global
+default.
+
+>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
+>>> s1.conf.get("k1") == "v1"
+True
In case an existing SparkSession is returned, the config
options specified
in this builder will be applied to the existing SparkSession.
+
+>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
+>>> s1.conf.get("k1") == s2.conf.get("k1")
+True
+>>> s1.conf.get("k2") == s2.conf.get("k2")
+True
"""
with self._lock:
-from pyspark.conf import SparkConf
from pyspark.context import SparkContext
-from pyspark.sql.context import SQLContext
-sparkConf = SparkConf()
+from pyspark.conf import SparkConf
+session = SparkSession._instantiatedContext
+if session is None:
+sparkConf = SparkConf()
+for key, value in self._options.items():
+sparkConf.set(key, value)
+sc = SparkContext.getOrCreate(sparkConf)
+session = SparkSession(sc)
--- End diff --
actually before this line we might have to explicitly set the confs through
`sc.conf.set`, since the `SparkContext` may be an existing one. There was a
patch that did this for scala recently: 01e7b9c85bb84924e279021f9748774dce9702c8
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221653189 Looks good. Merging into master 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221454949 **[Test build #59248 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59248/consoleFull)** for PR 13289 at commit [`d972a4a`](https://github.com/apache/spark/commit/d972a4ae90d5736c912a18aad9c1ce9543835b7e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221455015 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59248/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221455014 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221452786 **[Test build #59248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59248/consoleFull)** for PR 13289 at commit [`d972a4a`](https://github.com/apache/spark/commit/d972a4ae90d5736c912a18aad9c1ce9543835b7e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221451538 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221451526 **[Test build #59247 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59247/consoleFull)** for PR 13289 at commit [`636e5a6`](https://github.com/apache/spark/commit/636e5a6a904442bd53612f34d4271bc377f57884). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221451539 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59247/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13289#issuecomment-221449490 **[Test build #59247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59247/consoleFull)** for PR 13289 at commit [`636e5a6`](https://github.com/apache/spark/commit/636e5a6a904442bd53612f34d4271bc377f57884). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...
GitHub user ericl opened a pull request: https://github.com/apache/spark/pull/13289 [SPARK-15520] [SQL] SparkSession builder in python should also allow overriding confs of existing sessions ## What changes were proposed in this pull request? This fixes the python SparkSession builder to allow setting confs correctly. This was a leftover TODO from https://github.com/apache/spark/pull/13200. ## How was this patch tested? Python doc tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ericl/spark spark-15520 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/13289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13289 commit 636e5a6a904442bd53612f34d4271bc377f57884 Author: Eric Liang Date: 2016-05-25T01:22:18Z Tue May 24 18:22:17 PDT 2016 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
