[GitHub] spark pull request #20927: [SPARK-23809][SQL] Active SparkSession should be ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20927 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20927: [SPARK-23809][SQL] Active SparkSession should be ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20927#discussion_r178996716
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
---
@@ -1027,6 +1028,17 @@ object SparkSession {
*/
def getDefaultSession: Option[SparkSession] = Option(defaultSession.get)
+ /**
+ * Returns the currently active SparkSession, otherwise the default one.
If there is no default
+ * SparkSession, throws an exception.
+ *
+ * @since 2.4.0
--- End diff --
@ericl Could you submit a PR to 4.x without this new API?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20927: [SPARK-23809][SQL] Active SparkSession should be ...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20927#discussion_r177915352 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala --- @@ -34,6 +34,9 @@ private[spark] class TestSparkSession(sc: SparkContext) extends SparkSession(sc) this(new SparkConf) } + SparkSession.setDefaultSession(this) --- End diff -- We will merge that one first. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20927: [SPARK-23809][SQL] Active SparkSession should be ...
Github user ericl commented on a diff in the pull request: https://github.com/apache/spark/pull/20927#discussion_r177910254 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala --- @@ -34,6 +34,9 @@ private[spark] class TestSparkSession(sc: SparkContext) extends SparkSession(sc) this(new SparkConf) } + SparkSession.setDefaultSession(this) --- End diff -- Intentional conflict with https://github.com/apache/spark/pull/20926 , should resolve once one is merged --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20927: [SPARK-23809][SQL] Active SparkSession should be ...
GitHub user ericl opened a pull request: https://github.com/apache/spark/pull/20927 [SPARK-23809][SQL] Active SparkSession should be set by getOrCreate ## What changes were proposed in this pull request? Currently, the active spark session is set inconsistently (e.g., in createDataFrame, prior to query execution). Many places in spark also incorrectly query active session when they should be calling activeSession.getOrElse(defaultSession). The semantics here can be cleaned up if we also set the active session when the default session is set. Related: https://github.com/apache/spark/pull/20926/files ## How was this patch tested? Unit test You can merge this pull request into a Git repository by running: $ git pull https://github.com/ericl/spark active-session-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20927.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20927 commit 1b192d2542884f378a0ccc5c1936ff72c20403ee Author: Eric Liang Date: 2018-03-28T22:19:39Z set active commit 7a1b0f60ac9b6828223b1fce7efc79cb41b6d198 Author: Eric Liang Date: 2018-03-28T22:25:54Z test commit 8f3cbf3399420a14f5ebe74b99b2739437fe3647 Author: Eric Liang Date: 2018-03-28T22:29:26Z mirror spark-23808 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
