This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 32bdf8a5b52 [SPARK-43509][PYTHON][CONNECT][FOLLOW-UP] Set SPARK_CONNECT_MODE_ENABLED when running pyspark shell with remote is local 32bdf8a5b52 is described below commit 32bdf8a5b52ab1370dd009cd5967042c7ac20446 Author: Hyukjin Kwon <gurwls...@apache.org> AuthorDate: Thu May 18 14:35:39 2023 +0900 [SPARK-43509][PYTHON][CONNECT][FOLLOW-UP] Set SPARK_CONNECT_MODE_ENABLED when running pyspark shell with remote is local ### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/41013 that sets `SPARK_CONNECT_MODE_ENABLED` when running PySpark shell via `bin/pyspark --remote=local` so it works. ### Why are the changes needed? It is broken after the PR: ```bash ./bin/pyspark --remote local ``` ```python ... Traceback (most recent call last): File "/.../python/pyspark/shell.py", line 78, in <module> sc = spark.sparkContext File "/.../python/pyspark/sql/connect/session.py", line 567, in sparkContext raise PySparkNotImplementedError( pyspark.errors.exceptions.base.PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented. ``` ### Does this PR introduce _any_ user-facing change? No to end users. Yes to the dev as described above. ### How was this patch tested? Manually tested via `./bin/pyspark --remote local` Closes #41206 from HyukjinKwon/SPARK-43509-followup. Authored-by: Hyukjin Kwon <gurwls...@apache.org> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .../org/apache/spark/launcher/SparkSubmitCommandBuilder.java | 1 + python/pyspark/shell.py | 9 +++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java index 289eb31db31..9618673b4ce 100644 --- a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java +++ b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java @@ -363,6 +363,7 @@ class SparkSubmitCommandBuilder extends AbstractCommandBuilder { } if (remoteStr != null) { env.put("SPARK_REMOTE", remoteStr); + env.put("SPARK_CONNECT_MODE_ENABLED", "1"); } if (!isEmpty(pyOpts)) { diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py index 86f576a3029..7e2093c1d31 100644 --- a/python/pyspark/shell.py +++ b/python/pyspark/shell.py @@ -100,10 +100,11 @@ print( % (platform.python_version(), platform.python_build()[0], platform.python_build()[1]) ) if is_remote(): - print( - "Client connected to the Spark Connect server at %s" - % urlparse(os.environ["SPARK_REMOTE"]).netloc - ) + url = os.environ.get("SPARK_REMOTE", None) + assert url is not None + if url.startswith("local"): + url = "sc://localhost" # only for display in the console. + print("Client connected to the Spark Connect server at %s" % urlparse(url).netloc) else: print("Spark context Web UI available at %s" % (sc.uiWebUrl)) # type: ignore[union-attr] print( --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org