Hyukjin Kwon created SPARK-61008:
------------------------------------
Summary: Get or create the sessions of SparkSession.builder of the
Python client as PySpark does
Key: SPARK-61008
URL: https://issues.apache.org/jira/browse/SPARK-61008
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark 4.3.0 Connect, {{SparkSession.builder.getOrCreate()}} returns the
active session of the thread, or else the default session, and creates one only
when there is neither; it applies the options of the builder to the session
that it returns. Its URL is that of {{remote()}}, or else of {{SPARK_REMOTE}},
and it raises {{CONNECT_URL_NOT_SET}} without either. The first session created
is the default session, and the active session is thread-local: {{create()}}
sets the default and the active session only when they are not set, and
{{stop()}} unsets them.
The Rust-backed Python client creates a new session at each {{getOrCreate()}},
so the configurations and temporary views of a session are not those of the
next one: after {{spark.conf.set("spark.sql.shuffle.partitions", "7")}} and
{{createOrReplaceTempView("v")}}, a second {{getOrCreate()}} returns a session
whose configuration is {{200}} and that has no view {{v}}. It raises
{{ValueError}} without {{remote()}}, even with {{SPARK_REMOTE}} set, and keeps
one active session for all threads, which {{create()}} replaces. Get or create
sessions as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
