Hyukjin Kwon created SPARK-62037:
------------------------------------

             Summary: Share the options and hooks of a builder's shallow 
copies, hold PySpark's session and builder locks, let SparkSession, Catalog and 
DataSourceRegistration be subclassed, and load PySpark's pickles of readers
                 Key: SPARK-62037
                 URL: https://issues.apache.org/jira/browse/SPARK-62037
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the documented differences of the session against PySpark 
>4.3.0:
- copy.copy of the builder of pyspark.sql.connect.session.SparkSession has its 
own options and hook factories, where PySpark's plain object shares its 
_options dict and _hook_factories list with the copy, so a config or a hook set 
on the copy is seen by the original.
- The builder makes and stops sessions under a Rust lock, where PySpark holds 
the class attribute SparkSession._lock (an RLock) for getOrCreate, around the 
default session of create and for stop, and the builder's own Builder._lock for 
config, remote, appName, channelBuilder and _registerHook, so a thread that 
holds SparkSession._lock does not hold off the drop-in; and getOrCreate of a 
new session sends the Config of its options once, where PySpark's create and 
getOrCreate both apply them.
- SparkSession, Catalog and DataSourceRegistration cannot be subclassed, where 
PySpark's plain classes can.
- PySpark's pickle of a DataFrameReader or a DataStreamReader does not load in 
the drop-in (TypeError: missing 'client').
- The drop-in's LogicalPlan has no to_proto, which plugins and tools call.
- The known differences of the default port, of a missing URL, of the rendering 
of deep plans and of the builder copies are stale, and they lack that PySpark 
4.3.0 hangs at exit for a session that is not the default and is not stopped.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to