[
https://issues.apache.org/jira/browse/SPARK-62037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62037:
-----------------------------------
Labels: pull-request-available (was: )
> Share the options and hooks of a builder's shallow copies, hold PySpark's
> session and builder locks, let SparkSession, Catalog and
> DataSourceRegistration be subclassed, and load PySpark's pickles of readers
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62037
> URL: https://issues.apache.org/jira/browse/SPARK-62037
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> From a review of the documented differences of the session against PySpark
> 4.3.0:
> - copy.copy of the builder of pyspark.sql.connect.session.SparkSession has
> its own options and hook factories, where PySpark's plain object shares its
> _options dict and _hook_factories list with the copy, so a config or a hook
> set on the copy is seen by the original.
> - The builder makes and stops sessions under a Rust lock, where PySpark holds
> the class attribute SparkSession._lock (an RLock) for getOrCreate, around the
> default session of create and for stop, and the builder's own Builder._lock
> for config, remote, appName, channelBuilder and _registerHook, so a thread
> that holds SparkSession._lock does not hold off the drop-in; and getOrCreate
> of a new session sends the Config of its options once, where PySpark's create
> and getOrCreate both apply them.
> - SparkSession, Catalog and DataSourceRegistration cannot be subclassed,
> where PySpark's plain classes can.
> - PySpark's pickle of a DataFrameReader or a DataStreamReader does not load
> in the drop-in (TypeError: missing 'client').
> - The drop-in's LogicalPlan has no to_proto, which plugins and tools call.
> - The known differences of the default port, of a missing URL, of the
> rendering of deep plans and of the builder copies are stale, and they lack
> that PySpark 4.3.0 hangs at exit for a session that is not the default and is
> not stopped.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]