[
https://issues.apache.org/jira/browse/SPARK-61852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61852:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Give the remaining objects of a session PySpark Connect's classes, modules
> and reprs, and make the classic conf, tvf, merge and observation classes
> PySpark's
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61852
> URL: https://issues.apache.org/jira/browse/SPARK-61852
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61758 and SPARK-61769 (the classes and modules of the
> objects of a session), found by a review of the last 100 commits, each
> checked against official pyspark-client 4.3.0:
> - spark.dataSource is a native class that claims
> pyspark.sql.datasource.DataSourceRegistration; PySpark gives
> pyspark.sql.connect.datasource.DataSourceRegistration.
> - pyspark.sql.connect.avro.functions and protobuf.functions re-export the
> classic functions.
> - pickle of pyspark.sql.connect.functions.random fails, and F.column, negate
> and power are not the aliases col, negative and pow.
> - repr(spark.catalog) is Catalog() instead of the default object repr.
> - The Connect builder's enableHiveSupport() is a no-op where PySpark raises
> NOT_IMPLEMENTED.
> - SparkSession.Hook is not nested (qualname Hook), and neither its class nor
> a subclass instance pickles.
> - The SparkSession.builder descriptor is a native class that claims
> pyspark.sql.session.classproperty.
> - ChannelBuilder, DefaultChannelBuilder and PlanObservedMetrics are not of
> pyspark.sql.connect.client.core.
> - The classic RuntimeConfig, TableValuedFunction, MergeIntoWriter and
> Observation name the Connect classes, so isinstance is True where PySpark
> gives False; pyspark.sql.connect.window.Window is the classic class.
> - The connect banner names pyspark.__version__ (4.3.0) instead of the package
> version (4.4.0).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]