[ 
https://issues.apache.org/jira/browse/SPARK-61852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61852:
---------------------------------
    Fix Version/s: connect-rust-4.4.0

> Give the remaining objects of a session PySpark Connect's classes, modules 
> and reprs, and make the classic conf, tvf, merge and observation classes 
> PySpark's
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61852
>                 URL: https://issues.apache.org/jira/browse/SPARK-61852
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61758 and SPARK-61769 (the classes and modules of the 
> objects of a session), found by a review of the last 100 commits, each 
> checked against official pyspark-client 4.3.0:
> - spark.dataSource is a native class that claims 
> pyspark.sql.datasource.DataSourceRegistration; PySpark gives 
> pyspark.sql.connect.datasource.DataSourceRegistration.
> - pyspark.sql.connect.avro.functions and protobuf.functions re-export the 
> classic functions.
> - pickle of pyspark.sql.connect.functions.random fails, and F.column, negate 
> and power are not the aliases col, negative and pow.
> - repr(spark.catalog) is Catalog() instead of the default object repr.
> - The Connect builder's enableHiveSupport() is a no-op where PySpark raises 
> NOT_IMPLEMENTED.
> - SparkSession.Hook is not nested (qualname Hook), and neither its class nor 
> a subclass instance pickles.
> - The SparkSession.builder descriptor is a native class that claims 
> pyspark.sql.session.classproperty.
> - ChannelBuilder, DefaultChannelBuilder and PlanObservedMetrics are not of 
> pyspark.sql.connect.client.core.
> - The classic RuntimeConfig, TableValuedFunction, MergeIntoWriter and 
> Observation name the Connect classes, so isinstance is True where PySpark 
> gives False; pyspark.sql.connect.window.Window is the classic class.
> - The connect banner names pyspark.__version__ (4.3.0) instead of the package 
> version (4.4.0).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to