[ 
https://issues.apache.org/jira/browse/SPARK-61737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61737:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Make the native ML classes subclasses of PySpark's classes, so that their 
> objects are instances of every class of PySpark's MRO
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61737
>                 URL: https://issues.apache.org/jira/browse/SPARK-61737
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The objects of PySpark's ML classes are instances of every class of their 
> MRO: MLWritable, MLReadable, Identifiable, JavaMLWritable and JavaMLReadable, 
> HasTrainingSummary and the summary bases, the param mixins (HasMaxIter, 
> HasFeaturesCol, ...), PredictionModel and ClassificationModel, the Java 
> wrappers, and the class itself.
> The native ML classes of Spark Connect Rust are PyO3 classes, registered as 
> virtual subclasses of PySpark's ABC bases (Estimator, Model, Params, 
> Evaluator) only: isinstance(model, MLWritable) is False for every native 
> class (MLWritable is a plain class), and so PySpark's own pyspark.ml.Pipeline 
> and validators refuse to save native stages ("... is not MLWritable"). Every 
> one of the 140 classes of the full API has a class of PySpark's MRO that it 
> is not a subclass of.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to