Hyukjin Kwon created SPARK-61737:
------------------------------------

             Summary: Make the native ML classes subclasses of PySpark's 
classes, so that their objects are instances of every class of PySpark's MRO
                 Key: SPARK-61737
                 URL: https://issues.apache.org/jira/browse/SPARK-61737
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The objects of PySpark's ML classes are instances of every class of their MRO: 
MLWritable, MLReadable, Identifiable, JavaMLWritable and JavaMLReadable, 
HasTrainingSummary and the summary bases, the param mixins (HasMaxIter, 
HasFeaturesCol, ...), PredictionModel and ClassificationModel, the Java 
wrappers, and the class itself.

The native ML classes of Spark Connect Rust are PyO3 classes, registered as 
virtual subclasses of PySpark's ABC bases (Estimator, Model, Params, Evaluator) 
only: isinstance(model, MLWritable) is False for every native class (MLWritable 
is a plain class), and so PySpark's own pyspark.ml.Pipeline and validators 
refuse to save native stages ("... is not MLWritable"). Every one of the 140 
classes of the full API has a class of PySpark's MRO that it is not a subclass 
of.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to