[ 
https://issues.apache.org/jira/browse/SPARK-61386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61386:
------------------------------------

    Assignee: Hyukjin Kwon

> Make the native ML classes consistent and match PySpark's params, 
> constructors, uids and errors
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61386
>                 URL: https://issues.apache.org/jira/browse/SPARK-61386
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Low-priority findings of the s16 review of the native ML classes 
> (spark_connect::ml and pyspark.ml.connect of the drop-in), each checked 
> against official PySpark 4.3.0 (pyspark.ml.connect and the JVM-backed 
> pyspark.ml classes over Spark Connect):
> - Rust API consistency: JVM Int params were i64 on LogisticRegression and 
> LinearSVC but i32 on LinearRegression and KMeans; KMeans, LinearSVC and 
> LinearRegression had no getters where LogisticRegression has them; 
> LogisticRegressionModel, StringIndexerModel and MaxAbsScalerModel exposed pub 
> fields where the other models keep them private; the public Params builder 
> could send only Long ints and no arrays; CrossValidator::set_seed sent a Long 
> where KMeans narrows the seed as PySpark sends a Python int.
> - Python params: values were not converted with PySpark's TypeConverters 
> (KMeans(k=3.0) and seed=1.0 raised TypeError, an int past a long raised 
> OverflowError, setWeightCol(None) raised), constructors sent every default 
> param where PySpark sends only the explicit ones, and uids were bare UUIDs 
> where PySpark's are <Class>_<12 hex>.
> - The older natives (evaluators, LogisticRegression, scalers, StringIndexer, 
> VectorAssembler, Pipeline, CrossValidator) took positional arguments, where 
> PySpark's @keyword_only raises "Method __init__ forces keyword arguments."
> - A setter called while another thread fitted the same estimator raised 
> "RuntimeError: Already borrowed".
> - The transform of a model on a DataFrame of another session raised 
> PySparkValueError, where PySpark raises CONNECT_ML.CACHE_INVALID from the 
> server.
> - release_ml_objects sent its Delete on a stopped session, where 
> release_cached_remote_relation skips stopped sessions.
> - Docs: the removal of BinaryClassificationEvaluator set_score_col/score_col 
> and Python scoreCol= (shipped in v4.2.0) was not in the migration notes, and 
> the docs said to set a seed for reproducible fits, though without one both 
> clients use the JVM's default seed.
> The fixes apply to every native class, RandomForestClassifier included.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to