Hyukjin Kwon created SPARK-61386:
------------------------------------

             Summary: Make the native ML classes consistent and match PySpark's 
params, constructors, uids and errors
                 Key: SPARK-61386
                 URL: https://issues.apache.org/jira/browse/SPARK-61386
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Low-priority findings of the s16 review of the native ML classes 
(spark_connect::ml and pyspark.ml.connect of the drop-in), each checked against 
official PySpark 4.3.0 (pyspark.ml.connect and the JVM-backed pyspark.ml 
classes over Spark Connect):
- Rust API consistency: JVM Int params were i64 on LogisticRegression and 
LinearSVC but i32 on LinearRegression and KMeans; KMeans, LinearSVC and 
LinearRegression had no getters where LogisticRegression has them; 
LogisticRegressionModel, StringIndexerModel and MaxAbsScalerModel exposed pub 
fields where the other models keep them private; the public Params builder 
could send only Long ints and no arrays; CrossValidator::set_seed sent a Long 
where KMeans narrows the seed as PySpark sends a Python int.
- Python params: values were not converted with PySpark's TypeConverters 
(KMeans(k=3.0) and seed=1.0 raised TypeError, an int past a long raised 
OverflowError, setWeightCol(None) raised), constructors sent every default 
param where PySpark sends only the explicit ones, and uids were bare UUIDs 
where PySpark's are <Class>_<12 hex>.
- The older natives (evaluators, LogisticRegression, scalers, StringIndexer, 
VectorAssembler, Pipeline, CrossValidator) took positional arguments, where 
PySpark's @keyword_only raises "Method __init__ forces keyword arguments."
- A setter called while another thread fitted the same estimator raised 
"RuntimeError: Already borrowed".
- The transform of a model on a DataFrame of another session raised 
PySparkValueError, where PySpark raises CONNECT_ML.CACHE_INVALID from the 
server.
- release_ml_objects sent its Delete on a stopped session, where 
release_cached_remote_relation skips stopped sessions.
- Docs: the removal of BinaryClassificationEvaluator set_score_col/score_col 
and Python scoreCol= (shipped in v4.2.0) was not in the migration notes, and 
the docs said to set a seed for reproducible fits, though without one both 
clients use the JVM's default seed.
The fixes apply to every native class, RandomForestClassifier included.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to