Hyukjin Kwon created SPARK-61386:
------------------------------------
Summary: Make the native ML classes consistent and match PySpark's
params, constructors, uids and errors
Key: SPARK-61386
URL: https://issues.apache.org/jira/browse/SPARK-61386
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Low-priority findings of the s16 review of the native ML classes
(spark_connect::ml and pyspark.ml.connect of the drop-in), each checked against
official PySpark 4.3.0 (pyspark.ml.connect and the JVM-backed pyspark.ml
classes over Spark Connect):
- Rust API consistency: JVM Int params were i64 on LogisticRegression and
LinearSVC but i32 on LinearRegression and KMeans; KMeans, LinearSVC and
LinearRegression had no getters where LogisticRegression has them;
LogisticRegressionModel, StringIndexerModel and MaxAbsScalerModel exposed pub
fields where the other models keep them private; the public Params builder
could send only Long ints and no arrays; CrossValidator::set_seed sent a Long
where KMeans narrows the seed as PySpark sends a Python int.
- Python params: values were not converted with PySpark's TypeConverters
(KMeans(k=3.0) and seed=1.0 raised TypeError, an int past a long raised
OverflowError, setWeightCol(None) raised), constructors sent every default
param where PySpark sends only the explicit ones, and uids were bare UUIDs
where PySpark's are <Class>_<12 hex>.
- The older natives (evaluators, LogisticRegression, scalers, StringIndexer,
VectorAssembler, Pipeline, CrossValidator) took positional arguments, where
PySpark's @keyword_only raises "Method __init__ forces keyword arguments."
- A setter called while another thread fitted the same estimator raised
"RuntimeError: Already borrowed".
- The transform of a model on a DataFrame of another session raised
PySparkValueError, where PySpark raises CONNECT_ML.CACHE_INVALID from the
server.
- release_ml_objects sent its Delete on a stopped session, where
release_cached_remote_relation skips stopped sessions.
- Docs: the removal of BinaryClassificationEvaluator set_score_col/score_col
and Python scoreCol= (shipped in v4.2.0) was not in the migration notes, and
the docs said to set a seed for reproducible fits, though without one both
clients use the JVM's default seed.
The fixes apply to every native class, RandomForestClassifier included.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]