[
https://issues.apache.org/jira/browse/SPARK-61111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61111.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 309
[https://github.com/apache/spark-connect-rust/pull/309]
> Support the ML API of pyspark.ml, of estimators, models, pipelines and
> evaluators on the server, in the Python client as PySpark does
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61111
> URL: https://issues.apache.org/jira/browse/SPARK-61111
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The PySpark client package ships {{pyspark.ml}}, the ML API of estimators,
> models, pipelines, evaluators and tuning ({{connect_packages}} of
> {{python/packaging/client/setup.py}}). In PySpark Connect, these run on the
> server through the Connect ML protocol. For example, {{from
> pyspark.ml.classification import LogisticRegression;
> LogisticRegression().fit(df)}} fits the model on the server, keeps it in the
> server's ML cache and reads its attributes and summary from it.
> {{pyspark.ml.util}} reaches the server through the Connect client:
> * {{session.client.execute_command}} of {{MlCommand}}s (fit, fetch, delete,
> write, read)
> * {{_delete_ml_cache}}
> * DataFrames of the plans of {{pyspark.ml.connect.proto}}
> ({{TransformerRelation}}, {{AttributeRelation}})
> * the plans of DataFrames ({{df._plan.plan(client)}})
> * {{pyspark.sql.connect.proto}}
> The Python client ships only {{pyspark.ml.connect}}, {{linalg}}, {{param}},
> {{torch}} and {{deepspeed}}. {{from pyspark.ml import Pipeline}},
> {{pyspark.ml.classification}}, {{feature}}, {{regression}}, {{clustering}},
> {{evaluation}}, {{tuning}} and the other modules of {{pyspark.ml}} raise
> {{ImportError}}.
> The fix is to vendor these modules, and the ML internals of
> {{pyspark.ml.connect}} that they use, from Apache Spark v4.3.0. The client
> then needs to give them what they use of the Connect client, over its Rust
> transport:
> * the generated {{pyspark.sql.connect.proto}} modules
> * a {{client}} of the commands and the ML cache
> * DataFrames of official plans, and the plans of DataFrames
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
