Hyukjin Kwon created SPARK-61731:
------------------------------------
Summary: Fetch StopWordsRemover's server defaults, and send the
datasets of PrefixSpan and the ALS subsets and fetch the ALS rank as PySpark
does
Key: SPARK-61731
URL: https://issues.apache.org/jira/browse/SPARK-61731
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Follow-up of the native feature, fpm and recommendation classes of Spark
Connect Rust, to match pyspark.ml 4.3.0:
- StopWordsRemover's constructor holds the server defaults (stopWords, locale)
that PySpark's constructor fetches of the ML helper; without a session it
raises PySpark's bare AssertionError.
- ALSModel: rank is fetched on each access, as PySpark does, and
recommendForAllUsers, recommendForAllItems, recommendForUserSubset and
recommendForItemSubset are members of the native model.
- PrefixSpan.findFrequentSequentialPatterns and
ALSModel.recommendForUserSubset/recommendForItemSubset send their dataset as
pyspark.ml.connect.serialize does, in a lazy DataFrame that the server rejects
of a value that is not a DataFrame, as with official PySpark.
- The Rust API's ALSModel::rank() fetches the rank on each call, and a load
sends its Read alone.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]