[ 
https://issues.apache.org/jira/browse/SPARK-61731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61731:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Fetch StopWordsRemover's server defaults, and send the datasets of PrefixSpan 
> and the ALS subsets and fetch the ALS rank as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61731
>                 URL: https://issues.apache.org/jira/browse/SPARK-61731
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> Follow-up of the native feature, fpm and recommendation classes of Spark 
> Connect Rust, to match pyspark.ml 4.3.0:
> - StopWordsRemover's constructor holds the server defaults (stopWords, 
> locale) that PySpark's constructor fetches of the ML helper; without a 
> session it raises PySpark's bare AssertionError.
> - ALSModel: rank is fetched on each access, as PySpark does, and 
> recommendForAllUsers, recommendForAllItems, recommendForUserSubset and 
> recommendForItemSubset are members of the native model.
> - PrefixSpan.findFrequentSequentialPatterns and 
> ALSModel.recommendForUserSubset/recommendForItemSubset send their dataset as 
> pyspark.ml.connect.serialize does, in a lazy DataFrame that the server 
> rejects of a value that is not a DataFrame, as with official PySpark.
> - The Rust API's ALSModel::rank() fetches the rank on each call, and a load 
> sends its Read alone.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to