Ruifeng Zheng created SPARK-61633:
-------------------------------------

             Summary: Support the remaining native clustering algorithms: LDA 
and PowerIterationClustering
                 Key: SPARK-61633
                 URL: https://issues.apache.org/jira/browse/SPARK-61633
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Ruifeng Zheng


Complete native clustering algorithm execution by adding the remaining PySpark 
4.3.0 algorithms: LDA and PowerIterationClustering.

LDA should support both online and EM optimizers, all 14 constructor parameters 
and setters, cached fitted models and DataFrame transforms. 
PowerIterationClustering should expose its six parameters and assignClusters, 
using Spark Connect's powerIterationClusteringAssignClusters helper instead of 
an estimator fit API. Implement both the Rust API and native Python bindings, 
preserving the Spark-DataFrame-only input contract.

Validate parameter serialization and helper requests against official PySpark 
fixtures, and run Rust and native Python live tests for inference, graph 
assignments, parameter validation and model lifetime. This completes algorithm 
execution coverage; richer model inspection and persistence remain separate 
work.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to