[ https://issues.apache.org/jira/browse/SPARK-15575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360099#comment-15360099 ]
Yanbo Liang edited comment on SPARK-15575 at 7/2/16 9:40 AM: ------------------------------------------------------------- Yes, {{LinearRegression}}, {{LogisticRegression}}, {{AFTSurvivalRegression}} and {{MultilayerPerceptronClassifier}} depend on L-BFGS/OWL-QN currently. We have also implemented Spark owned L-BFGS/SGD optimizers in mllib package. But the four estimators mentioned above use different implementation. || Estimators || Optimizer implementation || | LinearRegression | breeze L-BFGS/OWL-QN | | LogisticRegression | breeze L-BFGS/OWL-QN | | AFTSurvivalRegression | breeze L-BFGS/OWL-QN | | MultilayerPerceptronClassifier | mllib L-BFGS/SGD | The L-BFGS implementation in mllib also calling breeze L-BFGS underneath. We should figure out a way to make the transformation smoothly. Since I'm also investigating the scalable version of L-BFGS (SPARK-10078) recently, I can start to write a draft to track the features and requirements of optimizers that Spark needed. Then we can discuss and design how to move to the new implementation. Thanks! was (Author: yanboliang): Yes, {{LinearRegression}}, {{LogisticRegression}}, {{AFTSurvivalRegression}} and {{MultilayerPerceptronClassifier}} depend on L-BFGS/OWL-QN currently. We have implemented Spark owned L-BFGS/SGD optimizers in mllib package. And the four estimators mentioned above use different implementation. || Estimators || Optimizer implementation || | LinearRegression | breeze L-BFGS/OWL-QN | | LogisticRegression | breeze L-BFGS/OWL-QN | | AFTSurvivalRegression | breeze L-BFGS/OWL-QN | | MultilayerPerceptronClassifier | mllib L-BFGS/SGD | The L-BFGS implementation in mllib also calling breeze L-BFGS underneath. We should figure out a way to make the transformation smoothly. Since I'm also investigating the scalable version of L-BFGS (SPARK-10078) recently, I can start to write a draft to track the features and requirements of optimizers that Spark needed. Then we can discuss and design how to move to the new implementation. Thanks! > Remove breeze from dependencies? > -------------------------------- > > Key: SPARK-15575 > URL: https://issues.apache.org/jira/browse/SPARK-15575 > Project: Spark > Issue Type: Improvement > Components: ML > Reporter: Joseph K. Bradley > > This JIRA is for discussing whether we should remove Breeze from the > dependencies of MLlib. The main issues with Breeze are Scala 2.12 support > and performance issues. > There are a few paths: > # Keep dependency. This could be OK, especially if the Scala version issues > are fixed within Breeze. > # Remove dependency > ## Implement our own linear algebra operators as needed > ## Design a way to build Spark using custom linalg libraries of the user's > choice. E.g., you could build MLlib using Breeze, or any other library > supporting the required operations. This might require significant work. > See [SPARK-6442] for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org