[ https://issues.apache.org/jira/browse/SPARK-13944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240322#comment-15240322 ]
Xiangrui Meng commented on SPARK-13944: --------------------------------------- There are more production workflows using RDD-based APIs than DataFrame-based APIs since many users are still running Spark 1.4 or earlier. It would be nice if we can keep binary compatibility on RDD-based APIs in Spark 2.0. Using type alias is not a good solution because 1) it is not Java-compatible, 2) it introduces dependency from the RDD-based API to mllib-local, which means future development on mllib-local might cause behavior changes or break changes to the RDD-based API. Since we already decided that the RDD-based API would go into maintenance mode in Spark 2.0. Leaving some old code there won't increase maintenance cost, compared with the type alias. We can provide a converter than converts all `mllib.linalg` types to `ml.linalg` types in Spark 2.0 to help users migrate to `ml.linalg`. > Separate out local linear algebra as a standalone module without Spark > dependency > --------------------------------------------------------------------------------- > > Key: SPARK-13944 > URL: https://issues.apache.org/jira/browse/SPARK-13944 > Project: Spark > Issue Type: New Feature > Components: Build, ML > Affects Versions: 2.0.0 > Reporter: Xiangrui Meng > Assignee: DB Tsai > Priority: Blocker > > Separate out linear algebra as a standalone module without Spark dependency > to simplify production deployment. We can call the new module > spark-mllib-local, which might contain local models in the future. > The major issue is to remove dependencies on user-defined types. > The package name will be changed from mllib to ml. For example, Vector will > be changed from `org.apache.spark.mllib.linalg.Vector` to > `org.apache.spark.ml.linalg.Vector`. The return vector type in the new ML > pipeline will be the one in ML package; however, the existing mllib code will > not be touched. As a result, this will potentially break the API. Also, when > the vector is loaded from mllib vector by Spark SQL, the vector will > automatically converted into the one in ml package. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org