[ https://issues.apache.org/jira/browse/SPARK-6442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629225#comment-14629225 ]
Rahul Palamuttam edited comment on SPARK-6442 at 7/16/15 5:33 AM: ------------------------------------------------------------------ I think instead of breeze, nd4j from deeplearning4j is a good library to look at that tackles all 4. I've been using it as an experimental backend for [SciSpark's sRDD | http://apache-spark-developers-list.1001551.n3.nabble.com/SciSpark-NASA-AIST14-proposal-td10115.html] It also has support for n-dimensions like numpy (unlike breeze). It's a very young project. They have one version published in mvnrepository. I use it by pulling from the git repo and and installing via maven locally. To give a general idea of how the [nd4j library | https://github.com/deeplearning4j/nd4j] works. a) You can choose which backend you want - jblas, netlib-java, or x86. The x86 one uses netlib-java for BLAS operations and drops down to C level for-loops (via JNI) for elementwise-operations. b) They also provide a scala-api backend (with the operators) via DSL in a separate project called nd4s. c) They've recently provided the option of ordering elements contiguously in memory. So element-wise-operations now benefit from cache locality. I guess the general consensus is the community is spread thin, so it could be worth it to wait for nd4j to mature a bit more and then tackle the problem. was (Author: rahul palamuttam): I think instead of breeze, nd4j from deeplearning4j is a good library to look at that tackles all 4. I've been using it as an experimental backend for [SciSpark's sRDD | http://apache-spark-developers-list.1001551.n3.nabble.com/SciSpark-NASA-AIST14-proposal-td10115.html] It also has support for n-dimensions like numpy (unlike breeze). It's a very young project. They have one version published in mvnrepository. I use it by pulling from the git repo and and installing via maven locally. To give an idea of how the [nd4j library | https://github.com/deeplearning4j/nd4j] works. a) You can choose which backend you want - jblas, netlib-java, or x86. The x86 one uses netlib-java for BLAS operations and drops down to C level for-loops (via JNI) for elementwise-operations. b) They also provide a scala-api backend (with the operators) via DSL in a separate project called nd4s. c) They've recently provided the option of ordering elements contiguously in memory. So element-wise-operations now benefit from cache locality. I guess the general consensus is the community is spread thin, so it could be worth it to wait for nd4j to mature a bit more and then tackle the problem. > MLlib Local Linear Algebra Package > ---------------------------------- > > Key: SPARK-6442 > URL: https://issues.apache.org/jira/browse/SPARK-6442 > Project: Spark > Issue Type: New Feature > Components: MLlib > Reporter: Burak Yavuz > Priority: Critical > > MLlib's local linear algebra package doesn't have any support for any type of > matrix operations. With 1.5, we wish to add support to a complete package of > optimized linear algebra operations for Scala/Java users. > The main goal is to support lazy operations so that element-wise can be > implemented in a single for-loop, and complex operations can be interfaced > through BLAS. > The design doc: http://goo.gl/sf5LCE -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org