Hi, We plan to add a more comprehensive local linear algebra package for MLlib 1.4. This local linear algebra package can then easily be extended to BlockMatrix to support the same operations in a distributed fashion.
You may find the JIRA to track this here: SPARK-6442 <https://issues.apache.org/jira/browse/SPARK-6442> The design doc is here: http://goo.gl/sf5LCE We would very much appreciate your feedback and input. Best, Burak On Thu, Mar 19, 2015 at 3:06 PM, Debasish Das <debasish.da...@gmail.com> wrote: > Yeah it will be better if we consolidate the development on one of > them...either Breeze or mllib.BLAS... > > On Thu, Mar 19, 2015 at 2:25 PM, Ulanov, Alexander < > alexander.ula...@hp.com> > wrote: > > > Thanks for quick response. > > > > I can use linealg.BLAS.gemm, and this means that I have to use MLlib > > Matrix. The latter does not support some useful functionality needed for > > optimization. For example, creation of Matrix given matrix size, array > and > > offset in this array. This means that I will need to create matrix in > > Breeze and convert it to MLlib. Also, linalg.BLAS misses some useful BLAS > > functions I need, that can be found in Breeze (and netlib-java). The same > > concerns are applicable to MLlib Vector. > > > > Best regards, Alexander > > > > 19.03.2015, в 14:16, "Debasish Das" <debasish.da...@gmail.com> > написал(а): > > > > I think for Breeze we are focused on dot and dgemv right now (along > > with several other matrix vector style operations)... > > > > For dgemm it is tricky since you need to do add dgemm for both > > DenseMatrix and CSCMatrix...and for CSCMatrix you need to get something > > like SuiteSparse which is under lgpl...so we have to think more on it.. > > > > For now can't you use dgemm directly from mllib.linalg.BLAS ? It's in > > master... > > > > > > On Thu, Mar 19, 2015 at 1:49 PM, Ulanov, Alexander < > > alexander.ula...@hp.com> wrote: > > > >> Thank you! When do you expect to have gemm in Breeze and that version > >> of Breeze to ship with MLlib? > >> > >> Also, could someone please elaborate on the linalg.BLAS and Matrix? Are > >> they going to be developed further, should in long term all developers > use > >> them? > >> > >> Best regards, Alexander > >> > >> 18.03.2015, в 23:21, "Debasish Das" <debasish.da...@gmail.com> > >> написал(а): > >> > >> dgemm dgemv and dot come to Breeze and Spark through netlib-java.... > >> > >> Right now both in dot and dgemv Breeze does a extra memory allocate but > >> we already found the issue and we are working on adding a common trait > that > >> will provide a sink operation (basically memory will be allocated by > >> user)...adding more BLAS operators in breeze will also help in general > as > >> lot more operations are defined over there... > >> > >> > >> On Wed, Mar 18, 2015 at 8:09 PM, Ulanov, Alexander < > >> alexander.ula...@hp.com> wrote: > >> > >>> Hi, > >>> > >>> Currently I am using Breeze within Spark MLlib for linear algebra. I > >>> would like to reuse previously allocated matrices for storing the > result of > >>> matrices multiplication, i.e. I need to use "gemm" function > C:=q*A*B+p*C, > >>> which is missing in Breeze (Breeze automatically allocates a new > matrix to > >>> store the result of multiplication). Also, I would like to minimize > gemm > >>> calls that Breeze does. Should I use mllib.linalg.BLAS functions > instead? > >>> While it has gemm and axpy, it has rather limited number of > operations. For > >>> example, I need sum of the matrix by row or by columns, or applying a > >>> function to all elements in a matrix. Also, MLlib Vector and Matrix > >>> interfaces that linalg.BLAS operates seems to be rather undeveloped. > Should > >>> I use plain netlib-java instead (will it remain in MLlib in future > >>> releases)? > >>> > >>> Best regards, Alexander > >>> > >> > >> > > >
