Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Thanks so much! I was just looking for it today on spark-packages - you've read my mind :) On Wed, Sep 9, 2015 at 5:53 PM, Burak Yavuz wrote: > By the way, I published > http://spark-packages.org/package/brkyvz/lazy-linalg that contains many > of the arithmetic operations for

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

By the way, I published http://spark-packages.org/package/brkyvz/lazy-linalg that contains many of the arithmetic operations for use in Scala. I really would appreciate any feedback! On Tue, Aug 25, 2015 at 11:06 AM, Kristina Rogale Plazonic wrote: > YES PLEASE! > > :)))

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

However I do think it's easier than it seems to write the implicits; it doesn't involve new classes or anything. Yes it's pretty much just what you wrote. There is a class Vector in Spark. This declaration can be in an object; you don't implement your own class. (Also you can use toBreeze to

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

What about declaring a few simple implicit conversions between the MLlib and Breeze Vector classes? if you import them then you should be able to write a lot of the source code just as you imagine it, as if the Breeze methods were available on the Vector object in MLlib. The problem is that

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Hmm. I have a lot of code on the local linear algebra operations using Spark's Matrix and Vector representations done for https://issues.apache.org/jira/browse/SPARK-6442. I can make a Spark package with that code if people are interested. Best, Burak On Tue, Aug 25, 2015 at 10:54 AM, Kristina

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

YES PLEASE! :))) On Tue, Aug 25, 2015 at 1:57 PM, Burak Yavuz brk...@gmail.com wrote: Hmm. I have a lot of code on the local linear algebra operations using Spark's Matrix and Vector representations done for https://issues.apache.org/jira/browse/SPARK-6442. I can make a Spark package

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Yes, you're right that it's quite on purpose to leave this API to Breeze, in the main. As you can see the Spark objects have already sprouted a few basic operations anyway; there's a slippery slope problem here. Why not addition, why not dot products, why not determinants, etc. What about

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Yes I get all that too and I think there's a legit question about whether moving a little further down the slippery slope is worth it and if so how far. The other catch here is: either you completely mimic another API (in which case why not just use it directly, which has its own problems) or you

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Well, yes, the hack below works (that's all I have time for), but is not satisfactory - it is not safe, and is verbose and very cumbersome to use, does not separately deal with SparseVector case and is not complete either. My question is, out of hundreds of users on this list, someone must have

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Kristina, Thanks for the discussion. I followed up on your problem and learned that Scala doesn't support multiple implicit conversions in a single expression http://stackoverflow.com/questions/8068346/can-scala-apply-multiple-implicit-conversions-in-one-expression for complexity reasons. I'm

Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

Hi all, I'm still not clear what is the best (or, ANY) way to add/subtract two org.apache.spark.mllib.Vector objects in Scala. Ok, I understand there was a conscious Spark decision not to support linear algebra operations in Scala and leave it to the user to choose a linear algebra library.

Re: Adding/subtracting org.apache.spark.mllib.linalg.Vector in Scala?

From what I have understood, you probably need to convert your vector to breeze and do your operations there. Check stackoverflow.com/questions/28232829/addition-of-two-rddmllib-linalg-vectors On Aug 25, 2015 7:06 PM, Kristina Rogale Plazonic kpl...@gmail.com wrote: Hi all, I'm still not clear