Contributing algorithms to MLlib
Hi, I'm a M.Sc. computer science student at Tel-Aviv College, Israel ( www.mta.ac.il) and as part of my final project that is dealing with Machine Learning algorithms in distributed systems, I would like to contribute couple of algorithms to MLlib. My advisor, Dr. Uzi Hadad, and I thought of starting with an implementation of* Fuzzy k - means* algorithm and continuing with *Hidden Markov Model* algorithm. Do you know if anyone is currently working on an implementation of these algorithms for MLlib? Regards, Alex
implementing the VectorAccumulatorParam
The programming-guide http://spark.apache.org/docs/latest/programming-guide.html has the following: However, when I try to use this I get an error: Last thing, am I posting on the wrong list? - Cheers, Stephanie -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6972.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
(The user@ list might be a bit better but I can see why it might look
like a dev@ question.)
Did you import org.apache.spark.mllib.linalg.Vector ? I think you are
picking up Scala's Vector class instead.
On Mon, Jun 9, 2014 at 11:57 AM, dataginjaninja
rickett.stepha...@gmail.com wrote:
The programming-guide
http://spark.apache.org/docs/latest/programming-guide.html has the
following:
object VectorAccumulatorParam extends AccumulatorParam[Vector] {
def zero(initialValue: Vector): Vector = {
Vector.zeros(initialValue.size)
}
def addInPlace(v1: Vector, v2: Vector): Vector = {
v1 += v2
}
}
// Then, create an Accumulator of this type:
val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam)
However, when I try to use this I get an error:
scala import org.apache.spark.AccumulatorParam
import org.apache.spark.AccumulatorParam
scala object VectorAccumulatorParam extends AccumulatorParam[Vector] {
| def zero(initialValue: Vector): Vector = {
| Vector.zeros(initialValue.size)
| }
| def addInPlace(v1: Vector, v2: Vector): Vector = {
| v1 += v2
| }
| }
console:12: error: type Vector takes type parameters
object VectorAccumulatorParam extends AccumulatorParam[Vector] {
^
Last thing, am I posting on the wrong list?
-
Cheers,
Stephanie
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
You are right. I was using the wrong vector class. Thanks. - Cheers, Stephanie -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6975.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
New error :-(
scala object VectorAccumulatorParam extends AccumulatorParam[Vector] {
| def zero(initialValue: Vector): Vector = {
| Vector.zeros(initialValue.size)
| }
| def addInPlace(v1: Vector, v2: Vector): Vector = {
| v1 += v2
| }
| }
console:12: error: not found: type AccumulatorParam
object VectorAccumulatorParam extends AccumulatorParam[Vector] {
^
console:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
Vector.zeros(initialValue.size)
^
console:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
v1 += v2
^
-
Cheers,
Stephanie
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6976.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
New error :-(
scala object VectorAccumulatorParam extends AccumulatorParam[Vector] {
| def zero(initialValue: Vector): Vector = {
| Vector.zeros(initialValue.size)
| }
| def addInPlace(v1: Vector, v2: Vector): Vector = {
| v1 += v2
| }
| }
console:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
Vector.zeros(initialValue.size)
^
console:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
v1 += v2
-
Cheers,
Stephanie
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: implementing the VectorAccumulatorParam
(BCC dev@)
The example is out of date with respect to current Vector class. The
zeros() method is on Vectors. There is not currently a += operation
for Vector anymore.
To be fair the example doesn't claim this illustrates use of the Spark
Vector class but it did work with the now-deprecated Vector.
Make sure you still have AccumulableParam imported.
You could make a PR to adjust the example to something that works with
the newer class once you have it working.
On Mon, Jun 9, 2014 at 12:27 PM, dataginjaninja
rickett.stepha...@gmail.com wrote:
New error :-(
scala object VectorAccumulatorParam extends AccumulatorParam[Vector] {
| def zero(initialValue: Vector): Vector = {
| Vector.zeros(initialValue.size)
| }
| def addInPlace(v1: Vector, v2: Vector): Vector = {
| v1 += v2
| }
| }
console:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
Vector.zeros(initialValue.size)
^
console:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
v1 += v2
-
Cheers,
Stephanie
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Emergency maintenace on jenkins
Just a heads up - due to an outage at UCB we've lost several of the Jenkins slaves. I'm trying to spin up new slaves on EC2 in order to compensate, but this might fail some ongoing builds. The good news is if we do get it working with EC2 workers, then we will have burst capability in the future - e.g. on release deadlines. So it's not all bad! - Patrick
