Re: Emergency maintenace on jenkins

[Henry Saputra]

Thanks for letting us know Patrick.

- Henry

On Monday, June 9, 2014, Patrick Wendell  wrote:

> Just a heads up - due to an outage at UCB we've lost several of the
> Jenkins slaves. I'm trying to spin up new slaves on EC2 in order to
> compensate, but this might fail some ongoing builds.
>
> The good news is if we do get it working with EC2 workers, then we
> will have burst capability in the future - e.g. on release deadlines.
> So it's not all bad!
>
> - Patrick
>

Emergency maintenace on jenkins

[Patrick Wendell]

Just a heads up - due to an outage at UCB we've lost several of the
Jenkins slaves. I'm trying to spin up new slaves on EC2 in order to
compensate, but this might fail some ongoing builds.

The good news is if we do get it working with EC2 workers, then we
will have burst capability in the future - e.g. on release deadlines.
So it's not all bad!

- Patrick

Re: implementing the VectorAccumulatorParam

[Sean Owen]

(BCC dev@)

The example is out of date with respect to current Vector class. The
zeros() method is on "Vectors". There is not currently a += operation
for Vector anymore.

To be fair the example doesn't claim this illustrates use of the Spark
Vector class but it did work with the now-deprecated Vector.

Make sure you still have AccumulableParam imported.

You could make a PR to adjust the example to something that works with
the newer class once you have it working.

On Mon, Jun 9, 2014 at 12:27 PM, dataginjaninja
 wrote:
> New error :-(
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>  |   def zero(initialValue: Vector): Vector = {
>  | Vector.zeros(initialValue.size)
>  |   }
>  |   def addInPlace(v1: Vector, v2: Vector): Vector = {
>  | v1 += v2
>  |   }
>  | }
> :14: error: value zeros is not a member of object
> scala.collection.immutable.Vector
>Vector.zeros(initialValue.size)
>   ^
> :17: error: value += is not a member of
> org.apache.spark.mllib.linalg.Vector
>v1 += v2
>
>
>
> -
> Cheers,
>
> Stephanie
> --
> View this message in context: 
> http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: implementing the VectorAccumulatorParam

[dataginjaninja]

New error :-(

scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
 |   def zero(initialValue: Vector): Vector = {
 | Vector.zeros(initialValue.size)
 |   }
 |   def addInPlace(v1: Vector, v2: Vector): Vector = {
 | v1 += v2
 |   }
 | }
:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
   Vector.zeros(initialValue.size)
  ^
:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
   v1 += v2



-
Cheers,

Stephanie
--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6978.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: implementing the VectorAccumulatorParam

[dataginjaninja]

New error :-(

scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
 |   def zero(initialValue: Vector): Vector = {
 | Vector.zeros(initialValue.size)
 |   }
 |   def addInPlace(v1: Vector, v2: Vector): Vector = {
 | v1 += v2
 |   }
 | }
:12: error: not found: type AccumulatorParam
   object VectorAccumulatorParam extends AccumulatorParam[Vector] {
 ^
:14: error: value zeros is not a member of object
scala.collection.immutable.Vector
   Vector.zeros(initialValue.size)
  ^
:17: error: value += is not a member of
org.apache.spark.mllib.linalg.Vector
   v1 += v2
  ^




-
Cheers,

Stephanie
--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6976.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: implementing the VectorAccumulatorParam

[dataginjaninja]

You are right. I was using the wrong vector class. Thanks.



-
Cheers,

Stephanie
--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973p6975.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: implementing the VectorAccumulatorParam

[Sean Owen]

(The user@ list might be a bit better but I can see why it might look
like a dev@ question.)

Did you import org.apache.spark.mllib.linalg.Vector ? I think you are
picking up Scala's Vector class instead.


On Mon, Jun 9, 2014 at 11:57 AM, dataginjaninja
 wrote:
> The  programming-guide
>    has the
> following:
>
> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>   def zero(initialValue: Vector): Vector = {
> Vector.zeros(initialValue.size)
>   }
>   def addInPlace(v1: Vector, v2: Vector): Vector = {
> v1 += v2
>   }
> }
>
>
> // Then, create an Accumulator of this type:
> val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam)
>
> However, when I try to use this I get an error:
>
> scala> import org.apache.spark.AccumulatorParam
> import org.apache.spark.AccumulatorParam
>
> scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>  |   def zero(initialValue: Vector): Vector = {
>  | Vector.zeros(initialValue.size)
>  |   }
>  |   def addInPlace(v1: Vector, v2: Vector): Vector = {
>  | v1 += v2
>  |   }
>  | }
> :12: error: type Vector takes type parameters
>object VectorAccumulatorParam extends AccumulatorParam[Vector] {
>   ^
>
>
> Last thing, am I posting on the wrong list?
>
>
>
> -
> Cheers,
>
> Stephanie
> --
> View this message in context: 
> http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

implementing the VectorAccumulatorParam

[dataginjaninja]

The  programming-guide
   has the
following:

object VectorAccumulatorParam extends AccumulatorParam[Vector] {
  def zero(initialValue: Vector): Vector = {
Vector.zeros(initialValue.size)
  }
  def addInPlace(v1: Vector, v2: Vector): Vector = {
v1 += v2
  }
}


// Then, create an Accumulator of this type:
val vecAccum = sc.accumulator(new Vector(...))(VectorAccumulatorParam)

However, when I try to use this I get an error:

scala> import org.apache.spark.AccumulatorParam
import org.apache.spark.AccumulatorParam

scala> object VectorAccumulatorParam extends AccumulatorParam[Vector] {
 |   def zero(initialValue: Vector): Vector = {
 | Vector.zeros(initialValue.size)
 |   }
 |   def addInPlace(v1: Vector, v2: Vector): Vector = {
 | v1 += v2
 |   }
 | }
:12: error: type Vector takes type parameters
   object VectorAccumulatorParam extends AccumulatorParam[Vector] {
  ^


Last thing, am I posting on the wrong list?



-
Cheers,

Stephanie
--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6973.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

implementing the VectorAccumulatorParam

[dataginjaninja]

The  programming-guide
   has the
following:

However, when I try to use this I get an error:


Last thing, am I posting on the wrong list?



-
Cheers,

Stephanie
--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/implementing-the-VectorAccumulatorParam-tp6972.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Contributing algorithms to MLlib

[Alex Levin]

Hi,



I'm a M.Sc. computer science student at Tel-Aviv College, Israel (
www.mta.ac.il) and as part of my final project that is dealing with Machine
Learning algorithms in distributed systems,

I would like to contribute couple of algorithms to MLlib.



My advisor, Dr. Uzi Hadad, and I thought of starting with an implementation
of* Fuzzy k - means* algorithm and continuing with *Hidden Markov Model*
algorithm.




Do you know if anyone is currently working on an implementation of these
algorithms for MLlib?







Regards,

Alex