Re: word2vec: how to save an mllib model and reload it?

2015-02-05 Thread Carsten Schnober
As a Spark newbie, I've come across this thread. I'm playing with Word2Vec in our Hadoop cluster and here's my issue with classic Java serialization of the model: I don't have SSH access to the cluster master node. Here's my code for computing the model: val input =

Word2Vec distributed?

2015-07-08 Thread Carsten Schnober
in Spark, has it? I just wonder whether I am interpreting the current situation correctly. Thanks! Carsten [1] https://issues.apache.org/jira/browse/SPARK-2510 -- Carsten Schnober Doctoral Researcher Ubiquitous Knowledge Processing (UKP) Lab FB 20 / Computer Science Department Technische

Word2Vec distributed?

2015-07-10 Thread Carsten Schnober
in Spark, has it? I just wonder whether I am interpreting the current situation correctly. Thanks! Carsten [1] https://issues.apache.org/jira/browse/SPARK-2510 -- Carsten Schnober Doctoral Researcher Ubiquitous Knowledge Processing (UKP) Lab FB 20 / Computer Science Department Technische

K Nearest Neighbours

2015-07-10 Thread Carsten Schnober
)] = ... sc.parallelize(v1).zipWithIndexes .map{ v = (computeDistances(v._1, v2), v._2) } Is there any good practice to approach problems like this? Thanks! Carsten -- Carsten Schnober Doctoral Researcher Ubiquitous Knowledge Processing (UKP) Lab FB 20 / Computer Science Department Technische Universität

Re: MLlib LDA implementation questions

2015-09-11 Thread Carsten Schnober
ementation? > > For reference, my prototype code can be found here: > https://github.com/marko-asplund/tech-protos/blob/master/mllib-lda/src/main/scala/fi/markoa/proto/mllib/LDADemo.scala > > > thanks, > marko -- Carsten Schnober Doctoral Researcher Ubiquitous Knowledge Processing (