I have ended up with the following piece of code but is turns out to be
really slow... Any other ideas provided that I can only use MLlib 1.2?
val data = test11.map(x= ((x(0) , x(1)) , x(2))).groupByKey().map(x=
(x._1 , x._2.toArray)).map{x=
var lt : Array[Double] = new Array[Double](test12.size)
val id = x._1._1
val cl = x._1._2
val dt = x._2
var i = -1
test12.foreach{y = i += 1; lt(i) = if(dt contains y) 1.0 else 0.0}
val vs = Vectors.dense(lt)
(id , cl , vs)
}
*// Adamantios*
On Fri, Aug 7, 2015 at 8:36 AM, Yanbo Liang yblia...@gmail.com wrote:
I think you want to flatten the 1M products to a vector of 1M elements, of
course mostly are zero.
It looks like HashingTF
https://spark.apache.org/docs/latest/ml-features.html#tf-idf-hashingtf-and-idf
can help you.
2015-08-07 11:02 GMT+08:00 praveen S mylogi...@gmail.com:
Use StringIndexer in MLib1.4 :
https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/ml/feature/StringIndexer.html
On Thu, Aug 6, 2015 at 8:49 PM, Adamantios Corais
adamantios.cor...@gmail.com wrote:
I have a set of data based on which I want to create a classification
model. Each row has the following form:
user1,class1,product1
user1,class1,product2
user1,class1,product5
user2,class1,product2
user2,class1,product5
user3,class2,product1
etc
There are about 1M users, 2 classes, and 1M products. What I would like
to do next is create the sparse vectors (something already supported by
MLlib) BUT in order to apply that function I have to create the dense
vectors
(with the 0s), first. In other words, I have to binarize my data. What's
the easiest (or most elegant) way of doing that?
*// Adamantios*
