I've been trying to implement cosine similarity using Spark and stumbled upon this article https://databricks.com/blog/2014/10/20/efficient-similarity-algorithm-now-in-spark-twitter.html The only problem I have with it is that it seems that they assume that in my input file each *column* is a separate tweet, hence by computing column similarity we will find similar tweets. Hope I understood it correctly?
Since it's a Twitter based method I guess there has to be some logic behind it but how would one go around preparing such a dataset? Personally after doing TF-IDF on a document I get an RDD of Vectors which I can then transform into a RowMatrix but this RowMatrix has each document as a separate row not column hence the aforementioned method won't really help me. I tried looking for some transpose methods but failed to find any. Am I missing something or just misusing the available methods? Mateusz -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-DIMSUM-row-similarity-tp24518.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
