Thanks for the response. Appreciate the help! Burke
On Tue, Oct 14, 2014 at 3:00 PM, Xiangrui Meng <[email protected]> wrote: > You cannot recover the document from the TF-IDF vector, because > HashingTF is not reversible. You can assign each document a unique ID, > and join back the result after training. HasingTF can transform > individual record: > > val docs: RDD[(String, Seq[String])] = ... > > val tf = new HashingTF() > val tfWithId: RDD[(String, Vector)] = docs.mapValues(tf.transform) > > ... > > Best, > Xiangrui > > On Tue, Oct 14, 2014 at 9:15 AM, Burke Webster <[email protected]> > wrote: > > I'm following the Mllib example for TF-IDF and ran into a problem due to > my > > lack of knowledge of Scala and spark. Any help would be greatly > > appreciated. > > > > Following the Mllib example I could do something like this: > > > > import org.apache.spark.rdd.RDD > > import org.apache.spark.SparkContext > > import org.apache.spark.mllib.feature.HashingTF > > import org.apache.spark.mllib.linalg.Vector > > import org.apache.spark.mllib.feature.IDF > > > > val sc: SparkContext = ... > > val documents: RDD[Seq[String]] = sc.textFile("...").map(_.split(" > ").toSeq) > > > > val hashingTF = new HashingTF() > > val tf: RDD[Vector] = hasingTF.transform(documents) > > tf.cache() > > > > val idf = new IDF().fit(tf) > > val tfidf: RDD[Vector] = idf.transform(tf) > > > > As a result I would have an RDD containing the TF-IDF vectors for the > input > > documents. My question is how do I map the vector back to the original > > input document? > > > > My end goal is to compute document similarity using cosine similarity. > From > > what I can tell, I can compute TF-IDF, apply the L2 norm, and then > compute > > the dot-product. Has anybody done this? > > > > Currently, my example looks more like this: > > > > import org.apache.spark.SparkContext._ > > import org.apache.spark.SparkConf > > import org.apache.spark.mllib.feature.HashingTF > > import org.apache.spark.mllib.feature.IDF > > import org.apache.spark.mllib.linalg.Vector > > import org.apache.spark.rdd.RDD > > import org.apache.spark.SparkContext > > > > val sc: SparkContext = ... > > > > // input is sequence file of the form (docid: Text, content: Text) > > val data: RDD[(String, String)] = sc.sequenceFile[String, > String](“corpus”) > > > > val docs: RDD[(String, Seq[String])] = data.mapValues(v => v.split(" > > ").toSeq) > > > > val hashingTF = new HashingTF() > > val tf: RDD[(String, Vector)] = hashingTF.?? > > > > I'm trying to maintain some linking from the document identifier to it's > > eventual vertex representation. I'm I going about this incorrectly? > > > > Thanks >
