Try this (replace ... with the appropriate values for your environment):
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext
import org.apache.spark.mllib.feature.HashingTF
import org.apache.spark.mllib.linalg.Vector
val sc = new SparkContext(...)
val documents = sc.wholeTextFile(...)
val tokenized = documents.map{ case(path, document) => (path,
document.split("\\s+"))}
val numFeatures = 100000
val hashingTF = new HashingTF(numFeatures)
val featurized = tokenized.map{case(path, words) => (path,
hashingTF.transform(words))}
Mohammed
From: rishikesh thakur [mailto:rishikeshtha...@hotmail.com]
Sent: Friday, July 17, 2015 12:33 AM
To: Mohammed Guller
Subject: Re: Feature Generation On Spark
Thanks I did look at the example. I am using Spark 1.2. The modules mentioned
there are not in 1.2 I guess. The import is failing
Rishi
________________________________
From: Mohammed Guller <moham...@glassbeam.com<mailto:moham...@glassbeam.com>>
Sent: Friday, July 10, 2015 2:31 AM
To: rishikesh thakur; ayan guha; Michal Čizmazia
Cc: user
Subject: RE: Feature Generation On Spark
Take a look at the examples here:
https://spark.apache.org/docs/latest/ml-guide.html
Mohammed
From: rishikesh thakur [mailto:rishikeshtha...@hotmail.com]
Sent: Saturday, July 4, 2015 10:49 PM
To: ayan guha; Michal Čizmazia
Cc: user
Subject: RE: Feature Generation On Spark
I have one document per file and each file is to be converted to a feature
vector. Pretty much like standard feature construction for document
classification.
Thanks
Rishi
________________________________
Date: Sun, 5 Jul 2015 01:44:04 +1000
Subject: Re: Feature Generation On Spark
From: guha.a...@gmail.com<mailto:guha.a...@gmail.com>
To: mici...@gmail.com<mailto:mici...@gmail.com>
CC: rishikeshtha...@hotmail.com<mailto:rishikeshtha...@hotmail.com>;
user@spark.apache.org<mailto:user@spark.apache.org>
Do you have one document per file or multiple document in the file?
On 4 Jul 2015 23:38, "Michal Čizmazia"
<mici...@gmail.com<mailto:mici...@gmail.com>> wrote:
Spark Context has a method wholeTextFiles. Is that what you need?
On 4 July 2015 at 07:04, rishikesh
<rishikeshtha...@hotmail.com<mailto:rishikeshtha...@hotmail.com>> wrote:
> Hi
>
> I am new to Spark and am working on document classification. Before model
> fitting I need to do feature generation. Each document is to be converted to
> a feature vector. However I am not sure how to do that. While testing
> locally I have a static list of tokens and when I parse a file I do a lookup
> and increment counters.
>
> In the case of Spark I can create an RDD which loads all the documents
> however I am not sure if one files goes to one executor or multiple. If the
> file is split then the feature vectors needs to be merged. But I am not able
> to figure out how to do that.
>
> Thanks
> Rishi
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Feature-Generation-On-Spark-tp23617.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
> For additional commands, e-mail:
> user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail:
user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
For additional commands, e-mail:
user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
