Hi
Thanks, I guess this will solve my problem. I will load mutiple files using
wildcard's likes *.csv. I guess if I use wholeTextFile instead of textFile, I
will get whole file contents as value which will in turn ensure one feature
vector per file.
thanksNitin
> Date: Sat, 4 Jul 2015 09:37:52 -0400
> Subject: Re: Feature Generation On Spark
> From: mici...@gmail.com
> To: rishikeshtha...@hotmail.com
> CC: user@spark.apache.org
>
> Spark Context has a method wholeTextFiles. Is that what you need?
>
> On 4 July 2015 at 07:04, rishikesh <rishikeshtha...@hotmail.com> wrote:
> > Hi
> >
> > I am new to Spark and am working on document classification. Before model
> > fitting I need to do feature generation. Each document is to be converted to
> > a feature vector. However I am not sure how to do that. While testing
> > locally I have a static list of tokens and when I parse a file I do a lookup
> > and increment counters.
> >
> > In the case of Spark I can create an RDD which loads all the documents
> > however I am not sure if one files goes to one executor or multiple. If the
> > file is split then the feature vectors needs to be merged. But I am not able
> > to figure out how to do that.
> >
> > Thanks
> > Rishi
> >
> >
> >
> > --
> > View this message in context:
> > http://apache-spark-user-list.1001560.n3.nabble.com/Feature-Generation-On-Spark-tp23617.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> > For additional commands, e-mail: user-h...@spark.apache.org
> >
