Re: Prediction using Classification with text attributes in Apache Spark MLLib

Trying to improve the old solution. Do we have a better text classifier now in Spark Mllib? Regards, lmk -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail:

RE: Prediction using Classification with text attributes in Apache Spark MLLib

Hi, Sorry to bounce back the old thread. What is the state now? Is this problem solved. How spark handle categorical data now? Regards, Ashutosh -- View this message in context:

Re: Prediction using Classification with text attributes in Apache Spark MLLib

This operation requires two transformers: 1) Indexer, which maps string features into categorical features 2) OneHotEncoder, which flatten categorical features into binary features We are working on the new dataset implementation, so we can easily express those transformations. Sorry for late!

RE: Prediction using Classification with text attributes in Apache Spark MLLib

...@gmail.commailto:lakshmi.muralikrish...@gmail.com] Sent: Wednesday, June 25, 2014 1:27 PM To: u...@spark.incubator.apache.orgmailto:u...@spark.incubator.apache.org Subject: RE: Prediction using Classification with text attributes in Apache Spark MLLib Hi Alexander, Just one more question

RE: Prediction using Classification with text attributes in Apache Spark MLLib

...@gmail.com] Sent: Wednesday, June 25, 2014 1:27 PM To: u...@spark.incubator.apache.org Subject: RE: Prediction using Classification with text attributes in Apache Spark MLLib Hi Alexander, Just one more question on a related note. Should I be following the same procedure even if my data

RE: Prediction using Classification with text attributes in Apache Spark MLLib

Hi, You need to convert your text to vector space model: http://en.wikipedia.org/wiki/Vector_space_model and then pass it to SVM. As far as I know, in previous versions of MLlib there was a special class for doing this:

RE: Prediction using Classification with text attributes in Apache Spark MLLib

Hi Alexander, Thanks for your prompt response. Earlier I was executing this Prediction using Weka only. But now we are moving to a huge dataset and hence to Apache Spark MLLib. Is there any other way to convert to libSVM format? Or is there any other simpler algorithm that I can use in mllib?

RE: Prediction using Classification with text attributes in Apache Spark MLLib

: Tuesday, June 24, 2014 3:41 PM To: u...@spark.incubator.apache.org Subject: RE: Prediction using Classification with text attributes in Apache Spark MLLib Hi Alexander, Thanks for your prompt response. Earlier I was executing this Prediction using Weka only. But now we are moving to a huge dataset

Re: Prediction using Classification with text attributes in Apache Spark MLLib

On Tue, Jun 24, 2014 at 12:28 PM, Ulanov, Alexander alexander.ula...@hp.com wrote: You need to convert your text to vector space model: http://en.wikipedia.org/wiki/Vector_space_model and then pass it to SVM. As far as I know, in previous versions of MLlib there was a special class for doing