1) The feature dimension should be a fixed number before you run
NaiveBayes. If you use bag of words, you need to handle the
word-to-index dictionary by yourself. You can either ignore the words
that never appear in training (because they have no effect in
prediction), or use hashing to randomly project words to a fixed-sized
feature space (collision may happen).
3) Yes, we saved the log conditional probabilities. So to compute the
likelihood, we only need summation.
Best,
Xiangrui
On Tue, Jul 8, 2014 at 12:01 AM, Rahul Bhojwani
<rahulbhojwani2...@gmail.com> wrote:
> I am really sorry. Its actually my mistake. My problem 2 is wrong because
> using a single feature is a senseless thing. Sorry for the inconvenience.
> But still I will be waiting for the solutions for problem 1 and 3.
>
> Thanks,
>
>
> On Tue, Jul 8, 2014 at 12:14 PM, Rahul Bhojwani
> <rahulbhojwani2...@gmail.com> wrote:
>>
>> Hello,
>>
>> I am a novice.I want to classify the text into two classes. For this
>> purpose I want to use Naive Bayes model. I am using Python for it.
>>
>> Here are the problems I am facing:
>>
>> Problem 1: I wanted to use all words as features for the bag of words
>> model. Which means my features will be count of individual words. In this
>> case whenever a new word comes in the test data (which was never present in
>> the train data) I need to increase the size of the feature vector to
>> incorporate that word as well. Correct me if I am wrong. Can I do that in
>> the present Mllib NaiveBayes. Or what is the way in which I can incorporate
>> this?
>>
>> Problem 2: As I was not able to proceed with all words I did some
>> pre-processing and figured out few features from the text. But using this
>> also is giving errors.
>> Right now I was testing for only one feature from the text that is count
>> of positive words. I am submitting the code below, along with the error:
>>
>>
>> #############Code
>>
>> import tokenizer
>> import gettingWordLists as gl
>> from pyspark.mllib.classification import NaiveBayes
>> from numpy import array
>> from pyspark import SparkContext, SparkConf
>>
>> conf = (SparkConf().setMaster("local[6]").setAppName("My
>> app").set("spark.executor.memory", "1g"))
>>
>> sc=SparkContext(conf = conf)
>>
>> # Getting the positive dict:
>> pos_list = []
>> pos_list = gl.getPositiveList()
>> tok = tokenizer.Tokenizer(preserve_case=False)
>>
>>
>> train_data = []
>>
>> with open("training_file.csv","r") as train_file:
>> for line in train_file:
>> tokens = line.split(",")
>> msg = tokens[0]
>> sentiment = tokens[1]
>> count = 0
>> tokens = set(tok.tokenize(msg))
>> for i in tokens:
>> if i.encode('utf-8') in pos_list:
>> count+=1
>> if sentiment.__contains__('NEG'):
>> label = 0.0
>> else:
>> label = 1.0
>> feature = []
>> feature.append(label)
>> feature.append(float(count))
>> train_data.append(feature)
>>
>>
>> model = NaiveBayes.train(sc.parallelize(array(train_data)))
>> print model.pi
>> print model.theta
>> print "\n\n\n\n\n" , model.predict(array([5.0]))
>>
>> ##############
>> This is the output:
>>
>> [-2.24512292 -0.11195389]
>> [[ 0.]
>> [ 0.]]
>>
>>
>>
>>
>>
>> Traceback (most recent call last):
>> File "naive_bayes_analyser.py", line 77, in <module>
>> print "\n\n\n\n\n" , model.predict(array([5.0]))
>> File
>> "F:\spark-0.9.1\spark-0.9.1\python\pyspark\mllib\classification.py", line
>> 101, in predict
>> return numpy.argmax(self.pi + dot(x, self.theta))
>> ValueError: matrices are not aligned
>>
>> ##############
>>
>> Problem 3: As you can see the output for model.pi is -ve. That is prior
>> probabilities are negative. Can someone explain that also. Is it the log of
>> the probability?
>>
>>
>>
>> Thanks,
>> --
>> Rahul K Bhojwani
>> 3rd Year B.Tech
>> Computer Science and Engineering
>> National Institute of Technology, Karnataka
>
>
>
>
> --
> Rahul K Bhojwani
> 3rd Year B.Tech
> Computer Science and Engineering
> National Institute of Technology, Karnataka
