[
https://issues.apache.org/jira/browse/MAHOUT-60?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Anil updated MAHOUT-60:
-----------------------------
Attachment: MAHOUT-60.patch
There are a lot of changes in this patch. Most of the Files have been
renamed. The trainer is now a bunch of *5 Map Reduce jobs*. The exact
functionality of each job is as follows. The trainer can support *any number of
maps and any number of reduces*. Also i am using Apache Lang library
commons-lang-2.4.jar ( which should be put in the classpath)
{noformat}
//Read the features in each document normalized by length of each document
CBayesFeatureDriver.runJob(input, output);
//Calculate the TfIdf for each word in each label
CBayesTfIdfDriver.runJob(input, output);
//Calculate the Sums of weights for each label, for each feature and for
each feature and for each label
CBayesWeightSummerDriver.runJob(input, output);
//Calculate the W_ij = log(Theta) for each label, feature. This step
actually generates the complement class
CBayesThetaDriver.runJob(input, output);
//Calculate the normalization factor Sigma_W_ij for each complement
class.
CBayesThetaNormalizerDriver.runJob(input, output);
{noformat}
I have tested it on a 6 system cluster. On 20 newsgroups dataset, it takes
around 4 minutes to train. It just used to take 20-30 seconds when creating
the CNB model in-memory. But the design is based on the assumption that the
datasets are going to be too huge to fit into memory.
There can be a lot of speed improvement if the Map-Reduce operations can be
somehow chained.
So Instead of Map1 -> Reduce1 - > Map1 -> Reduce2....
if it is possible to do. Map1 -> Reduce1 - > Reduce2 -> Reduce3 ->... then we
could save a lot of time on IO. I am not sure if such a functionality exists in
hadoop
I will test it out on Dmoz or Wikipedia dataset (if i can preprocess it within
a reasonable amount of time)
The other changes are that there is no longer a model file. The model is
stored in multiple part files in the folders trainer-theta and
trainer-thetaNormalizer
To Train
{noformat}
$bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar
org.apache.mahout.examples.classifiers.cbayes.TrainTwentyNewsgroups -t -i
20newsinput -o 20newsoutput
{noformat}
To Test
{noformat}
$bin/hadoop jar <MAHOUT_HOME>/build/apache-mahout-0.1-dev-ex.jar
org.apache.mahout.examples.classifiers.cbayes.TestTwentyNewsgroups -p
20newsoutput -t work/20news-18828
{noformat}
Next Step, to make the Classifier and the Testing completely Map Reduce.
> Complementary Naive Bayes
> -------------------------
>
> Key: MAHOUT-60
> URL: https://issues.apache.org/jira/browse/MAHOUT-60
> Project: Mahout
> Issue Type: Sub-task
> Components: Classification
> Reporter: Robin Anil
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 0.1
>
> Attachments: MAHOUT-60.patch, MAHOUT-60.patch, MAHOUT-60.patch
>
>
> The focus is to implement an improved text classifier based on this paper
> http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
