For information purpose, this is the program creating the sequence file : public static void main(String[] args) throws IOException, > InterruptedException, ClassNotFoundException { > Configuration conf = new Configuration(true); > FileSystem fs = FileSystem.get(conf); > > // The input file is not in hdfs > BufferedReader reader = new BufferedReader(new FileReader(args[1])); > Path filePath = new Path(args[2]); > // Delete previous file if exists > if (fs.exists(filePath)) > fs.delete(filePath, true); > SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, > filePath, Text.class, VectorWritable.class); > // Run through the input file > String line; > while ((line = reader.readLine()) != null) { > // We surround with try catch to get rid of the exception when header is > included in file > try { > // Split with the given separator > String[] c = line.split(args[3]); > if (c.length > 1) { > double[] d = new double[c.length]; > // Get the feature set > for (int i = 1; i < c.length; i++) > d[i] = Double.parseDouble(c[i]); > // Put it in a vector > Vector vec = new RandomAccessSparseVector(c.length); > vec.assign(d); > VectorWritable writable = new VectorWritable(); > writable.set(vec); > > // Create a label with a / and the class label > String label = c[0] + "/" + c[0]; > > // Write all in the seqfile > writer.append(new Text(label), writable); > } > } catch (NumberFormatException e) { > continue; > } > } > writer.close(); > reader.close(); > }
2014-02-25 16:25 GMT+01:00 Kevin Moulart <kevinmoul...@gmail.com>: > I finally managed to make it run, I had to format the class label in the > input file with a / in the name so I put Yes/1 or No/0 instead of just 1 or > 0. > > But then I noticed when testing the model that it doesn't classify all the > data : > 14/02/25 16:16:30 INFO mapred.JobClient: Map-Reduce Framework > 14/02/25 16:16:30 INFO mapred.JobClient: Map input records=*300000* > 14/02/25 16:16:30 INFO mapred.JobClient: Map output records=300000 > 14/02/25 16:16:30 INFO mapred.JobClient: Input split bytes=476 > 14/02/25 16:16:30 INFO mapred.JobClient: Spilled Records=0 > 14/02/25 16:16:30 INFO mapred.JobClient: CPU time spent (ms)=32000 > 14/02/25 16:16:30 INFO mapred.JobClient: Physical memory (bytes) > snapshot=834502656 > 14/02/25 16:16:30 INFO mapred.JobClient: Virtual memory (bytes) > snapshot=3738030080 > 14/02/25 16:16:30 INFO mapred.JobClient: Total committed heap usage > (bytes)=918552576 > 14/02/25 16:16:31 INFO test.TestNaiveBayesDriver: Standard NB Results: > ======================================================= > Summary > ------------------------------------------------------- > Correctly Classified Instances : 36078 91.3552% > Incorrectly Classified Instances : 3414 8.6448% > Total Classified Instances : *39492* > > ======================================================= > Confusion Matrix > ------------------------------------------------------- > a b <--Classified as > 34445 2114 | 36559 a = 0 > 1300 1633 | 2933 b = 1 > > > I did the testnb with the exact same file I used to train the model. > > Any idea ? > > > 2014-02-25 11:33 GMT+01:00 Kevin Moulart <kevinmoul...@gmail.com>: > > All right I've manage to narrow it down to the LabelIndex, I went to see >> the code but it isnt realy clear at all for me. What exactly should I >> provide as Label Index ? >> >> As a reminder, one line of my original file i=looks like : >> 0, 0.3222, 0, 1.543, ... >> 1, 0, 1.42, 1.12, ... >> >> With the 0, 1 being the labels I'm trying to learn and the rest being the >> data. >> >> For now I have the previously mentionned java code that creates the >> SequenceFile from my CSV, but when I then try to run the trainnb on it it >> tries to create a LabelIndex and fails with an ArrayOutOfBoundException: 1. >> >> Could someone tell me how to create the index, even manually at this >> point ? >> >> Thanks in advance ! >> >> >> 2014-02-24 15:41 GMT+01:00 Kevin Moulart <kevinmoul...@gmail.com>: >> >> I'll do that as soon as I manage to make it work ^^', that's a great idea >>> ! >>> >>> I'm stuck with this for now : >>> >>> public static void main(String[] args) throws IOException, >>>> InterruptedException, ClassNotFoundException { >>>> Configuration conf = new Configuration(true); >>>> FileSystem fs = FileSystem.get(conf); >>>> BufferedReader reader = new BufferedReader(new FileReader(args[1])); >>>> Path filePath = new Path(args[2]); >>>> if (fs.exists(filePath)) >>>> fs.delete(filePath, true); >>>> SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, >>>> filePath, Text.class, VectorWritable.class); >>>> try { >>>> String line; >>>> while ((line = reader.readLine()) != null) { >>>> String[] c = line.split(args[3]); >>>> if (c.length > 1) { >>>> double[] d = new double[c.length]; >>>> for (int i = 1; i < c.length; i++) >>>> d[i] = Double.parseDouble(c[i]); >>>> Vector vec = new RandomAccessSparseVector(c.length); >>>> vec.assign(d); >>>> VectorWritable writable = new VectorWritable(); >>>> writable.set(vec); >>>> writer.append(new Text(c[0]), writable); >>>> } >>>> } >>>> writer.close(); >>>> } catch (Throwable t) { >>>> t.printStackTrace(); >>>> } >>>> reader.close(); >>>> } >>> >>> >>> Which produces a sequence file but Mahout's trainnb doesn't seem to like >>> it that much, so I'm working on it for the moment. >>> >>> >>> 2014-02-24 15:37 GMT+01:00 Ted Dunning <ted.dunn...@gmail.com>: >>> >>> Kevin, >>>> >>>> While this is fresh in your mind can you prepare a javadoc patch that >>>> would >>>> have helped you out? And suggest other doc patches as well? >>>> >>>> >>>> >>>> On Mon, Feb 24, 2014 at 3:00 AM, Kevin Moulart <kevinmoul...@gmail.com >>>> >wrote: >>>> >>>> > Thanks, that's about the clearest answer I got so far :) >>>> > >>>> > >>>> > 2014-02-24 11:59 GMT+01:00 Sebastian Schelter <s...@apache.org>: >>>> > >>>> > > NaiveBayes expects a SequenceFile as input. The key is the class >>>> label as >>>> > > Text, the value are the features as VectorWritable. >>>> > > >>>> > > --sebastian >>>> > > >>>> > > >>>> > > On 02/24/2014 11:51 AM, Kevin Moulart wrote: >>>> > > >>>> > >> Hi again, >>>> > >> I finally set my mind on going through java to make a sequence >>>> file for >>>> > >> the >>>> > >> naive bayes, >>>> > >> but I still can't manage to find anyplace stating exactly what >>>> should be >>>> > >> in >>>> > >> the sequence file >>>> > >> for mahout to process it with Naive Bayes. >>>> > >> >>>> > >> I tried virtually every piece of code i found related to this >>>> subject, >>>> > >> with >>>> > >> no luck. >>>> > >> >>>> > >> My CSV file is like this : >>>> > >> Label that I want to predict, feature 1, feature 2, ..., feature >>>> 1628 >>>> > >> >>>> > >> Could someone tell me exactly what Naive Bayes training procedure >>>> > expects >>>> > >> ? >>>> > >> >>>> > >> >>>> > >> 2014-02-20 13:56 GMT+01:00 Jay Vyas <jayunit...@gmail.com>: >>>> > >> >>>> > >> This relates to a previous question I have: Does mahout have a >>>> concept >>>> > >>> of >>>> > >>> adapters which allow us to read data csv style data with filters >>>> to >>>> > >>> create >>>> > >>> exact format for its various inputs (i.e. Recommender three >>>> column >>>> > >>> format).? If not is it worth a jira? >>>> > >>> >>>> > >>> >>>> > >>> On Feb 20, 2014, at 7:50 AM, Kevin Moulart < >>>> kevinmoul...@gmail.com> >>>> > >>>> >>>> > >>> wrote: >>>> > >>> >>>> > >>>> >>>> > >>>> Hi and thanks ! >>>> > >>>> >>>> > >>>> What about the command line, is there a way to do that using the >>>> > >>>> existing >>>> > >>>> command line ? >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> 2014-02-20 12:02 GMT+01:00 Suneel Marthi < >>>> suneel_mar...@yahoo.com>: >>>> > >>>> >>>> > >>>> To convert input CSV to vectors, u can either: >>>> > >>>>> >>>> > >>>>> a) Use CSVIterator >>>> > >>>>> b) use InputDriver >>>> > >>>>> >>>> > >>>>> Either of the above should generate vectors from input CSV that >>>> could >>>> > >>>>> >>>> > >>>> then >>>> > >>> >>>> > >>>> be fed into Mahout classifier/clustering jobs. >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> On Thursday, February 20, 2014 5:57 AM, Kevin Moulart < >>>> > >>>>> kevinmoul...@gmail.com> wrote: >>>> > >>>>> >>>> > >>>>> Hi I'm trying to apply a Naive Bayes Classifier to a large CSV >>>> file >>>> > >>>>> from >>>> > >>>>> the command line. >>>> > >>>>> >>>> > >>>>> I know I have to feed the classifier with a seq file, so I >>>> tried to >>>> > put >>>> > >>>>> >>>> > >>>> my >>>> > >>> >>>> > >>>> csv into one using the command seqdirectory, but even when I try >>>> with >>>> > a >>>> > >>>>> really small csv (less than 100Mo) I instantly get an >>>> > >>>>> >>>> > >>>> outOfMemoryException >>>> > >>> >>>> > >>>> from java heap space : >>>> > >>>>> >>>> > >>>>> mahout seqdirectory -i "/user/cacf/Echant/testSeq" -o >>>> > >>>>> >>>> > >>>> "/user/cacf/resSeq" >>>> > >>> >>>> > >>>> -ow >>>> > >>>>>> MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. >>>> > >>>>>> Running on hadoop, using >>>> > >>>>>> >>>> > >>>>> /opt/cloudera/parcels/CDH/lib/hadoop/bin/hadoop >>>> > >>> >>>> > >>>> and HADOOP_CONF_DIR=/etc/hadoop/conf >>>> > >>>>>> MAHOUT-JOB: >>>> /usr/lib/mahout/mahout-examples-0.7-cdh4.5.0-job.jar >>>> > >>>>>> 14/02/20 11:47:22 INFO common.AbstractJob: Command line >>>> arguments: >>>> > >>>>>> {--charset=[UTF-8], --chunkSize=[64], --endPhase=[2147483647], >>>> > >>>>>> >>>> --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter], >>>> > >>>>>> --input=[/user/cacf/Echant/testSeq], --keyPrefix=[], >>>> > >>>>>> --output=[/user/cacf/resSeq], >>>> > >>>>>> >>>> > >>>>> --overwrite=null, --startPhase=[0], >>>> > >>>>> >>>> > >>>>>> --tempDir=[temp]} >>>> > >>>>>> 14/02/20 11:47:22 INFO common.HadoopUtil: Deleting >>>> /user/cacf/resSeq >>>> > >>>>>> Exception in thread "main" java.lang.OutOfMemoryError: Java >>>> heap >>>> > space >>>> > >>>>>> at java.util.Arrays.copyOf(Arrays.java:2367) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> java.lang.AbstractStringBuilder.expandCapacity( >>>> > >>> AbstractStringBuilder.java:130) >>>> > >>> >>>> > >>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> java.lang.AbstractStringBuilder.ensureCapacityInternal( >>>> > >>> AbstractStringBuilder.java:114) >>>> > >>> >>>> > >>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >>>> > >>>>> >>>> > >>>>>> at java.lang.StringBuilder.append(StringBuilder.java:132) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> org.apache.mahout.text.PrefixAdditionFilter.process( >>>> > >>> PrefixAdditionFilter.java:62) >>>> > >>> >>>> > >>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> org.apache.mahout.text.SequenceFilesFromDirectoryFilter.accept( >>>> > >>> SequenceFilesFromDirectoryFilter.java:90) >>>> > >>> >>>> > >>>> at >>>> org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1468) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1502) >>>> > >>>>> >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> org.apache.mahout.text.SequenceFilesFromDirectory.run( >>>> > >>> SequenceFilesFromDirectory.java:98) >>>> > >>> >>>> > >>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>> > >>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> org.apache.mahout.text.SequenceFilesFromDirectory.main( >>>> > >>> SequenceFilesFromDirectory.java:53) >>>> > >>> >>>> > >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> sun.reflect.NativeMethodAccessorImpl.invoke( >>>> > >>> NativeMethodAccessorImpl.java:57) >>>> > >>> >>>> > >>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke( >>>> > >>> DelegatingMethodAccessorImpl.java:43) >>>> > >>> >>>> > >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke( >>>> > >>> ProgramDriver.java:72) >>>> > >>> >>>> > >>>> at >>>> > >>>>>> >>>> > >>>>> >>>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) >>>> > >>>>> >>>> > >>>>>> at >>>> org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:196) >>>> > >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> > >>>>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> sun.reflect.NativeMethodAccessorImpl.invoke( >>>> > >>> NativeMethodAccessorImpl.java:57) >>>> > >>> >>>> > >>>> at >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke( >>>> > >>> DelegatingMethodAccessorImpl.java:43) >>>> > >>> >>>> > >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> > >>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208) >>>> > >>>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> Do you have an idea or a simple way to use Naive Bayes against >>>> my >>>> > large >>>> > >>>>> >>>> > >>>> CSV >>>> > >>> >>>> > >>>> ? >>>> > >>>>> >>>> > >>>>> Thanks in advance ! >>>> > >>>>> -- >>>> > >>>>> Kévin Moulart >>>> > >>>>> GSM France : +33 7 81 06 10 10 >>>> > >>>>> GSM Belgique : +32 473 85 23 85 >>>> > >>>>> Téléphone fixe : +32 2 771 88 45 >>>> > >>>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> -- >>>> > >>>> Kévin Moulart >>>> > >>>> GSM France : +33 7 81 06 10 10 >>>> > >>>> GSM Belgique : +32 473 85 23 85 >>>> > >>>> Téléphone fixe : +32 2 771 88 45 >>>> > >>>> >>>> > >>> >>>> > >>> >>>> > >> >>>> > >> >>>> > >> >>>> > > >>>> > >>>> > >>>> > -- >>>> > Kévin Moulart >>>> > GSM France : +33 7 81 06 10 10 >>>> > GSM Belgique : +32 473 85 23 85 >>>> > Téléphone fixe : +32 2 771 88 45 >>>> > >>>> >>> >>> >>> >>> -- >>> Kévin Moulart >>> GSM France : +33 7 81 06 10 10 >>> GSM Belgique : +32 473 85 23 85 >>> Téléphone fixe : +32 2 771 88 45 >>> >> >> >> >> -- >> Kévin Moulart >> GSM France : +33 7 81 06 10 10 >> GSM Belgique : +32 473 85 23 85 >> Téléphone fixe : +32 2 771 88 45 >> > > > > -- > Kévin Moulart > GSM France : +33 7 81 06 10 10 > GSM Belgique : +32 473 85 23 85 > Téléphone fixe : +32 2 771 88 45 > -- Kévin Moulart GSM France : +33 7 81 06 10 10 GSM Belgique : +32 473 85 23 85 Téléphone fixe : +32 2 771 88 45