Yes, open an issue for the name samples, that should be fixed. Jörn
On Mon, Mar 6, 2017 at 2:17 PM, Damiano Porta <damianopo...@gmail.com> wrote: > I have to redesign it, reading the wiki you gave me i have noticed that i > should not create two partitions (one for trainiing and one for testing). > It avoids overfitting, so i will pass all the data! > Thanks Jorn! > > P.S. Did you read my previous email about the bug in namesamples? Should i > open an issue? > > 2017-03-06 13:43 GMT+01:00 Damiano Porta <damianopo...@gmail.com>: > > > Oh I see. Thanks! > > > > Basically i have 30k sentences i apply the labels with a script and then > i > > pass 0-15k to train the model (to build the .bin) and 15k-30k to evaluate > > it. > > > > I am trying to build the model with 300 iterations again. > > > > 2017-03-06 13:31 GMT+01:00 Joern Kottmann <kottm...@gmail.com>: > > > >> You should understand how it works, have a look at this wikipedia > article, > >> the picture on the right side explains it quite nicely. > >> https://en.wikipedia.org/wiki/Cross-validation_(statistics) > >> > >> The idea is to split the data into n partitions and then use n-1 for > >> training and 1 for testing, this is repeated n times, so that each > >> partition was once used for testing. > >> > >> It really should be three times as long in your case, maybe there is > >> something else wrong?' > >> > >> Jörn > >> > >> On Mon, Mar 6, 2017 at 12:36 PM, Damiano Porta <damianopo...@gmail.com> > >> wrote: > >> > >> > Unfortunately not, 100 iterations ~ 30 minutes 300 iterations > 2 days > >> and > >> > it is still running... i will block it > >> > > >> > i still do not understand what number should i set as *folds*. Ok i > will > >> > set a number > 1 but, should i have to pay more attention to this > >> > parameter? if i set 8 or 10 does it matter anything? > >> > > >> > > >> > > >> > 2017-03-06 12:19 GMT+01:00 Joern Kottmann <kottm...@gmail.com>: > >> > > >> > > test.evaluate(samples, 1), here the second parameter is the number > of > >> > > folds, usually you use 10 or a number larger than 1. > >> > > > >> > > The amount of times you need for training with perceptron is linear > to > >> > the > >> > > iterations, if you use 300 instead of 100 it should take three times > >> as > >> > > long. > >> > > > >> > > Jörn > >> > > > >> > > On Mon, Mar 6, 2017 at 11:12 AM, Damiano Porta < > >> damianopo...@gmail.com> > >> > > wrote: > >> > > > >> > > > Jorn, > >> > > > I am training and testing the model via api. If it is not a > training > >> > > > problem. How is that possible that the evaluation is taking 2 days > >> (and > >> > > > still running) to evaluate the model? As i told you with 100 > >> > iterations i > >> > > > can get the model and the test in ~30 minutes. > >> > > > > >> > > > I only have a doubt about evaluation, this is the code: > >> > > > > >> > > > try (ObjectStream<NameSample> samples = > >> > > > ObjectStreamUtils.createObjectStream(evaluation)) { > >> > > > > >> > > > TrainingParameters mlParams = new > TrainingParameters(); > >> > > > mlParams.put(TrainingParameters.ALGORITHM_PARAM, > >> > > > PerceptronTrainer.PERCEPTRON_VALUE); > >> > > > mlParams.put(TrainingParameters.ITERATIONS_PARAM, > >> > > > Integer.toString(100)); > >> > > > mlParams.put(TrainingParameters.CUTOFF_PARAM, > >> > > > Integer.toString(0)); > >> > > > > >> > > > TokenNameFinderCrossValidator test = new > >> > > > TokenNameFinderCrossValidator("it", > >> > > > null, mlParams, null, > >> > > > (TokenNameFinderEvaluationMonitor)null); > >> > > > > >> > > > test.evaluate(samples, 1); *// <---- SECOND PARAMETER > >> HERE* > >> > > > > >> > > > FMeasure result = test.getFMeasure(); > >> > > > > >> > > > System.out.println(result.toString()); > >> > > > } > >> > > > > >> > > > What should i put on the second parameter of test.evaluate() ? > Each > >> > > sample > >> > > > (in samples variable) represents a document. There are no > relations > >> > with > >> > > > other samples. > >> > > > > >> > > > 2017-03-06 10:56 GMT+01:00 Joern Kottmann <kottm...@gmail.com>: > >> > > > > >> > > > > Hello, > >> > > > > > >> > > > > the model is only available after the training finished, hard to > >> > guess > >> > > > what > >> > > > > you are doing. > >> > > > > > >> > > > > Do you use the command line? Which command? > >> > > > > > >> > > > > Jörn > >> > > > > > >> > > > > On Mon, Mar 6, 2017 at 10:29 AM, Damiano Porta < > >> > damianopo...@gmail.com > >> > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Hello Jorn, > >> > > > > > I tried with 300 iterations and it takes forever, reducing > that > >> > > number > >> > > > to > >> > > > > > 100 i can finally get the model in half an hour. > >> > > > > > > >> > > > > > The problem with 300 iterations is that i can see the model > >> (.bin) > >> > in > >> > > > > half > >> > > > > > an hour too but the computations are still running. So i do > not > >> > > really > >> > > > > > understand what it is doing. > >> > > > > > > >> > > > > > Damiano > >> > > > > > > >> > > > > > 2017-03-06 10:19 GMT+01:00 Joern Kottmann <kottm...@gmail.com > >: > >> > > > > > > >> > > > > > > Hello, > >> > > > > > > > >> > > > > > > this looks like output from the cross validator. > >> > > > > > > > >> > > > > > > Jörn > >> > > > > > > > >> > > > > > > On Sun, Mar 5, 2017 at 11:34 AM, Damiano Porta < > >> > > > damianopo...@gmail.com > >> > > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hello, > >> > > > > > > > > >> > > > > > > > I am training a NER model with perceptron classifier > (using > >> > > OpenNLP > >> > > > > > > 1.7.0) > >> > > > > > > > > >> > > > > > > > the output of the training is: > >> > > > > > > > > >> > > > > > > > Indexing events using cutoff of 0 > >> > > > > > > > > >> > > > > > > > Computing event counts... done. 11861603 events > >> > > > > > > > Indexing... done. > >> > > > > > > > Collecting events... Done indexing. > >> > > > > > > > Incorporating indexed data for training... > >> > > > > > > > done. > >> > > > > > > > Number of Event Tokens: 11861603 > >> > > > > > > > Number of Outcomes: 23 > >> > > > > > > > Number of Predicates: 6623489 > >> > > > > > > > Computing model parameters... > >> > > > > > > > Performing 300 iterations. > >> > > > > > > > 1: . (11795234/11861603) 0.9944047191597966 > >> > > > > > > > 2: . (11820243/11861603) 0.9965131188423689 > >> > > > > > > > 3: . (11829329/11861603) 0.9972791198626357 > >> > > > > > > > 4: . (11834935/11861603) 0.9977517372651908 > >> > > > > > > > 5: . (11838996/11861603) 0.9980941024581584 > >> > > > > > > > 6: . (11841501/11861603) 0.9983052880795286 > >> > > > > > > > 7: . (11843704/11861603) 0.998491013398442 > >> > > > > > > > 8: . (11845304/11861603) 0.9986259024180796 > >> > > > > > > > 9: . (11846421/11861603) 0.9987200718149141 > >> > > > > > > > 10: . (11847181/11861603) 0.9987841440992419 > >> > > > > > > > 20: . (11852226/11861603) 0.9992094660392866 > >> > > > > > > > 30: . (11853947/11861603) 0.9993545560410343 > >> > > > > > > > 40: . (11854831/11861603) 0.999429082224384 > >> > > > > > > > 50: . (11855471/11861603) 0.999483037832239 > >> > > > > > > > Stopping: change in training set accuracy less than 1.0E-5 > >> > > > > > > > Stats: (11846242/11861603) 0.998704981105842 > >> > > > > > > > ...done. > >> > > > > > > > Compressed 6623489 parameters to 554312 > >> > > > > > > > 6892 outcome patterns > >> > > > > > > > Indexing events using cutoff of 0 > >> > > > > > > > > >> > > > > > > > Computing event counts... done. 6370206 events > >> > > > > > > > Indexing... done. > >> > > > > > > > Collecting events... Done indexing. > >> > > > > > > > Incorporating indexed data for training... > >> > > > > > > > done. > >> > > > > > > > Number of Event Tokens: 6370206 > >> > > > > > > > Number of Outcomes: 23 > >> > > > > > > > Number of Predicates: 3737425 > >> > > > > > > > Computing model parameters... > >> > > > > > > > Performing 300 iterations. > >> > > > > > > > 1: . (6330365/6370206) 0.9937457281601254 > >> > > > > > > > 2: . (6345859/6370206) 0.9961779885925196 > >> > > > > > > > 3: . (6351552/6370206) 0.9970716802564941 > >> > > > > > > > 4: . (6354847/6370206) 0.9975889319748843 > >> > > > > > > > 5: . (6356872/6370206) 0.997906818084062 > >> > > > > > > > 6: . (6358350/6370206) 0.998138835698563 > >> > > > > > > > 7: . (6359611/6370206) 0.9983367884806237 > >> > > > > > > > 8: . (6360473/6370206) 0.9984721059256169 > >> > > > > > > > 9: . (6361138/6370206) 0.9985764981540628 > >> > > > > > > > 10: . (6361532/6370206) 0.9986383485871572 > >> > > > > > > > 20: . (6364161/6370206) 0.9990510510963068 > >> > > > > > > > 30: . (6365106/6370206) 0.9991993979472563 > >> > > > > > > > Stopping: change in training set accuracy less than 1.0E-5 > >> > > > > > > > Stats: (6360617/6370206) 0.9984947111600473 > >> > > > > > > > ...done. > >> > > > > > > > Indexing events using cutoff of 0 > >> > > > > > > > > >> > > > > > > > Computing event counts... done. 6370114 events > >> > > > > > > > Indexing... done. > >> > > > > > > > Collecting events... Done indexing. > >> > > > > > > > Incorporating indexed data for training... > >> > > > > > > > done. > >> > > > > > > > Number of Event Tokens: 6370114 > >> > > > > > > > Number of Outcomes: 23 > >> > > > > > > > Number of Predicates: 3737390 > >> > > > > > > > Computing model parameters... > >> > > > > > > > Performing 300 iterations. > >> > > > > > > > 1: . (6330266/6370114) 0.9937445389517362 > >> > > > > > > > 2: . (6345810/6370114) 0.9961846836650019 > >> > > > > > > > 3: . (6351374/6370114) 0.9970581374210885 > >> > > > > > > > 4: . (6354747/6370114) 0.9975876412886803 > >> > > > > > > > 5: . (6356872/6370114) 0.9979212302950936 > >> > > > > > > > 6: . (6358429/6370114) 0.998165652922381 > >> > > > > > > > 7: . (6359417/6370114) 0.9983207521874805 > >> > > > > > > > 8: . (6360292/6370114) 0.9984581123665919 > >> > > > > > > > 9: . (6361076/6370114) 0.9985811870870757 > >> > > > > > > > 10: . (6361693/6370114) 0.998678045636232 > >> > > > > > > > 20: . (6364109/6370114) 0.9990573167136413 > >> > > > > > > > 30: . (6365008/6370114) 0.9991984444862368 > >> > > > > > > > 40: . (6365478/6370114) 0.9992722265253023 > >> > > > > > > > Stopping: change in training set accuracy less than 1.0E-5 > >> > > > > > > > Stats: (6359985/6370114) 0.9984099185666065 > >> > > > > > > > ...done. > >> > > > > > > > Indexing events using cutoff of 0 > >> > > > > > > > > >> > > > > > > > Computing event counts... done. 6370480 events > >> > > > > > > > Indexing... done. > >> > > > > > > > Collecting events... Done indexing. > >> > > > > > > > Incorporating indexed data for training... > >> > > > > > > > done. > >> > > > > > > > Number of Event Tokens: 6370480 > >> > > > > > > > Number of Outcomes: 23 > >> > > > > > > > Number of Predicates: 3737798 > >> > > > > > > > Computing model parameters... > >> > > > > > > > Performing 300 iterations. > >> > > > > > > > 1: . (6330685/6370480) 0.9937532179678769 > >> > > > > > > > 2: . (6346153/6370480) 0.9961812924614786 > >> > > > > > > > 3: . (6351726/6370480) 0.9970561088018485 > >> > > > > > > > 4: . (6355089/6370480) 0.9975840125076917 > >> > > > > > > > 5: . (6357173/6370480) 0.9979111464128292 > >> > > > > > > > 6: . (6358780/6370480) 0.9981634036995642 > >> > > > > > > > 7: . (6359845/6370480) 0.9983305810551167 > >> > > > > > > > 8: . (6360827/6370480) 0.9984847295651191 > >> > > > > > > > 9: . (6361316/6370480) 0.9985614898720347 > >> > > > > > > > 10: . (6362076/6370480) 0.9986807901445417 > >> > > > > > > > 20: . (6364506/6370480) 0.9990622370684784 > >> > > > > > > > 30: . (6365415/6370480) 0.9992049264733583 > >> > > > > > > > Stopping: change in training set accuracy less than 1.0E-5 > >> > > > > > > > Stats: (6362594/6370480) 0.9987621026986977 > >> > > > > > > > ...done. > >> > > > > > > > Indexing events using cutoff of 0 > >> > > > > > > > > >> > > > > > > > Computing event counts... done. 6370008 events > >> > > > > > > > Indexing... done. > >> > > > > > > > Collecting events... Done indexing. > >> > > > > > > > Incorporating indexed data for training... > >> > > > > > > > done. > >> > > > > > > > Number of Event Tokens: 6370008 > >> > > > > > > > Number of Outcomes: 23 > >> > > > > > > > Number of Predicates: 3737824 > >> > > > > > > > Computing model parameters... > >> > > > > > > > Performing 300 iterations. > >> > > > > > > > 1: . (6330200/6370008) 0.9937507142848172 > >> > > > > > > > 2: . (6345643/6370008) 0.9961750440501802 > >> > > > > > > > 3: . (6351415/6370008) 0.9970811653611737 > >> > > > > > > > 4: . (6354522/6370008) 0.9975689198506501 > >> > > > > > > > 5: . (6356723/6370008) 0.9979144453193779 > >> > > > > > > > 6: . (6358164/6370008) 0.9981406616757781 > >> > > > > > > > 7: . (6359399/6370008) 0.9983345389833106 > >> > > > > > > > 8: . (6360274/6370008) 0.9984719014481614 > >> > > > > > > > 9: . (6360694/6370008) 0.9985378354312899 > >> > > > > > > > 10: . (6361531/6370008) 0.9986692324405244 > >> > > > > > > > .... > >> > > > > > > > .... > >> > > > > > > > .... > >> > > > > > > > > >> > > > > > > > etc etc is that normal ? The parameters are; *0 cutoff* > and > >> > *300 > >> > > > > > > > iterators*. > >> > > > > > > > > >> > > > > > > > The corpus is relative small, it has 20k sentences. > >> > > > > > > > > >> > > > > > > > I do not remember an output like that using MAXENT > >> classifier. > >> > > > > > > > > >> > > > > > > > Damiano > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
