Sorry for late reply. Did you clean the data before you created the language model? Or you can try binarizing with KenLM instead
On 3 July 2014 14:50, Judah Schvimer <[email protected]> wrote: > Hi, > > I have a script that basically does all of the training steps for me and > it's worked many times before, but suddenly I got this error with a > different corpus. It doesn't occur when I use the moses.ini file in > train/model/ or in mert-work/, just when I use the one in filtered/. Any > idea what's wrong? Something appears to be weird around the 39-grams step. > > Defined parameters (per moses.ini or switch): > config: /home/judah/archive8/3/working/filtered/moses.ini > distortion-limit: 6 > feature: UnknownWordPenalty WordPenalty PhrasePenalty > PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4 > path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 > input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 > num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0 > output-factor=0 > path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe > Distortion IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/ > train.en-es.blm.es order=3 > input-factors: 0 > mapping: 0 T 0 > weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875 > 0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0= > 0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722 > 0.266723 0.0198795 UnknownWordPenalty0= 1 > /home/judah/mosesdecoder/bin > line=UnknownWordPenalty > FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 > line=WordPenalty > FeatureFunction: WordPenalty0 start: 1 end: 1 > line=PhrasePenalty > FeatureFunction: PhrasePenalty0 start: 2 end: 2 > line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20 > num-features=4 > path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 > input-factor=0 output-factor=0 > FeatureFunction: TranslationModel0 start: 3 end: 6 > line=LexicalReordering name=LexicalReordering0 num-features=6 > type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe > FeatureFunction: LexicalReordering0 start: 7 end: 12 > Initializing LexicalReordering.. > line=Distortion > FeatureFunction: Distortion0 start: 13 end: 13 > line=IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/ > train.en-es.blm.es order=3 > FeatureFunction: LM0 start: 14 end: 14 > Loading UnknownWordPenalty0 > Loading WordPenalty0 > Loading PhrasePenalty0 > Loading LexicalReordering0 > binary file loaded, default OFF_T: -1 > Loading Distortion0 > Loading LM0 > In LanguageModelIRST::Load: nGramOrder = 3 > Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1 > Language Model Type is 1 > mmap > loadtxt_ram() > 3-grams: reading 0 entries > done level 3 > 1-grams: reading 0 entries > done level 1 > 8-grams: reading 0 entries > done level 8 > 2-grams: reading 0 entries > done level 2 > 39-grams: reading 3991252117 entries > moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&, > float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed. > [1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f < > > > > > Thanks, > Judah > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
