Hi, I have a script that basically does all of the training steps for me and it's worked many times before, but suddenly I got this error with a different corpus. It doesn't occur when I use the moses.ini file in train/model/ or in mert-work/, just when I use the one in filtered/. Any idea what's wrong? Something appears to be weird around the 39-grams step.
Defined parameters (per moses.ini or switch): config: /home/judah/archive8/3/working/filtered/moses.ini distortion-limit: 6 feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4 path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe Distortion IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/ train.en-es.blm.es order=3 input-factors: 0 mapping: 0 T 0 weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875 0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0= 0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722 0.266723 0.0198795 UnknownWordPenalty0= 1 /home/judah/mosesdecoder/bin line=UnknownWordPenalty FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 line=WordPenalty FeatureFunction: WordPenalty0 start: 1 end: 1 line=PhrasePenalty FeatureFunction: PhrasePenalty0 start: 2 end: 2 line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4 path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 input-factor=0 output-factor=0 FeatureFunction: TranslationModel0 start: 3 end: 6 line=LexicalReordering name=LexicalReordering0 num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe FeatureFunction: LexicalReordering0 start: 7 end: 12 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 13 end: 13 line=IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/ train.en-es.blm.es order=3 FeatureFunction: LM0 start: 14 end: 14 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 binary file loaded, default OFF_T: -1 Loading Distortion0 Loading LM0 In LanguageModelIRST::Load: nGramOrder = 3 Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1 Language Model Type is 1 mmap loadtxt_ram() 3-grams: reading 0 entries done level 3 1-grams: reading 0 entries done level 1 8-grams: reading 0 entries done level 8 2-grams: reading 0 entries done level 2 39-grams: reading 3991252117 entries moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&, float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed. [1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f < > Thanks, Judah
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
