Hi, I am not an expert on the logging of the LM loading, but the following looks pretty wacky:
3-grams: reading 0 entries done level 3 1-grams: reading 0 entries done level 1 8-grams: reading 0 entries done level 8 2-grams: reading 0 entries done level 2 39-grams: reading 3991252117 entries No 1-3 grams, but gazillion 39-grams? Wait, 39-grams? -phi On Wed, Jul 9, 2014 at 1:20 PM, Judah Schvimer <[email protected]> wrote: > I did, I ended up just removing the filtering since it didn't seem so > necessary. binarizing normally worked fine > > > On Wed, Jul 9, 2014 at 12:02 PM, Hieu Hoang <[email protected]> wrote: >> >> Sorry for late reply. >> >> Did you clean the data before you created the language model? Or you can >> try binarizing with KenLM instead >> >> >> On 3 July 2014 14:50, Judah Schvimer <[email protected]> wrote: >>> >>> Hi, >>> >>> I have a script that basically does all of the training steps for me and >>> it's worked many times before, but suddenly I got this error with a >>> different corpus. It doesn't occur when I use the moses.ini file in >>> train/model/ or in mert-work/, just when I use the one in filtered/. Any >>> idea what's wrong? Something appears to be weird around the 39-grams step. >>> >>> Defined parameters (per moses.ini or switch): >>> config: /home/judah/archive8/3/working/filtered/moses.ini >>> distortion-limit: 6 >>> feature: UnknownWordPenalty WordPenalty PhrasePenalty >>> PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4 >>> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 >>> input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 >>> num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0 >>> output-factor=0 >>> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe >>> Distortion IRSTLM name=LM0 factor=0 >>> path=/home/judah/archive8/3/lm/train.en-es.blm.es order=3 >>> input-factors: 0 >>> mapping: 0 T 0 >>> weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875 >>> 0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0= >>> 0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722 >>> 0.266723 0.0198795 UnknownWordPenalty0= 1 >>> /home/judah/mosesdecoder/bin >>> line=UnknownWordPenalty >>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 >>> line=WordPenalty >>> FeatureFunction: WordPenalty0 start: 1 end: 1 >>> line=PhrasePenalty >>> FeatureFunction: PhrasePenalty0 start: 2 end: 2 >>> line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20 >>> num-features=4 >>> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 >>> input-factor=0 output-factor=0 >>> FeatureFunction: TranslationModel0 start: 3 end: 6 >>> line=LexicalReordering name=LexicalReordering0 num-features=6 >>> type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe >>> FeatureFunction: LexicalReordering0 start: 7 end: 12 >>> Initializing LexicalReordering.. >>> line=Distortion >>> FeatureFunction: Distortion0 start: 13 end: 13 >>> line=IRSTLM name=LM0 factor=0 >>> path=/home/judah/archive8/3/lm/train.en-es.blm.es order=3 >>> FeatureFunction: LM0 start: 14 end: 14 >>> Loading UnknownWordPenalty0 >>> Loading WordPenalty0 >>> Loading PhrasePenalty0 >>> Loading LexicalReordering0 >>> binary file loaded, default OFF_T: -1 >>> Loading Distortion0 >>> Loading LM0 >>> In LanguageModelIRST::Load: nGramOrder = 3 >>> Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1 >>> Language Model Type is 1 >>> mmap >>> loadtxt_ram() >>> 3-grams: reading 0 entries >>> done level 3 >>> 1-grams: reading 0 entries >>> done level 1 >>> 8-grams: reading 0 entries >>> done level 8 >>> 2-grams: reading 0 entries >>> done level 2 >>> 39-grams: reading 3991252117 entries >>> moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&, >>> float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed. >>> [1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f < >>> > >>> >>> >>> Thanks, >>> Judah >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >> >> >> >> -- >> Hieu Hoang >> Research Associate >> University of Edinburgh >> http://www.hoang.co.uk/hieu >> > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
