I did, I ended up just removing the filtering since it didn't seem so necessary. binarizing normally worked fine
On Wed, Jul 9, 2014 at 12:02 PM, Hieu Hoang <[email protected]> wrote: > Sorry for late reply. > > Did you clean the data before you created the language model? Or you can > try binarizing with KenLM instead > > > On 3 July 2014 14:50, Judah Schvimer <[email protected]> wrote: > >> Hi, >> >> I have a script that basically does all of the training steps for me and >> it's worked many times before, but suddenly I got this error with a >> different corpus. It doesn't occur when I use the moses.ini file in >> train/model/ or in mert-work/, just when I use the one in filtered/. Any >> idea what's wrong? Something appears to be weird around the 39-grams step. >> >> Defined parameters (per moses.ini or switch): >> config: /home/judah/archive8/3/working/filtered/moses.ini >> distortion-limit: 6 >> feature: UnknownWordPenalty WordPenalty PhrasePenalty >> PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4 >> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 >> input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 >> num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0 >> output-factor=0 >> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe >> Distortion IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/ >> train.en-es.blm.es order=3 >> input-factors: 0 >> mapping: 0 T 0 >> weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875 >> 0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0= >> 0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722 >> 0.266723 0.0198795 UnknownWordPenalty0= 1 >> /home/judah/mosesdecoder/bin >> line=UnknownWordPenalty >> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 >> line=WordPenalty >> FeatureFunction: WordPenalty0 start: 1 end: 1 >> line=PhrasePenalty >> FeatureFunction: PhrasePenalty0 start: 2 end: 2 >> line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20 >> num-features=4 >> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1 >> input-factor=0 output-factor=0 >> FeatureFunction: TranslationModel0 start: 3 end: 6 >> line=LexicalReordering name=LexicalReordering0 num-features=6 >> type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe >> FeatureFunction: LexicalReordering0 start: 7 end: 12 >> Initializing LexicalReordering.. >> line=Distortion >> FeatureFunction: Distortion0 start: 13 end: 13 >> line=IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/ >> train.en-es.blm.es order=3 >> FeatureFunction: LM0 start: 14 end: 14 >> Loading UnknownWordPenalty0 >> Loading WordPenalty0 >> Loading PhrasePenalty0 >> Loading LexicalReordering0 >> binary file loaded, default OFF_T: -1 >> Loading Distortion0 >> Loading LM0 >> In LanguageModelIRST::Load: nGramOrder = 3 >> Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1 >> Language Model Type is 1 >> mmap >> loadtxt_ram() >> 3-grams: reading 0 entries >> done level 3 >> 1-grams: reading 0 entries >> done level 1 >> 8-grams: reading 0 entries >> done level 8 >> 2-grams: reading 0 entries >> done level 2 >> 39-grams: reading 3991252117 entries >> moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&, >> float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed. >> [1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f < >> > >> >> >> Thanks, >> Judah >> >> _______________________________________________ >> Moses-support mailing list >> [email protected] >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > > > -- > Hieu Hoang > Research Associate > University of Edinburgh > http://www.hoang.co.uk/hieu > >
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
