Hi, by default, the maximum number of factors is 4, but you are using 5 output factors.
You can recompile Moses with an additional switch: bjam --max-factors=5 I am a bit concerned about your model since it generates a large number of output factors independently, which may lead to an explosion of possible translation options where all good choices get pruned out before you get to the generation step which will eliminate all the impossible combinations. It would be probably a good idea to use this complex factored setup only as a back-off for unknown words. For details, please look at the following paper: http://amta2012.amtaweb.org/AMTA2012Files/papers/147.pdf -phi On Mon, Mar 17, 2014 at 3:01 PM, Peter Kleiweg <p.c.j.klei...@rug.nl> wrote: > Hieu Hoang schreef op de 17e dag van de lentemaand van het jaar 2014: > >> can you send me me the exact command you executed, the moses.ini file used, >> and a few lines of the input file > > Here is the training command: > > en_word=0 > en_root=1 > en_pos=2 > en_rel=3 > nl_word=0 > nl_root=1 > nl_pos=2 > nl_postag=3 > nl_rel=4 > /net/aps/64/opt/moses/mosesdecoder/scripts/training/train-model.perl \ > -cores 12 \ > -root-dir train \ > -corpus > /net/aistaff/kleiweg/moses/europarl/factored/corpus/train-small.clean \ > -f en -e nl \ > -alignment grow-diag-final-and \ > -reordering msd-bidirectional-fe \ > --lm > $nl_word:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word:8 \ > --lm > $nl_root:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root:8 \ > --lm $nl_pos:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos:8 > \ > --lm > $nl_postag:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag:8 \ > --lm $nl_rel:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel:8 > \ > --alignment-factors $en_root-$nl_root \ > --translation-factors > $en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel > \ > --reordering-factors > $en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel > \ > --generation-factors $nl_root,$nl_pos,$nl_postag,$nl_rel-$nl_word \ > --decoding-steps t0:t1,t2,t3,t4,g0 \ > -external-bin-dir /net/aps/64/opt/moses/mgizapp/bin \ > -mgiza \ > -mgiza-cpus=12 \ > > training.out \ > 2> training.err > > Here are three lines of training data, input: > > I|I|prp|nsubj declare|declare|vbp|hd resumed|resume|vbd|hd the|the|dt|det > session|session|nn|hd of|of|in|hd the|the|dt|det European|European|nnp|nn > Parliament|Parliament|nnp|hd adjourned|adjourn|vbn|hd on|on|in|hd > Friday|Friday|nnp|hd 17|17|cd|num December|December|nnp|hd 1999|1999|cd|num > ,|,|,|punct and|and|cc|cc I|I|prp|nsubj would|would|md|aux like|like|vb|hd > once|once|rb|advmod again|again|rb|hd to|to|to|aux wish|wish|vb|hd > you|you|prp|nsubj a|a|dt|det happy|happy|jj|amod new|new|jj|amod > year|year|nn|hd in|in|in|hd the|the|dt|det hope|hope|nn|hd that|that|in|mark > you|you|prp|nsubj enjoyed|enjoy|vbd|hd a|a|dt|det pleasant|pleasant|jj|amod > festive|festive|jj|amod period|period|nn|hd .|.|.|punct > > although|although|in|mark ,|,|,|punct as|as|in|mark you|you|prp|nsubj > will|will|md|aux have|have|vb|aux seen|see|vbn|hd ,|,|,|punct the|the|dt|det > dreaded|dreaded|jj|amod '|'|"|punct > millennium|millennium|nn|nn bug|bug|nn|hd '|'|pos|punct > failed|fail|vbn|hd to|to|to|hd materialise|materialise|nn|pobj ,|,|,|punct > still|still|rb|advmod the|the|dt|det people|people|nns|hd in|in|in|hd > a|a|dt|det number|number|nn|hd of|of|in|hd countries|country|nns|pobj > suffered|suffer|vbd|hd a|a|dt|det series|series|nn|hd of|of|in|hd > natural|natural|jj|amod disasters|disaster|nns|hd that|that|wdt|nsubj > truly|truly|rb|advmod were|be|vbd|cop dreadful|dreadful|jj|hd .|.|.|punct > > you|you|prp|nsubj have|have|vbp|aux requested|request|vbn|hd a|a|dt|det > debate|debate|nn|hd on|on|in|hd this|this|dt|det subject|subject|nn|hd > in|in|in|hd the|the|dt|det course|course|nn|hd of|of|in|hd the|the|dt|det > next|next|jj|amod few|few|jj|amod days|day|nns|hd ,|,|,|punct > during|during|in|hd this|this|dt|det part-session|part-session|nn|hd > .|.|.|punct > > And three lines of training data, target: > > ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su > verklaar|verklaar|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det > zitting|zitting|noun|N(soort,ev,basis,zijd,stan)|hd van|van|prep|VZ(init)|hd > het|het|det|LID(bep,stan,evon)|det Europees|Europees|name|SPEC(deeleigen)|mwp > Parlement|Parlement|name|SPEC(deeleigen)|mwp ,|,|punct|LET()|-- > die|die|pron|VNW(betr,pron,stan,vol,persoon,getal)|rhd op|op|prep|VZ(init)|hd > vrijdag|vrijdag|noun|N(eigen,ev,basis,zijd,stan)|hd > 17|17|noun|TW(hoofd,vrij)|mwp december|december|noun|TW(hoofd,vrij)|mwp > werd|word|verb|WW(pv,verl,ev)|hd > onderbroken|onderbreek|verb|WW(vd,vrij,zonder)|hd ,|,|punct|LET()|-- > te|te|comp|VZ(init)|cmp zijn|ben|verb|WW(inf,vrij,zonder)|hd > hervat|hervat|verb|WW(vd,vrij,zonder)|vc .|.|punct|LET()|-- > ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su wens|wens|verb|WW(pv,tgw,ev)|hd > u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|hd allen|allen|adv|BW()|mod > een|een|det|LID(onbep,stan,agr)|det > gelukkig|gelukkig|adj|ADJ(prenom,basis,zond! er! > )|mod nieuwjaar|nieuwjaar|noun|N(soort,ev,basis,onz,stan)|hd > en|en|vg|VG(neven)|crd hoop|hoop|verb|WW(pv,tgw,ev)|hd > dat|dat|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su > een|een|det|LID(onbep,stan,agr)|det > goede|goed|adj|ADJ(prenom,basis,met-e,stan)|mod > vakantie|vakantie|noun|N(soort,ev,basis,zijd,stan)|hd > heeft|heb|verb|WW(pv,tgw,met-t)|hd gehad|heb|verb|WW(vd,vrij,zonder)|hd > .|.|punct|LET()|-- > > zoals|zoals|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su > heeft|heb|verb|WW(pv,tgw,met-t)|hd kunnen|kan|verb|WW(inf,vrij,zonder)|hd > constateren|constateer|verb|WW(inf,vrij,zonder)|hd ,|,|punct|LET()|-- > is|ben|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det > grote|groot|adj|ADJ(prenom,basis,met-e,stan)|mod "|"|punct|LET()|-- > millenniumbug|millenniumbug|name|N(eigen,ev,basis,zijd,stan)|hd > "|"|punct|LET()|-- uitgebleven|blijf_uit|verb|WW(vd,vrij,zonder)|hd > .|.|punct|LET()|-- de|de|det|LID(bep,stan,rest)|det > burgers|burger|noun|N(soort,mv,basis)|hd van|van|prep|VZ(init)|hd > een|een|det|LID(onbep,stan,agr)|det > aantal|aantal|noun|N(soort,ev,basis,onz,stan)|hd van|van|prep|VZ(init)|hd > onze|ons|det|VNW(bez,det,stan,vol,1,mv,prenom,met-e,rest)|det > lidstaten|lidstaat|noun|N(soort,mv,basis)|hd zijn|ben|verb|WW(pv,tgw,mv)|hd > daarentegen|daarentegen|adv|BW()|mod door|door|prep|VZ(init)|hd > verschrikkelijke|verschrikkelijk|adj|ADJ(prenom,basis,! me! > t-e,stan)|mod natuurrampen|natuur_ramp|noun|N(soort,mv,basis)|hd > getroffen|tref|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|-- > > u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su > heeft|heb|verb|WW(pv,tgw,met-t)|hd > aangegeven|geef_aan|verb|WW(vd,vrij,zonder)|hd dat|dat|comp|VG(onder)|cmp > u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su > deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det > vergaderperiode|vergader_periode|noun|N(soort,ev,basis,zijd,stan)|hd > een|een|det|LID(onbep,stan,agr)|det > debat|debat|noun|N(soort,ev,basis,onz,stan)|hd > wilt|wil|verb|WW(pv,tgw,met-t)|hd over|over|prep|VZ(init)|hd > deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det > rampen|ramp|noun|N(soort,mv,basis)|hd .|.|punct|LET()|-- > > Here is the moses.ini created by the training: > > ######################### > ### MOSES CONFIG FILE ### > ######################### > > # input factors > [input-factors] > 0 > 1 > 2 > 3 > > # mapping steps > [mapping] > 0 T 0 > 1 T 1 > 1 T 2 > 1 T 3 > 1 T 4 > 1 G 0 > > [distortion-limit] > 6 > > # feature functions > [feature] > UnknownWordPenalty > WordPenalty > PhrasePenalty > PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz > input-factor=0 output-factor=0 > PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz > input-factor=1 output-factor=1 > PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz > input-factor=2 output-factor=2 > PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz > input-factor=2 output-factor=3 > PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz > input-factor=3 output-factor=4 > Generation name=GenerationModel0 num-features=2 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz > input-factor=1,2,3,4 output-factor=0 > LexicalReordering name=LexicalReordering0 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering1 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering2 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering3 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=3 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering4 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz > Distortion > KENLM lazyken=0 name=LM0 factor=0 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3 > KENLM lazyken=0 name=LM1 factor=1 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3 > KENLM lazyken=0 name=LM2 factor=2 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3 > KENLM lazyken=0 name=LM3 factor=3 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3 > KENLM lazyken=0 name=LM4 factor=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3 > > # dense weights for feature functions > [weight] > UnknownWordPenalty0= 1 > WordPenalty0= -1 > PhrasePenalty0= 0.2 > TranslationModel0= 0.2 0.2 0.2 0.2 > TranslationModel1= 0.2 0.2 0.2 0.2 > TranslationModel2= 0.2 0.2 0.2 0.2 > TranslationModel3= 0.2 0.2 0.2 0.2 > TranslationModel4= 0.2 0.2 0.2 0.2 > GenerationModel0= 0.3 0 > LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 > LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3 > LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3 > LexicalReordering3= 0.3 0.3 0.3 0.3 0.3 0.3 > LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3 > Distortion0= 0.3 > LM0= 0.5 > LM1= 0.5 > LM2= 0.5 > LM3= 0.5 > LM4= 0.5 > > This is the command I used: > > moses -f train/model/moses.ini > > And this is the output of that command, until it exits, > before any actual input: > > Defined parameters (per moses.ini or switch): > config: train/model/moses.ini > distortion-limit: 6 > feature: UnknownWordPenalty WordPenalty PhrasePenalty > PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz > input-factor=0 output-factor=0 PhraseDictionaryMemory name=TranslationModel1 > table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz > input-factor=1 output-factor=1 PhraseDictionaryMemory name=TranslationModel2 > table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz > input-factor=2 output-factor=2 PhraseDictionaryMemory name=TranslationModel3 > table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz > input-factor=2 output-factor=3 PhraseDictionaryMemory name=TranslationModel4 > table-limit=0 num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/tr! ai! > n/model/phrase-table.3-4.gz input-factor=3 output-factor=4 Generation > name=GenerationModel0 num-features=2 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz > input-factor=1,2,3,4 output-factor=0 LexicalReordering > name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff > input-factor=0 output-factor=0 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering1 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering2 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz > LexicalReorderi! ng! > name=LexicalReordering3 num-features=6 type=wbe-msd-bidirectional-fe- > allff input-factor=2 output-factor=3 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz > LexicalReordering name=LexicalReordering4 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz > Distortion KENLM lazyken=0 name=LM0 factor=0 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3 > KENLM lazyken=0 name=LM1 factor=1 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3 > KENLM lazyken=0 name=LM2 factor=2 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3 KENLM > lazyken=0 name=LM3 factor=3 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3 > KENLM lazyken=0 name=LM4 factor=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3 > input-factors: 0 1 2 3 > mapping: 0 T 0 1 T 1 1 T 2 1 T 3 1 T 4 1 G 0 > weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 > TranslationModel0= 0.2 0.2 0.2 0.2 TranslationModel1= 0.2 0.2 0.2 0.2 > TranslationModel2= 0.2 0.2 0.2 0.2 TranslationModel3= 0.2 0.2 0.2 0.2 > TranslationModel4= 0.2 0.2 0.2 0.2 GenerationModel0= 0.3 0 > LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering1= 0.3 0.3 0.3 > 0.3 0.3 0.3 LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering3= > 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3 > Distortion0= 0.3 LM0= 0.5 LM1= 0.5 LM2= 0.5 LM3= 0.5 LM4= 0.5 > /net/aps/64/opt/moses/mosesdecoder/bin > line=UnknownWordPenalty > FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 > line=WordPenalty > FeatureFunction: WordPenalty0 start: 1 end: 1 > line=PhrasePenalty > FeatureFunction: PhrasePenalty0 start: 2 end: 2 > line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 > num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz > input-factor=0 output-factor=0 > FeatureFunction: TranslationModel0 start: 3 end: 6 > line=PhraseDictionaryMemory name=TranslationModel1 table-limit=0 > num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz > input-factor=1 output-factor=1 > FeatureFunction: TranslationModel1 start: 7 end: 10 > line=PhraseDictionaryMemory name=TranslationModel2 table-limit=0 > num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz > input-factor=2 output-factor=2 > FeatureFunction: TranslationModel2 start: 11 end: 14 > line=PhraseDictionaryMemory name=TranslationModel3 table-limit=0 > num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz > input-factor=2 output-factor=3 > FeatureFunction: TranslationModel3 start: 15 end: 18 > line=PhraseDictionaryMemory name=TranslationModel4 table-limit=0 > num-features=4 > path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz > input-factor=3 output-factor=4 > FeatureFunction: TranslationModel4 start: 19 end: 22 > Exception: bitset::set > > > > >> >> On 17 March 2014 17:21, Peter Kleiweg <p.c.j.klei...@rug.nl> wrote: >> >> > >> > I run train-model.perl, and it ends with "(9) create moses.ini" >> > without any error. >> > >> > When I try to start moses with "moses -f moses.ini" it fails >> > with "Exception: bitset::set" >> > >> > Is this a bug? What do I do? >> > What does it even mean, bitset::set? > > > > -- > Peter Kleiweg > http://pkleiweg.home.xs4all.nl/ > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
