Re: [Moses-support] Tuning crashed
Hi Tomasz The error message about missing the ini file is a consequence of the tuning crash, so just ignore this. To find out why Moses is failing, run it again in the console like this: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights If necessary, increase the verbosity of the debug (-v). cheers - Barry On 08/06/16 08:42, Tomasz Gawryl wrote: Hi, I have a problem with tuning crashed. It seems that moses.ini is missing in temporary folder but I have no idea why. I attached link to my config file. Please help. Regards Thomas moses@smtserver:~/working/experiments/NGRAM5/steps/2$ more TUNING_tune.2.STDERR Using SCRIPTS_ROOTDIR: /home/moses/src/mosesdecoder/scripts Asking moses for feature names and values from /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights exec: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights > ./features.list 2> /dev/null Exit code: 1 ERROR: Failed to run '/home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights'. at /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1748. cp: cannot stat ‘/home/moses/working/experiments/NGRAM5/tuning/tmp.2/moses.ini’: No such file or directory https://docs.google.com/document/d/1gI7YVUx8VoktIfIQvvU54jKSm5Ta6UZjYcszBFtP-V8/edit?usp=sharing ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Tuning crashed
Hi, I have a problem with tuning crashed. It seems that moses.ini is missing in temporary folder but I have no idea why. I attached link to my config file. Please help. Regards Thomas moses@smtserver:~/working/experiments/NGRAM5/steps/2$ more TUNING_tune.2.STDERR Using SCRIPTS_ROOTDIR: /home/moses/src/mosesdecoder/scripts Asking moses for feature names and values from /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights exec: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights Executing: /home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights > ./features.list 2> /dev/null Exit code: 1 ERROR: Failed to run '/home/moses/src/mosesdecoder/bin/moses -threads 16 -v 0 -config /home/moses/working/experiments/NGRAM5/model/moses.bin.ini.2 -show-weights'. at /home/moses/src/mosesdecoder/scripts/training/mert-moses.pl line 1748. cp: cannot stat '/home/moses/working/experiments/NGRAM5/tuning/tmp.2/moses.ini': No such file or directory https://docs.google.com/document/d/1gI7YVUx8VoktIfIQvvU54jKSm5Ta6UZjYcszBFtP -V8/edit?usp=sharing ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Tuning crashed
Thanks, Barry, Both batches-mira and pro tuning don't crash. But the result of batch-miratuning is very unstable due to the size of dev set. Best, Wei On Sat, May 24, 2014 at 11:34 AM, Wei Qiu wrote: > Thanks, Barry. I will try mira. > > Anyway, I share my model file here > > https://www.dropbox.com/s/jko7aitnk9mqa8a/model.tar.gz > > Best, > Wei > > > On Sat, May 24, 2014 at 10:13 AM, Barry Haddow > wrote: > >> Hi Wei >> >> This error (as far as I recall) tends to mean that one of your features >> is flat (ie uninformative). I don't see that this is the case for your >> features. The only thing I could think of is that your tuning set is quite >> small (500 sentences) but that doesn't usually cause a problem. >> >> My suggestion would be to try tuning with batch mira or pro. >> >> cheers - Barry >> >> >> On 24/05/14 08:31, Wei Qiu wrote: >> >> Hi, >> >> I tried to train a phrase-based baseline system from German to English. >> >> But during the tuning step, mert crashed in the first run. I tracked it >> to this error, >> >> Check abs(leftmost->first-gradient.rbegin()->first) < 0.0001 failed in >> mert/Optimizer.cpp:170 >> >> I've already checked similar posts on the mailing list, but my run1. outand >> run1. Scores >> .data look alright. >> >> The tuning directory is shared here: >> https://www.dropbox.com/s/j6tflbtft3uqvuu/tmp.1.tar.gz >> >> The model directory will be shared as well(it's still uploading). >> >> Thanks in advance. >> >> Best, >> Wei Qiu >> >> >> >> >> ___ >> Moses-support mailing >> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support >> >> >> > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Tuning crashed
Thanks, Barry. I will try mira. Anyway, I share my model file here https://www.dropbox.com/s/jko7aitnk9mqa8a/model.tar.gz Best, Wei On Sat, May 24, 2014 at 10:13 AM, Barry Haddow wrote: > Hi Wei > > This error (as far as I recall) tends to mean that one of your features is > flat (ie uninformative). I don't see that this is the case for your > features. The only thing I could think of is that your tuning set is quite > small (500 sentences) but that doesn't usually cause a problem. > > My suggestion would be to try tuning with batch mira or pro. > > cheers - Barry > > > On 24/05/14 08:31, Wei Qiu wrote: > > Hi, > > I tried to train a phrase-based baseline system from German to English. > > But during the tuning step, mert crashed in the first run. I tracked it > to this error, > > Check abs(leftmost->first-gradient.rbegin()->first) < 0.0001 failed in > mert/Optimizer.cpp:170 > > I've already checked similar posts on the mailing list, but my run1. outand > run1. Scores > .data look alright. > > The tuning directory is shared here: > https://www.dropbox.com/s/j6tflbtft3uqvuu/tmp.1.tar.gz > > The model directory will be shared as well(it's still uploading). > > Thanks in advance. > > Best, > Wei Qiu > > > > > ___ > Moses-support mailing > listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support > > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Tuning crashed
Hi Wei This error (as far as I recall) tends to mean that one of your features is flat (ie uninformative). I don't see that this is the case for your features. The only thing I could think of is that your tuning set is quite small (500 sentences) but that doesn't usually cause a problem. My suggestion would be to try tuning with batch mira or pro. cheers - Barry On 24/05/14 08:31, Wei Qiu wrote: Hi, I tried to train a phrase-based baseline system from German to English. But during the tuning step, mert crashed in the first run. I tracked it to this error, Check abs(leftmost->first-gradient.rbegin()->first) < 0.0001 failed in mert/Optimizer.cpp:170 I've already checked similar posts on the mailing list, but my run1. out and run1. Scores.data look alright. The tuning directory is shared here: https://www.dropbox.com/s/j6tflbtft3uqvuu/tmp.1.tar.gz The model directory will be shared as well(it's still uploading). Thanks in advance. Best, Wei Qiu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Tuning crashed
Hi, I tried to train a phrase-based baseline system from German to English. But during the tuning step, mert crashed in the first run. I tracked it to this error, Check abs(leftmost->first-gradient.rbegin()->first) < 0.0001 failed in mert /Optimizer.cpp:170 I've already checked similar posts on the mailing list, but my run1. outand run1. Scores .data look alright. The tuning directory is shared here: https://www.dropbox.com/s/j6tflbtft3uqvuu/tmp.1.tar.gz The model directory will be shared as well(it's still uploading). Thanks in advance. Best, Wei Qiu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
Thank you barry that was most helpful. I was running the EMS for factored models(POS), there have been few mistakes on my part while giving the inputs properly, which I have corrected. The EMS now has run fine. I gave a factorized reference file (target language) where as the output is just surface form of words, because of which the BLEU score is initialized to zero. Similar thing with the language model input, instead of plain file I have given a factorized file as input for language model leading to a lot of OOVs. Though the final BLEU score seems to be lower than the base run with just surface forms. Thanking you, Y. Jayendra Rakesh, B.Tech. Computer Science Dual Degree, LTRC, IIIT Hyderabad. On Sun, Jun 9, 2013 at 10:45 PM, Barry Haddow wrote: > Hi Jayendra > > The fact that your bleu precisions are all zero (run1.scores.dat) suggests > that something went badly wrong > with the translation, or your reference set is incorrect. What do the > translations (in run1.out) look like? > > The zeros in run1.features.dat are normal, but the very low lm scores (eg > -1804.31 in first row) are not. This > suggests that all the tokens in your input sentences are OOVs. Check your > setup to make sure that you have not, > for example, swapped the source and target language files accidentally at > some point, > > cheers - Barry > > > Quoting jayendra rakesh on Sun, 9 Jun 2013 > 00:08:10 +0530: > > Sorry for bumping the thread again. >> here's what I found after running the command by hand as suggested. >> >> run1.init.opt: >> * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 >> >> -1.00 0.20 0.20 0.20 0.20 0.20 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> * >> Things noticed. >> *run1.features.dat has columns 3,6 all zeroes >> * >> *run1.scores.dat has columns 1,3,5,7 all zeroes >> * >> >> run1.features.dat: >> *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 >> >> tm_1 tm_2 tm_3 tm_4 >> -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 >> -48.9898 -27.8995 17.9981 >> 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 >> -27.871 >> 17.9981 >> -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 >> -49.7377 -28.9981 17.9981 >> 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 >> -28.9696 >> 17.9981 >> -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 >> -50.0519 -28.2473 17.9981 >> -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 >> -48.9898 -28.83 17.9981* >> >> >> run1.scores.dat >> >> * >> SCORES_TXT_BEGIN_0 0 100 9 BLEU >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31* >> >> *Issue: in the MIRA run BLEU score is initialized to zero >> >> * >> Hoping the information would help in solving the issue. >> >> Thanking you, >> Jayendra Rakesh. >> >> >> On Wed, Jun 5, 2013 at 1:52 PM, jayendra rakesh >> **wrote: >> >> Hi Phi, >>> >>> Thanks for the reply, I made a hand run of the command as you have >>> suggested and was able to repeat the crash. I have checked run1.init.opt >>> file, it seems to be fine >>> >>> run1.init.opt: >>> * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 >>> >>> -1.00 0.20 0.20 0.20 0.20 0.20 >>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >>> * >>> But I noticed a thing in both run1.scores.dat and run1.features.dat >>> files, >>> that many *features*(*columns 3,6 are all zero)* and *scores*(*columns >>> 1,3,5,7 are all zero*) are having zero values, because of which i think >>> >>> BLEU score is initialized to zero in mert.log >>> >>> run1.features.dat: >>> *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 >>> >>> tm_1 tm_2 tm_3 tm_4 >>> -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 >>> -48.9898 -27.8995 17.9981 >>> 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 >>> -27.871 17.9981 >>> -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 >>> -49.7377 -28.9981 17.9981 >>> 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 >>> -28.9696 17.9981 >>> -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 >>> -50.0519 -28.2473 17.9981 >>> -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 >>> -48.9898 -28.83 17.9981* >>> >>> >>> run1.scores.dat >>> >>> * >>> SCORES_TXT_BEGIN_0 0 100 9 BLEU >>> 0 18 0 17 0 16 0 15 31 >>> 0 18 0 17 0 16 0 15 31 >>> 0 18 0 17 0 16 0 15 31 >>> 0 18 0 17 0 16 0 15 31 >>> 0 18 0 17 0 16 0 15 31* >>> >>> >>> >>> Hoping the information would help in pointing out the issue. >>> >>> Regards, >>> Jayendra Rakesh >>> >>> >>> >>> On Sun, Jun 2, 2013 at 5:19 PM, Philipp Koehn >>> wrote: >>> >>> Hi, I have been running kbest MIRA with factored
Re: [Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
Hi Jayendra The fact that your bleu precisions are all zero (run1.scores.dat) suggests that something went badly wrong with the translation, or your reference set is incorrect. What do the translations (in run1.out) look like? The zeros in run1.features.dat are normal, but the very low lm scores (eg -1804.31 in first row) are not. This suggests that all the tokens in your input sentences are OOVs. Check your setup to make sure that you have not, for example, swapped the source and target language files accidentally at some point, cheers - Barry Quoting jayendra rakesh on Sun, 9 Jun 2013 00:08:10 +0530: > Sorry for bumping the thread again. > here's what I found after running the command by hand as suggested. > > run1.init.opt: > * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 > -1.00 0.20 0.20 0.20 0.20 0.20 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > * > Things noticed. > *run1.features.dat has columns 3,6 all zeroes > * > *run1.scores.dat has columns 1,3,5,7 all zeroes > * > > run1.features.dat: > *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 > tm_1 tm_2 tm_3 tm_4 > -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 > -48.9898 -27.8995 17.9981 > 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 -27.871 > 17.9981 > -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 > -49.7377 -28.9981 17.9981 > 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 -28.9696 > 17.9981 > -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 > -50.0519 -28.2473 17.9981 > -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 > -48.9898 -28.83 17.9981* > > > run1.scores.dat > * > SCORES_TXT_BEGIN_0 0 100 9 BLEU > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31* > > *Issue: in the MIRA run BLEU score is initialized to zero > * > Hoping the information would help in solving the issue. > > Thanking you, > Jayendra Rakesh. > > > On Wed, Jun 5, 2013 at 1:52 PM, jayendra rakesh > wrote: > >> Hi Phi, >> >> Thanks for the reply, I made a hand run of the command as you have >> suggested and was able to repeat the crash. I have checked run1.init.opt >> file, it seems to be fine >> >> run1.init.opt: >> * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 >> -1.00 0.20 0.20 0.20 0.20 0.20 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 1 1 1 1 1 1 1 1 1 1 1 1 1 1 >> * >> But I noticed a thing in both run1.scores.dat and run1.features.dat files, >> that many *features*(*columns 3,6 are all zero)* and *scores*(*columns >> 1,3,5,7 are all zero*) are having zero values, because of which i think >> BLEU score is initialized to zero in mert.log >> >> run1.features.dat: >> *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 >> tm_1 tm_2 tm_3 tm_4 >> -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 >> -48.9898 -27.8995 17.9981 >> 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 >> -27.871 17.9981 >> -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 >> -49.7377 -28.9981 17.9981 >> 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 >> -28.9696 17.9981 >> -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 >> -50.0519 -28.2473 17.9981 >> -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 >> -48.9898 -28.83 17.9981* >> >> >> run1.scores.dat >> * >> SCORES_TXT_BEGIN_0 0 100 9 BLEU >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31 >> 0 18 0 17 0 16 0 15 31* >> >> >> Hoping the information would help in pointing out the issue. >> >> Regards, >> Jayendra Rakesh >> >> >> >> On Sun, Jun 2, 2013 at 5:19 PM, Philipp Koehn wrote: >> >>> Hi, >>> >>> I have been running kbest MIRA with factored models many times, never >>> with any problems, so "this should work". >>> >>> The error is in the step: /tools/mosesdecoder-master_2/bin/kbmira -J >>> 100 -C 0.001 --dense-init run1.init.opt --ffile run1.features.data >>> [...] >>> >>> so that's where to start. >>> >>> Check if the features file looks sane. >>> Check the run1.init.opt file. >>> Run the step by hand. >>> >>> If this does not work, send us the input files for this command (maybe >>> even a smaller subset, if you can reproduce the error). >>> >>> -phi >>> >>> On Sun, Jun 2, 2013 at 10:56 AM, jayendra rakesh >>> wrote: >>> > Hi, >>> > My EMS setup (factored,MIRA) crashes at tuning stage after single run. >>> > config.toy: (attaching only training and tuning sections) >>> > # TRANSLATION MODEL TRAINING >>> > >>> > [TRAINING] >>> > >>> > ### training script to be used: either a legacy script or >>> > # current moses training script (default) >>> > # >>> > script = $m
Re: [Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
It's quite difficult to debug from this output. Can you make your model files available for download via dropbox or google drive and i will try and run it on my laptop. Does it run ok with mert or pro? Hieu Sent while bumping into things On 8 Jun 2013, at 07:39 PM, jayendra rakesh wrote: Sorry for bumping the thread again. here's what I found after running the command by hand as suggested. run1.init.opt: * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 -1.00 0.20 0.20 0.20 0.20 0.20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 * Things noticed. *run1.features.dat has columns 3,6 all zeroes * *run1.scores.dat has columns 1,3,5,7 all zeroes * run1.features.dat: *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 tm_1 tm_2 tm_3 tm_4 -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 -48.9898 -27.8995 17.9981 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 -27.871 17.9981 -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 -49.7377 -28.9981 17.9981 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 -28.9696 17.9981 -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 -50.0519 -28.2473 17.9981 -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 -48.9898 -28.83 17.9981* run1.scores.dat * SCORES_TXT_BEGIN_0 0 100 9 BLEU 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31* *Issue: in the MIRA run BLEU score is initialized to zero * Hoping the information would help in solving the issue. Thanking you, Jayendra Rakesh. On Wed, Jun 5, 2013 at 1:52 PM, jayendra rakesh wrote: > Hi Phi, > > Thanks for the reply, I made a hand run of the command as you have > suggested and was able to repeat the crash. I have checked run1.init.opt > file, it seems to be fine > > run1.init.opt: > * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 > -1.00 0.20 0.20 0.20 0.20 0.20 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > * > But I noticed a thing in both run1.scores.dat and run1.features.dat files, > that many *features*(*columns 3,6 are all zero)* and *scores*(*columns > 1,3,5,7 are all zero*) are having zero values, because of which i think > BLEU score is initialized to zero in mert.log > > run1.features.dat: > *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 > tm_1 tm_2 tm_3 tm_4 > -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 > -48.9898 -27.8995 17.9981 > 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 > -27.871 17.9981 > -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 > -49.7377 -28.9981 17.9981 > 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 > -28.9696 17.9981 > -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 > -50.0519 -28.2473 17.9981 > -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 > -48.9898 -28.83 17.9981* > > > run1.scores.dat > * > SCORES_TXT_BEGIN_0 0 100 9 BLEU > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31* > > > Hoping the information would help in pointing out the issue. > > Regards, > Jayendra Rakesh > > > > On Sun, Jun 2, 2013 at 5:19 PM, Philipp Koehn wrote: > >> Hi, >> >> I have been running kbest MIRA with factored models many times, never >> with any problems, so "this should work". >> >> The error is in the step: /tools/mosesdecoder-master_2/bin/kbmira -J >> 100 -C 0.001 --dense-init run1.init.opt --ffile run1.features.data >> [...] >> >> so that's where to start. >> >> Check if the features file looks sane. >> Check the run1.init.opt file. >> Run the step by hand. >> >> If this does not work, send us the input files for this command (maybe >> even a smaller subset, if you can reproduce the error). >> >> -phi >> >> On Sun, Jun 2, 2013 at 10:56 AM, jayendra rakesh >> wrote: >> > Hi, >> > My EMS setup (factored,MIRA) crashes at tuning stage after single run. >> > config.toy: (attaching only training and tuning sections) >> > # TRANSLATION MODEL TRAINING >> > >> > [TRAINING] >> > >> > ### training script to be used: either a legacy script or >> > # current moses training script (default) >> > # >> > script = $moses-script-dir/training/train-model.perl >> > >> > ### general options >> > # these are options that are passed on to train-model.perl, for instance >> > # * "-mgiza -mgiza-cpus 8" to use mgiza instead of giza >> > # * "-sort-buffer-size 8G -sort-compress gzip" to reduce on-disk sorting >> > # * "-sort-parallel 8 -cores 8" to speed up phrase table building >> > # >> > #training-options = "" >> > >> > ### factored training: specify here which factors used >> > # if none specified, single factor training is ass
Re: [Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
Sorry for bumping the thread again. here's what I found after running the command by hand as suggested. run1.init.opt: * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 -1.00 0.20 0.20 0.20 0.20 0.20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 * Things noticed. *run1.features.dat has columns 3,6 all zeroes * *run1.scores.dat has columns 1,3,5,7 all zeroes * run1.features.dat: *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 tm_1 tm_2 tm_3 tm_4 -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 -48.9898 -27.8995 17.9981 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 -27.871 17.9981 -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 -49.7377 -28.9981 17.9981 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 -28.9696 17.9981 -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 -50.0519 -28.2473 17.9981 -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 -48.9898 -28.83 17.9981* run1.scores.dat * SCORES_TXT_BEGIN_0 0 100 9 BLEU 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31 0 18 0 17 0 16 0 15 31* *Issue: in the MIRA run BLEU score is initialized to zero * Hoping the information would help in solving the issue. Thanking you, Jayendra Rakesh. On Wed, Jun 5, 2013 at 1:52 PM, jayendra rakesh wrote: > Hi Phi, > > Thanks for the reply, I made a hand run of the command as you have > suggested and was able to repeat the crash. I have checked run1.init.opt > file, it seems to be fine > > run1.init.opt: > * 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50 > -1.00 0.20 0.20 0.20 0.20 0.20 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 1 1 1 1 1 1 1 1 1 1 1 1 1 1 > * > But I noticed a thing in both run1.scores.dat and run1.features.dat files, > that many *features*(*columns 3,6 are all zero)* and *scores*(*columns > 1,3,5,7 are all zero*) are having zero values, because of which i think > BLEU score is initialized to zero in mert.log > > run1.features.dat: > *FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0 > tm_1 tm_2 tm_3 tm_4 > -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959 > -48.9898 -27.8995 17.9981 > 0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 > -27.871 17.9981 > -8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714 > -49.7377 -28.9981 17.9981 > 0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 > -28.9696 17.9981 > -8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433 > -50.0519 -28.2473 17.9981 > -8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766 > -48.9898 -28.83 17.9981* > > > run1.scores.dat > * > SCORES_TXT_BEGIN_0 0 100 9 BLEU > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31 > 0 18 0 17 0 16 0 15 31* > > > Hoping the information would help in pointing out the issue. > > Regards, > Jayendra Rakesh > > > > On Sun, Jun 2, 2013 at 5:19 PM, Philipp Koehn wrote: > >> Hi, >> >> I have been running kbest MIRA with factored models many times, never >> with any problems, so "this should work". >> >> The error is in the step: /tools/mosesdecoder-master_2/bin/kbmira -J >> 100 -C 0.001 --dense-init run1.init.opt --ffile run1.features.data >> [...] >> >> so that's where to start. >> >> Check if the features file looks sane. >> Check the run1.init.opt file. >> Run the step by hand. >> >> If this does not work, send us the input files for this command (maybe >> even a smaller subset, if you can reproduce the error). >> >> -phi >> >> On Sun, Jun 2, 2013 at 10:56 AM, jayendra rakesh >> wrote: >> > Hi, >> > My EMS setup (factored,MIRA) crashes at tuning stage after single run. >> > config.toy: (attaching only training and tuning sections) >> > # TRANSLATION MODEL TRAINING >> > >> > [TRAINING] >> > >> > ### training script to be used: either a legacy script or >> > # current moses training script (default) >> > # >> > script = $moses-script-dir/training/train-model.perl >> > >> > ### general options >> > # these are options that are passed on to train-model.perl, for instance >> > # * "-mgiza -mgiza-cpus 8" to use mgiza instead of giza >> > # * "-sort-buffer-size 8G -sort-compress gzip" to reduce on-disk sorting >> > # * "-sort-parallel 8 -cores 8" to speed up phrase table building >> > # >> > #training-options = "" >> > >> > ### factored training: specify here which factors used >> > # if none specified, single factor training is assumed >> > # (one translation step, surface to surface) >> > # >> > input-factors = word pos >> > output-factors = word pos >> > alignment-factors = "word -> word" >> > translation-factors = "word+pos -> word+pos" >> > reordering-factors = "word -> word" >> > #generation-factors = "pos -> word"
Re: [Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
Hi Phi,
Thanks for the reply, I made a hand run of the command as you have
suggested and was able to repeat the crash. I have checked run1.init.opt
file, it seems to be fine
run1.init.opt:
* 0.30 0.30 0.30 0.30 0.30 0.30 0.30 0.50
-1.00 0.20 0.20 0.20 0.20 0.20
0 0 0 0 0 0 0 0 0 0 0 0 0 0
1 1 1 1 1 1 1 1 1 1 1 1 1 1
*
But I noticed a thing in both run1.scores.dat and run1.features.dat files,
that many *features*(*columns 3,6 are all zero)* and *scores*(*columns
1,3,5,7 are all zero*) are having zero values, because of which i think
BLEU score is initialized to zero in mert.log
run1.features.dat:
*FEATURES_TXT_BEGIN_0 0 100 14 d_0 d_1 d_2 d_3 d_4 d_5 d_6 lm_0 w_0 tm_0
tm_1 tm_2 tm_3 tm_4
-8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.6613 -80.0959
-48.9898 -27.8995 17.9981
0 -20.6133 0 0 -27.4523 0 0 -1804.31 -18 -81.3889 -80.0948 -50.4425 -27.871
17.9981
-8 -16.9628 0 -1.28775 -22.4881 0 -1.16898 -1804.31 -18 -77.4669 -80.9714
-49.7377 -28.9981 17.9981
0 -21.414 0 0 -26.4089 0 0 -1804.31 -18 -79.1945 -80.9702 -51.1904 -28.9696
17.9981
-8 -16.8088 0 -1.28775 -23.0796 0 -1.16898 -1804.31 -18 -78.6891 -79.6433
-50.0519 -28.2473 17.9981
-8 -16.1621 0 -1.28775 -23.5316 0 -1.16898 -1804.31 -18 -79.1464 -79.9766
-48.9898 -28.83 17.9981*
run1.scores.dat
*
SCORES_TXT_BEGIN_0 0 100 9 BLEU
0 18 0 17 0 16 0 15 31
0 18 0 17 0 16 0 15 31
0 18 0 17 0 16 0 15 31
0 18 0 17 0 16 0 15 31
0 18 0 17 0 16 0 15 31*
Hoping the information would help in pointing out the issue.
Regards,
Jayendra Rakesh
On Sun, Jun 2, 2013 at 5:19 PM, Philipp Koehn wrote:
> Hi,
>
> I have been running kbest MIRA with factored models many times, never
> with any problems, so "this should work".
>
> The error is in the step: /tools/mosesdecoder-master_2/bin/kbmira -J
> 100 -C 0.001 --dense-init run1.init.opt --ffile run1.features.data
> [...]
>
> so that's where to start.
>
> Check if the features file looks sane.
> Check the run1.init.opt file.
> Run the step by hand.
>
> If this does not work, send us the input files for this command (maybe
> even a smaller subset, if you can reproduce the error).
>
> -phi
>
> On Sun, Jun 2, 2013 at 10:56 AM, jayendra rakesh
> wrote:
> > Hi,
> > My EMS setup (factored,MIRA) crashes at tuning stage after single run.
> > config.toy: (attaching only training and tuning sections)
> > # TRANSLATION MODEL TRAINING
> >
> > [TRAINING]
> >
> > ### training script to be used: either a legacy script or
> > # current moses training script (default)
> > #
> > script = $moses-script-dir/training/train-model.perl
> >
> > ### general options
> > # these are options that are passed on to train-model.perl, for instance
> > # * "-mgiza -mgiza-cpus 8" to use mgiza instead of giza
> > # * "-sort-buffer-size 8G -sort-compress gzip" to reduce on-disk sorting
> > # * "-sort-parallel 8 -cores 8" to speed up phrase table building
> > #
> > #training-options = ""
> >
> > ### factored training: specify here which factors used
> > # if none specified, single factor training is assumed
> > # (one translation step, surface to surface)
> > #
> > input-factors = word pos
> > output-factors = word pos
> > alignment-factors = "word -> word"
> > translation-factors = "word+pos -> word+pos"
> > reordering-factors = "word -> word"
> > #generation-factors = "pos -> word"
> > decoding-steps = "t0"
> >
> > ### parallelization of data preparation step
> > # the two directions of the data preparation can be run in parallel
> > # comment out if not needed
> > #
> > parallel = yes
> >
> > ### pre-computation for giza++
> > # giza++ has a more efficient data structure that needs to be
> > # initialized with snt2cooc. if run in parallel, this may reduces
> > # memory requirements. set here the number of parts
> > #
> > #run-giza-in-parts = 5
> >
> > ### symmetrization method to obtain word alignments from giza output
> > # (commonly used: grow-diag-final-and)
> > #
> > alignment-symmetrization-method = grow-diag-final-and
> >
> > ### use of berkeley aligner for word alignment
> > #
> > #use-berkeley = true
> > #alignment-symmetrization-method = berkeley
> > #berkeley-train = $moses-script-dir/ems/support/berkeley-train.sh
> > #berkeley-process = $moses-script-dir/ems/support/berkeley-process.sh
> > #berkeley-jar = /your/path/to/berkeleyaligner-1.1/berkeleyaligner.jar
> > #berkeley-java-options = "-server -mx3m -ea"
> > #berkeley-training-options = "-Main.iters 5 5 -EMWordAligner.numThreads
> 8"
> > #berkeley-process-options = "-EMWordAligner.numThreads 8"
> > #berkeley-posterior = 0.5
> >
> > ### use of baseline alignment model (incremental training)
> > #
> > #baseline = 68
> > #baseline-alignment-model =
> > "$working-dir/training/prepared.$baseline/$input-extension.vcb \
> > # $working-dir/training/prepared.$baseline/$output-extension.vcb \
> > #
> >
> $working-dir/training/giza.$baseline/${output-extension}-$input-extension.cooc
> > \
> > #
> >
> $working-dir/trainin
Re: [Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
Hi,
I have been running kbest MIRA with factored models many times, never
with any problems, so "this should work".
The error is in the step: /tools/mosesdecoder-master_2/bin/kbmira -J
100 -C 0.001 --dense-init run1.init.opt --ffile run1.features.data
[...]
so that's where to start.
Check if the features file looks sane.
Check the run1.init.opt file.
Run the step by hand.
If this does not work, send us the input files for this command (maybe
even a smaller subset, if you can reproduce the error).
-phi
On Sun, Jun 2, 2013 at 10:56 AM, jayendra rakesh
wrote:
> Hi,
> My EMS setup (factored,MIRA) crashes at tuning stage after single run.
> config.toy: (attaching only training and tuning sections)
> # TRANSLATION MODEL TRAINING
>
> [TRAINING]
>
> ### training script to be used: either a legacy script or
> # current moses training script (default)
> #
> script = $moses-script-dir/training/train-model.perl
>
> ### general options
> # these are options that are passed on to train-model.perl, for instance
> # * "-mgiza -mgiza-cpus 8" to use mgiza instead of giza
> # * "-sort-buffer-size 8G -sort-compress gzip" to reduce on-disk sorting
> # * "-sort-parallel 8 -cores 8" to speed up phrase table building
> #
> #training-options = ""
>
> ### factored training: specify here which factors used
> # if none specified, single factor training is assumed
> # (one translation step, surface to surface)
> #
> input-factors = word pos
> output-factors = word pos
> alignment-factors = "word -> word"
> translation-factors = "word+pos -> word+pos"
> reordering-factors = "word -> word"
> #generation-factors = "pos -> word"
> decoding-steps = "t0"
>
> ### parallelization of data preparation step
> # the two directions of the data preparation can be run in parallel
> # comment out if not needed
> #
> parallel = yes
>
> ### pre-computation for giza++
> # giza++ has a more efficient data structure that needs to be
> # initialized with snt2cooc. if run in parallel, this may reduces
> # memory requirements. set here the number of parts
> #
> #run-giza-in-parts = 5
>
> ### symmetrization method to obtain word alignments from giza output
> # (commonly used: grow-diag-final-and)
> #
> alignment-symmetrization-method = grow-diag-final-and
>
> ### use of berkeley aligner for word alignment
> #
> #use-berkeley = true
> #alignment-symmetrization-method = berkeley
> #berkeley-train = $moses-script-dir/ems/support/berkeley-train.sh
> #berkeley-process = $moses-script-dir/ems/support/berkeley-process.sh
> #berkeley-jar = /your/path/to/berkeleyaligner-1.1/berkeleyaligner.jar
> #berkeley-java-options = "-server -mx3m -ea"
> #berkeley-training-options = "-Main.iters 5 5 -EMWordAligner.numThreads 8"
> #berkeley-process-options = "-EMWordAligner.numThreads 8"
> #berkeley-posterior = 0.5
>
> ### use of baseline alignment model (incremental training)
> #
> #baseline = 68
> #baseline-alignment-model =
> "$working-dir/training/prepared.$baseline/$input-extension.vcb \
> # $working-dir/training/prepared.$baseline/$output-extension.vcb \
> #
> $working-dir/training/giza.$baseline/${output-extension}-$input-extension.cooc
> \
> #
> $working-dir/training/giza-inverse.$baseline/${input-extension}-$output-extension.cooc
> \
> #
> $working-dir/training/giza.$baseline/${output-extension}-$input-extension.thmm.5
> \
> #
> $working-dir/training/giza.$baseline/${output-extension}-$input-extension.hhmm.5
> \
> #
> $working-dir/training/giza-inverse.$baseline/${input-extension}-$output-extension.thmm.5
> \
> #
> $working-dir/training/giza-inverse.$baseline/${input-extension}-$output-extension.hhmm.5"
>
> ### if word alignment should be skipped,
> # point to word alignment files
> #
> #word-alignment = $working-dir/model/aligned.1
>
> ### filtering some corpora with modified Moore-Lewis
> # specify corpora to be filtered and ratio to be kept, either before or
> after word alignment
> #mml-filter-corpora = toy
> #mml-before-wa = "-proportion 0.9"
> #mml-after-wa = "-proportion 0.9"
>
> ### create a bilingual concordancer for the model
> #
> #biconcor = $moses-script-dir/ems/biconcor/biconcor
>
> ### lexicalized reordering: specify orientation type
> # (default: only distance-based reordering model)
> #
> lexicalized-reordering = msd-bidirectional-fe
>
> ### hierarchical rule set
> #
> #hierarchical-rule-set = true
>
> ### settings for rule extraction
> #
> #extract-settings = ""
> max-phrase-length = 5
>
> ### add extracted phrases from baseline model
> #
> #baseline-extract = $working-dir/model/extract.$baseline
> #
> # requires aligned parallel corpus for re-estimating lexical translation
> probabilities
> #baseline-corpus = $working-dir/training/corpus.$baseline
> #baseline-alignment =
> $working-dir/model/aligned.$baseline.$alignment-symmetrization-method
>
> ### unknown word labels (target syntax only)
> # enables use of unknown word labels during decoding
> # label file is generated during rule extraction
> #
> #use-unknown-word-labels = true
>
> ##
[Moses-support] Tuning crashed, mert.out - no such file or directory. intial BLEU=0
Hi,
My EMS setup (factored,MIRA) crashes at tuning stage after single run.
config.toy: (attaching only training and tuning sections)
*# TRANSLATION MODEL TRAINING
[TRAINING]
### training script to be used: either a legacy script or
# current moses training script (default)
#
script = $moses-script-dir/training/train-model.perl
### general options
# these are options that are passed on to train-model.perl, for instance
# * "-mgiza -mgiza-cpus 8" to use mgiza instead of giza
# * "-sort-buffer-size 8G -sort-compress gzip" to reduce on-disk sorting
# * "-sort-parallel 8 -cores 8" to speed up phrase table building
#
#training-options = ""
### factored training: specify here which factors used
# if none specified, single factor training is assumed
# (one translation step, surface to surface)
#
input-factors = word pos
output-factors = word pos
alignment-factors = "word -> word"
translation-factors = "word+pos -> word+pos"
reordering-factors = "word -> word"
#generation-factors = "pos -> word"
decoding-steps = "t0"
### parallelization of data preparation step
# the two directions of the data preparation can be run in parallel
# comment out if not needed
#
parallel = yes
### pre-computation for giza++
# giza++ has a more efficient data structure that needs to be
# initialized with snt2cooc. if run in parallel, this may reduces
# memory requirements. set here the number of parts
#
#run-giza-in-parts = 5
### symmetrization method to obtain word alignments from giza output
# (commonly used: grow-diag-final-and)
#
alignment-symmetrization-method = grow-diag-final-and
### use of berkeley aligner for word alignment
#
#use-berkeley = true
#alignment-symmetrization-method = berkeley
#berkeley-train = $moses-script-dir/ems/support/berkeley-train.sh
#berkeley-process = $moses-script-dir/ems/support/berkeley-process.sh
#berkeley-jar = /your/path/to/berkeleyaligner-1.1/berkeleyaligner.jar
#berkeley-java-options = "-server -mx3m -ea"
#berkeley-training-options = "-Main.iters 5 5 -EMWordAligner.numThreads 8"
#berkeley-process-options = "-EMWordAligner.numThreads 8"
#berkeley-posterior = 0.5
### use of baseline alignment model (incremental training)
#
#baseline = 68
#baseline-alignment-model =
"$working-dir/training/prepared.$baseline/$input-extension.vcb \
# $working-dir/training/prepared.$baseline/$output-extension.vcb \
#
$working-dir/training/giza.$baseline/${output-extension}-$input-extension.cooc
\
#
$working-dir/training/giza-inverse.$baseline/${input-extension}-$output-extension.cooc
\
#
$working-dir/training/giza.$baseline/${output-extension}-$input-extension.thmm.5
\
#
$working-dir/training/giza.$baseline/${output-extension}-$input-extension.hhmm.5
\
#
$working-dir/training/giza-inverse.$baseline/${input-extension}-$output-extension.thmm.5
\
#
$working-dir/training/giza-inverse.$baseline/${input-extension}-$output-extension.hhmm.5"
### if word alignment should be skipped,
# point to word alignment files
#
#word-alignment = $working-dir/model/aligned.1
### filtering some corpora with modified Moore-Lewis
# specify corpora to be filtered and ratio to be kept, either before or
after word alignment
#mml-filter-corpora = toy
#mml-before-wa = "-proportion 0.9"
#mml-after-wa = "-proportion 0.9"
### create a bilingual concordancer for the model
#
#biconcor = $moses-script-dir/ems/biconcor/biconcor
### lexicalized reordering: specify orientation type
# (default: only distance-based reordering model)
#
lexicalized-reordering = msd-bidirectional-fe
### hierarchical rule set
#
#hierarchical-rule-set = true
### settings for rule extraction
#
#extract-settings = ""
max-phrase-length = 5
### add extracted phrases from baseline model
#
#baseline-extract = $working-dir/model/extract.$baseline
#
# requires aligned parallel corpus for re-estimating lexical translation
probabilities
#baseline-corpus = $working-dir/training/corpus.$baseline
#baseline-alignment =
$working-dir/model/aligned.$baseline.$alignment-symmetrization-method
### unknown word labels (target syntax only)
# enables use of unknown word labels during decoding
# label file is generated during rule extraction
#
#use-unknown-word-labels = true
### if phrase extraction should be skipped,
# point to stem for extract files
#
# extracted-phrases =
### settings for rule scoring
#
score-settings = "--GoodTuring"
### include word alignment in phrase table
#
include-word-alignment-in-rules = yes
### sparse lexical features
#
#sparse-lexical-features = "target-word-insertion top 50,
source-word-deletion top 50, word-translation top 50 50, phrase-length"
### domain adaptation settings
# options: sparse, any of: indicator, subset, ratio
#domain-features = "subset"
### if phrase table training should be skipped,
# point to phrase translation table
#
# phrase-translation-table =
### if reordering table training should be skipped,
# point to reordering table
#
# reordering-table =
### filtering the phrase table based on significance tests
# Johnson, Martin,
