Re: [Moses-support] EMS help
Thanks Barry. Answers and other questions below. Le 28/07/2015 10:25, Barry Haddow a écrit : Hi Vincent 2 bugs report : in the LM Corpus definition for Europarl : the $pair-extension is missing before .$output-extension in the step 5 (maybe for others too) generation of the moses.tuned.ini.5 file there is a missing .gz at the end of phrase-table.5 in the PhraseDictionaryMemory definition. These seem OK to me. For europarl, it points to the monolingual corpus, and for the phrase table the .gz is implicitly added. Did they not work for you? I am NOT talking about the [CORPUS:europarl] section but the [LM:europarl] I think in this section you need the $pair-extension same as [LM:nc] where it was fine Anyway :yes I had an error Also : when.gz is missing yes it stops and give an error message. in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. ok will do. I tried to remove the IGNORE for the Interpolated-LM section I am still using KenLM. BUT I get a message saying I need to define srilm-dir is SRILM mandatory to turn on the interpolated-lm with KenLM only ? That's right, the interpolated LM uses some code from SRILM. You can still use KenLM to create the individual language models, and use KenLM during decoding, OK. But related question : If I do not interpolate, and if I keep the 2 (or more) LM in the moses.ini file does the decoder work similarly as if I had interpolated 2 LMs ? cheers - Barry On 26/07/15 08:36, Vincent Nguyen wrote: Hi, I worked with the config.basic file 2 bugs report : in the LM Corpus definition for Europarl : the $pair-extension is missing before .$output-extension in the step 5 (maybe for others too) generation of the moses.tuned.ini.5 file there is a missing .gz at the end of phrase-table.5 in the PhraseDictionaryMemory definition. Then questions : in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. Last question : I tried to remove the IGNORE for the Interpolated-LM section I am still using KenLM. BUT I get a message saying I need to define srilm-dir is SRILM mandatory to turn on the interpolated-lm with KenLM only ? ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS help
Hi Vincent 2 bugs report : in the LM Corpus definition for Europarl : the $pair-extension is missing before .$output-extension in the step 5 (maybe for others too) generation of the moses.tuned.ini.5 file there is a missing .gz at the end of phrase-table.5 in the PhraseDictionaryMemory definition. These seem OK to me. For europarl, it points to the monolingual corpus, and for the phrase table the .gz is implicitly added. Did they not work for you? in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. I tried to remove the IGNORE for the Interpolated-LM section I am still using KenLM. BUT I get a message saying I need to define srilm-dir is SRILM mandatory to turn on the interpolated-lm with KenLM only ? That's right, the interpolated LM uses some code from SRILM. You can still use KenLM to create the individual language models, and use KenLM during decoding, cheers - Barry On 26/07/15 08:36, Vincent Nguyen wrote: Hi, I worked with the config.basic file 2 bugs report : in the LM Corpus definition for Europarl : the $pair-extension is missing before .$output-extension in the step 5 (maybe for others too) generation of the moses.tuned.ini.5 file there is a missing .gz at the end of phrase-table.5 in the PhraseDictionaryMemory definition. Then questions : in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. Last question : I tried to remove the IGNORE for the Interpolated-LM section I am still using KenLM. BUT I get a message saying I need to define srilm-dir is SRILM mandatory to turn on the interpolated-lm with KenLM only ? ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Errors while running phrase based tutorial from Moses manual
Hello I want to run the following command with moses %train-model.perl \ --root-dir unfactored \ --corpus factored-corpus/proj-syndicate \ --f de --e en \ --lm 0:3:factored-corpus/surface.lm:0 \ --external-bin-dir .../tools \ --input-factor-max 4 This creates a moses.ini file but the POS table created in unfactored/phrase-table.o-o.gz come sout to be empty. Also After running the following script for mert tuning: #!/bin/bash myPATH=/media/sudip/linuxdata1 ###Tuning moses.ini, cd to working directory nohup nice $myPATH/mosesdecoder/scripts/training/mert-moses.pl \ $myPATH/working/tuningcorpus/prepared-corpus/news-test2008.clean.fr \ $myPATH/working/tuningcorpus/prepared-corpus/news-test2008.clean.en \ $myPATH/mosesdecoder/bin/moses \ $myPATH/working/train/model/moses.ini \ --mertdir $myPATH/working/train/model/mert \ --decoder-flags='-threads 3' mert.out My mert.out shows the following error: sudip@sudip-pc:/media/sudip/linuxdata1/working$ cat mert.out tune.sh: line 9: /media/sudip/linuxdata1/working/train/model/moses.ini: Permission denied Why is this happening?/ Thanks.. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS help
same error: #!/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games cd /home/moses/working echo 'starting at '`date`' on '`hostname` mkdir -p /home/moses/working/training mkdir -p /home/moses/working/model /home/moses/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/model/moses.ini.5 /home/moses/working/model/moses.bin.ini.6 -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 echo 'finished at '`date` touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE Le 28/07/2015 14:47, Barry Haddow a écrit : Hi Vincent It could be a bug. Could you edit mosesdecoder/scripts/ems/experiment.meta and change the line: template: $binarize-all IN OUT -Binarizer $ttable-binarizer to template: $binarize-all IN OUT -Binarizer $ttable-binarizer Note that I have added quotes. Then you'll have to delete the most recent run, and re-run experiment.perl. If it works, fine. If it doesn't, could you post the steps/6/TRAINING_binarize-config.6 script (hopefully I got the name right - you may need to change the number) cheers - Barry On 28/07/15 13:11, Vincent Nguyen wrote: I know but this is what I have in my config.basic now: # conversion of rule table into binary on-disk format ttable-binarizer = $moses-bin-dir/CreateOnDiskPt 1 1 4 100 2 binarize-all = $moses-script-dir/training/binarize-model.perl I don't where else I can add the 5 arguments or if I need to reference ttable-binarizer somewhere Le 28/07/2015 13:49, Barry Haddow a écrit : Hi Vincent If you look at the error log, you will see: Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath You are missing the first 5 arguments to CreateOnDiskPt, as given in config.basic. cheers - Barry On 28/07/15 12:37, Vincent Nguyen wrote: I don't know why but the binarize crashes see below in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. Executing: /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/model/moses.bin.ini.6.tables /home/moses/working/model/moses.ini.5 /dev/null -nofilter -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables Stripping XML... Executing: /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl /dev/null /home/moses/working/model/moses.bin.ini.6.tables/input.34384 pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0 Considering factor 0 ro:LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz Considering factor 0 Filtering files... filtering /home/moses/working/model/phrase-table.5 - /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... Executing: ln -s /home/moses/working/model/phrase-table.5.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz binarizing... Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath Exit code: 1 Can't binarize at /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl line 417. Exit code: 1 binarising failed at /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS help
Hi Vincent I think the quotes are getting stripped off further down the pipeline. You could work around by changing to the compact phrase table. Or try editing binarize-model.perl to change safesystem($RealBin/filter-model-given-input.pl $targetdir $input_config /dev/null $hierarchical -nofilter -Binarizer $binarizer) || die binarising failed; to safesystem($RealBin/filter-model-given-input.pl $targetdir $input_config /dev/null $hierarchical -nofilter -Binarizer \$binarizer\) || die binarising failed; Note the escaped quotes around the $binarizer. cheers - Barry On 28/07/15 14:09, Vincent Nguyen wrote: same error: #!/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games cd /home/moses/working echo 'starting at '`date`' on '`hostname` mkdir -p /home/moses/working/training mkdir -p /home/moses/working/model /home/moses/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/model/moses.ini.5 /home/moses/working/model/moses.bin.ini.6 -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 echo 'finished at '`date` touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE Le 28/07/2015 14:47, Barry Haddow a écrit : Hi Vincent It could be a bug. Could you edit mosesdecoder/scripts/ems/experiment.meta and change the line: template: $binarize-all IN OUT -Binarizer $ttable-binarizer to template: $binarize-all IN OUT -Binarizer $ttable-binarizer Note that I have added quotes. Then you'll have to delete the most recent run, and re-run experiment.perl. If it works, fine. If it doesn't, could you post the steps/6/TRAINING_binarize-config.6 script (hopefully I got the name right - you may need to change the number) cheers - Barry On 28/07/15 13:11, Vincent Nguyen wrote: I know but this is what I have in my config.basic now: # conversion of rule table into binary on-disk format ttable-binarizer = $moses-bin-dir/CreateOnDiskPt 1 1 4 100 2 binarize-all = $moses-script-dir/training/binarize-model.perl I don't where else I can add the 5 arguments or if I need to reference ttable-binarizer somewhere Le 28/07/2015 13:49, Barry Haddow a écrit : Hi Vincent If you look at the error log, you will see: Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath You are missing the first 5 arguments to CreateOnDiskPt, as given in config.basic. cheers - Barry On 28/07/15 12:37, Vincent Nguyen wrote: I don't know why but the binarize crashes see below in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. Executing: /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/model/moses.bin.ini.6.tables /home/moses/working/model/moses.ini.5 /dev/null -nofilter -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables Stripping XML... Executing: /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl /dev/null /home/moses/working/model/moses.bin.ini.6.tables/input.34384 pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0 Considering factor 0 ro:LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz Considering factor 0 Filtering files... filtering /home/moses/working/model/phrase-table.5 - /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... Executing: ln -s /home/moses/working/model/phrase-table.5.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz binarizing... Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin Usage:
Re: [Moses-support] EMS help
I don't know why but the binarize crashes see below in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. Executing: /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/model/moses.bin.ini.6.tables /home/moses/working/model/moses.ini.5 /dev/null -nofilter -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables Stripping XML... Executing: /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl /dev/null /home/moses/working/model/moses.bin.ini.6.tables/input.34384 pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0 Considering factor 0 ro:LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz Considering factor 0 Filtering files... filtering /home/moses/working/model/phrase-table.5 - /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... Executing: ln -s /home/moses/working/model/phrase-table.5.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz binarizing... Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath Exit code: 1 Can't binarize at /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl line 417. Exit code: 1 binarising failed at /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS help
Hi Vincent If you look at the error log, you will see: Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath You are missing the first 5 arguments to CreateOnDiskPt, as given in config.basic. cheers - Barry On 28/07/15 12:37, Vincent Nguyen wrote: I don't know why but the binarize crashes see below in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. Executing: /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/model/moses.bin.ini.6.tables /home/moses/working/model/moses.ini.5 /dev/null -nofilter -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables Stripping XML... Executing: /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl /dev/null /home/moses/working/model/moses.bin.ini.6.tables/input.34384 pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0 Considering factor 0 ro:LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz Considering factor 0 Filtering files... filtering /home/moses/working/model/phrase-table.5 - /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... Executing: ln -s /home/moses/working/model/phrase-table.5.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz binarizing... Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath Exit code: 1 Can't binarize at /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl line 417. Exit code: 1 binarising failed at /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS help
Hi Vincent It could be a bug. Could you edit mosesdecoder/scripts/ems/experiment.meta and change the line: template: $binarize-all IN OUT -Binarizer $ttable-binarizer to template: $binarize-all IN OUT -Binarizer $ttable-binarizer Note that I have added quotes. Then you'll have to delete the most recent run, and re-run experiment.perl. If it works, fine. If it doesn't, could you post the steps/6/TRAINING_binarize-config.6 script (hopefully I got the name right - you may need to change the number) cheers - Barry On 28/07/15 13:11, Vincent Nguyen wrote: I know but this is what I have in my config.basic now: # conversion of rule table into binary on-disk format ttable-binarizer = $moses-bin-dir/CreateOnDiskPt 1 1 4 100 2 binarize-all = $moses-script-dir/training/binarize-model.perl I don't where else I can add the 5 arguments or if I need to reference ttable-binarizer somewhere Le 28/07/2015 13:49, Barry Haddow a écrit : Hi Vincent If you look at the error log, you will see: Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath You are missing the first 5 arguments to CreateOnDiskPt, as given in config.basic. cheers - Barry On 28/07/15 12:37, Vincent Nguyen wrote: I don't know why but the binarize crashes see below in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. Executing: /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/model/moses.bin.ini.6.tables /home/moses/working/model/moses.ini.5 /dev/null -nofilter -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables Stripping XML... Executing: /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl /dev/null /home/moses/working/model/moses.bin.ini.6.tables/input.34384 pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0 Considering factor 0 ro:LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz Considering factor 0 Filtering files... filtering /home/moses/working/model/phrase-table.5 - /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... Executing: ln -s /home/moses/working/model/phrase-table.5.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz binarizing... Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath Exit code: 1 Can't binarize at /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl line 417. Exit code: 1 binarising failed at /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Generation step for factorized translation
Hi, I'm using EMS and factorized translation. For instance, I have this setting: input-factors = word output-factors = word brown50 alignment-factors = word - word translation-factors = word - word+brown50 generation-factors = brown50 reordering-factors = word - word decoding-steps = t0 that is, I have brown clusters as an output factor. Does someone can point me the proper way to add a generation step to create the surface word using the cluster factor? I tried: generation-factors = brown50 - word but the experiment crashed durinh tuning Thanks in advance Marco ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Problem in translation
Dear All, I try to build a Model but I get an attached error file . is this mean that there are a problem in model . Because I test it by word in training data but the output was wrong. kindly find the attached file. thank you, Fatma El-Zahraa El -Taher Teaching Assistant at Computer System department Faculty of Engineering, Azhar University Email : fatmaelta...@gmail.com mobile: +201141600434 filterphrases.err Description: Binary data ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Problem in translation
Hi Fatma I don't see any error in the file. What do you mean the output was wrong. ? cheers - Barry On 28/07/15 19:13, fatma elzahraa Eltaher wrote: Dear All, I try to build a Model but I get an attached error file . is this mean that there are a problem in model . Because I test it by word in training data but the output was wrong. kindly find the attached file. thank you, Fatma El-Zahraa El -Taher Teaching Assistant at Computer System department Faculty of Engineering, Azhar University Email : fatmaelta...@gmail.com mailto:fatmaelta...@gmail.com mobile: +201141600434 ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] EMS help
Many thanks it seems to have fixed the issue, maybe worth to fix it on github. Le 28/07/2015 15:44, Barry Haddow a écrit : Hi Vincent I think the quotes are getting stripped off further down the pipeline. You could work around by changing to the compact phrase table. Or try editing binarize-model.perl to change safesystem($RealBin/filter-model-given-input.pl $targetdir $input_config /dev/null $hierarchical -nofilter -Binarizer $binarizer) || die binarising failed; to safesystem($RealBin/filter-model-given-input.pl $targetdir $input_config /dev/null $hierarchical -nofilter -Binarizer \$binarizer\) || die binarising failed; Note the escaped quotes around the $binarizer. cheers - Barry On 28/07/15 14:09, Vincent Nguyen wrote: same error: #!/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games cd /home/moses/working echo 'starting at '`date`' on '`hostname` mkdir -p /home/moses/working/training mkdir -p /home/moses/working/model /home/moses/mosesdecoder/scripts/training/binarize-model.perl /home/moses/working/model/moses.ini.5 /home/moses/working/model/moses.bin.ini.6 -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 echo 'finished at '`date` touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE Le 28/07/2015 14:47, Barry Haddow a écrit : Hi Vincent It could be a bug. Could you edit mosesdecoder/scripts/ems/experiment.meta and change the line: template: $binarize-all IN OUT -Binarizer $ttable-binarizer to template: $binarize-all IN OUT -Binarizer $ttable-binarizer Note that I have added quotes. Then you'll have to delete the most recent run, and re-run experiment.perl. If it works, fine. If it doesn't, could you post the steps/6/TRAINING_binarize-config.6 script (hopefully I got the name right - you may need to change the number) cheers - Barry On 28/07/15 13:11, Vincent Nguyen wrote: I know but this is what I have in my config.basic now: # conversion of rule table into binary on-disk format ttable-binarizer = $moses-bin-dir/CreateOnDiskPt 1 1 4 100 2 binarize-all = $moses-script-dir/training/binarize-model.perl I don't where else I can add the 5 arguments or if I need to reference ttable-binarizer somewhere Le 28/07/2015 13:49, Barry Haddow a écrit : Hi Vincent If you look at the error log, you will see: Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath You are missing the first 5 arguments to CreateOnDiskPt, as given in config.basic. cheers - Barry On 28/07/15 12:37, Vincent Nguyen wrote: I don't know why but the binarize crashes see below in my working directory I have 2 subdir, tuning with inside moses.filtered.ini.5 moses.ini.5 moses.tuned.ini.5 and model with inside moses.ini.5 (apparently this one does not have the tuned weights) those in the tuning subdir : the tuned one moses.tuned.ini.5 generated after the moses.ini.5 seems to point on phrase-table.5.gz not binarized and the moses.5.ini seem to point on the binarized within tuning/filtered.5/... unclear to me on which one I should use. If you run EMS, there will be a filtered ini file inside the evaluation directory which can be used to translate the test set using the tuned weights. However this model is filtered for the test set, so you cannot use it on other sentences. If you want the full model binarised, then you should add: binarize-all = $moses-script-dir/training/binarize-model.perl to the [GENERAL] section of the EMS config and rerun EMS. In this case the moses.tuned.ini in tuning can be used to translate any sentences. Executing: /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl /home/moses/working/model/moses.bin.ini.6.tables /home/moses/working/model/moses.ini.5 /dev/null -nofilter -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables Stripping XML... Executing: /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl /dev/null /home/moses/working/model/moses.bin.ini.6.tables/input.34384 pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0 Considering factor 0 ro:LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz Considering factor 0 Filtering files... filtering /home/moses/working/model/phrase-table.5 - /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... Executing: ln -s /home/moses/working/model/phrase-table.5.gz /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz binarizing... Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt
