Re: [Moses-support] error during testing
Hi, I have also the same problem in evaluation step with EMS and I would be thankful if you could help me. the lexical reordering file is emtpy and the log of the output in evaluation_test_filter.2.stderr is: Using SCRIPTS_ROOTDIR: /opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/scripts (9) create moses.ini @ Sat Dec 7 04:50:15 PST 2013 Executing: mkdir -p /opt/tools/workingEms/evaluation/test.filtered.2 Considering factor 0 Considering factor 0 filtering /opt/tools/workingEms/model/phrase-table.2 - /opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1... 0 of 2197240 phrases pairs used (0.00%) - note: max length 10 binarizing...cat /opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1 | LC_ALL=C sort -T /opt/tools/workingEms/evaluation/test.filtered.2 | /opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/bin/processPhraseTable -ttable 0 0 - -nscores 5 -out /opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1 processing ptree for stdin Segmentation fault (core dumped) filtering /opt/tools/workingEms/model/reordering-table.2.wbe-msd-bidirectional-fe.gz - /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe... 0 of 2197240 phrases pairs used (0.00%) - note: max length 10 binarizing.../opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/bin/processLexicalTable -in /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe -out /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe processLexicalTable v0.1 by Konrad Rawlik processing /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe to /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe.* ERROR: empty lexicalised reordering file Barry Haddow bhaddow@... writes: Hi Irene But the output is empty. And the errors are 1. segmentation fault 2. error: empty lexicalized reordering file Is this lexicalised reordering file empty then? It would be helpful if you could post the full log of the output when your run the filter command, cheers - Barry On 26/10/12 17:59, Irene Huang wrote: Hi, I have trained and tuned the model, now I am using ~/mosesdecoder/scripts/training/filter-model-given-input.pl http://filter-model-given-input.pl filtered-newstest2011 mert-work/moses.ini ~/corpus/newstest2011.true.fr http://newstest2011.true.fr \ -Binarizer ~/mosesdecoder/bin/processPhraseTable to filter the phrase table. But the output is empty. And the errors are 1. segmentation fault 2. error: empty lexicalized reordering file So does this mean it's out of memory error? Thanks ___ Moses-support mailing list Moses-support@... http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Specifying LM type in moses build
On 6 December 2013 22:50, Jeffrey Micher jmic...@cs.cmu.edu wrote: Hi, I’m trying to use KenLM instead of SRILM. I initially built moses with SRILM support. The documentation for using the various LMs has changed today. Now it’s not clear what I should do. The documentation has been changed to reflect the new moses.ini file format. The new format is used in the current github code, and in the forthcoming RELEASE-2.0. The old format was used in RELEASE-1.0 I seem to remember reading that I should use 8 for KENLM, 0 for SRILM when I’m training, so I used to use: -lm 0:3:path_to_srilm (3 gram) I changed it to: -lm 8:5:path_to_kenlm (5 gram) to create a moses.ini file that says KENLM the train-model.perl argument is -lm 0:5:path_to_kenlm:8 When I did this, my moses.ini file had something like: SRILM name=LM0 factor=8 path=/home/jmicher/experiments/exp13/data/exp13.lm.bin order=5 I went to check the documentation again in two places: first in the Language Modeling page, where I finally figured out that the documentation had been changed earlier today… so I can edit my moses.ini file now to have KENLM name=LM0 order=5 …etc But how do I specify this when I’m training? I went to the Training Reference page and found: --lm -- language model: factor:order:filename (option can be repeated) On the Build Baseline page, the training parameter for specifying lm info is given as: -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8 Why is there an 8 at the end of this string? And how should I now specify that I want to use KenLM when run the training script? Thanks. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] error during testing
are you sure the parallel data is encoded in UTF8? Was it tokenized, cleaned and escaped by the Moses scripts or by another external script? Can you please send me you EMS config file too On 7 December 2013 14:03, amir haghighi amir.haghighi...@gmail.com wrote: Hi, I have also the same problem in evaluation step with EMS and I would be thankful if you could help me. the lexical reordering file is emtpy and the log of the output in evaluation_test_filter.2.stderr is: Using SCRIPTS_ROOTDIR: /opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/scripts (9) create moses.ini @ Sat Dec 7 04:50:15 PST 2013 Executing: mkdir -p /opt/tools/workingEms/evaluation/test.filtered.2 Considering factor 0 Considering factor 0 filtering /opt/tools/workingEms/model/phrase-table.2 - /opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1... 0 of 2197240 phrases pairs used (0.00%) - note: max length 10 binarizing...cat /opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1 | LC_ALL=C sort -T /opt/tools/workingEms/evaluation/test.filtered.2 | /opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/bin/processPhraseTable -ttable 0 0 - -nscores 5 -out /opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1 processing ptree for stdin Segmentation fault (core dumped) filtering /opt/tools/workingEms/model/reordering-table.2.wbe-msd-bidirectional-fe.gz - /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe... 0 of 2197240 phrases pairs used (0.00%) - note: max length 10 binarizing.../opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/bin/processLexicalTable -in /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe -out /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe processLexicalTable v0.1 by Konrad Rawlik processing /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe to /opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe.* ERROR: empty lexicalised reordering file Barry Haddow bhaddow@... writes: Hi Irene But the output is empty. And the errors are 1. segmentation fault 2. error: empty lexicalized reordering file Is this lexicalised reordering file empty then? It would be helpful if you could post the full log of the output when your run the filter command, cheers - Barry On 26/10/12 17:59, Irene Huang wrote: Hi, I have trained and tuned the model, now I am using ~/mosesdecoder/scripts/training/filter-model-given-input.pl http://filter-model-given-input.pl filtered-newstest2011 mert-work/moses.ini ~/corpus/newstest2011.true.fr http://newstest2011.true.fr \ -Binarizer ~/mosesdecoder/bin/processPhraseTable to filter the phrase table. But the output is empty. And the errors are 1. segmentation fault 2. error: empty lexicalized reordering file So does this mean it's out of memory error? Thanks ___ Moses-support mailing list Moses-support@... http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
