it will binarize AFTER it filters. It can't filter already binarized tables
On 16/09/2016 11:10, Steve Braich wrote:
Hi Hieu,
OK, but then why does the filtering script have an option for using
the binary "-Binarizer" Phrase Table?
Thanks,
Steve
*~/mosesdecoder/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl/> filtered-newstest2011
mert-work/moses.ini ~/corpus/newstest2011.true.fr
<http://newstest2011.true.fr/> -Binarizer
~/mosesdecoder/bin/processPhraseTableMin*
On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang <hieuho...@gmail.com
<mailto:hieuho...@gmail.com>> wrote:
ah. from the moses.ini file, it looks like you've binarized the
phrase-table (and the re-ordering model?) already.
The filtering script only works with moses.ini which has the
original text pt and reordering files
Hieu Hoang
http://www.hoang.co.uk/hieu
On 16 September 2016 at 09:13, Steve Braich <steveb...@gmail.com
<mailto:steveb...@gmail.com>> wrote:
I included the moses.ini file in the original post.
I checked in VIM using :set list. Everything looks good. I
repost it here:
# MERT optimized configuration
# decoder /home/steve/mosesdecoder/bin/moses
# BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr
<http://news-test2008.true.fr>
# We were before running iteration 5
# finished Thu Sep 15 01:53:22 PDT 2016
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
[distortion-limit]
6
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/home/steve/working/binarised-model/phrase-table.minphr
input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/home/steve/working/binarised-model/reordering-table
Distortion
KENLM name=LM0 factor=0
path=/home/steve/lm/news-commentary-v8.fr-en.blm.en order=3
# dense weights for feature functions
[threads]
4
[weight]
LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621
0.0141898 0.0348721
Distortion0= 0.0721205
LM0= 0.0908638
WordPenalty0= -0.153776
PhrasePenalty0= 0.0874059
TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132
UnknownWordPenalty0= 1
On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang
<hieuho...@gmail.com <mailto:hieuho...@gmail.com>> wrote:
look inside the file
mert-work/moses.ini
is there anything strange going on? *
*
Hieu Hoang
http://www.hoang.co.uk/hieu
On 16 September 2016 at 09:02, Steve Braich
<steveb...@gmail.com <mailto:steveb...@gmail.com>> wrote:
Hi Hieu,
I changed from relative path to absolute. Still same
error.
Thanks,
Steve
On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang
<hieuho...@gmail.com <mailto:hieuho...@gmail.com>> wrote:
don't use relative paths, ie. change
filtered-newstest2011 ->
/whatever/filtered-newstest2011
mert-work/moses.ini -> /whatever/mert-work/moses.ini
*
*
Hieu Hoang
http://www.hoang.co.uk/hieu
On 16 September 2016 at 05:42, Steve Braich
<steveb...@gmail.com <mailto:steveb...@gmail.com>>
wrote:
Hello,
I am going thru the steps described in the
baseline system instructions. I have no
problems until the very end when I try to
filter out the test data from the binarised model.
Here is the step in the instructions where the
error appears:
*The model that we've trained can then be
filtered for this test set, meaning that we
only retain the entries needed translate the
test set. This will make the translation a lot
faster.*
**
*cd ~/working*
*~/mosesdecoder/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl>
filtered-newstest2011 mert-work/moses.ini
~/corpus/newstest2011.true.fr
<http://newstest2011.true.fr> -Binarizer
~/mosesdecoder/bin/processPhraseTableMin*
Here is the error message:
*Can't open '<
/home/steve/working/binarised-model/reordering-table'
at
/home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> line 439.*
Keep in mind the following:
* I compiled moses with the following
parameters. It is the same compilation. I
did not recompile moses at any point
during the process.
./bjam -a
--with-boost=/home/steve/mosesdecoder/boost_1_55_0
--with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0
NOTE: I did try recompiling with the
absolute path for cmph. Same error.
* I checked all paths first to make sure
they exist.
* I am able to translate segments using the
binarised model with no problem
* I provided my moses.ini file below.
Questions:
* Can I recompile moses after training
and/or tuning, and expect that it should
work for testing? I ask because I want to
know if the solution is to modify the
compile options, do I have to start from
scratch again? (Can you filter a
binarised model using moses with different
compile options that were used to train
the model?)
Thanks,
Steve
Moses.ini file
# MERT optimized configuration
# decoder /home/steve/mosesdecoder/bin/moses
# BLEU 0.18611 on dev
/home/steve/corpus/news-test2008.true.fr
<http://news-test2008.true.fr>
# We were before running iteration 5
# finished Thu Sep 15 01:53:22 PDT 2016
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
[distortion-limit]
6
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0
num-features=4
path=/home/steve/working/binarised-model/phrase-table.minphr
input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0
num-features=6
type=wbe-msd-bidirectional-fe-allff
input-factor=0 output-factor=0
path=/home/steve/working/binarised-model/reordering-table
Distortion
KENLM name=LM0 factor=0
path=/home/steve/lm/news-commentary-v8.fr-en.blm.en
order=3
# dense weights for feature functions
[threads]
4
[weight]
LexicalReordering0= 0.0760973 0.0426988
0.0872591 0.0872621 0.0141898 0.0348721
Distortion0= 0.0721205
LM0= 0.0908638
WordPenalty0= -0.153776
PhrasePenalty0= 0.0874059
TranslationModel0= 0.0162038 0.0871216
0.0687163 0.0814132
UnknownWordPenalty0= 1
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
<mailto:Moses-support@mit.edu>
http://mailman.mit.edu/mailman/listinfo/moses-support
<http://mailman.mit.edu/mailman/listinfo/moses-support>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
