it will binarize AFTER it filters. It can't filter already binarized tables
Hieu Hoang http://www.hoang.co.uk/hieu On 16 September 2016 at 11:10, Steve Braich <steveb...@gmail.com> wrote: > Hi Hieu, > > OK, but then why does the filtering script have an option for using the > binary "-Binarizer" Phrase Table? > > Thanks, > Steve > > *~/mosesdecoder/scripts/training/filter-model-given-input.pl > <http://filter-model-given-input.pl/> filtered-newstest2011 > mert-work/moses.ini ~/corpus/newstest2011.true.fr > <http://newstest2011.true.fr/> -Binarizer > ~/mosesdecoder/bin/processPhraseTableMin* > > On Fri, Sep 16, 2016 at 1:49 AM, Hieu Hoang <hieuho...@gmail.com> wrote: > >> ah. from the moses.ini file, it looks like you've binarized the >> phrase-table (and the re-ordering model?) already. >> >> The filtering script only works with moses.ini which has the original >> text pt and reordering files >> >> Hieu Hoang >> http://www.hoang.co.uk/hieu >> >> On 16 September 2016 at 09:13, Steve Braich <steveb...@gmail.com> wrote: >> >>> I included the moses.ini file in the original post. >>> >>> I checked in VIM using :set list. Everything looks good. I repost it >>> here: >>> >>> # MERT optimized configuration >>> # decoder /home/steve/mosesdecoder/bin/moses >>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >>> # We were before running iteration 5 >>> # finished Thu Sep 15 01:53:22 PDT 2016 >>> ### MOSES CONFIG FILE ### >>> ######################### >>> >>> # input factors >>> [input-factors] >>> 0 >>> >>> # mapping steps >>> [mapping] >>> 0 T 0 >>> >>> [distortion-limit] >>> 6 >>> >>> # feature functions >>> [feature] >>> UnknownWordPenalty >>> WordPenalty >>> PhrasePenalty >>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>> path=/home/steve/working/binarised-model/phrase-table.minphr >>> input-factor=0 output-factor=0 >>> LexicalReordering name=LexicalReordering0 num-features=6 >>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>> path=/home/steve/working/binarised-model/reordering-table >>> Distortion >>> KENLM name=LM0 factor=0 path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >>> order=3 >>> >>> # dense weights for feature functions >>> >>> [threads] >>> 4 >>> [weight] >>> >>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 0.0141898 >>> 0.0348721 >>> Distortion0= 0.0721205 >>> LM0= 0.0908638 >>> WordPenalty0= -0.153776 >>> PhrasePenalty0= 0.0874059 >>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >>> UnknownWordPenalty0= 1 >>> >>> On Fri, Sep 16, 2016 at 12:10 AM, Hieu Hoang <hieuho...@gmail.com> >>> wrote: >>> >>>> look inside the file >>>> mert-work/moses.ini >>>> is there anything strange going on? >>>> >>>> Hieu Hoang >>>> http://www.hoang.co.uk/hieu >>>> >>>> On 16 September 2016 at 09:02, Steve Braich <steveb...@gmail.com> >>>> wrote: >>>> >>>>> Hi Hieu, >>>>> I changed from relative path to absolute. Still same error. >>>>> >>>>> Thanks, >>>>> Steve >>>>> >>>>> On Thu, Sep 15, 2016 at 8:54 PM, Hieu Hoang <hieuho...@gmail.com> >>>>> wrote: >>>>> >>>>>> don't use relative paths, ie. change >>>>>> filtered-newstest2011 -> /whatever/filtered-newstest2011 >>>>>> mert-work/moses.ini -> /whatever/mert-work/moses.ini >>>>>> >>>>>> >>>>>> Hieu Hoang >>>>>> http://www.hoang.co.uk/hieu >>>>>> >>>>>> On 16 September 2016 at 05:42, Steve Braich <steveb...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I am going thru the steps described in the baseline system >>>>>>> instructions. I have no problems until the very end when I try to >>>>>>> filter >>>>>>> out the test data from the binarised model. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Here is the step in the instructions where the error appears: >>>>>>> >>>>>>> *The model that we've trained can then be filtered for this test >>>>>>> set, meaning that we only retain the entries needed translate the test >>>>>>> set. >>>>>>> This will make the translation a lot faster.* >>>>>>> >>>>>>> >>>>>>> >>>>>>> *cd ~/working* >>>>>>> >>>>>>> *~/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>>>> <http://filter-model-given-input.pl> filtered-newstest2011 >>>>>>> mert-work/moses.ini ~/corpus/newstest2011.true.fr >>>>>>> <http://newstest2011.true.fr> -Binarizer >>>>>>> ~/mosesdecoder/bin/processPhraseTableMin* >>>>>>> >>>>>>> >>>>>>> >>>>>>> Here is the error message: >>>>>>> >>>>>>> *Can't open '< /home/steve/working/binarised-model/reordering-table' >>>>>>> at /home/steve/mosesdecoder/scripts/training/filter-model-given-input.pl >>>>>>> <http://filter-model-given-input.pl> line 439.* >>>>>>> >>>>>>> >>>>>>> >>>>>>> Keep in mind the following: >>>>>>> >>>>>>> - I compiled moses with the following parameters. It is the >>>>>>> same compilation. I did not recompile moses at any point during the >>>>>>> process. >>>>>>> ./bjam -a --with-boost=/home/steve/mosesdecoder/boost_1_55_0 >>>>>>> --with-cmph=$HOME/mosesdecoder/cmph/cmph-2.0 >>>>>>> NOTE: I did try recompiling with the absolute path for cmph. >>>>>>> Same error. >>>>>>> - I checked all paths first to make sure they exist. >>>>>>> - I am able to translate segments using the binarised model with >>>>>>> no problem >>>>>>> - I provided my moses.ini file below. >>>>>>> >>>>>>> Questions: >>>>>>> >>>>>>> - Can I recompile moses after training and/or tuning, and expect >>>>>>> that it should work for testing? I ask because I want to know if the >>>>>>> solution is to modify the compile options, do I have to start from >>>>>>> scratch >>>>>>> again? (Can you filter a binarised model using moses with different >>>>>>> compile options that were used to train the model?) >>>>>>> >>>>>>> Thanks, >>>>>>> Steve >>>>>>> >>>>>>> >>>>>>> Moses.ini file >>>>>>> >>>>>>> # MERT optimized configuration >>>>>>> >>>>>>> # decoder /home/steve/mosesdecoder/bin/moses >>>>>>> >>>>>>> # BLEU 0.18611 on dev /home/steve/corpus/news-test2008.true.fr >>>>>>> >>>>>>> # We were before running iteration 5 >>>>>>> >>>>>>> # finished Thu Sep 15 01:53:22 PDT 2016 >>>>>>> >>>>>>> ### MOSES CONFIG FILE ### >>>>>>> >>>>>>> ######################### >>>>>>> >>>>>>> >>>>>>> # input factors >>>>>>> >>>>>>> [input-factors] >>>>>>> >>>>>>> 0 >>>>>>> >>>>>>> >>>>>>> # mapping steps >>>>>>> >>>>>>> [mapping] >>>>>>> >>>>>>> 0 T 0 >>>>>>> >>>>>>> >>>>>>> [distortion-limit] >>>>>>> >>>>>>> 6 >>>>>>> >>>>>>> >>>>>>> # feature functions >>>>>>> >>>>>>> [feature] >>>>>>> >>>>>>> UnknownWordPenalty >>>>>>> >>>>>>> WordPenalty >>>>>>> >>>>>>> PhrasePenalty >>>>>>> >>>>>>> PhraseDictionaryCompact name=TranslationModel0 num-features=4 >>>>>>> path=/home/steve/working/binarised-model/phrase-table.minphr >>>>>>> input-factor=0 output-factor=0 >>>>>>> >>>>>>> LexicalReordering name=LexicalReordering0 num-features=6 >>>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>>>>>> path=/home/steve/working/binarised-model/reordering-table >>>>>>> >>>>>>> Distortion >>>>>>> >>>>>>> KENLM name=LM0 factor=0 >>>>>>> path=/home/steve/lm/news-commentary-v8.fr-en.blm.en >>>>>>> order=3 >>>>>>> >>>>>>> >>>>>>> # dense weights for feature functions >>>>>>> >>>>>>> >>>>>>> [threads] >>>>>>> >>>>>>> 4 >>>>>>> >>>>>>> [weight] >>>>>>> >>>>>>> >>>>>>> LexicalReordering0= 0.0760973 0.0426988 0.0872591 0.0872621 >>>>>>> 0.0141898 0.0348721 >>>>>>> >>>>>>> Distortion0= 0.0721205 >>>>>>> >>>>>>> LM0= 0.0908638 >>>>>>> >>>>>>> WordPenalty0= -0.153776 >>>>>>> >>>>>>> PhrasePenalty0= 0.0874059 >>>>>>> >>>>>>> TranslationModel0= 0.0162038 0.0871216 0.0687163 0.0814132 >>>>>>> >>>>>>> UnknownWordPenalty0= 1 >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Moses-support mailing list >>>>>>> Moses-support@mit.edu >>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
