Re: [Moses-support] Adding new aligned phrases to the existing phrase table
Hi, It might be better to do phrase table fill-up. You would add entries from a second phrase table ("background phrase table") to your first phrase table ("foreground phrase table") only if they're not present yet. You end up with a single table without duplicates. Added background phrases can be distinguished from foreground phrases via a binary feature. There's a tool for this in Moses: contrib/combine-ptables/combine-ptables.pl (Use `--mode fillup` for what I've described above. The script provides further functionality which you can also try if you want.) Cf. Arianna Bisazza, Nick Ruiz, and Marcello Federico. 2011. Fill-up versus Interpolation Methods for Phrase-based SMT Adaptation. In Proc. of the Int. Workshop on Spoken Language Translation (IWSLT), pages 136–143, San Francisco, CA, USA, December. or Jan Niehues and Alex Waibel. 2012. Detailed Analysis of Different Strategies for Phrase Table Adaptation in SMT. In Proc. of the Conf. of the Assoc. for Machine Translation in the Americas (AMTA), San Diego, CA, USA, October/November. Cheers, Matthias On Tue, 2017-04-11 at 09:48 +0100, Hieu Hoang wrote: > from the webpage: > http://www.statmt.org/moses/?n=Advanced.Models#ntoc7 > Add 2 phrase-table to the [feature] section > > > [feature] > PhraseDictionaryMemory path=/my-dir/table1 ... > PhraseDictionaryMemory path=/my-dir/table2 ... > > Add an entry to the [mapping] section > [mapping] > 0 T 0 > 1 T 1 > > Add weights to the [weight] section > > [weight] > PhraseDictionaryMemory0= 0 0 1 0 > PhraseDictionaryMemory1= 0 0 1 0 > > You don't need to use PhraseDictionaryGroup > > > > * Looking for MT/NLP opportunities * > Hieu Hoang > http://moses-smt.org/ > > > On 9 April 2017 at 05:21, sriram wrote: > > > Hi Hieu, > > > > Thanks for the suggestion. > > > > In regard to point 2 . How can I use multiple phrase table inside moses? > > > > > > Regards, > > Sriram > > > > On Fri, Apr 7, 2017 at 5:44 PM, Hieu Hoang wrote: > > > > > there's no tools to do this but you can write it yourself. You need to > > > make up some scores to give each phrase. > > > > > > The other methods to use your phrases are: > > >1. Add it to the training data and retrain your model. > > >2. Create a 2nd phrase-table with just your phrases and get the > > > decoder to use it, in addition to the existing phrase-table > > > > > > * Looking for MT/NLP opportunities * > > > Hieu Hoang > > > http://moses-smt.org/ > > > > > > > > > On 6 April 2017 at 19:14, sriram wrote: > > > > > > > Hi, > > > > > > > > I have some good aligned phrases collection and I want to add to the > > > > existing phrase table. Is there any existing tool to add the same in > > > > Moses. > > > > > > > > Thanks, > > > > Sriram > > > > > > > > > > > > > > > > ___ > > > > Moses-support mailing list > > > > Moses-support@mit.edu > > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > > > > > > > > > > > > > > -- > > > > Open source English-Hindi MT system > > http://anusaaraka.iiit.ac.in/ > > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Adding new aligned phrases to the existing phrase table
from the webpage: http://www.statmt.org/moses/?n=Advanced.Models#ntoc7 Add 2 phrase-table to the [feature] section [feature] PhraseDictionaryMemory path=/my-dir/table1 ... PhraseDictionaryMemory path=/my-dir/table2 ... Add an entry to the [mapping] section [mapping] 0 T 0 1 T 1 Add weights to the [weight] section [weight] PhraseDictionaryMemory0= 0 0 1 0 PhraseDictionaryMemory1= 0 0 1 0 You don't need to use PhraseDictionaryGroup * Looking for MT/NLP opportunities * Hieu Hoang http://moses-smt.org/ On 9 April 2017 at 05:21, sriram wrote: > Hi Hieu, > > Thanks for the suggestion. > > In regard to point 2 . How can I use multiple phrase table inside moses? > > > Regards, > Sriram > > On Fri, Apr 7, 2017 at 5:44 PM, Hieu Hoang wrote: > >> there's no tools to do this but you can write it yourself. You need to >> make up some scores to give each phrase. >> >> The other methods to use your phrases are: >>1. Add it to the training data and retrain your model. >>2. Create a 2nd phrase-table with just your phrases and get the >> decoder to use it, in addition to the existing phrase-table >> >> * Looking for MT/NLP opportunities * >> Hieu Hoang >> http://moses-smt.org/ >> >> >> On 6 April 2017 at 19:14, sriram wrote: >> >>> Hi, >>> >>> I have some good aligned phrases collection and I want to add to the >>> existing phrase table. Is there any existing tool to add the same in Moses. >>> >>> Thanks, >>> Sriram >>> >>> >>> >>> ___ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> > > > -- > > Open source English-Hindi MT system > http://anusaaraka.iiit.ac.in/ > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Adding new aligned phrases to the existing phrase table
Hi Hieu, I got the following pointer to use multiple translation models. http://www.statmt.org/moses/?n=Advanced.Models#ntoc7 but when I use the options , I am getting the following error. Feature name PhraseDictionaryGroup is not registered. Does it mean I have to re-compile moses with the "PhraseDictionaryGroup" options. Please provide help for the correct information. Thanks, sriram On Sun, Apr 9, 2017 at 9:51 AM, sriram wrote: > Hi Hieu, > > Thanks for the suggestion. > > In regard to point 2 . How can I use multiple phrase table inside moses? > > > Regards, > Sriram > > On Fri, Apr 7, 2017 at 5:44 PM, Hieu Hoang wrote: > >> there's no tools to do this but you can write it yourself. You need to >> make up some scores to give each phrase. >> >> The other methods to use your phrases are: >>1. Add it to the training data and retrain your model. >>2. Create a 2nd phrase-table with just your phrases and get the >> decoder to use it, in addition to the existing phrase-table >> >> * Looking for MT/NLP opportunities * >> Hieu Hoang >> http://moses-smt.org/ >> >> >> On 6 April 2017 at 19:14, sriram wrote: >> >>> Hi, >>> >>> I have some good aligned phrases collection and I want to add to the >>> existing phrase table. Is there any existing tool to add the same in Moses. >>> >>> Thanks, >>> Sriram >>> >>> >>> >>> ___ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> > > > -- > > Open source English-Hindi MT system > http://anusaaraka.iiit.ac.in/ > > -- Open source English-Hindi MT system http://anusaaraka.iiit.ac.in/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Adding new aligned phrases to the existing phrase table
Hi Hieu, Thanks for the suggestion. In regard to point 2 . How can I use multiple phrase table inside moses? Regards, Sriram On Fri, Apr 7, 2017 at 5:44 PM, Hieu Hoang wrote: > there's no tools to do this but you can write it yourself. You need to > make up some scores to give each phrase. > > The other methods to use your phrases are: >1. Add it to the training data and retrain your model. >2. Create a 2nd phrase-table with just your phrases and get the decoder > to use it, in addition to the existing phrase-table > > * Looking for MT/NLP opportunities * > Hieu Hoang > http://moses-smt.org/ > > > On 6 April 2017 at 19:14, sriram wrote: > >> Hi, >> >> I have some good aligned phrases collection and I want to add to the >> existing phrase table. Is there any existing tool to add the same in Moses. >> >> Thanks, >> Sriram >> >> >> >> ___ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > -- Open source English-Hindi MT system http://anusaaraka.iiit.ac.in/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Adding new aligned phrases to the existing phrase table
there's no tools to do this but you can write it yourself. You need to make up some scores to give each phrase. The other methods to use your phrases are: 1. Add it to the training data and retrain your model. 2. Create a 2nd phrase-table with just your phrases and get the decoder to use it, in addition to the existing phrase-table * Looking for MT/NLP opportunities * Hieu Hoang http://moses-smt.org/ On 6 April 2017 at 19:14, sriram wrote: > Hi, > > I have some good aligned phrases collection and I want to add to the > existing phrase table. Is there any existing tool to add the same in Moses. > > Thanks, > Sriram > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support