Re: [Moses-support] Steps for setting up Cache Based Models
Hi all, Moving CBPT to foreground as suggested by Prashant working for my use case. Could any please suggest, how to generate word alignments for CBPT? Is there some tool or script available for generating alignments? I tried fast_align but the output format is totally different. Any help is much appreciated. Kind Regards, Parmpreet > On 21/07/2016, at 11:08 AM, Prashant Mathur <prash...@fbk.eu> wrote: > > Hi Parmpreet, > > Just FYI... > CBPT is meant to work as a foreground PT but it depends on the use case of > course. The chances of decoder using the phrases in the cache are more if the > 0th phrase table is CBPT and not the background PT. In the sense that instead > of this order of PTs > > PhraseDictionaryMemory name=TranslationModel0 num-features=4 > path=/path/to/model/phrase-table.1.gz input-factor=0 output-factor=0 > DynamicCacheBasedLanguageModel name=CBLM0 num-features=1 cblm-score-type=12 > cblm-query-type=1 cblm-max-age=1000 > > you have this order > > DynamicCacheBasedLanguageModel name=CBLM0 num-features=1 cblm-score-type=12 > cblm-query-type=1 cblm-max-age=1000 > PhraseDictionaryMemory name=TranslationModel0 num-features=4 > path=/path/to/model/phrase-table.1.gz input-factor=0 output-factor=0 > > —Prashant > >> On Jul 19, 2016, at 4:10 AM, Parmpreet Singh <p...@shadocms.com >> <mailto:p...@shadocms.com>> wrote: >> >> Hi All, >> >> >> Can anyone help me setting up Cache based Phrase tables? >> >> Please see detailed information below. >> >> Thanks, >> Parmpreet Singh >> >>> On 13/07/2016, at 11:52 AM, Parmpreet Singh <p...@shadocms.com >>> <mailto:p...@shadocms.com>> wrote: >>> >>> Hi All, >>> >>> I am trying to setup Dynamic Cache-Based Phrase Table for post edit >>> translation adaptation. >>> >>> I am following this tutorial: >>> http://www.statmt.org/moses/?n=Advanced.CacheBased >>> <http://www.statmt.org/moses/?n=Advanced.CacheBased> >>> Adaptive MT server: >>> https://307d7cc8-a-db0463cf-s-sites.googlegroups.com/a/fbk.eu/mt4cat/file-cabinet/AdaptiveMTserver-manual.pdf?attachauth=ANoY7crb4vDqMv94wuQREg76SnBs0jk3KdfMwTml0T78EAwNQbUgbmnvJQzUNJRbAnSm6TQwduhwfmvMa84r7JX1TXqhJjtsUSx2mOdkyrWUEiXXhBrCbWFPwcuyw575nm8Co1DP_l1aSa5Ur3v4AsFRtvmevFoLnNRuYa0bM2g7nsjsI_4s_sooPMODoVAoM7a-mcn1wI6dKgHLfpOO2DDyA0kyvTFkK4xj4w87FUGGKXglNj1uTyg%3D=0 >>> >>> <https://307d7cc8-a-db0463cf-s-sites.googlegroups.com/a/fbk.eu/mt4cat/file-cabinet/AdaptiveMTserver-manual.pdf?attachauth=ANoY7crb4vDqMv94wuQREg76SnBs0jk3KdfMwTml0T78EAwNQbUgbmnvJQzUNJRbAnSm6TQwduhwfmvMa84r7JX1TXqhJjtsUSx2mOdkyrWUEiXXhBrCbWFPwcuyw575nm8Co1DP_l1aSa5Ur3v4AsFRtvmevFoLnNRuYa0bM2g7nsjsI_4s_sooPMODoVAoM7a-mcn1wI6dKgHLfpOO2DDyA0kyvTFkK4xj4w87FUGGKXglNj1uTyg==0> >>> >>> Moses.ini is: >>> >>> # >>> ### MOSES CONFIG FILE ### >>> # >>> >>> # input factors >>> [input-factors] >>> 0 >>> >>> [xml-input] >>> inclusive >>> >>> # mapping steps >>> [mapping] >>> 0 T 0 >>> 1 T 1 >>> >>> >>> [distortion-limit] >>> 6 >>> >>> # feature functions >>> [feature] >>> KENLM lazyken=0 name=LM0 factor=0 path=/path/to/lm/nc.binlm.1 order=5 >>> Distortion >>> LexicalReordering name=LexicalReordering0 num-features=6 >>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 >>> path=/path/to/model/reordering-table.1.wbe-msd-bidirectional-fe.gz >>> UnknownWordPenalty >>> WordPenalty >>> PhrasePenalty >>> PhraseDictionaryMemory name=TranslationModel0 num-features=4 >>> path=/path/to/model/phrase-table.1.gz input-factor=0 output-factor=0 >>> >>> PhraseDictionaryDynamicCacheBased name=CBTM0 input-factor=0 output-factor=0 >>> num-features=1 table-limit=20 cbtm-score-type=12 >>> DynamicCacheBasedLanguageModel name=CBLM0 num-features=1 cblm-score-type=12 >>> cblm-query-type=1 cblm-max-age=1000 >>> >>> # dense weights for feature functions >>> [weight] >>> # The default weights are NOT optimized for translation quality. You MUST >>> tune the weights. >>> # Documentation for tuning is here: >>> http://www.statmt.org/moses/?n=FactoredTraining.Tuning >>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> >>> UnknownWordPenalty0= 1 >>> WordPenalty0= -1 >>
Re: [Moses-support] Steps for setting up Cache Based Models
Hi All, Can anyone help me setting up Cache based Phrase tables? Please see detailed information below. Thanks, Parmpreet Singh > On 13/07/2016, at 11:52 AM, Parmpreet Singh <p...@shadocms.com> wrote: > > Hi All, > > I am trying to setup Dynamic Cache-Based Phrase Table for post edit > translation adaptation. > > I am following this tutorial: > http://www.statmt.org/moses/?n=Advanced.CacheBased > <http://www.statmt.org/moses/?n=Advanced.CacheBased> > Adaptive MT server: > https://307d7cc8-a-db0463cf-s-sites.googlegroups.com/a/fbk.eu/mt4cat/file-cabinet/AdaptiveMTserver-manual.pdf?attachauth=ANoY7crb4vDqMv94wuQREg76SnBs0jk3KdfMwTml0T78EAwNQbUgbmnvJQzUNJRbAnSm6TQwduhwfmvMa84r7JX1TXqhJjtsUSx2mOdkyrWUEiXXhBrCbWFPwcuyw575nm8Co1DP_l1aSa5Ur3v4AsFRtvmevFoLnNRuYa0bM2g7nsjsI_4s_sooPMODoVAoM7a-mcn1wI6dKgHLfpOO2DDyA0kyvTFkK4xj4w87FUGGKXglNj1uTyg%3D=0 > > <https://307d7cc8-a-db0463cf-s-sites.googlegroups.com/a/fbk.eu/mt4cat/file-cabinet/AdaptiveMTserver-manual.pdf?attachauth=ANoY7crb4vDqMv94wuQREg76SnBs0jk3KdfMwTml0T78EAwNQbUgbmnvJQzUNJRbAnSm6TQwduhwfmvMa84r7JX1TXqhJjtsUSx2mOdkyrWUEiXXhBrCbWFPwcuyw575nm8Co1DP_l1aSa5Ur3v4AsFRtvmevFoLnNRuYa0bM2g7nsjsI_4s_sooPMODoVAoM7a-mcn1wI6dKgHLfpOO2DDyA0kyvTFkK4xj4w87FUGGKXglNj1uTyg==0> > > Moses.ini is: > > # > ### MOSES CONFIG FILE ### > # > > # input factors > [input-factors] > 0 > > [xml-input] > inclusive > > # mapping steps > [mapping] > 0 T 0 > 1 T 1 > > > [distortion-limit] > 6 > > # feature functions > [feature] > KENLM lazyken=0 name=LM0 factor=0 path=/path/to/lm/nc.binlm.1 order=5 > Distortion > LexicalReordering name=LexicalReordering0 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/path/to/model/reordering-table.1.wbe-msd-bidirectional-fe.gz > UnknownWordPenalty > WordPenalty > PhrasePenalty > PhraseDictionaryMemory name=TranslationModel0 num-features=4 > path=/path/to/model/phrase-table.1.gz input-factor=0 output-factor=0 > > PhraseDictionaryDynamicCacheBased name=CBTM0 input-factor=0 output-factor=0 > num-features=1 table-limit=20 cbtm-score-type=12 > DynamicCacheBasedLanguageModel name=CBLM0 num-features=1 cblm-score-type=12 > cblm-query-type=1 cblm-max-age=1000 > > # dense weights for feature functions > [weight] > # The default weights are NOT optimized for translation quality. You MUST > tune the weights. > # Documentation for tuning is here: > http://www.statmt.org/moses/?n=FactoredTraining.Tuning > <http://www.statmt.org/moses/?n=FactoredTraining.Tuning> > UnknownWordPenalty0= 1 > WordPenalty0= -1 > PhrasePenalty0= 0.2 > TranslationModel0= 0.2 0.2 0.2 0.2 > LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 > Distortion0= 0.3 > LM0= 0.5 > CBTM0= 1 > CBLM0= 1 > > > > Command to start moses SMT is: mosesdecoder/bin/moses -f /path/to/moses.ini -t > > Output of the above command is: > Defined parameters (per moses.ini or switch): > beam-threshold: 0.03 > config: /path/to/moses.ini > distortion-limit: 6 > feature: KENLM lazyken=0 name=LM0 factor=0 path=/path/to/lm/nc.binlm.1 > order=5 Distortion LexicalReordering name=LexicalReordering0 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/mnt/data/apps/models/baseline/en_fr/model/reordering-table.1.wbe-msd-bidirectional-fe.gz > UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory > name=TranslationModel0 num-features=4 > path=/mnt/data/apps/models/baseline/en_fr/model/phrase-table.1.gz > input-factor=0 output-factor=0 PhraseDictionaryDynamicCacheBased name=CBTM0 > input-factor=0 output-factor=0 num-features=1 table-limit=20 > cbtm-score-type=12 cbtm-constant=true DynamicCacheBasedLanguageModel > name=CBLM0 num-features=1 cblm-score-type=12 cblm-query-type=1 > cblm-max-age=1000 cblm-constant=true > input-factors: 0 > mapping: 0 T 0 1 T 1 > report-segmentation: > weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 > TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 > 0.3 Distortion0= 0.3 LM0= 0.5 CBTM0= 1.0 CBLM0= 1.0 > xml-input: exclusive > line=KENLM lazyken=0 name=LM0 factor=0 path=/path/to/lm/nc.binlm.1 order=5 > FeatureFunction: LM0 start: 0 end: 0 > line=Distortion > FeatureFunction: Distortion0 start: 1 end: 1 > line=LexicalReordering name=LexicalReordering0 num-features=6 > type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 > path=/path/to/model/reordering-table.1.wbe-msd-bidirectional-fe.gz > Initializing Lexical Reordering Feature.. > FeatureFunction: Le
