Re: [Moses-support] Target-syntax
Hi Rico and thank you for your response, I have implemented your suggestions, the training goes well but still produces empty rule-table and gluerule. I noticed that as I turn the switch (-target-syntax or source-syntax) off for a regular hierarchical, the rule-table are not empty anymore, so I thought maybe I'm missing some extraction options? Regards Massinissa 2015-02-27 18:17 GMT+01:00 Rico Sennrich rico.sennr...@gmx.ch: Massinissa Ahmim massinissa.ahmim@... writes: Dear all, I'm trying to train a syntactic model english to german. I did the annotation on the target part using bitpar and ran :nohup /mosesdecoder/scripts/training/train-model.perl --glue-grammar --max-phrase-length 10 --extract-options=--MaxSpan 15 --score-options=--GoodTuring -root-dir /home/Massi/KIID_ENDE/syntax/ -corpus /home/Massi/KIID_DEEN/kiid.10 -f en -e de -lm 0:5:/home/Massi/KIID_ENDE/LM/atelier/lm.kiid5.blm.de.mm -hierarchical -target-syntax /home/Massi/KIID_DEEN/tagged.kiid.10.de -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out the training went very well but outputs empty ruletable, I double-checked my paths but everything seems to be okay,Any ideas? Many thanks Massinissa Hi Massinissa, I cannot tell from here if your files are in the right format. As to your training parameters, what springs to mind is that your extract options are only suited for hierarchical models, not for syntactic ones. You can use the option '-ghkm' to use the GHKM extractor, which has more sensible defaults for string-to-tree systems. Alternatively, you should consider changing the following extract-options: --NonTermConsecSource (to allow consecutive non-terminal symbols on the source side of a rule) --MinHoleSource 1 (to allow nonterminals that only span 1 word) --MinWords 0 (to allow non-lexical rules) --MaxNonTerm SIZE (to allow SIZE nonterminals per rule (default 2)) The full list is here: http://www.statmt.org/moses/?n=Moses.SyntaxTutorial#ntoc14 best wishes, Rico ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Target-syntax
Dear all, I'm trying to train a syntactic model english to german. I did the annotation on the target part using bitpar and ran : nohup /mosesdecoder/scripts/training/train-model.perl --glue-grammar --max-phrase-length 10 --extract-options=--MaxSpan 15 --score-options=--GoodTuring -root-dir /home/Massi/KIID_ENDE/syntax/ -corpus /home/Massi/KIID_DEEN/kiid.10 -f en -e de -lm 0:5:/home/Massi/KIID_ENDE/LM/atelier/lm.kiid5.blm.de.mm -hierarchical -target-syntax /home/Massi/KIID_DEEN/tagged.kiid.10.de -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out the training went very well but outputs empty ruletable, I double-checked my paths but everything seems to be okay, Any ideas? Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Syntax training crashing
I'm running it on the parsed data...how to implement the syntactic annotation then? 2014-12-24 19:57 GMT+01:00 Hieu Hoang hieu.ho...@ed.ac.uk: You're running the word alignment on the unparsed data, right? Mgiza doesn't know anything about parsing Sent while bumping into things On 24 Dec 2014 20:01, Massinissa Ahmim massinissa.ah...@linguacustodia.com wrote: Dear all, I'm trying to train a syntactic model english to german. I did the annotation on the target part using bitpar and ran : nohup /mosesdecoder/scripts/training/train-model.perl -target-syntax --glue-grammar --max-phrase-length 10 --extract-options=--MaxSpan 15 --score-options=--GoodTuring -root-dir /home/Massi/KIID_ENDE/syntax/ -corpus /home/Massi/KIID_DEEN/tagged.kiid -f en -e de -lm 0:5:/home/Massi/KIID_ENDE/LM/atelier/lm.kiid5.blm.de.mm -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out but the training is crashing with the following message : ERROR: Execution of: /root/external-bin-dir/mgiza -CoocurrenceFile /home/Massi/KIID_ENDE/syntax/giza.de-en/de-en.cooc -c /home/Massi/KIID_ENDE/syntax/corpus/de-en-int-train.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -ncpus 30 -nodumps 1 -nsmooth 4 -o /home/Massi/KIID_ENDE/syntax/giza.de-en/de-en -onlyaldumps 1 -p0 0.999 -s /home/Massi/KIID_ENDE/syntax/corpus/en.vcb -t /home/Massi/KIID_ENDE/syntax/corpus/de.vcb died with signal 11, without coredump I tried to look for null byts but there isn't any... Many thanks -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Syntax training crashing
Dear all, I'm trying to train a syntactic model english to german. I did the annotation on the target part using bitpar and ran : nohup /mosesdecoder/scripts/training/train-model.perl -target-syntax --glue-grammar --max-phrase-length 10 --extract-options=--MaxSpan 15 --score-options=--GoodTuring -root-dir /home/Massi/KIID_ENDE/syntax/ -corpus /home/Massi/KIID_DEEN/tagged.kiid -f en -e de -lm 0:5:/home/Massi/KIID_ENDE/LM/atelier/lm.kiid5.blm.de.mm -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out but the training is crashing with the following message : ERROR: Execution of: /root/external-bin-dir/mgiza -CoocurrenceFile /home/Massi/KIID_ENDE/syntax/giza.de-en/de-en.cooc -c /home/Massi/KIID_ENDE/syntax/corpus/de-en-int-train.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -ncpus 30 -nodumps 1 -nsmooth 4 -o /home/Massi/KIID_ENDE/syntax/giza.de-en/de-en -onlyaldumps 1 -p0 0.999 -s /home/Massi/KIID_ENDE/syntax/corpus/en.vcb -t /home/Massi/KIID_ENDE/syntax/corpus/de.vcb died with signal 11, without coredump I tried to look for null byts but there isn't any... Many thanks -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] Running Moses with a compact lexical reordering
Dear all, I'm having troubles running moses with a compact lexical reordering table. After sorting the initial table, I ran the following command : /home/Moses/mosesdecoder/bin/processLexicalTableMin -in reordering-table.sort -out reordering-table -threads 32 the process has been successfully completed so I've updated my moses.ini file : LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-deg aulle-fren/training/model/reordering-table.minlexr and ran moses like this : /home/Moses/mosesdecoder/bin/moses -f moses2.ini -minlexr-memory and I'm getting the following error : line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr FeatureFunction: LexicalReordering0 start: 15 end: 20 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 21 end: 21 line=KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/Corps-lm.blm.en.mm order=5 FeatureFunction: LM0 start: 22 end: 22 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading table into memory...Exception: vector::_M_range_check Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Running Moses with a compact lexical reordering
Hi Marcin, the reordering table is over 12 GB, can I send a sample of the inital file instead? here is my moses.ini : ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 2 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty #PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/Update4/train-pros-fren-update4/binarised-model/phrase-table-pros-fren-update4.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-firsthalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel1 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-secondhalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryMultiModel num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=interpolate lambda=0.5,0.5 components=TranslationModel0,TranslationModel1 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe- allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/Corps-lm.blm.en.mm order=5 # dense weights for feature functions [threads] all [weight] LexicalReordering0= 0.0564145 -0.00809962 0.0939308 0.114042 0.0630804 0.0683044 Distortion0= 0.0452205 LM0= 0.140316 WordPenalty0= 0.0793205 PhrasePenalty0= 0.113696 PhraseDictionaryMultiModel0= 0.0608326 0.102939 0.0533373 0.000466626 TranslationModel0= 0 0 1 0 TranslationModel1= 0 0 1 0 UnknownWordPenalty0= 1 2014-12-01 16:30 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, weird. In that case you should keep the suffix. Can you make the reordering table available for download somehow? Can I see your full INI-file? W dniu 01.12.2014 o 16:27, Massinissa Ahmim pisze: Hi Marcin, I'm getting this now : Can't read /home/train-degaulle-fren/training/model/reordering-table I double checked the paths, they seem to be correct Thanks Massinissa 2014-12-01 16:17 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, try removing the .minlexr suffix from the ini file. Does it work after that? W dniu 01.12.2014 o 16:14, Massinissa Ahmim pisze: Dear all, I'm having troubles running moses with a compact lexical reordering table. After sorting the initial table, I ran the following command : /home/Moses/mosesdecoder/bin/processLexicalTableMin -in reordering-table.sort -out reordering-table -threads 32 the process has been successfully completed so I've updated my moses.ini file : LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-deg aulle-fren/training/model/reordering-table.minlexr and ran moses like this : /home/Moses/mosesdecoder/bin/moses -f moses2.ini -minlexr-memory and I'm getting the following error : line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr FeatureFunction: LexicalReordering0 start: 15 end: 20 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 21 end: 21 line=KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/ Corps-lm.blm.en.mm order=5 FeatureFunction: LM0 start: 22 end: 22 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading table into memory...Exception: vector::_M_range_check Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44
Re: [Moses-support] Running Moses with a compact lexical reordering
I'm getting the same message : Can't read /home/train-degaulle-fren/training/model/reordering-table 2014-12-01 16:49 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Could you check whether this happens if you do not use the phrase table combination? Just one normal phrase table and a standard decoding path 0 T 0 ? W dniu 01.12.2014 o 16:43, Massinissa Ahmim pisze: Hi Marcin, the reordering table is over 12 GB, can I send a sample of the inital file instead? here is my moses.ini : ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 2 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty #PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/Update4/train-pros-fren-update4/binarised-model/phrase-table-pros-fren-update4.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-firsthalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel1 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-secondhalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryMultiModel num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=interpolate lambda=0.5,0.5 components=TranslationModel0,TranslationModel1 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe- allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/Corps-lm.blm.en.mm order=5 # dense weights for feature functions [threads] all [weight] LexicalReordering0= 0.0564145 -0.00809962 0.0939308 0.114042 0.0630804 0.0683044 Distortion0= 0.0452205 LM0= 0.140316 WordPenalty0= 0.0793205 PhrasePenalty0= 0.113696 PhraseDictionaryMultiModel0= 0.0608326 0.102939 0.0533373 0.000466626 TranslationModel0= 0 0 1 0 TranslationModel1= 0 0 1 0 UnknownWordPenalty0= 1 2014-12-01 16:30 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, weird. In that case you should keep the suffix. Can you make the reordering table available for download somehow? Can I see your full INI-file? W dniu 01.12.2014 o 16:27, Massinissa Ahmim pisze: Hi Marcin, I'm getting this now : Can't read /home/train-degaulle-fren/training/model/reordering-table I double checked the paths, they seem to be correct Thanks Massinissa 2014-12-01 16:17 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, try removing the .minlexr suffix from the ini file. Does it work after that? W dniu 01.12.2014 o 16:14, Massinissa Ahmim pisze: Dear all, I'm having troubles running moses with a compact lexical reordering table. After sorting the initial table, I ran the following command : /home/Moses/mosesdecoder/bin/processLexicalTableMin -in reordering-table.sort -out reordering-table -threads 32 the process has been successfully completed so I've updated my moses.ini file : LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-deg aulle-fren/training/model/reordering-table.minlexr and ran moses like this : /home/Moses/mosesdecoder/bin/moses -f moses2.ini -minlexr-memory and I'm getting the following error : line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr FeatureFunction: LexicalReordering0 start: 15 end: 20 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 21 end: 21 line=KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/ Corps-lm.blm.en.mm order=5 FeatureFunction: LM0 start: 22 end: 22 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading table into memory...Exception: vector::_M_range_check Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* A Young Innovative Company recognised by the French Research Ministry and Paris Finance Innovation cluster [image: Logo labellisé FI small] ___ Moses-support
Re: [Moses-support] Running Moses with a compact lexical reordering
Hi, I'm getting the first error message now : Loading table into memory...Exception: vector::_M_range_check 2014-12-01 16:56 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Put the suffix .minlexr back, that was a failed advise. W dniu 01.12.2014 o 16:56, Massinissa Ahmim pisze: I'm getting the same message : Can't read /home/train-degaulle-fren/training/model/reordering-table 2014-12-01 16:49 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Could you check whether this happens if you do not use the phrase table combination? Just one normal phrase table and a standard decoding path 0 T 0 ? W dniu 01.12.2014 o 16:43, Massinissa Ahmim pisze: Hi Marcin, the reordering table is over 12 GB, can I send a sample of the inital file instead? here is my moses.ini : ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 2 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty #PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/Update4/train-pros-fren-update4/binarised-model/phrase-table-pros-fren-update4.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-firsthalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel1 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-secondhalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryMultiModel num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=interpolate lambda=0.5,0.5 components=TranslationModel0,TranslationModel1 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe- allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/Corps-lm.blm.en.mm order=5 # dense weights for feature functions [threads] all [weight] LexicalReordering0= 0.0564145 -0.00809962 0.0939308 0.114042 0.0630804 0.0683044 Distortion0= 0.0452205 LM0= 0.140316 WordPenalty0= 0.0793205 PhrasePenalty0= 0.113696 PhraseDictionaryMultiModel0= 0.0608326 0.102939 0.0533373 0.000466626 TranslationModel0= 0 0 1 0 TranslationModel1= 0 0 1 0 UnknownWordPenalty0= 1 2014-12-01 16:30 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, weird. In that case you should keep the suffix. Can you make the reordering table available for download somehow? Can I see your full INI-file? W dniu 01.12.2014 o 16:27, Massinissa Ahmim pisze: Hi Marcin, I'm getting this now : Can't read /home/train-degaulle-fren/training/model/reordering-table I double checked the paths, they seem to be correct Thanks Massinissa 2014-12-01 16:17 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, try removing the .minlexr suffix from the ini file. Does it work after that? W dniu 01.12.2014 o 16:14, Massinissa Ahmim pisze: Dear all, I'm having troubles running moses with a compact lexical reordering table. After sorting the initial table, I ran the following command : /home/Moses/mosesdecoder/bin/processLexicalTableMin -in reordering-table.sort -out reordering-table -threads 32 the process has been successfully completed so I've updated my moses.ini file : LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-deg aulle-fren/training/model/reordering-table.minlexr and ran moses like this : /home/Moses/mosesdecoder/bin/moses -f moses2.ini -minlexr-memory and I'm getting the following error : line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr FeatureFunction: LexicalReordering0 start: 15 end: 20 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 21 end: 21 line=KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/ Corps-lm.blm.en.mm order=5 FeatureFunction: LM0 start: 22 end: 22 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading table into memory...Exception: vector::_M_range_check Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http
Re: [Moses-support] Running Moses with a compact lexical reordering
Hi, I'm getting the first error message now : Loading table into memory...Exception: vector::_M_range_check 2014-12-01 17:00 GMT+01:00 Massinissa Ahmim massinissa.ah...@linguacustodia.com: Hi, I'm getting the first error message now : Loading table into memory...Exception: vector::_M_range_check 2014-12-01 16:56 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Put the suffix .minlexr back, that was a failed advise. W dniu 01.12.2014 o 16:56, Massinissa Ahmim pisze: I'm getting the same message : Can't read /home/train-degaulle-fren/training/model/reordering-table 2014-12-01 16:49 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Could you check whether this happens if you do not use the phrase table combination? Just one normal phrase table and a standard decoding path 0 T 0 ? W dniu 01.12.2014 o 16:43, Massinissa Ahmim pisze: Hi Marcin, the reordering table is over 12 GB, can I send a sample of the inital file instead? here is my moses.ini : ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 2 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty #PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/Update4/train-pros-fren-update4/binarised-model/phrase-table-pros-fren-update4.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-firsthalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel1 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-secondhalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryMultiModel num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=interpolate lambda=0.5,0.5 components=TranslationModel0,TranslationModel1 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe- allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/Corps-lm.blm.en.mm order=5 # dense weights for feature functions [threads] all [weight] LexicalReordering0= 0.0564145 -0.00809962 0.0939308 0.114042 0.0630804 0.0683044 Distortion0= 0.0452205 LM0= 0.140316 WordPenalty0= 0.0793205 PhrasePenalty0= 0.113696 PhraseDictionaryMultiModel0= 0.0608326 0.102939 0.0533373 0.000466626 TranslationModel0= 0 0 1 0 TranslationModel1= 0 0 1 0 UnknownWordPenalty0= 1 2014-12-01 16:30 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, weird. In that case you should keep the suffix. Can you make the reordering table available for download somehow? Can I see your full INI-file? W dniu 01.12.2014 o 16:27, Massinissa Ahmim pisze: Hi Marcin, I'm getting this now : Can't read /home/train-degaulle-fren/training/model/reordering-table I double checked the paths, they seem to be correct Thanks Massinissa 2014-12-01 16:17 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hi, try removing the .minlexr suffix from the ini file. Does it work after that? W dniu 01.12.2014 o 16:14, Massinissa Ahmim pisze: Dear all, I'm having troubles running moses with a compact lexical reordering table. After sorting the initial table, I ran the following command : /home/Moses/mosesdecoder/bin/processLexicalTableMin -in reordering-table.sort -out reordering-table -threads 32 the process has been successfully completed so I've updated my moses.ini file : LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-deg aulle-fren/training/model/reordering-table.minlexr and ran moses like this : /home/Moses/mosesdecoder/bin/moses -f moses2.ini -minlexr-memory and I'm getting the following error : line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr FeatureFunction: LexicalReordering0 start: 15 end: 20 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 21 end: 21 line=KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/ Corps-lm.blm.en.mm order=5 FeatureFunction: LM0 start: 22 end: 22 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading table into memory...Exception: vector::_M_range_check Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux
Re: [Moses-support] Running Moses with a compact lexical reordering
Hi, Yes, I just finished processing another one and still I'm getting the same message : Loading table into memory...Exception: vector::_M_range_check 2014-12-01 17:13 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, OK, paths look good. Do you have another compact reordering table somewhere that you can substitute there and try? I am starting to suspect that the current one is corrupted for some reason. If it works with another table, then we would have proof. W dniu 01.12.2014 o 17:08, Massinissa Ahmim pisze: Hi Marcin, Please find here attached the file Many thanks Massinissa 2014-12-01 17:02 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, slighlty clueless by now. Can you execute find /home/train-degaulle-fren paths.txt and send me the paths.txt file to my e-mails address? Do not put that on the support-list. W dniu 01.12.2014 o 17:00, Massinissa Ahmim pisze: Hi, I'm getting the first error message now : Loading table into memory...Exception: vector::_M_range_check 2014-12-01 16:56 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Put the suffix .minlexr back, that was a failed advise. W dniu 01.12.2014 o 16:56, Massinissa Ahmim pisze: I'm getting the same message : Can't read /home/train-degaulle-fren/training/model/reordering-table 2014-12-01 16:49 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Could you check whether this happens if you do not use the phrase table combination? Just one normal phrase table and a standard decoding path 0 T 0 ? W dniu 01.12.2014 o 16:43, Massinissa Ahmim pisze: Hi Marcin, the reordering table is over 12 GB, can I send a sample of the inital file instead? here is my moses.ini : ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 2 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty #PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/Update4/train-pros-fren-update4/binarised-model/phrase-table-pros-fren-update4.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel0 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-firsthalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryCompact tuneable=false name=TranslationModel1 num-features=4 path=/home/train-degaulle-fren/binarised-model/phrase-table-degaulle-secondhalf.minphr input-factor=0 output-factor=0 table-limit=20 PhraseDictionaryMultiModel num-features=4 input-factor=0 output-factor=0 table-limit=20 mode=interpolate lambda=0.5,0.5 components=TranslationModel0,TranslationModel1 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe- allff input-factor=0 output-factor=0 path=/home/train-degaulle-fren/training/model/reordering-table.minlexr Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/DATA/LM/Corps-lm.blm.en.mm order=5 # dense weights for feature functions [threads] all [weight] LexicalReordering0= 0.0564145 -0.00809962 0.0939308 0.114042 0.0630804 0.0683044 Distortion0= 0.0452205 LM0= 0.140316 WordPenalty0= 0.0793205 PhrasePenalty0= 0.113696 PhraseDictionaryMultiModel0= 0.0608326 0.102939 0.0533373 0.000466626 TranslationModel0= 0 0 1 0 TranslationModel1= 0 0 1 0 UnknownWordPenalty0= 1 2014-12-01 16:30 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl: Hm, weird. In that case you should keep the suffix. Can you make the reordering table available for download somehow? Can I see your full INI-file? W dniu 01.12.2014 o 16:27, Massinissa Ahmim pisze: Hi Marcin, I'm getting this now : Can't read /home/train-degaulle-fren/training/model/reordering-table I double checked the paths, they seem to be correct Thanks Massinissa 2014-12-01 16:17 GMT+01:00 Marcin Junczys-Dowmunt junc...@amu.edu.pl : Hi, try removing the .minlexr suffix from the ini file. Does it work after that? W dniu 01.12.2014 o 16:14, Massinissa Ahmim pisze: Dear all, I'm having troubles running moses with a compact lexical reordering table. After sorting the initial table, I ran the following command : /home/Moses/mosesdecoder/bin/processLexicalTableMin -in reordering-table.sort -out reordering-table -threads 32 the process has been successfully completed so I've updated my moses.ini file : LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/train-deg aulle-fren/training/model/reordering-table.minlexr and ran moses like this : /home/Moses/mosesdecoder/bin/moses -f moses2.ini -minlexr-memory and I'm getting the following error : line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input
Re: [Moses-support] Loading binary Phrasetable
Hi Barry, Thanks a lot ! I switched both to 0 but I'm still getting this message : FeatureFunction: TranslationModel0 start: 3 end: 3 Exception: moses/ScoreComponentCollection.cpp:242 in void Moses::ScoreComponentCollection::Assign(const Moses::FeatureFunction*, const std::vectorfloat) threw util::Exception'. Feature function TranslationModel0 specified 1 dense scores or weights. Actually has 4 2014-06-26 22:29 GMT+02:00 Barry Haddow bhad...@staffmail.ed.ac.uk: Hi Massinissa Are you loading the right ini file? The one in the debug message is different from the one you show at the bottom of your mail. In particular, the debug message shows that you are mixing TranslationModel0 and TranslationModel1 which is why Moses complains about missing weights, cheers - Barry On 26/06/14 20:48, Massinissa Ahmim wrote: Dear all, I'm having some troubles loading the binary phrasetable using the latest version of moses. When querying my moses.ini using : echo 'le monde' |/home/mosesdecoder/bin/moses -f moses.ini I'm getting this error : Defined parameters (per moses.ini or switch): config: moses.ini distortion-limit: 6 feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 path=/home/training/binarised-model/phrase-table input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/training/binarised-model/reordering-table Distortion KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en order=5 input-factors: 0 mapping: 0 T 0 weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel1= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 line=UnknownWordPenalty FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 line=WordPenalty FeatureFunction: WordPenalty0 start: 1 end: 1 line=PhrasePenalty FeatureFunction: PhrasePenalty0 start: 2 end: 2 line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 path=/home/training/binarised-model/phrase-table input-factor=0 output-factor=0 FeatureFunction: TranslationModel0 start: 3 end: 3 Exception: moses/ScoreComponentCollection.cpp:242 in void Moses:: ScoreComponentCollection::Assign(const Moses::FeatureFunction*, const std::vectorfloat) threw util::Exception'. Feature function TranslationModel0 specified 1 dense scores or weights. Actually has 0 Here is my moses.ini # ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 0 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/home/training/binarised-model/phrase-table input-factor=0 outp ut-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/training/ binarised-model/reordering-table Distortion KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en order=5 # dense weights for feature functions [weight] UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 Any suggestion ? Many thanks Massinissa -- Description : Description : lingua_custodia_final full logo */The Translation Trustee/* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :massinissa.ah...@linguacustodia.com mailto:massinissa.ahmim@ linguacustodia.com*** *Website :www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* üPensez à l’environnement, n’imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l’environnement, n’imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely
[Moses-support] Segmentation fault
Dear All, I'm having some troubles to run the lastest version of Moses on a Debian server, I've tried to train models with different corporas, but I'm getting the same error. When I try to query Moses with an unknown word, the process goes normally to completion, but when the word is contained in the phrasetable I'm getting the error below : Defined parameters (per moses.ini or switch): config: moses.ini distortion-limit: 6 feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/home/training/model/phrase-table.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/training/model/reordering-table.wbe-msd-bidirectional-fe.gz Distortion KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en order=5 input-factors: 0 mapping: 0 T 0 weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 line=UnknownWordPenalty FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 line=WordPenalty FeatureFunction: WordPenalty0 start: 1 end: 1 line=PhrasePenalty FeatureFunction: PhrasePenalty0 start: 2 end: 2 line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/home/training/model/phrase-table.gz input-factor=0 output-factor=0 FeatureFunction: TranslationModel0 start: 3 end: 6 line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/training/model/reordering-table.wbe-msd-bidirectional-fe.gz FeatureFunction: LexicalReordering0 start: 7 end: 12 Initializing LexicalReordering.. line=Distortion FeatureFunction: Distortion0 start: 13 end: 13 line=KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en order=5 FeatureFunction: LM0 start: 14 end: 14 Loading UnknownWordPenalty0 Loading WordPenalty0 Loading PhrasePenalty0 Loading LexicalReordering0 Loading table into memory...done. Loading Distortion0 Loading LM0 Loading TranslationModel0 Start loading text phrase table. Moses format : [0.031] seconds Reading /home/training/model/phrase-table.gz 5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 IO from STDOUT/STDIN Created input-output object : [0.031] seconds Translating line 0 in thread id 140175290300160 Translating: mmm Line 0: Initialize search took 0.000 seconds total Line 0: Collecting options took 0.000 seconds *** Segmentation fault Register dump: RAX: RBX: RCX: RDX: 01dfe000 RSI: RDI: 01b2e340 RBP: 01b24500 R8 : 01b480c0 R9 : 0003 R10: 01b0f9a0 R11: R12: 01b48a40 R13: 01b34100 R14: 01b24500 R15: 01b2e340 RSP: 7f7d1a625c50 RIP: 00500a61 EFLAGS: 00010246 CS: 0033 FS: GS: Trap: 000e Error: 0004 OldMask: CR2: 0010 FPUCW: 037f FPUSW: TAG: RIP: RDP: ST(0) ST(1) ST(2) ST(3) ST(4) ST(5) ST(6) ST(7) mxcsr: 1fa4 XMM0: XMM1: XMM2: XMM3: XMM4: XMM5: XMM6: XMM7: XMM8: XMM9: XMM10: XMM11: XMM12: XMM13: XMM14: XMM15: Backtrace: /home/Moses/mosesdecoder/bin/moses[0x500a61] /home/Moses/mosesdecoder/bin/moses[0x48fa19] /home/Moses/mosesdecoder/bin/moses[0x529525] /home/Moses/mosesdecoder/bin/moses[0x52975a] /home/Moses/mosesdecoder/bin/moses[0x529b13] /home/Moses/mosesdecoder/bin/moses[0x529e8c] /home/Moses/mosesdecoder/bin/moses[0x4798a0] /home/Moses/mosesdecoder/bin/moses[0x41c895] /home/Moses/mosesdecoder/bin/moses[0x5126a5] /home/Moses/mosesdecoder/bin/moses[0x685384] /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7f7d1c0d0b50] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f7d1be1b0ed] Memory map: 0040-0075e000 r-xp 08:02 10619160 /home/Moses/mosesdecoder/bin/moses
[Moses-support] Loading binary Phrasetable
Dear all, I'm having some troubles loading the binary phrasetable using the latest version of moses. When querying my moses.ini using : echo 'le monde' |/home/mosesdecoder/bin/moses -f moses.ini I'm getting this error : Defined parameters (per moses.ini or switch): config: moses.ini distortion-limit: 6 feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 path=/home/training/binarised-model/phrase-table input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/training/binarised-model/reordering-table Distortion KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en order=5 input-factors: 0 mapping: 0 T 0 weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel1= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 line=UnknownWordPenalty FeatureFunction: UnknownWordPenalty0 start: 0 end: 0 line=WordPenalty FeatureFunction: WordPenalty0 start: 1 end: 1 line=PhrasePenalty FeatureFunction: PhrasePenalty0 start: 2 end: 2 line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 path=/home/training/binarised-model/phrase-table input-factor=0 output-factor=0 FeatureFunction: TranslationModel0 start: 3 end: 3 Exception: moses/ScoreComponentCollection.cpp:242 in void Moses::ScoreComponentCollection::Assign(const Moses::FeatureFunction*, const std::vectorfloat) threw util::Exception'. Feature function TranslationModel0 specified 1 dense scores or weights. Actually has 0 Here is my moses.ini # ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 0 [distortion-limit] 6 # feature functions [feature] UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/home/training/binarised-model/phrase-table input-factor=0 outp ut-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/training/ binarised-model/reordering-table Distortion KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en order=5 # dense weights for feature functions [weight] UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 Any suggestion ? Many thanks Massinissa -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l’environnement, n’imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Loading a binary rule table
Hi Hieu, I used the same binarisation command as the one stated here-above I get this error when I try to translate something like : echo 'ceci est un test' | /mosesdecoder/bin/moses_chart -f moses.ini Here is my moses.ini : # ### MOSES CONFIG FILE ### # # input factors [input-factors] 0 # mapping steps [mapping] 0 T 0 1 T 1 # translation tables: table type (hierarchical(0), textual (0), binary (1)), source-factors, target-factors, number of scores, file # OLD FORMAT is still handled for back-compatibility # OLD FORMAT translation tables: source-factors, target-factors, number of scores, file # OLD FORMAT a binary table type (1) is assumed [ttable-file] 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar # no generation models, no generation-file section # language models: type(srilm/irstlm), factors, order, file [lmodel-file] 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm # limit on how many phrase translations e for each phrase f are loaded # 0 = all elements loaded [ttable-limit] 20 # language model weights [weight-l] 0.5000 # translation model weights [weight-t] 0.20 0.20 0.20 0.20 0.20 1.0 # no generation models, no weight-generation section # word penalty [weight-w] -1 [cube-pruning-pop-limit] 1000 [non-terminals] X [search-algorithm] 3 [inputtype] 3 [max-chart-span] 20 1000 2014-03-21 20:49 GMT+01:00 Hieu Hoang hieu.ho...@ed.ac.uk: i've never seen that error before. Did you change anything else, other than rebinarizing the phrase-table? What is the exact command you ran when you got this error? Can I have a look at your moses.ini file On 21 March 2014 18:32, Massinissa Ahmim massinissa.ah...@linguacustodia.com wrote: Thank you Hieu, I've binarised my rule-table again and it seems to be successfully loaded but I'm getting this message now : Reading /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table 5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 Exception: fd:3: Is a directory from zlib What can I do? Thanks ! Regards Massinissa 2014-03-21 11:18 GMT+01:00 Hieu Hoang hieu.ho...@ed.ac.uk: everything seems to be ok. Are you sure the server you are running on has access to the directory /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table Also, you should consider updating your version of Moses to the current github version, or version 2.1.1 On 20 March 2014 11:20, Massinissa Ahmim massinissa.ah...@linguacustodia.com wrote: Dear all, I've managed to train a hierarchical model using the following command : nohup /mosesdecoder/scripts/training/train-model.perl --hierarchical --glue-grammar --score-options=--GoodTuring -root-dir /disque2/Preparation/syntactic/hierarchical_PSCT -corpus /disque2/Preparation/backoff/PSCT.tok.uni.low -f fr -e en -lm 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out The resulting model works great but it is very slow. So I used CreatOnDiskPt to binarise the rule table as follows : /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 5 20 2 model/rule-table.gz rules-table This outputs the following files -rw-r--r-- 1 root root 85 20 mars 11:35 Misc.dat -rw-r--r-- 1 root root 781509322 20 mars 11:35 Source.dat -rw-r--r-- 1 root root 1758588137 20 mars 11:35 TargetColl.dat -rw-r--r-- 1 root root 1555353459 20 mars 11:35 TargetInd.dat -rw-r--r-- 1 root root 400714 20 mars 11:35 Vocab.dat and I've updated my moses.ini as follows : 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar But as I try to use it, I'm getting this : Defined parameters (per moses.ini or switch): config: moses.ini cube-pruning-pop-limit: 1000 input-factors: 0 inputtype: 3 lmodel-file: 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm mapping: 0 T 0 1 T 1 max-chart-span: 20 1000 non-terminals: X search-algorithm: 3 ttable-file: 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar ttable-limit: 20 weight-l: 0.5000 weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 weight-w: -1 /mosesdecoder/bin ScoreProducer: WordPenalty start: 0 end: 1 ScoreProducer: !UnknownWordPenalty start: 1 end: 2 Loading lexical distortion models...have 0 models Start loading LanguageModel /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm : [0.000] seconds /disque2/Preparation
Re: [Moses-support] Loading a binary rule table
Thank you Hieu, I've binarised my rule-table again and it seems to be successfully loaded but I'm getting this message now : Reading /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table 5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100 Exception: fd:3: Is a directory from zlib What can I do? Thanks ! Regards Massinissa 2014-03-21 11:18 GMT+01:00 Hieu Hoang hieu.ho...@ed.ac.uk: everything seems to be ok. Are you sure the server you are running on has access to the directory /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table Also, you should consider updating your version of Moses to the current github version, or version 2.1.1 On 20 March 2014 11:20, Massinissa Ahmim massinissa.ah...@linguacustodia.com wrote: Dear all, I've managed to train a hierarchical model using the following command : nohup /mosesdecoder/scripts/training/train-model.perl --hierarchical --glue-grammar --score-options=--GoodTuring -root-dir /disque2/Preparation/syntactic/hierarchical_PSCT -corpus /disque2/Preparation/backoff/PSCT.tok.uni.low -f fr -e en -lm 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out The resulting model works great but it is very slow. So I used CreatOnDiskPt to binarise the rule table as follows : /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 5 20 2 model/rule-table.gz rules-table This outputs the following files -rw-r--r-- 1 root root 85 20 mars 11:35 Misc.dat -rw-r--r-- 1 root root 781509322 20 mars 11:35 Source.dat -rw-r--r-- 1 root root 1758588137 20 mars 11:35 TargetColl.dat -rw-r--r-- 1 root root 1555353459 20 mars 11:35 TargetInd.dat -rw-r--r-- 1 root root 400714 20 mars 11:35 Vocab.dat and I've updated my moses.ini as follows : 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar But as I try to use it, I'm getting this : Defined parameters (per moses.ini or switch): config: moses.ini cube-pruning-pop-limit: 1000 input-factors: 0 inputtype: 3 lmodel-file: 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm mapping: 0 T 0 1 T 1 max-chart-span: 20 1000 non-terminals: X search-algorithm: 3 ttable-file: 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar ttable-limit: 20 weight-l: 0.5000 weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 weight-w: -1 /mosesdecoder/bin ScoreProducer: WordPenalty start: 0 end: 1 ScoreProducer: !UnknownWordPenalty start: 1 end: 2 Loading lexical distortion models...have 0 models Start loading LanguageModel /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm : [0.000] seconds /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm: line 80492: reached EOF before \end\ ScoreProducer: LM start: 2 end: 3 Finished loading LanguageModels : [0.056] seconds Using uniform ttable-limit of 20 for all translation tables. Start loading PhraseTable /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056] seconds filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table ScoreProducer: PhraseModel start: 3 end: 4 Start loading PhraseTable /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar : [0.056] seconds filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar ScoreProducer: PhraseModel:2 start: 4 end: 5 Finished loading phrase tables : [0.056] seconds max-chart-span: 20 max-chart-span: 1000 Start loading phrase table from /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056] seconds Can't read /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table So I wondering if I did something wrong with my training command/binarisation or with the parameters in the moses.ini Many thanks Regads MA -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. ___ Moses-support mailing list Moses-support@mit.edu http
[Moses-support] Loading a binary rule table
Dear all, I've managed to train a hierarchical model using the following command : nohup /mosesdecoder/scripts/training/train-model.perl --hierarchical --glue-grammar --score-options=--GoodTuring -root-dir /disque2/Preparation/syntactic/hierarchical_PSCT -corpus /disque2/Preparation/backoff/PSCT.tok.uni.low -f fr -e en -lm 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 training.out The resulting model works great but it is very slow. So I used CreatOnDiskPt to binarise the rule table as follows : /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 5 20 2 model/rule-table.gz rules-table This outputs the following files -rw-r--r-- 1 root root 85 20 mars 11:35 Misc.dat -rw-r--r-- 1 root root 781509322 20 mars 11:35 Source.dat -rw-r--r-- 1 root root 1758588137 20 mars 11:35 TargetColl.dat -rw-r--r-- 1 root root 1555353459 20 mars 11:35 TargetInd.dat -rw-r--r-- 1 root root 400714 20 mars 11:35 Vocab.dat and I've updated my moses.ini as follows : 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar But as I try to use it, I'm getting this : Defined parameters (per moses.ini or switch): config: moses.ini cube-pruning-pop-limit: 1000 input-factors: 0 inputtype: 3 lmodel-file: 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm mapping: 0 T 0 1 T 1 max-chart-span: 20 1000 non-terminals: X search-algorithm: 3 ttable-file: 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar ttable-limit: 20 weight-l: 0.5000 weight-t: 0.20 0.20 0.20 0.20 0.20 1.0 weight-w: -1 /mosesdecoder/bin ScoreProducer: WordPenalty start: 0 end: 1 ScoreProducer: !UnknownWordPenalty start: 1 end: 2 Loading lexical distortion models...have 0 models Start loading LanguageModel /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm : [0.000] seconds /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm: line 80492: reached EOF before \end\ ScoreProducer: LM start: 2 end: 3 Finished loading LanguageModels : [0.056] seconds Using uniform ttable-limit of 20 for all translation tables. Start loading PhraseTable /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056] seconds filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table ScoreProducer: PhraseModel start: 3 end: 4 Start loading PhraseTable /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar : [0.056] seconds filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar ScoreProducer: PhraseModel:2 start: 4 end: 5 Finished loading phrase tables : [0.056] seconds max-chart-span: 20 max-chart-span: 1000 Start loading phrase table from /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056] seconds Can't read /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table So I wondering if I did something wrong with my training command/binarisation or with the parameters in the moses.ini Many thanks Regads MA -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] tools for factored corpora
Hi Viktor, As far I know, you can use the wrapper scripts (treetagger or mxpost) located at /mosesdecoder/scripts/training/wrappers for this purpose. Regards Massinissa 2014-03-20 17:37 GMT+01:00 Viktor Pless viktor.pl...@gmail.com: Hi, what tools can be used to lemmatize/POS-tag/etc. a corpus in moses format (with the pipes)? I need them regarding Spanish, English, Hungarian. Thanks in advance. Viktor ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] manual: 3.2.1 Train an unfactored model
Hi Peter, There is obviously an issue with your training data, have you used the proj-syndicate corpora? if so please paste here your training command I'd be interested to see that. Regards M 2014-03-11 19:25 GMT+01:00 Peter Kleiweg p.c.j.klei...@rug.nl: Hi, From the manual: http://www.statmt.org/moses/manual/manual.pdf I have run the example from part 3.2.1, page 64, to create unfactored/model/moses.ini At the bottom of page 67 is this example: moses -f unfactored/model/moses.ini in -d 0.2 The file 'in' contains: menschen beschreibt putin . (The -d 0.2 causes an error in all examples, so I leave that out) I get an error: Exception: moses/Word.cpp:112 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vectorlong unsigned int, const StringPiece, bool) threw StrayFactorException because `fit'. You have configured 1 factors but the word !|!|. contains factor delimiter | too many times. What is wrong, and how do I fix this? All other examples in section 3.2 work OK. I have moses built with both IRSTLM and SRILM. -- Peter Kleiweg http://pkleiweg.home.xs4all.nl/ ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] train-model log file?
Hi Victor Yes, all you have to do is to add training.out after to your training command Regards Massinissa 2014-03-11 19:39 GMT+01:00 Viktor Pless viktor.pl...@gmail.com: Hi, does the train-model.perl script (or more specifically, mgiza) create a log file? thanks, ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Binarising the phrase table
Hi Per, Could you please past here the command your ran to binarise the Phrase-table? The current paths in your moses.ini should be to the out-put of processPhraseTable, for instance, if the output was /home/per/working/train/model/whatever/phrase-table/ , then you have to change your current path by this one making sure to use the stemming (taking the .gz off as I did above) and switching the first moses parameter (from 0 to 1) as follows : 1 0 0 5 /home/per/working/train/model/whatever/phrase-table/ Same thing with the reordering table Regards Massinissa 2014-02-28 7:47 GMT+01:00 Per Tunedal per.tune...@operamail.com: Hi, tried to binarise the phrase table and got in to trouble. 1. Error messages, as below. What's that? distinct source phrases: 439511 distinct first words of source phrases: 67600 number of phrase pairs (line count): 940438 Count of lines with missing alignments: 0/940438 WARNING: there are src voc entries with no phrase translation: count 2168 There exists phrase translations for 65432 entries 2. Modify the moses.ini file. I've found this on the Moses/Baseline page: 1. Change PhraseDictionaryMemory to PhraseDictionaryBinary 2. Set the path of the PhraseDictionary feature to point to $HOME/working/train/binarised-model/Kryptering1.sv-fr.phrase-table 3. Set the path of the LexicalReordering feature to point to $HOME/working/train/binarised-model/Kryptering1.sv-fr.reordering-table But I cannot find any such entries in my moses.ini - maybe because I'm running a somewhat older version of Moses. I've found e.g. the following lines in my ini-file: [ttable-file] 0 0 0 5 /home/per/working/train/model/phrase-table.gz # distortion (reordering) files [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz How should I change the entries to use my binarised model? Yours, Per Tunedal ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] Binarising the phrase table
Hi Per, Yes, you have change the 0 into 1 to use binary files. Actually you should do that only for the phrase-table, meaning : 1 0 0 5 /home/per/working/train/binarised-model/Kryptering1. sv-fr.phrase-table 0-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/ binarised-model/Kryptering1.sv-fr.reordering-table My last comment could be kinda misleading but by the 'same thing' I meant changing the paths. It should work now assuming the paths are accurate Regards Massinissa 2014-02-28 10:42 GMT+01:00 Per Tunedal per.tune...@operamail.com: Hi Massinissa, my commands: ~/mosesdecoder/bin/processPhraseTable \ -ttable 0 0 train/model/Kryptering1.sv-fr.phrase-table.pruned \ -nscores 5 -out train/binarised-model/Kryptering1.sv-fr.phrase-table ~/mosesdecoder/bin/processLexicalTable \ -in train/model/reordering-table.wbe-msd-bidirectional-fe.gz \ -out train/binarised-model/Kryptering1.sv-fr.reordering-table I tried the following changes in moses.ini: # OLD FORMAT a binary table type (1) is assumed [ttable-file] #0 0 0 5 /home/per/working/train/model/phrase-table.gz 0 0 0 5 /home/per/working/train/binarised-model/Kryptering1.sv-fr.phrase-table # distortion (reordering) files [distortion-file] #0-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz 0-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/binarised-model/Kryptering1.sv-fr.reordering-table But it doesn't work. Maybe changing the zeros to ones would do the trick? 1 0 0 5 /home/per/working/train/binarised-model/Kryptering1.sv-fr.phrase-table 1-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/binarised-model/Kryptering1.sv-fr.reordering-table Yours, Per Tunedal On Fri, Feb 28, 2014, at 9:55, Massinissa Ahmim wrote: Hi Per, Could you please past here the command your ran to binarise the Phrase-table? The current paths in your moses.ini should be to the out-put of processPhraseTable, for instance, if the output was /home/per/working/train/model/whatever/phrase-table/ , then you have to change your current path by this one making sure to use the stemming (taking the .gz off as I did above) and switching the first moses parameter (from 0 to 1) as follows : 1 0 0 5 /home/per/working/train/model/whatever/phrase-table/ Same thing with the reordering table Regards Massinissa 2014-02-28 7:47 GMT+01:00 Per Tunedal per.tune...@operamail.com: Hi, tried to binarise the phrase table and got in to trouble. 1. Error messages, as below. What's that? distinct source phrases: 439511 distinct first words of source phrases: 67600 number of phrase pairs (line count): 940438 Count of lines with missing alignments: 0/940438 WARNING: there are src voc entries with no phrase translation: count 2168 There exists phrase translations for 65432 entries 2. Modify the moses.ini file. I've found this on the Moses/Baseline page: 1. Change PhraseDictionaryMemory to PhraseDictionaryBinary 2. Set the path of the PhraseDictionary feature to point to $HOME/working/train/binarised-model/Kryptering1.sv-fr.phrase-table 3. Set the path of the LexicalReordering feature to point to $HOME/working/train/binarised-model/Kryptering1.sv-fr.reordering-table But I cannot find any such entries in my moses.ini - maybe because I'm running a somewhat older version of Moses. I've found e.g. the following lines in my ini-file: [ttable-file] 0 0 0 5 /home/per/working/train/model/phrase-table.gz # distortion (reordering) files [distortion-file] 0-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz How should I change the entries to use my binarised model? Yours, Per Tunedal ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. Email had 1 attachment: + image001.jpg 6k (image/jpeg) -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33
Re: [Moses-support] Very slow tuning with binarised kenlm language model
Hi Felipe, Tuning is usually very slow, it is recommended to use a muti-threaded mert if you have several CPUs at your disposal You can run a multi-threaded tuning by adding the option --decoder-flags -threads all to your command Regards Massinissa 2014-02-28 12:56 GMT+01:00 Felipe Sánchez Martínez fsanc...@dlsi.ua.es: Hello all, I am tuning a system that uses a binarised kenlm language model. This model was binarised with default parameters and is 22 GB in size after binarisation. The thing is that the language model and the (filtered) phrase table fit into memory (32 GB ; so, no swaping) but moses is translating very very slowly, it is only using around 15% of CPU. Is there anything I can do to make it faster? Thank you very much for your help Regards -- Felipe ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] ERROR: Can't generate symmetrized alignment file
Hi Cyrine, Try to run this command instead : ...train-model.perl -corpus /home/corpus/Apprentissage/ train..tok.true.clean.low1 -f en -e fr -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:/home/corpus/Apprentissage/FR3GR.lm:8 -external-bin-dir /home/moses/bin You don't have to call the files twice just the extensions, for instance if your parallel files are named corpus.en and corpus.fr use the path ~/whatever/corpus -f en -e fr and make sure both are located in the same directory Regards Massinissa 2014-02-26 16:47 GMT+01:00 cyrine.na...@univ-lorraine.fr cyrine.na...@gmail.com: Hello, I try to train phrase model with this command : train-model.perl -corpus /home/corpus/Apprentissage/train..tok.true.clean.low1 /home/corpus/Apprentissage/train..tok.true.clean.low1 -f en -e fr -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:/home/corpus/Apprentissage/FR3GR.lm:8 -external-bin-dir /home/moses/bin But it indicates this message Executing: mkdir -p ./model Executing: /home/moses/mosesdecoder/scripts/training/giza2bal.pl -d gzip -cd ./giza.fr-en/fr-en.A3.final.gz -i gzip -cd ./giza.en-fr/en-fr.A3.final.gz |/home/moses/mosesdecoder/scripts/../bin/symal -alignment=grow -diagonal=yes -final=yes -both=yes ./model/aligned.grow-diag-final-and sh: 1: /home/moses/mosesdecoder/scripts/../bin/symal: not found Exit code: 127 ERROR: Can't generate symmetrized alignment file Any idea to solve this problem? Thank you in advance -- *Cyrine Ph.D. Student in Computer Science* ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] ERROR: Can't generate symmetrized alignment file
Use absolute paths to all the scripts, and make sure your parallel files have the same names but the extension 2014-02-26 18:25 GMT+01:00 cyrine.na...@univ-lorraine.fr cyrine.na...@gmail.com: Thank you Massinissahttps://plus.google.com/u/0/106330732665975122814?prsrc=4 , I try to do the command that you propose, but it stills the same problem The symal script is in anothe directory.. 2014-02-26 17:46 GMT+01:00 Massinissa Ahmim massinissa.ah...@linguacustodia.com: Hi Cyrine, Try to run this command instead : ...train-model.perl -corpus /home/corpus/Apprentissage/ train..tok.true.clean.low1 -f en -e fr -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:/home/corpus/Apprentissage/FR3GR.lm:8 -external-bin-dir /home/moses/bin You don't have to call the files twice just the extensions, for instance if your parallel files are named corpus.en and corpus.fr use the path ~/whatever/corpus -f en -e fr and make sure both are located in the same directory Regards Massinissa 2014-02-26 16:47 GMT+01:00 cyrine.na...@univ-lorraine.fr cyrine.na...@gmail.com: Hello, I try to train phrase model with this command : train-model.perl -corpus /home/corpus/Apprentissage/train..tok.true.clean.low1 /home/corpus/Apprentissage/train..tok.true.clean.low1 -f en -e fr -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:3:/home/corpus/Apprentissage/FR3GR.lm:8 -external-bin-dir /home/moses/bin But it indicates this message Executing: mkdir -p ./model Executing: /home/moses/mosesdecoder/scripts/training/giza2bal.pl -d gzip -cd ./giza.fr-en/fr-en.A3.final.gz -i gzip -cd ./giza.en-fr/en-fr.A3.final.gz |/home/moses/mosesdecoder/scripts/../bin/symal -alignment=grow -diagonal=yes -final=yes -both=yes ./model/aligned.grow-diag-final-and sh: 1: /home/moses/mosesdecoder/scripts/../bin/symal: not found Exit code: 127 ERROR: Can't generate symmetrized alignment file Any idea to solve this problem? Thank you in advance -- *Cyrine Ph.D. Student in Computer Science* ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. -- *Cyrine NASRIPh.D. Student in Computer Science* -- [image: Description : Description : lingua_custodia_final full logo] *The Translation Trustee* *1, Place Charles de Gaulle* *78180 Montigny-le-Bretonneux* *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84* *Email :* *massinissa.ah...@linguacustodia.com massinissa.ah...@linguacustodia.com* *Website :* *www.linguacustodia.com http://www.linguacustodia.com/ - www.thetranslationtrustee.com http://www.thetranslationtrustee.com* ü Pensez à l'environnement, n'imprimez ce courriel que si nécessaire. Please do not print this email unless it is absolutely necessary. Spread environmental awareness. inline: image001.jpg___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
