Re: [Moses-support] Issue with tuning - decoder died

Fri, 23 Oct 2015 05:28:20 -0700 

doubt it. Look at the script. The problem chars are
   []|<>&'"

On 23/10/2015 13:01, Ya-chi Lin wrote:


Hi Hieu,
Thank for pointing this out. Could you please let me know if Chinese square brackets 「」are also a problem? 


Best,

Ya-Chi


------------------------------------------------------------------------
*寄件者:* Hieu Hoang <hieuho...@gmail.com>
*寄件日期:* 2015年10月23日 下午 02:32
*收件者:* Ya-chi Lin; moses-support@mit.edu
*主旨:* Re: [Moses-support] Issue with tuning - decoder died
you must escape the [ and ] characters. If you had used the moses tokenizer, it will do this for you. If you use a different tokenizer, then run your tokenized output through 
   scripts/tokenizer/escape-special-chars.perl

On 23/10/2015 01:22, Ya-chi Lin wrote:


Hi ALL,
I'm building an English to Traditional Chinese system but have encountered a problem while performing tuning. Below is the content of mert.out file. Can anyone please tell me what went wrong here? Any help will be greatly appreciated. 


Using SCRIPTS_ROOTDIR: /opt/mosesdecoder/scripts
Assuming the tables are already filtered, reusing filtered/moses.ini
Using cached features list: ./features.list
MERT starting values and ranges for random generation:
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  Distortion0 =   0.300 ( 0.00 ..  1.00)
    LM0 =   0.500 ( 0.00 ..  1.00)
  WordPenalty0 =  -1.000 ( 0.00 ..  1.00)
  PhrasePenalty0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: Distortion0=0.300000
featlist: LM0=0.500000
featlist: WordPenalty0=-1.000000
featlist: PhrasePenalty0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
run 1 start at Fri Oct 23 03:07:31 EEST 2015
Parsing --decoder-flags: ||
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
Normalizing lambdas: 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 0.500000 -1.000000 0.200000 0.200000 0.200000 0.200000 0.200000 DECODER_CFG = -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391' Executing: /opt/mosesdecoder/bin/moses -config filtered/moses.ini -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391' -n-best-list run1.best100.out 100 distinct -input-file /home/yachilin/working/tune.true.en > run1.out Executing: /opt/mosesdecoder/bin/moses -config filtered/moses.ini -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391' -n-best-list run1.best100.out 100 distinct -input-file /home/yachilin/working/tune.true.en > run1.out 1-10.20.2 0.2 0.2 0.20.3 0.3 0.3 0.3 0.3 0.30.30.5(1) run decoder to produce n-best lists 
params =
decoder_config = -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391' 
Defined parameters (per moses.ini or switch):
        config: filtered/moses.ini
        distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe Distortion KENLM lazyken=0 name=LM0 factor=0 path=/home/yachilin/lm/train.blm.tch order=3 
        input-factors: 0
        input-file: /home/yachilin/working/tune.true.en
        mapping: 0 T 0
        n-best-list: run1.best100.out 100 distinct
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 weight-overwrite: LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391 
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0 
FeatureFunction: TranslationModel0 start: 3 end: 6
line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe 
Initializing Lexical Reordering Feature..
FeatureFunction: LexicalReordering0 start: 7 end: 12
line=Distortion
FeatureFunction: Distortion0 start: 13 end: 13
line=KENLM lazyken=0 name=LM0 factor=0 path=/home/yachilin/lm/train.blm.tch order=3 
FeatureFunction: LM0 start: 14 end: 14
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
Loading table into memory...done.
Loading Distortion0
Loading LM0
Loading TranslationModel0
Start loading text phrase table. Moses format : [0.719] seconds
Reading /home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Exception: moses/Phrase.cpp:217 in void Moses::Phrase::CreateFromString(Moses::FactorDirection, const std::vector<long unsigned int>&, const StringPiece&, Moses::Word**) threw util::Exception because `nextPos == string::npos'. Incorrect formatting of non-terminal. Should have 2 non-terms, eg. [X][X]. Current string: [HIV] 
Exit code: 1
The decoder died. CONFIG WAS -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391' [1]+ Exit 2 $MOSES/scripts/training/mert-moses.pl tune.true.en tune.true.tch $MOSES/bin/moses train/model/moses.ini --mertdir $MOSES/bin/ &> mert.out 


_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


--
Hieu Hoang
http://www.hoang.co.uk/hieu


--
Hieu Hoang
http://www.hoang.co.uk/hieu


_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Reply via email to