date:20151023

Re: [Moses-support] Issue with tuning - decoder died

2015-10-23 Thread Hieu Hoang


doubt it. Look at the script. The problem chars are
   []|<>&'"

On 23/10/2015 13:01, Ya-chi Lin wrote:


Hi Hieu,


Thank for pointing this out. Could you please let me know if Chinese 
square brackets 「」are also a problem?



Best,

Ya-Chi



*寄件者:* Hieu Hoang 
*寄件日期:* 2015年10月23日 下午 02:32
*收件者:* Ya-chi Lin; moses-support@mit.edu
*主旨:* Re: [Moses-support] Issue with tuning - decoder died
you must escape the [ and ] characters. If you had used the moses 
tokenizer, it will do this for you. If you use a different tokenizer, 
then run your tokenized output through

   scripts/tokenizer/escape-special-chars.perl

On 23/10/2015 01:22, Ya-chi Lin wrote:


Hi ALL,


I'm building an English to Traditional Chinese system but have 
encountered a problem while performing tuning. Below is the content 
of mert.out file. Can anyone please tell me what went wrong here? Any 
help will be greatly appreciated.



Using SCRIPTS_ROOTDIR: /opt/mosesdecoder/scripts
Assuming the tables are already filtered, reusing filtered/moses.ini
Using cached features list: ./features.list
MERT starting values and ranges for random generation:
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  Distortion0 =   0.300 ( 0.00 ..  1.00)
LM0 =   0.500 ( 0.00 ..  1.00)
  WordPenalty0 =  -1.000 ( 0.00 ..  1.00)
  PhrasePenalty0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: Distortion0=0.30
featlist: LM0=0.50
featlist: WordPenalty0=-1.00
featlist: PhrasePenalty0=0.20
featlist: TranslationModel0=0.20
featlist: TranslationModel0=0.20
featlist: TranslationModel0=0.20
featlist: TranslationModel0=0.20
run 1 start at Fri Oct 23 03:07:31 EEST 2015
Parsing --decoder-flags: ||
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
Normalizing lambdas: 0.30 0.30 0.30 0.30 0.30 
0.30 0.30 0.50 -1.00 0.20 0.20 0.20 
0.20 0.20
DECODER_CFG = -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 
0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 
TranslationModel0= 0.043478 0.043478 0.043478 0.043478 
PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
Executing: /opt/mosesdecoder/bin/moses   -config filtered/moses.ini 
-weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 
0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 
0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 
Distortion0= 0.065217 WordPenalty0= -0.217391'  -n-best-list 
run1.best100.out 100 distinct  -input-file 
/home/yachilin/working/tune.true.en > run1.out
Executing: /opt/mosesdecoder/bin/moses   -config filtered/moses.ini 
-weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 
0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 
0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 
Distortion0= 0.065217 WordPenalty0= -0.217391'  -n-best-list 
run1.best100.out 100 distinct  -input-file 
/home/yachilin/working/tune.true.en > run1.out
1-10.20.2 0.2 0.2 0.20.3 0.3 0.3 0.3 0.3 0.30.30.5(1) run decoder to 
produce n-best lists

params =
decoder_config = -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 
0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 
TranslationModel0= 0.043478 0.043478 0.043478 0.043478 
PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'

Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty 
PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0 
output-factor=0 LexicalReordering name=LexicalReordering0 
num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 
output-factor=0 
path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe 
Distortion KENLM lazyken=0 name=LM0 factor=0 
path=/home/yachilin/lm/train.blm.tch order=3

input-factors: 0
input-file: /home/yachilin/working/tune.true.en
mapping: 0 T 0
n-best-list: run1.best100.out 100 distinct
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 
PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 
LexicalReordering0= 0.3 0.3 0.3

Re: [Moses-support] Issue with tuning - decoder died

2015-10-23 Thread Hieu Hoang

you must escape the [ and ] characters. If you had used the moses 
tokenizer, it will do this for you. If you use a different tokenizer, 
then run your tokenized output through

   scripts/tokenizer/escape-special-chars.perl

On 23/10/2015 01:22, Ya-chi Lin wrote:


Hi ALL,


I'm building an English to Traditional Chinese system but have 
encountered a problem while performing tuning. Below is the content of 
mert.out file. Can anyone please tell me what went wrong here? Any 
help will be greatly appreciated.



Using SCRIPTS_ROOTDIR: /opt/mosesdecoder/scripts
Assuming the tables are already filtered, reusing filtered/moses.ini
Using cached features list: ./features.list
MERT starting values and ranges for random generation:
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  LexicalReordering0 =   0.300 ( 0.00 ..  1.00)
  Distortion0 =   0.300 ( 0.00 ..  1.00)
LM0 =   0.500 ( 0.00 ..  1.00)
  WordPenalty0 =  -1.000 ( 0.00 ..  1.00)
  PhrasePenalty0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
  TranslationModel0 =   0.200 ( 0.00 ..  1.00)
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: LexicalReordering0=0.30
featlist: Distortion0=0.30
featlist: LM0=0.50
featlist: WordPenalty0=-1.00
featlist: PhrasePenalty0=0.20
featlist: TranslationModel0=0.20
featlist: TranslationModel0=0.20
featlist: TranslationModel0=0.20
featlist: TranslationModel0=0.20
run 1 start at Fri Oct 23 03:07:31 EEST 2015
Parsing --decoder-flags: ||
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
Normalizing lambdas: 0.30 0.30 0.30 0.30 0.30 
0.30 0.30 0.50 -1.00 0.20 0.20 0.20 
0.20 0.20
DECODER_CFG = -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 
0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 
TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 
0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
Executing: /opt/mosesdecoder/bin/moses   -config filtered/moses.ini 
-weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 
0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 
0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 
0.065217 WordPenalty0= -0.217391'  -n-best-list run1.best100.out 100 
distinct  -input-file /home/yachilin/working/tune.true.en > run1.out
Executing: /opt/mosesdecoder/bin/moses   -config filtered/moses.ini 
-weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217 
0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478 
0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0= 
0.065217 WordPenalty0= -0.217391'  -n-best-list run1.best100.out 100 
distinct  -input-file /home/yachilin/working/tune.true.en > run1.out
1-10.20.2 0.2 0.2 0.20.3 0.3 0.3 0.3 0.3 0.30.30.5(1) run decoder to 
produce n-best lists

params =
decoder_config = -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 
0.065217 0.065217 0.065217 0.065217 0.065217 0.065217 
TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 
0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'

Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty 
PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz 
input-factor=0 output-factor=0 LexicalReordering 
name=LexicalReordering0 num-features=6 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe 
Distortion KENLM lazyken=0 name=LM0 factor=0 
path=/home/yachilin/lm/train.blm.tch order=3

input-factors: 0
input-file: /home/yachilin/working/tune.true.en
mapping: 0 T 0
n-best-list: run1.best100.out 100 distinct
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 
PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
weight-overwrite: LM0= 0.108696 LexicalReordering0= 0.065217 
0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 
0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 
Distortion0= 0.065217 WordPenalty0= -0.217391

line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePena

Re: [Moses-support] Issue with tuning - decoder died

Re: [Moses-support] Issue with tuning - decoder died

2 matches

Site Navigation

Mail list logo

Footer information