Re: [Moses-support] EMS help

[Vincent Nguyen]

Many thanks it seems to have fixed the issue, maybe worth to fix it on 
github.


Le 28/07/2015 15:44, Barry Haddow a écrit :

Hi Vincent


I think the quotes are getting stripped off further down the pipeline. 
You could work around by changing to the compact phrase table. Or try 
editing binarize-model.perl to change


safesystem("$RealBin/filter-model-given-input.pl  $targetdir 
$input_config /dev/null $hierarchical -nofilter -Binarizer 
$binarizer") || die "binarising failed";


to

safesystem("$RealBin/filter-model-given-input.pl  $targetdir 
$input_config /dev/null $hierarchical -nofilter -Binarizer 
\"$binarizer\"") || die "binarising failed";


Note the escaped quotes around the $binarizer.

cheers - Barry

On 28/07/15 14:09, Vincent Nguyen wrote:

same error:

#!/bin/bash
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
cd /home/moses/working
echo 'starting at '`date`' on '`hostname`
mkdir -p /home/moses/working/training
mkdir -p /home/moses/working/model
/home/moses/mosesdecoder/scripts/training/binarize-model.perl 
/home/moses/working/model/moses.ini.5 
/home/moses/working/model/moses.bin.ini.6 -Binarizer 
"/home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"


echo 'finished at '`date`
touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE




Le 28/07/2015 14:47, Barry Haddow a écrit :

Hi Vincent

It could be a bug. Could you edit 
mosesdecoder/scripts/ems/experiment.meta and change the line:


  template: $binarize-all IN OUT -Binarizer $ttable-binarizer

to

  template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"

Note that I have added quotes. Then you'll have to delete the most 
recent run, and re-run experiment.perl. If it works, fine. If it 
doesn't, could you post the steps/6/TRAINING_binarize-config.6 
script (hopefully I got the name right - you may need to change the 
number)


cheers - Barry


On 28/07/15 13:11, Vincent Nguyen wrote:

I know but this is what I have in my config.basic now:
# conversion of rule table into binary on-disk format
ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
binarize-all = $moses-script-dir/training/binarize-model.perl

I don't where else I can add the 5 arguments or if I need to 
reference ttable-binarizer somewhere



Le 28/07/2015 13:49, Barry Haddow a écrit :

Hi Vincent

If you look at the error log, you will see:

Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
numSourceFactors numTargetFactors numScores tableLimit 
sortScoreIndex inputPath outputPath 
You are missing the first 5 arguments to CreateOnDiskPt, as given 
in config.basic.


cheers - Barry

On 28/07/15 12:37, Vincent Nguyen wrote:

I don't know why but the binarize crashes see below 




in my working directory I have 2 subdir,
"tuning" with inside moses.filtered.ini.5 moses.ini.5 
moses.tuned.ini.5

and
"model" with inside moses.ini.5 (apparently this one does not 
have the

tuned weights)

those in the tuning subdir : the "tuned" one moses.tuned.ini.5 
generated
after the moses.ini.5 seems to point on phrase-table.5.gz not 
binarized

and the moses.5.ini seem to point on the binarized within
tuning/filtered.5/...
unclear to me on which one I should use.
If you run EMS, there will be a filtered ini file inside the 
evaluation directory which can be used to translate the test set 
using the tuned weights. However this model is filtered for the 
test set, so you cannot use it on other sentences.


If you want the full model binarised, then you should add:

binarize-all = $moses-script-dir/training/binarize-model.perl

to the [GENERAL] section of the EMS config and rerun EMS. In 
this case the moses.tuned.ini in tuning can be used to translate 
any sentences.





Executing: 
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
/home/moses/working/model/moses.bin.ini.6.tables 
/home/moses/working/model/moses.ini.5 /dev/null -nofilter 
-Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt

Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
Stripping XML...
Executing: 
/home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl 
< /dev/null > 
/home/moses/working/model/moses.bin.ini.6.tables/input.34384
pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/home/moses/working/model/phrase-table.5 input-factor=0 
output-factor=0

Considering factor 0
ro:LexicalReordering name=LexicalReordering0 num-features=6 
type=wbe-msd-bidirectional-fe-allff input-factor=0 
output-factor=0 
path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz 


Considering factor 0
Filtering files...
filtering /home/moses/working/model/phrase-table.5 -> 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... 

Executing: ln -s /home/moses/working/model/phrase-table.5.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 


binarizing...
Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
/home/mos

Re: [Moses-support] Problem in translation

[Barry Haddow]

Hi Fatma

I don't see any error in the file. What do you mean "the output was 
wrong." ?


cheers - Barry

On 28/07/15 19:13, fatma elzahraa Eltaher wrote:

Dear All,

I try to build a Model but I get an attached error file . is this mean 
that there are a problem in model . Because I test it by word in 
training data but the output was wrong.


kindly find the attached file.

thank you,



Fatma El-Zahraa El -Taher

Teaching Assistant at Computer & System department

 Faculty of Engineering, Azhar University

Email : fatmaelta...@gmail.com 
mobile: +201141600434



___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Problem in translation

[fatma elzahraa Eltaher]

Dear All,

I try to build a Model but I get an attached error file . is this mean that
there are a problem in model . Because I test it by word in training data
but the output was wrong.

kindly find the attached file.


thank you,



Fatma El-Zahraa El -Taher

Teaching Assistant at Computer & System department

 Faculty of Engineering, Azhar University

Email : fatmaelta...@gmail.com
mobile: +201141600434


filterphrases.err
Description: Binary data
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Generation step for factorized translation

[Marco Damonte]

Hi,

I'm using EMS and factorized translation. For instance, I have this setting:

input-factors = word
output-factors = word brown50
alignment-factors = "word -> word"
translation-factors = "word -> word+brown50"
generation-factors = "brown50"
reordering-factors = "word -> word"
decoding-steps = "t0"

that is, I have brown clusters as an output factor.

Does someone can point me the proper way to add a generation step to create
the surface word using the cluster factor?

I tried:
generation-factors = "brown50 -> word"
but the experiment crashed durinh tuning

Thanks in advance
Marco
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Barry Haddow]

Hi Vincent


I think the quotes are getting stripped off further down the pipeline. 
You could work around by changing to the compact phrase table. Or try 
editing binarize-model.perl to change


safesystem("$RealBin/filter-model-given-input.pl  $targetdir 
$input_config /dev/null $hierarchical -nofilter -Binarizer $binarizer") 
|| die "binarising failed";


to

safesystem("$RealBin/filter-model-given-input.pl  $targetdir 
$input_config /dev/null $hierarchical -nofilter -Binarizer 
\"$binarizer\"") || die "binarising failed";


Note the escaped quotes around the $binarizer.

cheers - Barry

On 28/07/15 14:09, Vincent Nguyen wrote:

same error:

#!/bin/bash
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
cd /home/moses/working
echo 'starting at '`date`' on '`hostname`
mkdir -p /home/moses/working/training
mkdir -p /home/moses/working/model
/home/moses/mosesdecoder/scripts/training/binarize-model.perl 
/home/moses/working/model/moses.ini.5 
/home/moses/working/model/moses.bin.ini.6 -Binarizer 
"/home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"


echo 'finished at '`date`
touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE




Le 28/07/2015 14:47, Barry Haddow a écrit :

Hi Vincent

It could be a bug. Could you edit 
mosesdecoder/scripts/ems/experiment.meta and change the line:


  template: $binarize-all IN OUT -Binarizer $ttable-binarizer

to

  template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"

Note that I have added quotes. Then you'll have to delete the most 
recent run, and re-run experiment.perl. If it works, fine. If it 
doesn't, could you post the steps/6/TRAINING_binarize-config.6 script 
(hopefully I got the name right - you may need to change the number)


cheers - Barry


On 28/07/15 13:11, Vincent Nguyen wrote:

I know but this is what I have in my config.basic now:
# conversion of rule table into binary on-disk format
ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
binarize-all = $moses-script-dir/training/binarize-model.perl

I don't where else I can add the 5 arguments or if I need to 
reference ttable-binarizer somewhere



Le 28/07/2015 13:49, Barry Haddow a écrit :

Hi Vincent

If you look at the error log, you will see:

Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
numSourceFactors numTargetFactors numScores tableLimit 
sortScoreIndex inputPath outputPath 
You are missing the first 5 arguments to CreateOnDiskPt, as given 
in config.basic.


cheers - Barry

On 28/07/15 12:37, Vincent Nguyen wrote:

I don't know why but the binarize crashes see below 




in my working directory I have 2 subdir,
"tuning" with inside moses.filtered.ini.5  moses.ini.5 
moses.tuned.ini.5

and
"model" with inside moses.ini.5 (apparently this one does not 
have the

tuned weights)

those in the tuning subdir : the "tuned" one moses.tuned.ini.5 
generated
after the moses.ini.5 seems to point on phrase-table.5.gz not 
binarized

and the moses.5.ini seem to point on the binarized within
tuning/filtered.5/...
unclear to me on which one I should use.
If you run EMS, there will be a filtered ini file inside the 
evaluation directory which can be used to translate the test set 
using the tuned weights. However this model is filtered for the 
test set, so you cannot use it on other sentences.


If you want the full model binarised, then you should add:

binarize-all = $moses-script-dir/training/binarize-model.perl

to the [GENERAL] section of the EMS config and rerun EMS. In this 
case the moses.tuned.ini in tuning can be used to translate any 
sentences.





Executing: 
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
/home/moses/working/model/moses.bin.ini.6.tables 
/home/moses/working/model/moses.ini.5 /dev/null  -nofilter 
-Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt

Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
Stripping XML...
Executing: 
/home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl < 
/dev/null > 
/home/moses/working/model/moses.bin.ini.6.tables/input.34384
pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/home/moses/working/model/phrase-table.5 input-factor=0 
output-factor=0

Considering factor 0
ro:LexicalReordering name=LexicalReordering0 num-features=6 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz 


Considering factor 0
Filtering files...
filtering /home/moses/working/model/phrase-table.5 -> 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... 

Executing: ln -s /home/moses/working/model/phrase-table.5.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 


binarizing...
Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table

Re: [Moses-support] EMS help

[Vincent Nguyen]

same error:

#!/bin/bash
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
cd /home/moses/working
echo 'starting at '`date`' on '`hostname`
mkdir -p /home/moses/working/training
mkdir -p /home/moses/working/model
/home/moses/mosesdecoder/scripts/training/binarize-model.perl 
/home/moses/working/model/moses.ini.5 
/home/moses/working/model/moses.bin.ini.6 -Binarizer 
"/home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"


echo 'finished at '`date`
touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE




Le 28/07/2015 14:47, Barry Haddow a écrit :

Hi Vincent

It could be a bug. Could you edit 
mosesdecoder/scripts/ems/experiment.meta and change the line:


  template: $binarize-all IN OUT -Binarizer $ttable-binarizer

to

  template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"

Note that I have added quotes. Then you'll have to delete the most 
recent run, and re-run experiment.perl. If it works, fine. If it 
doesn't, could you post the steps/6/TRAINING_binarize-config.6 script 
(hopefully I got the name right - you may need to change the number)


cheers - Barry


On 28/07/15 13:11, Vincent Nguyen wrote:

I know but this is what I have in my config.basic now:
# conversion of rule table into binary on-disk format
ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
binarize-all = $moses-script-dir/training/binarize-model.perl

I don't where else I can add the 5 arguments or if I need to 
reference ttable-binarizer somewhere



Le 28/07/2015 13:49, Barry Haddow a écrit :

Hi Vincent

If you look at the error log, you will see:

Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
numTargetFactors numScores tableLimit sortScoreIndex inputPath 
outputPath 
You are missing the first 5 arguments to CreateOnDiskPt, as given in 
config.basic.


cheers - Barry

On 28/07/15 12:37, Vincent Nguyen wrote:

I don't know why but the binarize crashes see below 




in my working directory I have 2 subdir,
"tuning" with inside moses.filtered.ini.5  moses.ini.5 
moses.tuned.ini.5

and
"model" with inside moses.ini.5 (apparently this one does not 
have the

tuned weights)

those in the tuning subdir : the "tuned" one moses.tuned.ini.5 
generated
after the moses.ini.5 seems to point on phrase-table.5.gz not 
binarized

and the moses.5.ini seem to point on the binarized within
tuning/filtered.5/...
unclear to me on which one I should use.
If you run EMS, there will be a filtered ini file inside the 
evaluation directory which can be used to translate the test set 
using the tuned weights. However this model is filtered for the 
test set, so you cannot use it on other sentences.


If you want the full model binarised, then you should add:

binarize-all = $moses-script-dir/training/binarize-model.perl

to the [GENERAL] section of the EMS config and rerun EMS. In this 
case the moses.tuned.ini in tuning can be used to translate any 
sentences.





Executing: 
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
/home/moses/working/model/moses.bin.ini.6.tables 
/home/moses/working/model/moses.ini.5 /dev/null  -nofilter 
-Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt

Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
Stripping XML...
Executing: 
/home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl 
< /dev/null > 
/home/moses/working/model/moses.bin.ini.6.tables/input.34384
pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/home/moses/working/model/phrase-table.5 input-factor=0 
output-factor=0

Considering factor 0
ro:LexicalReordering name=LexicalReordering0 num-features=6 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz 


Considering factor 0
Filtering files...
filtering /home/moses/working/model/phrase-table.5 -> 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... 

Executing: ln -s /home/moses/working/model/phrase-table.5.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 


binarizing...
Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin 

Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
numTargetFactors numScores tableLimit sortScoreIndex inputPath 
outputPath

Exit code: 1
Can't binarize at 
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
line 417.

Exit code: 1
binarising failed at 
/home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43.












___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Barry Haddow]

Hi Vincent

It could be a bug. Could you edit 
mosesdecoder/scripts/ems/experiment.meta and change the line:

   template: $binarize-all IN OUT -Binarizer $ttable-binarizer

to

   template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"

Note that I have added quotes. Then you'll have to delete the most 
recent run, and re-run experiment.perl. If it works, fine. If it 
doesn't, could you post the steps/6/TRAINING_binarize-config.6 script 
(hopefully I got the name right - you may need to change the number)

cheers - Barry


On 28/07/15 13:11, Vincent Nguyen wrote:
> I know but this is what I have in my config.basic now:
> # conversion of rule table into binary on-disk format
> ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
> binarize-all = $moses-script-dir/training/binarize-model.perl
>
> I don't where else I can add the 5 arguments or if I need to reference 
> ttable-binarizer somewhere
>
>
> Le 28/07/2015 13:49, Barry Haddow a écrit :
>> Hi Vincent
>>
>> If you look at the error log, you will see:
>>
>>> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
>>> numTargetFactors numScores tableLimit sortScoreIndex inputPath 
>>> outputPath 
>> You are missing the first 5 arguments to CreateOnDiskPt, as given in 
>> config.basic.
>>
>> cheers - Barry
>>
>> On 28/07/15 12:37, Vincent Nguyen wrote:
>>> I don't know why but the binarize crashes see below 
>>>

> in my working directory I have 2 subdir,
> "tuning" with inside moses.filtered.ini.5  moses.ini.5 
> moses.tuned.ini.5
> and
> "model" with inside moses.ini.5 (apparently this one does not have 
> the
> tuned weights)
>
> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 
> generated
> after the moses.ini.5 seems to point on phrase-table.5.gz not 
> binarized
> and the moses.5.ini seem to point on the binarized within
> tuning/filtered.5/...
> unclear to me on which one I should use.
 If you run EMS, there will be a filtered ini file inside the 
 evaluation directory which can be used to translate the test set 
 using the tuned weights. However this model is filtered for the 
 test set, so you cannot use it on other sentences.

 If you want the full model binarised, then you should add:

 binarize-all = $moses-script-dir/training/binarize-model.perl

 to the [GENERAL] section of the EMS config and rerun EMS. In this 
 case the moses.tuned.ini in tuning can be used to translate any 
 sentences.

>>>
>>>
>>> Executing: 
>>> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
>>> /home/moses/working/model/moses.bin.ini.6.tables 
>>> /home/moses/working/model/moses.ini.5 /dev/null  -nofilter 
>>> -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>> Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
>>> Stripping XML...
>>> Executing: 
>>> /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl 
>>> < /dev/null > 
>>> /home/moses/working/model/moses.bin.ini.6.tables/input.34384
>>> pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 
>>> path=/home/moses/working/model/phrase-table.5 input-factor=0 
>>> output-factor=0
>>> Considering factor 0
>>> ro:LexicalReordering name=LexicalReordering0 num-features=6 
>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
>>> path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz
>>>  
>>>
>>> Considering factor 0
>>> Filtering files...
>>> filtering /home/moses/working/model/phrase-table.5 -> 
>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1... 
>>>
>>> Executing: ln -s /home/moses/working/model/phrase-table.5.gz 
>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 
>>>
>>> binarizing...
>>> Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 
>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin 
>>>
>>> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
>>> numTargetFactors numScores tableLimit sortScoreIndex inputPath 
>>> outputPath
>>> Exit code: 1
>>> Can't binarize at 
>>> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl line 
>>> 417.
>>> Exit code: 1
>>> binarising failed at 
>>> /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43.
>>>
>>
>>
>


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Barry Haddow]

Hi Vincent

If you look at the error log, you will see:

> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
> numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath 
You are missing the first 5 arguments to CreateOnDiskPt, as given in 
config.basic.

cheers - Barry

On 28/07/15 12:37, Vincent Nguyen wrote:
> I don't know why but the binarize crashes see below 
>
>>
>>> in my working directory I have 2 subdir,
>>> "tuning" with inside moses.filtered.ini.5  moses.ini.5 
>>> moses.tuned.ini.5
>>> and
>>> "model" with inside moses.ini.5 (apparently this one does not have the
>>> tuned weights)
>>>
>>> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 
>>> generated
>>> after the moses.ini.5 seems to point on phrase-table.5.gz not binarized
>>> and the moses.5.ini seem to point on the binarized within
>>> tuning/filtered.5/...
>>> unclear to me on which one I should use.
>> If you run EMS, there will be a filtered ini file inside the 
>> evaluation directory which can be used to translate the test set 
>> using the tuned weights. However this model is filtered for the test 
>> set, so you cannot use it on other sentences.
>>
>> If you want the full model binarised, then you should add:
>>
>> binarize-all = $moses-script-dir/training/binarize-model.perl
>>
>> to the [GENERAL] section of the EMS config and rerun EMS. In this 
>> case the moses.tuned.ini in tuning can be used to translate any 
>> sentences.
>>
>
>
> Executing: 
> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
> /home/moses/working/model/moses.bin.ini.6.tables 
> /home/moses/working/model/moses.ini.5 /dev/null  -nofilter -Binarizer 
> /home/moses/mosesdecoder/bin/CreateOnDiskPt
> Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
> Stripping XML...
> Executing: 
> /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl < 
> /dev/null > /home/moses/working/model/moses.bin.ini.6.tables/input.34384
> pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 
> path=/home/moses/working/model/phrase-table.5 input-factor=0 
> output-factor=0
> Considering factor 0
> ro:LexicalReordering name=LexicalReordering0 num-features=6 
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
> path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz 
>
> Considering factor 0
> Filtering files...
> filtering /home/moses/working/model/phrase-table.5 -> 
> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1...
> Executing: ln -s /home/moses/working/model/phrase-table.5.gz 
> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
> binarizing...
> Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 
> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin
> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
> numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath
> Exit code: 1
> Can't binarize at 
> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
> line 417.
> Exit code: 1
> binarising failed at 
> /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43.
>


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Vincent Nguyen]

I don't know why but the binarize crashes see below 

>
>> in my working directory I have 2 subdir,
>> "tuning" with inside moses.filtered.ini.5  moses.ini.5 moses.tuned.ini.5
>> and
>> "model" with inside moses.ini.5 (apparently this one does not have the
>> tuned weights)
>>
>> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 generated
>> after the moses.ini.5 seems to point on phrase-table.5.gz not binarized
>> and the moses.5.ini seem to point on the binarized within
>> tuning/filtered.5/...
>> unclear to me on which one I should use.
> If you run EMS, there will be a filtered ini file inside the 
> evaluation directory which can be used to translate the test set using 
> the tuned weights. However this model is filtered for the test set, so 
> you cannot use it on other sentences.
>
> If you want the full model binarised, then you should add:
>
> binarize-all = $moses-script-dir/training/binarize-model.perl
>
> to the [GENERAL] section of the EMS config and rerun EMS. In this case 
> the moses.tuned.ini in tuning can be used to translate any sentences.
>


Executing: 
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
/home/moses/working/model/moses.bin.ini.6.tables 
/home/moses/working/model/moses.ini.5 /dev/null  -nofilter -Binarizer 
/home/moses/mosesdecoder/bin/CreateOnDiskPt
Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
Stripping XML...
Executing: 
/home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl < 
/dev/null > /home/moses/working/model/moses.bin.ini.6.tables/input.34384
pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4 
path=/home/moses/working/model/phrase-table.5 input-factor=0 output-factor=0
Considering factor 0
ro:LexicalReordering name=LexicalReordering0 num-features=6 
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 
path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz
Considering factor 0
Filtering files...
filtering /home/moses/working/model/phrase-table.5 -> 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1...
Executing: ln -s /home/moses/working/model/phrase-table.5.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
binarizing...
Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz 
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin
Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors 
numTargetFactors numScores tableLimit sortScoreIndex inputPath outputPath
Exit code: 1
Can't binarize at 
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl 
line 417.
Exit code: 1
binarising failed at 
/home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43.
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Vincent Nguyen]

I am NOT talking about the [CORPUS:europarl] section but
the [LM:europarl] I think in this section you need the $pair-extension
same as [LM:nc] where it was fine
Anyway :yes I had an error


Europarl releases usually contain the parallel files (e.g. 
europarl-v7.fr-en.fr) and monolingual files (e.g. europarl-v7.en).


If I am not mistaken, http://www.statmt.org/wmt12/training-parallel.tgz 
do not, this could be the reason.

Also : when.gz is missing yes it stops and give an error message.


OK, this used to work.
Sorry I was not specific enough. the errors pops up when I use 
./daemon.pl for the web translation. Could be just there 




That's right, the interpolated LM uses some code from SRILM. You can 
still use KenLM to create the individual language models, and use 
KenLM during decoding,


OK. But related question :
If I do not interpolate, and if I keep the 2 (or more) LM in the 
moses.ini file

does the decoder work similarly as if I had interpolated 2 LMs ?


If you do not interpolate using EMS then both LMs will be features in 
the model - i.e. you get log-linear interpolation. See here for an 
early comparison of linear and log-linear interpolation - 
https://aclweb.org/anthology/W/W07/W07-0717.pdf - there has been other 
work since then. Note that SRILM does not do linear interpolation 
correctly,




Many thanks.

Just as a general question.
The baseline Tuto mentions this at the end :
This gives me a BLEU score of 23.5 (in comparison, the best result at 
WMT11 was 30.5 , 
although it should be cautioned that this uses NIST BLEU, which does its 
own tokenisation, so there will be 1-2 points difference in the score 
anyway)


The Baseline tuto is done with NewsCommentary_V8. I did it and got 22/23 
as a Bleu score.
My EMS with the config.basic file gives me around 26 (EuroparlV7+NCv10) 
test set out of NC2011.


Is the reason for my "low" score versus 30s because I am using KenLM 
only ? am I missing something else ?






cheers - Barry




___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Barry Haddow]

Hi Vincent

On 28/07/15 10:18, Vincent Nguyen wrote:
> Thanks Barry. Answers and other questions below.
>
> Le 28/07/2015 10:25, Barry Haddow a écrit :
>> Hi Vincent
>>
>>> 2 bugs report :
>>> in the LM Corpus definition for Europarl : the $pair-extension is
>>> missing before .$output-extension
>>> in the step 5 (maybe for others too) generation of the 
>>> moses.tuned.ini.5
>>> file there is a missing ".gz" at the end of phrase-table.5
>>> in the PhraseDictionaryMemory definition.
>> These seem OK to me. For europarl, it points to the monolingual 
>> corpus, and for the phrase table the .gz is implicitly added. Did 
>> they not work for you?
>
> I am NOT talking about the [CORPUS:europarl] section but
> the [LM:europarl] I think in this section you need the $pair-extension
> same as [LM:nc] where it was fine
> Anyway :yes I had an error

Europarl releases usually contain the parallel files (e.g. 
europarl-v7.fr-en.fr) and monolingual files (e.g. europarl-v7.en).

>
> Also : when.gz is missing yes it stops and give an error message.

OK, this used to work.

>
>
>>
>>> I tried to remove the "IGNORE" for the Interpolated-LM section
>>> I am still using KenLM.
>>> BUT I get a message saying I need to define srilm-dir
>>> is SRILM mandatory to turn on the interpolated-lm with KenLM only ?
>> That's right, the interpolated LM uses some code from SRILM. You can 
>> still use KenLM to create the individual language models, and use 
>> KenLM during decoding,
>
> OK. But related question :
> If I do not interpolate, and if I keep the 2 (or more) LM in the 
> moses.ini file
> does the decoder work similarly as if I had interpolated 2 LMs ?

If you do not interpolate using EMS then both LMs will be features in 
the model - i.e. you get log-linear interpolation. See here for an early 
comparison of linear and log-linear interpolation - 
https://aclweb.org/anthology/W/W07/W07-0717.pdf - there has been other 
work since then. Note that SRILM does not do linear interpolation correctly,


cheers - Barry


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Vincent Nguyen]

Thanks Barry. Answers and other questions below.

Le 28/07/2015 10:25, Barry Haddow a écrit :
> Hi Vincent
>
>> 2 bugs report :
>> in the LM Corpus definition for Europarl : the $pair-extension is
>> missing before .$output-extension
>> in the step 5 (maybe for others too) generation of the moses.tuned.ini.5
>> file there is a missing ".gz" at the end of phrase-table.5
>> in the PhraseDictionaryMemory definition.
> These seem OK to me. For europarl, it points to the monolingual 
> corpus, and for the phrase table the .gz is implicitly added. Did they 
> not work for you?

I am NOT talking about the [CORPUS:europarl] section but
the [LM:europarl] I think in this section you need the $pair-extension
same as [LM:nc] where it was fine
Anyway :yes I had an error

Also : when.gz is missing yes it stops and give an error message.



>
>> in my working directory I have 2 subdir,
>> "tuning" with inside moses.filtered.ini.5  moses.ini.5 moses.tuned.ini.5
>> and
>> "model" with inside moses.ini.5 (apparently this one does not have the
>> tuned weights)
>>
>> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 generated
>> after the moses.ini.5 seems to point on phrase-table.5.gz not binarized
>> and the moses.5.ini seem to point on the binarized within
>> tuning/filtered.5/...
>> unclear to me on which one I should use.
> If you run EMS, there will be a filtered ini file inside the 
> evaluation directory which can be used to translate the test set using 
> the tuned weights. However this model is filtered for the test set, so 
> you cannot use it on other sentences.
>
> If you want the full model binarised, then you should add:
>
> binarize-all = $moses-script-dir/training/binarize-model.perl
>
> to the [GENERAL] section of the EMS config and rerun EMS. In this case 
> the moses.tuned.ini in tuning can be used to translate any sentences.


ok will do.


>
>> I tried to remove the "IGNORE" for the Interpolated-LM section
>> I am still using KenLM.
>> BUT I get a message saying I need to define srilm-dir
>> is SRILM mandatory to turn on the interpolated-lm with KenLM only ?
> That's right, the interpolated LM uses some code from SRILM. You can 
> still use KenLM to create the individual language models, and use 
> KenLM during decoding,

OK. But related question :
If I do not interpolate, and if I keep the 2 (or more) LM in the 
moses.ini file
does the decoder work similarly as if I had interpolated 2 LMs ?


>
> cheers - Barry
>
> On 26/07/15 08:36, Vincent Nguyen wrote:
>> Hi,
>>
>> I worked with the config.basic file
>>
>> 2 bugs report :
>> in the LM Corpus definition for Europarl : the $pair-extension is
>> missing before .$output-extension
>> in the step 5 (maybe for others too) generation of the moses.tuned.ini.5
>> file there is a missing ".gz" at the end of phrase-table.5
>> in the PhraseDictionaryMemory definition.
>>
>> Then questions :
>>
>> in my working directory I have 2 subdir,
>> "tuning" with inside moses.filtered.ini.5  moses.ini.5 moses.tuned.ini.5
>> and
>> "model" with inside moses.ini.5 (apparently this one does not have the
>> tuned weights)
>>
>> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 generated
>> after the moses.ini.5 seems to point on phrase-table.5.gz not binarized
>> and the moses.5.ini seem to point on the binarized within
>> tuning/filtered.5/...
>>
>> unclear to me on which one I should use.
>>
>>
>> Last question :
>> I tried to remove the "IGNORE" for the Interpolated-LM section
>> I am still using KenLM.
>> BUT I get a message saying I need to define srilm-dir
>> is SRILM mandatory to turn on the interpolated-lm with KenLM only ?
>>
>>
>>
>>
>>
>> ___
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

Re: [Moses-support] EMS help

[Barry Haddow]

Hi Vincent

> 2 bugs report :
> in the LM Corpus definition for Europarl : the $pair-extension is
> missing before .$output-extension
> in the step 5 (maybe for others too) generation of the moses.tuned.ini.5
> file there is a missing ".gz" at the end of phrase-table.5
> in the PhraseDictionaryMemory definition.
These seem OK to me. For europarl, it points to the monolingual corpus, 
and for the phrase table the .gz is implicitly added. Did they not work 
for you?

> in my working directory I have 2 subdir,
> "tuning" with inside moses.filtered.ini.5  moses.ini.5 moses.tuned.ini.5
> and
> "model" with inside moses.ini.5 (apparently this one does not have the
> tuned weights)
>
> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 generated
> after the moses.ini.5 seems to point on phrase-table.5.gz not binarized
> and the moses.5.ini seem to point on the binarized within
> tuning/filtered.5/...
> unclear to me on which one I should use.
If you run EMS, there will be a filtered ini file inside the evaluation 
directory which can be used to translate the test set using the tuned 
weights. However this model is filtered for the test set, so you cannot 
use it on other sentences.

If you want the full model binarised, then you should add:

binarize-all = $moses-script-dir/training/binarize-model.perl

to the [GENERAL] section of the EMS config and rerun EMS. In this case 
the moses.tuned.ini in tuning can be used to translate any sentences.

> I tried to remove the "IGNORE" for the Interpolated-LM section
> I am still using KenLM.
> BUT I get a message saying I need to define srilm-dir
> is SRILM mandatory to turn on the interpolated-lm with KenLM only ?
That's right, the interpolated LM uses some code from SRILM. You can 
still use KenLM to create the individual language models, and use KenLM 
during decoding,

cheers - Barry

On 26/07/15 08:36, Vincent Nguyen wrote:
> Hi,
>
> I worked with the config.basic file
>
> 2 bugs report :
> in the LM Corpus definition for Europarl : the $pair-extension is
> missing before .$output-extension
> in the step 5 (maybe for others too) generation of the moses.tuned.ini.5
> file there is a missing ".gz" at the end of phrase-table.5
> in the PhraseDictionaryMemory definition.
>
> Then questions :
>
> in my working directory I have 2 subdir,
> "tuning" with inside moses.filtered.ini.5  moses.ini.5 moses.tuned.ini.5
> and
> "model" with inside moses.ini.5 (apparently this one does not have the
> tuned weights)
>
> those in the tuning subdir : the "tuned" one moses.tuned.ini.5 generated
> after the moses.ini.5 seems to point on phrase-table.5.gz not binarized
> and the moses.5.ini seem to point on the binarized within
> tuning/filtered.5/...
>
> unclear to me on which one I should use.
>
>
> Last question :
> I tried to remove the "IGNORE" for the Interpolated-LM section
> I am still using KenLM.
> BUT I get a message saying I need to define srilm-dir
> is SRILM mandatory to turn on the interpolated-lm with KenLM only ?
>
>
>
>
>
> ___
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support

[Moses-support] Errors while running phrase based tutorial from Moses manual

[Sudip Mahajan]

Hello I want to run the following command with moses

%train-model.perl \
--root-dir unfactored \
--corpus factored-corpus/proj-syndicate \
--f de --e en \
--lm 0:3:factored-corpus/surface.lm:0 \
--external-bin-dir .../tools \
--input-factor-max 4

This creates a moses.ini file but the POS table created in
"unfactored/phrase-table.o-o.gz" come sout to be empty.

Also

After running the following script for mert tuning:

#!/bin/bash
myPATH="/media/sudip/linuxdata1"
###Tuning moses.ini, cd to working directory
nohup nice $myPATH/mosesdecoder/scripts/training/mert-moses.pl \
$myPATH/working/tuningcorpus/prepared-corpus/news-test2008.clean.fr \
$myPATH/working/tuningcorpus/prepared-corpus/news-test2008.clean.en \
$myPATH/mosesdecoder/bin/moses \
$myPATH/working/train/model/moses.ini \
--mertdir $myPATH/working/train/model/mert \
--decoder-flags='-threads 3' >& mert.out &


My mert.out shows the following error:
sudip@sudip-pc:/media/sudip/linuxdata1/working$ cat mert.out
tune.sh: line 9: /media/sudip/linuxdata1/working/train/model/moses.ini:
Permission denied

Why is this happening?/

Thanks..
___
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support