HI Vincent
The compact phrase table is a direct replacement for the on disk phrase
table. Just use processPhraseTableMin as your binarizer (no arguments
required)
cheers - Barry
On 30/07/15 21:01, Vincent Nguyen wrote:
Barry,
If I want the end result to be Compact Tables and not OnDisk.
Do I have to binarize first or can I convert directly to Compact ?
(ie can I skip the CreateOnDisk stuff)
if so is there a predefined script or should do it manually ?
thanks
Le 28/07/2015 15:44, Barry Haddow a écrit :
Hi Vincent
I think the quotes are getting stripped off further down the
pipeline. You could work around by changing to the compact phrase
table. Or try editing binarize-model.perl to change
safesystem("$RealBin/filter-model-given-input.pl $targetdir
$input_config /dev/null $hierarchical -nofilter -Binarizer
$binarizer") || die "binarising failed";
to
safesystem("$RealBin/filter-model-given-input.pl $targetdir
$input_config /dev/null $hierarchical -nofilter -Binarizer
\"$binarizer\"") || die "binarising failed";
Note the escaped quotes around the $binarizer.
cheers - Barry
On 28/07/15 14:09, Vincent Nguyen wrote:
same error:
#!/bin/bash
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
cd /home/moses/working
echo 'starting at '`date`' on '`hostname`
mkdir -p /home/moses/working/training
mkdir -p /home/moses/working/model
/home/moses/mosesdecoder/scripts/training/binarize-model.perl
/home/moses/working/model/moses.ini.5
/home/moses/working/model/moses.bin.ini.6 -Binarizer
"/home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"
echo 'finished at '`date`
touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE
Le 28/07/2015 14:47, Barry Haddow a écrit :
Hi Vincent
It could be a bug. Could you edit
mosesdecoder/scripts/ems/experiment.meta and change the line:
template: $binarize-all IN OUT -Binarizer $ttable-binarizer
to
template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"
Note that I have added quotes. Then you'll have to delete the most
recent run, and re-run experiment.perl. If it works, fine. If it
doesn't, could you post the steps/6/TRAINING_binarize-config.6
script (hopefully I got the name right - you may need to change the
number)
cheers - Barry
On 28/07/15 13:11, Vincent Nguyen wrote:
I know but this is what I have in my config.basic now:
# conversion of rule table into binary on-disk format
ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
binarize-all = $moses-script-dir/training/binarize-model.perl
I don't where else I can add the 5 arguments or if I need to
reference ttable-binarizer somewhere
Le 28/07/2015 13:49, Barry Haddow a écrit :
Hi Vincent
If you look at the error log, you will see:
Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt
numSourceFactors numTargetFactors numScores tableLimit
sortScoreIndex inputPath outputPath
You are missing the first 5 arguments to CreateOnDiskPt, as given
in config.basic.
cheers - Barry
On 28/07/15 12:37, Vincent Nguyen wrote:
I don't know why but the binarize crashes see below ....
in my working directory I have 2 subdir,
"tuning" with inside moses.filtered.ini.5 moses.ini.5
moses.tuned.ini.5
and
"model" with inside moses.ini.5 (apparently this one does not
have the
tuned weights)
those in the tuning subdir : the "tuned" one moses.tuned.ini.5
generated
after the moses.ini.5 seems to point on phrase-table.5.gz not
binarized
and the moses.5.ini seem to point on the binarized within
tuning/filtered.5/...
unclear to me on which one I should use.
If you run EMS, there will be a filtered ini file inside the
evaluation directory which can be used to translate the test
set using the tuned weights. However this model is filtered for
the test set, so you cannot use it on other sentences.
If you want the full model binarised, then you should add:
binarize-all = $moses-script-dir/training/binarize-model.perl
to the [GENERAL] section of the EMS config and rerun EMS. In
this case the moses.tuned.ini in tuning can be used to
translate any sentences.
Executing:
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl
/home/moses/working/model/moses.bin.ini.6.tables
/home/moses/working/model/moses.ini.5 /dev/null -nofilter
-Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt
Executing: mkdir -p
/home/moses/working/model/moses.bin.ini.6.tables
Stripping XML...
Executing:
/home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl
< /dev/null >
/home/moses/working/model/moses.bin.ini.6.tables/input.34384
pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/home/moses/working/model/phrase-table.5 input-factor=0
output-factor=0
Considering factor 0
ro:LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz
Considering factor 0
Filtering files...
filtering /home/moses/working/model/phrase-table.5 ->
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1...
Executing: ln -s /home/moses/working/model/phrase-table.5.gz
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
binarizing...
Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
/home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin
Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt
numSourceFactors numTargetFactors numScores tableLimit
sortScoreIndex inputPath outputPath
Exit code: 1
Can't binarize at
/home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl
line 417.
Exit code: 1
binarising failed at
/home/moses/mosesdecoder/scripts/training/binarize-model.perl
line 43.
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
