You are right, that's seems to be a mistake. "-threads" should not be
specified twice. Anyone speaks EMS?
W dniu 28.02.2016 o 09:51, Vincent Nguyen pisze:
Marcin, (or others since it relates to EMS...),
look at my TRAINING_binarize-config.7.STDERR file below.
I have highlighted in Bold the important stuff.
I am running the binarizer in EMS with -threads 8
but in the detailed execution, especially when gzip is run, then I see
a "threads 1" parameter.
is this normal ?
Thanks,
Vincent
Executing:
/netshr/mosesdecoder/scripts/training/filter-model-given-input.pl
/netshr/working-fr-en/model/moses.bin.ini.7.tables
/netshr/working-fr-en/model/moses.ini.7 /dev/null -nofilter
-Binarizer "/netshr/mosesdecoder/bin/processPhraseTableMin *-threads 8*"
Executing: mkdir -p /netshr/working-fr-en/model/moses.bin.ini.7.tables
Stripping XML...
Executing:
/netshr/mosesdecoder/scripts/training/../generic/strip-xml.perl <
/dev/null > /netshr/working-fr-en/model/moses.bin.ini.7.tables/input.49617
pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/netshr/working-fr-en/model/phrase-table.7 input-factor=0
output-factor=0
Considering factor 0
ro:LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/netshr/working-fr-en/model/reordering-table.7.wbe-msd-bidirectional-fe.gz
Considering factor 0
Filtering files...
filtering /netshr/working-fr-en/model/phrase-table.7 ->
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1...
Executing: ln -s /netshr/working-fr-en/model/phrase-table.7.gz
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.gz
binarizing...
*Executing: gzip *-cd
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.gz
| LC_ALL=C sort -T /netshr/working-fr-en/model/moses.bin.ini.7.tables
>
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.gz.sorted
&& /netshr/mosesdecoder/bin/processPhraseTableMin *-threads 8* -in
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.gz.sorted
-out
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1 -nscores
4*-threads 1* && rm
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.gz.sorted
Used options:
Text phrase table will be read from:
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.gz.sorted
Output phrase table will be written to:
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.minphr
Step size for source landmark phrases: 2^10=1024
Source phrase fingerprint size: 16 bits / P(fp)=1.52588e-05
Selected target phrase encoding: Huffman + PREnc
Maxiumum allowed rank for PREnc: 100
Number of score components in phrase table: 4
Single Huffman code set for score components: no
Using score quantization: no
Explicitly included alignment information: yes
*Running with 1 threads*
Pass 1/3: Creating hash function for rank assignment
..................................................[5000000]
..................................................[10000000]
..................................................[15000000]
..................................................[20000000]
..................................................[25000000]
...........................................
Pass 2/3: Creating source phrase index + Encoding target phrases
..................................................[5000000]
..................................................[10000000]
..................................................[15000000]
..................................................[20000000]
..................................................[25000000]
...........................................
Intermezzo: Calculating Huffman code sets
Creating Huffman codes for 94477 target phrase symbols
Creating Huffman codes for 211655 scores
Creating Huffman codes for 9108470 scores
Creating Huffman codes for 167461 scores
Creating Huffman codes for 7856800 scores
Creating Huffman codes for 26 alignment points
Pass 3/3: Compressing target phrases
..................................................[5000000]
..................................................[10000000]
..................................................[15000000]
..................................................[20000000]
..................................................[25000000]
...........................................
Saving to
/netshr/working-fr-en/model/moses.bin.ini.7.tables/phrase-table.0-0.1.1.minphr
Done
filtering
/netshr/working-fr-en/model/reordering-table.7.wbe-msd-bidirectional-fe.gz
->
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe...
Executing: ln -s
/netshr/working-fr-en/model/reordering-table.7.wbe-msd-bidirectional-fe.gz/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.gz
binarizing...
*Executing: gzip* -cd
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.gz
| LC_ALL=C sort -T /netshr/working-fr-en/model/moses.bin.ini.7.tables
>
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.gz.sorted
&& /netshr/mosesdecoder/bin/processLexicalTableMin*-threads 8* -in
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.gz.sorted
-out
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe
*-threads 1* && rm
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.gz.sorted
Used options:
Text reordering table will be read from:
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.gz.sorted
Output reordering table will be written to:
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.minlexr
Step size for source landmark phrases: 2^10=1024
Phrase fingerprint size: 16 bits / P(fp)=1.52588e-05
Single Huffman code set for score components: no
Using score quantization: no
*Running with 1 threads*
Pass 1/2: Creating phrase index + Counting scores
..................................................[5000000]
..................................................[10000000]
..................................................[15000000]
..................................................[20000000]
..................................................[25000000]
..................................................[30000000]
..................................................[35000000]
..................................................[40000000]
..................................................[45000000]
..................................................[50000000]
..................................................[55000000]
..................................................[60000000]
..................................................[65000000]
.............................................
Intermezzo: Calculating Huffman code sets
Creating Huffman codes for 38763 scores
Creating Huffman codes for 22686 scores
Creating Huffman codes for 38050 scores
Creating Huffman codes for 38284 scores
Creating Huffman codes for 21654 scores
Creating Huffman codes for 37554 scores
Pass 2/2: Compressing scores
..................................................[5000000]
..................................................[10000000]
..................................................[15000000]
..................................................[20000000]
..................................................[25000000]
..................................................[30000000]
..................................................[35000000]
..................................................[40000000]
..................................................[45000000]
..................................................[50000000]
..................................................[55000000]
..................................................[60000000]
..................................................[65000000]
.............................................
Saving to
/netshr/working-fr-en/model/moses.bin.ini.7.tables/reordering-table.7.wbe-msd-bidirectional-fe.minlexr
Done
Executing: rm -f /netshr/working-fr-en/model/moses.bin.ini.7; ln -s
/netshr/working-fr-en/model/moses.bin.ini.7.tables/moses.ini
/netshr/working-fr-en/model/moses.bin.ini.7
Le 17/02/2016 23:16, Marcin Junczys-Dowmunt a écrit :
I just checked, it's really weirdly slow now. Apparently using more than
4 threads is a bad idea. But 4 threads seems to be about 2 times faster
than just one. I remember that used to work better. Maybe because I
haven't tcmalloc linked?
On 17.02.2016 23:07, Marcin Junczys-Dowmunt wrote:
It is, just not very well done. It generally does not make sense to have
more than 8-10 threads. That should however be somewhat faster than only
a single thread.
On 17.02.2016 22:44, Vincent Nguyen wrote:
I have the feeling it's not.
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
