Hi Vito
The 0.2 difference is after retuning? That's normal then.
But a difference of 5 bleu without retuning suggests a bug. Did you say
that this only happens with PhraseDictionaryMultiModel?
cheers - Barry
On 25/11/15 13:53, Vito Mandorino wrote:
Thank you. In our tests it seems that with the OnDisk table the
quality is basically the same between the two versions of Moses
(average 0.2 difference in score Bleu) but for the CompactPhraseTable
the difference is larger (2 points Bleu loss in average after
re-tuning with the new version of Moses, and more than 5 points Bleu
in average without re-tuning).
Do you think a better quality would be obtained by running a complete
re-training of the model with the new version of Moses?
Best regards,
Vito
2015-11-24 16:31 GMT+01:00 Hieu Hoang <hieuho...@gmail.com
<mailto:hieuho...@gmail.com>>:
There was a change in the underlying datastructure for stacks, it
changed from std::set (ordered) to boost::unordered_set.
https://github.com/moses-smt/mosesdecoder/commit/6b182ee5e987a5b2823aea7eaaa7ef0457c6a30d
This got some speed gains
1 5 10 15 20 25 30 35
56 real4m57.795s real1m19.005s real0m51.636s real0m49.624s
real0m49.869s real0m52.475s real0m53.806s real 0m54.957s
13/10 baseline user4m41.255s user5m45.086s user6m34.053s
user8m12.430s user8m10.667s user8m16.486s user8m10.592s user
8m13.859s
sys0m16.514s sys0m35.494s sys0m54.513s sys1m10.643s
sys1m18.449s sys1m21.738s sys1m23.133s sys 1m25.048s
57 real4m41.148s real1m16.002s real0m50.747s real0m48.711s
real0m49.130s real0m51.473s real0m53.141s real 0m54.513s
(56) + unordered set stack user4m23.968s user5m30.356s
user6m26.167s user7m39.286s user7m56.229s user7m52.669s
user7m56.978s user 7m56.216s
sys0m17.231s sys0m35.063s sys0m54.081s sys1m10.137s
sys1m17.194s sys1m22.912s sys1m25.948s sys 1m26.247s
However, the hypotheses are now added to the stack in a different
order so there will be slight differences in results
On 24/11/2015 13:53, Vito Mandorino wrote:
Hi,
in some of our tests a recent version of Moses (pulled from
github last week) and an older one do not give the same
translations on the same source segment (with the same moses.ini).
Here is the 5-best list for the translation of 'test' with the
last week version:
0 ||| test ||| LexicalReordering0= -1.1969 0 0 0 0 0
Distortion0= 0 LM0= -51.1788 WordPenalty0= -1 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -3.03811 -2.5834 -2.08503 -1.83075
||| -1.27754
0 ||| testing ||| LexicalReordering0= 0 0 0 0 0 0 Distortion0= 0
LM0= -35.1495 WordPenalty0= -1 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -5.21045 -5.04877 -4.71131 -4.66382
||| -1.70337
0 ||| funds ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -11.3753 WordPenalty0= -1 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -10.8209 -10.6835 -5.14555 -5.73388
||| -1.77009
0 ||| known as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -58.8877 WordPenalty0= -3 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -4.42285 -11.9339 -5.14555 -18.0392
||| -1.89152
0 ||| as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -35.5353 WordPenalty0= -2 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -9.34698 -11.9339 -5.14555 -9.14874
||| -1.89159
and with the older version of Moses:
0 ||| funds ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -11.3753 WordPenalty0= -1 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -2.52548 -2.52544 -2.45544 -2.48609
||| -0.815668
0 ||| as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -35.5353 WordPenalty0= -2 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -2.52464 -2.52565 -2.45544 -2.5244
||| -0.953799
0 ||| as ||| LexicalReordering0= -3.1355 0 0 0 0 0 Distortion0=
0 LM0= -34.1633 WordPenalty0= -1 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -2.5256 -2.52565 -2.45544 -2.48609
||| -1.07254
0 ||| known as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -58.8877 WordPenalty0= -3 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -2.38597 -2.52565 -2.45544 -2.52573
||| -1.07536
0 ||| is known as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
Distortion0= 0 LM0= -80.8518 WordPenalty0= -4 PhrasePenalty0= 1
PhraseDictionaryMultiModel0= -2.37158 -2.52565 -2.45544 -2.52573
||| -1.18753
This looks very strange. The only difference is in the
phrase-table scores. Do you have any idea of what is going on?
The only possibility which come to mind is maybe a different
handling of the PhraseDictionaryMultiModel feature.
The moses.ini is in attachment.
Best regards,
Vito
--
*M**. Vito MANDORINO -- Chief Scientist*
Description : Description : lingua_custodia_final full logo
*/The Translation Trustee/*
*1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
*Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
<tel:%2B33%206%2084%2065%2068%2089>*
*Email
:****<mailto:massinissa.ah...@linguacustodia.com>vito.mandor...@linguacustodia.com
<mailto:vito.mandor...@linguacustodia.com>***
*Website :****www.linguacustodia.com
<http://www.linguacustodia.com/> - www.thetranslationtrustee.com
<http://www.thetranslationtrustee.com/>*
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu <mailto:Moses-support@mit.edu>
http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
http://www.hoang.co.uk/hieu
--
*M**. Vito MANDORINO -- Chief Scientist*
Description : Description : lingua_custodia_final full logo
*/The Translation Trustee/*
*1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
*Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
<tel:%2B33%206%2084%2065%2068%2089>*
*Email :****vito.mandor...@linguacustodia.com
<mailto:massinissa.ah...@linguacustodia.com>***
*Website :****www.linguacustodia.com
<http://www.linguacustodia.com/> - www.thetranslationtrustee.com
<http://www.thetranslationtrustee.com/>*
This body part will be downloaded on demand.
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
