Hi Roko
Apart from compact phrase tables, which you should try:
You could start several Moses servers on different ports, with different
decoding parameters. Would that not be feasible for you?
Regards
Mathias
On 05.04.17 13:33, RR wrote:
Hi,
I am currently working on a system to explore MOSES decoding parameter
space - running MOSES in decoding mode with one parameter set, getting
a BLEU score and rerunning with different parameters.
More than 50% of my time is being spent repeatedly "loading" the
phrase table, language model and reordering model. For example,
$ ./example.fr <http://example.fr> | .../moses -f ./run11.moses.ini
-threads 31 > ./trans_baseline
gives: "Start loading text phrase table. Moses format : [37.296]
seconds."
37 seconds just to load a 590MB phrase table, even if it is on
RAMDISK. htop indicates that for those 37 seconds, a single core is
100% utilized.
My understanding is that some kind of optimized data structure (a hash
table of some kind?) is being created in those 37 seconds, which is
then lost and recreated when I re-run MOSES with a different value of
a decoding parameter.
If I want to eliminate this loading time, what is the best way
forward? Is there a phrase table format that avoids a significant
loading time (i.e. a 500MB phrase table loads in < 1 sec)? Should I
try to find a way to run a single MOSES server but with different
decoding parameters? I have a system with 130GB RAM and I am using a
small phrase table that has been filtered for my decoding set.
Thanks in advance,
Roko
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
