Size reduction and binarizing time are normal. At WIPO we use only
quantized models, with no quality loss so far. Your speed issues were
caused by insufficient RAM then. Interesting, since I was able to use
98GB models on my 128GB server with several moses instances running in
parallel, but maybe you had other stuff running.
W dniu 26.04.2015 o 21:20, liling tan pisze:
Dear Moses devs/users,
@Ken, I'm working with 128 RAM, the default binarized LM works but
it's kind of slow when tuning.
I've tried the trie and it's wonderful!! Effectively, it brought down
the size of the LM:
Text: 16GB
ARPA: 38G
Binary (no trie): 71GB
Trie Binary: 17GB
*Does the small trie binary look normal? *(it looks magically small,
but i've tested and it didn't give me a difference in BLEU).
The decoding speed reduces, I'm waiting to see how much time saved
when tuning.
The only issue is that using trie binarization takes much longer than
normal binary. It took a night's sleep (5-6 hours) to complete while
the normal binarization takes 1-2 hours. Still it's worth the time
since it reduces the size of the model and decoding speed.
@Marcin, @Ken thanks for the tips on working with big LM and phrase
models.
Regards,
Liling
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
