Dear list,
lmplz crashed on my machine recently. Command is
lmplz -o 4 -S 70% --text zhc-simp.txt --arpa zhc.lm --prune 0 1 1 2
=== 1/5 Counting and sorting n-grams ===
Reading /home/gumble/docs/E/corpus/zhs/zhc-simp.txt
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
tcmalloc: large alloc 2340552704 bytes == 0x55e7ed4f4000 @
tcmalloc: large alloc 9362194432 bytes == 0x55e878d14000 @
****************************************************************************************************
Unigram tokens 886453003 types 66249
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:794988 2:1961835648 3:3678441728 4:5885507072
tcmalloc: large alloc 5885509632 bytes == 0x55e7ed4f4000 @
tcmalloc: large alloc 1961836544 bytes == 0x55e94c29c000 @
tcmalloc: large alloc 3678445568 bytes == 0x55e9c1190000 @
Statistics:
1 66249 D1=0.549028 D2=1.18255 D3+=0.99644
2 14266408/22790840 D1=0.615082 D2=1.06095 D3+=1.47555
3 87810872/205978808 D1=0.742285 D2=1.17282 D3+=1.49899
4 62909089/415283792 D1=0.698985 D2=1.20588 D3+=1.54463
Memory estimate for binary LM:
type MB
probing 3417 assuming -p 1.5
probing 4002 assuming -r models -p 1.5
trie 1653 without quantization
trie 908 assuming -q 8 -b 8 quantization
trie 1418 assuming -a 22 array pointer compression
trie 674 assuming -a 22 -q 8 -b 8 array pointer compression and
quantization
=== 3/5 Calculating and sorting initial probabilities ===
tcmalloc: large alloc 4119576576 bytes == 0x55e94c1d8000 @
tcmalloc: large alloc 9966813184 bytes == 0x55eaaf630000 @
Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
##**********###############################################################-----##**********++#############################################################-----##************#############################################################-----##************####################################################################************####################################################################************+###################################################################*************###################################################################*************#####################################################################################
=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
---------------------------------------------------------------------------------------------------terminate
called after throwing an instance of 'lm::FormatLoadException'
what(): ./lm/common/joint_order.hh:61 in void lm::JointOrder(const
util::stream::ChainPositions&, Callback&) [with Callback =
lm::builder::{anonymous}::Callback<lm::builder::{anonymous}::OutputProbBackoff>;
Compare = lm::SuffixOrder] threw FormatLoadException because `order !=
current + 1'.
Detected n-gram without matching suffix
--
Dingyuan Wang
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
