Hello everyone,
I have tried using processPhraseTableMin on my model trained on
Europarl v7 that's just too large to load into memory uncompressed,
but I get a segfault at each attempt.
Any idea why, or any workaround to try?
Here is the command used and the output:
$ processPhraseTableMin -threads 3 -in train/model/phrase-table.gz
-nscores 4 -out binarised-model/phrase-table
Used options:
Text phrase table will be read from: train/model/phrase-table.gz
Output phrase table will be written to: binarised-model/phrase-table.minphr
Step size for source landmark phrases: 2^10=1024
Source phrase fingerprint size: 16 bits / P(fp)=1.52588e-05
Selected target phrase encoding: Huffman + PREnc
Maxiumum allowed rank for PREnc: 100
Number of score components in phrase table: 4
Single Huffman code set for score components: no
Using score quantization: no
Explicitly included alignment information: yes
Running with 3 threads
Pass 1/3: Creating hash function for rank assignment
..................................................[5000000]
..................................................[10000000]
..................................................[15000000]
..................................................[20000000]
..................................................[25000000]
..................................................[30000000]
..................................................[35000000]
..................................................[40000000]
..................................................[45000000]
..................................................[50000000]
..................................................[55000000]
..................................................[60000000]
...........................
Pass 2/3: Creating source phrase index + Encoding target phrases
*** Segmentation fault
Register dump:
RAX: 0000000000683fc0 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 00007ffc83939418
RBP: 0000000000000008 R8 : 00007fc2575af960 R9 : 00007fc258a35740
R10: 0000000000000008 R11: 00007fc257279260 R12: 00000000014b8560
R13: 0000000000000003 R14: 00007ffc83939350 R15: 00007ffc839393f8
RSP: 00007ffc83938f30
RIP: 00007fc25727927b EFLAGS: 00010202
CS: 0033 FS: 0000 GS: 0000
Trap: 0000000e Error: 00000004 OldMask: 00000000 CR2: 00000000
FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000
RIP: 00000000 RDP: 00000000
ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000
ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000
ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000
ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000
mxcsr: 1f80
XMM0: 0000000000000000000000003cdd67ba XMM1: 0000000000000000000000003cdd67ba
XMM2: 0000000000000000000000003cdd67ba XMM3: 0000000000000000000000003cdd67ba
XMM4: 0000000000000000000000003cdd67ba XMM5: 0000000000000000000000003cdd67ba
XMM6: 0000000000000000000000003cdd67ba XMM7: 0000000000000000000000003cdd67ba
XMM8: 0000000000000000000000003cdd67ba XMM9: 0000000000000000000000003cdd67ba
XMM10: 0000000000000000000000003cdd67ba XMM11: 0000000000000000000000003cdd67ba
XMM12: 0000000000000000000000003cdd67ba XMM13: 0000000000000000000000003cdd67ba
XMM14: 0000000000000000000000003cdd67ba XMM15: 0000000000000000000000003cdd67ba
Backtrace:
/usr/lib/libc.so.6(fwrite+0x1b)[0x7fc25727927b]
processPhraseTableMin[0x42c4a4]
processPhraseTableMin[0x42c662]
processPhraseTableMin[0x417dc3]
processPhraseTableMin[0x405f18]
/usr/lib/libc.so.6(__libc_start_main+0xf0)[0x7fc257230800]
processPhraseTableMin[0x406d29]
Memory map:
00400000-00483000 r-xp 00000000 08:01 8130298
/opt/mosesdecoder/bin/processPhraseTableMin
00682000-00684000 rw-p 00082000 08:01 8130298
/opt/mosesdecoder/bin/processPhraseTableMin
00684000-00685000 rw-p 00000000 00:00 0
014b7000-014d8000 rw-p 00000000 00:00 0 [heap]
7fc1f2f90000-7fc215a76000 rw-p 00000000 00:00 0
7fc21c000000-7fc21f32f000 rw-p 00000000 00:00 0
7fc21f32f000-7fc220000000 ---p 00000000 00:00 0
7fc224000000-7fc2276e6000 rw-p 00000000 00:00 0
7fc2276e6000-7fc228000000 ---p 00000000 00:00 0
7fc228000000-7fc22b34a000 rw-p 00000000 00:00 0
7fc22b34a000-7fc22c000000 ---p 00000000 00:00 0
7fc22c000000-7fc230000000 rw-p 00000000 00:00 0
7fc230000000-7fc234000000 rw-p 00000000 00:00 0
7fc234000000-7fc237fff000 rw-p 00000000 00:00 0
7fc237fff000-7fc238000000 ---p 00000000 00:00 0
7fc238000000-7fc23c000000 rw-p 00000000 00:00 0
7fc23c000000-7fc240000000 rw-p 00000000 00:00 0
7fc240000000-7fc244000000 rw-p 00000000 00:00 0
7fc244000000-7fc2442f6000 rw-p 00000000 00:00 0
7fc2442f6000-7fc248000000 ---p 00000000 00:00 0
7fc248000000-7fc2486b5000 rw-p 00000000 00:00 0
7fc2486b5000-7fc24c000000 ---p 00000000 00:00 0
7fc24c000000-7fc24cb09000 rw-p 00000000 00:00 0
7fc24cb09000-7fc250000000 ---p 00000000 00:00 0
7fc253a09000-7fc253a0a000 ---p 00000000 00:00 0
7fc253a0a000-7fc25420a000 rw-p 00000000 00:00 0
7fc25520c000-7fc25520d000 ---p 00000000 00:00 0
7fc25520d000-7fc255a0d000 rw-p 00000000 00:00 0
7fc255a0d000-7fc255a0e000 ---p 00000000 00:00 0
7fc255a0e000-7fc25620e000 rw-p 00000000 00:00 0
7fc25620e000-7fc25620f000 ---p 00000000 00:00 0
7fc25620f000-7fc256a0f000 rw-p 00000000 00:00 0
7fc256a0f000-7fc256a10000 ---p 00000000 00:00 0
7fc256a10000-7fc257210000 rw-p 00000000 00:00 0
[stack:22662]
7fc257210000-7fc2573a9000 r-xp 00000000 08:01 9178200
/usr/lib/libc-2.21.so
7fc2573a9000-7fc2575a9000 ---p 00199000 08:01 9178200
/usr/lib/libc-2.21.so
7fc2575a9000-7fc2575ad000 r--p 00199000 08:01 9178200
/usr/lib/libc-2.21.so
7fc2575ad000-7fc2575af000 rw-p 0019d000 08:01 9178200
/usr/lib/libc-2.21.so
7fc2575af000-7fc2575b3000 rw-p 00000000 00:00 0
7fc2575b3000-7fc2575cb000 r-xp 00000000 08:01 9178220
/usr/lib/libpthread-2.21.so
7fc2575cb000-7fc2577ca000 ---p 00018000 08:01 9178220
/usr/lib/libpthread-2.21.so
7fc2577ca000-7fc2577cb000 r--p 00017000 08:01 9178220
/usr/lib/libpthread-2.21.so
7fc2577cb000-7fc2577cc000 rw-p 00018000 08:01 9178220
/usr/lib/libpthread-2.21.so
7fc2577cc000-7fc2577d0000 rw-p 00000000 00:00 0
7fc2577d0000-7fc2577e6000 r-xp 00000000 08:01 9178553
/usr/lib/libgcc_s.so.1
7fc2577e6000-7fc2579e5000 ---p 00016000 08:01 9178553
/usr/lib/libgcc_s.so.1
7fc2579e5000-7fc2579e6000 rw-p 00015000 08:01 9178553
/usr/lib/libgcc_s.so.1
7fc2579e6000-7fc257ae9000 r-xp 00000000 08:01 9178248
/usr/lib/libm-2.21.so
7fc257ae9000-7fc257ce9000 ---p 00103000 08:01 9178248
/usr/lib/libm-2.21.so
7fc257ce9000-7fc257cea000 r--p 00103000 08:01 9178248
/usr/lib/libm-2.21.so
7fc257cea000-7fc257ceb000 rw-p 00104000 08:01 9178248
/usr/lib/libm-2.21.so
7fc257ceb000-7fc257ddb000 r-xp 00000000 08:01 9178557
/usr/lib/libstdc++.so.6.0.20
7fc257ddb000-7fc257fdb000 ---p 000f0000 08:01 9178557
/usr/lib/libstdc++.so.6.0.20
7fc257fdb000-7fc257fe3000 r--p 000f0000 08:01 9178557
/usr/lib/libstdc++.so.6.0.20
7fc257fe3000-7fc257fe5000 rw-p 000f8000 08:01 9178557
/usr/lib/libstdc++.so.6.0.20
7fc257fe5000-7fc257ffa000 rw-p 00000000 00:00 0
7fc257ffa000-7fc258001000 r-xp 00000000 08:01 9178221
/usr/lib/librt-2.21.so
7fc258001000-7fc258200000 ---p 00007000 08:01 9178221
/usr/lib/librt-2.21.so
7fc258200000-7fc258201000 r--p 00006000 08:01 9178221
/usr/lib/librt-2.21.so
7fc258201000-7fc258202000 rw-p 00007000 08:01 9178221
/usr/lib/librt-2.21.so
7fc258202000-7fc258206000 r-xp 00000000 08:01 9178197
/usr/lib/libSegFault.so
7fc258206000-7fc258405000 ---p 00004000 08:01 9178197
/usr/lib/libSegFault.so
7fc258405000-7fc258406000 r--p 00003000 08:01 9178197
/usr/lib/libSegFault.so
7fc258406000-7fc258407000 rw-p 00004000 08:01 9178197
/usr/lib/libSegFault.so
7fc258407000-7fc258416000 r-xp 00000000 08:01 9181283
/usr/lib/libbz2.so.1.0.6
7fc258416000-7fc258615000 ---p 0000f000 08:01 9181283
/usr/lib/libbz2.so.1.0.6
7fc258615000-7fc258617000 rw-p 0000e000 08:01 9181283
/usr/lib/libbz2.so.1.0.6
7fc258617000-7fc25863c000 r-xp 00000000 08:01 9186133
/usr/lib/liblzma.so.5.2.1
7fc25863c000-7fc25883b000 ---p 00025000 08:01 9186133
/usr/lib/liblzma.so.5.2.1
7fc25883b000-7fc25883c000 r--p 00024000 08:01 9186133
/usr/lib/liblzma.so.5.2.1
7fc25883c000-7fc25883d000 rw-p 00025000 08:01 9186133
/usr/lib/liblzma.so.5.2.1
7fc25883d000-7fc25885f000 r-xp 00000000 08:01 9178244
/usr/lib/ld-2.21.so
7fc258a34000-7fc258a3c000 rw-p 00000000 00:00 0
7fc258a5e000-7fc258a5f000 r--p 00021000 08:01 9178244
/usr/lib/ld-2.21.so
7fc258a5f000-7fc258a60000 rw-p 00022000 08:01 9178244
/usr/lib/ld-2.21.so
7fc258a60000-7fc258a61000 rw-p 00000000 00:00 0
7ffc8391a000-7ffc8393b000 rw-p 00000000 00:00 0 [stack]
7ffc8394d000-7ffc8394f000 r--p 00000000 00:00 0 [vvar]
7ffc8394f000-7ffc83951000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
Segmentation fault (core dumped)
Best,
Valmi
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
