Hi Ellie
Quoting ellie timoney <el...@fastmail.com>:
Hi,
Are you confident that the backtrace in your earlier email is from
the thread that crashed?
no, as I had no experience in debugging multithreaded programs before,
but looking at all threads of multiple core dumps DH_free is a
common pattern.
mupdate is multithreaded, so obtaining the correct backtrace from a
crash is a bit of work (but maybe you already did this)
I used Redhats abrtd to collect the coredumps. This program also collects some
other information and produces an core_backtrace json output and
groups similar
coredumps together. I can provide these if anyone is interested, but they
are to big to mail.
It's been a while since I've done any multithreaded debugging with
gcc (since Cyrus is predominantly single-threaded), but my fingers
remember using "thread apply all bt" and going from there, if that's
any help :)
I have attached the output of "thread apply all bt" form three different
core dumps.
On Thu, Jul 5, 2018, at 7:40 PM, Michael Menge wrote:
Quoting Michael Menge <michael.me...@zdv.uni-tuebingen.de>:
> Hi,
>
> we are in the process of setting up our new production mailserver with
> cyrus-imapd 3.0.7 on RHEL 7.5 Servers.
>
> At the moment we encounter many crashes (SIGSEGV) of the mupdate
process on
> the mupdate master instance. As soon as we issue a command that updates
> multiple mailboxes in short time we trigger a SIGSEGV.
>
>> sam user/zrstes* cyrus all
> Setting ACL on user/zrstes1...OK.
> Setting ACL on user/zrstes1/Mail...OK.
> Setting ACL on user/zrstes1/Mail/drafts...OK.
> Setting ACL on user/zrstes1/Mail/s-spam...OK.
> Setting ACL on user/zrstes1/Mail/sent...OK.
> Setting ACL on user/zrstes1/Mail/trash...OK.
> Setting ACL on user/zrstes1/Mail/v-spam...OK.
> Setting ACL on user/zrstes2...cyrus: lrswipkxtea: no connection to server
>
> I suspect that at this time the connection from the backend to the
> mupdate master
> is lost as the mupdate process received the SIGSEGV
>
> (gdb) bt
> #0 0x00007fb49c000098 in ?? ()
> #1 0x00007fb4a9a59a85 in DH_free (r=0x7fb49c1b9600) at dh_lib.c:194
> #2 0x00007fb4aa84dbef in tls_shutdown_serverengine () at imap/tls.c:1311
> #3 0x0000000000404075 in conn_free (C=0x7fb4840009f0) at
imap/mupdate.c:379
> #4 0x00000000004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
> #5 0x00007fb4a771add5 in start_thread (arg=0x7fb4a22d9700) at
> pthread_create.c:308
> #6 0x00007fb4a718fb3d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
>
>
> Further information, as DH_free is in the bt, we have NOT configured
> dh_params in
> our ssl certificate/key or in the imapd.conf
>
As a follow up, we did generate dh_params, but the mupdate still crashes
with SIGSEGV. Any ideas how to debug/fix this? Is there any
information that is missing?
Regards
Michael Menge
--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
michael.me...@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/libexec/mupdate...done.
[New LWP 10478]
[New LWP 10482]
[New LWP 10481]
[New LWP 10479]
[New LWP 10480]
[New LWP 10476]
[New LWP 10477]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `mupdate -m -C /etc/imapd_mu.conf'.
Program terminated with signal 11, Segmentation fault.
#0 OPENSSL_cleanse () at x86_64cpuid.s:192
192 movq %rax,(%rdi)
(gdb)
Thread 7 (Thread 0x7fc98341f700 (LWP 10477)):
#0 0x00007fc986062a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1 0x00000000004098fa in mupdate_placebo_kick_start (rock=0x0) at
imap/mupdate-slave.c:351
#2 0x00007fc98605bdd5 in start_thread (arg=0x7fc98341f700) at
pthread_create.c:308
#3 0x00007fc985ad0b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x7fc98b43c8c0 (LWP 10476)):
#0 0x00007fc986062a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1 0x000000000040a232 in main (argc=4, argv=0x7ffc0cfcfd28,
envp=0x7ffc0cfcfd50) at master/service-thread.c:268
Thread 5 (Thread 0x7fc97bfff700 (LWP 10480)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007fc98605bdd5 in start_thread (arg=0x7fc97bfff700) at
pthread_create.c:308
#3 0x00007fc985ad0b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x7fc98241d700 (LWP 10479)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007fc98605bdd5 in start_thread (arg=0x7fc98241d700) at
pthread_create.c:308
#3 0x00007fc985ad0b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x7fc981c1c700 (LWP 10481)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007fc98605bdd5 in start_thread (arg=0x7fc981c1c700) at
pthread_create.c:308
#3 0x00007fc985ad0b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x7fc98141b700 (LWP 10482)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007fc98605bdd5 in start_thread (arg=0x7fc98141b700) at
pthread_create.c:308
#3 0x00007fc985ad0b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x7fc982c1e700 (LWP 10478)):
#0 OPENSSL_cleanse () at x86_64cpuid.s:192
#1 0x00007fc988357993 in BN_clear_free (a=0x7fc974000078) at bn_lib.c:235
#2 0x00007fc98839aab5 in DH_free (r=0x7fc9741b9600) at dh_lib.c:203
#3 0x00007fc98918ebef in tls_shutdown_serverengine () at imap/tls.c:1311
#4 0x0000000000404075 in conn_free (C=0x7fc970001520) at imap/mupdate.c:379
#5 0x00000000004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
#6 0x00007fc98605bdd5 in start_thread (arg=0x7fc982c1e700) at
pthread_create.c:308
#7 0x00007fc985ad0b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) quit
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/libexec/mupdate...done.
[New LWP 20804]
[New LWP 20803]
[New LWP 20801]
[New LWP 20800]
[New LWP 20799]
[New LWP 20802]
[New LWP 20798]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `mupdate -m -C /etc/imapd_mu.conf'.
Program terminated with signal 11, Segmentation fault.
#0 BN_clear_free (a=0x6120646e6966206f) at bn_lib.c:234
234 if (a->d != NULL) {
(gdb)
Thread 7 (Thread 0x7f58bbaf08c0 (LWP 20798)):
#0 0x00007f58b6716a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1 0x000000000040a232 in main (argc=4, argv=0x7ffd75303d68,
envp=0x7ffd75303d90) at master/service-thread.c:268
Thread 6 (Thread 0x7f58b22d0700 (LWP 20802)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007f58b670fdd5 in start_thread (arg=0x7f58b22d0700) at
pthread_create.c:308
#3 0x00007f58b6184b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 5 (Thread 0x7f58b3ad3700 (LWP 20799)):
#0 0x00007f58b6716a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1 0x00000000004098fa in mupdate_placebo_kick_start (rock=0x0) at
imap/mupdate-slave.c:351
#2 0x00007f58b670fdd5 in start_thread (arg=0x7f58b3ad3700) at
pthread_create.c:308
#3 0x00007f58b6184b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x7f58b32d2700 (LWP 20800)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007f58b670fdd5 in start_thread (arg=0x7f58b32d2700) at
pthread_create.c:308
#3 0x00007f58b6184b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x7f58b2ad1700 (LWP 20801)):
#0 0x00007f58b617bcd9 in __pselect (nfds=15, readfds=0x7f58b2ad0bf0,
writefds=0x0, exceptfds=0x0, timeout=<optimized out>, sigmask=<optimized out>)
at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007f58b951cfc2 in signals_select (nfds=15, rfds=0x7f58b2ad0bf0,
wfds=0x0, efds=0x0, tout=0x7f58b2ad0be0) at lib/signals.c:257
#2 0x00007f58b951b483 in prot_select (readstreams=0x7f58a40008c0,
extra_read_fd=10, out=0x7f58b2ad0d28, extra_read_flag=0x7f58b2ad0cfc,
timeout=0x7f58b2ad0be0) at lib/prot.c:1578
#3 0x0000000000405e6e in thread_main (rock=0x0) at imap/mupdate.c:1199
#4 0x00007f58b670fdd5 in start_thread (arg=0x7f58b2ad1700) at
pthread_create.c:308
#5 0x00007f58b6184b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x7f58b1acf700 (LWP 20803)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007f58b670fdd5 in start_thread (arg=0x7f58b1acf700) at
pthread_create.c:308
#3 0x00007f58b6184b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x7f58b12ce700 (LWP 20804)):
#0 BN_clear_free (a=0x6120646e6966206f) at bn_lib.c:234
#1 0x00007f58b8a4eab5 in DH_free (r=0x7f58a41be720) at dh_lib.c:203
#2 0x00007f58b9842bef in tls_shutdown_serverengine () at imap/tls.c:1311
#3 0x0000000000404075 in conn_free (C=0x7f58a4113a10) at imap/mupdate.c:379
#4 0x00000000004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
#5 0x00007f58b670fdd5 in start_thread (arg=0x7f58b12ce700) at
pthread_create.c:308
#6 0x00007f58b6184b3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) quit
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/libexec/mupdate...done.
[New LWP 31410]
[New LWP 31407]
[New LWP 31409]
[New LWP 31405]
[New LWP 31404]
[New LWP 31406]
[New LWP 31408]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `mupdate -m -C /etc/imapd_mu.conf'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fb49c000098 in ?? ()
(gdb)
Thread 7 (Thread 0x7fb4a32db700 (LWP 31408)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007fb4a771add5 in start_thread (arg=0x7fb4a32db700) at
pthread_create.c:308
#3 0x00007fb4a718fb3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x7fb4a42dd700 (LWP 31406)):
#0 0x00007fb4a7186cd9 in __pselect (nfds=11, readfds=0x7fb4a42dcbf0,
writefds=0x0, exceptfds=0x0, timeout=<optimized out>, sigmask=<optimized out>)
at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007fb4aa527fc2 in signals_select (nfds=11, rfds=0x7fb4a42dcbf0,
wfds=0x0, efds=0x0, tout=0x0) at lib/signals.c:257
#2 0x00007fb4aa526483 in prot_select (readstreams=0x7fb49c0008c0,
extra_read_fd=10, out=0x7fb4a42dcd28, extra_read_flag=0x7fb4a42dccfc,
timeout=0x0) at lib/prot.c:1578
#3 0x0000000000405e6e in thread_main (rock=0x0) at imap/mupdate.c:1199
#4 0x00007fb4a771add5 in start_thread (arg=0x7fb4a42dd700) at
pthread_create.c:308
#5 0x00007fb4a718fb3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 5 (Thread 0x7fb4acafb8c0 (LWP 31404)):
#0 0x00007fb4a7721a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1 0x000000000040a232 in main (argc=4, argv=0x7fff12ca3e38,
envp=0x7fff12ca3e60) at master/service-thread.c:268
Thread 4 (Thread 0x7fb4a4ade700 (LWP 31405)):
#0 0x00007fb4a7721a2d in accept () at ../sysdeps/unix/syscall-template.S:81
#1 0x00000000004098fa in mupdate_placebo_kick_start (rock=0x0) at
imap/mupdate-slave.c:351
#2 0x00007fb4a771add5 in start_thread (arg=0x7fb4a4ade700) at
pthread_create.c:308
#3 0x00007fb4a718fb3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 3 (Thread 0x7fb4a2ada700 (LWP 31409)):
#0 0x00007fb4a7186cd9 in __pselect (nfds=16, readfds=0x7fb4a2ad9b50,
writefds=0x0, exceptfds=0x0, timeout=<optimized out>, sigmask=<optimized out>)
at ../sysdeps/unix/sysv/linux/pselect.c:77
#1 0x00007fb4aa527fc2 in signals_select (nfds=16, rfds=0x7fb4a2ad9b50,
wfds=0x0, efds=0x0, tout=0x7fb4a2ad9bd0) at lib/signals.c:257
#2 0x00007fb4aa52429e in prot_fill (s=0x7fb4940013f0) at lib/prot.c:730
#3 0x00007fb4aa7f52f8 in getword (in=0x7fb4940013f0, buf=0x7fb494001338) at
imap/imapparse.c:70
#4 0x00000000004048ad in docmd (c=0x7fb4940009f0) at imap/mupdate.c:677
#5 0x00000000004062af in thread_main (rock=0x0) at imap/mupdate.c:1329
#6 0x00007fb4a771add5 in start_thread (arg=0x7fb4a2ada700) at
pthread_create.c:308
#7 0x00007fb4a718fb3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x7fb4a3adc700 (LWP 31407)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at
../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1 0x0000000000405cd4 in thread_main (rock=0x0) at imap/mupdate.c:1142
#2 0x00007fb4a771add5 in start_thread (arg=0x7fb4a3adc700) at
pthread_create.c:308
#3 0x00007fb4a718fb3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x7fb4a22d9700 (LWP 31410)):
#0 0x00007fb49c000098 in ?? ()
#1 0x00007fb4a9a59a85 in DH_free (r=0x7fb49c1b9600) at dh_lib.c:194
#2 0x00007fb4aa84dbef in tls_shutdown_serverengine () at imap/tls.c:1311
#3 0x0000000000404075 in conn_free (C=0x7fb4840009f0) at imap/mupdate.c:379
#4 0x00000000004062c0 in thread_main (rock=0x0) at imap/mupdate.c:1330
#5 0x00007fb4a771add5 in start_thread (arg=0x7fb4a22d9700) at
pthread_create.c:308
#6 0x00007fb4a718fb3d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) quit
