After around 24 hours, backtrace is the same: (gdb) thr apply all bt
Thread 5 (Thread 1106889024 (LWP 9361)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized out>, ptr=<value optimized out>) at tools.c:1124 #2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 #3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6 Thread 4 (Thread 1115281728 (LWP 9362)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>, ptr=<value optimized out>) at tools.c:1147 #2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 #3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6 Thread 3 (Thread 1123674432 (LWP 9363)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>, ptr=<value optimized out>) at tools.c:1147 #2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at ../../../libraries/libldap_r/tpool.c:663 #3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 #4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6 Thread 2 (Thread 1132067136 (LWP 9360)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value optimized out>, e=0x2b7b521d2068) at tools.c:475 #2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545 #3 0x00000000004906f9 in slapadd (argc=<value optimized out>, argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 #4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at ../../../servers/slapd/main.c:391 Thread 1 (Thread 47794824997056 (LWP 9360)): #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value optimized out>, e=0x2b7b521d2068) at tools.c:475 #2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545 #3 0x00000000004906f9 in slapadd (argc=<value optimized out>, argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 #4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at ../../../servers/slapd/main.c:391 #0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 --On Sunday, March 22, 2009 2:37 AM +0000 [email protected] wrote: > --On Saturday, March 21, 2009 7:09 PM -0700 Howard Chu <[email protected]> > wrote: > >> [email protected] wrote: >>> Full_Name: Quanah Gibson-Mount >>> Version: RE24 3/19/2009 >>> OS: Linux 2.6 >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (75.111.29.239) >>> >>> >>> While testing various load scenarios, slapadd deadlocked on me. >> >> The trace for thread 1 and 2 look identical, so I'm assuming they're the >> same thread. In that case I see 3 tool threads running. How many did you >> have configured? If 1 and 2 are really different threads (certainly their >> thread IDs look quite different), then there's something very wrong in >> the thread library... >> >> print bdb_tool_index_tcount >> > > (gdb) print bdb_tool_index_tcount > $1 = 1 > > >> It looks like one thread has exited early. I'm not sure how that can >> happen since the only way for any index thread to exit is if >> slapd_shutdown is set. Is it set? > > (gdb) print slapd_shutdown > $2 = 0 > > > It's configured with 4 tool threads. > > --Quanah > >>> (gdb) thr apply all bt >>> >>> Thread 5 (Thread 1106889024 (LWP 9361)): >>> # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> # 1 0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized >>> # out>, >>> ptr=<value optimized out>) at tools.c:1124 >>> # 2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper >>> # (xpool=0x17e07fd0) at >>> ../../../libraries/libldap_r/tpool.c:663 >>> # 3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 >>> # 4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6 >>> >>> Thread 4 (Thread 1115281728 (LWP 9362)): >>> # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> # 1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized >>> # out>, >>> ptr=<value optimized out>) at tools.c:1147 >>> # 2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper >>> # (xpool=0x17e07fd0) at >>> ../../../libraries/libldap_r/tpool.c:663 >>> # 3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 >>> # 4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6 >>> >>> Thread 3 (Thread 1123674432 (LWP 9363)): >>> # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> # 1 0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized >>> # out>, >>> ptr=<value optimized out>) at tools.c:1147 >>> # 2 0x00002b781715a2ca in ldap_int_thread_pool_wrapper >>> # (xpool=0x17e07fd0) at >>> ../../../libraries/libldap_r/tpool.c:663 >>> # 3 0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0 >>> # 4 0x000000395e0d1e3d in clone () from /lib64/libc.so.6 >>> >>> Thread 2 (Thread 1132067136 (LWP 9360)): >>> # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> # 1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, >>> # txn=<value >>> optimized out>, e=0x2b7b521d2068) at tools.c:475 >>> # 2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, >>> # e=0x2b7b521d2068, >>> text=0x7fff93960cc0) at tools.c:545 >>> # 3 0x00000000004906f9 in slapadd (argc=<value optimized out>, >>> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 >>> # 4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at >>> ../../../servers/slapd/main.c:391 >>> >>> Thread 1 (Thread 47794824997056 (LWP 9360)): >>> # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >>> # 1 0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, >>> # txn=<value >>> optimized out>, e=0x2b7b521d2068) at tools.c:475 >>> # 2 0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, >>> # e=0x2b7b521d2068, >>> text=0x7fff93960cc0) at tools.c:545 >>> # 3 0x00000000004906f9 in slapadd (argc=<value optimized out>, >>> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350 >>> # 4 0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at >>> ../../../servers/slapd/main.c:391 >>> # 0 0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib64/libpthread.so.0 >> >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > > > > -- > > Quanah Gibson-Mount > Principal Software Engineer > Zimbra, Inc > -------------------- > Zimbra :: the leader in open source messaging and collaboration > > -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
