[ https://issues.apache.org/jira/browse/DISPATCH-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17334743#comment-17334743 ]
michael goulish commented on DISPATCH-2088: ------------------------------------------- Here you go! (gdb) thread apply all bt {color:#172b4d}Thread 33{color} (Thread 0x7fa320ff9640 (LWP 53393)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2f0000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 {color:#172b4d}Thread 32{color} (Thread 0x7fa2e8ff9640 (LWP 53408)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2a8000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 {color:#172b4d}Thread 31{color} (Thread 0x7fa2e37fe640 (LWP 53409)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2bc000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fa30effd640 (LWP 53396)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa300000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 --Type <RET> for more, q to quit, c to continue without paging-- #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fa30f7fe640 (LWP 53395)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2fc000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fa30cff9640 (LWP 53400)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2e4000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-di--Type <RET> for more, q to quit, c to continue without paging--c spatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 *{color:#de350b}Thread 27{color}* (Thread 0x7fa2eb7fe640 (LWP 53403)): #0 0x00007fa343d8350c in send () from /lib64/libpthread.so.0 #1 0x00007fa343dbe718 in snd (s=512, b=<optimized out>, fd=25) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:333 #2 pni_raw_write (send=<optimized out>, set_error=<optimized out>, sock=<optimized out>, conn=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/raw_connection.c:566 #3 pni_raw_write (send=<optimized out>, set_error=<optimized out>, sock=25, conn=0x7fa2dc129cf0) at /home/mick/latest/qpid-proton/c/src/proactor/raw_connection.c:554 #4 pni_raw_connection_process (sched_ready=<optimized out>, t=0x7fa2dc129c30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:388 #5 process (tsk=0x7fa2dc129c30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2230 #6 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2419 #7 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #8 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 {color:#de350b}*Thread 26*{color} (Thread 0x7fa2eaffd640 (LWP 53404)): #0 0x00007fa343dbafa5 in memory_barrier (ee=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:123 #1 rearm_polling (epollfd=<optimized out>, ee=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:186 #2 pni_raw_connection_done (rc=0x7fa2dc12ac70) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:413 #3 pn_proactor_done (batch=0x7fa2dc12b228, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2692 #4 pn_proactor_done (p=0xd46d30, batch=batch@entry=0x7fa2dc12b228) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2672 #5 0x00007fa343e9d081 in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1138 #6 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fa3217fa640 (LWP 53392)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa304000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fa2ebfff640 (LWP 53401)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2c0000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 *{color:#de350b}Thread 23{color}* (Thread 0x7fa2e3fff640 (LWP 53402)): #0 0x00007fa343d8350c in send () from /lib64/libpthread.so.0 #1 0x00007fa343dbe718 in snd (s=512, b=<optimized out>, fd=27) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:333 #2 pni_raw_write (send=<optimized out>, set_error=<optimized out>, sock=<optimized out>, conn=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/raw_connection.c:566 #3 pni_raw_write (send=<optimized out>, set_error=<optimized out>, sock=27, conn=0x7fa2f413a820) at /home/mick/latest/qpid-proton/c/src/proactor/raw_connection.c:554 #4 pni_raw_connection_process (sched_ready=<optimized out>, t=0x7fa2f413a760) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:388 #5 process (tsk=0x7fa2f413a760) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2230 #6 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2419 #7 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #8 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fa2ea7fc640 (LWP 53405)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2cc000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fa30d7fa640 (LWP 53399)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2c8000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fa321ffb640 (LWP 53391)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2ec000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fa2e9ffb640 (LWP 53406)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2d0000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fa323fff640 (LWP 53387)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2d4000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fa30dffb640 (LWP 53398)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2c4000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fa30ffff640 (LWP 53394)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2dc000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 {color:#de350b}*Thread 15*{color} (Thread 0x7fa330c7a640 (LWP 53386)): #0 0x00007fa343d82ea0 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fa343d7b763 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00007fa343dbd469 in lock (m=0xd47050) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:330 #3 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2407 #4 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #5 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fa322ffd640 (LWP 53389)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2f4000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fa33347f640 (LWP 53381)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa318000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fa3237fe640 (LWP 53388)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa310000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fa33247d640 (LWP 53383)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa31c000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 {color:#de350b}*Thread 10*{color} (Thread 0x7fa33147b640 (LWP 53385)): #0 0x00007fa343d7ce99 in __pthread_mutex_unlock_usercnt () from /lib64/libpthread.so.0 #1 0x00007fa343dbde0a in unlock (m=0xd47050) at /home/mick/latest/qpid-proton/c/src/proactor/epoll-internal.h:331 #2 poller_do_epoll (can_block=true, ts=0x7fa314000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2490 #3 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2438 #4 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #5 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fa332c7e640 (LWP 53382)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa328000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fa331c7c640 (LWP 53384)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2ac000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fa30e7fc640 (LWP 53397)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2b0000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fa333c80640 (LWP 53380)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa324000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fa3227fc640 (LWP 53390)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2f8000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fa2e2ffd640 (LWP 53410)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa29c000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fa2e97fa640 (LWP 53407)): #0 0x00007fa343d7f6c2 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa343dbdcbb in suspend (ts=0x7fa2a4000b60, p=0xd46d30) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:393 #2 next_event_batch (p=0xd46d30, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2455 #3 0x00007fa343e9cf9f in thread_run (arg=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #4 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 {color:#de350b}*Thread 2*{color} (Thread 0x7fa3431672c0 (LWP 53378)): #0 0x00007fa343d8350c in send () from /lib64/libpthread.so.0 #1 0x00007fa343dbe718 in snd (s=512, b=<optimized out>, fd=31) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:333 #2 pni_raw_write (send=<optimized out>, set_error=<optimized out>, sock=<optimized out>, conn=<optimized out>) at /home/mick/latest/qpid-proton/c/src/proactor/raw_connection.c:566 #3 pni_raw_write (send=<optimized out>, set_error=<optimized out>, sock=31, conn=0x7fa2f413bba0) at /home/mick/latest/qpid-proton/c/src/proactor/raw_connection.c:554 #4 pni_raw_connection_process (sched_ready=<optimized out>, t=0x7fa2f413bae0) at /home/mick/latest/qpid-proton/c/src/proactor/epoll_raw_connection.c:388 #5 process (tsk=0x7fa2f413bae0) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2230 #6 next_event_batch (p=<optimized out>, can_block=true) at /home/mick/latest/qpid-proton/c/src/proactor/epoll.c:2419 #7 0x00007fa343e9cf9f in thread_run (arg=arg@entry=0xb52c00) at /home/mick/latest/qpid-dispatch/src/server.c:1105 #8 0x00007fa343e9d730 in qd_server_run (qd=<optimized out>) at /home/mick/latest/qpid-dispatch/src/server.c:1487 #9 0x00000000004026a8 in main_process (config_path=0x7ffc80b5bd99 "./Br_1.conf", python_pkgdir=<optimized out>, test_hooks=<optimized out>, fd=2) at /home/mick/latest/qpid-dispatch/router/src/main.c:115 #10 0x000000000040240b in main (argc=3, argv=0x7ffc80b5a808) at /home/mick/latest/qpid-dispatch/router/src/main.c:369 *{color:#de350b}Thread 1{color}* (Thread 0x7fa334481640 (LWP 53379)): #0 sys_atomic_sub (value=1, ref=0x14) at /home/mick/latest/qpid-dispatch/include/qpid/dispatch/atomic.h:48 #1 sys_atomic_dec (ref=0x14) at /home/mick/latest/qpid-dispatch/include/qpid/dispatch/atomic.h:212 #2 qd_buffer_dec_fanout (buf=0x0) at /home/mick/latest/qpid-dispatch/include/qpid/dispatch/buffer.h:177 #3 qd_message_stream_data_release (stream_data=0x7fa2a817acc8) at /home/mick/latest/qpid-dispatch/src/message.c:2627 #4 0x00007fa343e48895 in flush_outgoing_buffs (conn=conn@entry=0x7fa2f411f888) at /home/mick/latest/qpid-dispatch/src/adaptors/tcp_adaptor.c:431 #5 0x00007fa343e4c05e in free_qdr_tcp_connection (tc=0x7fa2f411f888) at /home/mick/latest/qpid-dispatch/src/adaptors/tcp_adaptor.c:455 #6 0x00007fa343e8791d in router_core_thread (arg=0xdb6cb0) at /home/mick/latest/qpid-dispatch/src/router_core/router_core_thread.c:239 #7 0x00007fa343d793f9 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa3439f8b53 in clone () from /lib64/libc.so.6 (gdb) > SEGV in qd_buffer_dec_fanout > ---------------------------- > > Key: DISPATCH-2088 > URL: https://issues.apache.org/jira/browse/DISPATCH-2088 > Project: Qpid Dispatch > Issue Type: Bug > Components: Protocol Adaptors > Reporter: michael goulish > Assignee: Charles E. Rolke > Priority: Blocker > Fix For: 1.16.0 > > > *code from 2021-04-26-afternoon* > { > dispatch: (main) 22689e4f95ae1945e61eec814d3ab3e2d4259f04 > proton: (main) 08b301a97c834e002d41ee852bba1288fe83b936 > } > > *Test* > * Doing 1-router TCP throughput testing across high-bandwidth link. > * Router has 32 worker threads. > * iperf client is using "-P 10" flag, i.e. doing 10 parallel streams. > * Router is sustaining 10+ Gbit/sec during test. > * SEGV happens at end of test. > > Here's the backtrace: > > {color:#de350b}#0 sys_atomic_sub (value=1, ref=0x14){color} > {color:#de350b} at > /home/mick/latest/qpid-dispatch/include/qpid/dispatch/atomic.h:48{color} > {color:#de350b}#1 sys_atomic_dec (ref=0x14){color} > {color:#de350b} at > /home/mick/latest/qpid-dispatch/include/qpid/dispatch/atomic.h:212{color} > {color:#de350b}#2 qd_buffer_dec_fanout (buf=0x0){color} > {color:#de350b} at > /home/mick/latest/qpid-dispatch/include/qpid/dispatch/buffer.h:177{color} > {color:#de350b}#3 qd_message_stream_data_release > (stream_data=0x7f01b80038c8){color} > {color:#de350b} at /home/mick/latest/qpid-dispatch/src/message.c:2627{color} > {color:#de350b}#4 0x00007f0237035895 in flush_outgoing_buffs > (conn=conn@entry=0x7f0218012a88){color} > {color:#de350b} at > /home/mick/latest/qpid-dispatch/src/adaptors/tcp_adaptor.c:431{color} > {color:#de350b}#5 0x00007f023703905e in free_qdr_tcp_connection > (tc=0x7f0218012a88){color} > {color:#de350b} at > /home/mick/latest/qpid-dispatch/src/adaptors/tcp_adaptor.c:455{color} > {color:#de350b}#6 0x00007f023707491d in router_core_thread > (arg=0x1e6ccb0){color} > {color:#de350b} at > /home/mick/latest/qpid-dispatch/src/router_core/router_core_thread.c:239{color} > {color:#de350b}#7 0x00007f0236f663f9 in start_thread () from > /lib64/libpthread.so.0{color} > {color:#de350b}#8 0x00007f0236be5b53 in clone () from /lib64/libc.so.6{color} > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org