[ https://issues.apache.org/jira/browse/QPID-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Stitcher resolved QPID-2256. ----------------------------------- Resolution: Fixed Fixed by removing unnecessary locking introduced by previous change for QPID-2214 > cluster_test hangs with theads deadlocked on mutex in DeletionManager. > ---------------------------------------------------------------------- > > Key: QPID-2256 > URL: https://issues.apache.org/jira/browse/QPID-2256 > Project: Qpid > Issue Type: Bug > Components: C++ Broker > Affects Versions: 0.6 > Reporter: Alan Conway > Assignee: Andrew Stitcher > Priority: Blocker > Fix For: 0.6 > > > Running cluster_test in a loop will fairly quickly result in a deadlock. The > test is blocked waiting for a child broker to exit. The broker appears > deadlocked around a mutex in DeletionManager, here are the stack traces of > the deadlocked broker: > Thread 10 (Thread 0x414b2940 (LWP 2351)): > #0 0x0000003da400af70 in pthread_cond_timedwait@@GLIBC_2.3.2 () from > /lib64/libpthread.so.0 > #1 0x00002af11703c21d in qpid::sys::Condition::wait (this=0x1accd4e0, > mut...@0x1accd4b8, absoluteti...@0x1acccae0) at > ../../cpp/include/qpid/sys/posix/Condition.h:69 > #2 0x00002af11703c48b in qpid::sys::Monitor::wait (this=0x1accd4b8, > absoluteti...@0x1acccae0) at ../../cpp/include/qpid/sys/Monitor.h:45 > #3 0x00002af1170396ac in qpid::sys::Timer::run (this=0x1accd4b0) at > ../../cpp/src/qpid/sys/Timer.cpp:139 > #4 0x00002af116f4f7cc in runRunnable (p=0x1accd4b0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #5 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #6 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 9 (Thread 0x4217c940 (LWP 2353)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x4217bd70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd4d00) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 8 (Thread 0x42b7d940 (LWP 2354)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x42b7cd70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd6160) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 7 (Thread 0x4357e940 (LWP 2355)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x1acd4bc0) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x4357da30, l...@0x1acd4bc0) at > ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5b162 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::handleAdder::operator() > (this=0x4357dad0, ptr=0x1acd4bc0) > at ../../cpp/src/qpid/sys/DeletionManager.h:128 > #6 0x00002af116f5b1d3 in > std::for_each<__gnu_cxx::__normal_iterator<qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::ThreadStatus**, > > std::vector<qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::ThreadStatus*, > > std::allocator<qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::ThreadStatus*> > > >, > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::handleAdder> > (__first={_M_current = 0x1acd6410}, __last={_M_current = 0x1acd6458}, > __f={handle = {px = 0x4357dad0, pn = {pi_ = 0x1acd6458}}}) > at > /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_algo.h:159 > #7 0x00002af116f5b268 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::addHandle > (this=0x2af11730c360, h= > {px = 0x4357db20, pn = {pi_ = 0x2af11730c360}}) at > ../../cpp/src/qpid/sys/DeletionManager.h:158 > #8 0x00002af116f5b310 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::markForDeletion > (handle=0x2aaaac001470) at ../../cpp/src/qpid/sys/DeletionManager.h:63 > #9 0x00002af116f550bf in ~PollerHandle (this=0x2aaaac0016b8) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:156 > #10 0x00002af117032938 in ~DispatchHandle (this=0x2aaaac0016b8) at > ../../cpp/src/qpid/sys/DispatchHandle.cpp:44 > #11 0x00002af116f46266 in ~AsynchIO (this=0x2aaaac0016b0) at > ../../cpp/src/qpid/sys/posix/AsynchIO.cpp:313 > #12 0x00002af11703175f in qpid::sys::DispatchHandle::processEvent > (this=0x2aaaac0016b8, type=qpid::sys::Poller::READ_WRITABLE) at > ../../cpp/src/qpid/sys/DispatchHandle.cpp:334 > #13 0x00002af116f5741d in qpid::sys::Poller::Event::process (this=0x4357deb0) > at ../../cpp/src/qpid/sys/Poller.h:123 > #14 0x00002af116f56b3e in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:483 > #15 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #16 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #17 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #18 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 6 (Thread 0x43f7f940 (LWP 2356)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x43f7ed70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd6230) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 5 (Thread 0x44980940 (LWP 2357)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x4497fd70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd62d0) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 4 (Thread 0x45381940 (LWP 2358)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x45380d70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd6320) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x45d82940 (LWP 2360)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x45d81d70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd6370) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x46783940 (LWP 2361)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x46782d70, l...@0x2af11730c360) > at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd63c0) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af116f4f7cc in runRunnable (p=0x7fff608801d0) at > ../../cpp/src/qpid/sys/posix/Thread.cpp:35 > #10 0x0000003da4006617 in start_thread () from /lib64/libpthread.so.0 > #11 0x0000003da34d3c2d in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x2af11732b940 (LWP 2335)): > #0 0x0000003da400d2e4 in __lll_lock_wait () from /lib64/libpthread.so.0 > #1 0x0000003da4008c55 in _L_lock_1127 () from /lib64/libpthread.so.0 > #2 0x0000003da4008b53 in pthread_mutex_lock () from /lib64/libpthread.so.0 > #3 0x00002af116965181 in qpid::sys::Mutex::lock (this=0x2af11730c360) at > ../../cpp/include/qpid/sys/posix/Mutex.h:116 > #4 0x00002af1169651e6 in ScopedLock (this=0x7fff6087fe80, > l...@0x2af11730c360) at ../../cpp/include/qpid/sys/Mutex.h:33 > #5 0x00002af116f5c29a in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::AllThreadsStatuses::delThreadStatus > (this=0x2af11730c360, t=0x1acd4bc0) > at ../../cpp/src/qpid/sys/DeletionManager.h:148 > #6 0x00002af116f5c340 in > qpid::sys::DeletionManager<qpid::sys::PollerHandlePrivate>::destroyThreadState > () at ../../cpp/src/qpid/sys/DeletionManager.h:83 > #7 0x00002af116f56b52 in qpid::sys::Poller::run (this=0x1acc9b50) at > ../../cpp/src/qpid/sys/epoll/EpollPoller.cpp:488 > #8 0x00002af1170379c2 in qpid::sys::Dispatcher::run (this=0x7fff608801d0) at > ../../cpp/src/qpid/sys/Dispatcher.cpp:37 > #9 0x00002af1169a8684 in qpid::broker::Broker::run (this=0x1accd460) at > ../../cpp/src/qpid/broker/Broker.cpp:318 > #10 0x000000000040ae56 in QpiddBroker::execute (this=0x7fff6088077f, > options=0x1acc6000) at ../../cpp/src/posix/QpiddBroker.cpp:176 > #11 0x0000000000409231 in main (argc=13, argv=0x7fff60880a38) at > ../../cpp/src/qpidd.cpp:80 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org