[ https://issues.apache.org/jira/browse/QPID-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932176#action_12932176 ]
Alan Conway commented on QPID-2874: ----------------------------------- To reproduce run make check TESTS=run_cluster_tests "CLUSTER_TESTS=*.test_management -DDURATION=4" in a loop. I've seen the failure in 2-6 iterations. Note you must use a debug build. A release build with -DNDEBUG has assertions compiled out so it will not show this problem. > Clustered broker crashes in assertion in cluster/ExpiryPolicy.cpp > ----------------------------------------------------------------- > > Key: QPID-2874 > URL: https://issues.apache.org/jira/browse/QPID-2874 > Project: Qpid > Issue Type: Bug > Components: C++ Clustering > Affects Versions: 0.6 > Reporter: Alan Conway > Assignee: Alan Conway > > From us...@qpid.apache.org: > We have a Java node connecting to one of two Qpid brokers (0.5 or 0.6, > doesn't matter, we get the same) as a client of another Java node > which acts > as the server. The client sends messages over a queue, awaiting a > response > message from the server on a temporary queue. Both brokers use > clustering > (and are part of the same cluster). At some point, after 1 to 3 > exchanges > between the client and server nodes, both Qpid brokers simply crash, > without > any error being logged. Broker logfile is attached. The typical > output that > appears in the broker log files - the last line in the log is always > the > same, so it seems that the place at which the failure occurs is > consistent. > Original message with logs and config: > http://apache-qpid-users.2158936.n2.nabble.com/Qpid-clustering-broker-crash-tt5511801.html#a5511801 > Stack trace: > [r...@ca-mtl1-tric-01 .qpidd]# gdb /opt/qpid/current/sbin/qpidd core.6709 > GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2) > Copyright (C) 2009 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "i386-redhat-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from /opt/qpid/qpid-0.5/sbin/qpidd...done. > [New Thread 6732] > [New Thread 6727] > [New Thread 6721] > [New Thread 6710] > [New Thread 6709] > warning: .dynamic section for "/usr/lib/libplc4.so" is not at the expected > address > warning: difference appears to be caused by prelink, adjusting expectations > warning: .dynamic section for "/usr/lib/libplds4.so" is not at the expected > address > warning: difference appears to be caused by prelink, adjusting expectations > warning: .dynamic section for "/usr/lib/openais/libcpg.so.2" is not at the > expected address > warning: difference appears to be caused by prelink, adjusting expectations > Reading symbols from /opt/qpid/current/lib/libqpidbroker.so.0...done. > Loaded symbols for /opt/qpid/current/lib/libqpidbroker.so.0 > Reading symbols from /opt/qpid/current/lib/libqpidcommon.so.0...done. > Loaded symbols for /opt/qpid/current/lib/libqpidcommon.so.0 > Reading symbols from /usr/lib/libboost_program_options.so.2...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib/libboost_program_options.so.2 > Reading symbols from /usr/lib/libboost_filesystem.so.2...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib/libboost_filesystem.so.2 > Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib/libuuid.so.1 > Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. > Loaded symbols for /lib/libdl.so.2 > Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib/librt.so.1 > Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libsasl2.so.2 > Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libstdc++.so.6 > Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. > Loaded symbols for /lib/libm.so.6 > Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib/libgcc_s.so.1 > Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. > Loaded symbols for /lib/libc.so.6 > Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. > Loaded symbols for /lib/ld-linux.so.2 > Reading symbols from /lib/libpthread.so.0...(no debugging symbols > found)...done. > Loaded symbols for /lib/libpthread.so.0 > Reading symbols from /lib/libresolv.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib/libresolv.so.2 > Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. > Loaded symbols for /lib/libcrypt.so.1 > Reading symbols from > /opt/qpid/current/lib/qpid/daemon/replicating_listener.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/daemon/replicating_listener.so > Reading symbols from > /opt/qpid/current/lib/qpid/daemon/replication_exchange.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/daemon/replication_exchange.so > Reading symbols from /opt/qpid/current/lib/qpid/daemon/acl.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/daemon/acl.so > Reading symbols from /opt/qpid/current/lib/qpid/daemon/ssl.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/daemon/ssl.so > Reading symbols from /opt/qpid/current/lib/libsslcommon.so.0...done. > Loaded symbols for /opt/qpid/current/lib/libsslcommon.so.0 > Reading symbols from /usr/lib/libnss3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libnss3.so > Reading symbols from /usr/lib/libssl3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libssl3.so > Reading symbols from /usr/lib/libnspr4.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libnspr4.so > Reading symbols from /usr/lib/libnssutil3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libnssutil3.so > Reading symbols from /usr/lib/libplc4.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libplc4.so > Reading symbols from /usr/lib/libplds4.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libplds4.so > Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. > Loaded symbols for /usr/lib/libz.so.1 > Reading symbols from /opt/qpid/current/lib/qpid/daemon/cluster.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/daemon/cluster.so > Reading symbols from /usr/lib/openais/libcpg.so.2...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/openais/libcpg.so.2 > Reading symbols from /usr/lib/libcman.so.2...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libcman.so.2 > Reading symbols from /opt/qpid/current/lib/libqpidclient.so.0...done. > Loaded symbols for /opt/qpid/current/lib/libqpidclient.so.0 > Reading symbols from > /opt/qpid/current/lib/qpid/client/rdmaconnector.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/client/rdmaconnector.so > Reading symbols from /opt/qpid/current/lib/librdmawrap.so.0...done. > Loaded symbols for /opt/qpid/current/lib/librdmawrap.so.0 > Reading symbols from /usr/lib/librdmacm.so.1...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/librdmacm.so.1 > Reading symbols from /usr/lib/libibverbs.so.1...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib/libibverbs.so.1 > Reading symbols from /opt/qpid/current/lib/qpid/client/sslconnector.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/client/sslconnector.so > Reading symbols from /opt/qpid/current/lib/qpid/daemon/rdma.so...done. > Loaded symbols for /opt/qpid/current/lib/qpid/daemon/rdma.so > Core was generated by `/opt/qpid/current/sbin/qpidd -p 10004 > --cluster-name=TEST_CL --no-data-dir --au'. > Program terminated with signal 6, Aborted. > #0 0x007cd402 in __kernel_vsyscall () > (gdb) thread apply all backtrace > Thread 6 (Thread 6709): > #0 0x007cd402 in __kernel_vsyscall () > #1 0x00993486 in epoll_wait () from /lib/libc.so.6 > #2 0x00475584 in qpid::sys::Poller::wait (this=0x8b70430, timeout=...) > at qpid/sys/epoll/EpollPoller.cpp:432 > #3 0x004763d3 in qpid::sys::Poller::run (this=0x8b70430) > at qpid/sys/epoll/EpollPoller.cpp:398 > #4 0x004c7ef4 in qpid::sys::Dispatcher::run (this=0xbfab0ea0) > at qpid/sys/Dispatcher.cpp:37 > #5 0x001d3f21 in qpid::broker::Broker::run (this=0x8b837b0) > at qpid/broker/Broker.cpp:319 > #6 0x08050155 in QpiddDaemon::child (this=0xbfab116c) > at posix/QpiddBroker.cpp:128 > #7 0x002140db in qpid::broker::Daemon::fork (this=0xbfab116c) > at qpid/broker/Daemon.cpp:85 > #8 0x0804ea7c in QpiddBroker::execute (this=0xbfab13f5, options=0x8b81248) > at posix/QpiddBroker.cpp:158 > #9 0x0804c8e7 in main (argc=14, argv=0xbfab14a4) at qpidd.cpp:77 > Thread 5 (Thread 6710): > #0 0x007cd402 in __kernel_vsyscall () > #1 0x00706ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > ---Type <return> to continue, or q <return> to quit--- > from /lib/libpthread.so.0 > #2 0x0099f524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 > #3 0x0029db60 in qpid::broker::Timer::run (this=0xb7fae0b0) > at ./qpid/sys/posix/Condition.h:69 > #4 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xb7fae0b0) > at qpid/sys/posix/Thread.cpp:35 > #5 0x00702832 in start_thread () from /lib/libpthread.so.0 > #6 0x00992e0e in clone () from /lib/libc.so.6 > Thread 4 (Thread 6721): > #0 0x007cd402 in __kernel_vsyscall () > #1 0x00706ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > from /lib/libpthread.so.0 > #2 0x0099f524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 > #3 0x0029db60 in qpid::broker::Timer::run (this=0x8b839b4) > at ./qpid/sys/posix/Condition.h:69 > #4 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0x8b839b4) > at qpid/sys/posix/Thread.cpp:35 > #5 0x00702832 in start_thread () from /lib/libpthread.so.0 > #6 0x00992e0e in clone () from /lib/libc.so.6 > Thread 3 (Thread 6727): > #0 0x007cd402 in __kernel_vsyscall () > ---Type <return> to continue, or q <return> to quit--- > #1 0x00706ef2 in pthread_cond_timedwait@@GLIBC_2.3.2 () > from /lib/libpthread.so.0 > #2 0x0099f524 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libc.so.6 > #3 0x0029db60 in qpid::broker::Timer::run (this=0x8b83a34) > at ./qpid/sys/posix/Condition.h:69 > #4 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0x8b83a34) > at qpid/sys/posix/Thread.cpp:35 > #5 0x00702832 in start_thread () from /lib/libpthread.so.0 > #6 0x00992e0e in clone () from /lib/libc.so.6 > Thread 2 (Thread 6732): > #0 0x007cd402 in __kernel_vsyscall () > #1 0x00993486 in epoll_wait () from /lib/libc.so.6 > #2 0x00475584 in qpid::sys::Poller::wait (this=0x8b70430, timeout=...) > at qpid/sys/epoll/EpollPoller.cpp:432 > #3 0x004763d3 in qpid::sys::Poller::run (this=0x8b70430) > at qpid/sys/epoll/EpollPoller.cpp:398 > #4 0x004c7ef4 in qpid::sys::Dispatcher::run (this=0xbfab0ea0) > at qpid/sys/Dispatcher.cpp:37 > #5 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfab0ea0) > at qpid/sys/posix/Thread.cpp:35 > #6 0x00702832 in start_thread () from /lib/libpthread.so.0 > #7 0x00992e0e in clone () from /lib/libc.so.6 > ---Type <return> to continue, or q <return> to quit--- > Thread 1 (Thread 6731): > #0 0x007cd402 in __kernel_vsyscall () > #1 0x008e9df0 in raise () from /lib/libc.so.6 > #2 0x008eb701 in abort () from /lib/libc.so.6 > #3 0x008e326b in __assert_fail () from /lib/libc.so.6 > #4 0x00f9cc46 in qpid::cluster::ExpiryPolicy::willExpire (this=0x8b87340, > m=...) at qpid/cluster/ExpiryPolicy.cpp:47 > #5 0x0024c6c4 in qpid::broker::Message::setExpiryPolicy (this=0xb4903128, > e=...) at qpid/broker/Message.cpp:338 > #6 0x0024e8bd in qpid::broker::Message::setTimestamp (this=0xb4903128, e=...) > at qpid/broker/Message.cpp:331 > #7 0x0027a419 in qpid::broker::SemanticState::route (this=0x8b755b0, msg=..., > strategy=...) at qpid/broker/SemanticState.cpp:358 > #8 0x0027b858 in qpid::broker::SemanticState::handle (this=0x8b755b0, > msg=...) > at qpid/broker/SemanticState.cpp:348 > #9 0x00292532 in qpid::broker::SessionState::handleContent (this=0x8b75488, > frame=..., id=...) at qpid/broker/SessionState.cpp:243 > #10 0x00292b23 in qpid::broker::SessionState::handleIn (this=0x8b75488, > frame=...) at qpid/broker/SessionState.cpp:321 > #11 0x0029640b in > qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, > > &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::ha---Type > <return> to continue, or q <return> to quit--- > ndle (this=0x8b75588, t=...) at ./qpid/framing/Handler.h:67 > #12 0x0049840a in qpid::amqp_0_10::SessionHandler::handleIn (this=0x8b74b78, > f=...) at qpid/amqp_0_10/SessionHandler.cpp:86 > #13 0x0029640b in > qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, > > &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn)>::handle > (this=0x8b74b80, t=...) at ./qpid/framing/Handler.h:67 > #14 0x00207fbc in operator() (this=0x8b73be4, frame=...) > at ./qpid/framing/Handler.h:42 > #15 qpid::broker::Connection::received (this=0x8b73be4, frame=...) > at qpid/broker/Connection.cpp:106 > #16 0x00f81462 in qpid::cluster::Connection::deliveredFrame (this=0x8b73b40, > f=...) at qpid/cluster/Connection.cpp:158 > #17 0x00f61b1d in qpid::cluster::Cluster::deliveredFrame (this=0x8b87470, > e=...) at qpid/cluster/Cluster.cpp:271 > #18 0x00f634fd in operator() (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/mem_fn_template.hpp:149 > #19 operator()<boost::_mfi::mf1<void, qpid::cluster::Cluster, > qpid::cluster::EventFrame const&>, > boost::_bi::list1<qpid::cluster::EventFrame const&> > ( > function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 > #20 operator()<qpid::cluster::EventFrame const> (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/bind_template.hpp:32 > #21 > boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,---Type > <return> to continue, or q <return> to quit--- > boost::_mfi::mf1<void, qpid::cluster::Cluster, qpid::cluster::EventFrame > const&>, boost::_bi::list2<boost::_bi::value<qpid::cluster::Cluster*>, > boost::arg<1> > >, void, qpid::cluster::EventFrame const&>::invoke > (function_obj_ptr=..., > a0=...) at /usr/include/boost/function/function_template.hpp:136 > #22 0x00f6a7c4 in boost::function1<void, qpid::cluster::EventFrame const&, > std::allocator<void> >::operator() (this=0x8b87908, a0=...) > at /usr/include/boost/function/function_template.hpp:576 > #23 0x00f6fb79 in > qpid::cluster::PollableQueue<qpid::cluster::EventFrame>::handleBatch > (this=0x8b87840, values=...) at qpid/cluster/PollableQueue.h:48 > #24 0x00f635fd in operator() (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/mem_fn_template.hpp:149 > #25 operator()<boost::_mfi::mf1<void, > qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, > std::deque<qpid::cluster::EventFrame, > std::allocator<qpid::cluster::EventFrame> >&>, > boost::_bi::list1<std::deque<qpid::cluster::EventFrame, > std::allocator<qpid::cluster::EventFrame> >&> > (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind.hpp:287 > #26 operator()<std::deque<qpid::cluster::EventFrame, > std::allocator<qpid::cluster::EventFrame> > > (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/bind_template.hpp:32 > #27 > boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, > boost::_mfi::mf1<void, > qpid::cluster::PollableQueue<qpid::cluster::EventFrame>, > std::deque<qpid::cluster::EventFrame, > std::allocator<qpid::cluster::EventFrame> >&>, > boost::_bi::list2<boost::_bi::value<qpid::cluster::PollableQueue<qpid::clu---Type > <return> to continue, or q <return> to quit--- > ster::EventFrame>*>, boost::arg<1> > >, void, > std::deque<qpid::cluster::EventFrame, > std::allocator<qpid::cluster::EventFrame> >&>::invoke ( > function_obj_ptr=..., a0=...) > at /usr/include/boost/function/function_template.hpp:136 > #28 0x00f6ab94 in boost::function1<void, > std::deque<qpid::cluster::EventFrame, > std::allocator<qpid::cluster::EventFrame> >&, std::allocator<void> > >::operator() > (this=0x8b87888, a0=...) > at /usr/include/boost/function/function_template.hpp:576 > #29 0x00f7327c in qpid::sys::PollableQueue<qpid::cluster::EventFrame>::process > (this=0x8b87840) at ./qpid/sys/PollableQueue.h:149 > #30 0x00f734a2 in > qpid::sys::PollableQueue<qpid::cluster::EventFrame>::dispatch > (this=0x8b87840, cond=...) at ./qpid/sys/PollableQueue.h:136 > #31 0x00f6363d in operator() (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/mem_fn_template.hpp:149 > #32 operator()<boost::_mfi::mf1<void, > qpid::sys::PollableQueue<qpid::cluster::EventFrame>, > qpid::sys::PollableCondition&>, > boost::_bi::list1<qpid::sys::PollableCondition&> > (function_obj_ptr=..., > a0=...) at /usr/include/boost/bind.hpp:287 > #33 operator()<qpid::sys::PollableCondition> (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/bind_template.hpp:32 > #34 > boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, > boost::_mfi::mf1<void, qpid::sys::PollableQueue<qpid::cluster::EventFrame>, > qpid::sys::PollableCondition&>, > boost::_bi::list2<boost::_bi::value<qpid::sys::PollableQueue<qpid::cluster::EventFrame>*>, > boost::arg<1> > >, void, qpid::sys::Poll---Type <return> to continue, or q > <return> to quit--- > ableCondition&>::invoke (function_obj_ptr=..., a0=...) > at /usr/include/boost/function/function_template.hpp:136 > #35 0x00473d67 in boost::function1<void, qpid::sys::PollableCondition&, > std::allocator<boost::function_base> >::operator() (this=0x8b88630, a0=...) > at /usr/include/boost/function/function_template.hpp:576 > #36 0x00472c37 in qpid::sys::PollableConditionPrivate::dispatch ( > this=0x8b88628) at qpid/sys/posix/PollableCondition.cpp:89 > #37 0x0047308d in operator() (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/mem_fn_template.hpp:149 > #38 operator()<boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, > qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > ( > function_obj_ptr=..., a0=...) at /usr/include/boost/bind.hpp:287 > #39 operator()<qpid::sys::DispatchHandle> (function_obj_ptr=..., a0=...) > at /usr/include/boost/bind/bind_template.hpp:32 > #40 > boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, > boost::_mfi::mf1<void, qpid::sys::PollableConditionPrivate, > qpid::sys::DispatchHandle&>, > boost::_bi::list2<boost::_bi::value<qpid::sys::PollableConditionPrivate*>, > boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke ( > function_obj_ptr=..., a0=...) > at /usr/include/boost/function/function_template.hpp:136 > #41 0x004cb547 in boost::function1<void, qpid::sys::DispatchHandle&, > std::allocator<boost::function_base> >::operator() (this=0x8b71010, a0=...) > at /usr/include/boost/function/function_template.hpp:576 > ---Type <return> to continue, or q <return> to quit--- > #42 0x004c9a48 in qpid::sys::DispatchHandle::processEvent (this=0x8b71008, > type=READABLE) at qpid/sys/DispatchHandle.cpp:428 > #43 0x004763b0 in process (this=0x8b70430) at ./qpid/sys/Poller.h:122 > #44 qpid::sys::Poller::run (this=0x8b70430) > at qpid/sys/epoll/EpollPoller.cpp:402 > #45 0x004c7ef4 in qpid::sys::Dispatcher::run (this=0xbfab0ea0) > at qpid/sys/Dispatcher.cpp:37 > #46 0x0046ced1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfab0ea0) > at qpid/sys/posix/Thread.cpp:35 > #47 0x00702832 in start_thread () from /lib/libpthread.so.0 > #48 0x00992e0e in clone () from /lib/libc.so.6 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org