[
https://issues.apache.org/jira/browse/QPID-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gordon Sim reassigned QPID-5793:
--------------------------------
Assignee: Gordon Sim
> [C++ broker] Memory leak _and_ deadlock due to qpid.auto_delete_timeout queues
> ------------------------------------------------------------------------------
>
> Key: QPID-5793
> URL: https://issues.apache.org/jira/browse/QPID-5793
> Project: Qpid
> Issue Type: Bug
> Components: C++ Broker
> Affects Versions: 0.26
> Reporter: Pavel Moravec
> Assignee: Gordon Sim
>
> Description of problem:
> There is a memory leak in the broker when a client declares auto-delete queue
> with qpid.auto_delete_timeout and re-connects just at the time the broker is
> about to delete the queue.
> Version-Release number of selected component (if applicable):
> any
> How reproducible:
> 100% within some time
> Steps to Reproduce:
> 1) Run script:
> queues=100
> sleep=0
> while true; do
> for i in $(seq 1 $queues); do
> qpid-receive -a "autoDelQueue_${i}; {create:always, node:{
> x-declare:{auto-delete:True, arguments:{'qpid.auto_delete_timeout':1}},
> x-bindings:[{exchange:'amq.fanout', queue:autoDelQueue_${i}}] }}"
> --print-content=no -m 10 &
> done
> wait
> sleep $sleep
> date
> done
> Modify parameters (esp. sleep) such that the loop is executed once every
> second.
> 2) Monitor qpid broker memory utilization over time.
> Actual results:
> Both VSZ and RSS of qpidd process grows.
> Expected results:
> No memory grow.
> Additional info:
> the memory consumption increase is followed after a while by deadlock (see
> below) that - depending on number of worker threads - can practically freeze
> the broker. Backtraces of relevant threads:
> Thread 4 (Thread 0x7fc08e803700 (LWP 26576)):
> #0 0x00000032c4c0e264 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00000032c4c09523 in _L_lock_892 () from /lib64/libpthread.so.0
> #2 0x00000032c4c09407 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x00007fc08f2e4842 in qpid::sys::Mutex::lock() ()
> #4 0x00007fc08f2e4900 in
> qpid::sys::ScopedLock<qpid::sys::Mutex>::ScopedLock(qpid::sys::Mutex&) ()
> #5 0x00007fc08f38bec5 in qpid::broker::Queue::canAutoDelete() const ()
> #6 0x00007fc08f39d0bd in bool boost::_mfi::cmf0<bool,
> qpid::broker::Queue>::call<boost::shared_ptr<qpid::broker::Queue>
> const>(boost::shared_ptr<qpid::broker::Queue> const&, void const*) const ()
> #7 0x00007fc08f39c2ef in bool boost::_mfi::cmf0<bool,
> qpid::broker::Queue>::operator()<boost::shared_ptr<qpid::broker::Queue>
> >(boost::shared_ptr<qpid::broker::Queue> const&) const ()
> #8 0x00007fc08f39b117 in bool
> boost::_bi::list1<boost::_bi::value<boost::shared_ptr<qpid::broker::Queue> >
> >::operator()<bool, boost::_mfi::cmf0<bool, qpid::broker::Queue>,
> boost::_bi::list0>(boost::_bi::type<bool>, boost::_mfi::cmf0<bool,
> qpid::broker::Queue>&, boost::_bi::list0&, long) ()
> #9 0x00007fc08f39974b in boost::_bi::bind_t<boost::_bi::unspecified,
> boost::_mfi::cmf0<bool, qpid::broker::Queue>,
> boost::_bi::list1<boost::_bi::value<boost::shared_ptr<qpid::broker::Queue> >
> > >::operator()() ()
> #10 0x00007fc08f397593 in bool
> qpid::broker::QueueRegistry::destroyIf<boost::_bi::bind_t<boost::_bi::unspecified,
> boost::_mfi::cmf0<bool, qpid::broker::Queue>,
> boost::_bi::list1<boost::_bi::value<boost::shared_ptr<qpid::broker::Queue> >
> > > >(std::basic_string<char, std::char_traits<char>, std::allocator<char> >
> const&, boost::_bi::bind_t<boost::_bi::unspecified, boost::_mfi::cmf0<bool,
> qpid::broker::Queue>,
> boost::_bi::list1<boost::_bi::value<boost::shared_ptr<qpid::broker::Queue> >
> > >) ()
> #11 0x00007fc08f38ed3e in qpid::broker::Queue::tryAutoDelete() ()
> #12 0x00007fc08f395a78 in qpid::broker::AutoDeleteTask::fire() ()
> #13 0x00007fc08ecb9847 in qpid::sys::TimerTask::fireTask() ()
> #14 0x00007fc08ecbaaa1 in
> qpid::sys::Timer::fire(boost::intrusive_ptr<qpid::sys::TimerTask>) ()
> #15 0x00007fc08ecba3d5 in qpid::sys::Timer::run() ()
> #16 0x00007fc08ec3b597 in qpid::sys::(anonymous
> namespace)::runRunnable(void*) ()
> #17 0x00000032c4c079d1 in start_thread () from /lib64/libpthread.so.0
> #18 0x00000032c44e8b6d in clone () from /lib64/libc.so.6
> Thread 3 (Thread 0x7fc08cdcd700 (LWP 26577)):
> #0 0x00000032c4c0e264 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1 0x00000032c4c09523 in _L_lock_892 () from /lib64/libpthread.so.0
> #2 0x00000032c4c09407 in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3 0x00007fc08f2e4842 in qpid::sys::Mutex::lock() ()
> #4 0x00007fc08f2e4900 in
> qpid::sys::ScopedLock<qpid::sys::Mutex>::ScopedLock(qpid::sys::Mutex&) ()
> #5 0x00007fc08f38f3a3 in qpid::broker::Queue::hasExclusiveOwner() const ()
> #6 0x00007fc08f346459 in qpid::broker::Broker::bind(std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >
> const&, std::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, qpid::framing::FieldTable const&, qpid::broker::OwnershipToken
> const*, std::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&) ()
> #7 0x00007fc08f462d55 in
> qpid::broker::SessionAdapter::ExchangeHandlerImpl::bind(std::basic_string<char,
> std::char_traits<char>, std::allocator<char> > const&,
> std::basic_string<char, std::char_traits<char>, std::allocator<char> >
> const&, std::basic_string<char, std::char_traits<char>, std::allocator<char>
> > const&, qpid::framing::FieldTable const&) ()
> #8 0x00007fc08ebd15d5 in void
> qpid::framing::ExchangeBindBody::invoke<qpid::framing::AMQP_ServerOperations::ExchangeHandler>(qpid::framing::AMQP_ServerOperations::ExchangeHandler&)
> const ()
> #9 0x00007fc08ebcf1f5 in
> qpid::framing::AMQP_ServerOperations::ExchangeHandler::Invoker::visit(qpid::framing::ExchangeBindBody
> const&) ()
> #10 0x00007fc08ebdcb7c in
> qpid::framing::ExchangeBindBody::accept(qpid::framing::MethodBodyConstVisitor&)
> const ()
> #11 0x00007fc08ebcd5a2 in
> qpid::framing::AMQP_ServerOperations::Invoker::visit(qpid::framing::ExchangeBindBody
> const&) ()
> #12 0x00007fc08ebdcb7c in
> qpid::framing::ExchangeBindBody::accept(qpid::framing::MethodBodyConstVisitor&)
> const ()
> #13 0x00007fc08f475545 in qpid::framing::Invoker::Result
> qpid::framing::invoke<qpid::broker::SessionAdapter>(qpid::broker::SessionAdapter&,
> qpid::framing::AMQMethodBody const&) ()
> #14 0x00007fc08f47152f in
> qpid::broker::SessionState::handleCommand(qpid::framing::AMQMethodBody*) ()
> #15 0x00007fc08f4722ba in
> qpid::broker::SessionState::handleIn(qpid::framing::AMQFrame&) ()
> #16 0x00007fc08f47a0a5 in
> qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface,
>
> &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle(qpid::framing::AMQFrame&)
> () #17 0x00007fc08ec7b6c7 in
> qpid::amqp_0_10::SessionHandler::handleIn(qpid::framing::AMQFrame&) ()
> #18 0x00007fc08f47a0a5 in
> qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface,
>
> &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle(qpid::framing::AMQFrame&)
> () #19 0x00007fc08f3ce456 in
> qpid::framing::Handler<qpid::framing::AMQFrame&>::operator()(qpid::framing::AMQFrame&)
> ()
> #20 0x00007fc08f3ca064 in
> qpid::broker::ConnectionHandler::handle(qpid::framing::AMQFrame&) ()
> #21 0x00007fc08f3c0a2e in
> qpid::broker::amqp_0_10::Connection::received(qpid::framing::AMQFrame&) ()
> #22 0x00007fc08f329780 in qpid::amqp_0_10::Connection::decode(char const*,
> unsigned long) ()
> #23 0x00007fc08f43b015 in qpid::broker::SecureConnection::decode(char const*,
> unsigned long) ()
> #24 0x00007fc08ecb031b in
> qpid::sys::AsynchIOHandler::readbuff(qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*) ()
> #25 0x00007fc08f4d8534 in boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler,
> qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>::operator()(qpid::sys::AsynchIOHandler*,
> qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*) const ()
> #26 0x00007fc08f4d7272 in void
> boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>,
> boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void,
> qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*&> >(boost::_bi::type<void>,
> boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>&, boost::_bi::list2<qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*&>&, int) ()
> #27 0x00007fc08f4d6704 in void boost::_bi::bind_t<void,
> boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>,
> boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>,
> boost::arg<1>, boost::arg<2> > >::operator---Type <return> to continue, or q
> <return> to quit---
> ()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*>(qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*&) ()
> #28 0x00007fc08f4d59dd in
> boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void,
> boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>,
> boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>,
> boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>::invoke(boost::detail::function::function_buffer&,
> qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*) ()
> #29 0x00007fc08ec233dc in boost::function2<void, qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*>::operator()(qpid::sys::AsynchIO&,
> qpid::sys::AsynchIOBufferBase*) const ()
> #30 0x00007fc08ec204cc in
> qpid::sys::posix::AsynchIO::readable(qpid::sys::DispatchHandle&) ()
> #31 0x00007fc08ec283da in boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
> qpid::sys::DispatchHandle&>::operator()(qpid::sys::posix::AsynchIO*,
> qpid::sys::DispatchHandle&) const ()
> #32 0x00007fc08ec2772b in void
> boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>,
> boost::arg<1> >::operator()<boost::_mfi::mf1<void,
> qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>,
> boost::_bi::list1<qpid::sys::DispatchHandle&> >(boost::_bi::type<void>,
> boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
> qpid::sys::DispatchHandle&>&, boost::_bi::list1<qpid::sys::DispatchHandle&>&,
> int) ()
> #33 0x00007fc08ec26a1c in void boost::_bi::bind_t<void,
> boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
> qpid::sys::DispatchHandle&>,
> boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>,
> boost::arg<1> >
> >::operator()<qpid::sys::DispatchHandle>(qpid::sys::DispatchHandle&) ()
> #34 0x00007fc08ec259f9 in
> boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
> boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO,
> qpid::sys::DispatchHandle&>,
> boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>,
> boost::arg<1> > >, void,
> qpid::sys::DispatchHandle&>::invoke(boost::detail::function::function_buffer&,
> qpid::sys::DispatchHandle&) ()
> #35 0x00007fc08ecb4332 in boost::function1<void,
> qpid::sys::DispatchHandle&>::operator()(qpid::sys::DispatchHandle&) const ()
> #36 0x00007fc08ecb38f8 in
> qpid::sys::DispatchHandle::processEvent(qpid::sys::Poller::EventType) ()
> #37 0x00007fc08ec48692 in qpid::sys::Poller::Event::process() ()
> #38 0x00007fc08ec47990 in qpid::sys::Poller::run() ()
> #39 0x00007fc08ecb24d9 in qpid::sys::Dispatcher::run() ()
> #40 0x00007fc08ec3b597 in qpid::sys::(anonymous
> namespace)::runRunnable(void*) ()
> #41 0x00000032c4c079d1 in start_thread () from /lib64/libpthread.so.0
> #42 0x00000032c44e8b6d in clone () from /lib64/libc.so.6
> (I saw variants of Thread 3 waiting on the lock acquire, e.g. within:
> #3 0x00007fc08ecb9c00 in qpid::sys::TimerTask::cancel() ()
> #4 0x00007fc08f389cb8 in qpid::broker::Queue::consume(..)
> )
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
