broker crash using rdma+sasl
----------------------------
Key: QPID-3174
URL: https://issues.apache.org/jira/browse/QPID-3174
Project: Qpid
Issue Type: Bug
Components: C++ Broker
Reporter: michael j. goulish
This is sometimes very hard to observe. I was lucky enough to see it three
times in the space of 500 iterations yesterday, but now, with same code &
scripts & system, I have no repetition after 6000 iterations.
my tree rev number is 1084895
Here is the backtrace -- all 3 have been identical:
#0 0x0000003f8d030265 in raise (sig=<value optimized out>)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x0000003f8d031d10 in abort () at abort.c:88
#2 0x0000003f8d0296e6 in __assert_fail (
assertion=0x2b28f90400aa "completionsNeeded.get() > 0",
file=0x2b28f903fe58 "./qpid/broker/AsyncCompletion.h", line=167,
function=0x2b28f9040440 "void
qpid::broker::AsyncCompletion::end(qpid::broker::AsyncCompletion::Callback&)")
at assert.c:78
#3 0x00002b28f8fe2935 in qpid::broker::AsyncCompletion::end
(this=0x2aaab60c2e50,
cb=<value optimized out>) at ./qpid/broker/AsyncCompletion.h:167
#4 0x00002b28f8fdf3f3 in qpid::broker::SessionState::handleContent (
this=0x2aaabc4157d0, frame=..., id=<value optimized out>)
at qpid/broker/SessionState.cpp:265
#5 0x00002b28f8fdf830 in qpid::broker::SessionState::handleIn
(this=0x2aaabc4157d0,
frame=...) at qpid/broker/SessionState.cpp:355
#6 0x00002b28f94799dc in qpid::amqp_0_10::SessionHandler::handleIn (
this=0x2aaab0025680, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#7 0x00002b28f8f2622b in operator() (this=0x2aaab38cffa0, frame=...)
at ./qpid/framing/Handler.h:42
#8 qpid::broker::Connection::received (this=0x2aaab38cffa0, frame=...)
at qpid/broker/Connection.cpp:164
#9 0x00002b28f8ef8300 in qpid::amqp_0_10::Connection::decode
(this=0x2aaabc47d240,
buffer=<value optimized out>, size=<value optimized out>)
at qpid/amqp_0_10/Connection.cpp:58
#10 0x00002b28f94c25e5 in qpid::sys::cyrus::CyrusSecurityLayer::decode (
this=0x2aaabc468480, input=0x2aaab2aa97d0 "", size=118)
at qpid/sys/cyrus/CyrusSecurityLayer.cpp:59
#11 0x00002b28f999c7d8 in qpid::sys::RdmaIOHandler::readbuff
(this=0x2aaab002d9e0,
buff=0xffffffffffffffff) at qpid/sys/RdmaIOPlugin.cpp:218
#12 0x00002b28f9bba75a in boost::function2<void, Rdma::AsynchIO&,
Rdma::Buffer*, std::allocator<boost::function_base> >::operator() (this=0x0,
a0=..., a1=0x6)
here is my broker start script:
-------------- start script -----------------------
#! /bin/bash
export LD_LIBRARY_PATH=$TRUNK/qpid/cpp/src/.libs
QPID_SRC=$TRUNK/qpid/cpp/src
QPIDD=${QPID_SRC}/.libs/qpidd
echo $QPIDD
rm -rf /tmp/mick
mkdir /tmp/mick
$QPIDD \
--no-module-dir \
--load-module ${QPID_SRC}/.libs/rdma.so \
--data-dir /tmp/mick/data_1 \
--auth=yes \
--mgmt-enable=yes \
--port 5813 \
--log-enable info+ \
--log-to-file /tmp/mick/qpidd_1.log \
--log-source yes \
--sasl-config=${QPID_SRC}/tests/sasl_config \
-d
echo "started broker from $QPIDD"
----------- end script ------------------------
and here is my client iterator script;
---------------------- start script ---------------------
#! /bin/bash
rm core.* ~/.qpidd/core*
count=0
while [ $count -lt 10000 ]
do
echo "==========================================="
echo "TEST $count"
echo "==========================================="
sleep 1
core_files=`ls -l core.* ~/.qpidd/core* | wc -l`
echo "core files: ${core_files}"
if [ ${core_files} -gt 0 ]; then
echo "core files found!"
exit 1
else
echo "no core files found."
fi
./qpid-perftest --username zig --password zig --protocol rdma --broker
20.0.40.14 --port 5813 --qt 4 --count 10
count=$(( $count + 1 ))
done
---------------------- end script -----------------------
the sasl config directory that the broker is pointing at was created by the
script cpp/src/tests/sasl_test_setup.sh
also, you need to set "ulimit -l 131072" (at least that value) before
starting the broker.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscr...@qpid.apache.org
