Hi Folks,
I'm seeing crashing in qpid proton (qpid-proton-c-devel-0.9-3 on fc22) when
running my publisher at a decent rate (1000 msg/s) for a minute or so.
Does something like this look familiar? Somehow got into a state where
pn_data_grow is called, but the data node coming back from it points to an
invalid memory location.
(gdb)
bt
[31/1873]
#0 0x0000000006bf55df in pn_data_new (data=data@entry=0x5e6a090) at
/usr/src/debug/qpid-proton-0.9/proton-c/src/codec/codec.c:1126
#1 0x0000000006bf5de8 in pn_data_add (data=data@entry=0x5e6a090) at
/usr/src/debug/qpid-proton-0.9/proton-c/src/codec/codec.c:1397
#2 0x0000000006bf5fa8 in pn_data_put_ubyte (data=data@entry=0x5e6a090,
ub=<optimized out>) at
/usr/src/debug/qpid-proton-0.9/proton-c/src/codec/codec.c:1485
#3 0x00000000069c4c14 in qpidBridgeMsgCodec_pack (bridgeMessage=0x5e61fb0,
target=target@entry=0x5e6b2c0, protonMessage=protonMessage@entry
=0xffefff528)
at mama/c_cpp/src/c/bridge/qpid/codec.c:119
#4 0x00000000069c5fd2 in qpidBridgePublisherImpl_enqueueMessageForAddress
(msg=msg@entry=0x5e6b2c0, url=<optimized out>, impl=impl@entry=0x5e61f30)
at mama/c_cpp/src/c/bridge/qpid/publisher.c:629
#5 0x00000000069c63cd in qpidBridgeMamaPublisher_send
(publisher=0x5e61f30, msg=0x5e6b2c0) at
mama/c_cpp/src/c/bridge/qpid/publisher.c:295
#6 0x0000000004e6be97 in mamaPublisher_send (publisher=0x5e61dc0,
msg=msg@entry=0x5e6b2c0) at mama/c_cpp/src/c/publisher.c:258
#7 0x000000000040577d in publishMessageRdtsc (pubIndex=<optimized out>,
msgSample=<optimized out>, nowTsc=<optimized out>) at
mama/c_cpp/src/testtools/performance/c/mamaproducerc_v2.c:1620
#8 0x0000000000403b20 in main (argc=<optimized out>, argv=<optimized out>)
at mama/c_cpp/src/testtools/performance/c/mamaproducerc_v2.c:782
(gdb) l
1121 {
1122 if (data->capacity <= data->size) {
1123 pn_data_grow(data);
1124 }
1125 pni_node_t *node = pn_data_node(data, ++(data->size));
1126 node->next = 0;
1127 node->down = 0;
1128 node->children = 0;
1129 return node;
1130 }
(gdb) print *data
$11 = {nodes = 0x0, buf = 0x5e6a5d0, decoder = 0x5e6a6c0, encoder =
0x5e6a790, error = 0x5e6a850, str = 0x5e6a8c0, capacity = 0, size = 32769,
parent = 1, current = 0, base_parent = 0,
base_current = 0}
(gdb) print node
$12 = (pni_node_t *) 0x240000 <---- invalid memory location
I ran it through valgrind as well and couldn't see any invalid memory
trouncing leading up to the event:
==2332== Invalid write of size 2
==2332== at 0x6BF55DF: pn_data_new (codec.c:1126)
==2332== by 0x6BF5DE7: pn_data_add (codec.c:1397)
==2332== by 0x6BF5FA7: pn_data_put_ubyte (codec.c:1485)
==2332== by 0x69C4C13: qpidBridgeMsgCodec_pack (codec.c:119)
==2332== by 0x69C5FD1: qpidBridgePublisherImpl_enqueueMessageForAddress
(publisher.c:629)
==2332== by 0x69C63CC: qpidBridgeMamaPublisher_send (publisher.c:295)
==2332== by 0x4E6BE96: mamaPublisher_send (publisher.c:258)
==2332== by 0x40577C: publishMessageRdtsc (mamaproducerc_v2.c:1620)
==2332== by 0x403B1F: main (mamaproducerc_v2.c:782)
==2332== Address 0x240034 is not stack'd, malloc'd or (recently) free'd
==2332==
==2332==
==2332== Process terminating with default action of signal 11 (SIGSEGV):
dumping core
==2332== Access not within mapped region at address 0x240034
==2332== at 0x6BF55DF: pn_data_new (codec.c:1126)
==2332== by 0x6BF5DE7: pn_data_add (codec.c:1397)
==2332== by 0x6BF5FA7: pn_data_put_ubyte (codec.c:1485)
==2332== by 0x69C4C13: qpidBridgeMsgCodec_pack (codec.c:119)
==2332== by 0x69C5FD1: qpidBridgePublisherImpl_enqueueMessageForAddress
(publisher.c:629)
==2332== by 0x69C63CC: qpidBridgeMamaPublisher_send (publisher.c:295)
==2332== by 0x4E6BE96: mamaPublisher_send (publisher.c:258)
==2332== by 0x40577C: publishMessageRdtsc (mamaproducerc_v2.c:1620)
==2332== by 0x403B1F: main (mamaproducerc_v2.c:782)
==2332== If you believe this happened as a result of a stack
==2332== overflow in your program's main thread (unlikely but
==2332== possible), you can try to increase the size of the
==2332== main thread stack using the --main-stacksize= flag.
==2332== The main thread stack size used in this run was 8388608.
Cheers,
Frank
