Re: Some stomp-messages are received with an empty text

[Arjen van der Meijden]

Noone? I've done some follow-up research, but still have no idea which 
part of ActiveMQ fails. It turns out that the incoming stomp-messages 
are all good, so they're received correctly. And I've also noticed that, 
by looking at the tcp-traffic, the outgoing messages indeed don't have 
the body so now and then.

So the text-message body disappears inside ActiveMQ. It looks to me as 
if some internal resource is shared in a place it shouldn't have been 
shared and is reset to null before others are done with it. But I 
couldn't find such a resource in the o.a...transport.stomp-package. Then 
again I don't know the code that well, so that doesn't mean much.

And while I can easily reproduce (on that specific AMQ-instance) the 
problem with any topic, I can't reproduce it on another AMQ. So I'm 
guessing the load inhibited on our production-instance has something to 
do with it as well (which supports the idea of some shared resource that 
shouldn't be shared amongst threads).

I've already filed a JIRA-issue, but would really like to figure this 
out and find a solution.
https://issues.apache.org/activemq/browse/AMQ-2871

Any ideas how to proceed from here?

Best regards,

Arjen

On 14-8-2010 20:05 Arjen van der Meijden wrote:
Hi List,

I have a fairly simple set-up with a single Activemq (currently 5.3.2,
but the behavior existed with previous versions).

On that AMQ we have a few queue's and topics, where the topics are
filled using PHP producers with Stomp and four Java subscribers with the
normal openwire protocol (via a Spring jms listener container). The PHP
producers are created on demand and send a single, tiny text message
when some data needs to be adjusted in the Java subscribers.

The problem is that all four consumers sometimes receive a message
without such a text body, where it should have contained one.

For instance, we have a hourly cronjob that sends an update after its
done ('views 0'). This cronjob is executed on the same machine, with the
same php-scripts etc. Today, the message was empty at 7u, 10u, 11u, 15u,
17u and 19u, although yesterday only at 6u, 10u and 20u.

So for now my question would be: how to debug this? I can trivially
reproduce the situation by just sending a few bogus-messages some of
those will fail. I can't see any difference in the php-side between
failed and successful messages. But the server is used quite heavily
with messages over the queues, so just changing the log level to debug
won't be a good start (unless only a specific topic could be in
debugmode?). I could separate the queues from the topics for now, but
than I have no guarantee the same situation occurs.

I've no idea where to look, but I'm actually suspecting ActiveMQ for now
as I can't really believe my php-scripts would fail only some of the
time (normally it fails or it works).

Any idea where I should look?

Best regards,

Arjen

My configuration is rather trivial, its the default config without
systemusage and destinationpolicy. With the stomp and openwire
transporters like so:
<transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?transport.closeAsync=false"/>


When logging the messages in the subscribers, it looks like this for a
correct one:
ActiveMQTextMessage {commandId = 3, responseRequired = false, messageId
= ID:argus-38958-1279890714168-2:66257231:-1:1:1, originalDestination =
null, originalTransactionId = null, producerId =
ID:argus-38958-1279890714168-2:66257231:-1:1, destination =
topic://pw_cache_products, transactionId = null, expiration = 0,
timestamp = 1281801902987, arrival = 0, brokerInTime = 1281801902987,
brokerOutTime = 1281801902988, correlationId = null, replyTo = null,
persistent = false, type = null, priority = 0, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = false, userID =
null, content = null, marshalledProperties = null, dataStructure = null,
redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties =
true, readOnlyBody = true, droppable = false, text = views 0}

And like this for an incorrect one:
ActiveMQTextMessage {commandId = 3, responseRequired = false, messageId
= ID:argus-38958-1279890714168-2:66384399:-1:1:1, originalDestination =
null, originalTransactionId = null, producerId =
ID:argus-38958-1279890714168-2:66384399:-1:1, destination =
topic://pw_cache_products, transactionId = null, expiration = 0,
timestamp = 1281805503252, arrival = 0, brokerInTime = 1281805503252,
brokerOutTime = 1281805503252, correlationId = null, replyTo = null,
persistent = false, type = null, priority = 0, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = false, userID =
null, content = null, marshalledProperties = null, dataStructure = null,
redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties =
true, readOnlyBody = true, droppable = false, text = null}

I.e. text = null vs text = views 0