Hi,
I'm working on ActiveMQ 5.8.0 with MySQL as a data source. The configuration
of ActiveMQ is attached (activemq.xml). It is used in a basic scenario: a
producer sends a short message (JSON object with 2 fields) with 5s delay to
queue T on which 3 consumers listen. The consumers are created using plain
Java API:
for (int i = 0; i < numberOfConsumer; i++)
{
Session session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
Destination destination =
session.createQueue(queueName);
MessageConsumer consumer =
session.createConsumer(destination);
MyConsumer myConsumer = new MyConsumer();
consumer.setMessageListener(myConsumer);
connection.setExceptionListener(myConsumer);
consumers.add(invoiceConsumer);
}
MyConsumer class implements both javax.jms.ExceptionListener and
javax.jms.MessageListener. URL used by consumer is:
failover://(tcp://localhost:61616)?startupMaxReconnectAttempts=3&initialReconnectDelay=1000&timeout=500&maxReconnectDelay=1000&maxReconnectAttempts=-1
The problem is as follows:
At times a message is not delivered to any of the consumer. However, the
following messages are. I can see an entry for the corresponding message in
ACTIVEMQ_MSGS table. Once I restart ActiveMQ (without a restart of an
application holding the consumers), the missing message gets delivered. I
enclose the full init.out in which one can see what happens under the hood
of ActiveMQ (including the failing delivery of a message).
Let me do a short analysis for you. Once a message is sent with 5s delay an
internal job in ActiveMQ is scheduled for that:
2013-08-22 10:42:38,546 | DEBUG | ActiveMQ Transport:
tcp:///127.0.0.1:43716@61616 |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | Scheduling Job
[id=ID:localhost.
localdomain-43080-1377160848385-1:1:5:1:1, startTime=Thu Aug 22 10:42:38
CEST 2013, delay=5000, period=0, repeat=0, nextTime=Thu Jan 01 01:00:00 CET
1970]
When the time comes, job is fired:
2013-08-22 10:42:43,006 | DEBUG | JobScheduler:JMS |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | Firing Job
[id=ID:localhost.localdomain-43080-1377160848385-1:1:
5:1:1, startTime=Thu Aug 22 10:42:38 CEST 2013, delay=5000, period=0,
repeat=0, nextTime=Thu Aug 22 10:42:43 CEST 2013]
2013-08-22 10:42:43,008 | DEBUG | JobScheduler:JMS |
org.apache.activemq.broker.region.Queue | localhost Message
ID:localhost.localdomain-43080-1377160848385-1:1:5:1:1 sent to que
ue://T
Then a bridging takes place - which *doesn't* occur for messages which
actually get delivered - which I don't understand:
2013-08-22 10:42:43,017 | DEBUG | ActiveMQ BrokerService[localhost] Task-5 |
org.apache.activemq.network.DemandForwardingBridgeSupport | bridging
(localhost -> localhost) ID:local
host.localdomain-43080-1377160848385-1:1:5:1:1, consumer:
ID:localhost.localdomain-38827-1377160835122-4:1:1:1, destination queue://T,
brokerPath: [ID:localhost.localdomain-38827-
1377160835122-1:1], message: ActiveMQObjectMessage {commandId = 16,
responseRequired = true, messageId =
ID:localhost.localdomain-43080-1377160848385-1:1:5:1:1, originalDestinatio
n = null, originalTransactionId = null, producerId =
ID:localhost.localdomain-38827-1377160835122-4:2:1:1, destination =
queue://T, transactionId = null, expiration = 0, timestamp
= 1377160958535, arrival = 0, brokerInTime = 1377160963007, brokerOutTime =
1377160963009, correlationId = null, replyTo = null, persistent = true, type
= null, priority = 4, gro
upID = null, groupSequence = 0, targetConsumerId = null, compressed = false,
userID = null, content = org.apache.activemq.util.ByteSequence@4a6e0a9,
marshalledProperties = org.apa
che.activemq.util.ByteSequence@67e779a1, dataStructure = null,
redeliveryCounter = 0, size = 1117, properties =
{scheduledJobId=ID:localhost.localdomain-43080-1377160848385-1:1:5:
1:1}, readOnlyProperties = false, readOnlyBody = false, droppable = false}
After that a rescheduling of a job takes place:
2013-08-22 10:42:43,025 | DEBUG | ActiveMQ Transport:
tcp:///127.0.0.1:43630@61616 |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | Scheduling Job
[id=ID:localhost.
localdomain-43080-1377160848385-1:1:5:1:1, startTime=Thu Aug 22 10:42:43
CEST 2013, delay=5000, period=0, repeat=0, nextTime=Thu Jan 01 01:00:00 CET
1970]
and firing of the job to send a message again:
2013-08-22 10:42:48,013 | DEBUG | JobScheduler:JMS |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerImpl | Firing Job
[id=ID:localhost.localdomain-43080-1377160848385-1:1:
5:1:1, startTime=Thu Aug 22 10:42:43 CEST 2013, delay=5000, period=0,
repeat=0, nextTime=Thu Aug 22 10:42:48 CEST 2013]
2013-08-22 10:42:48,015 | DEBUG | JobScheduler:JMS |
org.apache.activemq.broker.region.Queue | localhost Message
ID:localhost.localdomain-43080-1377160848385-1:1:5:1:1 sent to que
ue://T
However, no message is seen on the consumer side whatsoever and the entry
for the message remains in ACTIVEMQ_MSGS (no ACK?):
ID, CONTAINER, MSGID_PROD, MSGID_SEQR, EXPIRATION, MSG, PRIORITY, XID
'3', 'queue://T', 'ID:localhost.localdomain-43080-1377160848385-1:1:5:1',
'1', '0', BLOB, '0', NULL
For comparison, one can see a correct message sent at 10:41:38,054 in the
logs.
To sum up, the problem is that from time to time a message isn't delivered
by ActiveMQ. Is there something wrong with the configuration either on
server or consumer side? A doubtful bug in ActiveMQ? A hack for that is to
restart ActiveMQ for the missing messages to be delivered.
Moreover, for maintenance purposes - is there documentation / blog post /
article about how ACTIVEMQ_MSGS table works (apart from the short
description in 'ActiveMQ in Action' book)? What should be an action taken by
DBA if a number of entries in ACTIVEMQ_MSGS table grow?
I'd be grateful for any support since I'm beginning to feel helpless.
Cheers,
Tomek
activemq.xml
<http://activemq.2283324.n4.nabble.com/file/n4670572/activemq.xml>
init.out <http://activemq.2283324.n4.nabble.com/file/n4670572/init.out>
--
View this message in context:
http://activemq.2283324.n4.nabble.com/No-message-delivery-vs-ACTIVEMQ-MSGS-tp4670572.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
