Active MQ stalls after Job Scheduler component shuts down.
----------------------------------------------------------
Key: AMQ-3103
URL: https://issues.apache.org/jira/browse/AMQ-3103
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.4.2
Environment: Redhat Enterprise Linux 5.X,
JDK 1.5 32-bit
JDK 1.6 64-bit
Active MQ 4.2
Reporter: Swapnonil Mukherjee
Observation
----
Active MQ stops accepting all incoming messages destined for a particular,
queue after the scheduler component processing scheduled messages on that queue
encounters an Null Pointer Exception.
Environment
----
We are using the Spring JMSTemplate component to post messages onto a queue. We
also place a delay of 30 seconds on each message before posting
{noformat}
message.setLongProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY,
Integer.parseInt("30") * 1000);
{noformat}
We use the Spring Default Message Listener Container to receive messages.
Normally the broker runs fine and we have seen messages appear under the
"Scheduled" tab on the Active MQ Console, after which they processed normally
and we can tally using the "Messages Enqueued" and the "Messages Dequeued"
numbers. But occasionally the Job Scheduler fails with the following exception.
{noformat}
2010-12-10 16:31:38,522 | ERROR | JMS Failed to schedule job |
org.apache.activemq.broker.scheduler.JobSchedulerImpl | JobScheduler:JMS
java.lang.NullPointerException
at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:264)
at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:225)
at org.apache.kahadb.index.BTreeNode.remove(BTreeNode.java:330)
at org.apache.kahadb.index.BTreeIndex.remove(BTreeIndex.java:194)
at
org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:347)
at
org.apache.activemq.broker.scheduler.JobSchedulerImpl$4.execute(JobSchedulerImpl.java:125)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
at
org.apache.activemq.broker.scheduler.JobSchedulerImpl.remove(JobSchedulerImpl.java:123)
at
org.apache.activemq.broker.scheduler.JobSchedulerImpl.mainLoop(JobSchedulerImpl.java:515)
at
org.apache.activemq.broker.scheduler.JobSchedulerImpl.run(JobSchedulerImpl.java:429)
at java.lang.Thread.run(Thread.java:619)
2010-12-10 16:31:39,561 | INFO |
JobSchedulerStore:activemq-data/primary/scheduler stopped |
org.apache.activemq.broker.scheduler.JobSchedulerStore | JobScheduler:JMS
{noformat}
Why does the Job Scheduler fail? One possible reason we have found is that the
clock time settings on the VMs producing the messages and the broker as well as
the consumers are all different. So the Job Scheduler may be shutting itself
down arbitrarily due to this difference in clock. We are in the process of
syncing all clocks but we are not sure whether this will solve the problem.
Bug
----
But the Bug really is, even if the Job Scheduler encounters an Null Pointer,
why should it shutdown? Even more problematic is the fact that the queue itself
stalls and does not accept anymore messages after the Job Scheduler shuts down.
We have tried to delete the db.redo log to recover from this type of shutdown.
The broker recovers fine, but all messages posted to this queue after the Job
Scheduler shut itself down were lost. We have not been able to recover those
messages.
I am attaching the activemq log and activemq configuration file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
