Rajagopal created AMQ-6354:
------------------------------
Summary: KahaDb journal file piling up because of Durable
subscriber destination
Key: AMQ-6354
URL: https://issues.apache.org/jira/browse/AMQ-6354
Project: ActiveMQ
Issue Type: Bug
Components: Broker, KahaDB, Message Store
Affects Versions: 5.5.1
Environment: Linux
Reporter: Rajagopal
AMQ Configuration:
1. KahaDb Persistence store
2. Topics and Queues destination are configured.
3. Durable Subscribers configured (added/removed) dynamically (All added
subscribers will be active) for Topic destination.
Issue:
• Nearly ~20 Durable Subscriber added/deleted to a particular destination
dynamically.
• After configuration, Once the message are sent to the Topic, the
messages are consumed and Acknowledged by each subscribers.
• There were 249 Subscribers and we could see from Journal file that all
subscribers acknowledged.
• In JMX logs after message is sent the below entries are observed for
yyyy destination,
o Consumer Count is 249
o In-Flight Count is 0
o Dispatch count is matching with the consumer count (messages * Consumer
count)
• We enabled kahaDB TRACE logs and it clearly points out that for a
particular Topic destination the journal files could not be gc collected
Snippet of activemq.log
----------------------------------------------
2016-06-30 01:41:43,980 org.apache.activemq.store.kahadb.MessageDatabase TRACE
(MessageDatabase.java:checkpointUpdate:1413) [ActiveMQ Journal Checkpoint
Worker] [24] | gc candidates after dest:0:xxxxx, [8066, 8067, ...........]
2016-06-30 01:41:43,980 org.apache.activemq.store.kahadb.MessageDatabase TRACE
(MessageDatabase.java:checkpointUpdate:1413) [ActiveMQ Journal Checkpoint
Worker] [24] | gc candidates after dest:1:yyyy, [8067, 8073, ............]
--------------------------------------------
• After analyzing the 8066 file, we understood that 8066 has one message
for yyyy Topic destination and 249 ACk’s in the same file. But Checkpoint
cleanup could not remove the file. So this pattern followed and all the files
containing the yyyy destination messages are not removed, which leads to
journal files pile-up.
Workaround :
• As a work around, first we restarted the process but still the files
were not cleaned up.
• Secondly, we stopped the process, removed the index (db.data) file and
restarted. Now all the files are cleaned up.
• By above workaround, we suspect that index file is corrupted, which is
causing the journal pile up.
• Our question,
o if index file is corrupted, what is the reason for corruption?
o Is there any bug in the 5.5.1 version of AMQ regarding index file
corruption for Durable Subscription?
This is sporadic issue and not reproducible always. (Not always Dynamic
Subscription configuration leads to journal pile up)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
