Journal files don't get cleaned up
----------------------------------
Key: AMQ-1786
URL: https://issues.apache.org/activemq/browse/AMQ-1786
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.1.0
Environment: Linux server
Sun JDK 1.6.0
Reporter: Brian Desai
Priority: Critical
Attachments: activemq.xml
I'm running ActiveMQ 5.1.0 with the AMQ persistence adapter, and it appears
that not all of the journal files get cleaned up. My setup is a little
abnormal, as I'm trying to test out ActiveMQ's ability to handle queue
messaging with consumers that may become inactive for periods of time.
So for this test, I have a single publisher pushing messages out to 21 queues.
These are persistent messages with an expiration time. I have a message
listener reading from all queues (reading from '>'). So, as soon as the
message is sent to the queues, it's read by the message listener, taking it off
the queue. So far, so good.
I have a 2 MB max file length set on the AMQ persistence adapter. So, I would
expect to see for the journal, 2 MB files that get cleaned up after the file
rolls over. However, the journal files don't always get cleaned up, as shown
in the file listing below. Out of 181 rollovers, 30 of the files did not get
cleaned up. The message listener showed no errors, and as far as I can tell,
it didn't drop any messages.
{noformat}
-rw-r--r-- 1 root root 2096753 2008-05-30 20:30 data/journal/data-13
-rw-r--r-- 1 root root 2096967 2008-05-30 20:31 data/journal/data-14
-rw-r--r-- 1 root root 2096899 2008-05-30 20:45 data/journal/data-25
-rw-r--r-- 1 root root 2097057 2008-05-30 21:20 data/journal/data-52
-rw-r--r-- 1 root root 2096916 2008-05-30 21:22 data/journal/data-54
-rw-r--r-- 1 root root 2096536 2008-05-30 21:45 data/journal/data-72
-rw-r--r-- 1 root root 2096894 2008-05-30 21:47 data/journal/data-73
-rw-r--r-- 1 root root 2097129 2008-05-30 21:49 data/journal/data-75
-rw-r--r-- 1 root root 2097101 2008-05-30 21:58 data/journal/data-82
-rw-r--r-- 1 root root 2097026 2008-05-30 21:59 data/journal/data-83
-rw-r--r-- 1 root root 2096906 2008-05-30 22:02 data/journal/data-85
-rw-r--r-- 1 root root 2096973 2008-05-30 22:13 data/journal/data-94
-rw-r--r-- 1 root root 2097105 2008-05-30 22:24 data/journal/data-102
-rw-r--r-- 1 root root 2097033 2008-05-30 22:41 data/journal/data-113
-rw-r--r-- 1 root root 2096730 2008-05-30 22:42 data/journal/data-114
-rw-r--r-- 1 root root 2096569 2008-05-30 22:45 data/journal/data-116
-rw-r--r-- 1 root root 2096870 2008-05-30 22:50 data/journal/data-118
-rw-r--r-- 1 root root 2096567 2008-05-30 22:52 data/journal/data-119
-rw-r--r-- 1 root root 2096766 2008-05-30 23:03 data/journal/data-128
-rw-r--r-- 1 root root 2096877 2008-05-30 23:06 data/journal/data-130
-rw-r--r-- 1 root root 2096888 2008-05-30 23:18 data/journal/data-140
-rw-r--r-- 1 root root 2096699 2008-05-30 23:20 data/journal/data-141
-rw-r--r-- 1 root root 2096973 2008-05-30 23:22 data/journal/data-143
-rw-r--r-- 1 root root 2096924 2008-05-30 23:31 data/journal/data-150
-rw-r--r-- 1 root root 2096936 2008-05-30 23:45 data/journal/data-161
-rw-r--r-- 1 root root 2096527 2008-05-30 23:57 data/journal/data-170
-rw-r--r-- 1 root root 2097151 2008-05-30 23:58 data/journal/data-171
-rw-r--r-- 1 root root 2096972 2008-05-31 00:11 data/journal/data-179
-rw-r--r-- 1 root root 2096703 2008-05-31 00:13 data/journal/data-180
-rw-r--r-- 1 root root 2097069 2008-05-31 00:14 data/journal/data-181
{noformat}
I've also tried taking out the wildcard '>' on a single consumer, and instead
used separate consumers for each queue, and I get the same result.
I haven't even gotten to the test yet where the listener is not running. So,
in this "normal" operation, all messages are consumed. Yet, not all journal
files get cleaned up. These left-over files don't ever get cleaned up. They
will eventually start filling the hard drive. I can understand files being
left behind when there's no consumer, but there is a consumer the whole time.
What I'm basically looking for is a persistence layer for messaging to multiple
clients, so that consumers can get messages retroactively when they start up.
I could try to use topics with durable clients, but I thought the queues would
be easier to setup, as messages in queues are persisted by default. However, I
don't want the consumer to process "stale" messages, which is why I set an
expiration time. So, I would think that, with a constant rate of messages, the
persistent disk store utilization would eventually level out as the messages
started to expire. I realize that if there's no consumer for a queue, expired
messages won't get cleaned up (am currently trying to figure out a work-around
for that - periodically checking the queues with a QueueBrowser seems to
trigger the removal of expired messages). However, even when all consumers are
active, the journal keeps growing, as it's not always cleaning up it's files!
I've attached my configuration to this ticket.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
