I've made some headway on this, but could still use some help.
I went through and deleted all of the Offline Durable Topic Subscribers.
That freed up a little space (1%) in the store.
I notice there's a parameter /JournalFilesForFullRecovery that shows//a
value of 6500+. When I deleted the Offline Subscriber, the number
reduced about 75. After I deleted the offline subscribers, I restarted
the Broker. I saw that where previously, the number of journal entries
processed during startup/recover was over 300,000, during this restart
cycle, there were none, and the broker came online very quickly./
//
/I've been monitoring it for a while. DLQs show 0, and it's done maybe
10 or so posts. So, things are "working", but the store percent used is
staying at 99%, and the number of JournalFilesForFullRecovery is still
staying very high at 6440./
/Does anyone have any ideas on how to pursue this? At this point, I'm
thinking I should bring up a new MQ Broker and change my app to point to
it./
/George/////
//
On 3/9/2023 4:18 PM, George Sexton wrote:
I'm having problems now with an ActiveMQ instance running on AWS. I'm
running version 5.15.15 on an mq.m5.large instance, which is 4 cores
and 16GB of memory.
There's a couple of DLQs that are empty, and one topic that sees maybe
50 messages a day. The AMQP connector shows 20 connections, Active
Durable Subscribers is 12, and Offline Durable Topic Subscribers is
something on the order of 400 or so.
The store has been steadily increasing, around 9% per day and now it's
at 100%. The one topic has had maybe 700 messages.
The configuration is very simple, it's:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<broker advisorySupport="false"
xmlns=http://activemq.apache.org/schema/core>
<plugins>
<forcePersistencyModeBrokerPlugin persistenceFlag="true"/>
<statisticsBrokerPlugin/>
<timeStampingBrokerPlugin ttlCeiling="86400000"
zeroExpirationOverride="86400000"/>
</plugins>
</broker>
Logging shows:
2023-03-09 23:12:14,774 | INFO |
Usage(default:store:queue://ActiveMQ.DLQ:store) percentUsage=0%,
usage=214796216383, limit=214748364800,
percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%,
usage=214796216383, limit=214748364800, percentUsageMinDelta=1%:
Persistent store is Full, 100% of 214748364800. Stopping producer
(ID:.mq.us-west-2.amazonaws.com-X:44:0:0)
to prevent flooding queue://ActiveMQ.DLQ.
Seehttp://activemq.apache.org/producer-flow-control.html for more
info (blocking for: 1472s) |
org.apache.activemq.broker.region.Queue |
ActiveMQ Broker[MyApp] Scheduler
Doing a little math shows it's writing something like 226KB/Second.
I've noticed Network utilization seems very high.
Can anyone give me some ideas? Should I be looking at clients? Is this
some sort of internal file corruption? Should I just kill this and
deploy a totally new ActiveMQ instance?
Thanks for any help.
--
George Sexton
(303) 438 9585 x102
MH Software, Inc.
