Hello, I check activemq.log for messages like these to determine if activemq has corrupt data files:
[kxn2@amq-a02 scheduler]$ sudo grep "Failed to start job scheduler store" /opt/local/activemq/data/activemq.log | head -1 2022-06-03 16:00:46,670 | ERROR | Failed to start job scheduler store: JobSchedulerStore: /opt/local/apache-activemq-5.16.4/data/amq-acceptance-cluster/scheduler | org.apache.activemq.broker.BrokerService | main Then I move scheduleDB files after stopping activemq.service on both brokers. cd /opt/local/activemq/data/kahadb/scheduler sudo mv scheduleDB.data scheduleDB.data.`date +%Y%m%d`; sudo mv scheduleDB.redo scheduleDB.redo.`date +%Y%m%d` After starting ActiveMQ, 7,500,000 entries were recovered. That takes 2.5 hours but it usually works but, there was a corrupt journal file. [kxn2@amq-a02 data]$ grep Corrupt activemq.log* 2022-06-02 07:55:40,066 | WARN | Corrupt journal records found in '/opt/local/apache-activemq-5.16.4/data/amq-acceptance-cluster/scheduler/db-1179.log' between offsets: 11558626..11559784 | org.apache.activemq.store.kahadb.disk.journal.Journal | main We eventually stopped activemq, moved schedule/ and started activemq. Part of the problem is that it took 2.5 hours to recover 7.5M entried then fail. After we restarted, we have this. [kxn2@amq-a02 scheduler]$ ls -l db-*.log | wc -l 1 [kxn2@amq-a02 scheduler]$ wc -l db-1.log 50,067 db-1.log Before we had lots of log files. [kxn2@amq-a02 scheduler.bkup]$ ls -l db-*.log | wc -l 125 [kxn2@amq-a02 scheduler.bkup]$ wc -l db-*.log ... 8,697,209 total How can I get activemq to clean up these logs? P.S. I did eventually test removing the journal entry at byte range 11558626..11559784. I could get ActiveMQ to start after that. --- Karl Nordström Systems Administrator Penn State IT | Application Platforms
