Hello
We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. It's
a very old system. However last night we started to get these errors and
the broker did not come up. Any ideas on how to fix this would be
appreciated. I can provide further log messages as needed, if sufficient
direction is provided. Thank you!
Here are the logs
2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
is starting |
org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,732 | INFO | Listening for connections at:
tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
.maxFrameSize=104857600 |
org.apache.activemq.transport.TransportServerThreadSupport | main
2020-08-07 02:44:54,733 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2020-08-07 02:44:54,823 | INFO | Listening for connections at:
ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
at
org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_232]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
at
org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
at
org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
2020-08-07 02:44:46,669 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date [Fri
Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated
LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,10.184.11.9:2181,
10.184.11.7:2181//activemq/ap
ps/mkt/lpf/prod-usc1a/BROKER01P] | org.apache.activemq.broker.BrokerService
| main
2020-08-07 02:44:51,492 | INFO | Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,494 | INFO | Client
environment:host.name=mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,496 | INFO | Client environment:java.version=1.8.0_232
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle
Corporation | org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,500 | INFO | Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,504 | INFO | Client
environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
|
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,505 | INFO | Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,506 | INFO | Client
environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,509 | INFO | Client
environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,510 | INFO | Client
environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,514 | INFO | Initiating client connection,
connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
sessionTimeout=2000 watcher=org.apache.activem
q.leveldb.replicated.groups.ZKClient@36060e |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state
change dispatcher thread
2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section
named 'Client' was found in specified JAAS
configuration file:
'/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue
connection to Zookeeper server without SASL authentication, if Zookeeper
server a
llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
10.184.11.9:2181)
2020-08-07 02:44:51,577 | INFO | Opening socket connection to server
10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
main-SendThread(10.184.11.9:2181)
2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed |
org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
2020-08-07 02:44:51,606 | INFO | Socket connection established to
10.184.11.9/10.184.11.9:2181, initiating session |
org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
11.9:2181)
2020-08-07 02:44:51,622 | INFO | Session establishment complete on server
10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated
timeout = 4000 | org.apache.
zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
2020-08-07 02:44:51,946 | INFO | Promoted to master |
org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
2020-08-07 02:44:52,110 | WARN | listeners are taking too long to process
the events |
org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
main-EventThread
2020-08-07 02:44:52,111 | WARN | listeners are taking too long to process
the events |
org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
main-EventThread
2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
BrokerService[BROKER01P] Task-1
2020-08-07 02:44:54,349 | INFO | Master started:
tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
| org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[BROKER01P] Task-2
2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
is starting | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,732 | INFO | Listening for connections at:
tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2020-08-07 02:44:54,733 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2020-08-07 02:44:54,823 | INFO | Listening for connections at:
ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
| org.apache.activemq.transport.TransportServerThreadSupport | main
2020-08-07 02:44:54,825 | INFO | Connector ssl started |
org.apache.activemq.broker.TransportConnector | main
2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
started | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,827 | INFO | For help or more information please see:
http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current store
usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb only
has 42345 mb of usable space. - resetting to maximum available disk space:
42345 mb | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
(current store usage is 25900 mb). The data directory:
/var/lib/activemq/data only has 42345 mb of usable space. - resetting to
maximum available disk space: 42345 mb |
org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due to
exception, java.io.EOFException: File
'/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.EOFException: File
'/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
