I setup a 3 node ActiveMQ / Leveldb cluster using 5.13 Activemq running on
RedHat in a VM.
I'm also using the java wrapper.
It seems to run fine I ran 50K messages thru for a test and failed over many
times.
But when left alone over a weekend some of the nodes fail. See below from
ActiveMQ log.
Has anyone run into this?
2016-01-07 15:05:49,435 | INFO | Unable to read additional data from server
sessionid 0x2520de0943f0000, likely server has closed socket, closing socket
connection and attempting reconnect | org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(appdev042.corp.local:2181)
2016-01-07 15:05:49,677 | INFO | Slave stopped |
org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[localhost] Task-8
2016-01-07 15:05:50,440 | INFO | Client will use DIGEST-MD5 as SASL
mechanism. | org.apache.zookeeper.client.ZooKeeperSaslClient |
WrapperSimpleAppMain-SendThread(appdev041.corp.local:2181)
2016-01-07 15:05:50,441 | INFO | Opening socket connection to server
appdev041.corp.local/10.150.32.101:2181. Will attempt to SASL-authenticate
using Login Context section 'Client' | org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(appdev041.corp.local:2181)
2016-01-07 15:05:50,442 | INFO | Socket connection established to
appdev041.corp.local/10.150.32.101:2181, initiating session |
org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(appdev041.corp.local:2181)
2016-01-07 15:05:50,448 | INFO | Session establishment complete on server
appdev041.corp.local/10.150.32.101:2181, sessionid = 0x2520de0943f0000,
negotiated timeout = 4000 | org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(appdev041.corp.local:2181)
2016-01-07 15:05:50,453 | WARN | unprocessed event state: SaslAuthenticated
| org.apache.activemq.leveldb.replicated.groups.ZKClient |
WrapperSimpleAppMain-EventThread
2016-01-07 15:05:50,490 | INFO | Using the pure java LevelDB
implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
BrokerService[localhost] Task-9
2016-01-07 15:05:50,492 | INFO | Attaching to master:
tcp://appdev043.corp.local:60854 |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | ActiveMQ
BrokerService[localhost] Task-9
2016-01-07 15:05:50,493 | INFO | Slave started |
org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[localhost] Task-9
2016-01-07 15:05:50,937 | ERROR | Could not refresh JMS Connection for
destination 'esb.*' - retrying using FixedBackOff{interval=5000,
currentAttempts=17673, maxAttempts=unlimited}. Cause: Could not create
Transport. Reason: java.io.IOException: Broker named 'localhost' does not
exist. | org.apache.camel.component.jms.DefaultJmsMessageListenerContainer |
Camel (esbRoute1) thread #0 - JmsConsumer[esb.*]
2016-01-07 15:05:51,030 | INFO | Slave skipping download of:
log/0000000012c00b6a.log |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,178 | INFO | Slave requested:
00000000135ad8bd.index/CURRENT |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,180 | INFO | Slave requested:
00000000135ad8bd.index/000029.sst |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,180 | INFO | Slave requested:
00000000135ad8bd.index/000028.log |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,181 | INFO | Slave requested:
00000000135ad8bd.index/MANIFEST-000026 |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,187 | INFO | Attaching... Downloaded 0.02/4670.44 kb
and 1/4 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,194 | INFO | Attaching... Downloaded 744.92/4670.44 kb
and 2/4 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,209 | INFO | Attaching... Downloaded 4669.99/4670.44 kb
and 3/4 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,210 | INFO | Attaching... Downloaded 4670.44/4670.44 kb
and 4/4 files | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:51,211 | INFO | Attached |
org.apache.activemq.leveldb.replicated.SlaveLevelDBStore |
hawtdispatch-DEFAULT-1
2016-01-07 15:05:58,082 | INFO | Client session timed out, have not heard
from server in 3592ms for sessionid 0x2520de0943f0000, closing socket
connection and attempting reconnect | org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(appdev041.corp.local:2181)
2016-01-07 15:05:58,227 | INFO | Slave stopped |
org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[localhost] Task-12
2016-01-07 15:05:58,435 | INFO | Client will use DIGEST-MD5 as SASL
mechanism. | org.apache.zookeeper.client.ZooKeeperSaslClient |
WrapperSimpleAppMain-SendThread(10.150.32.103:2181)
2016-01-07 15:05:58,436 | INFO | Opening socket connection to server
10.150.32.103/10.150.32.103:2181. Will attempt to SASL-authenticate using
Login Context section 'Client' | org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(10.150.32.103:2181)
2016-01-07 15:05:58,454 | ERROR | Unable to open socket to
10.150.32.103/10.150.32.103:2181 | org.apache.zookeeper.ClientCnxnSocketNIO
| WrapperSimpleAppMain-SendThread(10.150.32.103:2181)
2016-01-07 15:05:58,460 | WARN | Session 0x2520de0943f0000 for server null,
unexpected error, closing socket connection and attempting reconnect |
org.apache.zookeeper.ClientCnxn |
WrapperSimpleAppMain-SendThread(10.150.32.103:2181)
java.net.SocketException: Network is unreachable
at sun.nio.ch.Net.connect0(Native Method)[:1.7.0_85]
at sun.nio.ch.Net.connect(Net.java:484)[:1.7.0_85]
at sun.nio.ch.Net.connect(Net.java:476)[:1.7.0_85]
at
sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:675)[:1.7.0_85]
at
org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)[zookeeper-3.4.6.jar:3.4.6-1569965]
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)[zookeeper-3.4.6.jar:3.4.6-1569965]
Thanks
Mike
--
View this message in context:
http://activemq.2283324.n4.nabble.com/ActiveMQ-5-13-leveldb-cluster-failing-over-weekend-tp4706507.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
