Hi
I have a 2 node akka cluster, which looses connection once every two days.
Often when the server i almost idle. We are still on akka 2.2.4, using
oracle java 1.8_25. Its a virtual servers running on vmware esx.
The application have been running fine for 3 years, but started with this a
few weeks ago.
Anyone have any ideas on were to start troubleshooting this?
My clusterconfig:
remote {
log-sent-messages = on
log-received-messages = on
log-remote-lifecycle-events = off
netty.tcp {
port = 45000
}
watch-failure-detector.threshold = 15
}
cluster {
log-info = on
auto-down = on
auto-join = on
failure-detector {
threshold = 15
min-std-deviation = 500 ms
acceptable-heartbeat-pause = 20 s
}
heartbeat-request {
grace-period = 20 s
expected-response-after = 10 s
time-to-live = 60 s
}
use-dispatcher = cluster-dispatcher
}
}
cluster-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 2
parallelism-max = 4
}
}
logoutput:
node1:
2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86
),Actor[ak
ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:27,666 [svarut-akka.actor.default-dispatcher-31] ERROR
a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
svarut@193.161.171.181:45000] - Marking node(s) as UNREACHABLE [Memb
er(address = akka.tcp://svarut@193.161.171.182:45000, status = Up)]
2015-01-24 03:30:28,504 [svarut-akka.actor.default-dispatcher-17] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
svarut@193.161.171.181:45000] - Leader is marking unreachable node
[akka.tcp://svarut@193.161.171.182:45000] as [Down]
2015-01-24 03:30:28,506 [svarut-akka.actor.default-dispatcher-25] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
svarut@193.161.171.181:45000] - Leader is removing unreachable node
[akka.tcp://svarut@193.161.171.182:45000]
Node2:
2015-01-24 03:30:02,040 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903])
2015-01-24 03:30:02,042 [svarut-akka.actor.default-dispatcher-21] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903])
2015-01-24 03:30:28,636 [svarut-akka.actor.default-dispatcher-2] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Acto
r[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980])
2015-01-24 03:30:28,637 [svarut-akka.actor.default-dispatcher-18] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980])
2015-01-24 03:30:41,457 [svarut-akka.actor.default-dispatcher-22] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
svarut@193.161.171.182:45000] - Trigger extra expected heartbeat fr
om [akka.tcp://svarut@193.161.171.181:45000]
2015-01-24 03:31:06,471 [svarut-akka.actor.default-dispatcher-2] ERROR
a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
svarut@193.161.171.182:45000] - Marking node(s) as UNREACHABLE [Membe
r(address = akka.tcp://svarut@193.161.171.181:45000, status = Up)]
2015-01-24 03:31:07,445 [svarut-akka.actor.default-dispatcher-12] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
svarut@193.161.171.182:45000] - Leader is marking unreachable node
[akka.tcp://svarut@193.161.171.181:45000] as [Down]
2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-12] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://
svarut@193.161.171.182:45000] - Leader is removing unreachable node
[akka.tcp://svarut@193.161.171.181:45000]
2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-2] INFO
Remoting - Address [akka.tcp://svarut@193.161.171.181:45000] is now
quarantined, all messages to this address will be
delivered to dead letters.
2015-01-24 03:31:07,516 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-1/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-0#-
1902872711])
2015-01-24 03:31:07,517 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-5#-2023131039])
--
Idar
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
