Re: [akka-user] Akka cluster unreachable
Hi Idar,
Happy to hear that, and thanks for the confirmation.
-Endre
On Wed, Feb 4, 2015 at 9:08 AM, Idar Borlaug wrote:
> I have upgraded to 2.3.9 it have now been running for 5 days, without any
> problems.
>
> On 28 January 2015 at 11:50, Idar Borlaug wrote:
>
>> I know, we have a few problems with upgradeing to 2.3.9. (we might wait
>> until eventstore has a cluster connector for java)
>>
>> The only thing i can see is that the nodes have been almost idle for a
>> good while. But there are a few requests coming in. It dosen work fine for
>> 2-3 days.
>>
>> Will the state actor die if say a firewall cuts the tcp connection
>> because its been alive too long?
>>
>> I will see if operations can monitor more data, and consider upgrading to
>> 2.3.9 again.
>>
>> On 27 January 2015 at 14:27, Björn Antonsson <
>> bjorn.antons...@typesafe.com> wrote:
>>
>>> Hi Idar,
>>>
>>> A lot of things have been fixed in akka since 2.2.4. Would it be
>>> possible to upgrade to 2.3.9?
>>>
>>> From the log that you pasted it seems like the akka protocol state actor
>>> for the connection to node 2 has died on node 1, before your log starts. Is
>>> there something unusual going on on the machines, like GC or something else
>>> that would consume resources at the time of the failure? Are you monitoring
>>> other health metrics like disk space and swap usage?
>>>
>>> B/
>>>
>>> On 26 January 2015 at 12:31:19, Idar Borlaug (idar.borl...@gmail.com)
>>> wrote:
>>>
>>> Hi
>>>
>>> I have a 2 node akka cluster, which looses connection once every two
>>> days. Often when the server i almost idle. We are still on akka 2.2.4,
>>> using oracle java 1.8_25. Its a virtual servers running on vmware esx.
>>> The application have been running fine for 3 years, but started with
>>> this a few weeks ago.
>>> Anyone have any ideas on were to start troubleshooting this?
>>>
>>> My clusterconfig:
>>> remote {
>>>log-sent-messages = on
>>>log-received-messages = on
>>>log-remote-lifecycle-events = off
>>>netty.tcp {
>>>port = 45000
>>> }
>>> watch-failure-detector.threshold = 15
>>>}
>>>
>>> cluster {
>>> log-info = on
>>> auto-down = on
>>> auto-join = on
>>> failure-detector {
>>> threshold = 15
>>> min-std-deviation = 500 ms
>>> acceptable-heartbeat-pause = 20 s
>>> }
>>> heartbeat-request {
>>> grace-period = 20 s
>>> expected-response-after = 10 s
>>> time-to-live = 60 s
>>> }
>>> use-dispatcher = cluster-dispatcher
>>> }
>>> }
>>>
>>> cluster-dispatcher {
>>> type = "Dispatcher"
>>> executor = "fork-join-executor"
>>> fork-join-executor {
>>> parallelism-min = 2
>>> parallelism-max = 4
>>> }
>>> }
>>>
>>>
>>> logoutput:
>>> node1:
>>> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
>>> n.k.e.DeadLetterLogger - Dead letter:
>>> DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
>>>
>>> dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>>> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
>>> n.k.e.DeadLetterLogger - Dead letter:
>>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>>
>>> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>>> n.k.e.DeadLetterLogger - Dead letter:
>>> DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
>>>
>>> or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>>> n.k.e.DeadLetterLogger - Dead letter:
>>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>>
>>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>>> n.k.e.DeadLetterLogger - Dead letter:
>>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>>
>>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>>> n.k.e.DeadLetterLogger - Dead letter:
>>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>>
>>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>>> 2015-01-24 0
Re: [akka-user] Akka cluster unreachable
I have upgraded to 2.3.9 it have now been running for 5 days, without any
problems.
On 28 January 2015 at 11:50, Idar Borlaug wrote:
> I know, we have a few problems with upgradeing to 2.3.9. (we might wait
> until eventstore has a cluster connector for java)
>
> The only thing i can see is that the nodes have been almost idle for a
> good while. But there are a few requests coming in. It dosen work fine for
> 2-3 days.
>
> Will the state actor die if say a firewall cuts the tcp connection because
> its been alive too long?
>
> I will see if operations can monitor more data, and consider upgrading to
> 2.3.9 again.
>
> On 27 January 2015 at 14:27, Björn Antonsson > wrote:
>
>> Hi Idar,
>>
>> A lot of things have been fixed in akka since 2.2.4. Would it be possible
>> to upgrade to 2.3.9?
>>
>> From the log that you pasted it seems like the akka protocol state actor
>> for the connection to node 2 has died on node 1, before your log starts. Is
>> there something unusual going on on the machines, like GC or something else
>> that would consume resources at the time of the failure? Are you monitoring
>> other health metrics like disk space and swap usage?
>>
>> B/
>>
>> On 26 January 2015 at 12:31:19, Idar Borlaug (idar.borl...@gmail.com)
>> wrote:
>>
>> Hi
>>
>> I have a 2 node akka cluster, which looses connection once every two
>> days. Often when the server i almost idle. We are still on akka 2.2.4,
>> using oracle java 1.8_25. Its a virtual servers running on vmware esx.
>> The application have been running fine for 3 years, but started with this
>> a few weeks ago.
>> Anyone have any ideas on were to start troubleshooting this?
>>
>> My clusterconfig:
>> remote {
>>log-sent-messages = on
>>log-received-messages = on
>>log-remote-lifecycle-events = off
>>netty.tcp {
>>port = 45000
>> }
>> watch-failure-detector.threshold = 15
>>}
>>
>> cluster {
>> log-info = on
>> auto-down = on
>> auto-join = on
>> failure-detector {
>> threshold = 15
>> min-std-deviation = 500 ms
>> acceptable-heartbeat-pause = 20 s
>> }
>> heartbeat-request {
>> grace-period = 20 s
>> expected-response-after = 10 s
>> time-to-live = 60 s
>> }
>> use-dispatcher = cluster-dispatcher
>> }
>> }
>>
>> cluster-dispatcher {
>> type = "Dispatcher"
>> executor = "fork-join-executor"
>> fork-join-executor {
>> parallelism-min = 2
>> parallelism-max = 4
>> }
>> }
>>
>>
>> logoutput:
>> node1:
>> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>
>> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
>>
>> or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86
>> ),Actor[ak
>>
>> ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointMa
Re: [akka-user] Akka cluster unreachable
I know, we have a few problems with upgradeing to 2.3.9. (we might wait
until eventstore has a cluster connector for java)
The only thing i can see is that the nodes have been almost idle for a good
while. But there are a few requests coming in. It dosen work fine for 2-3
days.
Will the state actor die if say a firewall cuts the tcp connection because
its been alive too long?
I will see if operations can monitor more data, and consider upgrading to
2.3.9 again.
On 27 January 2015 at 14:27, Björn Antonsson
wrote:
> Hi Idar,
>
> A lot of things have been fixed in akka since 2.2.4. Would it be possible
> to upgrade to 2.3.9?
>
> From the log that you pasted it seems like the akka protocol state actor
> for the connection to node 2 has died on node 1, before your log starts. Is
> there something unusual going on on the machines, like GC or something else
> that would consume resources at the time of the failure? Are you monitoring
> other health metrics like disk space and swap usage?
>
> B/
>
> On 26 January 2015 at 12:31:19, Idar Borlaug (idar.borl...@gmail.com)
> wrote:
>
> Hi
>
> I have a 2 node akka cluster, which looses connection once every two days.
> Often when the server i almost idle. We are still on akka 2.2.4, using
> oracle java 1.8_25. Its a virtual servers running on vmware esx.
> The application have been running fine for 3 years, but started with this
> a few weeks ago.
> Anyone have any ideas on were to start troubleshooting this?
>
> My clusterconfig:
> remote {
>log-sent-messages = on
>log-received-messages = on
>log-remote-lifecycle-events = off
>netty.tcp {
>port = 45000
> }
> watch-failure-detector.threshold = 15
>}
>
> cluster {
> log-info = on
> auto-down = on
> auto-join = on
> failure-detector {
> threshold = 15
> min-std-deviation = 500 ms
> acceptable-heartbeat-pause = 20 s
> }
> heartbeat-request {
> grace-period = 20 s
> expected-response-after = 10 s
> time-to-live = 60 s
> }
> use-dispatcher = cluster-dispatcher
> }
> }
>
> cluster-dispatcher {
> type = "Dispatcher"
> executor = "fork-join-executor"
> fork-join-executor {
> parallelism-min = 2
> parallelism-max = 4
> }
> }
>
>
> logoutput:
> node1:
> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
>
> dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>
> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
>
> or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>
> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>
> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>
> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
> n.k.e.DeadLetterLogger - Dead letter:
> DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86
> ),Actor[ak
>
> ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
> 2015-01-24 03:30:27,666 [svarut-akka.actor.default-dispatcher-31] ERROR
> a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
> svarut@193.161.171.181:45000] - Marking node(s) as
Re: [akka-user] Akka cluster unreachable
Hi Idar,
A lot of things have been fixed in akka since 2.2.4. Would it be possible to
upgrade to 2.3.9?
>From the log that you pasted it seems like the akka protocol state actor for
>the connection to node 2 has died on node 1, before your log starts. Is there
>something unusual going on on the machines, like GC or something else that
>would consume resources at the time of the failure? Are you monitoring other
>health metrics like disk space and swap usage?
B/
On 26 January 2015 at 12:31:19, Idar Borlaug (idar.borl...@gmail.com) wrote:
Hi
I have a 2 node akka cluster, which looses connection once every two days.
Often when the server i almost idle. We are still on akka 2.2.4, using oracle
java 1.8_25. Its a virtual servers running on vmware esx.
The application have been running fine for 3 years, but started with this a few
weeks ago.
Anyone have any ideas on were to start troubleshooting this?
My clusterconfig:
remote {
log-sent-messages = on
log-received-messages = on
log-remote-lifecycle-events = off
netty.tcp {
port = 45000
}
watch-failure-detector.threshold = 15
}
cluster {
log-info = on
auto-down = on
auto-join = on
failure-detector {
threshold = 15
min-std-deviation = 500 ms
acceptable-heartbeat-pause = 20 s
}
heartbeat-request {
grace-period = 20 s
expected-response-after = 10 s
time-to-live = 60 s
}
use-dispatcher = cluster-dispatcher
}
}
cluster-dispatcher {
type = "Dispatcher"
executor = "fork-join-executor"
fork-join-executor {
parallelism-min = 2
parallelism-max = 4
}
}
logoutput:
node1:
2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
/svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86),Actor[ak
ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
2015-01-24 03:30:27,666 [svarut-akka.actor.default-dispatcher-31] ERROR
a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://svarut@193.161.171.181:45000]
- Marking node(s) as UNREACHABLE [Memb
er(address = akka.tcp://svarut@193.161.171.182:45000, status = Up)]
2015-01-24 03:30:28,504 [svarut-akka.actor.default-dispatcher-17] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://svarut@193.161.171.181:45000]
- Leader is marking unreachable node
[akka.tcp://svarut@193.161.171.182:45000] as [Down]
2015-01-24 03:30:28,506 [svarut-akka.actor.default-dispatcher-25] INFO
Cluster(akka://svarut) - Cluster Node [akka.tcp://svarut@193.161.171.181:45000]
- Leader is removing unreachable node
[akka.tcp://svarut@193.161.171.182:45000]
Node2:
2015-01-24 03:30:02,040 [svarut-akka.actor.default-dispatcher-12] WARN
n.k.e.DeadLetterLogger - Dead letter:
DeadLetter
