I have upgraded to 2.3.9 it have now been running for 5 days, without any
problems.
On 28 January 2015 at 11:50, Idar Borlaug <idar.borl...@gmail.com> wrote:
> I know, we have a few problems with upgradeing to 2.3.9. (we might wait
> until eventstore has a cluster connector for java)
>
> The only thing i can see is that the nodes have been almost idle for a
> good while. But there are a few requests coming in. It dosen work fine for
> 2-3 days.
>
> Will the state actor die if say a firewall cuts the tcp connection because
> its been alive too long?
>
> I will see if operations can monitor more data, and consider upgrading to
> 2.3.9 again.
>
> On 27 January 2015 at 14:27, Björn Antonsson <bjorn.antons...@typesafe.com
> > wrote:
>
>> Hi Idar,
>>
>> A lot of things have been fixed in akka since 2.2.4. Would it be possible
>> to upgrade to 2.3.9?
>>
>> From the log that you pasted it seems like the akka protocol state actor
>> for the connection to node 2 has died on node 1, before your log starts. Is
>> there something unusual going on on the machines, like GC or something else
>> that would consume resources at the time of the failure? Are you monitoring
>> other health metrics like disk space and swap usage?
>>
>> B/
>>
>> On 26 January 2015 at 12:31:19, Idar Borlaug (idar.borl...@gmail.com)
>> wrote:
>>
>> Hi
>>
>> I have a 2 node akka cluster, which looses connection once every two
>> days. Often when the server i almost idle. We are still on akka 2.2.4,
>> using oracle java 1.8_25. Its a virtual servers running on vmware esx.
>> The application have been running fine for 3 years, but started with this
>> a few weeks ago.
>> Anyone have any ideas on were to start troubleshooting this?
>>
>> My clusterconfig:
>> remote {
>> log-sent-messages = on
>> log-received-messages = on
>> log-remote-lifecycle-events = off
>> netty.tcp {
>> port = 45000
>> }
>> watch-failure-detector.threshold = 15
>> }
>>
>> cluster {
>> log-info = on
>> auto-down = on
>> auto-join = on
>> failure-detector {
>> threshold = 15
>> min-std-deviation = 500 ms
>> acceptable-heartbeat-pause = 20 s
>> }
>> heartbeat-request {
>> grace-period = 20 s
>> expected-response-after = 10 s
>> time-to-live = 60 s
>> }
>> use-dispatcher = cluster-dispatcher
>> }
>> }
>>
>> cluster-dispatcher {
>> type = "Dispatcher"
>> executor = "fork-join-executor"
>> fork-join-executor {
>> parallelism-min = 2
>> parallelism-max = 4
>> }
>> }
>>
>>
>> logoutput:
>> node1:
>> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(heartbeat-timer,HeartbeatTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>> 2015-01-24 03:30:02,242 [svarut-akka.actor.default-dispatcher-25] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>
>> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
>>
>> or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.182%3A53119-2#-1998086621])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Timer(AckIdleTimer,AckIdleCheckTimer,true,0),Actor[akka://svarut/dea
>>
>> dLetters],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:02,312 [svarut-akka.actor.default-dispatcher-24] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(TakeOver(akka.remote.transport.AkkaProtocolHandle@40711a86
>> ),Actor[ak
>>
>> ka://svarut/system/endpointManager#1220933312],Actor[akka://svarut/system/endpointManager/endpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.182%3A45000-1#1403595424])
>> 2015-01-24 03:30:27,666 [svarut-akka.actor.default-dispatcher-31] ERROR
>> a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
>> svarut@193.161.171.181:45000] - Marking node(s) as UNREACHABLE [Memb
>> er(address = akka.tcp://svarut@193.161.171.182:45000, status = Up)]
>> 2015-01-24 03:30:28,504 [svarut-akka.actor.default-dispatcher-17] INFO
>> Cluster(akka://svarut) - Cluster Node [akka.tcp://
>> svarut@193.161.171.181:45000] - Leader is marking unreachable node
>> [akka.tcp://svarut@193.161.171.182:45000] as [Down]
>> 2015-01-24 03:30:28,506 [svarut-akka.actor.default-dispatcher-25] INFO
>> Cluster(akka://svarut) - Cluster Node [akka.tcp://
>> svarut@193.161.171.181:45000] - Leader is removing unreachable node
>> [akka.tcp://svarut@193.161.171.182:45000]
>>
>> Node2:
>> 2015-01-24 03:30:02,040 [svarut-akka.actor.default-dispatcher-12] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Act
>>
>> or[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903])
>> 2015-01-24 03:30:02,042 [svarut-akka.actor.default-dispatcher-21] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>
>> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-3#72034903])
>> 2015-01-24 03:30:28,636 [svarut-akka.actor.default-dispatcher-2] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(DisassociateUnderlying(Unknown),Actor[akka://svarut/deadLetters],Acto
>>
>> r[akka://svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980])
>> 2015-01-24 03:30:28,637 [svarut-akka.actor.default-dispatcher-18] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>
>> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A44793-4#-747477980])
>> 2015-01-24 03:30:41,457 [svarut-akka.actor.default-dispatcher-22] INFO
>> Cluster(akka://svarut) - Cluster Node [akka.tcp://
>> svarut@193.161.171.182:45000] - Trigger extra expected heartbeat fr
>> om [akka.tcp://svarut@193.161.171.181:45000]
>> 2015-01-24 03:31:06,471 [svarut-akka.actor.default-dispatcher-2] ERROR
>> a.c.ClusterCoreDaemon - Cluster Node [akka.tcp://
>> svarut@193.161.171.182:45000] - Marking node(s) as UNREACHABLE [Membe
>> r(address = akka.tcp://svarut@193.161.171.181:45000, status = Up)]
>> 2015-01-24 03:31:07,445 [svarut-akka.actor.default-dispatcher-12] INFO
>> Cluster(akka://svarut) - Cluster Node [akka.tcp://
>> svarut@193.161.171.182:45000] - Leader is marking unreachable node
>> [akka.tcp://svarut@193.161.171.181:45000] as [Down]
>> 2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-12] INFO
>> Cluster(akka://svarut) - Cluster Node [akka.tcp://
>> svarut@193.161.171.182:45000] - Leader is removing unreachable node
>> [akka.tcp://svarut@193.161.171.181:45000]
>> 2015-01-24 03:31:07,446 [svarut-akka.actor.default-dispatcher-2] INFO
>> Remoting - Address [akka.tcp://svarut@193.161.171.181:45000] is now
>> quarantined, all messages to this address will be
>> delivered to dead letters.
>> 2015-01-24 03:31:07,516 [svarut-akka.actor.default-dispatcher-12] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>
>> /svarut/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-1/endpointWriter/endpointReader-akka.tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-0#-
>> 1902872711])
>> 2015-01-24 03:31:07,517 [svarut-akka.actor.default-dispatcher-12] WARN
>> n.k.e.DeadLetterLogger - Dead letter:
>> DeadLetter(Disassociated(Unknown),Actor[akka://svarut/deadLetters],Actor[akka:/
>>
>> /svarut/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fsvarut%40193.161.171.181%3A45000-5#-2023131039])
>>
>> --
>> Idar
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+unsubscr...@googlegroups.com.
>> To post to this group, send email to akka-user@googlegroups.com.
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> Björn Antonsson
>> Typesafe <http://typesafe.com/> – Reactive Apps on the JVM
>> twitter: @bantonsson <http://twitter.com/#!/bantonsson>
>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+unsubscr...@googlegroups.com.
>> To post to this group, send email to akka-user@googlegroups.com.
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Idar
>
--
Idar
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
