Hello Martynas,
Well, I think I can rule this option out because:
- without any load on the system (my scenario 1 in my orig. post) a restart
works fine.
- also, most of the time node3 can send back messages to node2. but node3
does not send to node1. (however, sometimes both nodes, node1 and node2 do
not hear back from node3)
- and we also tried with ActorSelection and it did not work.
is it suspicious to see Disassociated messages? or is this just a symptom?
Thanks,
Robert
On Friday, November 7, 2014 9:15:42 AM UTC-8, Martynas Mickevičius wrote:
>
> Hi Robert,
>
> as you mentioned and from the logs you provided its seems that messages
> are flowing from node{1,2} to node3 after restart, but not to the other
> direction.
>
> Would it be possible that your application tries to send messages from
> node3 to node{1,2} using ActorRefs which were resolved before the restart
> of node{1,2}? ActorRef includes actor UID which changes after Actor is
> stopped and started again, which happens upon node restart. Here is
> <https://github.com/2m/sandbox-akka-remote/blob/fd61036875bcb622e6a5657c16d879bcc7b6b21b/src/main/scala/NodeRestart.scala>
>
> a quick example code that illustrates that situation.
>
> If so, you should send messages using ActorSelection or re-resolve
> ActorRefs after node restart or periodically.
>
> On Fri, Nov 7, 2014 at 3:58 AM, Robert Preissl <rob...@ticketfly.com
> <javascript:>> wrote:
>
>> Hello Endre,
>>
>> First of all, thanks for replying so quickly!
>>
>> Second, I need to mention that we use Akka remoting. and not Akka
>> clustering (yet). not sure if this makes a difference.
>>
>> What I mean that in our restart scenario (where first node1 and node2 are
>> simultaneously restarted. and then node3) when node1 and node2 are coming
>> back up, it seems that the connection node1 -> node3 works fine. but the
>> connection node3 -> node1 does not.
>>
>> So, to answer your question, yes, it seems we loose messages from node3.
>>
>> I attached more detail logs below. (and please excuse the many log lines;
>> i tried to clean it up as much as possible)
>>
>> what is interesting to see is this line:
>> *processing Event(Disassociated [akka.tcp://DivaPCluster@10.57.0.43:8900
>> <http://DivaPCluster@10.57.0.43:8900>] ->
>> [akka.tcp://DivaPCluster@10.57.0.41:8900
>> <http://DivaPCluster@10.57.0.41:8900>]*
>>
>> 10.57.0.43 is node3. and 10.57.0.41 is node1, by the way.
>>
>> so, the connection between node3 and node1 is Disassociated; which
>> explains maybe why node1 never hears back from node3 when it tries to sync
>> up.
>>
>> We looked a bit in the akka source code and found that stopping an
>> EndpointWriter (I think) triggers a "Disassociated" to be fired, right? and
>> we can see this stop in a few log lines above:
>>
>> *[akka://DivaPCluster/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FDivaPCluster%4010.57.0.41%3A8900-0/endpointWriter]
>>
>> akka.remote.EndpointWriter - stopping*
>>
>> so, why is it stopping? is this our problem here?
>>
>> the logs from node3 are attached as a file.
>>
>> Thank you!
>>
>> Robert
>>
>>
>>
>> On Wednesday, November 5, 2014 12:55:04 PM UTC-8, Robert Preissl wrote:
>>>
>>> hello!
>>>
>>> I am having a problem in my remote Akka production system, which
>>> consists of 3 nodes running with the latest version of Akka (2.3.6.):
>>>
>>> In more details, I am experiencing errors with "*rolling restarts*" of
>>> the cluster (for deployment, etc. we cannot afford any downtime), where a
>>> restart happens in the following sequence
>>> 1.) restart node1 and node2.
>>> 2.) once 1. completed, restart node3.
>>>
>>> *but we only observe failures once there is a load (even small load) on
>>> the system*. So, I want to describe two scenarios:
>>>
>>>
>>> *Scenario 1 - no load on the system: Restart works.*
>>>
>>> if there is no load on the system at all, the restarting seems to work
>>> fine. I.e., with detailed logging I can observe that node3 logs the
>>> following events: (in chronological order)
>>>
>>> 13:09:48.769 WARN [akka.tcp://DivaPCluster@NODE_
>>> 3:8900/system/endpointManager/reliableEndpointWriter-akka.tcp0-1]
>>> akka.remote.ReliableDeliverySupervisor - Association with remote system
>>> [akka.tcp://DivaPCluster@NODE_2:8900] has failed, address is now gated
>>> for [5000] ms. Reason is: [Disassociated].
>>> 13:09:48.823 WARN [akka.tcp://DivaPCluster@NODE_
>>> 3:8900/system/endpointManager/reliableEndpointWriter-akka.tcp0-0]
>>> akka.remote.ReliableDeliverySupervisor - Association with remote system
>>> [akka.tcp://DivaPCluster@NODE_1:8900] has failed, address is now gated
>>> for [5000] ms. Reason is: [Disassociated].
>>>
>>> 13:10:10.661 DEBUG [Remoting] Remoting - Associated
>>> [akka.tcp://DivaPCluster@NODE_3:8900] <- [akka.tcp://DivaPCluster@NODE_
>>> 2:8900]
>>> 13:10:10.987 DEBUG [Remoting] Remoting - Associated
>>> [akka.tcp://DivaPCluster@NODE_3:8900] <- [akka.tcp://DivaPCluster@NODE_
>>> 1:8900]
>>>
>>> Since node1 and node2 restart, it is fine that the association is gated
>>> between node3 -> node1 (and between node3 -> node2) for a while.
>>> And I assume it becomes active again since "a successful inbound
>>> connection is accepted from a remote system during Gate it automatically
>>> transitions to Active" (as you describe in http://doc.akka.io/docs/akka/
>>> snapshot/java/remoting.html)
>>>
>>> this can be verified since I can see the logs on node1 that it tries to
>>> connect at this point in time after the restart: 13:10:10.861 (and the
>>> connection becomes active on node3; managing node3 -> node1; at time
>>> 13:10:10.987 as you can see above)
>>>
>>> so, everything cool here and the system restarts fine!
>>>
>>>
>>>
>>> *Scenario 2 - easy load on the system: Restart fails due to
>>> Unrecoverable "gated" state*
>>>
>>> Similar to Scenario 1 above, I can observe the "gated" messages for
>>> links node3 -> node1 and node3 -> node2.
>>>
>>> However, I never see that the links become active again! and the restart
>>> never recovers and I need to manually stop my nodes and start up again.
>>>
>>> This is surprising since I clearly see that node1 and node2 (after they
>>> restarted) send message to node3. and node3 successfully logs the reception
>>> of these messages.
>>>
>>> So, why does in this scenario the connection not become active again??
>>> It is a successful inbound connection that should make the link active
>>> again as you describe on your site?
>>>
>>>
>>>
>>> Any help on this is greatly appreciated. otherwise we need to roll back
>>> to Scala 2.10 (or 2.9) and an older version of Akka.
>>>
>>> Thanks,
>>> Robert
>>>
>> --
>> >>>>>>>>>> Read the docs: http://akka.io/docs/
>> >>>>>>>>>> Check the FAQ:
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to akka-user+...@googlegroups.com <javascript:>.
>> To post to this group, send email to akka...@googlegroups.com
>> <javascript:>.
>> Visit this group at http://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Martynas Mickevičius
> Typesafe <http://typesafe.com/> – Reactive
> <http://www.reactivemanifesto.org/> Apps on the JVM
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
