Hi,
On Mon, Oct 24, 2016 at 6:42 PM, <fh...@mails.tsinghua.edu.cn> wrote: > Hi guys, > I'm recently studying the source code of akka-cluster and there's > something that I really cannot understand. > Actually, I was trying to reproduce what akka-cluster document said "If > system messages cannot be delivered to a node it will be quarantined and > then it cannot come back from unreachable". So I've down these steps: > > 1. Create Actor A on node1, Actor B on node2; > 2. Let A watch B; > 3. Cut the wire between node1 and node2; > 4. Let B send a user message, namely UMSG, to A; > 5. Close B, and I've seen that Actor B tried to send a message with > type DeathWatchNotification; > 6. Reconnect node1 and node2, and the cluster "revived" with both > nodes had changed state of each other from UNREACHABLE to UP; > 7. ActorA immediately received Terminated message from ActorB, while > it didn't receive the user message (UMSG, see Step4) from B. > > Yes, this is exactly how it should work. User messages have no guaranteed delivery, and hence they are simply dropped during connection problems. System messages (like Watch and DeathWatchNotification) on the other hand are buffered until they are acknowledged by remote system. Once the connection comes back, they are attempted to be redelivered (as you see at step 7). So what happens 2. Watch sent from A to B and acknowledged as received by B 4. UMSG is dropped as connection is severed 5. DeathWatchNotification is attempted to be sent by B, but ack does not arrive 7. above DWN is resent, processed by actor A and acknowledged the reception of the message > Here is my questions > > 1. DeathWatchNotification is also a kind of system message, and, which > might not be true, it failed issuing to its watcher. According to the > document mentioned above, the state of the node containing the watcher > actor should have been put to DOWN, but it didn't. Why is this? Where can I > find the code related to this behavior? > > This is a misunderstanding. System message delivery is only considered failed once all the buffer space designated for containing system messages for redelivery are depleted. Otherwise these messages are kept around for a long time. If a remote system is not responding for a long time then it is considered dead. This can happen in two ways: - In ordinary remoting DeathWatch is sending heartbeats between systems of the watcher and watchee. If a limit is reached (watch-failure-detector) then the remote system is considered dead, is quarantined and all system messages are dropped, and watched actors declared dead. - if cluster is used (your case), cluster nodes monitor each other via heartbeats, and if there are missed heartbeats from a system, members designate that as unreachable. This is not the final state and recoverable, it is only if a system is DOWN-ed when it is deemed dead, and watcher actors are informed of this fact. DOWNing can be manual, programmatic, or by using auto-downing: http://doc.akka.io/docs/akka/2.4/scala/cluster-usage.html#Auto-downing__DO_NOT_USE_ > > 1. Who is the real actor that sends this Terminated system message? > > It is converted to a user message from the DWN system message. > > 1. I've noted that in akka-cluster both system message and user > message are delivered in a same way using akka-remote. So why is UMSG > dropped but Terminated received? > > See above. > > Thanks for helping in advance > Hangyu > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c > urrent/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
