Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

I have a supervisor actor per node which is already subscribed to cluster 
events,
I avoided using ask when the actor system is going down anyway and it is 
driven by the JVM shutdown hook,
so instead I came up with this hack, maybe a bit ugly but it works:

For supervisor actor which once it receives MemberRemoved for itself it 
terminates the actor system:
  if (message instanceof MemberRemoved) {
final Member member = ((MemberRemoved) message).member();
final ActorSystem system = context().system();
if (Cluster.get(system).selfAddress().equals(member.address())) {
  // It was me that left so, terminate actor system.
  system.terminate();
} else {
  // Invalidates that other supervisor cache that left.
  removeSupervisor(member);
}
  }

For main class which once the system is shutdown it also shutdowns the 
logger manager:
  public static void registerShutdownHook(ActorSystem system) {
final CountDownLatch latch = new CountDownLatch(1);
system.registerOnTermination(() -> {
  try {
Configurator.shutdown((LoggerContext) LogManager.getContext());
  } finally {
latch.countDown();
  }
});
Runtime.getRuntime().addShutdownHook(new Thread() {
  @Override
  public void run() {
final Cluster cluster = Cluster.get(system);
cluster.leave(cluster.selfAddress());
try {
  latch.await();
} catch (InterruptedException ignored) {
}
  }
});
  }

Regards,

Guido.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

You could if system.terminate() could do that for you as part of the 
cluster extension then I could simply Await for Future to be 
ready.

On Monday, March 21, 2016 at 9:36:43 AM UTC, Akka Team wrote:
>
> But then you cannot expect a graceful removal from the cluster. 
>
> -Endre
>
> On Fri, Mar 18, 2016 at 6:47 PM, Guido Medina  > wrote:
>
>> Won't happen, the micro-services are being sent Linux SIGTERM, which is 
>> why I'm hooking on the JVM shutdown.
>>
>> On Friday, March 18, 2016 at 5:26:01 PM UTC, Konrad Malawski wrote:
>>>
>>> I precisely explained for what event you need to wait :-)
>>>
>>> Proper graceful shutdown means you need to wait for *_your own_ address* 
>>> to come back as *MemberRemoved* event,
>>> and THEN you can shut down the actor system and JVM.
>>>
>>>
>>> -- 
>>> Cheers,
>>> Konrad 'ktoso’ Malawski
>>> Akka  @ Lightbend 
>>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Akka Team
> Typesafe - Reactive apps on the JVM
> Blog: letitcrash.com
> Twitter: @akkateam
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Akka Team]

But then you cannot expect a graceful removal from the cluster.

-Endre

On Fri, Mar 18, 2016 at 6:47 PM, Guido Medina  wrote:

> Won't happen, the micro-services are being sent Linux SIGTERM, which is
> why I'm hooking on the JVM shutdown.
>
> On Friday, March 18, 2016 at 5:26:01 PM UTC, Konrad Malawski wrote:
>>
>> I precisely explained for what event you need to wait :-)
>>
>> Proper graceful shutdown means you need to wait for *_your own_ address*
>> to come back as *MemberRemoved* event,
>> and THEN you can shut down the actor system and JVM.
>>
>>
>> --
>> Cheers,
>> Konrad 'ktoso’ Malawski
>> Akka  @ Lightbend 
>>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Akka Team
Typesafe - Reactive apps on the JVM
Blog: letitcrash.com
Twitter: @akkateam

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

How long is enough time? I tried sleep 1 sec between leave and terminate 
calls and it didn't work.

On Friday, March 18, 2016 at 5:13:50 PM UTC, Konrad Malawski wrote:
>
> Also, please don't do that:
>cluster.leave(cluster.selfAddress());
> system.terminate();
> It is wrong.
>
> You need to give the cluster gossip enough time to complete the leaving 
> "nicely".
>
> Proper graceful shutdown means you need to wait for _your own_ address to 
> come back as MemberRemoved event,
> and THEN you can shut down the actor system and JVM.
>
> Related: 
> this mistake is exactly something I warn against in my recent talk "Zen of 
> Akka", you may enjoy it :-)
> https://youtu.be/tC-joPMPJLs
>
>
>
> On Fri, Mar 18, 2016 at 5:58 PM, Patrik Nordwall  > wrote:
>
> Hi Ben
>
> There are some subtle exceptions to the convergence rule. Unreachable 
> members with status Down or Exiting are not counted.
>
> /Patrik
>
> On Thu, Mar 17, 2016 at 11:25 PM, Guido Medina  > wrote:
>
> In fact in the other nodes you get the following:
>
> INFO  22:10:20,383 ster) - Cluster Node [akka.tcp://
> DevCluster@127.0.0.1:2552] - Leader is moving node [akka.tcp://
> DevCluster@127.0.0.1:38845] to [Exiting]
> INFO  22:10:20,384 orRef - Message [akka.cluster.GossipEnvelope] from 
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
> Actor[akka://DevCluster/deadLetters] was not delivered. [1] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:20,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>  
> to Actor[akka://DevCluster/deadLetters] was not delivered. [2] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:21,383 orRef - Message [akka.cluster.GossipEnvelope] from 
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
> Actor[akka://DevCluster/deadLetters] was not delivered. [3] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:21,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>  
> to Actor[akka://DevCluster/deadLetters] was not delivered. [4] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:22,383 orRef - Message [akka.cluster.GossipEnvelope] from 
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
> Actor[akka://DevCluster/deadLetters] was not delivered. [5] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:22,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>  
> to Actor[akka://DevCluster/deadLetters] was not delivered. [6] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:23,383 orRef - Message [akka.cluster.GossipEnvelope] from 
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
> Actor[akka://DevCluster/deadLetters] was not delivered. [7] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:23,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>  
> to Actor[akka://DevCluster/deadLetters] was not delivered. [8] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:24,382 orRef - Message [akka.cluster.GossipEnvelope] from 
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
> Actor[akka://DevCluster/deadLetters] was not delivered. [9] dead letters 
> encountered. This logging can be turned off or adjusted with configuration 
> settings 'akka.log-dead-letters' and 
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:24,384 ster) - Cluster Node [akka.tcp://
> DevCluster@127.0.0.1:2552] - Marking exiting node(s) as UNREACHABLE 
> [Member(address = 

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Konrad Malawski]

Also, please don't do that:
   cluster.leave(cluster.selfAddress());
system.terminate();
It is wrong.

You need to give the cluster gossip enough time to complete the leaving
"nicely".

Proper graceful shutdown means you need to wait for _your own_ address to
come back as MemberRemoved event,
and THEN you can shut down the actor system and JVM.

Related:
this mistake is exactly something I warn against in my recent talk "Zen of
Akka", you may enjoy it :-)
https://youtu.be/tC-joPMPJLs



On Fri, Mar 18, 2016 at 5:58 PM, Patrik Nordwall 
wrote:

> Hi Ben
>
> There are some subtle exceptions to the convergence rule. Unreachable
> members with status Down or Exiting are not counted.
>
> /Patrik
>
> On Thu, Mar 17, 2016 at 11:25 PM, Guido Medina  wrote:
>
>> In fact in the other nodes you get the following:
>>
>> INFO  22:10:20,383 ster) - Cluster Node [akka.tcp://
>> DevCluster@127.0.0.1:2552] - Leader is moving node [akka.tcp://
>> DevCluster@127.0.0.1:38845] to [Exiting]
>> INFO  22:10:20,384 orRef - Message [akka.cluster.GossipEnvelope] from
>> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
>> Actor[akka://DevCluster/deadLetters] was not delivered. [1] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:20,403 orRef - Message [akka.cluster.
>> ClusterHeartbeatSender$Heartbeat] from 
>> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>> to Actor[akka://DevCluster/deadLetters] was not delivered. [2] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:21,383 orRef - Message [akka.cluster.GossipEnvelope] from
>> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
>> Actor[akka://DevCluster/deadLetters] was not delivered. [3] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:21,403 orRef - Message [akka.cluster.
>> ClusterHeartbeatSender$Heartbeat] from 
>> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>> to Actor[akka://DevCluster/deadLetters] was not delivered. [4] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:22,383 orRef - Message [akka.cluster.GossipEnvelope] from
>> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
>> Actor[akka://DevCluster/deadLetters] was not delivered. [5] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:22,403 orRef - Message [akka.cluster.
>> ClusterHeartbeatSender$Heartbeat] from 
>> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>> to Actor[akka://DevCluster/deadLetters] was not delivered. [6] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:23,383 orRef - Message [akka.cluster.GossipEnvelope] from
>> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
>> Actor[akka://DevCluster/deadLetters] was not delivered. [7] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:23,403 orRef - Message [akka.cluster.
>> ClusterHeartbeatSender$Heartbeat] from 
>> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
>> to Actor[akka://DevCluster/deadLetters] was not delivered. [8] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:24,382 orRef - Message [akka.cluster.GossipEnvelope] from
>> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
>> Actor[akka://DevCluster/deadLetters] was not delivered. [9] dead letters
>> encountered. This logging can be turned off or adjusted with configuration
>> settings 'akka.log-dead-letters' and
>> 'akka.log-dead-letters-during-shutdown'.
>> INFO  22:10:24,384 ster) - Cluster Node [akka.tcp://
>> DevCluster@127.0.0.1:2552] - Marking exiting node(s) as UNREACHABLE
>> [Member(address = akka.tcp://DevCluster@127.0.0.1:38845, status =
>> Exiting)]. This is expected and they will be removed.
>> INFO  22:10:24,403 orRef - Message [akka.cluster.
>> ClusterHeartbeatSender$Heartbeat] from 
>> 

[akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

Hi Benjamin,

I also rely on cluster events and AFAIK you can expect (and trust) 
*MemberUp* and *MemberRemoved*, these IMHO are the only two consistent 
states you can trust.
In other words, I register some actors only when their nodes reach 
*MemberUp* and unregister only when their nodes reach *MemberRemoved*
Any other state in between I would treat them as information only.

So far I haven't got any issue with my mini-shard implementation relying on 
these only 2 statuses, the draw back is that it will only have to wait for 
a longer time to react.

HTH,

Guido.

On Thursday, March 17, 2016 at 6:07:48 PM UTC, Benjamin Black wrote:
>
> Hello,
>
> I'm adding logic to our service so that when a node is being restarted it 
> gracefully leaves the cluster using cluster.leave(cluster.selfAddress). 
> In the cluster specification doc it states:
>
> If a node is unreachable then gossip convergence is not possible and 
> therefore any leader actions are also not possible (for instance, 
> allowing a node to become a part of the cluster). To be able to move 
> forward the state of theunreachable nodes must be changed. It must become 
> reachable again or marked as down
>
> Is this totally true? If a node is unreachable and is the 
> leaving/exiting/removed state will this stop the leader from adding a new 
> node? I ask because I have an actor that subscribes to cluster events and I 
> can see a node is being added whilst another node is considered unreachable 
> and in the exiting status:
>
> 14:02:46.843 INFO  Exited member Member(address = akka.tcp://
> geyser@172.16.120.160:7000, status = Exiting)
> 14:02:51.842 INFO  Unreachable member Member(address = akka.tcp://
> geyser@172.16.120.160:7000, status = Exiting)
> 14:02:53.843 INFO  Removing member Member(address = akka.tcp://
> geyser@172.16.120.160:7000, status = Removed)
> 14:02:57.843 INFO  Exited member Member(address = akka.tcp://
> geyser@172.16.119.46:7000, status = Exiting)
> 14:03:02.760 INFO  Unreachable member Member(address = akka.tcp://
> geyser@172.16.119.46:7000, status = Exiting)
> 14:03:04.843 INFO  Adding member Member(address = akka.tcp://
> geyser@172.16.120.160:7000, status = Up)
>
> Thanks,
> Ben
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Patrik Nordwall]

Hi Ben

There are some subtle exceptions to the convergence rule. Unreachable
members with status Down or Exiting are not counted.

/Patrik

On Thu, Mar 17, 2016 at 11:25 PM, Guido Medina  wrote:

> In fact in the other nodes you get the following:
>
> INFO  22:10:20,383 ster) - Cluster Node [akka.tcp://
> DevCluster@127.0.0.1:2552] - Leader is moving node [akka.tcp://
> DevCluster@127.0.0.1:38845] to [Exiting]
> INFO  22:10:20,384 orRef - Message [akka.cluster.GossipEnvelope] from
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
> Actor[akka://DevCluster/deadLetters] was not delivered. [1] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:20,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
> to Actor[akka://DevCluster/deadLetters] was not delivered. [2] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:21,383 orRef - Message [akka.cluster.GossipEnvelope] from
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
> Actor[akka://DevCluster/deadLetters] was not delivered. [3] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:21,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
> to Actor[akka://DevCluster/deadLetters] was not delivered. [4] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:22,383 orRef - Message [akka.cluster.GossipEnvelope] from
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
> Actor[akka://DevCluster/deadLetters] was not delivered. [5] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:22,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
> to Actor[akka://DevCluster/deadLetters] was not delivered. [6] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:23,383 orRef - Message [akka.cluster.GossipEnvelope] from
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
> Actor[akka://DevCluster/deadLetters] was not delivered. [7] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:23,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
> to Actor[akka://DevCluster/deadLetters] was not delivered. [8] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:24,382 orRef - Message [akka.cluster.GossipEnvelope] from
> Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to
> Actor[akka://DevCluster/deadLetters] was not delivered. [9] dead letters
> encountered. This logging can be turned off or adjusted with configuration
> settings 'akka.log-dead-letters' and
> 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:24,384 ster) - Cluster Node [akka.tcp://
> DevCluster@127.0.0.1:2552] - Marking exiting node(s) as UNREACHABLE
> [Member(address = akka.tcp://DevCluster@127.0.0.1:38845, status =
> Exiting)]. This is expected and they will be removed.
> INFO  22:10:24,403 orRef - Message [akka.cluster.
> ClusterHeartbeatSender$Heartbeat] from 
> Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769]
> to Actor[akka://DevCluster/deadLetters] was not delivered. [10] dead
> letters encountered, no more dead letters will be logged. This logging can
> be turned off or adjusted with configuration settings
> 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
> INFO  22:10:25,387 ster) - Cluster Node [akka.tcp://
> DevCluster@127.0.0.1:2552] - Leader is removing exiting node [akka.tcp://
> DevCluster@127.0.0.1:38845]
>
> Notice the 1st status is [Exiting], then you see from [Exiting] to
> [Unreachable] and then you get [MemberRemoved] which isn't logged by Akka
> but I can assure you it happens, and 

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

Won't happen, the micro-services are being sent Linux SIGTERM, which is why 
I'm hooking on the JVM shutdown.

On Friday, March 18, 2016 at 5:26:01 PM UTC, Konrad Malawski wrote:
>
> I precisely explained for what event you need to wait :-)
>
> Proper graceful shutdown means you need to wait for *_your own_ address* 
> to come back as *MemberRemoved* event,
> and THEN you can shut down the actor system and JVM.
>
>
> -- 
> Cheers,
> Konrad 'ktoso’ Malawski
> Akka  @ Lightbend 
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Clarification on unreachable nodes in cluster

[Benjamin Black]

Hi Guido,

I think in your case you are shutting down before the node has communicated 
to the leader that it wants to leave. I wait to get the MemberExited 
message before shutting down the node. Maybe I should wait for the 
MemberRemoved? Either way the ultimate aim is to not have the unreachable 
logic kick in and have to wait x seconds (I use 10 seconds) for the node to 
be auto downed by the leader. And the reason why I don't want to wait is 
according to the docs the leader wouldn't be able to add nodes whilst any 
node in the cluster is considered unreachable, which is a problem if I'm 
doing a rolling restart of all the nodes.

Regards,
Ben

On Thursday, March 17, 2016 at 4:51:30 PM UTC-4, Guido Medina wrote:
>
> As for cluster.leave(cluster.selfAddress) my micro-services use the 
> following to leave:
>
> Runtime.getRuntime().addShutdownHook(new Thread() {
>   @Override
>   public void run() {
> final Cluster cluster = Cluster.get(system);
> cluster.leave(cluster.selfAddress());
> system.terminate();
> Configurator.shutdown((LoggerContext) LogManager.getContext());
>   }
> });
>
> But honestly I have never seen that work, the other nodes just report it 
> as unreachable until it times out and it is completely removed,
> maybe the shutdown happens so fast that it is useless in my case.
>
> HTH,
>
> Guido.
>
> On Thursday, March 17, 2016 at 8:33:29 PM UTC, Guido Medina wrote:
>>
>> Hi Benjamin,
>>
>> I also rely on cluster events and AFAIK you can expect (and trust) 
>> *MemberUp* and *MemberRemoved*, these IMHO are the only two consistent 
>> states you can trust.
>> In other words, I register some actors only when their nodes reach 
>> *MemberUp* and unregister only when their nodes reach *MemberRemoved*
>> Any other state in between I would treat them as information only.
>>
>> So far I haven't got any issue with my mini-shard implementation relying 
>> on these only 2 statuses, the draw back is that it will only have to wait 
>> for a longer time to react.
>>
>> HTH,
>>
>> Guido.
>>
>> On Thursday, March 17, 2016 at 6:07:48 PM UTC, Benjamin Black wrote:
>>>
>>> Hello,
>>>
>>> I'm adding logic to our service so that when a node is being restarted 
>>> it gracefully leaves the cluster using cluster.leave(cluster
>>> .selfAddress). In the cluster specification doc it states:
>>>
>>> If a node is unreachable then gossip convergence is not possible and 
>>> therefore any leader actions are also not possible (for instance, 
>>> allowing a node to become a part of the cluster). To be able to move 
>>> forward the state of theunreachable nodes must be changed. It must 
>>> become reachable again or marked as down
>>>
>>> Is this totally true? If a node is unreachable and is the 
>>> leaving/exiting/removed state will this stop the leader from adding a new 
>>> node? I ask because I have an actor that subscribes to cluster events and I 
>>> can see a node is being added whilst another node is considered unreachable 
>>> and in the exiting status:
>>>
>>> 14:02:46.843 INFO  Exited member Member(address = akka.tcp://
>>> geyser@172.16.120.160:7000, status = Exiting)
>>> 14:02:51.842 INFO  Unreachable member Member(address = akka.tcp://
>>> geyser@172.16.120.160:7000, status = Exiting)
>>> 14:02:53.843 INFO  Removing member Member(address = akka.tcp://
>>> geyser@172.16.120.160:7000, status = Removed)
>>> 14:02:57.843 INFO  Exited member Member(address = akka.tcp://
>>> geyser@172.16.119.46:7000, status = Exiting)
>>> 14:03:02.760 INFO  Unreachable member Member(address = akka.tcp://
>>> geyser@172.16.119.46:7000, status = Exiting)
>>> 14:03:04.843 INFO  Adding member Member(address = akka.tcp://
>>> geyser@172.16.120.160:7000, status = Up)
>>>
>>> Thanks,
>>> Ben
>>>
>>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

I just tried this:

final Cluster cluster = Cluster.get(system);
cluster.leave(cluster.selfAddress());
try {
  Thread.sleep(1000);
} catch (InterruptedException ignored) {
}
system.terminate();

and it didn't work, still the dead node has to timeout in the others,
in other words I don't see any cluster event message coming to the node 
that just left so by default I'm just shutting down letting gossip failures 
take care of it,
and reacting only upon MemberUp and MemberRemoved instead, I have that 
working for like 10 months now no problem.

Regards,

Guido.

On Thursday, March 17, 2016 at 10:04:27 PM UTC, Benjamin Black wrote:
>
> Hi Guido,
>
> I think in your case you are shutting down before the node has 
> communicated to the leader that it wants to leave. I wait to get the 
> MemberExited message before shutting down the node. Maybe I should wait for 
> the MemberRemoved? Either way the ultimate aim is to not have the 
> unreachable logic kick in and have to wait x seconds (I use 10 seconds) for 
> the node to be auto downed by the leader. And the reason why I don't want 
> to wait is according to the docs the leader wouldn't be able to add nodes 
> whilst any node in the cluster is considered unreachable, which is a 
> problem if I'm doing a rolling restart of all the nodes.
>
> Regards,
> Ben
>
> On Thursday, March 17, 2016 at 4:51:30 PM UTC-4, Guido Medina wrote:
>>
>> As for cluster.leave(cluster.selfAddress) my micro-services use the 
>> following to leave:
>>
>> Runtime.getRuntime().addShutdownHook(new Thread() {
>>   @Override
>>   public void run() {
>> final Cluster cluster = Cluster.get(system);
>> cluster.leave(cluster.selfAddress());
>> system.terminate();
>> Configurator.shutdown((LoggerContext) LogManager.getContext());
>>   }
>> });
>>
>> But honestly I have never seen that work, the other nodes just report it 
>> as unreachable until it times out and it is completely removed,
>> maybe the shutdown happens so fast that it is useless in my case.
>>
>> HTH,
>>
>> Guido.
>>
>> On Thursday, March 17, 2016 at 8:33:29 PM UTC, Guido Medina wrote:
>>>
>>> Hi Benjamin,
>>>
>>> I also rely on cluster events and AFAIK you can expect (and trust) 
>>> *MemberUp* and *MemberRemoved*, these IMHO are the only two consistent 
>>> states you can trust.
>>> In other words, I register some actors only when their nodes reach 
>>> *MemberUp* and unregister only when their nodes reach *MemberRemoved*
>>> Any other state in between I would treat them as information only.
>>>
>>> So far I haven't got any issue with my mini-shard implementation relying 
>>> on these only 2 statuses, the draw back is that it will only have to wait 
>>> for a longer time to react.
>>>
>>> HTH,
>>>
>>> Guido.
>>>
>>> On Thursday, March 17, 2016 at 6:07:48 PM UTC, Benjamin Black wrote:

 Hello,

 I'm adding logic to our service so that when a node is being restarted 
 it gracefully leaves the cluster using cluster.leave(cluster
 .selfAddress). In the cluster specification doc it states:

 If a node is unreachable then gossip convergence is not possible and 
 therefore any leader actions are also not possible (for instance, 
 allowing a node to become a part of the cluster). To be able to move 
 forward the state of theunreachable nodes must be changed. It must 
 become reachable again or marked as down

 Is this totally true? If a node is unreachable and is the 
 leaving/exiting/removed state will this stop the leader from adding a new 
 node? I ask because I have an actor that subscribes to cluster events and 
 I 
 can see a node is being added whilst another node is considered 
 unreachable 
 and in the exiting status:

 14:02:46.843 INFO  Exited member Member(address = akka.tcp://
 geyser@172.16.120.160:7000, status = Exiting)
 14:02:51.842 INFO  Unreachable member Member(address = akka.tcp://
 geyser@172.16.120.160:7000, status = Exiting)
 14:02:53.843 INFO  Removing member Member(address = akka.tcp://
 geyser@172.16.120.160:7000, status = Removed)
 14:02:57.843 INFO  Exited member Member(address = akka.tcp://
 geyser@172.16.119.46:7000, status = Exiting)
 14:03:02.760 INFO  Unreachable member Member(address = akka.tcp://
 geyser@172.16.119.46:7000, status = Exiting)
 14:03:04.843 INFO  Adding member Member(address = akka.tcp://
 geyser@172.16.120.160:7000, status = Up)

 Thanks,
 Ben



-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

[akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

In fact in the other nodes you get the following:

INFO  22:10:20,383 ster) - Cluster Node [akka.tcp://DevCluster@127.0.0.1:2552] 
- Leader is moving node [akka.tcp://DevCluster@127.0.0.1:38845] to [Exiting]
INFO  22:10:20,384 orRef - Message [akka.cluster.GossipEnvelope] from Actor[
akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [1] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:20,403 orRef - Message [akka.cluster.
ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [2] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:21,383 orRef - Message [akka.cluster.GossipEnvelope] from Actor[
akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [3] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:21,403 orRef - Message [akka.cluster.
ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [4] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:22,383 orRef - Message [akka.cluster.GossipEnvelope] from Actor[
akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [5] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:22,403 orRef - Message [akka.cluster.
ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [6] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:23,383 orRef - Message [akka.cluster.GossipEnvelope] from Actor[
akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [7] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:23,403 orRef - Message [akka.cluster.
ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [8] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:24,382 orRef - Message [akka.cluster.GossipEnvelope] from Actor[
akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [9] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 
'akka.log-dead-letters-during-shutdown'.
INFO  22:10:24,384 ster) - Cluster Node [akka.tcp://DevCluster@127.0.0.1:2552] 
- Marking exiting node(s) as UNREACHABLE [Member(address = 
akka.tcp://DevCluster@127.0.0.1:38845, status = Exiting)]. This is expected 
and they will be removed.
INFO  22:10:24,403 orRef - Message [akka.cluster.
ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [10] dead 
letters encountered, no more dead letters will be logged. This logging can 
be turned off or adjusted with configuration settings 
'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:25,387 ster) - Cluster Node [akka.tcp://DevCluster@127.0.0.1:2552] 
- Leader is removing exiting node [akka.tcp://DevCluster@127.0.0.1:38845]

Notice the 1st status is [Exiting], then you see from [Exiting] to 
[Unreachable] and then you get [MemberRemoved] which isn't logged by Akka 
but I can assure you it happens, and if you log the events the last two 
will be:

INFO  22:23:00,968 ster) - Cluster Node [akka.tcp://DevCluster@127.0.0.1:2552] 
- Leader is auto-downing unreachable node 
[akka.tcp://DevCluster@127.0.0.1:54621]
INFO  22:23:00,969 ster) - Cluster Node [akka.tcp://DevCluster@127.0.0.1:2552] 
- Marking unreachable node 

[akka-user] Re: Clarification on unreachable nodes in cluster

[Guido Medina]

As for cluster.leave(cluster.selfAddress) my micro-services use the 
following to leave:

Runtime.getRuntime().addShutdownHook(new Thread() {
  @Override
  public void run() {
final Cluster cluster = Cluster.get(system);
cluster.leave(cluster.selfAddress());
system.terminate();
Configurator.shutdown((LoggerContext) LogManager.getContext());
  }
});

But honestly I have never seen that work, the other nodes just report it as 
unreachable until it times out and it is completely removed,
maybe the shutdown happens so fast that it is useless in my case.

HTH,

Guido.

On Thursday, March 17, 2016 at 8:33:29 PM UTC, Guido Medina wrote:
>
> Hi Benjamin,
>
> I also rely on cluster events and AFAIK you can expect (and trust) 
> *MemberUp* and *MemberRemoved*, these IMHO are the only two consistent 
> states you can trust.
> In other words, I register some actors only when their nodes reach 
> *MemberUp* and unregister only when their nodes reach *MemberRemoved*
> Any other state in between I would treat them as information only.
>
> So far I haven't got any issue with my mini-shard implementation relying 
> on these only 2 statuses, the draw back is that it will only have to wait 
> for a longer time to react.
>
> HTH,
>
> Guido.
>
> On Thursday, March 17, 2016 at 6:07:48 PM UTC, Benjamin Black wrote:
>>
>> Hello,
>>
>> I'm adding logic to our service so that when a node is being restarted it 
>> gracefully leaves the cluster using cluster.leave(cluster.selfAddress). 
>> In the cluster specification doc it states:
>>
>> If a node is unreachable then gossip convergence is not possible and 
>> therefore any leader actions are also not possible (for instance, 
>> allowing a node to become a part of the cluster). To be able to move 
>> forward the state of theunreachable nodes must be changed. It must 
>> become reachable again or marked as down
>>
>> Is this totally true? If a node is unreachable and is the 
>> leaving/exiting/removed state will this stop the leader from adding a new 
>> node? I ask because I have an actor that subscribes to cluster events and I 
>> can see a node is being added whilst another node is considered unreachable 
>> and in the exiting status:
>>
>> 14:02:46.843 INFO  Exited member Member(address = akka.tcp://
>> geyser@172.16.120.160:7000, status = Exiting)
>> 14:02:51.842 INFO  Unreachable member Member(address = akka.tcp://
>> geyser@172.16.120.160:7000, status = Exiting)
>> 14:02:53.843 INFO  Removing member Member(address = akka.tcp://
>> geyser@172.16.120.160:7000, status = Removed)
>> 14:02:57.843 INFO  Exited member Member(address = akka.tcp://
>> geyser@172.16.119.46:7000, status = Exiting)
>> 14:03:02.760 INFO  Unreachable member Member(address = akka.tcp://
>> geyser@172.16.119.46:7000, status = Exiting)
>> 14:03:04.843 INFO  Adding member Member(address = akka.tcp://
>> geyser@172.16.120.160:7000, status = Up)
>>
>> Thanks,
>> Ben
>>
>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Clarification on unreachable nodes in cluster

[Konrad Malawski]

I precisely explained for what event you need to wait :-)

Proper graceful shutdown means you need to wait for _your own_ address to come 
back as MemberRemoved event,
and THEN you can shut down the actor system and JVM.

-- 
Cheers,
Konrad 'ktoso’ Malawski
Akka @ Lightbend

On 18 March 2016 at 18:24:58, Guido Medina (oxyg...@gmail.com) wrote:

How long is enough time? I tried sleep 1 sec between leave and terminate calls 
and it didn't work.

On Friday, March 18, 2016 at 5:13:50 PM UTC, Konrad Malawski wrote:
Also, please don't do that:
       cluster.leave(cluster.selfAddress());
        system.terminate();
It is wrong.

You need to give the cluster gossip enough time to complete the leaving 
"nicely".

Proper graceful shutdown means you need to wait for _your own_ address to come 
back as MemberRemoved event,
and THEN you can shut down the actor system and JVM.

Related: 
this mistake is exactly something I warn against in my recent talk "Zen of 
Akka", you may enjoy it :-)
https://youtu.be/tC-joPMPJLs



On Fri, Mar 18, 2016 at 5:58 PM, Patrik Nordwall  wrote:
Hi Ben

There are some subtle exceptions to the convergence rule. Unreachable members 
with status Down or Exiting are not counted.

/Patrik

On Thu, Mar 17, 2016 at 11:25 PM, Guido Medina  wrote:
In fact in the other nodes you get the following:

INFO  22:10:20,383 ster) - Cluster Node [akka.tcp://DevCluster@127.0.0.1:2552] 
- Leader is moving node [akka.tcp://DevCluster@127.0.0.1:38845] to [Exiting]
INFO  22:10:20,384 orRef - Message [akka.cluster.GossipEnvelope] from 
Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [1] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:20,403 orRef - Message 
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [2] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:21,383 orRef - Message [akka.cluster.GossipEnvelope] from 
Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [3] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:21,403 orRef - Message 
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [4] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:22,383 orRef - Message [akka.cluster.GossipEnvelope] from 
Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [5] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:22,403 orRef - Message 
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [6] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:23,383 orRef - Message [akka.cluster.GossipEnvelope] from 
Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [7] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:23,403 orRef - Message 
[akka.cluster.ClusterHeartbeatSender$Heartbeat] from 
Actor[akka://DevCluster/system/cluster/core/daemon/heartbeatSender#1423910769] 
to Actor[akka://DevCluster/deadLetters] was not delivered. [8] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:24,382 orRef - Message [akka.cluster.GossipEnvelope] from 
Actor[akka://DevCluster/system/cluster/core/daemon#1435915897] to 
Actor[akka://DevCluster/deadLetters] was not delivered. [9] dead letters 
encountered. This logging can be turned off or adjusted with configuration 
settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
INFO  22:10:24,384 ster) - Cluster