Re: [akka-user] Akka configured seed nodes are leaving cluster and unreachable/reachable warning message is getting logged.

[Patrik Nordwall]

I would suggest that you turn on remote debug logging and hopefully find
the root cause of the "Failed to write message to the transport":
http://doc.akka.io/docs/akka/2.4/java/logging.html#Auxiliary_remote_logging_options

/Patrik

On Tue, Aug 23, 2016 at 5:27 PM, Rob Crawford  wrote:

> Trying to pass around an entire Spring context is a bad idea. It contains
> MANY objects -- you're serializing your entire application.
>
> Focus on serializing the state -- changeable data -- of your actor, and
> let Spring wire up the dependencies on the other node.
>
>
> On Tuesday, August 23, 2016 at 10:19:35 AM UTC-4, vipin mandloi wrote:
>>
>> Hello,
>>
>> Thanks for your quick response.
>>
>>
>> As soon i add 3rd node, i am getting this* unreachable/reachable message
>> . and cluster is getting deformed*. till 2 nodes there is no issue.
>>
>> One more things added *serialize-creators = on and serialize-messages = *
>> *on* to verify serialization for props and message. although i have
>> added serialize for *ClassPathXmlApplicationContext (refer 1st email
>> application.conf) getting serialization error*
>>
>>
>> *Error:-*
>>
>> ERROR] [08/23/2016 07:02:10.705] 
>> [ClusterSystem-akka.actor.default-dispatcher-5]
>> [akka://ClusterSystem/user/singleton] pre-creation serialization check
>> failed at [akka://ClusterSystem/user/singleton/ConsumerActor]
>>
>> java.lang.IllegalArgumentException: pre-creation serialization check
>> failed at [akka://ClusterSystem/user/singleton/ConsumerActor]
>>
>>at akka.actor.dungeon.Children$cl
>> ass.makeChild(Children.scala:193)
>>
>>at akka.actor.dungeon.Children$cl
>> ass.actorOf(Children.scala:37)
>>
>>at akka.actor.ActorCell.actorOf(ActorCell.scala:369)
>>
>>at akka.contrib.pattern.ClusterSi
>> ngletonManager.gotoOldest(ClusterSingletonManager.scala:516)
>>
>>at akka.contrib.pattern.ClusterSi
>> ngletonManager$$anonfun$6.applyOrElse(ClusterSingletonManager.scala:428)
>>
>>at akka.contrib.pattern.ClusterSi
>> ngletonManager$$anonfun$6.applyOrElse(ClusterSingletonManager.scala:417)
>>
>>at scala.runtime.AbstractPartialF
>> unction.apply(AbstractPartialFunction.scala:33)
>>
>>at akka.actor.FSM$class.processEvent(FSM.scala:604)
>>
>>at akka.contrib.pattern.ClusterSi
>> ngletonManager.processEvent(ClusterSingletonManager.scala:336)
>>
>>at akka.actor.FSM$class.akka$acto
>> r$FSM$$processMsg(FSM.scala:598)
>>
>>at akka.actor.FSM$$anonfun$receiv
>> e$1.applyOrElse(FSM.scala:592)
>>
>>at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>
>>at akka.contrib.pattern.ClusterSi
>> ngletonManager.aroundReceive(ClusterSingletonManager.scala:336)
>>
>>at akka.actor.ActorCell.receiveMe
>> ssage(ActorCell.scala:516)
>>
>>at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>>
>>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>
>>at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>
>>at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>
>>at scala.concurrent.forkjoin.Fork
>> JoinTask.doExec(ForkJoinTask.java:260)
>>
>>at scala.concurrent.forkjoin.Fork
>> JoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>>
>>at scala.concurrent.forkjoin.Fork
>> JoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>>
>>at scala.concurrent.forkjoin.Fork
>> JoinPool.runWorker(ForkJoinPool.java:1979)
>>
>>at scala.concurrent.forkjoin.Fork
>> JoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>
>> *Caused by: java.lang.IllegalArgumentException: Can't serialize object of
>> type class org.springframework.context.su
>> pport.ClassPathXmlApplicationContext*
>>
>>at akka.cluster.protobuf.ClusterM
>> essageSerializer.toBinary(ClusterMessageSerializer.scala:74)
>>
>>at akka.serialization.Serializati
>> on$$anonfun$serialize$1.apply(Serialization.scala:90)
>>
>>
>>
>>
>>
>>
>>
>> On Monday, August 22, 2016 at 10:22:49 PM UTC+5:30, Justin du coeur wrote:
>>>
>>> On Mon, Aug 22, 2016 at 11:09 AM, vipin mandloi 
>>> wrote:
>>>
 Please let me know if i am missing any configuration.

>>>
>>> It's not a matter of *missing* -- the problem is that this:
>>>
>>>
 *My application.conf is configured as below-*


   cluster {
 auto-down-unreachable-after = 10s
   }

>>>
>>> Should never, ever, *ever* be used except for initial testing.
>>> Auto-down is intended just as a way to get you started, but it's very
>>> primitive and dumb, and must *never* be used in production.  Auto-downing
>>> is always likely to cause cluster disasters, and having such a short
>>> auto-down timeout essentially guarantees it.
>>>
>>> Basically, you're getting split-brai

Re: [akka-user] Akka configured seed nodes are leaving cluster and unreachable/reachable warning message is getting logged.

[Rob Crawford]

Trying to pass around an entire Spring context is a bad idea. It contains 
MANY objects -- you're serializing your entire application.

Focus on serializing the state -- changeable data -- of your actor, and let 
Spring wire up the dependencies on the other node.

On Tuesday, August 23, 2016 at 10:19:35 AM UTC-4, vipin mandloi wrote:
>
> Hello,
>
> Thanks for your quick response.
>
>
> As soon i add 3rd node, i am getting this* unreachable/reachable message 
> . and cluster is getting deformed*. till 2 nodes there is no issue.
>
> One more things added *serialize-creators = on and serialize-messages = *
> *on* to verify serialization for props and message. although i have added 
> serialize for *ClassPathXmlApplicationContext (refer 1st email 
> application.conf) getting serialization error*
>
>
> *Error:-*
>
> ERROR] [08/23/2016 07:02:10.705] 
> [ClusterSystem-akka.actor.default-dispatcher-5] 
> [akka://ClusterSystem/user/singleton] pre-creation serialization check 
> failed at [akka://ClusterSystem/user/singleton/ConsumerActor]
>
> java.lang.IllegalArgumentException: pre-creation serialization check 
> failed at [akka://ClusterSystem/user/singleton/ConsumerActor]
>
>at 
> akka.actor.dungeon.Children$class.makeChild(Children.scala:193)
>
>at 
> akka.actor.dungeon.Children$class.actorOf(Children.scala:37)
>
>at akka.actor.ActorCell.actorOf(ActorCell.scala:369)
>
>at 
> akka.contrib.pattern.ClusterSingletonManager.gotoOldest(ClusterSingletonManager.scala:516)
>
>at 
> akka.contrib.pattern.ClusterSingletonManager$$anonfun$6.applyOrElse(ClusterSingletonManager.scala:428)
>
>at 
> akka.contrib.pattern.ClusterSingletonManager$$anonfun$6.applyOrElse(ClusterSingletonManager.scala:417)
>
>at 
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
>
>at akka.actor.FSM$class.processEvent(FSM.scala:604)
>
>at 
> akka.contrib.pattern.ClusterSingletonManager.processEvent(ClusterSingletonManager.scala:336)
>
>at 
> akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598)
>
>at 
> akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592)
>
>at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>
>at 
> akka.contrib.pattern.ClusterSingletonManager.aroundReceive(ClusterSingletonManager.scala:336)
>
>at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>
>at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>
>at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>
>at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>
>at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>
>at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>
>at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
>
>at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
>
>at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>
>at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> *Caused by: java.lang.IllegalArgumentException: Can't serialize object of 
> type class 
> org.springframework.context.support.ClassPathXmlApplicationContext*
>
>at 
> akka.cluster.protobuf.ClusterMessageSerializer.toBinary(ClusterMessageSerializer.scala:74)
>
>at 
> akka.serialization.Serialization$$anonfun$serialize$1.apply(Serialization.scala:90)
>
>
>
>
>
>
>
> On Monday, August 22, 2016 at 10:22:49 PM UTC+5:30, Justin du coeur wrote:
>>
>> On Mon, Aug 22, 2016 at 11:09 AM, vipin mandloi  
>> wrote:
>>
>>> Please let me know if i am missing any configuration.
>>>
>>
>> It's not a matter of *missing* -- the problem is that this:
>>  
>>
>>> *My application.conf is configured as below-*
>>>
>>>  
>>>   cluster { 
>>> auto-down-unreachable-after = 10s
>>>   }  
>>>
>>
>> Should never, ever, *ever* be used except for initial testing.  Auto-down 
>> is intended just as a way to get you started, but it's very primitive and 
>> dumb, and must *never* be used in production.  Auto-downing is always 
>> likely to cause cluster disasters, and having such a short auto-down 
>> timeout essentially guarantees it.
>>
>> Basically, you're getting split-brain -- you need to replace auto-down 
>> with a more-sophisticated downing strategy.  See this section of the 
>> documentation 
>> 
>>  
>> for more information.
>>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the ar

Re: [akka-user] Akka configured seed nodes are leaving cluster and unreachable/reachable warning message is getting logged.

[vipin mandloi]

Hello,

Thanks for your quick response.


As soon i add 3rd node, i am getting this* unreachable/reachable message . 
and cluster is getting deformed*. till 2 nodes there is no issue.

One more things added *serialize-creators = on and serialize-messages = *
*on* to verify serialization for props and message. although i have added 
serialize for *ClassPathXmlApplicationContext (refer 1st email 
application.conf) getting serialization error*


*Error:-*

ERROR] [08/23/2016 07:02:10.705] 
[ClusterSystem-akka.actor.default-dispatcher-5] 
[akka://ClusterSystem/user/singleton] pre-creation serialization check 
failed at [akka://ClusterSystem/user/singleton/ConsumerActor]

java.lang.IllegalArgumentException: pre-creation serialization check failed 
at [akka://ClusterSystem/user/singleton/ConsumerActor]

   at 
akka.actor.dungeon.Children$class.makeChild(Children.scala:193)

   at 
akka.actor.dungeon.Children$class.actorOf(Children.scala:37)

   at akka.actor.ActorCell.actorOf(ActorCell.scala:369)

   at 
akka.contrib.pattern.ClusterSingletonManager.gotoOldest(ClusterSingletonManager.scala:516)

   at 
akka.contrib.pattern.ClusterSingletonManager$$anonfun$6.applyOrElse(ClusterSingletonManager.scala:428)

   at 
akka.contrib.pattern.ClusterSingletonManager$$anonfun$6.applyOrElse(ClusterSingletonManager.scala:417)

   at 
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)

   at akka.actor.FSM$class.processEvent(FSM.scala:604)

   at 
akka.contrib.pattern.ClusterSingletonManager.processEvent(ClusterSingletonManager.scala:336)

   at 
akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598)

   at 
akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592)

   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)

   at 
akka.contrib.pattern.ClusterSingletonManager.aroundReceive(ClusterSingletonManager.scala:336)

   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

   at akka.actor.ActorCell.invoke(ActorCell.scala:487)

   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)

   at akka.dispatch.Mailbox.run(Mailbox.scala:221)

   at akka.dispatch.Mailbox.exec(Mailbox.scala:231)

   at 
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

   at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)

   at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)

   at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

   at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

*Caused by: java.lang.IllegalArgumentException: Can't serialize object of 
type class 
org.springframework.context.support.ClassPathXmlApplicationContext*

   at 
akka.cluster.protobuf.ClusterMessageSerializer.toBinary(ClusterMessageSerializer.scala:74)

   at 
akka.serialization.Serialization$$anonfun$serialize$1.apply(Serialization.scala:90)







On Monday, August 22, 2016 at 10:22:49 PM UTC+5:30, Justin du coeur wrote:
>
> On Mon, Aug 22, 2016 at 11:09 AM, vipin mandloi  > wrote:
>
>> Please let me know if i am missing any configuration.
>>
>
> It's not a matter of *missing* -- the problem is that this:
>  
>
>> *My application.conf is configured as below-*
>>
>>  
>>   cluster { 
>> auto-down-unreachable-after = 10s
>>   }  
>>
>
> Should never, ever, *ever* be used except for initial testing.  Auto-down 
> is intended just as a way to get you started, but it's very primitive and 
> dumb, and must *never* be used in production.  Auto-downing is always 
> likely to cause cluster disasters, and having such a short auto-down 
> timeout essentially guarantees it.
>
> Basically, you're getting split-brain -- you need to replace auto-down 
> with a more-sophisticated downing strategy.  See this section of the 
> documentation 
> 
>  
> for more information.
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Akka configured seed nodes are leaving cluster and unreachable/reachable warning message is getting logged.

[Justin du coeur]

On Mon, Aug 22, 2016 at 11:09 AM, vipin mandloi 
wrote:

> Please let me know if i am missing any configuration.
>

It's not a matter of *missing* -- the problem is that this:


> *My application.conf is configured as below-*
>
>
>   cluster {
> auto-down-unreachable-after = 10s
>   }
>

Should never, ever, *ever* be used except for initial testing.  Auto-down
is intended just as a way to get you started, but it's very primitive and
dumb, and must *never* be used in production.  Auto-downing is always
likely to cause cluster disasters, and having such a short auto-down
timeout essentially guarantees it.

Basically, you're getting split-brain -- you need to replace auto-down with
a more-sophisticated downing strategy.  See this section of the
documentation

for more information.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Akka configured seed nodes are leaving cluster and unreachable/reachable warning message is getting logged.

[vipin mandloi]

Hello All,

I am setting akka cluster *(using akka library version 2.3.9* ) with one 
master actor and multiple worker actor. Master Actor is configured with 
pool cluster aware router.

I am deploying application with forming 4 cluster nodes. *Initially all the 
nodes are joining to cluster properly but after few minutes nodes are 
leaving to cluster and forming their own cluster. *

Getting below *warn *message when i add another node in the cluster.


*[WARN*] [08/22/2016 04:52:42.145] 
[ClusterSystem-akka.remote.default-remote-dispatcher-8] 
[akka.tcp://ClusterSystem@host:port/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%host%port-4]
 
Association with remote system [akka.tcp://ClusterSystem@host:port] has 
failed, address is now gated for [5000] ms. Reason is: [Failed to write 
message to the transport].


And *after few minutes of joining, * frequent unreachable reachable 
*warning *message is getting logged.


*[WARN] *[08/22/2016 05:42:13.586] 
[ClusterSystem-akka.actor.default-dispatcher-8] 
[akka.tcp://ClusterSystem@host:port/system/cluster/core/daemon] Cluster 
Node [akka.tcp://ClusterSystem@host:port] - Marking node(s) as UNREACHABLE 
[Member(address = akka.tcp://ClusterSystem@xldcarcsi01:port, status = Up)]
[INFO] [08/22/2016 05:42:14.586] 
[ClusterSystem-akka.actor.default-dispatcher-6] 
[Cluster(akka://ClusterSystem)] Cluster Node 
[akka.tcp://ClusterSystem@host:port] - Marking node(s) as REACHABLE 
[Member(address = akka.tcp://ClusterSystem@host:port, status = Up), 
Member(address = akka.tcp://ClusterSystem@xldcarcsi01:port, status = Up)]
*[WARN]* [08/22/2016 05:42:14.592] 
[ClusterSystem-akka.remote.default-remote-dispatcher-10] 
[akka.tcp://ClusterSystem@host:port/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40host%3Aport-0]
 
Association with remote system [akka.tcp://ClusterSystem@host:port] has 
failed, address is now gated for [5000] ms. Reason is: [Failed to write 
message to the transport].
*[WARN]* [08/22/2016 05:42:14.593] 
[ClusterSystem-akka.remote.default-remote-dispatcher-10] 
[akka.tcp://ClusterSystem@host:port/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40xldcarcsi01%3Aport-1]
 
Association with remote system [akka.tcp://ClusterSystem@xldcarcsi01:port] 
has failed, address is now gated for [5000] ms. Reason is: [Failed to write 
message to the transport].
[WARN] [08/22/2016 05:42:18.584] 
[ClusterSystem-akka.actor.default-dispatcher-2] 
[akka.tcp://ClusterSystem@host:port/system/cluster/core/daemon] Cluster 
Node [akka.tcp://ClusterSystem@host:port] - Marking node(s) as UNREACHABLE 
[Member(address = akka.tcp://ClusterSystem@host:port, status = Up)]




Please let me know if i am missing any configuration.


*My application.conf is configured as below-*

my-dispatcher {
type = PinnedDispatcher 
executor = "thread-pool-executor"
thread-pool-executor {
core-pool-size-min = 1
core-pool-size-factor = 2.0
core-pool-size-max = 2
}
throughput = 3
mailbox-capacity = 3
}

akka {
 
 bounded-mailbox {
 mailbox-type = "akka.dispatch.BoundedMailbox"
 mailbox-capacity = 2
 mailbox-push-timeout-time = 100ms
  }

  actor {
provider = "akka.cluster.ClusterActorRefProvider"
serializers {
akka-cluster = "akka.cluster.protobuf.ClusterMessageSerializer"
}
serialization-bindings {
"akka.cluster.ClusterMessage" = akka-cluster
"org.springframework.context.support.ClassPathXmlApplicationContext" = 
akka-cluster
} 
  }
  remote {
log-remote-lifecycle-events = off   
netty.tcp {
  hostname = ${akka.node.hostname}
  port = ${akka.node.port}
}
  }
 
  cluster {
seed-nodes = [${akka.seed.nodes}]
 
  roles = ["master"]
  
auto-down-unreachable-after = 10s
  }  
}
akka.actor.deployment {
  /singleton/MyConsumerActor/workerRouter {
router = consistent-hashing-pool
nr-of-instances = 100
cluster {
  enabled = on
  max-nr-of-instances-per-node = 3
  allow-local-routees = on
  use-role = master
}
  }
}
*//tried with customizing transport-failure-detector*
akka.remote.transport-failure-detector {
heartbeat-interval = 1000s 
acceptable-heartbeat-pause = 6000s 
}



Thanks,
Vipin Mandloi

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.