Hi again, I have found the problem and have a solution. Ticket here<https://www.assembla.com/spaces/akka/tickets/3880>. Pull request tomorrow.
/Patrik On Tue, Feb 18, 2014 at 2:57 PM, Patrik Nordwall <patrik.nordw...@gmail.com>wrote: > Hi James, > > Thanks for reporting. I'm still investigating your scenario. On the way I > found another (maybe related) > issue<https://www.assembla.com/spaces/akka/tickets/3879> > . > Stay tuned. > > Regards > Patrik > > > On Tue, Feb 18, 2014 at 12:58 AM, James Bellenger <ja...@kixeye.com>wrote: > >> Hi gang, a question regarding shard failover >> >> Here's my 3-node setup (using akka-2.3.0 rc3): >> >> - backend 1 >> - backend 2 >> - frontend >> >> All nodes initialize ClusterSharding at startup -- frontend supplies an >> empty entryProps so that it does not host any regions. >> >> Frontend starts pinging a range of sharded actors host on the backends -- >> I can see that the entries are evenly distributed across the backends. >> >> When I invoke a clean shutdown of one of the backends, the frontend node >> is left in a bad state. It continually tries to connect to the dead backend >> node. It does this forever. Restarting the frontend is required to get it >> to find the failed over shards on the remaining backend. >> >> Am I missing something with shutting down cluster sharding? >> Some logs from frontend. They're a bit noisy -- I've tried to call out >> the interesting bits. >> >> DEBUG 15:40:42,389 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [0] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >> DEBUG 15:40:43,405 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [1] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >> DEBUG 15:40:44,425 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [2] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >> INFO 15:40:44,825 akka.actor.LocalActorRef - Message >> [akka.remote.transport.AssociationHandle$Disassociated] from >> Actor[akka://ghost/deadLetters] to >> Actor[akka://ghost/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1#-206867083] >> was not delivered. [2] dead letters encountered. This logging can be turned >> off or adjusted with configuration settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> INFO 15:40:44,881 akka.actor.LocalActorRef - Message >> [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from >> Actor[akka://ghost/deadLetters] to >> Actor[akka://ghost/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1#-206867083] >> was not delivered. [3] dead letters encountered. This logging can be turned >> off or adjusted with configuration settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> *DEBUG 15:40:44,884 akka.remote.EndpointWriter - Disassociated >> [akka.tcp://ghost@127.0.0.1:50373 <http://ghost@127.0.0.1:50373>] -> >> [akka.tcp://ghost@127.0.0.1:50351 <http://ghost@127.0.0.1:50351>]* >> INFO 15:40:44,884 akka.actor.LocalActorRef - Message >> [akka.actor.FSM$Timer] from Actor[akka://ghost/deadLetters] to >> Actor[akka://ghost/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1/endpointWriter#1517188237] >> was not delivered. [4] dead letters encountered. This logging can be turned >> off or adjusted with configuration settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> INFO 15:40:45,091 >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >> [akka.cluster.GossipEnvelope] from >> Actor[akka://ghost/system/cluster/core/daemon#-1507678631] to >> Actor[akka://ghost/deadLetters] was not delivered. [5] dead letters >> encountered. This logging can be turned off or adjusted with configuration >> settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> INFO 15:40:45,097 com.kixeye.common.log.AkkaLogger - Cluster Node >> [akka.tcp://ghost@127.0.0.1:50373] - Marking exiting node(s) as >> UNREACHABLE [Member(address = akka.tcp://ghost@127.0.0.1:50351, status = >> Exiting)]. This is expected and they will be removed. >> *INFO 15:40:45,106 com.kixeye.common.cluster.ClusterModule - member is >> unreachable: Member(address = akka.tcp://ghost@127.0.0.1:50351 >> <http://ghost@127.0.0.1:50351>, status = Exiting)* >> ^-- node becomes unreachable >> >> DEBUG 15:40:45,445 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [3] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >> INFO 15:40:45,535 >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >> [akka.cluster.ClusterHeartbeatSender$Heartbeat] from >> Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#214131267] to >> Actor[akka://ghost/deadLetters] was not delivered. [6] dead letters >> encountered. This logging can be turned off or adjusted with configuration >> settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> *DEBUG 15:40:46,465 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [4] to >> [Actor[akka.tcp://ghost@127.0.0.1:50351/user/sharding/user#-576014717 >> <http://ghost@127.0.0.1:50351/user/sharding/user#-576014717>]]* >> *^-- *forwarding msg to a known-unreachable node >> >> INFO 15:40:46,465 >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >> [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from >> Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] >> to Actor[akka://ghost/deadLetters] was not delivered. [7] dead letters >> encountered. This logging can be turned off or adjusted with configuration >> settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> INFO 15:40:46,535 >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >> [akka.cluster.ClusterHeartbeatSender$Heartbeat] from >> Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#214131267] to >> Actor[akka://ghost/deadLetters] was not delivered. [8] dead letters >> encountered. This logging can be turned off or adjusted with configuration >> settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> INFO 15:40:47,103 com.kixeye.common.cluster.ClusterModule - event: >> MemberRemoved(Member(address = akka.tcp://ghost@127.0.0.1:50351, status >> = Removed),Exiting) >> INFO 15:40:47,104 akka.contrib.pattern.ClusterSingletonManager - Member >> removed [akka.tcp://ghost@127.0.0.1:50351] >> DEBUG 15:40:47,486 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [5] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >> DEBUG 15:40:48,506 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [6] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >> INFO 15:40:48,506 >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >> [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from >> Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] >> to Actor[akka://ghost/deadLetters] was not delivered. [9] dead letters >> encountered. This logging can be turned off or adjusted with configuration >> settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> DEBUG 15:40:49,526 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [7] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >> DEBUG 15:40:50,546 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [8] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >> DEBUG 15:40:51,552 akka.remote.EndpointWriter - AssociationError >> [akka.tcp://ghost@127.0.0.1:50373] -> [akka.tcp://ghost@127.0.0.1:50351]: >> Error [Association failed with [akka.tcp://ghost@127.0.0.1:50351]] [ >> akka.remote.EndpointAssociationException: Association failed with >> [akka.tcp://ghost@127.0.0.1:50351] >> Caused by: >> akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: >> Connection refused: no further information: /127.0.0.1:50351 >> ] >> *WARN 15:40:51,554 akka.remote.ReliableDeliverySupervisor - Association >> with remote system [akka.tcp://ghost@127.0.0.1:50351 >> <http://ghost@127.0.0.1:50351>] has failed, address is now gated for [5000] >> ms. Reason is: [Association failed with >> [akka.tcp://ghost@127.0.0.1:50351]].* >> ^-- this repeats every 5s, forever >> >> DEBUG 15:40:51,566 akka.contrib.pattern.ShardRegion - Forwarding request >> for shard [9] to [Actor[akka.tcp:// >> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >> DEBUG 15:40:51,579 akka.remote.EndpointWriter - Disassociated [akka.tcp:// >> ghost@127.0.0.1:50373] -> [akka.tcp://ghost@127.0.0.1:50351] >> INFO 15:40:51,579 >> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >> [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from >> Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] >> to Actor[akka://ghost/deadLetters] was not delivered. [10] dead letters >> encountered. This logging can be turned off or adjusted with configuration >> settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
