On Thu, Feb 20, 2014 at 5:33 PM, James Bellenger <ja...@kixeye.com> wrote:
> Everything working as advertised :) > It's really cool to see it in full loadbalancing+failover action. > Thanks, Patrik > Glad to hear that. Thanks for verifying. /Patrik > > > On Thu, Feb 20, 2014 at 2:20 AM, Patrik Nordwall < > patrik.nordw...@gmail.com> wrote: > >> Hi James, >> We released 2.3.0-RC4 yesterday, including fixes for these issues. Please >> try it and report back. >> Thanks, >> Patrik >> >> >> On Tue, Feb 18, 2014 at 6:05 PM, Patrik Nordwall < >> patrik.nordw...@gmail.com> wrote: >> >>> Hi again, >>> >>> I have found the problem and have a solution. Ticket >>> here<https://www.assembla.com/spaces/akka/tickets/3880>. >>> Pull request tomorrow. >>> >>> /Patrik >>> >>> >>> On Tue, Feb 18, 2014 at 2:57 PM, Patrik Nordwall < >>> patrik.nordw...@gmail.com> wrote: >>> >>>> Hi James, >>>> >>>> Thanks for reporting. I'm still investigating your scenario. On the way >>>> I found another (maybe related) >>>> issue<https://www.assembla.com/spaces/akka/tickets/3879> >>>> . >>>> Stay tuned. >>>> >>>> Regards >>>> Patrik >>>> >>>> >>>> On Tue, Feb 18, 2014 at 12:58 AM, James Bellenger <ja...@kixeye.com>wrote: >>>> >>>>> Hi gang, a question regarding shard failover >>>>> >>>>> Here's my 3-node setup (using akka-2.3.0 rc3): >>>>> >>>>> - backend 1 >>>>> - backend 2 >>>>> - frontend >>>>> >>>>> All nodes initialize ClusterSharding at startup -- frontend supplies >>>>> an empty entryProps so that it does not host any regions. >>>>> >>>>> Frontend starts pinging a range of sharded actors host on the backends >>>>> -- I can see that the entries are evenly distributed across the backends. >>>>> >>>>> When I invoke a clean shutdown of one of the backends, the frontend >>>>> node is left in a bad state. It continually tries to connect to the dead >>>>> backend node. It does this forever. Restarting the frontend is required to >>>>> get it to find the failed over shards on the remaining backend. >>>>> >>>>> Am I missing something with shutting down cluster sharding? >>>>> Some logs from frontend. They're a bit noisy -- I've tried to call out >>>>> the interesting bits. >>>>> >>>>> DEBUG 15:40:42,389 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [0] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >>>>> DEBUG 15:40:43,405 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [1] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >>>>> DEBUG 15:40:44,425 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [2] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >>>>> INFO 15:40:44,825 akka.actor.LocalActorRef - Message >>>>> [akka.remote.transport.AssociationHandle$Disassociated] from >>>>> Actor[akka://ghost/deadLetters] to >>>>> Actor[akka://ghost/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1#-206867083] >>>>> was not delivered. [2] dead letters encountered. This logging can be >>>>> turned >>>>> off or adjusted with configuration settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> INFO 15:40:44,881 akka.actor.LocalActorRef - Message >>>>> [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from >>>>> Actor[akka://ghost/deadLetters] to >>>>> Actor[akka://ghost/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1#-206867083] >>>>> was not delivered. [3] dead letters encountered. This logging can be >>>>> turned >>>>> off or adjusted with configuration settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> *DEBUG 15:40:44,884 akka.remote.EndpointWriter - Disassociated >>>>> [akka.tcp://ghost@127.0.0.1:50373 <http://ghost@127.0.0.1:50373>] -> >>>>> [akka.tcp://ghost@127.0.0.1:50351 <http://ghost@127.0.0.1:50351>]* >>>>> INFO 15:40:44,884 akka.actor.LocalActorRef - Message >>>>> [akka.actor.FSM$Timer] from Actor[akka://ghost/deadLetters] to >>>>> Actor[akka://ghost/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2Fghost%40127.0.0.1%3A50351-1/endpointWriter#1517188237] >>>>> was not delivered. [4] dead letters encountered. This logging can be >>>>> turned >>>>> off or adjusted with configuration settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> INFO 15:40:45,091 >>>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >>>>> [akka.cluster.GossipEnvelope] from >>>>> Actor[akka://ghost/system/cluster/core/daemon#-1507678631] to >>>>> Actor[akka://ghost/deadLetters] was not delivered. [5] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> INFO 15:40:45,097 com.kixeye.common.log.AkkaLogger - Cluster Node >>>>> [akka.tcp://ghost@127.0.0.1:50373] - Marking exiting node(s) as >>>>> UNREACHABLE [Member(address = akka.tcp://ghost@127.0.0.1:50351, >>>>> status = Exiting)]. This is expected and they will be removed. >>>>> *INFO 15:40:45,106 com.kixeye.common.cluster.ClusterModule - member is >>>>> unreachable: Member(address = akka.tcp://ghost@127.0.0.1:50351 >>>>> <http://ghost@127.0.0.1:50351>, status = Exiting)* >>>>> ^-- node becomes unreachable >>>>> >>>>> DEBUG 15:40:45,445 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [3] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >>>>> INFO 15:40:45,535 >>>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >>>>> [akka.cluster.ClusterHeartbeatSender$Heartbeat] from >>>>> Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#214131267] >>>>> to >>>>> Actor[akka://ghost/deadLetters] was not delivered. [6] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> *DEBUG 15:40:46,465 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [4] to >>>>> [Actor[akka.tcp://ghost@127.0.0.1:50351/user/sharding/user#-576014717 >>>>> <http://ghost@127.0.0.1:50351/user/sharding/user#-576014717>]]* >>>>> *^-- *forwarding msg to a known-unreachable node >>>>> >>>>> INFO 15:40:46,465 >>>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >>>>> [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from >>>>> Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] >>>>> to Actor[akka://ghost/deadLetters] was not delivered. [7] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> INFO 15:40:46,535 >>>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >>>>> [akka.cluster.ClusterHeartbeatSender$Heartbeat] from >>>>> Actor[akka://ghost/system/cluster/core/daemon/heartbeatSender#214131267] >>>>> to >>>>> Actor[akka://ghost/deadLetters] was not delivered. [8] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> INFO 15:40:47,103 com.kixeye.common.cluster.ClusterModule - event: >>>>> MemberRemoved(Member(address = akka.tcp://ghost@127.0.0.1:50351, >>>>> status = Removed),Exiting) >>>>> INFO 15:40:47,104 akka.contrib.pattern.ClusterSingletonManager - >>>>> Member removed [akka.tcp://ghost@127.0.0.1:50351] >>>>> DEBUG 15:40:47,486 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [5] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >>>>> DEBUG 15:40:48,506 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [6] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >>>>> INFO 15:40:48,506 >>>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >>>>> [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from >>>>> Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] >>>>> to Actor[akka://ghost/deadLetters] was not delivered. [9] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> DEBUG 15:40:49,526 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [7] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >>>>> DEBUG 15:40:50,546 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [8] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50351/user/sharding/user#-576014717]] >>>>> DEBUG 15:40:51,552 akka.remote.EndpointWriter - AssociationError >>>>> [akka.tcp://ghost@127.0.0.1:50373] -> [akka.tcp:// >>>>> ghost@127.0.0.1:50351]: Error [Association failed with [akka.tcp:// >>>>> ghost@127.0.0.1:50351]] [ >>>>> akka.remote.EndpointAssociationException: Association failed with >>>>> [akka.tcp://ghost@127.0.0.1:50351] >>>>> Caused by: >>>>> akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: >>>>> Connection refused: no further information: /127.0.0.1:50351 >>>>> ] >>>>> *WARN 15:40:51,554 akka.remote.ReliableDeliverySupervisor - >>>>> Association with remote system [akka.tcp://ghost@127.0.0.1:50351 >>>>> <http://ghost@127.0.0.1:50351>] has failed, address is now gated for >>>>> [5000] >>>>> ms. Reason is: [Association failed with >>>>> [akka.tcp://ghost@127.0.0.1:50351]].* >>>>> ^-- this repeats every 5s, forever >>>>> >>>>> DEBUG 15:40:51,566 akka.contrib.pattern.ShardRegion - Forwarding >>>>> request for shard [9] to [Actor[akka.tcp:// >>>>> ghost@127.0.0.1:50324/user/sharding/user#-498786510]] >>>>> DEBUG 15:40:51,579 akka.remote.EndpointWriter - Disassociated >>>>> [akka.tcp://ghost@127.0.0.1:50373] -> [akka.tcp:// >>>>> ghost@127.0.0.1:50351] >>>>> INFO 15:40:51,579 >>>>> akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message >>>>> [com.kixeye.common.cluster.UserBackend$UserMessageEnvelope] from >>>>> Actor[akka://ghost/user/ghost-sheppard/module-com.kixeye.common.cluster.UserClientModule-8c0d9b97-8b4e-4988-a40b-91d054080afa#1779934259] >>>>> to Actor[akka://ghost/deadLetters] was not delivered. [10] dead letters >>>>> encountered. This logging can be turned off or adjusted with configuration >>>>> settings 'akka.log-dead-letters' and >>>>> 'akka.log-dead-letters-during-shutdown'. >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to akka-user+unsubscr...@googlegroups.com. >>>>> To post to this group, send email to akka-user@googlegroups.com. >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Patrik Nordwall >>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>>> Twitter: @patriknw >>>> >>>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> >>> >> >> >> -- >> >> Patrik Nordwall >> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >> Twitter: @patriknw >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://akka.io/faq/ > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/groups/opt_out. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://akka.io/faq/ >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/groups/opt_out.
