Hi,
INFO [2015-10-01 09:01:31,823] [] [] [dispatcher-4] a.a.EmptyLocalActorRef > akka://ClusterSystem/user/worker - Message [mycompany.akka.cluster.Work] > from > Actor[akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton#-83415853] > to Actor[akka://ClusterSystem/user/worker#-2093331180] was not delivered. > [1] dead letters encountered. This logging can be turned off or adjusted > with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > This can result from two reasons as far as I see: A) there is no actor on that path. B) the ActorRef you use to send is referring to an older actor and the UID does not match with the new instance. Anyway, you can enable some of the debug flags to see what is going on: akka.loglevel = DEBUG akka.remote.log-received-messages=on akka.remote.log-sent-messages=on That should give a bit more idea of what is going on. What can also cause problems if you run all systems on a single JVM and mix ActorRefs aquired through remoting and via shared variables. For example: val ref1 = system1.actorOf(...) system2.actorOf( .. passing ref1 directly to an actor ...) will circumvent the remoting subsystem, which properly serializes ActorRefs so they include all information (host, port, etc.) so the remote system can properly reply for example. In detail, there are local and remote actorRefs, where the local ones bypass remoting and are the fastest, and remote refs that are acquired via remoting and do serialzation and sending via the wire. When you share two systems on the JVM you can accidentally share the local ref of one with the other and cause confusion. I am not sure, these are speculations. If you can create a simple standalone reproducer, we can look at it. -Endre > INFO [2015-10-01 09:01:31,831] [] [] [dispatcher-3] > a.a.EmptyLocalActorRef akka://ClusterSystem/user/worker - Message > [mycompany.akka.cluster.Work] from > Actor[akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton#-83415853] > to Actor[akka://ClusterSystem/user/worker] was not delivered. [2] dead > letters encountered. This logging can be turned off or adjusted with > configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > DEBUG [2015-10-01 09:01:35,261] [] [] [dispatcher-4] c.t.a.cluster.Worker > akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker with path > akka://ClusterSystem/user/worker received ping. > > > > Box 2 - Leader/Master > > INFO [2015-10-01 09:01:09,667] [] [] [dispatcher-17] akka.remote.Remoting > akka.remote.Remoting - Starting remoting > INFO [2015-10-01 09:01:10,224] [] [] [dispatcher-17] akka.remote.Remoting > akka.remote.Remoting - Remoting started; listening on addresses > :[akka.tcp://ClusterSystem@MASTERIP:12489] > INFO [2015-10-01 09:01:10,225] [] [] [dispatcher-17] akka.remote.Remoting > akka.remote.Remoting - Remoting now listens on addresses: > [akka.tcp://ClusterSystem@MASTERIP:12489] > INFO [2015-10-01 09:01:10,226] [] [] [dispatcher-17] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12489] - Starting up... > INFO [2015-10-01 09:01:10,235] [] [] [dispatcher-17] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12489] - Started up successfully > INFO [2015-10-01 09:01:10,275] [] [] [dispatcher-4] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12489] - No seed-nodes configured, > manual cluster join required > DEBUG [2015-10-01 09:01:10,508] [] [] [dispatcher-18] c.t.a.cluster.Worker > akka.tcp://ClusterSystem@MASTERIP:12489/user/worker - Worker > d5f55d4b-1198-4f25-b72c-814325b0d198 with path > akka://ClusterSystem/user/worker sending > RegisterWorker(d5f55d4b-1198-4f25-b72c-814325b0d198) to master at > Actor[akka://ClusterSystem/user/$b#643942838] > INFO [2015-10-01 09:01:10,628] [] [] [dispatcher-16] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12489] - Node > [akka.tcp://ClusterSystem@MASTERIP:12489] is JOINING, roles [] > INFO [2015-10-01 09:01:11,344] [] [] [dispatcher-3] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12487] - Node > [akka.tcp://ClusterSystem@MASTERIP:12487] is JOINING, roles [master] > INFO [2015-10-01 09:01:11,351] [] [] [dispatcher-3] a.actor.LocalActorRef > akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1 - > Message [akka.cluster.InternalClusterAction$InitJoinNack] from > Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/system/cluster/core/daemon#-867905712] > to > Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#2098879220] > was not delivered. [1] dead letters encountered. This logging can be turned > off or adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO [2015-10-01 09:01:11,394] [] [] [dispatcher-17] > a.actor.LocalActorRef > akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1 - > Message [akka.cluster.InternalClusterAction$InitJoinNack] from > Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/system/cluster/core/daemon#-867905712] > to > Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#2098879220] > was not delivered. [2] dead letters encountered. This logging can be turned > off or adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > INFO [2015-10-01 09:01:11,418] [] [] [dispatcher-16] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12489] - Leader is moving node > [akka.tcp://ClusterSystem@MASTERIP:12489] to [Up] > INFO [2015-10-01 09:01:12,019] [] [] [dispatcher-3] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12487] - Leader is moving node > [akka.tcp://ClusterSystem@MASTERIP:12487] to [Up] > INFO [2015-10-01 09:01:12,097] [] [] [dispatcher-5] > a.c.s.ClusterSingletonManager > akka.tcp://ClusterSystem@MASTERIP:12487/user/master > - Singleton manager starting singleton actor > [akka://ClusterSystem/user/master/singleton] > INFO [2015-10-01 09:01:12,100] [] [] [dispatcher-5] > a.c.s.ClusterSingletonManager > akka.tcp://ClusterSystem@MASTERIP:12487/user/master > - ClusterSingletonManager state change [Start -> Oldest] > INFO [2015-10-01 09:01:16,434] [] [] [dispatcher-18] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12487] - Node > [akka.tcp://ClusterSystem@SLAVEIP:12487] is JOINING, roles [master] > INFO [2015-10-01 09:01:17,018] [] [] [dispatcher-17] > a.c.Cluster(akka://ClusterSystem) > akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node > [akka.tcp://ClusterSystem@MASTERIP:12487] - Leader is moving node > [akka.tcp://ClusterSystem@SLAVEIP:12487] to [Up] > INFO [2015-10-01 09:01:18,544] [] [] [dispatcher-17] > c.t.m.l.a.LoadTestMaster > akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton > - WorkerId 3d5e4a3f-2a87-4256-9067-ec60cb377f53 registered for > Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker#-2093331180] > INFO [2015-10-01 09:01:31,772] [] [] [dispatcher-3] > c.t.m.l.a.LoadTestMaster > akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton > - Accepted work: dac76d63-ddd3-4603-8d80-c1fb017e9246. Sending ack to > Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/temp/$a] > DEBUG [2015-10-01 09:01:31,804] [] [] [dispatcher-3] > c.t.m.l.a.LoadTestMaster > akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton > - Master sending > Work(dac76d63-ddd3-4603-8d80-c1fb017e9246,RunLocust(DryRun(),Some(MASTERIP))) > to Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker#-2093331180] > with path akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker > > Thanks! > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - Reactive apps on the JVM Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
