TL;DR: Following TypeSafe activator Distributed worker code, messages from Master/Leader based on actor ref or actor path go to dead letters on target system
1. Worker sends RegisterWorker(id) to Master via Master Proxy WORKER SYSTEM: DEBUG [2015-10-01 09:01:05,285] [] [] [dispatcher-17] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker 3d5e4a3f-2a87-4256-9067-ec60cb377f53 with path akka://ClusterSystem/user/worker sending RegisterWorker(3d5e4a3f-2a87-4256-9067-ec60cb377f53) to master at Actor[akka://ClusterSystem/user/$b#-1210252092] MASTER SYSTEM: INFO [2015-10-01 09:01:18,544] [] [] [dispatcher-17] c.t.m.l.a.LoadTestMaster akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton - WorkerId 3d5e4a3f-2a87-4256-9067-ec60cb377f53 registered for Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker#-2093331180] 2. Master sends work to Worker (once using ActorRef, once using Actor Path, just for fun) MASTER SYSTEM: DEBUG [2015-10-01 09:01:31,804] [] [] [dispatcher-3] c.t.m.l.a.LoadTestMaster akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton - Master sending Work(dac76d63-ddd3-4603-8d80-c1fb017e9246,RunLocust(DryRun(),Some(MASTERIP))) to Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker#-2093331180] with path akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker 3..Cluster on worker machine records dead letter WORKER SYSTEM: INFO [2015-10-01 09:01:31,823] [] [] [dispatcher-4] a.a.EmptyLocalActorRef akka://ClusterSystem/user/worker - Message [mycompany.akka.cluster.Work] from Actor[akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton#-83415853] to Actor[akka://ClusterSystem/user/worker#-2093331180] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. WORKER SYSTEM: INFO [2015-10-01 09:01:31,831] [] [] [dispatcher-3] a.a.EmptyLocalActorRef akka://ClusterSystem/user/worker - Message [mycompany.akka.cluster.Work] from Actor[akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton#-83415853] to Actor[akka://ClusterSystem/user/worker] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Here's the relevant output. Added local pinging of worker to show that it is indeed up and at the correct path Box 1 - Not Leader; Worker running here INFO [2015-10-01 09:01:05,017] [] [] [dispatcher-16] akka.remote.Remoting akka.remote.Remoting - Starting remoting INFO [2015-10-01 09:01:05,134] [] [] [dispatcher-16] akka.remote.Remoting akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://ClusterSystem@SLAVEIP:12489] INFO [2015-10-01 09:01:05,135] [] [] [dispatcher-16] akka.remote.Remoting akka.remote.Remoting - Remoting now listens on addresses: [akka.tcp://ClusterSystem@SLAVEIP:12489] INFO [2015-10-01 09:01:05,164] [] [] [dispatcher-16] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12489] - Starting up... INFO [2015-10-01 09:01:05,168] [] [] [dispatcher-16] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12489] - Started up successfully INFO [2015-10-01 09:01:05,172] [] [] [dispatcher-4] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12489] - Metrics collection has started successfully INFO [2015-10-01 09:01:05,182] [] [] [dispatcher-3] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12489] - No seed-nodes configured, manual cluster join required DEBUG [2015-10-01 09:01:05,285] [] [] [dispatcher-17] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker 3d5e4a3f-2a87-4256-9067-ec60cb377f53 with path akka://ClusterSystem/user/worker sending RegisterWorker(3d5e4a3f-2a87-4256-9067-ec60cb377f53) to master at Actor[akka://ClusterSystem/user/$b#-1210252092] DEBUG [2015-10-01 09:01:05,342] [] [] [dispatcher-17] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker with path akka://ClusterSystem/user/worker received ping. INFO [2015-10-01 09:01:05,384] [] [] [dispatcher-18] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12489] - Node [akka.tcp://ClusterSystem@SLAVEIP:12489] is JOINING, roles [] INFO [2015-10-01 09:01:06,215] [] [] [dispatcher-18] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12489] - Leader is moving node [akka.tcp://ClusterSystem@SLAVEIP:12489] to [Up] DEBUG [2015-10-01 09:01:10,262] [] [] [dispatcher-18] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker with path akka://ClusterSystem/user/worker received ping. DEBUG [2015-10-01 09:01:15,272] [] [] [dispatcher-3] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker with path akka://ClusterSystem/user/worker received ping. INFO [2015-10-01 09:01:16,792] [] [] [dispatcher-4] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@SLAVEIP:12487] - Welcome from [akka.tcp://ClusterSystem@MASTERIP:12487] INFO [2015-10-01 09:01:17,660] [] [] [dispatcher-4] a.c.s.ClusterSingletonProxy akka.tcp://ClusterSystem@SLAVEIP:12487/user/$b - Singleton identified at [akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton] INFO [2015-10-01 09:01:18,091] [] [] [dispatcher-4] a.c.s.ClusterSingletonManager akka.tcp://ClusterSystem@SLAVEIP:12487/user/master - ClusterSingletonManager state change [Start -> Younger] DEBUG [2015-10-01 09:01:20,265] [] [] [dispatcher-17] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker with path akka://ClusterSystem/user/worker received ping. DEBUG [2015-10-01 09:01:31,810] [] [] [] c.t.m.l.a.TestManager - Received ack from master for work: Work(dac76d63-ddd3-4603-8d80-c1fb017e9246,RunLocust(DryRun(),None)) INFO [2015-10-01 09:01:31,823] [] [] [dispatcher-4] a.a.EmptyLocalActorRef akka://ClusterSystem/user/worker - Message [mycompany.akka.cluster.Work] from Actor[akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton#-83415853] to Actor[akka://ClusterSystem/user/worker#-2093331180] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO [2015-10-01 09:01:31,831] [] [] [dispatcher-3] a.a.EmptyLocalActorRef akka://ClusterSystem/user/worker - Message [mycompany.akka.cluster.Work] from Actor[akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton#-83415853] to Actor[akka://ClusterSystem/user/worker] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. DEBUG [2015-10-01 09:01:35,261] [] [] [dispatcher-4] c.t.a.cluster.Worker akka.tcp://ClusterSystem@SLAVEIP:12489/user/worker - Worker with path akka://ClusterSystem/user/worker received ping. Box 2 - Leader/Master INFO [2015-10-01 09:01:09,667] [] [] [dispatcher-17] akka.remote.Remoting akka.remote.Remoting - Starting remoting INFO [2015-10-01 09:01:10,224] [] [] [dispatcher-17] akka.remote.Remoting akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://ClusterSystem@MASTERIP:12489] INFO [2015-10-01 09:01:10,225] [] [] [dispatcher-17] akka.remote.Remoting akka.remote.Remoting - Remoting now listens on addresses: [akka.tcp://ClusterSystem@MASTERIP:12489] INFO [2015-10-01 09:01:10,226] [] [] [dispatcher-17] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12489] - Starting up... INFO [2015-10-01 09:01:10,235] [] [] [dispatcher-17] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12489] - Started up successfully INFO [2015-10-01 09:01:10,275] [] [] [dispatcher-4] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12489] - No seed-nodes configured, manual cluster join required DEBUG [2015-10-01 09:01:10,508] [] [] [dispatcher-18] c.t.a.cluster.Worker akka.tcp://ClusterSystem@MASTERIP:12489/user/worker - Worker d5f55d4b-1198-4f25-b72c-814325b0d198 with path akka://ClusterSystem/user/worker sending RegisterWorker(d5f55d4b-1198-4f25-b72c-814325b0d198) to master at Actor[akka://ClusterSystem/user/$b#643942838] INFO [2015-10-01 09:01:10,628] [] [] [dispatcher-16] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12489] - Node [akka.tcp://ClusterSystem@MASTERIP:12489] is JOINING, roles [] INFO [2015-10-01 09:01:11,344] [] [] [dispatcher-3] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12487] - Node [akka.tcp://ClusterSystem@MASTERIP:12487] is JOINING, roles [master] INFO [2015-10-01 09:01:11,351] [] [] [dispatcher-3] a.actor.LocalActorRef akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1 - Message [akka.cluster.InternalClusterAction$InitJoinNack] from Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/system/cluster/core/daemon#-867905712] to Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#2098879220] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO [2015-10-01 09:01:11,394] [] [] [dispatcher-17] a.actor.LocalActorRef akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1 - Message [akka.cluster.InternalClusterAction$InitJoinNack] from Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/system/cluster/core/daemon#-867905712] to Actor[akka://ClusterSystem/system/cluster/core/daemon/firstSeedNodeProcess-1#2098879220] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. INFO [2015-10-01 09:01:11,418] [] [] [dispatcher-16] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12489] - Leader is moving node [akka.tcp://ClusterSystem@MASTERIP:12489] to [Up] INFO [2015-10-01 09:01:12,019] [] [] [dispatcher-3] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12487] - Leader is moving node [akka.tcp://ClusterSystem@MASTERIP:12487] to [Up] INFO [2015-10-01 09:01:12,097] [] [] [dispatcher-5] a.c.s.ClusterSingletonManager akka.tcp://ClusterSystem@MASTERIP:12487/user/master - Singleton manager starting singleton actor [akka://ClusterSystem/user/master/singleton] INFO [2015-10-01 09:01:12,100] [] [] [dispatcher-5] a.c.s.ClusterSingletonManager akka.tcp://ClusterSystem@MASTERIP:12487/user/master - ClusterSingletonManager state change [Start -> Oldest] INFO [2015-10-01 09:01:16,434] [] [] [dispatcher-18] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12487] - Node [akka.tcp://ClusterSystem@SLAVEIP:12487] is JOINING, roles [master] INFO [2015-10-01 09:01:17,018] [] [] [dispatcher-17] a.c.Cluster(akka://ClusterSystem) akka.cluster.Cluster(akka://ClusterSystem) - Cluster Node [akka.tcp://ClusterSystem@MASTERIP:12487] - Leader is moving node [akka.tcp://ClusterSystem@SLAVEIP:12487] to [Up] INFO [2015-10-01 09:01:18,544] [] [] [dispatcher-17] c.t.m.l.a.LoadTestMaster akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton - WorkerId 3d5e4a3f-2a87-4256-9067-ec60cb377f53 registered for Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker#-2093331180] INFO [2015-10-01 09:01:31,772] [] [] [dispatcher-3] c.t.m.l.a.LoadTestMaster akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton - Accepted work: dac76d63-ddd3-4603-8d80-c1fb017e9246. Sending ack to Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/temp/$a] DEBUG [2015-10-01 09:01:31,804] [] [] [dispatcher-3] c.t.m.l.a.LoadTestMaster akka.tcp://ClusterSystem@MASTERIP:12487/user/master/singleton - Master sending Work(dac76d63-ddd3-4603-8d80-c1fb017e9246,RunLocust(DryRun(),Some(MASTERIP))) to Actor[akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker#-2093331180] with path akka.tcp://ClusterSystem@SLAVEIP:12487/user/worker Thanks! -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
