Hi Chanan, I am looking at your code, and I think I found your problem.
You launch the GameActor (and thus, the Worker actor) on the system stored in the actorSystem variable (named "Application"). You use however the system stored in the querySystem variable (named "GameClusterSystemQuery") to look up the extension. What happens from here: - actorSystem passes the message directly to the foreign systems sharding extension ActorRef, therefore it is a local send (does not contain any remoting information, since you use the actual ActorRef *instance* that is completely owned by the other system). This in itself works - now querySystem's shard proxy will forward this message to the backend system. Since the sender ActorRef contains *no remoting information*, it attaches its own address when serializing the ref - now backend receives query, and sends reply back to the sender, which contains the remoting address of *querySystem* - querySystem gets the reply, and since it sees its own address, it tries to look up the actor, which he considers to be his own. - lookup fails, message ends up in dead letters The underlying issue here is that ActorRefs has two major kinds: - LocalActorRef implements "tell" as a direct enqueue to a mailbox. It does not involve any remoting metadata, etc. otherwise local message sending would be prohibitively slow. It simply emqueues the message to the local mailbox using local method calls on the JVM. - RemoteActorRef implements "tell" as passing the message down to remoting, which attaches to all actor-refs leaving the system the necessary remoting metadata, so any other system can reply with a message to the proper destination, no matter how many systems it travelled through. What you did in your code is that you aquired the LocalActorRef of the extension of querySystem from actorSystem. Since you called "tell" on that ActorRef, it circumvented the whole remoting infrastructure, causing querySystem to treat it as a local send. Then querySystem used the RemoteActorRef of the necessary cluster infrastructure actors, and therefore remoting dutifully added the necessary metadata: assuming that the sender reference was local. In general, while it is allowed to use the LocalActorRef's of foreign systems direcly for performance reasons, this is highly discouraged if any of the references involved (e.g. sender()) can leak through remoting. If for example you aquired the reference of the extension in querySystem from actorSystem via a remote ActorSelection, then you would get the RemoteActorRef of the extension. As a general rule, you should not acquire extensions directly from other systems. I don't know if you intentionally used a foreign extension ActorRef, or if it was a mistake, but if you for some reason want to keep those actor systems separate, your best bet would be to start up the extension *also* on actorSystem, and use that one directly, or set up proxy actors on querySystem which resend the message to the extension on that system, saving the original sender an using themselves as sender for the query, then replying to the original sender when an answer arrived. I am not sure this is worth it though. I attached a patch that launches the actors on querySystem directly instead of actorSystem, and then your sample works. -Endre On Sat, May 9, 2015 at 1:52 PM, Chanan Braunstein <chan...@gmail.com> wrote: > Endre, > > I am sorry, but I do not see how this error can be in my code. It is not > even reach my actor even though the print out from deadletter has the right > actor. > > I am using the correct logger, I just happened to paste the previous by > mistake from Master which uses the play logger (which by the way has no > problem printing from within akka): > > > https://github.com/chanan/java-cqrs-starter/blob/feature/ask-not-working/app/actors/GameActor.java#L42 > > This line is never printed instead I get: > > [INFO] [05/09/2015 07:44:15.456] > [GameClusterSystemQuery-akka.actor.default-dispatcher-21] > [akka://GameClusterSystemQuery/user/$a/$a] Message > [com.example.protocols.Queries$PlayerNameResponse] from Actor[akka.tcp:// > GameClusterSystemQuery@127.0.0.1:2553/user/sharding/PlayerRegionQuery/Test-1#1475000342] > to Actor[akka://GameClusterSystemQuery/user/$a/$a#234612887] was not > delivered. [1] dead letters encountered. This logging can be turned off or > adjusted with configuration settings 'akka.log-dead-letters' and > 'akka.log-dead-letters-during-shutdown'. > > Which exactly matches the sender() that the cluster is using: > > [INFO] [05/09/2015 07:44:15.450] > [GameClusterSystemQuery-akka.actor.default-dispatcher-30] [akka.tcp:// > GameClusterSystemQuery@127.0.0.1:2553/user/sharding/PlayerRegionQuery/Test-1] > Sender: Actor[akka.tcp:// > GameClusterSystemQuery@127.0.0.1:61165/user/$a/$a#234612887] > > > I am not sure how my code has anything to do with this issue since my code > is never reached. > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Don_t_use_foreign_extensions_from_another_actor_system.patch
Description: Binary data
