[ https://issues.apache.org/jira/browse/SPARK-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335252#comment-14335252 ]
Marcelo Vanzin edited comment on SPARK-5124 at 2/24/15 7:03 PM: ---------------------------------------------------------------- Hi @rxin, Yeah, the current {{receive()}} gives you the message and the sender, but it doesn't give you a way to explicitly reply to a message. That means two things: - It's not clear whether calling something like {{sender.send()}} would be thread-safe. For example, in akka it would because akka processes one message at a time for each actor, so you're sending replies in order. But that forces you to implement the same semantics on any new RPC layer, and that is one akka feature thing that I'd prefer not to inherit (as you say, let each endpoint decide whether it needs it). - It precludes anynchronously handling that message for much of the same reasons. e.g., let's say some incoming message requires a long computation before sending the reply, and you use a thread pool for that. With {{sender.send()}}, you'd need to build correlation logic into your application. If you have something like {{message.reply()}} instead, you don't need to - the RPC layer handles correlations and timing out if a reply doesn't arrive. Granted, I have no idea how you'd implement that second bullet in akka, and perhaps for that reason I don't think Spark currently uses that pattern anywhere. But it sounds like a useful feature to have. was (Author: vanzin): Hi @rxin, Yeah, the current {{receive()}} gives you the message and the sender, but it doesn't give you a way to explicit reply to a message. That means two things: - It's not clear whether calling something like {{sender.send()}} would be thread-safe. For example, in akka it would because akka processes one message at a time for each actor, so you're sending replies in order. But that forces you to implement the same semantics on any new RPC layer, and that is one akka feature thing that I'd prefer not to inherit (as you say, let each endpoint decide whether it needs it). - It precludes anynchronously handling that message for much of the same reasons. e.g., let's say some incoming message requires a long computation before sending the reply, and you use a thread pool for that. With {{sender.send()}}, you'd need to build correlation logic into your application. If you have something like {{message.reply()}} instead, you don't need to - the RPC layer handles correlations and timing out if a reply doesn't arrive. Granted, I have no idea how you'd implement that second bullet in akka, and perhaps for that reason I don't think Spark currently uses that pattern anywhere. But it sounds like a useful feature to have. > Standardize internal RPC interface > ---------------------------------- > > Key: SPARK-5124 > URL: https://issues.apache.org/jira/browse/SPARK-5124 > Project: Spark > Issue Type: Sub-task > Components: Spark Core > Reporter: Reynold Xin > Assignee: Shixiong Zhu > Attachments: Pluggable RPC - draft 1.pdf, Pluggable RPC - draft 2.pdf > > > In Spark we use Akka as the RPC layer. It would be great if we can > standardize the internal RPC interface to facilitate testing. This will also > provide the foundation to try other RPC implementations in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org