[ https://issues.apache.org/jira/browse/SPARK-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348215#comment-14348215 ]
Shixiong Zhu edited comment on SPARK-5124 at 3/5/15 6:09 AM: ------------------------------------------------------------- I made some changes to the above APIs. 1. I added a new trait RpcResponse because RpcResponseCallback can not be used directly. Both `receive` and `receiveAndReply` methods can receive messages if the sender is an RpcEndpoint, so if the receiver wants to reply the sender, it needs an approach to specify which methods it wants to send. Here is the interface of RpcResponse: {quote} trait RpcResponse { def reply(response: Any): Unit def replyWithSender(response: Any, sender: RpcEndpointRef): Unit def fail(e: Throwable): Unit } {quote} Calling `reply` will send the message to RpcEndpoint.receive, and calling `replyWithSender` will send the message to RpcEndpoint.replyWithSender. 2.Insteading of adding a trait like ThreadSafeRpcEndpoint, I added a new method `setupThreadSafeEndpoint` in RpcEnv, so that RpcEnv can decide how to implement the thread-safe semantics internally. For `setupEndpoint`, RpcEnv won't guarantee sending messages thread-safely. [~vanzin] For the local Endpoint idea, I think we need a local message dispatcher to send/receive messages between different Endpoint. It can not be implemented just in the Endpoint. We will also need EndpointRef to refer to an Endpoint, and a generic trait for the local message dispatcher and RpcEnv. It looks a bit complex. What do you think? Please help review my PR: https://github.com/apache/spark/pull/4588 was (Author: zsxwing): I made some changes to the above APIs. 1. I added a new trait RpcResponse because RpcResponseCallback can not be used directly. Both `receive` and `receiveAndReply` methods can receive messages if the sender is an RpcEndpoint, so if the receiver wants to reply the sender, it needs an approach to specify which methods it wants to send. Here is the interface of RpcResponse: {quote} private[spark] trait RpcResponse { def reply(response: Any): Unit def replyWithSender(response: Any, sender: RpcEndpointRef): Unit def fail(e: Throwable): Unit } {quote} Calling `reply` will send the message to RpcEndpoint.receive, and calling `replyWithSender` will send the message to RpcEndpoint.replyWithSender. 2.Insteading of adding a trait like ThreadSafeRpcEndpoint, I added a new method `setupThreadSafeEndpoint` in RpcEnv, so that RpcEnv can decide how to implement the thread-safe semantics internally. For `setupEndpoint`, RpcEnv won't guarantee sending messages thread-safely. [~vanzin] For the local Endpoint idea, I think we need a local message dispatcher to send/receive messages between different Endpoint. It can not be implemented just in the Endpoint. We will also need EndpointRef to refer to an Endpoint, and a generic trait for the local message dispatcher and RpcEnv. It looks a bit complex. What do you think? Please help review my PR: https://github.com/apache/spark/pull/4588 > Standardize internal RPC interface > ---------------------------------- > > Key: SPARK-5124 > URL: https://issues.apache.org/jira/browse/SPARK-5124 > Project: Spark > Issue Type: Sub-task > Components: Spark Core > Reporter: Reynold Xin > Assignee: Shixiong Zhu > Attachments: Pluggable RPC - draft 1.pdf, Pluggable RPC - draft 2.pdf > > > In Spark we use Akka as the RPC layer. It would be great if we can > standardize the internal RPC interface to facilitate testing. This will also > provide the foundation to try other RPC implementations in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org