[
https://issues.apache.org/jira/browse/SPARK-11095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu reopened SPARK-11095:
----------------------------------
> Simplify Netty RPC implementation by using a separate thread pool for each
> endpoint
> -----------------------------------------------------------------------------------
>
> Key: SPARK-11095
> URL: https://issues.apache.org/jira/browse/SPARK-11095
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Reynold Xin
> Assignee: Shixiong Zhu
> Priority: Major
> Labels: bulk-closed
>
> The dispatcher class and the inbox class of the current Netty-based RPC
> implementation is fairly complicated. It uses a single, shared thread pool to
> execute all the endpoints. This is similar to how Akka does actor message
> dispatching. The benefit of this design is that this RPC implementation can
> support a very large number of endpoints, as they are all multiplexed into a
> single thread pool for execution. The downside is the complexity resulting
> from synchronization and coordination.
> An alternative implementation is to have a separate message queue and thread
> pool for each endpoint. The dispatcher simply routes the messages to the
> appropriate message queue, and the threads poll the queue for messages to
> process.
> If the endpoint is single threaded, then the thread pool should contain only
> a single thread. If the endpoint supports concurrent execution, then the
> thread pool should contain more threads.
> Two additional things we need to be careful with are:
> 1. An endpoint should only process normal messages after OnStart is called.
> This can be done by having the thread that starts the endpoint processing
> OnStart.
> 2. An endpoint should process OnStop after all normal messages have been
> processed. I think this can be done by having a busy loop to spin until the
> size of the message queue is 0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
