I'll admit, I didn't know you could deploy multiple workers per machine. I agree, I don't see the use case for it? multiple executors, yes of course. And I guess you could imagine multiple distinct Spark clusters running a worker on one machine. I don't have an informed opinion therefore, but agree that it seems like a best practice enough to enforce 1 worker per machine, if it makes things simpler rather than harder.
On Fri, Feb 28, 2020 at 1:21 PM Xingbo Jiang <jiangxb1...@gmail.com> wrote: > > Hi all, > > Based on my experience, there is no scenario that necessarily requires > deploying multiple Workers on the same node with Standalone backend. A worker > should book all the resources reserved to Spark on the host it is launched, > then it can allocate those resources to one or more executors launched by > this worker. Since each executor runs in a separated JVM, we can limit the > memory of each executor to avoid long GC pause. > > The remaining concern is the local-cluster mode is implemented by launching > multiple workers on the local host, we might need to re-implement > LocalSparkCluster to launch only one Worker and multiple executors. It should > be fine because local-cluster mode is only used in running Spark unit test > cases, thus end users should not be affected by this change. > > Removing multiple workers on the same host support could simplify the deploy > model of Standalone backend, and also reduce the burden to support legacy > deploy pattern in the future feature developments. (There is an example in > https://issues.apache.org/jira/browse/SPARK-27371 , where we designed a > complex approach to coordinate resource requirements from different workers > launched on the same host). > > The proposal is to update the document to deprecate the support of system > environment `SPARK_WORKER_INSTANCES` in Spark 3.0, and remove the support in > the next major version (Spark 3.1). > > Please kindly let me know if you have use cases relying on this feature. > > Thanks! > > Xingbo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
