[jira] [Commented] (SPARK-4848) On a stand-alone cluster, several worker-specific variables are read only on the master
[ https://issues.apache.org/jira/browse/SPARK-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376239#comment-14376239 ] Apache Spark commented on SPARK-4848: - User 'nkronenfeld' has created a pull request for this issue: https://github.com/apache/spark/pull/5140 > On a stand-alone cluster, several worker-specific variables are read only on > the master > --- > > Key: SPARK-4848 > URL: https://issues.apache.org/jira/browse/SPARK-4848 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 1.0.0 > Environment: stand-alone spark cluster >Reporter: Nathan Kronenfeld > Original Estimate: 24h > Remaining Estimate: 24h > > On a stand-alone spark cluster, much of the determination of worker > specifics, especially one has multiple instances per node, is done only on > the master. > The master loops over instances, and starts a worker per instance on each > node. > This means, if your workers have different values of SPARK_WORKER_INSTANCES > or SPARK_WORKER_WEBUI_PORT from each other (or from the master), all values > are ignored except the one on the master. > SPARK_WORKER_PORT looks like it is unread in scripts, but read in code - I'm > not sure how it will behave, since all instances will read the same value > from the environment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4848) On a stand-alone cluster, several worker-specific variables are read only on the master
[ https://issues.apache.org/jira/browse/SPARK-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246391#comment-14246391 ] Apache Spark commented on SPARK-4848: - User 'nkronenfeld' has created a pull request for this issue: https://github.com/apache/spark/pull/3699 > On a stand-alone cluster, several worker-specific variables are read only on > the master > --- > > Key: SPARK-4848 > URL: https://issues.apache.org/jira/browse/SPARK-4848 > Project: Spark > Issue Type: Bug > Components: Project Infra > Environment: stand-alone spark cluster >Reporter: Nathan Kronenfeld > Original Estimate: 24h > Remaining Estimate: 24h > > On a stand-alone spark cluster, much of the determination of worker > specifics, especially one has multiple instances per node, is done only on > the master. > The master loops over instances, and starts a worker per instance on each > node. > This means, if your workers have different values of SPARK_WORKER_INSTANCES > or SPARK_WORKER_WEBUI_PORT from each other (or from the master), all values > are ignored except the one on the master. > SPARK_WORKER_PORT looks like it is unread in scripts, but read in code - I'm > not sure how it will behave, since all instances will read the same value > from the environment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
