[jira] [Commented] (SPARK-32333) Drop references to Master
[
https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405904#comment-17405904
]
Erik Krogen commented on SPARK-32333:
-
Personally, tackling publicly-facing things is my highest priority. This blocks
other projects/users from being able to pursue their own efforts in this area.
For example, internally we have code with references to the {{--master}}
argument, and can't drop these references because of lack of an alternative. So
this would be (1) above, though I'll note that things like
{{SparkConf.setMaster}} are used beyond the context of standalone mode.
For (2) / (3), I'm actually kind of surprised to see there's not _too_ many:
{code}
. find . -name "*.scala" -or -name "*.java" | xargs grep -o -i -E "class
[[:alpha:][:digit:]]*master[[:alpha:][:digit:]]*"
./resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ApplicationMasterSuite.scala:class
ApplicationMasterSuite
./resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala:class
ApplicationMasterArguments
./resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:class
ApplicationMaster
./resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterSource.scala:class
ApplicationMasterSource
./core/src/test/scala/org/apache/spark/deploy/master/ui/MasterWebUISuite.scala:class
MasterWebUISuite
./core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala:class
MasterSuite
./core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala:class
DummyMaster
./core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala:class
SmarterMaster
./core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:class
MyBlockManagerMaster
./core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:class
MyMapOutputTrackerMaster
./core/src/test/scala/org/apache/spark/storage/BlockManagerMasterSuite.scala:class
BlockManagerMasterSuite
./core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala:class
MasterChangeAcknowledged
./core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala:class
MasterChanged
./core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala:class
MasterStateResponse
./core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala:class
TestMasterInfo
./core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:class
MasterPage
./core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala:class
MasterWebUI
./core/src/main/scala/org/apache/spark/deploy/master/MasterArguments.scala:class
MasterArguments
./core/src/main/scala/org/apache/spark/deploy/master/Master.scala:class Master
./core/src/main/scala/org/apache/spark/deploy/master/MasterSource.scala:class
MasterSource
./core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:class
BlockManagerMaster
./core/src/main/scala/org/apache/spark/storage/BlockManagerMasterHeartbeatEndpoint.scala:class
BlockManagerMasterHeartbeatEndpoint
./core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala:class
BlockManagerMasterEndpoint
./core/src/main/scala/org/apache/spark/MapOutputTracker.scala:class
MapOutputTrackerMasterEndpoint
./core/src/main/scala/org/apache/spark/MapOutputTracker.scala:class
MapOutputTrackerMaster
{code}
So it seems that it's likely reasonable to tackle given the breakdown you've
outlined. Though the entire {{org.apache.spark.deploy.master}} package will
also have to be renamed as part of (2).
> Drop references to Master
> -
>
> Key: SPARK-32333
> URL: https://issues.apache.org/jira/browse/SPARK-32333
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
>Affects Versions: 3.0.0
>Reporter: Thomas Graves
>Priority: Major
>
> We have a lot of references to "master" in the code base. It will be
> beneficial to remove references to problematic language that can alienate
> potential community members.
> SPARK-32004 removed references to slave
>
> Here is a IETF draft to fix up some of the most egregious examples
> (master/slave, whitelist/backlist) with proposed alternatives.
> https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405814#comment-17405814 ] Thomas Graves commented on SPARK-32333: --- I was looking to break this up into subtasks but not sure how much we will be able to, perhaps something like this: Note we need to keep backwards compatibility so rename is copy for any public api's/scripts. Also note, one thing we may not want to change or discuss more is rename ApplicationMaster since that is the name Hadoop uses for it. We can certainly change internal api's and functions but external we may not want to. 1) Rename the standalone Master and public apis (SparkConf.setMaster), scripts, docs that reference those scripts 2) Rename standalone Master classes that aren't public - MasterArguments, MasterUI, MasterMessages, etc 3) Rename internal classes and messages with Master in name, note we could also rename class with Master in the name of them like BlockManagerMaster, MapOutputTrackerMaster, etc. We could break this up further if people are interesting in helping. > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403348#comment-17403348 ] Neil Shah-Quinn commented on SPARK-32333: - I like "leader" too. > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379239#comment-17379239 ] Erik Krogen commented on SPARK-32333: - +1 on leader from my side > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378093#comment-17378093 ] Thomas Graves commented on SPARK-32333: --- Getting back to this, now that spark 3.2 branch is cut, perhaps we can target for 3.3. >From the discussion thread on spark-dev mailing list Leader was mentioned the >most, Scheduler a second. One reason against controller, coordinator, application manager, primary as it perhaps implies being needed and if the standalone master goes down the apps are unaffected. Based on the feedback, I propose "Leader" based on feedback and it being short. > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270058#comment-17270058 ] Jacek Laskowski commented on SPARK-32333: - Just today when I was reading about a new ASF project - Apache Gobblin I found (highlighting mine): "Runs as a standalone cluster with *primary* and worker nodes." This "primary node" makes a lot of sense. > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264303#comment-17264303 ] Neil Shah-Quinn commented on SPARK-32333: - I'm glad there's a plan to improve this language! For what it's worth, I like "Scheduler" or "Coordinator". They're short and accurately reflect that (as I understand it) its role is simply to assign executors which then communicate directly with the driver program. > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184051#comment-17184051 ] Thomas Graves commented on SPARK-32333: --- I send email to the dev list to get feedback, some other suggestions in that email here: A few name possibilities: - ApplicationManager - StandaloneClusterManager - Coordinator - Primary - Controller That chain can be found here: [http://apache-spark-developers-list.1001551.n3.nabble.com/Removing-references-to-Master-td29948.html] > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32333) Drop references to Master
[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170451#comment-17170451 ] Thomas Graves commented on SPARK-32333: --- What about renaming Master to be ApplicationManager or StandaloneClusterManager? > Drop references to Master > - > > Key: SPARK-32333 > URL: https://issues.apache.org/jira/browse/SPARK-32333 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Thomas Graves >Priority: Major > > We have a lot of references to "master" in the code base. It will be > beneficial to remove references to problematic language that can alienate > potential community members. > SPARK-32004 removed references to slave > > Here is a IETF draft to fix up some of the most egregious examples > (master/slave, whitelist/backlist) with proposed alternatives. > https://tools.ietf.org/id/draft-knodel-terminology-00.html#rfc.section.1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
