[ https://issues.apache.org/jira/browse/SPARK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14116890#comment-14116890 ]
Apache Spark commented on SPARK-3332: ------------------------------------- User 'douglaz' has created a pull request for this issue: https://github.com/apache/spark/pull/2223 > Tags shouldn't be the main strategy for machine membership on clusters > ---------------------------------------------------------------------- > > Key: SPARK-3332 > URL: https://issues.apache.org/jira/browse/SPARK-3332 > Project: Spark > Issue Type: Bug > Components: EC2 > Reporter: Allan Douglas R. de Oliveira > > The implementation for SPARK-2333 changed the machine membership mechanism > from security groups to tags. > This is a fundamentally flawed strategy as there aren't guarantees at all the > machines will have a tag (even with a retry mechanism). > For instance, if the script is killed after launching the instances but > before setting the tags the machines will be "invisible" to a destroy > command, leaving a unmanageable cluster behind. > The initial proposal is to go back to the previous behavior for all cases but > when the new flag (--security-group-prefix) is used. > Also it's worthwhile to mention that SPARK-3180 introduced the > --additional-security-group flag which is a reasonable solution to SPARK-2333 > (but isn't a full replacement to all use cases of --security-group-prefix). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org