Allan Douglas R. de Oliveira created SPARK-3332: ---------------------------------------------------
Summary: Tags shouldn't be the main strategy for machine membership on clusters Key: SPARK-3332 URL: https://issues.apache.org/jira/browse/SPARK-3332 Project: Spark Issue Type: Bug Components: EC2 Reporter: Allan Douglas R. de Oliveira The implementation for SPARK-2333 changed the machine membership mechanism from security groups to tags. This is a fundamentally flawed strategy as there aren't guarantees at all the machines will have a tag (even with a retry mechanism). For instance, if the script is killed after launching the instances but before setting the tags the machines will be "invisible" to a destroy command, leaving a unmanageable cluster behind. The initial proposal is to go back to the previous behavior for all cases but when the new flag (--security-group-prefix) is used. Also it's worthwhile to mention that SPARK-3180 introduced the --additional-security-group flag which is a reasonable solution to SPARK-2333 (but isn't a full replacement to all use cases of --security-group-prefix). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org