[jira] [Updated] (SPARK-3332) Tagging is not atomic with launching instances on EC2
[ https://issues.apache.org/jira/browse/SPARK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3332: --- Summary: Tagging is not atomic with launching instances on EC2 (was: Tags shouldn't be the main strategy for machine membership on clusters) Tagging is not atomic with launching instances on EC2 - Key: SPARK-3332 URL: https://issues.apache.org/jira/browse/SPARK-3332 Project: Spark Issue Type: Bug Components: EC2 Reporter: Allan Douglas R. de Oliveira The implementation for SPARK-2333 changed the machine membership mechanism from security groups to tags. This is a fundamentally flawed strategy as there aren't guarantees at all the machines will have a tag (even with a retry mechanism). For instance, if the script is killed after launching the instances but before setting the tags the machines will be invisible to a destroy command, leaving a unmanageable cluster behind. The initial proposal is to go back to the previous behavior for all cases but when the new flag (--security-group-prefix) is used. Also it's worthwhile to mention that SPARK-3180 introduced the --additional-security-group flag which is a reasonable solution to SPARK-2333 (but isn't a full replacement to all use cases of --security-group-prefix). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-3332) Tagging is not atomic with launching instances on EC2
[ https://issues.apache.org/jira/browse/SPARK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3332: --- Target Version/s: 1.2.0 Tagging is not atomic with launching instances on EC2 - Key: SPARK-3332 URL: https://issues.apache.org/jira/browse/SPARK-3332 Project: Spark Issue Type: Bug Components: EC2 Reporter: Allan Douglas R. de Oliveira The implementation for SPARK-2333 changed the machine membership mechanism from security groups to tags. This is a fundamentally flawed strategy as there aren't guarantees at all the machines will have a tag (even with a retry mechanism). For instance, if the script is killed after launching the instances but before setting the tags the machines will be invisible to a destroy command, leaving a unmanageable cluster behind. The initial proposal is to go back to the previous behavior for all cases but when the new flag (--security-group-prefix) is used. Also it's worthwhile to mention that SPARK-3180 introduced the --additional-security-group flag which is a reasonable solution to SPARK-2333 (but isn't a full replacement to all use cases of --security-group-prefix). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
