[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138590#comment-15138590 ] Sean Owen commented on SPARK-5629: -- I'm bulk closing some EC2 issues as the ones that are still live can be 'ported' to the new amplab repo > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15124613#comment-15124613 ] Shivaram Venkataraman commented on SPARK-5629: -- I manually cleaned up some of the issues and pointed them to open issues on spark-ec2. I think for some of the issues we should just ping the issue and see if its still a relevant issue. Finally I think some of the S3 reading issues aren't spark-ec2 issues but more an issue with jets3t etc. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118972#comment-15118972 ] Sean Owen commented on SPARK-5629: -- OK, I'm not sure how to implement that. There are lots of EC2 issues that haven't been touched in a year. I tend to think those are ones nobody is concerned with any longer. Maybe the reverse -- when someone cares enough to open the spark-ec2 issues for business, go back and search for "EC2" issues and port those that matter? > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111056#comment-15111056 ] Shivaram Venkataraman commented on SPARK-5629: -- Yes - though I think its beneficial to see if the ticket is still valid and if it is, we can open a corresponding issue at github.com/amplab/spark-ec2. Then we can leave a marker here saying where this issue is being followed up at. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110853#comment-15110853 ] Sean Owen commented on SPARK-5629: -- Are all of the EC2 tickets becoming essentially "wont fix" as the support moves out of Spark itself? > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329940#comment-14329940 ] Florian Verhein commented on SPARK-5629: Agree. My point was more about avoiding tying `--machine-readable` to a particular encoding, as adding any more later (if needed, and it appears from this discussion that this is possible) would then make backwards compatibility ugly/hard. My vote would be for json too. Aside: I saw some value in bash (export variables) because spark_ec2.py is suited for use via its CLI, so scripting bash around it is natural. However on further thought I don't think that would be a good idea, because a) its ugly to implement, and b) once a script gets to the complexity of requiring these variables, it should really be refactored into something more suitable like python. Also the existing `--get-master` should be sufficient for most use cases. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329771#comment-14329771 ] Nicholas Chammas commented on SPARK-5629: - [~florianverhein] - Hmm... Thinking about this for a bit, I'd be against introducing such flexibility right off the bat. Every additional option or non-standard flow will impose a maintenance burden. By that argument, actually, we should also just stick to JSON even though its less readable. As in, {{describe}} outputs in pretty-printed JSON and that's it. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328658#comment-14328658 ] Florian Verhein commented on SPARK-5629: Good idea. Can I suggest keeping a format specifier and leaving encodings open? For example YAML is great for readability, but json is much more easily consumed in python etc. Something like `--encoding [json|yaml|...]`. Then start with the "most useful" encoding (or the "simplest" -- json output is a one liner once you have the data structure). > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325372#comment-14325372 ] Nicholas Chammas commented on SPARK-5629: - YAML is not part of the Python standard library, unfortunately. Agree on somehow marking this as experimental. I think YAML is like JSON in that adding or removing fields shouldn't break any parsers. There is no pre-defined schema. It should just affect you if you are trying to access a field that was removed, for example, just like it would with JSON. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325354#comment-14325354 ] Shivaram Venkataraman commented on SPARK-5629: -- This sounds fine to me and I really like YAML -- Does Python have native support for printing out YAML ? One thing we should do is probably marking this as experimental as we might not be able maintain backwards compatibility etc. (On that note are YAML parsers backwards compatible ? i.e. if we add a new field in the next release will it break things ?) > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325346#comment-14325346 ] Nicholas Chammas commented on SPARK-5629: - For example, you run: {code} $ spark-ec2 describe my-spark-cluster {code} And you get back something like this: {code} my-spark-cluster: launched: "2015-02-18 14:03:22 UTC" status: running nodes: master: ec2-54-69-105-224.us-west-2.compute.amazonaws.com slaves: - ec2-54-69-1215-97.us-west-2.compute.amazonaws.com - ec2-54-69-186-101.us-west-2.compute.amazonaws.com - ec2-54-69-186-109.us-west-2.compute.amazonaws.com {code} Actually, since this is both valid YAML and very human-readable, we probably don't need the {{--machine-readable}} option mentioned in the ticket body. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324907#comment-14324907 ] Shivaram Venkataraman commented on SPARK-5629: -- Is there an example output for `describe` you have in mind ? And I am not sure it'll be easy to list all the clusters as spark-ec2 looks up clusters by the security group / cluster-id ? > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-5629) Add spark-ec2 action to return info about an existing cluster
[ https://issues.apache.org/jira/browse/SPARK-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324888#comment-14324888 ] Nicholas Chammas commented on SPARK-5629: - cc [~joshrosen] / [~shivaram] I see that we already have a {{get-master}} action which will probably serve most use cases where spark-ec2 is being used as part of some automated pipeline (e.g. spark-perf testing). Typically, you just want the master address so you can ssh in and do stuff. Still, I'm looking for your initial reaction to this proposal. > Add spark-ec2 action to return info about an existing cluster > - > > Key: SPARK-5629 > URL: https://issues.apache.org/jira/browse/SPARK-5629 > Project: Spark > Issue Type: Improvement > Components: EC2 >Reporter: Nicholas Chammas >Priority: Minor > > You can launch multiple clusters using spark-ec2. At some point, you might > just want to get some information about an existing cluster. > Use cases include: > * Wanting to check something about your cluster in the EC2 web console. > * Wanting to feed information about your cluster to another tool (e.g. as > described in [SPARK-5627]). > So, in addition to the [existing > actions|https://github.com/apache/spark/blob/9b746f380869b54d673e3758ca5e4475f76c864a/ec2/spark_ec2.py#L115]: > * {{launch}} > * {{destroy}} > * {{login}} > * {{stop}} > * {{start}} > * {{get-master}} > * {{reboot-slaves}} > We add a new action, {{describe}}, which describes an existing cluster if > given a cluster name, and all clusters if not. > Some examples: > {code} > # describes all clusters launched by spark-ec2 > spark-ec2 describe > {code} > {code} > # describes cluster-1 > spark-ec2 describe cluster-1 > {code} > In combination with the proposal in [SPARK-5627]: > {code} > # describes cluster-3 in a machine-readable way (e.g. JSON) > spark-ec2 describe cluster-3 --machine-readable > {code} > Parallels in similar tools include: > * [{{juju status}}|https://juju.ubuntu.com/docs/] from Ubuntu Juju > * [{{starcluster > listclusters}}|http://star.mit.edu/cluster/docs/latest/manual/getting_started.html?highlight=listclusters#logging-into-a-worker-node] > from MIT StarCluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org