[ https://issues.apache.org/jira/browse/SPARK-5879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14328651#comment-14328651 ]
Florian Verhein commented on SPARK-5879: ---------------------------------------- Yeah that's more flexible - rather than deciding for the user what/how we should expose when they run spark_ec2.py, the user must instead request it. No significant downsides in having the additional invocation to spark_ec2.py required. > spary_ec2.py should expose/return master and slave lists (e.g. write to file) > ----------------------------------------------------------------------------- > > Key: SPARK-5879 > URL: https://issues.apache.org/jira/browse/SPARK-5879 > Project: Spark > Issue Type: Improvement > Components: EC2 > Reporter: Florian Verhein > > After running spark_ec2.py, it is often useful/necessary to know the master's > ip / dn. Particularly if running spark_ec2.py is part of a larger pipeline. > For example, consider a wrapper that launches a cluster, then waits for > completion of some application running on it (e.g. polling via ssh), before > destroying the cluster. > Some options: > - write `launch-variables.sh` with MASTERS and SLAVES exports (i.e. basically > a subset of the ec2_variables.sh that is temporarily created as part of > deploy_files variable substitution) > - launch-variables.json (same info but as json) > Both would be useful depending on the wrapper language. > I think we should incorporate the cluster name for the case that multiple > clusters are launched. E.g. <cluster_name>_variables.sh/.json > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org