[
https://issues.apache.org/jira/browse/SPARK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187900#comment-14187900
]
Nicholas Chammas edited comment on SPARK-3398 at 10/29/14 2:48 AM:
-------------------------------------------------------------------
If that fixes it for you, then I think the solution is simple. -We just need to
set {{cwd}} to the user's current working directory in all our calls to
[{{subprocess.check_call()}}|https://docs.python.org/2/library/subprocess.html#subprocess.check_call].
Right now it defaults to the {{spark-ec2}} directory, which will be
problematic if you call {{spark-ec2}} from another directory.-
We need to fix [how the script gets called
here|https://github.com/apache/spark/blob/4b55482abf899c27da3d55401ad26b4e9247b327/ec2/spark-ec2#L22].
was (Author: nchammas):
If that fixes it for you, then I think the solution is simple. We just need to
set {{cwd}} to the user's current working directory in all our calls to
[{{subprocess.check_call()}}|https://docs.python.org/2/library/subprocess.html#subprocess.check_call].
Right now it defaults to the {{spark-ec2}} directory, which will be
problematic if you call {{spark-ec2}} from another directory.
> Have spark-ec2 intelligently wait for specific cluster states
> -------------------------------------------------------------
>
> Key: SPARK-3398
> URL: https://issues.apache.org/jira/browse/SPARK-3398
> Project: Spark
> Issue Type: Improvement
> Components: EC2
> Reporter: Nicholas Chammas
> Assignee: Nicholas Chammas
> Priority: Minor
> Fix For: 1.2.0
>
>
> {{spark-ec2}} currently has retry logic for when it tries to install stuff on
> a cluster and for when it tries to destroy security groups.
> It would be better to have some logic that allows {{spark-ec2}} to explicitly
> wait for when all the nodes in a cluster it is working on have reached a
> specific state.
> Examples:
> * Wait for all nodes to be up
> * Wait for all nodes to be up and accepting SSH connections (then start
> installing stuff)
> * Wait for all nodes to be down
> * Wait for all nodes to be terminated (then delete the security groups)
> Having a function in the {{spark_ec2.py}} script that blocks until the
> desired cluster state is reached would reduce the need for various retry
> logic. It would probably also eliminate the need for the {{--wait}} parameter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
