[jira] [Assigned] (SPARK-925) Allow ec2 scripts to load default options from a json file
[
https://issues.apache.org/jira/browse/SPARK-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-925:
--
Assignee: Apache Spark
Allow ec2 scripts to load default options from a json file
--
Key: SPARK-925
URL: https://issues.apache.org/jira/browse/SPARK-925
Project: Spark
Issue Type: Improvement
Components: EC2
Affects Versions: 0.8.0
Reporter: Shay Seng
Assignee: Apache Spark
Priority: Minor
The option list for ec2 script can be a little irritating to type in,
especially things like path to identity-file, region , zone, ami etc.
It would be nice if ec2 script looks for an options.json file in the
following order: (1) CWD, (2) ~/spark-ec2, (3) same dir as spark_ec2.py
Something like:
def get_defaults_from_options():
# Check to see if a options.json file exists, if so load it.
# However, values in the options.json file can only overide values in opts
# if the Opt values are None or
# i.e. commandline options take presidence
defaults =
{'aws-access-key-id':'','aws-secret-access-key':'','key-pair':'',
'identity-file':'', 'region':'ap-southeast-1', 'zone':'',
'ami':'','slaves':1, 'instance-type':'m1.large'}
# Look for options.json in directory cluster was called from
# Had to modify the spark_ec2 wrapper script since it mangles the pwd
startwd = os.environ['STARTWD']
if os.path.exists(os.path.join(startwd,options.json)):
optionspath = os.path.join(startwd,options.json)
else:
optionspath = os.path.join(os.getcwd(),options.json)
try:
print Loading options file: , optionspath
with open (optionspath) as json_data:
jdata = json.load(json_data)
for k in jdata:
defaults[k]=jdata[k]
except IOError:
print 'Warning: options.json file not loaded'
# Check permissions on identity-file, if defined, otherwise launch will
fail late and will be irritating
if defaults['identity-file']!='':
st = os.stat(defaults['identity-file'])
user_can_read = bool(st.st_mode stat.S_IRUSR)
grp_perms = bool(st.st_mode stat.S_IRWXG)
others_perm = bool(st.st_mode stat.S_IRWXO)
if (not user_can_read):
print No read permission to read , defaults['identify-file']
sys.exit(1)
if (grp_perms or others_perm):
print Permissions are too open, please chmod 600 file ,
defaults['identify-file']
sys.exit(1)
# if defaults contain AWS access id or private key, set it to environment.
# required for use with boto to access the AWS console
if defaults['aws-access-key-id'] != '':
os.environ['AWS_ACCESS_KEY_ID']=defaults['aws-access-key-id']
if defaults['aws-secret-access-key'] != '':
os.environ['AWS_SECRET_ACCESS_KEY'] = defaults['aws-secret-access-key']
return defaults
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-925) Allow ec2 scripts to load default options from a json file
[
https://issues.apache.org/jira/browse/SPARK-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-925:
--
Assignee: (was: Apache Spark)
Allow ec2 scripts to load default options from a json file
--
Key: SPARK-925
URL: https://issues.apache.org/jira/browse/SPARK-925
Project: Spark
Issue Type: Improvement
Components: EC2
Affects Versions: 0.8.0
Reporter: Shay Seng
Priority: Minor
The option list for ec2 script can be a little irritating to type in,
especially things like path to identity-file, region , zone, ami etc.
It would be nice if ec2 script looks for an options.json file in the
following order: (1) CWD, (2) ~/spark-ec2, (3) same dir as spark_ec2.py
Something like:
def get_defaults_from_options():
# Check to see if a options.json file exists, if so load it.
# However, values in the options.json file can only overide values in opts
# if the Opt values are None or
# i.e. commandline options take presidence
defaults =
{'aws-access-key-id':'','aws-secret-access-key':'','key-pair':'',
'identity-file':'', 'region':'ap-southeast-1', 'zone':'',
'ami':'','slaves':1, 'instance-type':'m1.large'}
# Look for options.json in directory cluster was called from
# Had to modify the spark_ec2 wrapper script since it mangles the pwd
startwd = os.environ['STARTWD']
if os.path.exists(os.path.join(startwd,options.json)):
optionspath = os.path.join(startwd,options.json)
else:
optionspath = os.path.join(os.getcwd(),options.json)
try:
print Loading options file: , optionspath
with open (optionspath) as json_data:
jdata = json.load(json_data)
for k in jdata:
defaults[k]=jdata[k]
except IOError:
print 'Warning: options.json file not loaded'
# Check permissions on identity-file, if defined, otherwise launch will
fail late and will be irritating
if defaults['identity-file']!='':
st = os.stat(defaults['identity-file'])
user_can_read = bool(st.st_mode stat.S_IRUSR)
grp_perms = bool(st.st_mode stat.S_IRWXG)
others_perm = bool(st.st_mode stat.S_IRWXO)
if (not user_can_read):
print No read permission to read , defaults['identify-file']
sys.exit(1)
if (grp_perms or others_perm):
print Permissions are too open, please chmod 600 file ,
defaults['identify-file']
sys.exit(1)
# if defaults contain AWS access id or private key, set it to environment.
# required for use with boto to access the AWS console
if defaults['aws-access-key-id'] != '':
os.environ['AWS_ACCESS_KEY_ID']=defaults['aws-access-key-id']
if defaults['aws-secret-access-key'] != '':
os.environ['AWS_SECRET_ACCESS_KEY'] = defaults['aws-secret-access-key']
return defaults
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
