Having worked with a few customers, we found the current organization of the scripts and configuration a bit confusing and inconvenient, so we propose to reorganize them. Below we described the specific reasons and the rough proposal, please kindly provide your opinions and suggestions. :)
Specific reasons for re-organization: 1) Usually the application developers/users and platform administrators belongs to two teams. So it's better to separate the scripts used by administrators and application users, e.g. put them in sbin and bin folders respectively 2) User level options and admin level options need to be separated. For example, an application user may never know how many spindles there are in the cluster nodes, so it's often the administrator's duty to specify spark.local.dir. 3) If there are multiple ways to specify an option, an overriding rule should be present and should not be error-prone. 4) Currently the options are set and get using System property. It's hard to manage and inconvenient for users. It's good to gather the options into one file using format like xml or json. Some previous work: 1) SPARK-544 contains a discussion about providing a configuration class for spark https://spark-project.atlassian.net/browse/SPARK-5<https://spark-project.atlassian.net/browse/SPARK-544> 44 2) Ankur has gathered spark options into a json format https://gist.github.com/ankurcha/5655646 Our rough proposal: - Scripts 1. make an "sbin" folder containing all the scripts for administrators, specifically, - all service administration scripts, i.e. start-*, stop-*, slaves.sh, *-daemons, *-daemon scripts - low-level or internally used utility scripts, i.e. compute-classpath, spark-config, spark-class, spark-executor 2. make a "bin" folder containing all the scripts for application developers/users, specifically, - user level app running scripts, i.e. pyspark, spark-shell, and we propose to add a script "spark" for users to run applications (very much like spark-class but may add some more control or convenient utilities) - scripts for status checking, e.g. spark and hadoop version checking, running applications checking, etc. We can make this a separate script or add functionality to "spark" script. 3. No wandering scripts outside the sbin and bin folders - Configurations/Options and overriding rule 1. Define a Configuration class which contains all the options available for Spark application. A Configuration instance can be de-/serialized from/to a json formatted file. 2. Each application (SparkContext) has one Configuration instance and it is initialized by the application which creates it (either read from file or passed from command line options or env SPARK_JAVA_OPTS). 3. When launching an Executor on a node, the Configuration is firstly initialized using the node-local configuration file as default. The Configuration passed from application driver context will override any options specified in default. Any comments are welcome. -- [email protected] *Shane Huang * *Intel Asia-Pacific R&D Ltd.* *Email: [email protected]*
