I think it's good to have Bigtop to package Spark. But in this track we're just targeting enhancing the usability of Spark itself without Bigtop. After all, few of our customers used Bigtop.
On Wed, Sep 25, 2013 at 1:20 PM, Shane Huang <[email protected]>wrote: > I think it's good to have Bigtop to package Spark. But I > > > On Wed, Sep 25, 2013 at 1:16 PM, Konstantin Boudnik <[email protected]>wrote: > >> Late to the game, but... Bigtop is packaging Spark now as a part of the >> standard distribution - our release 0.7.0 is around the corner. And we do >> it >> in the same way that has been done for Hadoop. Perhaps it worth looking >> into... >> >> Lemme know if you have any questions, >> Cos >> >> On Sun, Sep 22, 2013 at 12:07PM, Shane Huang wrote: >> > And I created a new issue SPARK-915 to track the re-org of scripts as >> > SPARK-544 only talks about Config. >> > https://spark-project.atlassian.net/browse/SPARK-915 >> > >> > >> > On Wed, Sep 18, 2013 at 1:42 AM, Matei Zaharia <[email protected] >> >wrote: >> > >> > > Hi Shane, >> > > >> > > I agree with all these points. Improving the configuration system is >> one >> > > of the main things I'd like to have in the next release. >> > > >> > > > 1) Usually the application developers/users and platform >> administrators >> > > > belongs to two teams. So it's better to separate the scripts used by >> > > > administrators and application users, e.g. put them in sbin and bin >> > > folders >> > > > respectively >> > > >> > > Yup, right now we don't have any attempt to install on standard system >> > > paths. >> > > >> > > > 3) If there are multiple ways to specify an option, an overriding >> rule >> > > > should be present and should not be error-prone. >> > > >> > > Yes, I think this should always be Configuration class in code > >> system >> > > properties > env vars. Over time we will deprecate the env vars and >> maybe >> > > even system properties. >> > > >> > > > 4) Currently the options are set and get using System property. >> It's hard >> > > > to manage and inconvenient for users. It's good to gather the >> options >> > > into >> > > > one file using format like xml or json. >> > > >> > > I think this is the main thing to do first -- pick one configuration >> class >> > > and change the code to use this. >> > > >> > > > Our rough proposal: >> > > > >> > > > - Scripts >> > > > >> > > > 1. make an "sbin" folder containing all the scripts for >> administrators, >> > > > specifically, >> > > > - all service administration scripts, i.e. start-*, stop-*, >> > > > slaves.sh, *-daemons, *-daemon scripts >> > > > - low-level or internally used utility scripts, i.e. >> > > > compute-classpath, spark-config, spark-class, spark-executor >> > > > 2. make a "bin" folder containing all the scripts for application >> > > > developers/users, specifically, >> > > > - user level app running scripts, i.e. pyspark, spark-shell, >> and we >> > > > propose to add a script "spark" for users to run applications >> (very >> > > much >> > > > like spark-class but may add some more control or convenient >> > > utilities) >> > > > - scripts for status checking, e.g. spark and hadoop version >> > > > checking, running applications checking, etc. We can make this >> a >> > > separate >> > > > script or add functionality to "spark" script. >> > > > 3. No wandering scripts outside the sbin and bin folders >> > > >> > > Makes sense. >> > > >> > > > - Configurations/Options and overriding rule >> > > > >> > > > 1. Define a Configuration class which contains all the options >> > > available >> > > > for Spark application. A Configuration instance can be >> de-/serialized >> > > > from/to a json formatted file. >> > > > 2. Each application (SparkContext) has one Configuration instance >> and >> > > it >> > > > is initialized by the application which creates it (either read >> from >> > > file >> > > > or passed from command line options or env SPARK_JAVA_OPTS). >> > > > 3. When launching an Executor on a node, the Configuration is >> firstly >> > > > initialized using the node-local configuration file as default. >> The >> > > > Configuration passed from application driver context will >> override any >> > > > options specified in default. >> > > >> > > This sounds great to me! The one thing I'll add is that we might want >> to >> > > prevent applications from overriding certain settings on each node, >> such as >> > > work directories. The best way is to probably just ignore the app's >> version >> > > of those settings in the Executor. >> > > >> > > If you guys would like, feel free to write up this design on >> SPARK-544 and >> > > start working on it. I think it looks good. >> > > >> > > Matei >> > >> > >> > >> > >> > -- >> > *Shane Huang * >> > *Intel Asia-Pacific R&D Ltd.* >> > *Email: [email protected]* >> > > > > -- > *Shane Huang * > *Intel Asia-Pacific R&D Ltd.* > *Email: [email protected]* > > -- *Shane Huang * *Intel Asia-Pacific R&D Ltd.* *Email: [email protected]*
