Hi Shane, Junluan, Definitely, let's cooperate. Should we chat offline?
-Evan On Thu, Sep 26, 2013 at 1:24 AM, Xia, Junluan <[email protected]> wrote: > Hi Chan > > Shane and I happened to try to contribute configure feature to spark, > could we cooperate to implement it? > > -----Original Message----- > From: Evan Chan [mailto:[email protected]] > Sent: Thursday, September 26, 2013 2:43 PM > To: [email protected] > Subject: Re: Propose to Re-organize the scripts and configurations > > Shane, and others, > > Let's work together on the configuration thing. I had proposed in a > separate thread to use Typesafe Config to hold all configuration > (essentially a configuration class, but which can read from both JSON files > as well as -D java command line args). > > Typesafe Config works much much better than a simple config class, and > also better than Hadoop configs. It also has advantages over JSON (more > readable, comments). It would also be the easiest to transition from the > current scheme, since the current java system properties can be seamlessly > integrated. > > I would be happy to contribute this back soon because it is also a big > pain point for us. I also have extensive experience with both Typesafe > Config and other config systems. > > I would definitely start with SparkContext and work our way out from there. > In fact I can submit a patch for everyone to test out fairly quickly > just for SparkContext. > > -Evan > > > > On Tue, Sep 24, 2013 at 10:26 PM, Shane Huang <[email protected] > >wrote: > > > I think it's good to have Bigtop to package Spark. But in this track > > we're just targeting enhancing the usability of Spark itself without > Bigtop. > > After all, few of our customers used Bigtop. > > > > > > On Wed, Sep 25, 2013 at 1:20 PM, Shane Huang <[email protected] > > >wrote: > > > > > I think it's good to have Bigtop to package Spark. But I > > > > > > > > > On Wed, Sep 25, 2013 at 1:16 PM, Konstantin Boudnik <[email protected] > > >wrote: > > > > > >> Late to the game, but... Bigtop is packaging Spark now as a part of > > >> the standard distribution - our release 0.7.0 is around the corner. > > >> And we > > do > > >> it > > >> in the same way that has been done for Hadoop. Perhaps it worth > > >> looking into... > > >> > > >> Lemme know if you have any questions, > > >> Cos > > >> > > >> On Sun, Sep 22, 2013 at 12:07PM, Shane Huang wrote: > > >> > And I created a new issue SPARK-915 to track the re-org of > > >> > scripts as > > >> > SPARK-544 only talks about Config. > > >> > https://spark-project.atlassian.net/browse/SPARK-915 > > >> > > > >> > > > >> > On Wed, Sep 18, 2013 at 1:42 AM, Matei Zaharia < > > [email protected] > > >> >wrote: > > >> > > > >> > > Hi Shane, > > >> > > > > >> > > I agree with all these points. Improving the configuration > > >> > > system is > > >> one > > >> > > of the main things I'd like to have in the next release. > > >> > > > > >> > > > 1) Usually the application developers/users and platform > > >> administrators > > >> > > > belongs to two teams. So it's better to separate the scripts > > >> > > > used > > by > > >> > > > administrators and application users, e.g. put them in sbin > > >> > > > and > > bin > > >> > > folders > > >> > > > respectively > > >> > > > > >> > > Yup, right now we don't have any attempt to install on standard > > system > > >> > > paths. > > >> > > > > >> > > > 3) If there are multiple ways to specify an option, an > > >> > > > overriding > > >> rule > > >> > > > should be present and should not be error-prone. > > >> > > > > >> > > Yes, I think this should always be Configuration class in code > > >> > > > > > >> system > > >> > > properties > env vars. Over time we will deprecate the env vars > > >> > > and > > >> maybe > > >> > > even system properties. > > >> > > > > >> > > > 4) Currently the options are set and get using System property. > > >> It's hard > > >> > > > to manage and inconvenient for users. It's good to gather the > > >> options > > >> > > into > > >> > > > one file using format like xml or json. > > >> > > > > >> > > I think this is the main thing to do first -- pick one > > >> > > configuration > > >> class > > >> > > and change the code to use this. > > >> > > > > >> > > > Our rough proposal: > > >> > > > > > >> > > > - Scripts > > >> > > > > > >> > > > 1. make an "sbin" folder containing all the scripts for > > >> administrators, > > >> > > > specifically, > > >> > > > - all service administration scripts, i.e. start-*, stop-*, > > >> > > > slaves.sh, *-daemons, *-daemon scripts > > >> > > > - low-level or internally used utility scripts, i.e. > > >> > > > compute-classpath, spark-config, spark-class, > spark-executor > > >> > > > 2. make a "bin" folder containing all the scripts for > > application > > >> > > > developers/users, specifically, > > >> > > > - user level app running scripts, i.e. pyspark, > > >> > > > spark-shell, > > >> and we > > >> > > > propose to add a script "spark" for users to run > > >> > > > applications > > >> (very > > >> > > much > > >> > > > like spark-class but may add some more control or > > >> > > > convenient > > >> > > utilities) > > >> > > > - scripts for status checking, e.g. spark and hadoop > version > > >> > > > checking, running applications checking, etc. We can > > >> > > > make > > this > > >> a > > >> > > separate > > >> > > > script or add functionality to "spark" script. > > >> > > > 3. No wandering scripts outside the sbin and bin folders > > >> > > > > >> > > Makes sense. > > >> > > > > >> > > > - Configurations/Options and overriding rule > > >> > > > > > >> > > > 1. Define a Configuration class which contains all the > > >> > > > options > > >> > > available > > >> > > > for Spark application. A Configuration instance can be > > >> de-/serialized > > >> > > > from/to a json formatted file. > > >> > > > 2. Each application (SparkContext) has one Configuration > > instance > > >> and > > >> > > it > > >> > > > is initialized by the application which creates it (either > > >> > > > read > > >> from > > >> > > file > > >> > > > or passed from command line options or env SPARK_JAVA_OPTS). > > >> > > > 3. When launching an Executor on a node, the Configuration > > >> > > > is > > >> firstly > > >> > > > initialized using the node-local configuration file as > default. > > >> The > > >> > > > Configuration passed from application driver context will > > >> override any > > >> > > > options specified in default. > > >> > > > > >> > > This sounds great to me! The one thing I'll add is that we > > >> > > might > > want > > >> to > > >> > > prevent applications from overriding certain settings on each > > >> > > node, > > >> such as > > >> > > work directories. The best way is to probably just ignore the > > >> > > app's > > >> version > > >> > > of those settings in the Executor. > > >> > > > > >> > > If you guys would like, feel free to write up this design on > > >> SPARK-544 and > > >> > > start working on it. I think it looks good. > > >> > > > > >> > > Matei > > >> > > > >> > > > >> > > > >> > > > >> > -- > > >> > *Shane Huang * > > >> > *Intel Asia-Pacific R&D Ltd.* > > >> > *Email: [email protected]* > > >> > > > > > > > > > > > > -- > > > *Shane Huang * > > > *Intel Asia-Pacific R&D Ltd.* > > > *Email: [email protected]* > > > > > > > > > > > > -- > > *Shane Huang * > > *Intel Asia-Pacific R&D Ltd.* > > *Email: [email protected]* > > > > > > -- > -- > Evan Chan > Staff Engineer > [email protected] | > > <http://www.ooyala.com/> > <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala>< > http://www.twitter.com/ooyala> > -- -- Evan Chan Staff Engineer [email protected] | <http://www.ooyala.com/> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
