There was another discussion on the old dev list about this: https://groups.google.com/forum/#!msg/spark-developers/GL2_DwAeh5s/9rwQ3iDa2t4J
I tend to agree with having configuration sitting in JSON (or properties files) and using the Typesafe Config library which can parse both. Something I've used in my apps is along these lines: https://gist.github.com/MLnick/6578146 It's then easy to have default config overridden with CLI for example: val conf = cliConf.withFallback(defaultConf) I'd be happy to be involved in working on this if there is a consensus about best approach N On Mon, Sep 16, 2013 at 9:29 AM, Mike <[email protected]> wrote: > Shane Huang wrote: > > we found the current organization of the scripts and configuration a > > bit confusing and inconvenient > > ditto > > > - Scripts > > I wonder why the work of these scripts wasn't mostly done in Scala. > Seems roundabout to use Bash (or Python, in spark-perf) to calculate > shell environment variables that are then read back into Scala code. > > > 1. Define a Configuration class which contains all the options > > available for Spark application. A Configuration instance can be > > de-/serialized from/to a json formatted file. > > 2. Each application (SparkContext) has one Configuration instance and > > it is initialized by the application which creates it (either read > > from file or passed from command line options or env SPARK_JAVA_OPTS). > > Reminiscent of what Hibernate's been doing for the past decade. Would > be nice if the Configuration was also exposed through an MBean or such > so that one can check it's values with certainty. >
