Shane Huang wrote: > we found the current organization of the scripts and configuration a > bit confusing and inconvenient
ditto > - Scripts I wonder why the work of these scripts wasn't mostly done in Scala. Seems roundabout to use Bash (or Python, in spark-perf) to calculate shell environment variables that are then read back into Scala code. > 1. Define a Configuration class which contains all the options > available for Spark application. A Configuration instance can be > de-/serialized from/to a json formatted file. > 2. Each application (SparkContext) has one Configuration instance and > it is initialized by the application which creates it (either read > from file or passed from command line options or env SPARK_JAVA_OPTS). Reminiscent of what Hibernate's been doing for the past decade. Would be nice if the Configuration was also exposed through an MBean or such so that one can check it's values with certainty.
