That later release should be at least 0.10.0, then, since use of config files won't be backward compatible with 0.9.0.
On Sat, Jan 18, 2014 at 12:11 PM, Matei Zaharia <[email protected]>wrote: > We can add config files in a later release. They were never officially > released, and were only in master for about a month. > > One other thing to note is that the config file feature is kind of limited > anyway. Users will want to have a separate config file with each app, which > they have to ship with its classpath, and there’s no great way of merging > them in the current setup. I’m not actually sure it’s a feature we want to > support, compared to say just a SparkConf.fromFile method that reads a Java > Properties file. > > Matei > > On Jan 18, 2014, at 12:01 PM, Mark Hamstra <[email protected]> > wrote: > > > Really? Disabling config files seems to me to be a bigger/more onerous > > change for users than spark.speculation=true|false => > > spark.speculation.enabled=true|false and spark.locality.wait => > > spark.locality.wait.default. > > > > > > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <[email protected] > >wrote: > > > >> This is definitely an important issue to fix. Instead of renaming > >> properties, one solution would be to replace Typesafe Config with just > >> reading Java system properties, and disable config files for this > release. > >> I kind of like that over renaming. > >> > >> Matei > >> > >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <[email protected]> > >> wrote: > >> > >>> Hi, > >>> > >>> Speculation was an example, there are others in spark which are > >>> affected by this ... > >>> Some of them have been around for a while, so will break existing > >> code/scripts. > >>> > >>> Regards, > >>> Mridul > >>> > >>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <[email protected]> > >> wrote: > >>>> change spark.speculation to spark.speculation.switch? > >>>> > >>>> maybe we can restrict that all properties in Spark should be "three > >> levels" > >>>> > >>>> > >>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan < > [email protected] > >>> wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> Unless I am mistaken, the change to using typesafe ConfigFactory has > >>>>> broken some of the system properties we use in spark. > >>>>> > >>>>> For example: if we have both > >>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 > >>>>> set, then the spark.speculation property is dropped. > >>>>> > >>>>> The rules of parseProperty actually document this clearly [1] > >>>>> > >>>>> > >>>>> I am not sure what the right fix here would be (other than replacing > >>>>> use of config that is). > >>>>> > >>>>> Any thoughts ? > >>>>> I would vote -1 for 0.9 to be released before this is fixed. > >>>>> > >>>>> > >>>>> Regards, > >>>>> Mridul > >>>>> > >>>>> > >>>>> [1] > >>>>> > >> > http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 > >>>>> > >> > >> > >
