Hey Mark - ya if we did add this I think it would be in the next major release.
On Sat, Jan 18, 2014 at 12:17 PM, Mark Hamstra <[email protected]> wrote: > That later release should be at least 0.10.0, then, since use of config > files won't be backward compatible with 0.9.0. > > > On Sat, Jan 18, 2014 at 12:11 PM, Matei Zaharia > <[email protected]>wrote: > >> We can add config files in a later release. They were never officially >> released, and were only in master for about a month. >> >> One other thing to note is that the config file feature is kind of limited >> anyway. Users will want to have a separate config file with each app, which >> they have to ship with its classpath, and there’s no great way of merging >> them in the current setup. I’m not actually sure it’s a feature we want to >> support, compared to say just a SparkConf.fromFile method that reads a Java >> Properties file. >> >> Matei >> >> On Jan 18, 2014, at 12:01 PM, Mark Hamstra <[email protected]> >> wrote: >> >> > Really? Disabling config files seems to me to be a bigger/more onerous >> > change for users than spark.speculation=true|false => >> > spark.speculation.enabled=true|false and spark.locality.wait => >> > spark.locality.wait.default. >> > >> > >> > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <[email protected] >> >wrote: >> > >> >> This is definitely an important issue to fix. Instead of renaming >> >> properties, one solution would be to replace Typesafe Config with just >> >> reading Java system properties, and disable config files for this >> release. >> >> I kind of like that over renaming. >> >> >> >> Matei >> >> >> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <[email protected]> >> >> wrote: >> >> >> >>> Hi, >> >>> >> >>> Speculation was an example, there are others in spark which are >> >>> affected by this ... >> >>> Some of them have been around for a while, so will break existing >> >> code/scripts. >> >>> >> >>> Regards, >> >>> Mridul >> >>> >> >>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <[email protected]> >> >> wrote: >> >>>> change spark.speculation to spark.speculation.switch? >> >>>> >> >>>> maybe we can restrict that all properties in Spark should be "three >> >> levels" >> >>>> >> >>>> >> >>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan < >> [email protected] >> >>> wrote: >> >>>> >> >>>>> Hi, >> >>>>> >> >>>>> Unless I am mistaken, the change to using typesafe ConfigFactory has >> >>>>> broken some of the system properties we use in spark. >> >>>>> >> >>>>> For example: if we have both >> >>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >> >>>>> set, then the spark.speculation property is dropped. >> >>>>> >> >>>>> The rules of parseProperty actually document this clearly [1] >> >>>>> >> >>>>> >> >>>>> I am not sure what the right fix here would be (other than replacing >> >>>>> use of config that is). >> >>>>> >> >>>>> Any thoughts ? >> >>>>> I would vote -1 for 0.9 to be released before this is fixed. >> >>>>> >> >>>>> >> >>>>> Regards, >> >>>>> Mridul >> >>>>> >> >>>>> >> >>>>> [1] >> >>>>> >> >> >> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >> >>>>> >> >> >> >> >> >>
