We can add config files in a later release. They were never officially released, and were only in master for about a month.
One other thing to note is that the config file feature is kind of limited anyway. Users will want to have a separate config file with each app, which they have to ship with its classpath, and there’s no great way of merging them in the current setup. I’m not actually sure it’s a feature we want to support, compared to say just a SparkConf.fromFile method that reads a Java Properties file. Matei On Jan 18, 2014, at 12:01 PM, Mark Hamstra <m...@clearstorydata.com> wrote: > Really? Disabling config files seems to me to be a bigger/more onerous > change for users than spark.speculation=true|false => > spark.speculation.enabled=true|false and spark.locality.wait => > spark.locality.wait.default. > > > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia > <matei.zaha...@gmail.com>wrote: > >> This is definitely an important issue to fix. Instead of renaming >> properties, one solution would be to replace Typesafe Config with just >> reading Java system properties, and disable config files for this release. >> I kind of like that over renaming. >> >> Matei >> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Speculation was an example, there are others in spark which are >>> affected by this ... >>> Some of them have been around for a while, so will break existing >> code/scripts. >>> >>> Regards, >>> Mridul >>> >>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >> wrote: >>>> change spark.speculation to spark.speculation.switch? >>>> >>>> maybe we can restrict that all properties in Spark should be "three >> levels" >>>> >>>> >>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com >>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Unless I am mistaken, the change to using typesafe ConfigFactory has >>>>> broken some of the system properties we use in spark. >>>>> >>>>> For example: if we have both >>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>>> set, then the spark.speculation property is dropped. >>>>> >>>>> The rules of parseProperty actually document this clearly [1] >>>>> >>>>> >>>>> I am not sure what the right fix here would be (other than replacing >>>>> use of config that is). >>>>> >>>>> Any thoughts ? >>>>> I would vote -1 for 0.9 to be released before this is fixed. >>>>> >>>>> >>>>> Regards, >>>>> Mridul >>>>> >>>>> >>>>> [1] >>>>> >> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>>> >> >>
