Yeah, I can get on board with that -- gives us another chance to re-think/re-work config files to address the limitations Matei mentioned before the interface is fixed for 1.0.
On Sat, Jan 18, 2014 at 12:27 PM, Patrick Wendell <pwend...@gmail.com>wrote: > Hey Mark - ya if we did add this I think it would be in the next major > release. > > On Sat, Jan 18, 2014 at 12:17 PM, Mark Hamstra <m...@clearstorydata.com> > wrote: > > That later release should be at least 0.10.0, then, since use of config > > files won't be backward compatible with 0.9.0. > > > > > > On Sat, Jan 18, 2014 at 12:11 PM, Matei Zaharia <matei.zaha...@gmail.com > >wrote: > > > >> We can add config files in a later release. They were never officially > >> released, and were only in master for about a month. > >> > >> One other thing to note is that the config file feature is kind of > limited > >> anyway. Users will want to have a separate config file with each app, > which > >> they have to ship with its classpath, and there’s no great way of > merging > >> them in the current setup. I’m not actually sure it’s a feature we want > to > >> support, compared to say just a SparkConf.fromFile method that reads a > Java > >> Properties file. > >> > >> Matei > >> > >> On Jan 18, 2014, at 12:01 PM, Mark Hamstra <m...@clearstorydata.com> > >> wrote: > >> > >> > Really? Disabling config files seems to me to be a bigger/more > onerous > >> > change for users than spark.speculation=true|false => > >> > spark.speculation.enabled=true|false and spark.locality.wait => > >> > spark.locality.wait.default. > >> > > >> > > >> > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia < > matei.zaha...@gmail.com > >> >wrote: > >> > > >> >> This is definitely an important issue to fix. Instead of renaming > >> >> properties, one solution would be to replace Typesafe Config with > just > >> >> reading Java system properties, and disable config files for this > >> release. > >> >> I kind of like that over renaming. > >> >> > >> >> Matei > >> >> > >> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> > >> >> wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> Speculation was an example, there are others in spark which are > >> >>> affected by this ... > >> >>> Some of them have been around for a while, so will break existing > >> >> code/scripts. > >> >>> > >> >>> Regards, > >> >>> Mridul > >> >>> > >> >>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> > >> >> wrote: > >> >>>> change spark.speculation to spark.speculation.switch? > >> >>>> > >> >>>> maybe we can restrict that all properties in Spark should be "three > >> >> levels" > >> >>>> > >> >>>> > >> >>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan < > >> mri...@gmail.com > >> >>> wrote: > >> >>>> > >> >>>>> Hi, > >> >>>>> > >> >>>>> Unless I am mistaken, the change to using typesafe ConfigFactory > has > >> >>>>> broken some of the system properties we use in spark. > >> >>>>> > >> >>>>> For example: if we have both > >> >>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 > >> >>>>> set, then the spark.speculation property is dropped. > >> >>>>> > >> >>>>> The rules of parseProperty actually document this clearly [1] > >> >>>>> > >> >>>>> > >> >>>>> I am not sure what the right fix here would be (other than > replacing > >> >>>>> use of config that is). > >> >>>>> > >> >>>>> Any thoughts ? > >> >>>>> I would vote -1 for 0.9 to be released before this is fixed. > >> >>>>> > >> >>>>> > >> >>>>> Regards, > >> >>>>> Mridul > >> >>>>> > >> >>>>> > >> >>>>> [1] > >> >>>>> > >> >> > >> > http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 > >> >>>>> > >> >> > >> >> > >> > >> >
