Hah! Stupid English language -- by "fixed" I mean established/stabilized, not repaired.
On Sat, Jan 18, 2014 at 12:42 PM, Mark Hamstra <[email protected]>wrote: > Yeah, I can get on board with that -- gives us another chance to > re-think/re-work config files to address the limitations Matei mentioned > before the interface is fixed for 1.0. > > > On Sat, Jan 18, 2014 at 12:27 PM, Patrick Wendell <[email protected]>wrote: > >> Hey Mark - ya if we did add this I think it would be in the next major >> release. >> >> On Sat, Jan 18, 2014 at 12:17 PM, Mark Hamstra <[email protected]> >> wrote: >> > That later release should be at least 0.10.0, then, since use of config >> > files won't be backward compatible with 0.9.0. >> > >> > >> > On Sat, Jan 18, 2014 at 12:11 PM, Matei Zaharia < >> [email protected]>wrote: >> > >> >> We can add config files in a later release. They were never officially >> >> released, and were only in master for about a month. >> >> >> >> One other thing to note is that the config file feature is kind of >> limited >> >> anyway. Users will want to have a separate config file with each app, >> which >> >> they have to ship with its classpath, and there’s no great way of >> merging >> >> them in the current setup. I’m not actually sure it’s a feature we >> want to >> >> support, compared to say just a SparkConf.fromFile method that reads a >> Java >> >> Properties file. >> >> >> >> Matei >> >> >> >> On Jan 18, 2014, at 12:01 PM, Mark Hamstra <[email protected]> >> >> wrote: >> >> >> >> > Really? Disabling config files seems to me to be a bigger/more >> onerous >> >> > change for users than spark.speculation=true|false => >> >> > spark.speculation.enabled=true|false and spark.locality.wait => >> >> > spark.locality.wait.default. >> >> > >> >> > >> >> > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia < >> [email protected] >> >> >wrote: >> >> > >> >> >> This is definitely an important issue to fix. Instead of renaming >> >> >> properties, one solution would be to replace Typesafe Config with >> just >> >> >> reading Java system properties, and disable config files for this >> >> release. >> >> >> I kind of like that over renaming. >> >> >> >> >> >> Matei >> >> >> >> >> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <[email protected] >> > >> >> >> wrote: >> >> >> >> >> >>> Hi, >> >> >>> >> >> >>> Speculation was an example, there are others in spark which are >> >> >>> affected by this ... >> >> >>> Some of them have been around for a while, so will break existing >> >> >> code/scripts. >> >> >>> >> >> >>> Regards, >> >> >>> Mridul >> >> >>> >> >> >>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <[email protected]> >> >> >> wrote: >> >> >>>> change spark.speculation to spark.speculation.switch? >> >> >>>> >> >> >>>> maybe we can restrict that all properties in Spark should be >> "three >> >> >> levels" >> >> >>>> >> >> >>>> >> >> >>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan < >> >> [email protected] >> >> >>> wrote: >> >> >>>> >> >> >>>>> Hi, >> >> >>>>> >> >> >>>>> Unless I am mistaken, the change to using typesafe ConfigFactory >> has >> >> >>>>> broken some of the system properties we use in spark. >> >> >>>>> >> >> >>>>> For example: if we have both >> >> >>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >> >> >>>>> set, then the spark.speculation property is dropped. >> >> >>>>> >> >> >>>>> The rules of parseProperty actually document this clearly [1] >> >> >>>>> >> >> >>>>> >> >> >>>>> I am not sure what the right fix here would be (other than >> replacing >> >> >>>>> use of config that is). >> >> >>>>> >> >> >>>>> Any thoughts ? >> >> >>>>> I would vote -1 for 0.9 to be released before this is fixed. >> >> >>>>> >> >> >>>>> >> >> >>>>> Regards, >> >> >>>>> Mridul >> >> >>>>> >> >> >>>>> >> >> >>>>> [1] >> >> >>>>> >> >> >> >> >> >> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >> >> >>>>> >> >> >> >> >> >> >> >> >> >> >> > >
