Yeah, this is exactly my reasoning as well. Matei
On Jan 18, 2014, at 12:14 PM, Mridul Muralidharan <[email protected]> wrote: > IMO we should shoot for more stable interfaces and not break them just > to workaround bugs - unless the benefit of breaking compatibility is > offset by the added functionality. > Since I was not around for a while, I am not sure how much config file > feature was requested ... > > Regards, > Mridul > > On Sun, Jan 19, 2014 at 1:31 AM, Mark Hamstra <[email protected]> wrote: >> Really? Disabling config files seems to me to be a bigger/more onerous >> change for users than spark.speculation=true|false => >> spark.speculation.enabled=true|false and spark.locality.wait => >> spark.locality.wait.default. >> >> >> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >> <[email protected]>wrote: >> >>> This is definitely an important issue to fix. Instead of renaming >>> properties, one solution would be to replace Typesafe Config with just >>> reading Java system properties, and disable config files for this release. >>> I kind of like that over renaming. >>> >>> Matei >>> >>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> Speculation was an example, there are others in spark which are >>>> affected by this ... >>>> Some of them have been around for a while, so will break existing >>> code/scripts. >>>> >>>> Regards, >>>> Mridul >>>> >>>> On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <[email protected]> >>> wrote: >>>>> change spark.speculation to spark.speculation.switch? >>>>> >>>>> maybe we can restrict that all properties in Spark should be "three >>> levels" >>>>> >>>>> >>>>> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <[email protected] >>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Unless I am mistaken, the change to using typesafe ConfigFactory has >>>>>> broken some of the system properties we use in spark. >>>>>> >>>>>> For example: if we have both >>>>>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>>>> set, then the spark.speculation property is dropped. >>>>>> >>>>>> The rules of parseProperty actually document this clearly [1] >>>>>> >>>>>> >>>>>> I am not sure what the right fix here would be (other than replacing >>>>>> use of config that is). >>>>>> >>>>>> Any thoughts ? >>>>>> I would vote -1 for 0.9 to be released before this is fixed. >>>>>> >>>>>> >>>>>> Regards, >>>>>> Mridul >>>>>> >>>>>> >>>>>> [1] >>>>>> >>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>>>> >>> >>>
