IMO we should shoot for more stable interfaces and not break them just to workaround bugs - unless the benefit of breaking compatibility is offset by the added functionality. Since I was not around for a while, I am not sure how much config file feature was requested ...
Regards, Mridul On Sun, Jan 19, 2014 at 1:31 AM, Mark Hamstra <[email protected]> wrote: > Really? Disabling config files seems to me to be a bigger/more onerous > change for users than spark.speculation=true|false => > spark.speculation.enabled=true|false and spark.locality.wait => > spark.locality.wait.default. > > > On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia > <[email protected]>wrote: > >> This is definitely an important issue to fix. Instead of renaming >> properties, one solution would be to replace Typesafe Config with just >> reading Java system properties, and disable config files for this release. >> I kind of like that over renaming. >> >> Matei >> >> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <[email protected]> >> wrote: >> >> > Hi, >> > >> > Speculation was an example, there are others in spark which are >> > affected by this ... >> > Some of them have been around for a while, so will break existing >> code/scripts. >> > >> > Regards, >> > Mridul >> > >> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <[email protected]> >> wrote: >> >> change spark.speculation to spark.speculation.switch? >> >> >> >> maybe we can restrict that all properties in Spark should be "three >> levels" >> >> >> >> >> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <[email protected] >> >wrote: >> >> >> >>> Hi, >> >>> >> >>> Unless I am mistaken, the change to using typesafe ConfigFactory has >> >>> broken some of the system properties we use in spark. >> >>> >> >>> For example: if we have both >> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >> >>> set, then the spark.speculation property is dropped. >> >>> >> >>> The rules of parseProperty actually document this clearly [1] >> >>> >> >>> >> >>> I am not sure what the right fix here would be (other than replacing >> >>> use of config that is). >> >>> >> >>> Any thoughts ? >> >>> I would vote -1 for 0.9 to be released before this is fixed. >> >>> >> >>> >> >>> Regards, >> >>> Mridul >> >>> >> >>> >> >>> [1] >> >>> >> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >> >>> >> >>
