Hey Mridul this was patched and we cut a new release candidate. There were several different config options which had a.b and a.b.c... they should all work in the new RC.
On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mri...@gmail.com> wrote: > Chanced upon spill related config which exhibit same pattern ... > > - Mridul > > On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <r...@databricks.com> wrote: >> I also just went over the config options to see how pervasive this is. In >> addition to speculation, there is one more "conflict" of this kind: >> >> spark.locality.wait >> spark.locality.wait.node >> spark.locality.wait.process >> spark.locality.wait.rack >> >> >> spark.speculation >> spark.speculation.interval >> spark.speculation.multiplier >> spark.speculation.quantile >> >> >> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >> <matei.zaha...@gmail.com>wrote: >> >>> This is definitely an important issue to fix. Instead of renaming >>> properties, one solution would be to replace Typesafe Config with just >>> reading Java system properties, and disable config files for this release. >>> I kind of like that over renaming. >>> >>> Matei >>> >>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >>> wrote: >>> >>> > Hi, >>> > >>> > Speculation was an example, there are others in spark which are >>> > affected by this ... >>> > Some of them have been around for a while, so will break existing >>> code/scripts. >>> > >>> > Regards, >>> > Mridul >>> > >>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >>> wrote: >>> >> change spark.speculation to spark.speculation.switch? >>> >> >>> >> maybe we can restrict that all properties in Spark should be "three >>> levels" >>> >> >>> >> >>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com >>> >wrote: >>> >> >>> >>> Hi, >>> >>> >>> >>> Unless I am mistaken, the change to using typesafe ConfigFactory has >>> >>> broken some of the system properties we use in spark. >>> >>> >>> >>> For example: if we have both >>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>> >>> set, then the spark.speculation property is dropped. >>> >>> >>> >>> The rules of parseProperty actually document this clearly [1] >>> >>> >>> >>> >>> >>> I am not sure what the right fix here would be (other than replacing >>> >>> use of config that is). >>> >>> >>> >>> Any thoughts ? >>> >>> I would vote -1 for 0.9 to be released before this is fixed. >>> >>> >>> >>> >>> >>> Regards, >>> >>> Mridul >>> >>> >>> >>> >>> >>> [1] >>> >>> >>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>> >>> >>> >>>
