Hi Mridul, Would you mind help to vet the latest 0.9 RC 4 candidate to make sure the fixes are included since you raised the issues in prev RC?
Thanks, Henry On Sun, Jan 19, 2014 at 10:33 AM, Mridul Muralidharan <mri...@gmail.com> wrote: > Oh great, just saw the PR from Matei ... for some odd reason, the dev > mails are coming to be horribly delayed. > > > Thanks, > Mridul > > On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <pwend...@gmail.com> wrote: >> Hey Mridul this was patched and we cut a new release candidate. There >> were several different config options which had a.b and a.b.c... they >> should all work in the new RC. >> >> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mri...@gmail.com> >> wrote: >>> Chanced upon spill related config which exhibit same pattern ... >>> >>> - Mridul >>> >>> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <r...@databricks.com> wrote: >>>> I also just went over the config options to see how pervasive this is. In >>>> addition to speculation, there is one more "conflict" of this kind: >>>> >>>> spark.locality.wait >>>> spark.locality.wait.node >>>> spark.locality.wait.process >>>> spark.locality.wait.rack >>>> >>>> >>>> spark.speculation >>>> spark.speculation.interval >>>> spark.speculation.multiplier >>>> spark.speculation.quantile >>>> >>>> >>>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >>>> <matei.zaha...@gmail.com>wrote: >>>> >>>>> This is definitely an important issue to fix. Instead of renaming >>>>> properties, one solution would be to replace Typesafe Config with just >>>>> reading Java system properties, and disable config files for this release. >>>>> I kind of like that over renaming. >>>>> >>>>> Matei >>>>> >>>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >>>>> wrote: >>>>> >>>>> > Hi, >>>>> > >>>>> > Speculation was an example, there are others in spark which are >>>>> > affected by this ... >>>>> > Some of them have been around for a while, so will break existing >>>>> code/scripts. >>>>> > >>>>> > Regards, >>>>> > Mridul >>>>> > >>>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >>>>> wrote: >>>>> >> change spark.speculation to spark.speculation.switch? >>>>> >> >>>>> >> maybe we can restrict that all properties in Spark should be "three >>>>> levels" >>>>> >> >>>>> >> >>>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com >>>>> >wrote: >>>>> >> >>>>> >>> Hi, >>>>> >>> >>>>> >>> Unless I am mistaken, the change to using typesafe ConfigFactory has >>>>> >>> broken some of the system properties we use in spark. >>>>> >>> >>>>> >>> For example: if we have both >>>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>>> >>> set, then the spark.speculation property is dropped. >>>>> >>> >>>>> >>> The rules of parseProperty actually document this clearly [1] >>>>> >>> >>>>> >>> >>>>> >>> I am not sure what the right fix here would be (other than replacing >>>>> >>> use of config that is). >>>>> >>> >>>>> >>> Any thoughts ? >>>>> >>> I would vote -1 for 0.9 to be released before this is fixed. >>>>> >>> >>>>> >>> >>>>> >>> Regards, >>>>> >>> Mridul >>>>> >>> >>>>> >>> >>>>> >>> [1] >>>>> >>> >>>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>>> >>> >>>>> >>>>>
