Yeah, it is in my TODO to test latest RC - unfortunately, the test setup is slightly busy running other things.
Regards, Mridul On Thu, Jan 23, 2014 at 10:16 PM, Henry Saputra <henry.sapu...@gmail.com> wrote: > Hi Mridul, > > Would you mind help to vet the latest 0.9 RC 4 candidate to make sure > the fixes are included since you raised the issues in prev RC? > > Thanks, > > Henry > > On Sun, Jan 19, 2014 at 10:33 AM, Mridul Muralidharan <mri...@gmail.com> > wrote: >> Oh great, just saw the PR from Matei ... for some odd reason, the dev >> mails are coming to be horribly delayed. >> >> >> Thanks, >> Mridul >> >> On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <pwend...@gmail.com> wrote: >>> Hey Mridul this was patched and we cut a new release candidate. There >>> were several different config options which had a.b and a.b.c... they >>> should all work in the new RC. >>> >>> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mri...@gmail.com> >>> wrote: >>>> Chanced upon spill related config which exhibit same pattern ... >>>> >>>> - Mridul >>>> >>>> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <r...@databricks.com> wrote: >>>>> I also just went over the config options to see how pervasive this is. In >>>>> addition to speculation, there is one more "conflict" of this kind: >>>>> >>>>> spark.locality.wait >>>>> spark.locality.wait.node >>>>> spark.locality.wait.process >>>>> spark.locality.wait.rack >>>>> >>>>> >>>>> spark.speculation >>>>> spark.speculation.interval >>>>> spark.speculation.multiplier >>>>> spark.speculation.quantile >>>>> >>>>> >>>>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >>>>> <matei.zaha...@gmail.com>wrote: >>>>> >>>>>> This is definitely an important issue to fix. Instead of renaming >>>>>> properties, one solution would be to replace Typesafe Config with just >>>>>> reading Java system properties, and disable config files for this >>>>>> release. >>>>>> I kind of like that over renaming. >>>>>> >>>>>> Matei >>>>>> >>>>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> > Hi, >>>>>> > >>>>>> > Speculation was an example, there are others in spark which are >>>>>> > affected by this ... >>>>>> > Some of them have been around for a while, so will break existing >>>>>> code/scripts. >>>>>> > >>>>>> > Regards, >>>>>> > Mridul >>>>>> > >>>>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >>>>>> wrote: >>>>>> >> change spark.speculation to spark.speculation.switch? >>>>>> >> >>>>>> >> maybe we can restrict that all properties in Spark should be "three >>>>>> levels" >>>>>> >> >>>>>> >> >>>>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com >>>>>> >wrote: >>>>>> >> >>>>>> >>> Hi, >>>>>> >>> >>>>>> >>> Unless I am mistaken, the change to using typesafe ConfigFactory has >>>>>> >>> broken some of the system properties we use in spark. >>>>>> >>> >>>>>> >>> For example: if we have both >>>>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>>>> >>> set, then the spark.speculation property is dropped. >>>>>> >>> >>>>>> >>> The rules of parseProperty actually document this clearly [1] >>>>>> >>> >>>>>> >>> >>>>>> >>> I am not sure what the right fix here would be (other than replacing >>>>>> >>> use of config that is). >>>>>> >>> >>>>>> >>> Any thoughts ? >>>>>> >>> I would vote -1 for 0.9 to be released before this is fixed. >>>>>> >>> >>>>>> >>> >>>>>> >>> Regards, >>>>>> >>> Mridul >>>>>> >>> >>>>>> >>> >>>>>> >>> [1] >>>>>> >>> >>>>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>>>> >>> >>>>>> >>>>>>
