Cool thanks, appreciate it =) - Henry
On Thu, Jan 23, 2014 at 10:06 AM, Mridul Muralidharan <mri...@gmail.com> wrote: > Yeah, it is in my TODO to test latest RC - unfortunately, the test > setup is slightly busy running other things. > > Regards, > Mridul > > On Thu, Jan 23, 2014 at 10:16 PM, Henry Saputra <henry.sapu...@gmail.com> > wrote: >> Hi Mridul, >> >> Would you mind help to vet the latest 0.9 RC 4 candidate to make sure >> the fixes are included since you raised the issues in prev RC? >> >> Thanks, >> >> Henry >> >> On Sun, Jan 19, 2014 at 10:33 AM, Mridul Muralidharan <mri...@gmail.com> >> wrote: >>> Oh great, just saw the PR from Matei ... for some odd reason, the dev >>> mails are coming to be horribly delayed. >>> >>> >>> Thanks, >>> Mridul >>> >>> On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <pwend...@gmail.com> >>> wrote: >>>> Hey Mridul this was patched and we cut a new release candidate. There >>>> were several different config options which had a.b and a.b.c... they >>>> should all work in the new RC. >>>> >>>> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <mri...@gmail.com> >>>> wrote: >>>>> Chanced upon spill related config which exhibit same pattern ... >>>>> >>>>> - Mridul >>>>> >>>>> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <r...@databricks.com> wrote: >>>>>> I also just went over the config options to see how pervasive this is. In >>>>>> addition to speculation, there is one more "conflict" of this kind: >>>>>> >>>>>> spark.locality.wait >>>>>> spark.locality.wait.node >>>>>> spark.locality.wait.process >>>>>> spark.locality.wait.rack >>>>>> >>>>>> >>>>>> spark.speculation >>>>>> spark.speculation.interval >>>>>> spark.speculation.multiplier >>>>>> spark.speculation.quantile >>>>>> >>>>>> >>>>>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >>>>>> <matei.zaha...@gmail.com>wrote: >>>>>> >>>>>>> This is definitely an important issue to fix. Instead of renaming >>>>>>> properties, one solution would be to replace Typesafe Config with just >>>>>>> reading Java system properties, and disable config files for this >>>>>>> release. >>>>>>> I kind of like that over renaming. >>>>>>> >>>>>>> Matei >>>>>>> >>>>>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> > Hi, >>>>>>> > >>>>>>> > Speculation was an example, there are others in spark which are >>>>>>> > affected by this ... >>>>>>> > Some of them have been around for a while, so will break existing >>>>>>> code/scripts. >>>>>>> > >>>>>>> > Regards, >>>>>>> > Mridul >>>>>>> > >>>>>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> >>>>>>> wrote: >>>>>>> >> change spark.speculation to spark.speculation.switch? >>>>>>> >> >>>>>>> >> maybe we can restrict that all properties in Spark should be "three >>>>>>> levels" >>>>>>> >> >>>>>>> >> >>>>>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan >>>>>>> >> <mri...@gmail.com >>>>>>> >wrote: >>>>>>> >> >>>>>>> >>> Hi, >>>>>>> >>> >>>>>>> >>> Unless I am mistaken, the change to using typesafe ConfigFactory >>>>>>> >>> has >>>>>>> >>> broken some of the system properties we use in spark. >>>>>>> >>> >>>>>>> >>> For example: if we have both >>>>>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>>>>> >>> set, then the spark.speculation property is dropped. >>>>>>> >>> >>>>>>> >>> The rules of parseProperty actually document this clearly [1] >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> I am not sure what the right fix here would be (other than replacing >>>>>>> >>> use of config that is). >>>>>>> >>> >>>>>>> >>> Any thoughts ? >>>>>>> >>> I would vote -1 for 0.9 to be released before this is fixed. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> Regards, >>>>>>> >>> Mridul >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> [1] >>>>>>> >>> >>>>>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>>>>> >>> >>>>>>> >>>>>>>
