Cool thanks, appreciate it =) - Henry
On Thu, Jan 23, 2014 at 10:06 AM, Mridul Muralidharan <[email protected]> wrote: > Yeah, it is in my TODO to test latest RC - unfortunately, the test > setup is slightly busy running other things. > > Regards, > Mridul > > On Thu, Jan 23, 2014 at 10:16 PM, Henry Saputra <[email protected]> > wrote: >> Hi Mridul, >> >> Would you mind help to vet the latest 0.9 RC 4 candidate to make sure >> the fixes are included since you raised the issues in prev RC? >> >> Thanks, >> >> Henry >> >> On Sun, Jan 19, 2014 at 10:33 AM, Mridul Muralidharan <[email protected]> >> wrote: >>> Oh great, just saw the PR from Matei ... for some odd reason, the dev >>> mails are coming to be horribly delayed. >>> >>> >>> Thanks, >>> Mridul >>> >>> On Sun, Jan 19, 2014 at 10:35 PM, Patrick Wendell <[email protected]> >>> wrote: >>>> Hey Mridul this was patched and we cut a new release candidate. There >>>> were several different config options which had a.b and a.b.c... they >>>> should all work in the new RC. >>>> >>>> On Sun, Jan 19, 2014 at 4:56 AM, Mridul Muralidharan <[email protected]> >>>> wrote: >>>>> Chanced upon spill related config which exhibit same pattern ... >>>>> >>>>> - Mridul >>>>> >>>>> On Sun, Jan 19, 2014 at 1:10 AM, Reynold Xin <[email protected]> wrote: >>>>>> I also just went over the config options to see how pervasive this is. In >>>>>> addition to speculation, there is one more "conflict" of this kind: >>>>>> >>>>>> spark.locality.wait >>>>>> spark.locality.wait.node >>>>>> spark.locality.wait.process >>>>>> spark.locality.wait.rack >>>>>> >>>>>> >>>>>> spark.speculation >>>>>> spark.speculation.interval >>>>>> spark.speculation.multiplier >>>>>> spark.speculation.quantile >>>>>> >>>>>> >>>>>> On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> This is definitely an important issue to fix. Instead of renaming >>>>>>> properties, one solution would be to replace Typesafe Config with just >>>>>>> reading Java system properties, and disable config files for this >>>>>>> release. >>>>>>> I kind of like that over renaming. >>>>>>> >>>>>>> Matei >>>>>>> >>>>>>> On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> > Hi, >>>>>>> > >>>>>>> > Speculation was an example, there are others in spark which are >>>>>>> > affected by this ... >>>>>>> > Some of them have been around for a while, so will break existing >>>>>>> code/scripts. >>>>>>> > >>>>>>> > Regards, >>>>>>> > Mridul >>>>>>> > >>>>>>> > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <[email protected]> >>>>>>> wrote: >>>>>>> >> change spark.speculation to spark.speculation.switch? >>>>>>> >> >>>>>>> >> maybe we can restrict that all properties in Spark should be "three >>>>>>> levels" >>>>>>> >> >>>>>>> >> >>>>>>> >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan >>>>>>> >> <[email protected] >>>>>>> >wrote: >>>>>>> >> >>>>>>> >>> Hi, >>>>>>> >>> >>>>>>> >>> Unless I am mistaken, the change to using typesafe ConfigFactory >>>>>>> >>> has >>>>>>> >>> broken some of the system properties we use in spark. >>>>>>> >>> >>>>>>> >>> For example: if we have both >>>>>>> >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 >>>>>>> >>> set, then the spark.speculation property is dropped. >>>>>>> >>> >>>>>>> >>> The rules of parseProperty actually document this clearly [1] >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> I am not sure what the right fix here would be (other than replacing >>>>>>> >>> use of config that is). >>>>>>> >>> >>>>>>> >>> Any thoughts ? >>>>>>> >>> I would vote -1 for 0.9 to be released before this is fixed. >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> Regards, >>>>>>> >>> Mridul >>>>>>> >>> >>>>>>> >>> >>>>>>> >>> [1] >>>>>>> >>> >>>>>>> http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 >>>>>>> >>> >>>>>>> >>>>>>>
