Really? Disabling config files seems to me to be a bigger/more onerous change for users than spark.speculation=true|false => spark.speculation.enabled=true|false and spark.locality.wait => spark.locality.wait.default.
On Sat, Jan 18, 2014 at 11:36 AM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > This is definitely an important issue to fix. Instead of renaming > properties, one solution would be to replace Typesafe Config with just > reading Java system properties, and disable config files for this release. > I kind of like that over renaming. > > Matei > > On Jan 18, 2014, at 11:30 AM, Mridul Muralidharan <mri...@gmail.com> > wrote: > > > Hi, > > > > Speculation was an example, there are others in spark which are > > affected by this ... > > Some of them have been around for a while, so will break existing > code/scripts. > > > > Regards, > > Mridul > > > > On Sun, Jan 19, 2014 at 12:51 AM, Nan Zhu <zhunanmcg...@gmail.com> > wrote: > >> change spark.speculation to spark.speculation.switch? > >> > >> maybe we can restrict that all properties in Spark should be "three > levels" > >> > >> > >> On Sat, Jan 18, 2014 at 2:10 PM, Mridul Muralidharan <mri...@gmail.com > >wrote: > >> > >>> Hi, > >>> > >>> Unless I am mistaken, the change to using typesafe ConfigFactory has > >>> broken some of the system properties we use in spark. > >>> > >>> For example: if we have both > >>> -Dspark.speculation=true -Dspark.speculation.multiplier=0.95 > >>> set, then the spark.speculation property is dropped. > >>> > >>> The rules of parseProperty actually document this clearly [1] > >>> > >>> > >>> I am not sure what the right fix here would be (other than replacing > >>> use of config that is). > >>> > >>> Any thoughts ? > >>> I would vote -1 for 0.9 to be released before this is fixed. > >>> > >>> > >>> Regards, > >>> Mridul > >>> > >>> > >>> [1] > >>> > http://typesafehub.github.io/config/latest/api/com/typesafe/config/ConfigFactory.html#parseProperties%28java.util.Properties,%20com.typesafe.config.ConfigParseOptions%29 > >>> > >
