Re: spark config params conventions

2014-03-14 Thread Chester Chen
Based on typesafe config maintainer's response, with latest version of typeconfig, the double quote is no longer needed for key like spark.speculation, so you don't need code to strip the quotes Chester Alpine data labs Sent from my iPhone On Mar 12, 2014, at 2:50 PM, Aaron Davidson

Re: spark config params conventions

2014-03-12 Thread Aaron Davidson
Should we try to deprecate these types of configs for 1.0.0? We can start by accepting both and giving a warning if you use the old one, and then actually remove them in the next minor release. I think spark.speculation.enabled=true is better than spark.speculation=true, and if we decide to use

Re: spark config params conventions

2014-03-12 Thread yao
+1. I agree to keep the old ones only for backward compatibility purpose. On Wed, Mar 12, 2014 at 12:38 PM, Evan Chan e...@ooyala.com wrote: +1. Not just for Typesafe Config, but if we want to consider hierarchical configs like JSON rather than flat key mappings, it is necessary. It is

Re: spark config params conventions

2014-03-12 Thread Aaron Davidson
One solution for typesafe config is to use spark.speculation = true Typesafe will recognize the key as a string rather than a path, so the name will actually be \spark.speculation\, so you need to handle this contingency when passing the config operations to spark (stripping the quotes from the