Re: Moving to Typesafe Config?
Hi Patrick, That sounds great, I just fired a JIRA ticket for this (https://spark-project.atlassian.net/browse/SPARK-1048). We are setting up spark in one of our data center (~100 machines) and would like to have this feature ASAP. Please let me know if there are things we can help. Thanks -Shengzhe -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Moving-to-Typesafe-Config-tp381p397.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Re: Moving to Typesafe Config?
Hey Heiko, Spark 0.9 introduced a common config class for Spark applications. It also (initially) supported loading config files in the nested typesafe format, but this was removed last minute due to a bug. In 1.0 we'll probably add support for config files, though it may not support typesafe's tree-style config files because that conflicts with the naming style of several spark options (we have options where x.y and x.y.z are both named keys, and the typesafe parser doesn't allow that). - Patrick On Mon, Jan 27, 2014 at 8:59 AM, Heiko Braun wrote: > Thanks. I found the discussion myself ;) > > /heiko > >> Am 27.01.2014 um 17:34 schrieb Mark Hamstra : >> >> And it would be more helpful if I gave you a usable link >> http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html >> >> Sent from my iPhone >> >>> On Jan 27, 2014, at 8:13 AM, Heiko Braun wrote: >>> >>> Thanks Mark. >>> On 27 Jan 2014, at 17:05, Mark Hamstra wrote: Been done and undone, and will probably be redone for 1.0. See https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun wrote: > > Is there any interest in moving to a more structured approach for > configuring spark components? I.e. moving to the typesafe config [1]. > Since > spark already leverages akka, this seems to be a reasonable choice IMO. > > [1] https://github.com/typesafehub/config > > Regards, Heiko >>>
Re: Moving to Typesafe Config?
Thanks. I found the discussion myself ;) /heiko > Am 27.01.2014 um 17:34 schrieb Mark Hamstra : > > And it would be more helpful if I gave you a usable link > http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html > > Sent from my iPhone > >> On Jan 27, 2014, at 8:13 AM, Heiko Braun wrote: >> >> Thanks Mark. >> >>> On 27 Jan 2014, at 17:05, Mark Hamstra wrote: >>> >>> Been done and undone, and will probably be redone for 1.0. See >>> https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 >>> >>> >>> On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun >>> wrote: >>> Is there any interest in moving to a more structured approach for configuring spark components? I.e. moving to the typesafe config [1]. Since spark already leverages akka, this seems to be a reasonable choice IMO. [1] https://github.com/typesafehub/config Regards, Heiko >>
Re: Moving to Typesafe Config?
And it would be more helpful if I gave you a usable link http://apache-spark-developers-list.1001551.n3.nabble.com/Config-properties-broken-in-master-td208.html Sent from my iPhone > On Jan 27, 2014, at 8:13 AM, Heiko Braun wrote: > > Thanks Mark. > >> On 27 Jan 2014, at 17:05, Mark Hamstra wrote: >> >> Been done and undone, and will probably be redone for 1.0. See >> https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 >> >> >> On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun wrote: >> >>> >>> Is there any interest in moving to a more structured approach for >>> configuring spark components? I.e. moving to the typesafe config [1]. Since >>> spark already leverages akka, this seems to be a reasonable choice IMO. >>> >>> [1] https://github.com/typesafehub/config >>> >>> Regards, Heiko >
Re: Moving to Typesafe Config?
Thanks Mark. On 27 Jan 2014, at 17:05, Mark Hamstra wrote: > Been done and undone, and will probably be redone for 1.0. See > https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 > > > On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun wrote: > >> >> Is there any interest in moving to a more structured approach for >> configuring spark components? I.e. moving to the typesafe config [1]. Since >> spark already leverages akka, this seems to be a reasonable choice IMO. >> >> [1] https://github.com/typesafehub/config >> >> Regards, Heiko >> >>
Re: Moving to Typesafe Config?
Been done and undone, and will probably be redone for 1.0. See https://mail.google.com/mail/ca/u/0/#search/config/143a6c39e3995882 On Mon, Jan 27, 2014 at 7:58 AM, Heiko Braun wrote: > > Is there any interest in moving to a more structured approach for > configuring spark components? I.e. moving to the typesafe config [1]. Since > spark already leverages akka, this seems to be a reasonable choice IMO. > > [1] https://github.com/typesafehub/config > > Regards, Heiko > >
Moving to Typesafe Config?
Is there any interest in moving to a more structured approach for configuring spark components? I.e. moving to the typesafe config [1]. Since spark already leverages akka, this seems to be a reasonable choice IMO. [1] https://github.com/typesafehub/config Regards, Heiko
