Re: [Puppet Users] how to implement cascading defaults based on puppet classes

2013-02-11 Thread jcbollinger 


On Sunday, February 10, 2013 12:58:48 PM UTC-6, Roman Shaposhnik wrote:

 On Sun, Feb 10, 2013 at 9:32 AM, Keith Burdis ke...@burdis.orgjavascript: 
 wrote: 
  Have you looked at using hiera [1] for your configuration? It is very 
 good 
  for cascading defaults with more specific config at the top of the 
 hierarchy 
  and the least specific at the bottom. It is part of Puppet 3 and an add 
 on 
  for 2.7. 

 Yes, I have looked at it. My problem with Hiera is that it resides 
 completely 
 outside of Puppet's DSL and is arguably pretty orthogonal to the 
 parameterized 
 classes API.



Separating configuration data from your manifests is precisely what Hiera 
is for.  The prevailing opinion around here is that that's a *good* thing.  
Besides, if you want the data in Puppet DSL then you can just declare it as 
variables of some class or classes.  Or your ENC can set it as the values 
of global variables.

Furthermore, Puppet 3 integrates Hiera with parameterized classes by using 
hiera to look up values for class parameters that are not explicitly 
declared, before ultimately falling back to any defaults written into the 
class itself.  There are good reasons to rely on that function completely, 
instead of declaring parameters explicitly anywhere, though those reasons 
are somewhat weaker if you use a custom ENC.

 

 I like parameterized classes precisely because they are 
 dynamic. IOW, I can have my ENC instantiate classes with particular 
 parameters depending on any criteria I want *without* affecting anything 
 else on the system. If I want my cluster instantiated during odd numbered 
 hours to have auth = 'simple' and the rest auth = 'kerberos' -- that's 
 just ENC's job. If I want to pull off that same trick with Hiera -- 
 I'm essentially 
 signing up to change a bunch of global variables that can potentially 
 affect my entire site. 



Is that a bona fide use case for you?  That would be unusual.

In any case, if you rely on the Hiera integration with parametrized classes 
then you can draw your data from Hiera under normal circumstances without 
losing the ability to override via your ENC.

 


 That said, I'm starting to suspect that any solution to my problem that 
 can 
 be hidden inside of parsing step has to either be parsing order dependent 
 (yuk!) or it has to rely on some global state that is guaranteed to be 
 consistent 
 before the parser takes over. Hiera fits that description, but it is too 
 global. 



I really don't understand what you mean by too global.  Hiera's 
architecture appears to be an excellent fit for your cascading defaults, 
and as I already discussed, you can apply local overrides via your ENC in 
the event that you actually need to do.  It is also easy to define your 
data hierarchy so that you can provide per-machine data where you want to 
do that (or per-rack, or per-datacenter, etc., provided only that Puppet 
can identify the target machines via their node facts).

 

 Perhaps I can simply teach my ENC to spew out the top-level parameter 
 called cascading_defaults which would be a hiera-like hash for looking up 
 those very same values. Then my params class could be made rewritten as: 
  class cluster::hdfs::params { 
   auth = hlookup('auth', $::cascading_defaults, ['cluster', 
 'hdfs'], 'simple') 
   } 

 Does it make sense? 



That sounds like a workable solution, but whether it makes sense depends on 
information I don't have.  I am inclined to suspect, however, that you are 
rejecting Hiera based on mis- or limited understanding of what it can do 
for you.  If Hiera can in fact do the job adequately, then I think it makes 
the *most* sense to use it instead of building your own custom solution.


John

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] how to implement cascading defaults based on puppet classes

2013-02-11 Thread Roman Shaposhnik 
On Mon, Feb 11, 2013 at 6:39 AM, jcbollinger john.bollin...@stjude.org wrote:
 I am inclined to suspect, however, that you are
 rejecting Hiera based on mis- or limited understanding of what it can do for
 you.  If Hiera can in fact do the job adequately, then I think it makes the
 most sense to use it instead of building your own custom solution.

Honestly, I started this thread to be educated on various options. If you
have the patience to bear with me -- that'll be mostly appreciated. I may
ask some silly questions, though -- so patience is the keyword here ;-)

Now, before you read further let me state a few assumptions that I have as
far as using Hiera with a very custom ENC is concerned and see if they
hold true. Here's my ideal state of things as far as how I'd like to use Puppet:
   #1 I'd like to have a very flexible set of Puppet classes capable of
deploying radically different topologies of Bigtop Hadoop clusters.
ASSUMPTION: to me this means a set of module(s) full of
 highly parameterized classes
   #2 Given that there's nothing static about my environment (I'm not
running a datacenter -- I'm running a utility that lets users deploy
random configurations of Bigtop Hadoop clusters on VMs) I'd rather
minimize the # of things I have to configure/interact with
when describing
how the next N VMs should look like.
ASSUMPTION: to me this means relying exclusively on a very custom
 ENC that would instantiate classes
with precisely the
 right set of parameters, instead of
updating Hiera backend
 everytime a new deployment is asked for.
   #3 Even though I can make my ENC spew out a setting for every single
parameter that each class needs I'd rather check with the collective
Puppet intelligence first to figure out whether there's a
DSL-level language
construct that would allow a parameter in a 'base' class
affect the defaults
in the 'child' one. NOTE: I'm NOT talking about dragging data/settings
into my classes, I'm purely asking whether there's a sane way for the
following bit of my ENC output:
classes:
   cluster::hdfs
auth: kerberos
to affect all the children such as cluster::hdfs::namenode so the
ENC doesn't have to do it explicitly.
ASSUMPTION: I do realize that Hiera is one option to make this happen,
 however, it seems that in my case the
trade off is: I need
 to teach my ENC to manipulate the
state of Hiera backend.
 If I decide to do that it seems that
I might as well make my
 ENC output extremely verbose.

 Separating configuration data from your manifests is precisely what Hiera is
 for.  The prevailing opinion around here is that that's a good thing.
 Besides, if you want the data in Puppet DSL then you can just declare it as
 variables of some class or classes.  Or your ENC can set it as the values of
 global variables.

An ideal place for me to keep this data is at the level of ENC. I
really *DO NOT*
want it at the level of the DSL itself. What I do need at the level of the DSL
are the hooks to make the kind of interfacing with the ENC possible.

 Furthermore, Puppet 3 integrates Hiera with parameterized classes by using
 hiera to look up values for class parameters that are not explicitly
 declared, before ultimately falling back to any defaults written into the
 class itself.  There are good reasons to rely on that function completely,
 instead of declaring parameters explicitly anywhere, though those reasons
 are somewhat weaker if you use a custom ENC.

That's precisely my use case -- a very custom ENC. In fact, I'd rather have
my ENC be a single source of truth for all *parameterization*

 Is that a bona fide use case for you?  That would be unusual.

It seems to be (unless given my description above you will draw
a different conclusion).

 I really don't understand what you mean by too global.

Sorry for the poor choice of words. What I really meant was that
in my case, where:
   # the knowledge of how defaults cascade changes with every
  single deployment
   # such knowledge originates with an ENC
having Hiera in the mix would mean yet another subsytem which
state I'd have to keep consistent with the state of my ENC.

 That sounds like a workable solution, but whether it makes sense depends on
 information I don't have.

Well, you know know the full story. If you could help making the right choice
here -- that'll be, as I said, appreciated.

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to

Re: [Puppet Users] how to implement cascading defaults based on puppet classes

2013-02-11 Thread jcbollinger 


On Monday, February 11, 2013 11:09:28 AM UTC-6, Roman Shaposhnik wrote:

 On Mon, Feb 11, 2013 at 6:39 AM, jcbollinger 
 john.bo...@stjude.orgjavascript: 
 wrote: 
  I am inclined to suspect, however, that you are 
  rejecting Hiera based on mis- or limited understanding of what it can do 
 for 
  you.  If Hiera can in fact do the job adequately, then I think it makes 
 the 
  most sense to use it instead of building your own custom solution. 

 Honestly, I started this thread to be educated on various options. If you 
 have the patience to bear with me -- that'll be mostly appreciated. I may 
 ask some silly questions, though -- so patience is the keyword here ;-) 

 Now, before you read further let me state a few assumptions that I have as 
 far as using Hiera with a very custom ENC is concerned and see if they 
 hold true. Here's my ideal state of things as far as how I'd like to use 
 Puppet: 
#1 I'd like to have a very flexible set of Puppet classes capable of 
 deploying radically different topologies of Bigtop Hadoop 
 clusters. 
 ASSUMPTION: to me this means a set of module(s) full of 
  highly parameterized classes 



False.  Any parametrized class you write can be trivially rewritten without 
parameters, instead pulling data from an external source (with hiera being 
the canonical example of such a source).  That changes only the protocol by 
which data is fed to your classes, not their expressive capability.

 

#2 Given that there's nothing static about my environment (I'm not 
 running a datacenter -- I'm running a utility that lets users 
 deploy 
 random configurations of Bigtop Hadoop clusters on VMs) I'd rather 
 minimize the # of things I have to configure/interact with 
 when describing 
 how the next N VMs should look like. 
 ASSUMPTION: to me this means relying exclusively on a very custom 
  ENC that would instantiate classes 
 with precisely the 
  right set of parameters, instead of 
 updating Hiera backend 
  everytime a new deployment is asked for. 



Questionable.  You posit a large and diverse collection of data 
characterizing the configuration that will be deployed.  You will certainly 
need to record that data somewhere, somehow, for each configuration.  You 
or your users will need to do this at least as part of the process of 
developing a configuration.  You or your users will probably want to have 
it on an ongoing basis if they want to *maintain* their configuration via 
Puppet or deploy the same configuration on a new set of VMs.  You or your 
users may also want to have it as a starting point for creating variant 
configurations, instead of starting from scratch each time.

So, since you need to record your data, why not do so in a form that Puppet 
can use directly?  As a bonus, your ENC can then be a lot simpler.

 

#3 Even though I can make my ENC spew out a setting for every single 
 parameter that each class needs I'd rather check with the 
 collective 
 Puppet intelligence first to figure out whether there's a 
 DSL-level language 
 construct that would allow a parameter in a 'base' class 
 affect the defaults 
 in the 'child' one. NOTE: I'm NOT talking about dragging 
 data/settings 
 into my classes, I'm purely asking whether there's a sane way for 
 the 
 following bit of my ENC output: 
 classes: 
cluster::hdfs 
 auth: kerberos 
 to affect all the children such as cluster::hdfs::namenode so the 
 ENC doesn't have to do it explicitly. 
 ASSUMPTION: I do realize that Hiera is one option to make this 
 happen, 
  however, it seems that in my case the 
 trade off is: I need 
  to teach my ENC to manipulate the 
 state of Hiera backend. 
  If I decide to do that it seems that 
 I might as well make my 
  ENC output extremely verbose. 



Questionable.  Although you could, in principle, make your ENC write or 
rewrite data for the hiera back end, that seems a strange way to go about 
it.  One would normally maintain the data separately from, but in parallel 
with the ENC itself.  Otherwise you're recomputing the data every time, or 
else you're reading it from somewhere else and transliterating it into the 
form that Puppet will consume, which seems like wasted effort to me.

To answer your question more directly, Puppet parametrized classes can draw 
on *non-parametrized* classes' variables for their own parameter defaults, 
but the mechanism for doing so reliably uses class inheritance, so

   1. Doing so prevents you using class inheritance for anything else 
   (which may be tolerable), and
   2. You cannot safely use

Re: [Puppet Users] how to implement cascading defaults based on puppet classes

2013-02-10 Thread Keith Burdis 
Have you looked at using hiera [1] for your configuration? It is very good
for cascading defaults with more specific config at the top of the
hierarchy and the least specific at the bottom. It is part of Puppet 3 and
an add on for 2.7.

  - Keith

[1]
http://www.devco.net/archives/2011/06/05/hiera_a_pluggable_hierarchical_data_store.php
 On 10 Feb 2013 06:00, Roman Shaposhnik ro...@shaposhnik.org wrote:

 Hi!

 suppose my puppet recipes are deploying a configuration
 for the entire cluster (lets say Hadoop) consisting of quite a
 few distributed systems (lets say distributed file-system --
 HDFS, scheduler -- YARN, coordinator -- zookeeper). Each
 of these distributed systems in turn consists of a few agents
 (e.g. in HDFS there's a NameNode, DataNode, etc.).

 My biggest concern is how to make it possible to cascade
 defaults in such a way that they can be specified at each
 of the levels: entire cluster, subsystem, an agent that
 is part of the subsystem.

 To be more explicit, suppose I have a node description
 that looks like this:
 -
 class { cluster::hdfs::datanode:
namenode_uri = hdfs://foo.company.com,
 auth = kerberos
 }
 class { cluster::hdfs::secondary_namenode:
 namenode_uri = hdfs://foo.company.com,
auth  = kerberos
 }
 class { cluster::zookeeper::server:
 ensemble = zoo1.company.com,
 auth = kerberos
 }
 -

 What would be the ideal way for me to not repeat myself
 and set auth as a cluster-level default, while having
 namenode_uri as an HDFS-level default?

 Really, what I need should look something like:
 --
 class { cluster:
auth = kerberos
 }
 class { hdfs:
namenode_uri = hdfs://foo.company.com
 }
 class { cluster::hdfs::datanode: }
 class { cluster::hdfs::secondary_namenode: }
 class { cluster::zookeeper::server:
 ensemble = zoo1.company.com,
 }
 

 Any ideas on how this could be implemented? I've tried
 a couple of things, but so far no luck. The natural way
 of solving it via inheritance (e.g. cluster::hdfs inherits cluster
 and cluster::hdfs::datanode inherits cluster::hdfs, etc.)
 seems to be of no help at all. Another thing I was considering
 was something along the lines of:
   class cluster::hdfs::datanode (
   $auth = $cluster::hdfs::params::auth
   ) inherits cluster::hdfs::params {...}
   ..
   class cluster::hdfs::params {
   if defined(Class['cluster::hdfs']) {
   $auth = $cluster::hdfs::auth
   } elsif defined(Class['cluster']) {
   $auth = $cluster::auth
   } else {
   $auth = 'simple'
   }
   }

 seems to be pretty ugly and worse yet  dependent on the parsing
 order.

 Am I missing something obvious here?

 Thanks,
 Roman.

 --
 You received this message because you are subscribed to the Google Groups
 Puppet Users group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to puppet-users+unsubscr...@googlegroups.com.
 To post to this group, send email to puppet-users@googlegroups.com.
 Visit this group at http://groups.google.com/group/puppet-users?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] how to implement cascading defaults based on puppet classes

2013-02-10 Thread Roman Shaposhnik 
On Sun, Feb 10, 2013 at 9:32 AM, Keith Burdis ke...@burdis.org wrote:
 Have you looked at using hiera [1] for your configuration? It is very good
 for cascading defaults with more specific config at the top of the hierarchy
 and the least specific at the bottom. It is part of Puppet 3 and an add on
 for 2.7.

Yes, I have looked at it. My problem with Hiera is that it resides completely
outside of Puppet's DSL and is arguably pretty orthogonal to the parameterized
classes API. I like parameterized classes precisely because they are
dynamic. IOW, I can have my ENC instantiate classes with particular
parameters depending on any criteria I want *without* affecting anything
else on the system. If I want my cluster instantiated during odd numbered
hours to have auth = 'simple' and the rest auth = 'kerberos' -- that's
just ENC's job. If I want to pull off that same trick with Hiera --
I'm essentially
signing up to change a bunch of global variables that can potentially
affect my entire site.

That said, I'm starting to suspect that any solution to my problem that can
be hidden inside of parsing step has to either be parsing order dependent
(yuk!) or it has to rely on some global state that is guaranteed to be
consistent
before the parser takes over. Hiera fits that description, but it is too global.
Perhaps I can simply teach my ENC to spew out the top-level parameter
called cascading_defaults which would be a hiera-like hash for looking up
those very same values. Then my params class could be made rewritten as:
 class cluster::hdfs::params {
  auth = hlookup('auth', $::cascading_defaults, ['cluster',
'hdfs'], 'simple')
      }

Does it make sense?

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] how to implement cascading defaults based on puppet classes

2013-02-09 Thread Roman Shaposhnik 
Hi!

suppose my puppet recipes are deploying a configuration
for the entire cluster (lets say Hadoop) consisting of quite a
few distributed systems (lets say distributed file-system --
HDFS, scheduler -- YARN, coordinator -- zookeeper). Each
of these distributed systems in turn consists of a few agents
(e.g. in HDFS there's a NameNode, DataNode, etc.).

My biggest concern is how to make it possible to cascade
defaults in such a way that they can be specified at each
of the levels: entire cluster, subsystem, an agent that
is part of the subsystem.

To be more explicit, suppose I have a node description
that looks like this:
-
class { cluster::hdfs::datanode:
   namenode_uri = hdfs://foo.company.com,
auth = kerberos
}
class { cluster::hdfs::secondary_namenode:
namenode_uri = hdfs://foo.company.com,
   auth  = kerberos
}
class { cluster::zookeeper::server:
ensemble = zoo1.company.com,
auth = kerberos
}
-

What would be the ideal way for me to not repeat myself
and set auth as a cluster-level default, while having
namenode_uri as an HDFS-level default?

Really, what I need should look something like:
--
class { cluster:
   auth = kerberos
}
class { hdfs:
   namenode_uri = hdfs://foo.company.com
}
class { cluster::hdfs::datanode: }
class { cluster::hdfs::secondary_namenode: }
class { cluster::zookeeper::server:
ensemble = zoo1.company.com,
}


Any ideas on how this could be implemented? I've tried
a couple of things, but so far no luck. The natural way
of solving it via inheritance (e.g. cluster::hdfs inherits cluster
and cluster::hdfs::datanode inherits cluster::hdfs, etc.)
seems to be of no help at all. Another thing I was considering
was something along the lines of:
  class cluster::hdfs::datanode (
  $auth = $cluster::hdfs::params::auth
  ) inherits cluster::hdfs::params {...}
  ..
  class cluster::hdfs::params {
  if defined(Class['cluster::hdfs']) {
  $auth = $cluster::hdfs::auth
  } elsif defined(Class['cluster']) {
  $auth = $cluster::auth
  } else {
  $auth = 'simple'
  }
  }

seems to be pretty ugly and worse yet  dependent on the parsing
order.

Am I missing something obvious here?

Thanks,
Roman.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.