[Puppet Users] Re: Hiera, define, class, putting all together

[jcbollinger]

On Saturday, June 4, 2016 at 7:20:19 PM UTC-5, Helmut Schneider wrote:
>
> Hi, 
>
> I'm new to puppet and read docs as much as possible, so forgive some 
> confusion in my code. 
>
>
[...]
 

> nodes/xyz.yaml: 
> packages: 
>   - postfix 
>   - wget 
> apacheModules: 
>   - mpm_prefork 
>   - php 
>   - ssl 
>
>
[...]
 

> nodes.pp: 
> case $operatingsystem { 
>   /^(Debian|Ubuntu)$/: { 
>
 
[...]

$ubuntuDefaultApacheModules = hiera_array ('apacheModules', '', 
> 'ubuntu-common') 
> $ubuntuExtraApacheModules = hiera_array ('apacheModules', '', 
> "nodes/$fqdn") 
> $ubuntuApacheModules = [ $ubuntuDefaultApacheModules, 
> $ubuntuExtraApacheModules ] 
> hiera_include ('classes', '', 'ubuntu-common') 
>   } 
>   default: { 
>} 
> } 
> hiera_include ('classes', '', "nodes/$fqdn") 
> [...] 
> install-packages { [ $ubuntuDefaultPackages, $packages ]: } <= works 
> apache24::modules { $ubuntuApacheModules: } <= fails 
>
> The error is: 
>
> Error: Could not retrieve catalog from remote server: Error 400 on 
> SERVER: Duplicate declaration: Apache24::Modules[mpm_prefork] is 
> already declared in file 
> /etc/puppet/environments/production/manifests/nodes.pp:54; cannot 
> redeclare at /etc/puppet/environments/production/manifests/nodes.pp:54 
> on node xyz 
>
> Apache24::Modules[mpm_prefork] varies. I'm using the apache class from 
> puppetlabs. Apache24::Modules is called only from nodes.pp, there are 
> no duplicates. 
>
>

If there are no duplicates among your Hiera data, then it follows that you 
are *creating* the duplication in your manifests, which indeed appears to 
be plausible.  In any event, you do appear to be using Hiera in a less than 
efficient manner.  Here are some important considerations that bear on the 
question:

   - The choice among functions hiera(), hiera_array(), and hiera_hash() 
   has little to do with the expected type of the value returned.  Although 
   the latter two do return an array and a hash, respectively, the plain 
   hiera() function will also return one of those if that is the type of 
   the data it finds associated with the given key.  The primary distinction 
   among these is the lookup strategy.  In particular, the hiera_array() 
   function performs an "array-merge" lookup, which collects data from the 
   whole hierarchy, whereas the plain hiera() function performs a priority 
   lookup, returning only the value from the highest-priority hierarchy level 
   that provides one.
   - The third parameter to the hiera-family functions is optional, and 
   usually unneeded.  Additionally, I suspect its docs are misleading for 
   hiera_hash() and hiera_array(): whereas it does cause the specified 
   hierarchy level to be consulted, first, it does not makes sense for these 
   two functions for the search to stop there in the event that the key is 
   found in the override level.  That ought to apply only to the hiera() 
   function, from which the docs for that argument in the other two functions 
   appear to have been copied.
   
 

> My question is: Is the approach itself correct? I want to pass 
> $packages and $apachemodules to a class that does the install. 
>
>

There's nothing inherently wrong with constructing those arrays and using 
them as resource titles.  There's nothing any more wrong about using 
array-titled, resource-like class declarations than there is about using 
resource-like class declarations in general.  If you use array titles, 
however, then you undertake responsibility to ensure that the arrays 
contain no duplicate elements.  In this case, it appears that you have 
ensured the opposite.

Consider this call:

hiera_array ('apacheModules', '', 'ubuntu-common')

It performs an array-merge lookup for key 'apacheModules', with hierarchy 
level 'ubuntu-common' inserted at the top of the hierarchy.  That level is 
already in the hierarchy for the node, or so I must presume.  Now remember 
that an array-merge lookup collects data from every level in the hierarchy, 
so if that duplicate level contains any data for the requested key then you 
will automatically get dupes for all those data.

Even if you weren't getting dupes by virtue of having the same hierarchy 
level consulted twice, it looks like you have a second avenue for dupes.  
You are performing two array-merge lookups and then concatenating the 
results.  Since each array-merge lookup will collect data from the whole 
hierarchy (plus, in your case, an extra level), the two sets of results you 
are concatenating will have many elements in common, so concatenating them 
produces dupes.

The bottom line is that you're making this a lot harder than it needs to 
be.  You have no need to use a manual hierarchy override if you simply 
define the hierarchy appropriately, as the details of your error suggest 
you have already done.  And you do not need separate array-merge lookups 
for each key: with the hierarchy defined correctly, a single one will do

Re: [Puppet Users] Re: Hiera, define, class, putting all together

[Helmut Schneider]

jcbollinger wrote:

> On Saturday, June 4, 2016 at 7:20:19 PM UTC-5, Helmut Schneider wrote:
> > 
> > Hi, 
> > 
> > I'm new to puppet and read docs as much as possible, so forgive
> > some confusion in my code. 

[Code]

> Consider this call:
> 
> hiera_array ('apacheModules', '', 'ubuntu-common')
> 
> It performs an array-merge lookup for key 'apacheModules', with
> hierarchy level 'ubuntu-common' inserted at the top of the hierarchy.
> That level is already in the hierarchy for the node, or so I must
> presume.  Now remember that an array-merge lookup collects data from
> every level in the hierarchy, so if that duplicate level contains any
> data for the requested key then you will automatically get dupes for
> all those data.
> 
> Even if you weren't getting dupes by virtue of having the same
> hierarchy level consulted twice, it looks like you have a second
> avenue for dupes.  You are performing two array-merge lookups and
> then concatenating the results.  Since each array-merge lookup will
> collect data from the whole hierarchy (plus, in your case, an extra
> level), the two sets of results you are concatenating will have many
> elements in common, so concatenating them produces dupes.

This in fact wasn't clear to me, I thought it would restrict the serach
to ubuntu-common.yaml and not extend it.

> In fact, even your 'case' statement is a bit suspect.

Without the case statement, how can I make sure that ubuntu only
receives classes for ubuntu and not e.g. for Windows then?

> With appropriate use of fact interpolation in your hierarchy
> definition, you should be able to reduce [the section you presented
> of] your nodes.pp to just this:
> 
> $packages = hiera_array ('packages', [])
> $apacheModules = hiera_array ('apacheModules', []) 
> hiera_include ('classes') 
> install-packages { $packages: }
> apache24::modules { $apacheModules: }

Works like a charm btw.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/xn0k6yybesbrkfk000%40news.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Hiera, define, class, putting all together

[jcbollinger]

On Tuesday, June 7, 2016 at 8:02:24 AM UTC-5, Helmut Schneider wrote:
>
> jcbollinger wrote: 
>
>
Without the case statement, how can I make sure that ubuntu only 
> receives classes for ubuntu and not e.g. for Windows then? 
>
>

That is where your data hierarchy comes in.  You have a level named 
"ubuntu-common"; it therefore stands to reason that if you have any Windows 
machines under management then there is also a "windows-common" (even if 
it's only notional).  These would be *alternatives* for a particular level 
of your hierarchy, and you would select which to use based on node facts.  
That's precisely what you presently do via the 'case' statement, but you 
can, and probably should, do it directly in your hierarchy definition.  
This is one of the primary uses of Hiera interpolation tokens.  The Hiera 
docs for defining a hierarchy 
 discuss this and provide 
an example.  (I've linked to the Hiera 3.1 docs, but substantially the same 
thing applies to older Hiera as well.)


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/e2eb4a36-5ae2-4dee-86fa-f938ade5f9d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Hiera, define, class, putting all together

[Helmut Schneider]

jcbollinger wrote:

> 
> On Tuesday, June 7, 2016 at 8:02:24 AM UTC-5, Helmut Schneider wrote:
> > 
> > jcbollinger wrote: 
> > 
> > 
> Without the case statement, how can I make sure that ubuntu only 
> > receives classes for ubuntu and not e.g. for Windows then? 
> > 
> > 
> 
> That is where your data hierarchy comes in.  You have a level named 
> "ubuntu-common"; it therefore stands to reason that if you have any
> Windows machines under management then there is also a
> "windows-common" (even if it's only notional).  These would be
> alternatives for a particular level of your hierarchy, and you would
> select which to use based on node facts.  That's precisely what you
> presently do via the 'case' statement, but you can, and probably
> should, do it directly in your hierarchy definition.  This is one of
> the primary uses of Hiera interpolation tokens.  The Hiera docs for
> defining a hierarchy
>  discuss this and
> provide an example.  (I've linked to the Hiera 3.1 docs, but
> substantially the same thing applies to older Hiera as well.)

I changed my config accordingly:

:hierarchy:
  - "nodes/%{::fqdn}"
  - "%{::operatingsystem}"
  - "%{::osfamily}"
  - common

Thank you!

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/xn0k7324iwgvkxm000%40news.gmane.org.
For more options, visit https://groups.google.com/d/optout.