Re: [Puppet Users] Puppet array merge failure?

[jcbollinger]

On Thursday, February 5, 2015 at 10:30:52 AM UTC-6, leam hall wrote:
>
> And on that note I moved the 
>
>   $servers   = hiera('ntp_servers') 
>
> to 
>
>   $servers   = hiera_array('ntp_servers') 
>
> and resolved the issue. Woo-hoo! 
>
>

Indeed.  As you discovered, the hiera() function (and automated data 
binding) perform priority lookups.  If you want array- or hash-merge 
lookups then you need to use the appropriate function specific to that kind 
of lookup.  This is an acknowledged limitation of Hiera -- in practice, the 
appropriate type of lookup for a given key is usually a property of the 
data, but there is no built-in way to represent that alongside the data.  
The data consumer somehow just has to know.  This is an especial problem 
for automated data binding, where you cannot even directly control the type 
of lookup performed.

Anyway, I'm glad you've got it sorted.  Yay!


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/2e257bcb-a29e-485f-b47c-cfaf1bbbe0fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[leam hall]

And on that note I moved the

  $servers   = hiera('ntp_servers')

to

  $servers   = hiera_array('ntp_servers')

and resolved the issue. Woo-hoo!

Leam

-- 
Mind on a Mission

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACv9p5qWdrUEZJqpyF822N_ZbiJLg0mO9nvs96KPxMeyE-cmmg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[leam hall]

We're using the PuppetLabs ntp module with minor changes.

environments/sandbox/modules/ntp/manifests:

|-- config.pp
|-- init.pp
|-- install.pp
|-- params.pp
`-- service.pp

In init.pp we're using servers in the class parameters:

  $servers   = $ntp::params::servers,

Didn't change the $servers validation:

   validate_array($servers)


In params.pp we use a hiera call to build $servers:

  $servers   = hiera('ntp_servers')

and commented out the centos pool servers under the 'RedHat' $osfamily
case statement.

Is that what you were looking for?

Leam

-- 
Mind on a Mission

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CACv9p5rdgm4zYjCFDuqwjEUWFY4ZJ6bkx1hCefUjMd4gr0mPOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[jcbollinger]

On Monday, February 2, 2015 at 10:19:33 AM UTC-6, leam hall wrote:
>
> Hey John, thanks! 
>
> To add to your points, I'm referencing 
> https://docs.puppetlabs.com/hiera/1/lookup_types.html#array-merge 
>
> Your recommended Hiera command works fine on the master. Thanks! All 
> NTP servers show up. 
>
> However, they are not getting put into the ntp.conf file. I've 
> modified both the sandbox and remotenode json files to add extra 
> servers. The new servers show up in a hiera call on the puppet master 
> but are not showing up in the remotenode ntp config file. 
>
> Back to trouble-shooting... 
>


At this point, it sounds like we can conclude that the Hiera data files are 
not your problem.  If you want further help troubleshooting then we'll need 
to see the resource declaration by which the target file is being managed.  
If you are templating the file then the relevant part(s) of the template, 
too.


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/f8e03b49-6468-4070-99aa-660df0636dbd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[leam hall]

Hey John, thanks!

To add to your points, I'm referencing
https://docs.puppetlabs.com/hiera/1/lookup_types.html#array-merge

Your recommended Hiera command works fine on the master. Thanks! All
NTP servers show up.

However, they are not getting put into the ntp.conf file. I've
modified both the sandbox and remotenode json files to add extra
servers. The new servers show up in a hiera call on the puppet master
but are not showing up in the remotenode ntp config file.

Back to trouble-shooting...

Leam



On Mon, Feb 2, 2015 at 9:56 AM, jcbollinger  wrote:
>
>
> On Friday, January 30, 2015 at 11:33:37 AM UTC-6, leam hall wrote:
>>
>>
>> On Friday, January 30, 2015 at 12:11:20 PM UTC-5, Matthew Burgess wrote:
>>>
>>>
>>> On 30 Jan 2015 14:09, "leam hall"  wrote:
>>> >
>>> > :merge_behavior: deep
>>>
>>> See
>>> https://docs.puppetlabs.com/hiera/1/lookup_types.html#deep-merging-in-hiera--120.
>>> In particular you probably want 'deeper' not 'deep' behavior.
>>>
>>> Matt.
>>
>>
>> Hey Matt, good catch. However, I changed to deep_merge in hiera.yaml [...]
>
>
>
> I suppose you mean you changed to
>
> :merge_behavior: deeper
>
> If not, then you did not make the change Matthew suggested.
>
>
>>
>> I think we have the deep_merge gem installed:
>>
>> /opt/puppet/lib/ruby/gems/1.9.1/gems/deep_merge-1.0.0
>>
>> Not sure if the gems just get picked up or have to be called specifically.
>
>
>
> Since it seems you are using Puppet PE, the installer should have ensured
> you have all necessary gems, in the right places.
>
> I suppose the relevant question at this point is why you expect different
> behavior than you see?  You did not present your data files, but I infer
> from what you did present that you have at least these:
>
> /var/lib/hiera/node/remotenode.example.com.yaml
> /var/lib/hiera/sandbox.yaml
>
> You might have .json files instead of or in addition to the .yaml files, and
> you might have more files than those, such as a global.yaml and/or a
> defaults.yaml, but only the fqdn- and environment-specific data sources have
> values associated with key 'ntp_servers'.  The "node/remotenode.example.com"
> data source(s) provides the "defiant" and "aquarius" values, and the
> "sandbox" data source(s) provides the "ntp1" and "ntp2" values.
>
> Now consider this lookup you report performing:
>
> hiera -a --array ntp_servers fqdn=remotenode.example.com
>
> Why do you give the "fqdn=..." argument?  Clearly you discovered, one way or
> another, that that is how you can present fact values to Hiera when you
> invoke it from the command line.  In this case, that allows Hiera to
> interpolate the given fact value into your hierarchy definition, so that it
> will consult your "node/%{fqdn}" data source(s) for that FQDN.
>
> Similarly, with this command ...
>
> hiera  -a ntp_servers environment=sandbox
>
> ... the "environment=sandbox" defines the "environment" fact to Hiera, so
> that it chooses the "sandbox" data source(s) in the "%{environment}" slot.
>
> By this point I'm sure you see where I'm going: in each of those cases, you
> define only one fact to Hiera.  You do not in either case give it the
> information to choose both data sources you want it to use.  Thus, this is
> not an array merge failure, it is a data source specification failure.  I
> predict you would see the results you want with this:
>
> hiera --array ntp_servers environment=sandbox fqdn=remotenode.example.com
>
> There are several things to note here:
>
> The -a and --array options are synonymous.  You don't need to specify both.
> The native / deep / deeper merge behavior pertains to hash-merge lookups
> only.  You are performing array-merge lookups, so that option is irrelevant
> to your tests.
> Hiera does not consider it an error for any data source to be missing -- it
> simply skips any it cannot open.  That is intentional, and generally what
> you want when the data source is genuinely missing, but it can produce
> surprising results if the data source is present but inaccessible (e.g.
> because of ownership / permissions / mandatory access controls, etc.).
> When Puppet invokes Hiera via its Ruby API, all the Puppet variables that
> are in scope are available to Hiera for interpolation, including in
> particular all fact values reported by the target node.  You won't encounter
> an issue of the type that tripped you up in conjunction with Hiera usage by
> Puppet.
>
>
> John
>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/bB-wGAOQAHI/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/360ff971-55e8-421e-b793-cd82c26050ad%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Mind on a 

Re: [Puppet Users] Puppet array merge failure?

[jcbollinger]

On Friday, January 30, 2015 at 11:33:37 AM UTC-6, leam hall wrote:
>
>
> On Friday, January 30, 2015 at 12:11:20 PM UTC-5, Matthew Burgess wrote:
>>
>>
>> On 30 Jan 2015 14:09, "leam hall"  wrote:
>> >
>> > :merge_behavior: deep
>>
>> See 
>> https://docs.puppetlabs.com/hiera/1/lookup_types.html#deep-merging-in-hiera--120.
>>  
>> In particular you probably want 'deeper' not 'deep' behavior.
>>
>> Matt.
>>
>
> Hey Matt, good catch. However, I changed to deep_merge in hiera.yaml [...]
>


I suppose you mean you changed to

:merge_behavior: deeper

If not, then you did not make the change Matthew suggested.

 

> I think we have the deep_merge gem installed:
>
> /opt/puppet/lib/ruby/gems/1.9.1/gems/deep_merge-1.0.0
>
> Not sure if the gems just get picked up or have to be called specifically.
>


Since it seems you are using Puppet PE, the installer should have ensured 
you have all necessary gems, in the right places.

I suppose the relevant question at this point is why you expect different 
behavior than you see?  You did not present your data files, but I infer 
from what you did present that you have at least these:

/var/lib/hiera/node/remotenode.example.com.yaml
/var/lib/hiera/sandbox.yaml

You might have .json files instead of or in addition to the .yaml files, 
and you might have more files than those, such as a global.yaml and/or a 
defaults.yaml, but only the fqdn- and environment-specific data sources 
have values associated with key 'ntp_servers'.  The 
"node/remotenode.example.com" data source(s) provides the "defiant" and 
"aquarius" values, and the "sandbox" data source(s) provides the "ntp1" and 
"ntp2" values.

Now consider this lookup you report performing:

hiera -a --array ntp_servers fqdn=remotenode.example.com 


Why do you give the "fqdn=..." argument?  Clearly you discovered, one way 
or another, that that is how you can present fact values to Hiera when you 
invoke it from the command line.  In this case, that allows Hiera to 
interpolate the given fact value into your hierarchy definition, so that it 
will consult your "node/%{fqdn}" data source(s) for *that* FQDN.

Similarly, with this command ...

hiera  -a ntp_servers environment=sandbox

... the "environment=sandbox" defines the "environment" fact to Hiera, so 
that it chooses the "sandbox" data source(s) in the "%{environment}" slot.

By this point I'm sure you see where I'm going: in each of those cases, you 
define only one fact to Hiera.  You do not in either case give it the 
information to choose both data sources you want it to use.  Thus, this is 
not an array merge failure, it is a data source specification failure.  I 
predict you would see the results you want with this:

hiera --array ntp_servers environment=sandbox fqdn=remotenode.example.com 


There are several things to note here:

   - The -a and --array options are synonymous.  You don't need to specify 
   both.
   - The native / deep / deeper merge behavior pertains to hash-merge 
   lookups only.  You are performing array-merge lookups, so that option is 
   irrelevant to your tests.
   - Hiera does not consider it an error for any data source to be missing 
   -- it simply skips any it cannot open.  That is intentional, and generally 
   what you want when the data source is genuinely missing, but it can produce 
   surprising results if the data source is present but inaccessible (e.g. 
   because of ownership / permissions / mandatory access controls, etc.).
   - When Puppet invokes Hiera via its Ruby API, all the Puppet variables 
   that are in scope are available to Hiera for interpolation, including in 
   particular all fact values reported by the target node.  You won't 
   encounter an issue of the type that tripped you up in conjunction with 
   Hiera usage by Puppet.


John


-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/360ff971-55e8-421e-b793-cd82c26050ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[Sans]

Did you check if have ruby-deep-merge (or the equivalent to your OS) 
installed? 

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/e65a5cfe-f689-437c-a5d3-0312eb4d4482%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[leam hall]

On Friday, January 30, 2015 at 12:11:20 PM UTC-5, Matthew Burgess wrote:
>
>
> On 30 Jan 2015 14:09, "leam hall" > wrote:
> >
> > :merge_behavior: deep
>
> See 
> https://docs.puppetlabs.com/hiera/1/lookup_types.html#deep-merging-in-hiera--120.
>  
> In particular you probably want 'deeper' not 'deep' behavior.
>
> Matt.
>

Hey Matt, good catch. However, I changed to deep_merge in hiera.yaml, 
restarted the pe-puppet service, and am still getting the behavior. I think 
we have the deep_merge gem installed:

/opt/puppet/lib/ruby/gems/1.9.1/gems/deep_merge-1.0.0

Not sure if the gems just get picked up or have to be called specifically.

Thanks!

Leam

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/46cb9eff-bb17-47e6-b962-3170b978c82b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Puppet array merge failure?

[Matthew Burgess]

On 30 Jan 2015 14:09, "leam hall"  wrote:
>
> :merge_behavior: deep

See
https://docs.puppetlabs.com/hiera/1/lookup_types.html#deep-merging-in-hiera--120.
In particular you probably want 'deeper' not 'deep' behavior.

Matt.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAKUTv3Kq1rEvFFhXzuc1uyTMY_Npmdv3be5O6cm_hnQi5naKag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Puppet array merge failure?

[leam hall]

Note that I'm still learning so this could easily be operator error. I'd 
like to get all 4 ntp servers in in the array. That is, I have other things 
that need to be in merged hashes and arrays and I'm using ntp_servers as a 
learning tool.

Thanks!

Leam

#

hiera.yaml
--
:backends:
  - yaml
  - json
:hierarchy:
  - "node/%{fqdn}"
  - "%{osfamily}"
  - "%{clientcert}"
  - "%{environment}"
  - global
  - defaults

:yaml:
  :datadir: "/var/lib/hiera"
:json:
  :datadir: "/var/lib/hiera"

:merge_behavior: deep



Hiera tests on puppet master
-

Note that remotenode.example.com is in the sandbox environment.

#  hiera -a --array ntp_servers fqdn=remotenode.example.com
["aquarius.example.com", "defiant.example.com"]


#  hiera  -a ntp_servers environment=sandbox
['ntp1.example.com", "ntp2.example.com"]


-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/969b1927-1c22-4c21-b736-12deaf992b32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.