Re: [Puppet Users] Hiera data repetition

[Martin Alfke]

Hi Karsten,

> On 27. Feb 2021, at 18:18, Karsten Heymann  wrote:
> 
> Hi Martin,
> 
> regarding your "no lookup" policy, how do you handle for example deep merge 
> lookups. They cannot be used with automatic data binding, or am I wrong?
> 

we set the merge behavior using hiera data lookup_options.
https://blog.example42.com/2021/02/11/hiera-merge-behavior/ (at the end of the 
post you can see an example).
This allows you to not only set merge behaviour globally, but use the 
hierarchies to even set different lookup options.

e.g. you will deep merge the user hash from all hierarchies, except for one 
system:

# data/common.yaml
lookup_options:
  profile::base::usermgmt::users:
merge: 'deep'

# data/nodes/.yaml
lookup_options:
  profile::base::usermgmt::users:
merge: 'first'

Best,
Martin

> Regards
> Karsten
> 
> Am Sa., 27. Feb. 2021 um 13:06 Uhr schrieb Martin Alfke :
> Hi
> 
> > On 25. Feb 2021, at 11:59, Dmitry Nurislamov  wrote:
> > 
> > Hello. Sometimes it is desirable to define Hiera variables for usage in 
> > Hiera itself, i.e. variables that won't be looked up from Puppet manifests. 
> > We do this to avoid repetition. Here's an example of a variable used only 
> > in the file it is defined:
> > 
> > _internal_api_host: 'int.api.example.com'
> > profile::keepalived::vrrp_script:
> >   ping_internal_api:
> > script: "curl https://%{lookup('_internal_api_host')}/ping"
> > profile::nginx::servers:
> >   "%{alias('_internal_api_host')}":
> > ...
> > 
> > Sometimes we also define this kind of variables in one file but do lookups 
> > somewhere else:
> > 
> > hieradata/project/foo.yaml:
> > profile::base::network::interfaces:
> >   eth0:
> > method: 'static'
> > address: "%{lookup('_INTERNAL_IP')}/24"
> > 
> > hieradata/project/foo/foo1.yaml:
> > _INTERNAL_IP: '10.1.2.3'
> > 
> > For us this became an issue. There are no conventions, and therefore no 
> > consistency in our quite large hiera. Sometimes we can't even tell whether 
> > some variable is looked up from manifests or is "local" for Hiera. For 
> > example, the "_INTERNAL_IP" above is named 
> > "profile::base::network::intern_ip" in some projects.
> > 
> > The question is... Is this a normal practice? How do other folks handle 
> > these cases? I couldn't find any information regarding this. Thanks.
> 
> This is part of the way how you can implement hiera.
> We heavily use the same pattern to avoid duplicate data.
> Unluckily there is (yet?) no best practices written down.
> 
> Usually we have a pattern that we strictly follow:
> - Never set class parameters on module (technical component or library 
> module) level. Only set data on profile classes.
> - Only do automatic data binding, using profile class namespace on hiera 
> data, never do explizit lookups in profile classes.
>   - e.g.:profile::base::usermgmt::users: {}
> - nested lookups always use short key names, usually prefixed with company or 
> department or team short name
>   - e.g. company_ldap_pass
> 
> This pattern allows us to strictly separate data which are needed multiple 
> times in hiera.
> Using automatic data binding and the name convention on nested lookups, we 
> can easily identify if data is consumed by a class or needed hiera internally.
> 
> hth,
> Martin
> 
> 
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups 
> > "Puppet Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to puppet-users+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/puppet-users/CAL017hDOZ7zoJqT0pbAyXVq4665TPDbOJ5Y-ELwUgzbaSofpKg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/849CA551-45AD-47D7-9CB4-369595DAD605%40gmail.com.

Re: [Puppet Users] Hiera data repetition

[Henrik Lindberg]

On 2021-02-27 18:18, Karsten Heymann wrote:

Hi Martin,

regarding your "no lookup" policy, how do you handle for example deep 
merge lookups. They cannot be used with automatic data binding, or am I 
wrong?


Regards
Karsten


You can specify lookup_options per key in the hiera data itself. One of 
the options is if the lookup should be a deep merge or not. This is 
supported for all lookups (automatic or not) and was introduced 
especially to make it possible to do deep merge (and other things) when 
using APL).


Best,
- henrik




Am Sa., 27. Feb. 2021 um 13:06 Uhr schrieb Martin Alfke 
mailto:tux...@gmail.com>>:


Hi

 > On 25. Feb 2021, at 11:59, Dmitry Nurislamov mailto:dmnu...@gmail.com>> wrote:
 >
 > Hello. Sometimes it is desirable to define Hiera variables for
usage in Hiera itself, i.e. variables that won't be looked up from
Puppet manifests. We do this to avoid repetition. Here's an example
of a variable used only in the file it is defined:
 >
 >     _internal_api_host: 'int.api.example.com
'
 >     profile::keepalived::vrrp_script:
 >       ping_internal_api:
 >         script: "curl https://%{lookup('_internal_api_host')}/ping"
 >     profile::nginx::servers:
 >       "%{alias('_internal_api_host')}":
 >         ...
 >
 > Sometimes we also define this kind of variables in one file but
do lookups somewhere else:
 >
 > hieradata/project/foo.yaml:
 >     profile::base::network::interfaces:
 >       eth0:
 >         method: 'static'
 >         address: "%{lookup('_INTERNAL_IP')}/24"
 >
 > hieradata/project/foo/foo1.yaml:
 >     _INTERNAL_IP: '10.1.2.3'
 >
 > For us this became an issue. There are no conventions, and
therefore no consistency in our quite large hiera. Sometimes we
can't even tell whether some variable is looked up from manifests or
is "local" for Hiera. For example, the "_INTERNAL_IP" above is named
"profile::base::network::intern_ip" in some projects.
 >
 > The question is... Is this a normal practice? How do other folks
handle these cases? I couldn't find any information regarding this.
Thanks.

This is part of the way how you can implement hiera.
We heavily use the same pattern to avoid duplicate data.
Unluckily there is (yet?) no best practices written down.

Usually we have a pattern that we strictly follow:
- Never set class parameters on module (technical component or
library module) level. Only set data on profile classes.
- Only do automatic data binding, using profile class namespace on
hiera data, never do explizit lookups in profile classes.
   - e.g.:profile::base::usermgmt::users: {}
- nested lookups always use short key names, usually prefixed with
company or department or team short name
   - e.g. company_ldap_pass

This pattern allows us to strictly separate data which are needed
multiple times in hiera.
Using automatic data binding and the name convention on nested
lookups, we can easily identify if data is consumed by a class or
needed hiera internally.

hth,
Martin


 >
 > --
 > You received this message because you are subscribed to the
Google Groups "Puppet Users" group.
 > To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-users+unsubscr...@googlegroups.com
.
 > To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com

.

-- 
You received this message because you are subscribed to the Google

Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-users+unsubscr...@googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com

.

--
You received this message because you are subscribed to the Google 
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to puppet-users+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAL017hDOZ7zoJqT0pbAyXVq4665TPDbOJ5Y-ELwUgzbaSofpKg%40mail.gmail.com 
.



--

Visit my Blog "Puppet on the Edge"

Re: [Puppet Users] Hiera data repetition

[Karsten Heymann]

Hi Martin,

regarding your "no lookup" policy, how do you handle for example deep merge
lookups. They cannot be used with automatic data binding, or am I wrong?

Regards
Karsten

Am Sa., 27. Feb. 2021 um 13:06 Uhr schrieb Martin Alfke :

> Hi
>
> > On 25. Feb 2021, at 11:59, Dmitry Nurislamov  wrote:
> >
> > Hello. Sometimes it is desirable to define Hiera variables for usage in
> Hiera itself, i.e. variables that won't be looked up from Puppet manifests.
> We do this to avoid repetition. Here's an example of a variable used only
> in the file it is defined:
> >
> > _internal_api_host: 'int.api.example.com'
> > profile::keepalived::vrrp_script:
> >   ping_internal_api:
> > script: "curl https://%{lookup('_internal_api_host')}/ping"
> > profile::nginx::servers:
> >   "%{alias('_internal_api_host')}":
> > ...
> >
> > Sometimes we also define this kind of variables in one file but do
> lookups somewhere else:
> >
> > hieradata/project/foo.yaml:
> > profile::base::network::interfaces:
> >   eth0:
> > method: 'static'
> > address: "%{lookup('_INTERNAL_IP')}/24"
> >
> > hieradata/project/foo/foo1.yaml:
> > _INTERNAL_IP: '10.1.2.3'
> >
> > For us this became an issue. There are no conventions, and therefore no
> consistency in our quite large hiera. Sometimes we can't even tell whether
> some variable is looked up from manifests or is "local" for Hiera. For
> example, the "_INTERNAL_IP" above is named
> "profile::base::network::intern_ip" in some projects.
> >
> > The question is... Is this a normal practice? How do other folks handle
> these cases? I couldn't find any information regarding this. Thanks.
>
> This is part of the way how you can implement hiera.
> We heavily use the same pattern to avoid duplicate data.
> Unluckily there is (yet?) no best practices written down.
>
> Usually we have a pattern that we strictly follow:
> - Never set class parameters on module (technical component or library
> module) level. Only set data on profile classes.
> - Only do automatic data binding, using profile class namespace on hiera
> data, never do explizit lookups in profile classes.
>   - e.g.:profile::base::usermgmt::users: {}
> - nested lookups always use short key names, usually prefixed with company
> or department or team short name
>   - e.g. company_ldap_pass
>
> This pattern allows us to strictly separate data which are needed multiple
> times in hiera.
> Using automatic data binding and the name convention on nested lookups, we
> can easily identify if data is consumed by a class or needed hiera
> internally.
>
> hth,
> Martin
>
>
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAL017hDOZ7zoJqT0pbAyXVq4665TPDbOJ5Y-ELwUgzbaSofpKg%40mail.gmail.com.

Re: [Puppet Users] Hiera data repetition

[Martin Alfke]

Hi

> On 25. Feb 2021, at 11:59, Dmitry Nurislamov  wrote:
> 
> Hello. Sometimes it is desirable to define Hiera variables for usage in Hiera 
> itself, i.e. variables that won't be looked up from Puppet manifests. We do 
> this to avoid repetition. Here's an example of a variable used only in the 
> file it is defined:
> 
> _internal_api_host: 'int.api.example.com'
> profile::keepalived::vrrp_script:
>   ping_internal_api:
> script: "curl https://%{lookup('_internal_api_host')}/ping"
> profile::nginx::servers:
>   "%{alias('_internal_api_host')}":
> ...
> 
> Sometimes we also define this kind of variables in one file but do lookups 
> somewhere else:
> 
> hieradata/project/foo.yaml:
> profile::base::network::interfaces:
>   eth0:
> method: 'static'
> address: "%{lookup('_INTERNAL_IP')}/24"
> 
> hieradata/project/foo/foo1.yaml:
> _INTERNAL_IP: '10.1.2.3'
> 
> For us this became an issue. There are no conventions, and therefore no 
> consistency in our quite large hiera. Sometimes we can't even tell whether 
> some variable is looked up from manifests or is "local" for Hiera. For 
> example, the "_INTERNAL_IP" above is named 
> "profile::base::network::intern_ip" in some projects.
> 
> The question is... Is this a normal practice? How do other folks handle these 
> cases? I couldn't find any information regarding this. Thanks.

This is part of the way how you can implement hiera.
We heavily use the same pattern to avoid duplicate data.
Unluckily there is (yet?) no best practices written down.

Usually we have a pattern that we strictly follow:
- Never set class parameters on module (technical component or library module) 
level. Only set data on profile classes.
- Only do automatic data binding, using profile class namespace on hiera data, 
never do explizit lookups in profile classes.
  - e.g.:profile::base::usermgmt::users: {}
- nested lookups always use short key names, usually prefixed with company or 
department or team short name
  - e.g. company_ldap_pass

This pattern allows us to strictly separate data which are needed multiple 
times in hiera.
Using automatic data binding and the name convention on nested lookups, we can 
easily identify if data is consumed by a class or needed hiera internally.

hth,
Martin


> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/77E8E860-ABA3-4B06-99C6-C294BB0EE750%40gmail.com.

[Puppet Users] Hiera data repetition

[Dmitry Nurislamov]

Hello. Sometimes it is desirable to define Hiera variables for usage in 
Hiera itself, i.e. variables that won't be looked up from Puppet manifests. 
We do this to avoid repetition. Here's an example of a variable used only 
in the file it is defined:

_internal_api_host: 'int.api.example.com'
profile::keepalived::vrrp_script:
  ping_internal_api:
script: "curl https://%{lookup('_internal_api_host')}/ping"
profile::nginx::servers:
  "%{alias('_internal_api_host')}":
...

Sometimes we also define this kind of variables in one file but do lookups 
somewhere else:

hieradata/project/foo.yaml:
profile::base::network::interfaces:
  eth0:
method: 'static'
address: "%{lookup('_INTERNAL_IP')}/24"

hieradata/project/foo/foo1.yaml:
_INTERNAL_IP: '10.1.2.3'

For us this became an issue. There are no conventions, and therefore no 
consistency in our quite large hiera. Sometimes we can't even tell whether 
some variable is looked up from manifests or is "local" for Hiera. For 
example, the "_INTERNAL_IP" above is named 
"profile::base::network::intern_ip" in some projects.

The question is... Is this a normal practice? How do other folks handle 
these cases? I couldn't find any information regarding this. Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/282b3812-287a-4eb0-a838-591f2c3f925an%40googlegroups.com.