Re: [Puppet Users] Re: Status of Data in modules

[Gregory Orange]

Hi Fraser,

On 02/04/15 19:56, Fraser Goffin wrote:

Here's a simple example that works for me :-

--snip--

Wonderful, thank you. Using your work plus 
https://github.com/zipkid/puppet3-hiera_data_in_module/tree/master/modules/test, 
I've got it working. I _think_ my only problem was that I was using a 
define - behaviour is different to classes. I wonder if that is on purpose.


Greg.

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/552353D9.5040801%40calorieking.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Status of Data in modules

[Fraser Goffin]

Here's a simple example that works for me :-

First you obviously have the ripienaar/module_data module installed and 
available to your puppet apply (I'm using masterless puppet).

My folder structure :-

- puppet
| modules
|--- sonarqube
|-- data
|- hiera.yaml
|- common.yaml
|- osfamily
| windows.yaml
| ubuntu.yaml
|- version
| 4.5.yaml
| 5.0.yaml

..\puppet\modules\sonarqube\data\hiera.yaml

:hierarchy:
- "osfamily/%{::osfamily}"
- "version/%{sonarqube::sonarqube_majmin_version}"
- common

:backends:
- yaml

:yaml:
  :datadir: puppet://sonarqube/data

common.yaml :-

sonarqube::sonar_runner_version: '2.4'
sonarqube::sonar_runner_zip: 
"sonar-runner-dist-%{hiera('sonarqube::sonar_runner_version')}.zip"
sonarqube::sonarqube_server_hostname: 'localhost'
sonarqube::sonarqube_server_port: '9000'
sonarqube::sonarqube_database_type: 'mysql'
...

4.5.yaml :-

sonarqube::sonarqube_full_version: '4.5.2'
sonarqube::sonarqube_installer_zip: 
"sonarqube-%{hiera('sonarqube::sonarqube_full_version')}.zip"
...

windows.yaml :-

sonarqube::sonarqube_package_location: 'C:/puppet-installs/sonarqube'
sonarqube::sonarqube_apps_install_basefolder: 'C:/Apps'
...


Then in  ...\puppet\modules\profiles\manifests\sonarqube.pp   (I'm 
following the roles and profiles pattern) :-

  class {'::sonarqube':
#sonarqube_install_path   => $sonarqube_install_path,
#sonarqube_package_location   => $sonarqube_package_location,
#sonarqube_installer_zip  => $sonarqube_installer_zip,
sonarqube_majmin_version => $sonarqube_majmin_version,
#sonar_runner_version => $sonar_runner_version,
#sonarqube_proxy_host=> $sonarqube_proxy_host,
#sonarqube_proxy_port=> $sonarqube_proxy_port
 ...
   }

Now, how you set the values you passed to the parameterised class 
(::sonarqube) is up to you (note: do hiera lookups only in your PROFILE 
class). First you could set all/some of them explicitly either by simply 
assigning a literal or doing a hiera call (obviously the first option here 
takes hiera completely out of the equation since it has the highest 
precedence when resolving the class params) :-

$sonarqube_server_port = '9000'

or (given the above common.yaml) :-

$sonarqube_majmin_version = hiera('sonarqube::sonarqube_server_port')

You might take the view that you are only going to pass explicit values for 
those that you want to over-ride (i.e. all others will be resolved by 
automatic parameter lookup). Keep in mind that, resolution order in the 
target class is :-

1. Explict param value passed.
2. Automatic param lookup (Hiera ... which goes to SITE level hiera FIRST 
and then to MODULE level)
3. Default param value (I would recommend NOT doing that in most cases - 
better to catch config errors rather than the potential of a difficult to 
find false positive)
4. Error

Not sure where your lookup is falling down, perhaps its resolution order 
(2) above ???

HTHs

Fraser.



On Tuesday, 24 March 2015 07:53:36 UTC, gregory...@calorieking.com wrote:
>
> I've been trying to use the module_data module, but cannot get it to 
> bring in values. Is there a worked example somewhere, including site.pp 
> and hiera.yaml? 
>
> https://github.com/zipkid/puppet3-hiera_data_in_module hasn't helped - I 
> get the class defaults from init.pp, not the data from the yaml files. 
>
> Cheers, 
> Greg. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/ab8d0a6d-424b-4333-8d6e-4d87662d94a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Status of Data in modules

[Gregory Orange]

I've been trying to use the module_data module, but cannot get it to 
bring in values. Is there a worked example somewhere, including site.pp 
and hiera.yaml?


https://github.com/zipkid/puppet3-hiera_data_in_module hasn't helped - I 
get the class defaults from init.pp, not the data from the yaml files.


Cheers,
Greg.

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/551117E8.6010702%40calorieking.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Status of Data in modules

[Fabio Sangiovanni]

Ok, I get it. First rule of data in modules: you don't talk about data in 
modules.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/350036b6-05b8-4c2c-a5a8-6c7b4ece25f6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Status of Data in modules

[Trevor Vaughan]

I also would like to know if this is going to get mainlined.

It would be an excellent addition for overall module reuse.

Trevor


On Tue, Dec 31, 2013 at 3:31 AM, Fabio Sangiovanni
wrote:

> Hi everybody,
>
> is there any news about this topic?
> I think it would be great to know Puppetlabs' official position on the
> matter (if any), and the status of activities about it (issues on
> tickets.puppetlabs.com don't say much).
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/f312d16c-0695-48a9-a630-2016fff1ba91%40googlegroups.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CANs%2BFoUsfEMCekaUmkJB3SPX-_aeD-Gtydi%2Bnb7%3D8tATX6viCA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: Status of Data in modules

[Fabio Sangiovanni]

Hi everybody,

is there any news about this topic?
I think it would be great to know Puppetlabs' official position on the 
matter (if any), and the status of activities about it (issues on 
tickets.puppetlabs.com don't say much).

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/f312d16c-0695-48a9-a630-2016fff1ba91%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "Alessandro Franceschi" 
> To: puppet-users@googlegroups.com
> Sent: Sunday, December 8, 2013 10:40:55 PM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> Wow, this looks promising: sane, plain and easy to use.
> Going to test it soon.
> Looking at things in perspective, how do you think this approach will go
> along with the one implemented in Puppet 3.3 ?

In another thread it was mentioned by Hendrik that the 3.3.x one is being 
removed
which prompted me to make my code into a module.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/1052559863.511.1386549474054.JavaMail.zimbra%40devco.net.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Status of Data in modules

[Alessandro Franceschi]

Wow, this looks promising: sane, plain and easy to use. 
Going to test it soon.
Looking at things in perspective, how do you think this approach will go 
along with the one implemented in Puppet 3.3 ?

al

On Sunday, December 8, 2013 11:19:37 PM UTC+1, R.I. Pienaar wrote:
>
>
>
> - Original Message - 
> > From: "R.I.Pienaar" > 
> > To: puppet...@googlegroups.com  
> > Sent: Tuesday, October 22, 2013 3:46:45 PM 
> > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > 
> > 
> > 
> > - Original Message - 
> > > From: "jcbollinger"  
> > > To: puppet...@googlegroups.com  
> > > Sent: Tuesday, October 22, 2013 3:13:14 PM 
> > > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > > 
> > > 
> > > 
> > > On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote: 
> > > > 
> > > > Another round of thanks for the replies to this thread. I apologize 
> that 
> > > > almost as soon as I posted it, I got pulled off onto another project 
> and 
> > > > wasn't able to follow up until now. Replies inline below, and there 
> are 
> > > > probably a couple more coming to different branches (damn I miss 
> Usenet 
> > > > threading!) 
> > > > 
> > > > John Bollinger wrote: 
> > > >> > We agree on most of what you said, but it doesn't seem very 
> responsive 
> > > >> to 
> > > >> > the comments to which they ostensibly reply.  I am in no way 
> arguing 
> > > >> > against the idea of the data in modules subsystem.  It is a 
> fantastic 
> > > >> idea, 
> > > >> > and long past due.  I *am* concerned, however, about the new 
> approach 
> > > >> Eric 
> > > >> > proposed.  I suggested a more general approach than (my 
> understanding 
> > > >> of) 
> > > >> > the one he described, one not tied specifically to ::params 
> classes. 
> > > >> > Inasmuch as you disfavor ::params classes, I would think that you 
> > > >> > would 
> > > >> > find much to like about my counterproposal.  Indeed, I think my 
> > > >> proposal is 
> > > >> > very much like the original prototype you floated. 
> > > > 
> > > > 
> > > > John I didn't see a more detailed description of what you're 
> proposing; 
> > > > is 
> > > > this section (quoted from upthread) what you're referring to? 
> > > > 
> > > 
> > > Yes. 
> > >   
> > > 
> > > > 
> > > > Do I understand correctly that you set out to get rid of the 
> ::params 
> > > >> class pattern, but now you favor an approach that depends on that 
> > > >> pattern? 
> > > > 
> > > > 
> > > > Heh, well when you put it that way... 
> > > >   
> > > > 
> > > 
> > > 
> > > Let's also keep in mind that the purpose of the ::params class pattern 
> is 
> > > not really to serve as a per-module general data repository.  Rather, 
> it is 
> > > specifically to provide a means for indirection of class parameter 
> > > defaults.  To the extent that ::params classes now do serve as data 
> > > repositories, it is -- or should be -- in service to that purpose, not 
> to a 
> > > broader one.  Data in modules is a *complementary*, but more general, 
> > > approach whereby default values expressed in DSL code can in some 
> cases be 
> > > replaced by default values drawn from per-module data.  Where data are 
> > > consumed by a module in other ways or for other purposes, there is no 
> > > particular reason why a ::params class should be involved. 
> > > 
> > >   
> > > 
> > > > Why is that better than being more general: enable an implicit 
> > > >> lowest-priority hierarchy level for values of form 
> > > >> 'modulename::variable', 
> > > >> drawing on data from per-module data files such as 
> > > >> modules/modulename/data.yaml? 
> > > > 
> > > > 
> > > > If I understand this correctly this is slightly different (and 
> probably 
> > > > inadequate from RI's standpoint), because it just adds another 
> 'category' 
> > > > (in the ARM-9 sense) to the end of each lookup, an

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "R.I.Pienaar" 
> To: puppet-users@googlegroups.com
> Sent: Tuesday, October 22, 2013 3:46:45 PM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> 
> 
> - Original Message -
> > From: "jcbollinger" 
> > To: puppet-users@googlegroups.com
> > Sent: Tuesday, October 22, 2013 3:13:14 PM
> > Subject: Re: [Puppet Users] Re: Status of Data in modules
> > 
> > 
> > 
> > On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:
> > >
> > > Another round of thanks for the replies to this thread. I apologize that
> > > almost as soon as I posted it, I got pulled off onto another project and
> > > wasn't able to follow up until now. Replies inline below, and there are
> > > probably a couple more coming to different branches (damn I miss Usenet
> > > threading!)
> > >
> > > John Bollinger wrote:
> > >> > We agree on most of what you said, but it doesn't seem very responsive
> > >> to
> > >> > the comments to which they ostensibly reply.  I am in no way arguing
> > >> > against the idea of the data in modules subsystem.  It is a fantastic
> > >> idea,
> > >> > and long past due.  I *am* concerned, however, about the new approach
> > >> Eric
> > >> > proposed.  I suggested a more general approach than (my understanding
> > >> of)
> > >> > the one he described, one not tied specifically to ::params classes.
> > >> > Inasmuch as you disfavor ::params classes, I would think that you
> > >> > would
> > >> > find much to like about my counterproposal.  Indeed, I think my
> > >> proposal is
> > >> > very much like the original prototype you floated.
> > >
> > >
> > > John I didn't see a more detailed description of what you're proposing;
> > > is
> > > this section (quoted from upthread) what you're referring to?
> > >
> > 
> > Yes.
> >  
> > 
> > >
> > > Do I understand correctly that you set out to get rid of the ::params
> > >> class pattern, but now you favor an approach that depends on that
> > >> pattern?
> > >
> > >
> > > Heh, well when you put it that way...
> > >  
> > >
> > 
> > 
> > Let's also keep in mind that the purpose of the ::params class pattern is
> > not really to serve as a per-module general data repository.  Rather, it is
> > specifically to provide a means for indirection of class parameter
> > defaults.  To the extent that ::params classes now do serve as data
> > repositories, it is -- or should be -- in service to that purpose, not to a
> > broader one.  Data in modules is a *complementary*, but more general,
> > approach whereby default values expressed in DSL code can in some cases be
> > replaced by default values drawn from per-module data.  Where data are
> > consumed by a module in other ways or for other purposes, there is no
> > particular reason why a ::params class should be involved.
> > 
> >  
> > 
> > > Why is that better than being more general: enable an implicit
> > >> lowest-priority hierarchy level for values of form
> > >> 'modulename::variable',
> > >> drawing on data from per-module data files such as
> > >> modules/modulename/data.yaml?
> > >
> > >
> > > If I understand this correctly this is slightly different (and probably
> > > inadequate from RI's standpoint), because it just adds another 'category'
> > > (in the ARM-9 sense) to the end of each lookup, and what RI and others
> > > propose is to have another _complete hiera invocation_ inside the module
> > > owning a class parameter's namespace the end of each unsuccessful
> > > site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
> > > defined, and a tree of data files. (params.pp does this by letting
> > > old-school puppet DSL logic determine your "hierarchy")
> > >
> > >
> > 
> > I don't have any particular objection to implementing data-in-modules as a
> > separate full-fledged lookup against a per-module fallback hierarchy, but
> > the qualitative differences from what I suggested are subtle.  For the most
> > part, I think it's just a question of how many levels you can or do add to
> > the bottom of the logical hierarchy, whe

[Puppet Users] Re: Status of Data in modules

[Jon Shanks]

Hey, 

Just to jump in at the end, (been following the thread) and looked at the 
implementatino of data in modules, but found that the complexity 
surrounding it was a bit much for people who were not experienced. Also, 
troubleshooting issues with data, i.e. some form of outcome of a puppet run 
that didn't match what was expected, created an extra overhead in trying to 
identify where the issue lied. We also really only want to return data for 
modules which are included on a host.

I therefore wrote a wrapper function which essentially utilises the 
hierarchy but with an addition

 - modules/%{module_name}/%{klass}

It retreives the classes from the API, using the node indirection.

%{klass} gets defined in the scope as it iterates, (ephemeral_from) and 
merges the data it finds, so, if a module as a dependency on another module 
i.e. 'foreman' has sudo rules it requires, you could place those within 
modules/sudo/foreman.yaml so that you are containing data relating to a 
module, without managing different hierarchical configurations throught 
your module structure. 

For defaults, we defined that as modules/%{module_name}/defaults, which is 
the last in the hierarchy. 

This probably would haev been nicer if the data sources could utlises the 
returned classes in the hierarchy and naturally iterate over the array, to 
save the wrapper functinoaity to do it. 

Obviously this wouldn't solve all the problems faced with modules or the 
principles of the forge modules being usable easily out of the box, but it 
solved the issue we had in categorising data within modules. Thought i 
would mention our use case. 

Jon


On Friday, 11 October 2013 19:09:23 UTC+1, Eric Sorenson wrote:
>
>
> Thanks to everyone who kicked the tires on the experimental data in 
> modules feature included in Puppet 3.3.0. We got a lot of feedback, some 
> cool proof-of-concept modules, and a definitive conclusion to the 
> experiment. 
>
> The idea of including a module-specific hiera backend is centered around 
> one primary use case: replacing the 'params class pattern', a common idiom 
> in Puppet modules that's described in the [Using Parameterized 
> Classes][param-classes] guide. The problem that most testers ran into 
> though is that for non-trivial modules they ended up having to re-implement 
> the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious 
> ways. The solutions to this led to more contortions until we'd ended up 
> with the ability to execute parser functions in the right-hand-side of a 
> yaml value. So something which started out trying to help separate data 
> from code ended up putting code back into data! 
>
> Additionally, even after multiple attempts to simplify the surface area 
> and user experience with the bindings system (described in ARM-9) that 
> underlay the data-in-modules implementation, users still found its 
> complexity daunting. There are some important bits of scaffolding (like an 
> actual type system for Puppet!) that will prove valuable as more of the 
> future parser and evaluator work that Henrik is building makes its way into 
> the product, but in the final analysis the data in modules feature was the 
> wrong vehicle to introduce them. 
>
> Refocusing on the problems users were trying to solve (and here I have to 
> give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the 
> dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] 
> module) and the problems with the 'params' pattern lent some clarity. We've 
> gotten into a situation of disparity with regard to hiera and data 
> bindings, because data bindings enable module _users_ to use their 
> site-wide hiera data but don't provide moduel _authors_ the same 
> affordance. But rather than introduce additional complexity, we can close 
> the gap for existing code patterns. 
>
> So the proposed solution at this point is: 
> - enable an implicit data-binding lookup against the hiera-puppet backend 
> for a value of 'classname::variable' in the file 
> 'modules/classname/manifests/params.pp', which simplifies class definition 
> and provides consistency with other hiera backends. As a module author, 
> you'd still leave your logic for variables in params.pp, but they'd be 
> implicitly looked up via data bindings as the class is declared, after 
> consulting site-wide hiera. 
> - remove the user-facing '--binder' functionality 
> - fix known problems with the hiera-puppet lookups ([Redmine 
> 15746][15746], namely, but if there are others that are important to you 
> please speak up!) 
>
> To show how this would work, I'll rework the ['smart parameter defaults' 
> example][param-classes] I linked above, with my commentary behind `##` 
> comments: 
>
> # /etc/puppet/modules/webserver/manifests/params.pp 
> 
> class webserver::params {   ## nothing changes here... 
>  $packages = $operatingsystem ? { 
>/(?i-mx:ubuntu|debian)/=> 'apache2', 
>  

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "jcbollinger" 
> To: puppet-users@googlegroups.com
> Sent: Tuesday, October 22, 2013 3:13:14 PM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> 
> 
> On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:
> >
> > Another round of thanks for the replies to this thread. I apologize that
> > almost as soon as I posted it, I got pulled off onto another project and
> > wasn't able to follow up until now. Replies inline below, and there are
> > probably a couple more coming to different branches (damn I miss Usenet
> > threading!)
> >
> > John Bollinger wrote:
> >> > We agree on most of what you said, but it doesn't seem very responsive
> >> to
> >> > the comments to which they ostensibly reply.  I am in no way arguing
> >> > against the idea of the data in modules subsystem.  It is a fantastic
> >> idea,
> >> > and long past due.  I *am* concerned, however, about the new approach
> >> Eric
> >> > proposed.  I suggested a more general approach than (my understanding
> >> of)
> >> > the one he described, one not tied specifically to ::params classes.
> >> > Inasmuch as you disfavor ::params classes, I would think that you would
> >> > find much to like about my counterproposal.  Indeed, I think my
> >> proposal is
> >> > very much like the original prototype you floated.
> >
> >
> > John I didn't see a more detailed description of what you're proposing; is
> > this section (quoted from upthread) what you're referring to?
> >
> 
> Yes.
>  
> 
> >
> > Do I understand correctly that you set out to get rid of the ::params
> >> class pattern, but now you favor an approach that depends on that pattern?
> >
> >
> > Heh, well when you put it that way...
> >  
> >
> 
> 
> Let's also keep in mind that the purpose of the ::params class pattern is
> not really to serve as a per-module general data repository.  Rather, it is
> specifically to provide a means for indirection of class parameter
> defaults.  To the extent that ::params classes now do serve as data
> repositories, it is -- or should be -- in service to that purpose, not to a
> broader one.  Data in modules is a *complementary*, but more general,
> approach whereby default values expressed in DSL code can in some cases be
> replaced by default values drawn from per-module data.  Where data are
> consumed by a module in other ways or for other purposes, there is no
> particular reason why a ::params class should be involved.
> 
>  
> 
> > Why is that better than being more general: enable an implicit
> >> lowest-priority hierarchy level for values of form 'modulename::variable',
> >> drawing on data from per-module data files such as
> >> modules/modulename/data.yaml?
> >
> >
> > If I understand this correctly this is slightly different (and probably
> > inadequate from RI's standpoint), because it just adds another 'category'
> > (in the ARM-9 sense) to the end of each lookup, and what RI and others
> > propose is to have another _complete hiera invocation_ inside the module
> > owning a class parameter's namespace the end of each unsuccessful
> > site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
> > defined, and a tree of data files. (params.pp does this by letting
> > old-school puppet DSL logic determine your "hierarchy")
> >
> >
> 
> I don't have any particular objection to implementing data-in-modules as a
> separate full-fledged lookup against a per-module fallback hierarchy, but
> the qualitative differences from what I suggested are subtle.  For the most
> part, I think it's just a question of how many levels you can or do add to
> the bottom of the logical hierarchy, whether it's implemented via one call
> to the hiera subsystem or two.  There is a difference, however, in the
> behavior of lookups that collected data from across the hierarchy, i.e.
> hiera_hash() and hiera_array().  Those aren't relevant to class parameter
> binding (at this point), but it is worth considering what semantics are
> wanted there, and whether there might be a way for the caller to choose.
> 
>  
> 
> > I also talked to a user today who wants data from modules (by doing hash
> > key merge on a parameter's class::subclass::varname) from *any* module in
> > the modulepath to contribute, say, sudoers rules to the sudo module from
> > other site-written modu

Re: [Puppet Users] Re: Status of Data in modules

[jcbollinger]

On Monday, October 21, 2013 8:14:59 PM UTC-5, Eric Sorenson wrote:
>
> Another round of thanks for the replies to this thread. I apologize that 
> almost as soon as I posted it, I got pulled off onto another project and 
> wasn't able to follow up until now. Replies inline below, and there are 
> probably a couple more coming to different branches (damn I miss Usenet 
> threading!)
>
> John Bollinger wrote:
>> > We agree on most of what you said, but it doesn't seem very responsive 
>> to 
>> > the comments to which they ostensibly reply.  I am in no way arguing 
>> > against the idea of the data in modules subsystem.  It is a fantastic 
>> idea, 
>> > and long past due.  I *am* concerned, however, about the new approach 
>> Eric 
>> > proposed.  I suggested a more general approach than (my understanding 
>> of) 
>> > the one he described, one not tied specifically to ::params classes. 
>> > Inasmuch as you disfavor ::params classes, I would think that you would 
>> > find much to like about my counterproposal.  Indeed, I think my 
>> proposal is 
>> > very much like the original prototype you floated.
>
>
> John I didn't see a more detailed description of what you're proposing; is 
> this section (quoted from upthread) what you're referring to?
>

Yes.
 

>
> Do I understand correctly that you set out to get rid of the ::params 
>> class pattern, but now you favor an approach that depends on that pattern?  
>
>
> Heh, well when you put it that way...
>  
>


Let's also keep in mind that the purpose of the ::params class pattern is 
not really to serve as a per-module general data repository.  Rather, it is 
specifically to provide a means for indirection of class parameter 
defaults.  To the extent that ::params classes now do serve as data 
repositories, it is -- or should be -- in service to that purpose, not to a 
broader one.  Data in modules is a *complementary*, but more general, 
approach whereby default values expressed in DSL code can in some cases be 
replaced by default values drawn from per-module data.  Where data are 
consumed by a module in other ways or for other purposes, there is no 
particular reason why a ::params class should be involved.

 

> Why is that better than being more general: enable an implicit 
>> lowest-priority hierarchy level for values of form 'modulename::variable', 
>> drawing on data from per-module data files such as 
>> modules/modulename/data.yaml?
>
>
> If I understand this correctly this is slightly different (and probably 
> inadequate from RI's standpoint), because it just adds another 'category' 
> (in the ARM-9 sense) to the end of each lookup, and what RI and others 
> propose is to have another _complete hiera invocation_ inside the module 
> owning a class parameter's namespace the end of each unsuccessful 
> site-hiera lookup. Separate hiera.yaml config file with its own hierarchy 
> defined, and a tree of data files. (params.pp does this by letting 
> old-school puppet DSL logic determine your "hierarchy")
>
>

I don't have any particular objection to implementing data-in-modules as a 
separate full-fledged lookup against a per-module fallback hierarchy, but 
the qualitative differences from what I suggested are subtle.  For the most 
part, I think it's just a question of how many levels you can or do add to 
the bottom of the logical hierarchy, whether it's implemented via one call 
to the hiera subsystem or two.  There is a difference, however, in the 
behavior of lookups that collected data from across the hierarchy, i.e. 
hiera_hash() and hiera_array().  Those aren't relevant to class parameter 
binding (at this point), but it is worth considering what semantics are 
wanted there, and whether there might be a way for the caller to choose.

 

> I also talked to a user today who wants data from modules (by doing hash 
> key merge on a parameter's class::subclass::varname) from *any* module in 
> the modulepath to contribute, say, sudoers rules to the sudo module from 
> other site-written modules that require particular sudoers stanzas. So I'm 
> trying to consider how to pull all of this together without making a O(n^n) 
> complexity explosion.
>


I'm with R.I. in suggesting that you get something solid and fundamentally 
sound out soon, even if it doesn't address every user request on the first 
go (or ever).  I understand how a confederated data source such as you now 
describe could be useful, but I think such a feature would require a 
significant effort in its own right.

Furthermore, I think you are fast approaching the point where the data 
subsystem cannot automagically do the right thing in every case.  I don't 
think it would be a sin to require some features to be explicitly declared 
or invoked by DSL code.  For example, perhaps you want a data access 
function that allows the caller to somehow specify the scope of the data to 
search.  Maybe a couple of releases down the line.


John


-- 
You received this message because you are 

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "Eric Sorenson" 
> To: puppet-users@googlegroups.com
> Sent: Tuesday, October 22, 2013 2:14:59 AM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> Another round of thanks for the replies to this thread. I apologize that
> almost as soon as I posted it, I got pulled off onto another project and
> wasn't able to follow up until now. Replies inline below, and there are
> probably a couple more coming to different branches (damn I miss Usenet
> threading!)
> 
> John Bollinger wrote:
> > > We agree on most of what you said, but it doesn't seem very responsive
> > to
> > > the comments to which they ostensibly reply.  I am in no way arguing
> > > against the idea of the data in modules subsystem.  It is a fantastic
> > idea,
> > > and long past due.  I *am* concerned, however, about the new approach
> > Eric
> > > proposed.  I suggested a more general approach than (my understanding
> > of)
> > > the one he described, one not tied specifically to ::params classes.
> > > Inasmuch as you disfavor ::params classes, I would think that you would
> > > find much to like about my counterproposal.  Indeed, I think my proposal
> > is
> > > very much like the original prototype you floated.
> 
> 
> John I didn't see a more detailed description of what you're proposing; is
> this section (quoted from upthread) what you're referring to?
> 
> Do I understand correctly that you set out to get rid of the ::params class
> > pattern, but now you favor an approach that depends on that pattern?
> 
> 
> Heh, well when you put it that way...
>  
> 
> > Why is that better than being more general: enable an implicit
> > lowest-priority hierarchy level for values of form 'modulename::variable',
> > drawing on data from per-module data files such as
> > modules/modulename/data.yaml?
> 
> 
> If I understand this correctly this is slightly different (and probably
> inadequate from RI's standpoint), because it just adds another 'category'
> (in the ARM-9 sense) to the end of each lookup, and what RI and others
> propose is to have another _complete hiera invocation_ inside the module
> owning a class parameter's namespace the end of each unsuccessful
> site-hiera lookup. Separate hiera.yaml config file with its own hierarchy
> defined, and a tree of data files. (params.pp does this by letting
> old-school puppet DSL logic determine your "hierarchy")
> 
> I also talked to a user today who wants data from modules (by doing hash
> key merge on a parameter's class::subclass::varname) from *any* module in
> the modulepath to contribute, say, sudoers rules to the sudo module from
> other site-written modules that require particular sudoers stanzas. So I'm
> trying to consider how to pull all of this together without making a O(n^n)
> complexity explosion.
>  
> 
> >
> > RI replied:
> > Your comments are good and addressed in later replies, especially related
> > to
> > data mangling.  This is a common problem in all languages, data almost
> > never
> > arrives in the final form and all programming languages have patterns for
> > retrieving data, validating and mangling it.  We just need to introduce
> > similar patterns.
> >
> 
> This is really interesting, and not something that's come up so far
> AFAIK. It ties in somewhat to https://projects.puppetlabs.com/issues/20199
>  , needing a way to indicate the data type of something that's looked up
> implicitly with data bindings, but introduces another layer around
> retrieving and modifying data as it flows back towards puppet, which I
> hadn't considered.  That is what the "code-in-data" people are asking for,
> like https://github.com/puppetlabs/hiera/pull/152 that ended up with
> arbitrary puppet functions inside hiera curly brace expansion.  Would love
> thoughts on how to do that in a generally useful, lightweight way.
>  
> 
> > I, obviously, share your concern with the current round of proposals.
> >  Data
> > in module querying only params.pp is literally the worst possible
> > suggestion
> > one can make in this regard.  It would be a massive step backward. Might
> > as
> > well just go ahead and unmerge hiera if the goal is to not learn anything
> > from
> > its design and incredibly wide adoption.
> >
> 
> Oh surely there's way worse suggestions out there  :)
>  
> 
> > > I do think it is a mistake to focus on eliminating all need for ::params
> > > classes as a 

Re: [Puppet Users] Re: Status of Data in modules

[Eric Sorenson]

Another round of thanks for the replies to this thread. I apologize that 
almost as soon as I posted it, I got pulled off onto another project and 
wasn't able to follow up until now. Replies inline below, and there are 
probably a couple more coming to different branches (damn I miss Usenet 
threading!)

John Bollinger wrote:
> > We agree on most of what you said, but it doesn't seem very responsive 
> to 
> > the comments to which they ostensibly reply.  I am in no way arguing 
> > against the idea of the data in modules subsystem.  It is a fantastic 
> idea, 
> > and long past due.  I *am* concerned, however, about the new approach 
> Eric 
> > proposed.  I suggested a more general approach than (my understanding 
> of) 
> > the one he described, one not tied specifically to ::params classes. 
> > Inasmuch as you disfavor ::params classes, I would think that you would 
> > find much to like about my counterproposal.  Indeed, I think my proposal 
> is 
> > very much like the original prototype you floated.


John I didn't see a more detailed description of what you're proposing; is 
this section (quoted from upthread) what you're referring to?

Do I understand correctly that you set out to get rid of the ::params class 
> pattern, but now you favor an approach that depends on that pattern?  


Heh, well when you put it that way...
 

> Why is that better than being more general: enable an implicit 
> lowest-priority hierarchy level for values of form 'modulename::variable', 
> drawing on data from per-module data files such as 
> modules/modulename/data.yaml?


If I understand this correctly this is slightly different (and probably 
inadequate from RI's standpoint), because it just adds another 'category' 
(in the ARM-9 sense) to the end of each lookup, and what RI and others 
propose is to have another _complete hiera invocation_ inside the module 
owning a class parameter's namespace the end of each unsuccessful 
site-hiera lookup. Separate hiera.yaml config file with its own hierarchy 
defined, and a tree of data files. (params.pp does this by letting 
old-school puppet DSL logic determine your "hierarchy")

I also talked to a user today who wants data from modules (by doing hash 
key merge on a parameter's class::subclass::varname) from *any* module in 
the modulepath to contribute, say, sudoers rules to the sudo module from 
other site-written modules that require particular sudoers stanzas. So I'm 
trying to consider how to pull all of this together without making a O(n^n) 
complexity explosion.
 

>
> RI replied:
> Your comments are good and addressed in later replies, especially related 
> to 
> data mangling.  This is a common problem in all languages, data almost 
> never 
> arrives in the final form and all programming languages have patterns for 
> retrieving data, validating and mangling it.  We just need to introduce 
> similar patterns.   
>

This is really interesting, and not something that's come up so far 
AFAIK. It ties in somewhat to https://projects.puppetlabs.com/issues/20199 
 , needing a way to indicate the data type of something that's looked up 
implicitly with data bindings, but introduces another layer around 
retrieving and modifying data as it flows back towards puppet, which I 
hadn't considered.  That is what the "code-in-data" people are asking for, 
like https://github.com/puppetlabs/hiera/pull/152 that ended up with 
arbitrary puppet functions inside hiera curly brace expansion.  Would love 
thoughts on how to do that in a generally useful, lightweight way.
 

> I, obviously, share your concern with the current round of proposals. 
>  Data 
> in module querying only params.pp is literally the worst possible 
> suggestion 
> one can make in this regard.  It would be a massive step backward. Might 
> as 
> well just go ahead and unmerge hiera if the goal is to not learn anything 
> from 
> its design and incredibly wide adoption. 
>

Oh surely there's way worse suggestions out there  :)
 

> > I do think it is a mistake to focus on eliminating all need for ::params 
> > classes as a goal of the initiative, however.  Likely *most* need for 
> them 
> > can be redirected to a relatively simple data-in-modules subsystem, and 
> > that would be well, but the initiative does not fail if some need for 
> the 
> > ::params class pattern remains. 
>
 

> yeah, as per the other replies - eliminate *storing data* in params.pp but 
> validate/mangle in something like params.pp.  That is in the event that 
> no-one delivers a layer of data validation around data bindings and hiera. 
>
>
So it doesn't seem helpful to get data-bindings integration via puppet 
code, even as a first step?

I definitely agree hiera data in general needs a way to do validation, but 
the semantics you described of "requiring an integer between 10 and 20" 
would be additional complexity on top of Henrik's type system. (That work 
was foundational BTW, not specific to the data-in-modules binder as you 
said 

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "jcbollinger" 
> To: puppet-users@googlegroups.com
> Sent: Wednesday, October 16, 2013 4:23:17 PM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> 
> 
> On Tuesday, October 15, 2013 8:35:00 AM UTC-5, R.I. Pienaar wrote:
> >
> >
> >
> > - Original Message -
> > > From: "jcbollinger"
> > > To: puppet...@googlegroups.com 
> > > Sent: Tuesday, October 15, 2013 2:08:10 PM
> > > Subject: Re: [Puppet Users] Re: Status of Data in modules
> > > 
> > > 
> > > 
> > > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
> > > > 
> > > > On 14.10.2013 16:16, jcbollinger wrote:
> > > > > So the proposed solution at this point is:
> > > > > - enable an implicit data-binding lookup against the
> > hiera-puppet
> > > > > backend for a value of 'classname::variable' in the file
> > > > > 'modules/classname/manifests/params.pp', which simplifies class
> > > > > definition and provides consistency with other hiera backends.
> > As a
> > > > > module author, you'd still leave your logic for variables in
> > > > > params.pp, but they'd be implicitly looked up via data bindings
> > as
> > > > > the class is declared, after consulting site-wide hiera.
> > > > > 
> > > > > 
> > > > > 
> > > > > Do I understand correctly that you set out to get rid of the
> > ::params
> > > > > class pattern, but now you favor an approach that depends on that
> > > > > pattern?  Why is that better than being more general: enable an
> > implicit
> > > > > lowest-priority hierarchy level for values of form
> > > > > 'modulename::variable', drawing on data from per-module data files
> > such
> > > > > as modules/modulename/data.yaml?
> > > > 
> > > > AIUI, prototyping revealed that many params classes contain logic that
> > > > cannot be expressed in .yaml to calculate default values. Making the
> > > > fallback lookup check the params class, the logic can be preserved
> > while
> > > > avoiding verbose boilerplate code like the params_lookup calls in
> > > > Alessandro's modules.
> > > > 
> > > > 
> > > 
> > > I'm not saying that categorically getting rid of ::params classes is a
> > > viable target.  In fact, I don't really understand why it was ever an
> > > objective in the first place.  On the other hand, I don't see why it
> > makes
> > > sense for Puppet to give special significance to that pattern, either.
> >  A
> > > more general data-in-modules feature such as I describe would give users
> > > the option to avoid ::params classes in some cases, and I'm inclined to
> > > think that it would be easier to implement, to understand, and to use.
> > > 
> >
> > there are many reasons to avoid params.pp.  It's *code* not *data* and
> > it's
> >
> 
> 
> Eric observes -- and this comes as no surprise to me -- that the reason it
> didn't work well to eliminate params.pp was that sometimes you *need* code
> to help set appropriate default parameter values.  Or at least that is the
> practical conclusion that arose from field testing the original approach.
> I am all for pushing the actual data out of manifests, but you still need a
> place to hang code.  Pushing code out into the data to allow ::params
> classes to be dropped does not achieve the objective of separating data
> from code.
> 
>  
> 
> > one file that tends to include data for many different roles/sources/uses.
> >
> 
> 
> That's a fair observation, though a bit usage-oriented.  Nothing prevents
> users from creating a separate class::params class for every single class,
> which would at least partially address that.  In any case, the proposed new
> data in modules approach does nothing to address that concern -- if
> anything, it magnifies that problem.  If the objective is still to avoid
> params.pp, then why does the revised data in modules proposal provide data *
> only* to ::params classes?
> 
>  
> 
> >
> > You have to consider the main reasons for separating data from code in
> > order
> > to understand the motivation.
> >
> > When you have a params.pp you end up with stuff like this:
> >
> >  

Re: [Puppet Users] Re: Status of Data in modules

[jcbollinger]

On Tuesday, October 15, 2013 8:35:00 AM UTC-5, R.I. Pienaar wrote:
>
>
>
> - Original Message - 
> > From: "jcbollinger"
> > To: puppet...@googlegroups.com  
> > Sent: Tuesday, October 15, 2013 2:08:10 PM 
> > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > 
> > 
> > 
> > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote: 
> > > 
> > > On 14.10.2013 16:16, jcbollinger wrote: 
> > > > So the proposed solution at this point is: 
> > > > - enable an implicit data-binding lookup against the 
> hiera-puppet 
> > > > backend for a value of 'classname::variable' in the file 
> > > > 'modules/classname/manifests/params.pp', which simplifies class 
> > > > definition and provides consistency with other hiera backends. 
> As a 
> > > > module author, you'd still leave your logic for variables in 
> > > > params.pp, but they'd be implicitly looked up via data bindings 
> as 
> > > > the class is declared, after consulting site-wide hiera. 
> > > > 
> > > > 
> > > > 
> > > > Do I understand correctly that you set out to get rid of the 
> ::params 
> > > > class pattern, but now you favor an approach that depends on that 
> > > > pattern?  Why is that better than being more general: enable an 
> implicit 
> > > > lowest-priority hierarchy level for values of form 
> > > > 'modulename::variable', drawing on data from per-module data files 
> such 
> > > > as modules/modulename/data.yaml? 
> > > 
> > > AIUI, prototyping revealed that many params classes contain logic that 
> > > cannot be expressed in .yaml to calculate default values. Making the 
> > > fallback lookup check the params class, the logic can be preserved 
> while 
> > > avoiding verbose boilerplate code like the params_lookup calls in 
> > > Alessandro's modules. 
> > > 
> > > 
> > 
> > I'm not saying that categorically getting rid of ::params classes is a 
> > viable target.  In fact, I don't really understand why it was ever an 
> > objective in the first place.  On the other hand, I don't see why it 
> makes 
> > sense for Puppet to give special significance to that pattern, either. 
>  A 
> > more general data-in-modules feature such as I describe would give users 
> > the option to avoid ::params classes in some cases, and I'm inclined to 
> > think that it would be easier to implement, to understand, and to use. 
> > 
>
> there are many reasons to avoid params.pp.  It's *code* not *data* and 
> it's 
>


Eric observes -- and this comes as no surprise to me -- that the reason it 
didn't work well to eliminate params.pp was that sometimes you *need* code 
to help set appropriate default parameter values.  Or at least that is the 
practical conclusion that arose from field testing the original approach.  
I am all for pushing the actual data out of manifests, but you still need a 
place to hang code.  Pushing code out into the data to allow ::params 
classes to be dropped does not achieve the objective of separating data 
from code.

 

> one file that tends to include data for many different roles/sources/uses. 
>


That's a fair observation, though a bit usage-oriented.  Nothing prevents 
users from creating a separate class::params class for every single class, 
which would at least partially address that.  In any case, the proposed new 
data in modules approach does nothing to address that concern -- if 
anything, it magnifies that problem.  If the objective is still to avoid 
params.pp, then why does the revised data in modules proposal provide data *
only* to ::params classes?

 

>
> You have to consider the main reasons for separating data from code in 
> order 
> to understand the motivation. 
>
> When you have a params.pp you end up with stuff like this: 
>
>
> https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
>  
>
>

That sort of thing is not really the issue.  I think we pretty much all 
agree that pushing that out to an external data source is a win.  The issue 
is with cases where the rules for choosing parameter defaults are more 
complex than a simple switch based on one fact.  As I understand Eric, 
field testing showed that users wanted to be able to perform essentially 
arbitrary logic to *compute* the data provided to modules.  I agree with 
his conclusion that rolling such logic into the data is as bad as putting 
the data into ::params classes

Re: [Puppet Users] Re: Status of Data in modules

[Alessandro Franceschi]

Il giorno mercoledì 16 ottobre 2013 12:14:12 UTC+2, R.I. Pienaar ha scritto:
>
>
>
> - Original Message - 
> > From: "Alessandro Franceschi" > 
> > To: puppet...@googlegroups.com  
> > Sent: Wednesday, October 16, 2013 10:54:14 AM 
> > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > 
> > 
> > 
> > On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote: 
> > > 
> > > 
> > > 
> > > - Original Message - 
> > > > From: "jcbollinger"  
> > > > To: puppet...@googlegroups.com  
> > > > Sent: Tuesday, October 15, 2013 2:08:10 PM 
> > > > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > > > 
> > > > 
> > > > 
> > > > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote: 
> > > > > 
> > > > > On 14.10.2013 16:16, jcbollinger wrote: 
> > > > > > So the proposed solution at this point is: 
> > > > > > - enable an implicit data-binding lookup against the 
> > > hiera-puppet 
> > > > > > backend for a value of 'classname::variable' in the file 
> > > > > > 'modules/classname/manifests/params.pp', which simplifies 
> class 
> > > > > > definition and provides consistency with other hiera 
> backends. 
> > > As a 
> > > > > > module author, you'd still leave your logic for variables in 
> > > > > > params.pp, but they'd be implicitly looked up via data 
> bindings 
> > > as 
> > > > > > the class is declared, after consulting site-wide hiera. 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Do I understand correctly that you set out to get rid of the 
> > > ::params 
> > > > > > class pattern, but now you favor an approach that depends on 
> that 
> > > > > > pattern?  Why is that better than being more general: enable an 
> > > implicit 
> > > > > > lowest-priority hierarchy level for values of form 
> > > > > > 'modulename::variable', drawing on data from per-module data 
> files 
> > > such 
> > > > > > as modules/modulename/data.yaml? 
> > > > > 
> > > > > AIUI, prototyping revealed that many params classes contain logic 
> that 
> > > > > cannot be expressed in .yaml to calculate default values. Making 
> the 
> > > > > fallback lookup check the params class, the logic can be preserved 
> > > while 
> > > > > avoiding verbose boilerplate code like the params_lookup calls in 
> > > > > Alessandro's modules. 
> > > > > 
> > > > > 
> > > > 
> > > > I'm not saying that categorically getting rid of ::params classes is 
> a 
> > > > viable target.  In fact, I don't really understand why it was ever 
> an 
> > > > objective in the first place.  On the other hand, I don't see why it 
> > > makes 
> > > > sense for Puppet to give special significance to that pattern, 
> either. 
> > >  A 
> > > > more general data-in-modules feature such as I describe would give 
> users 
> > > > the option to avoid ::params classes in some cases, and I'm inclined 
> to 
> > > > think that it would be easier to implement, to understand, and to 
> use. 
> > > > 
> > > 
> > > there are many reasons to avoid params.pp.  It's *code* not *data* and 
> > > it's 
> > > one file that tends to include data for many different 
> roles/sources/uses. 
> > > 
> > > You have to consider the main reasons for separating data from code in 
> > > order 
> > > to understand the motivation. 
> > > 
> > > When you have a params.pp you end up with stuff like this: 
> > > 
> > > 
> > > 
> https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
>  
> > > 
> > > No-one would call that maintainable or readable vs say having 
> AIX.json, 
> > > Debian.json and so forth. 
> > > 
> > > For a community member who wants to add support for a new OS this 
> > > simplifies things a LOT. They can see what operating systems are 
> supported 
> > > already and they can easily add a new one 

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "R.I.Pienaar" 
> To: puppet-users@googlegroups.com
> Sent: Wednesday, October 16, 2013 11:14:12 AM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> 
> 
> - Original Message -
> > From: "Alessandro Franceschi" 
> > To: puppet-users@googlegroups.com
> > Sent: Wednesday, October 16, 2013 10:54:14 AM
> > Subject: Re: [Puppet Users] Re: Status of Data in modules
> > 
> > 
> > 
> > On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:
> > >
> > >
> > >
> > > - Original Message -
> > > > From: "jcbollinger" 
> > > > To: puppet...@googlegroups.com 
> > > > Sent: Tuesday, October 15, 2013 2:08:10 PM
> > > > Subject: Re: [Puppet Users] Re: Status of Data in modules
> > > > 
> > > > 
> > > > 
> > > > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
> > > > > 
> > > > > On 14.10.2013 16:16, jcbollinger wrote:
> > > > > > So the proposed solution at this point is:
> > > > > > - enable an implicit data-binding lookup against the
> > > hiera-puppet
> > > > > > backend for a value of 'classname::variable' in the file
> > > > > > 'modules/classname/manifests/params.pp', which simplifies class
> > > > > > definition and provides consistency with other hiera backends.
> > > As a
> > > > > > module author, you'd still leave your logic for variables in
> > > > > > params.pp, but they'd be implicitly looked up via data bindings
> > > as
> > > > > > the class is declared, after consulting site-wide hiera.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Do I understand correctly that you set out to get rid of the
> > > ::params
> > > > > > class pattern, but now you favor an approach that depends on that
> > > > > > pattern?  Why is that better than being more general: enable an
> > > implicit
> > > > > > lowest-priority hierarchy level for values of form
> > > > > > 'modulename::variable', drawing on data from per-module data files
> > > such
> > > > > > as modules/modulename/data.yaml?
> > > > > 
> > > > > AIUI, prototyping revealed that many params classes contain logic
> > > > > that
> > > > > cannot be expressed in .yaml to calculate default values. Making the
> > > > > fallback lookup check the params class, the logic can be preserved
> > > while
> > > > > avoiding verbose boilerplate code like the params_lookup calls in
> > > > > Alessandro's modules.
> > > > > 
> > > > > 
> > > > 
> > > > I'm not saying that categorically getting rid of ::params classes is a
> > > > viable target.  In fact, I don't really understand why it was ever an
> > > > objective in the first place.  On the other hand, I don't see why it
> > > makes
> > > > sense for Puppet to give special significance to that pattern, either.
> > >  A
> > > > more general data-in-modules feature such as I describe would give
> > > > users
> > > > the option to avoid ::params classes in some cases, and I'm inclined to
> > > > think that it would be easier to implement, to understand, and to use.
> > > > 
> > >
> > > there are many reasons to avoid params.pp.  It's *code* not *data* and
> > > it's
> > > one file that tends to include data for many different
> > > roles/sources/uses.
> > >
> > > You have to consider the main reasons for separating data from code in
> > > order
> > > to understand the motivation.
> > >
> > > When you have a params.pp you end up with stuff like this:
> > >
> > >
> > > https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
> > >
> > > No-one would call that maintainable or readable vs say having AIX.json,
> > > Debian.json and so forth.
> > >
> > > For a community member who wants to add support for a new OS this
> > > simplifies things a LOT. They can see what operating systems are
> > > supported
> > > alre

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "Alessandro Franceschi" 
> To: puppet-users@googlegroups.com
> Sent: Wednesday, October 16, 2013 10:54:14 AM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> 
> 
> On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:
> >
> >
> >
> > - Original Message -
> > > From: "jcbollinger" 
> > > To: puppet...@googlegroups.com 
> > > Sent: Tuesday, October 15, 2013 2:08:10 PM
> > > Subject: Re: [Puppet Users] Re: Status of Data in modules
> > > 
> > > 
> > > 
> > > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
> > > > 
> > > > On 14.10.2013 16:16, jcbollinger wrote:
> > > > > So the proposed solution at this point is:
> > > > > - enable an implicit data-binding lookup against the
> > hiera-puppet
> > > > > backend for a value of 'classname::variable' in the file
> > > > > 'modules/classname/manifests/params.pp', which simplifies class
> > > > > definition and provides consistency with other hiera backends.
> > As a
> > > > > module author, you'd still leave your logic for variables in
> > > > > params.pp, but they'd be implicitly looked up via data bindings
> > as
> > > > > the class is declared, after consulting site-wide hiera.
> > > > > 
> > > > > 
> > > > > 
> > > > > Do I understand correctly that you set out to get rid of the
> > ::params
> > > > > class pattern, but now you favor an approach that depends on that
> > > > > pattern?  Why is that better than being more general: enable an
> > implicit
> > > > > lowest-priority hierarchy level for values of form
> > > > > 'modulename::variable', drawing on data from per-module data files
> > such
> > > > > as modules/modulename/data.yaml?
> > > > 
> > > > AIUI, prototyping revealed that many params classes contain logic that
> > > > cannot be expressed in .yaml to calculate default values. Making the
> > > > fallback lookup check the params class, the logic can be preserved
> > while
> > > > avoiding verbose boilerplate code like the params_lookup calls in
> > > > Alessandro's modules.
> > > > 
> > > > 
> > > 
> > > I'm not saying that categorically getting rid of ::params classes is a
> > > viable target.  In fact, I don't really understand why it was ever an
> > > objective in the first place.  On the other hand, I don't see why it
> > makes
> > > sense for Puppet to give special significance to that pattern, either.
> >  A
> > > more general data-in-modules feature such as I describe would give users
> > > the option to avoid ::params classes in some cases, and I'm inclined to
> > > think that it would be easier to implement, to understand, and to use.
> > > 
> >
> > there are many reasons to avoid params.pp.  It's *code* not *data* and
> > it's
> > one file that tends to include data for many different roles/sources/uses.
> >
> > You have to consider the main reasons for separating data from code in
> > order
> > to understand the motivation.
> >
> > When you have a params.pp you end up with stuff like this:
> >
> >
> > https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
> >
> > No-one would call that maintainable or readable vs say having AIX.json,
> > Debian.json and so forth.
> >
> > For a community member who wants to add support for a new OS this
> > simplifies things a LOT. They can see what operating systems are supported
> > already and they can easily add a new one by dropping a single file.
> >
> > This improves the contributor life cycle significantly:
> >
> >  * Adding FooOS support will not break existing supported OS support.
> >FooOS.json is only going to be read on FooOS machines.
> >  * They do not have to worry about complex merge conflicts on busy modules
> >such as the ones you'd find internal to large companies vs many team
> >members editing a single param.pp
> >  * There's no syntax and stuff to bother about, its pretty easy evaluate
> >the data and to pre/post commit check this stuff.  Contributor dont
> > have
> >to test extensively 

Re: [Puppet Users] Re: Status of Data in modules

[Alessandro Franceschi]

On Tuesday, October 15, 2013 3:35:00 PM UTC+2, R.I. Pienaar wrote:
>
>
>
> - Original Message - 
> > From: "jcbollinger"  
> > To: puppet...@googlegroups.com  
> > Sent: Tuesday, October 15, 2013 2:08:10 PM 
> > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > 
> > 
> > 
> > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote: 
> > > 
> > > On 14.10.2013 16:16, jcbollinger wrote: 
> > > > So the proposed solution at this point is: 
> > > > - enable an implicit data-binding lookup against the 
> hiera-puppet 
> > > > backend for a value of 'classname::variable' in the file 
> > > > 'modules/classname/manifests/params.pp', which simplifies class 
> > > > definition and provides consistency with other hiera backends. 
> As a 
> > > > module author, you'd still leave your logic for variables in 
> > > > params.pp, but they'd be implicitly looked up via data bindings 
> as 
> > > > the class is declared, after consulting site-wide hiera. 
> > > > 
> > > > 
> > > > 
> > > > Do I understand correctly that you set out to get rid of the 
> ::params 
> > > > class pattern, but now you favor an approach that depends on that 
> > > > pattern?  Why is that better than being more general: enable an 
> implicit 
> > > > lowest-priority hierarchy level for values of form 
> > > > 'modulename::variable', drawing on data from per-module data files 
> such 
> > > > as modules/modulename/data.yaml? 
> > > 
> > > AIUI, prototyping revealed that many params classes contain logic that 
> > > cannot be expressed in .yaml to calculate default values. Making the 
> > > fallback lookup check the params class, the logic can be preserved 
> while 
> > > avoiding verbose boilerplate code like the params_lookup calls in 
> > > Alessandro's modules. 
> > > 
> > > 
> > 
> > I'm not saying that categorically getting rid of ::params classes is a 
> > viable target.  In fact, I don't really understand why it was ever an 
> > objective in the first place.  On the other hand, I don't see why it 
> makes 
> > sense for Puppet to give special significance to that pattern, either. 
>  A 
> > more general data-in-modules feature such as I describe would give users 
> > the option to avoid ::params classes in some cases, and I'm inclined to 
> > think that it would be easier to implement, to understand, and to use. 
> > 
>
> there are many reasons to avoid params.pp.  It's *code* not *data* and 
> it's 
> one file that tends to include data for many different roles/sources/uses. 
>
> You have to consider the main reasons for separating data from code in 
> order 
> to understand the motivation. 
>
> When you have a params.pp you end up with stuff like this: 
>
>
> https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
>  
>
> No-one would call that maintainable or readable vs say having AIX.json, 
> Debian.json and so forth. 
>
> For a community member who wants to add support for a new OS this 
> simplifies things a LOT. They can see what operating systems are supported 
> already and they can easily add a new one by dropping a single file. 
>
> This improves the contributor life cycle significantly: 
>
>  * Adding FooOS support will not break existing supported OS support.   
>FooOS.json is only going to be read on FooOS machines. 
>  * They do not have to worry about complex merge conflicts on busy modules 
>such as the ones you'd find internal to large companies vs many team 
>members editing a single param.pp 
>  * There's no syntax and stuff to bother about, its pretty easy evaluate 
>the data and to pre/post commit check this stuff.  Contributor dont 
> have 
>to test extensively to ensure he didnt accidentally mess up params.pp 
>complex nested statements in some subtle manner 
>  * In large environments if you have strict change control etc, the 
> previous 
>points help things a lot, you can easily reason about the implications 
> and you 
>can be sure they wont affect existing systems.  It's just data that 
> will affect 
>a small subset users. 
>
> This improves the maintainers life because: 
>
>  * They can find it easier to merge new OS support because the change is 
> contained 
>in seperate files and easy to evaluate 
>

[Puppet Users] Re: Status of Data in modules

[John Julien]

On Friday, October 11, 2013 7:12:32 PM UTC-5, Chuck wrote:
>
> What would be nice but not necessary:
>
> 1) defined variable automatically have classname added to avoid global 
> conflicts.
>
>
> eg.
>
> Module: apache
>
>   variable: port
>
>   becomes global hiera:   apache::port
>
+1
Defining a variable in a module but having it's scope be global seems 
counter intuitive and could lead to conflicts.  If someone wants a topscope 
variable they should probably define it in a topscope location (ENC, 
Centralized Hiera Files, etc)

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Status of Data in modules

[John Julien]

On Tuesday, October 15, 2013 8:08:10 AM UTC-5, jcbollinger wrote:
>
>
>
> I'm not saying that categorically getting rid of ::params classes is a 
> viable target.  In fact, I don't really understand why it was ever an 
> objective in the first place.  On the other hand, I don't see why it makes 
> sense for Puppet to give special significance to that pattern, either.  A 
> more general data-in-modules feature such as I describe would give users 
> the option to avoid ::params classes in some cases, and I'm inclined to 
> think that it would be easier to implement, to understand, and to use.
>
>
> John
>
>
I agree with this. It seems the proposed new solution adds complexity and 
non-intuitive value lookups that could confuse users.  I think the 
data-in-modules feature sounds useful.  Perhaps as John suggests a change 
in scope is needed.  Instead of getting rid of params.pp the scope should 
be to move hierarchical data out of params.pp and leave variables derived 
from logic in params.pp.
 

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Status of Data in modules

[Chuck]

I totally agree with R.I. on this.

On Tuesday, October 15, 2013 8:35:00 AM UTC-5, R.I. Pienaar wrote:
>
>
>
> - Original Message - 
> > From: 
> > To: puppet...@googlegroups.com  
> > Sent: Tuesday, October 15, 2013 2:08:10 PM 
> > Subject: Re: [Puppet Users] Re: Status of Data in modules 
> > 
> > 
> > 
> > On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote: 
> > > 
> > > On 14.10.2013 16:16, jcbollinger wrote: 
> > > > So the proposed solution at this point is: 
> > > > - enable an implicit data-binding lookup against the 
> hiera-puppet 
> > > > backend for a value of 'classname::variable' in the file 
> > > > 'modules/classname/manifests/params.pp', which simplifies class 
> > > > definition and provides consistency with other hiera backends. 
> As a 
> > > > module author, you'd still leave your logic for variables in 
> > > > params.pp, but they'd be implicitly looked up via data bindings 
> as 
> > > > the class is declared, after consulting site-wide hiera. 
> > > > 
> > > > 
> > > > 
> > > > Do I understand correctly that you set out to get rid of the 
> ::params 
> > > > class pattern, but now you favor an approach that depends on that 
> > > > pattern?  Why is that better than being more general: enable an 
> implicit 
> > > > lowest-priority hierarchy level for values of form 
> > > > 'modulename::variable', drawing on data from per-module data files 
> such 
> > > > as modules/modulename/data.yaml? 
> > > 
> > > AIUI, prototyping revealed that many params classes contain logic that 
> > > cannot be expressed in .yaml to calculate default values. Making the 
> > > fallback lookup check the params class, the logic can be preserved 
> while 
> > > avoiding verbose boilerplate code like the params_lookup calls in 
> > > Alessandro's modules. 
> > > 
> > > 
> > 
> > I'm not saying that categorically getting rid of ::params classes is a 
> > viable target.  In fact, I don't really understand why it was ever an 
> > objective in the first place.  On the other hand, I don't see why it 
> makes 
> > sense for Puppet to give special significance to that pattern, either. 
>  A 
> > more general data-in-modules feature such as I describe would give users 
> > the option to avoid ::params classes in some cases, and I'm inclined to 
> > think that it would be easier to implement, to understand, and to use. 
> > 
>
> there are many reasons to avoid params.pp.  It's *code* not *data* and 
> it's 
> one file that tends to include data for many different roles/sources/uses. 
>
> You have to consider the main reasons for separating data from code in 
> order 
> to understand the motivation. 
>
> When you have a params.pp you end up with stuff like this: 
>
>
> https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140
>  
>
> No-one would call that maintainable or readable vs say having AIX.json, 
> Debian.json and so forth. 
>
> For a community member who wants to add support for a new OS this 
> simplifies things a LOT. They can see what operating systems are supported 
> already and they can easily add a new one by dropping a single file. 
>
> This improves the contributor life cycle significantly: 
>
>  * Adding FooOS support will not break existing supported OS support.   
>FooOS.json is only going to be read on FooOS machines. 
>  * They do not have to worry about complex merge conflicts on busy modules 
>such as the ones you'd find internal to large companies vs many team 
>members editing a single param.pp 
>  * There's no syntax and stuff to bother about, its pretty easy evaluate 
>the data and to pre/post commit check this stuff.  Contributor dont 
> have 
>to test extensively to ensure he didnt accidentally mess up params.pp 
>complex nested statements in some subtle manner 
>  * In large environments if you have strict change control etc, the 
> previous 
>points help things a lot, you can easily reason about the implications 
> and you 
>can be sure they wont affect existing systems.  It's just data that 
> will affect 
>a small subset users. 
>
> This improves the maintainers life because: 
>
>  * They can find it easier to merge new OS support because the change is 
> contained 
>in seperate files and easy to evaluat

Re: [Puppet Users] Re: Status of Data in modules

[R.I.Pienaar]

- Original Message -
> From: "jcbollinger" 
> To: puppet-users@googlegroups.com
> Sent: Tuesday, October 15, 2013 2:08:10 PM
> Subject: Re: [Puppet Users] Re: Status of Data in modules
> 
> 
> 
> On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
> >
> > On 14.10.2013 16:16, jcbollinger wrote:
> > > So the proposed solution at this point is:
> > > - enable an implicit data-binding lookup against the hiera-puppet
> > > backend for a value of 'classname::variable' in the file
> > > 'modules/classname/manifests/params.pp', which simplifies class
> > > definition and provides consistency with other hiera backends. As a
> > > module author, you'd still leave your logic for variables in
> > > params.pp, but they'd be implicitly looked up via data bindings as
> > > the class is declared, after consulting site-wide hiera.
> > > 
> > > 
> > > 
> > > Do I understand correctly that you set out to get rid of the ::params
> > > class pattern, but now you favor an approach that depends on that
> > > pattern?  Why is that better than being more general: enable an implicit
> > > lowest-priority hierarchy level for values of form
> > > 'modulename::variable', drawing on data from per-module data files such
> > > as modules/modulename/data.yaml?
> >
> > AIUI, prototyping revealed that many params classes contain logic that
> > cannot be expressed in .yaml to calculate default values. Making the
> > fallback lookup check the params class, the logic can be preserved while
> > avoiding verbose boilerplate code like the params_lookup calls in
> > Alessandro's modules.
> >
> >
> 
> I'm not saying that categorically getting rid of ::params classes is a
> viable target.  In fact, I don't really understand why it was ever an
> objective in the first place.  On the other hand, I don't see why it makes
> sense for Puppet to give special significance to that pattern, either.  A
> more general data-in-modules feature such as I describe would give users
> the option to avoid ::params classes in some cases, and I'm inclined to
> think that it would be easier to implement, to understand, and to use.
> 

there are many reasons to avoid params.pp.  It's *code* not *data* and it's
one file that tends to include data for many different roles/sources/uses.

You have to consider the main reasons for separating data from code in order
to understand the motivation.

When you have a params.pp you end up with stuff like this:

   
https://github.com/puppetlabs/puppetlabs-ntp/blob/master/manifests/params.pp#L28-140

No-one would call that maintainable or readable vs say having AIX.json, 
Debian.json and so forth.

For a community member who wants to add support for a new OS this
simplifies things a LOT. They can see what operating systems are supported 
already and they can easily add a new one by dropping a single file.

This improves the contributor life cycle significantly:

 * Adding FooOS support will not break existing supported OS support.  
   FooOS.json is only going to be read on FooOS machines.
 * They do not have to worry about complex merge conflicts on busy modules
   such as the ones you'd find internal to large companies vs many team
   members editing a single param.pp
 * There's no syntax and stuff to bother about, its pretty easy evaluate 
   the data and to pre/post commit check this stuff.  Contributor dont have 
   to test extensively to ensure he didnt accidentally mess up params.pp
   complex nested statements in some subtle manner
 * In large environments if you have strict change control etc, the previous 
   points help things a lot, you can easily reason about the implications and 
you 
   can be sure they wont affect existing systems.  It's just data that will 
affect 
   a small subset users.

This improves the maintainers life because:

 * They can find it easier to merge new OS support because the change is 
contained 
   in seperate files and easy to evaluate
 * Fewer complex merge commits and easier, cleaner, commit history
 * The code is simpler and generally easier to maintain in the long term

This improves the module user life because:

 * He can just look at existing data files and know without having to parse 
complex 
   nested case statements what the available overridable data is and what the 
keys
   would be etc.

There are more but these are the basics

>From a Puppet system perspective its important that this feature behave 
>consistently
and predictably with the current more or less universally accepted standard of 
data
seperation - hiera.  And hiera is all about data, the hiera

Re: [Puppet Users] Re: Status of Data in modules

[jcbollinger]

On Tuesday, October 15, 2013 12:52:39 AM UTC-5, David Schmitt wrote:
>
> On 14.10.2013 16:16, jcbollinger wrote: 
> > So the proposed solution at this point is: 
> > - enable an implicit data-binding lookup against the hiera-puppet 
> > backend for a value of 'classname::variable' in the file 
> > 'modules/classname/manifests/params.pp', which simplifies class 
> > definition and provides consistency with other hiera backends. As a 
> > module author, you'd still leave your logic for variables in 
> > params.pp, but they'd be implicitly looked up via data bindings as 
> > the class is declared, after consulting site-wide hiera. 
> > 
> > 
> > 
> > Do I understand correctly that you set out to get rid of the ::params 
> > class pattern, but now you favor an approach that depends on that 
> > pattern?  Why is that better than being more general: enable an implicit 
> > lowest-priority hierarchy level for values of form 
> > 'modulename::variable', drawing on data from per-module data files such 
> > as modules/modulename/data.yaml? 
>
> AIUI, prototyping revealed that many params classes contain logic that 
> cannot be expressed in .yaml to calculate default values. Making the 
> fallback lookup check the params class, the logic can be preserved while 
> avoiding verbose boilerplate code like the params_lookup calls in 
> Alessandro's modules. 
>
>

I'm not saying that categorically getting rid of ::params classes is a 
viable target.  In fact, I don't really understand why it was ever an 
objective in the first place.  On the other hand, I don't see why it makes 
sense for Puppet to give special significance to that pattern, either.  A 
more general data-in-modules feature such as I describe would give users 
the option to avoid ::params classes in some cases, and I'm inclined to 
think that it would be easier to implement, to understand, and to use.


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Status of Data in modules

[David Schmitt]

On 14.10.2013 16:16, jcbollinger wrote:

So the proposed solution at this point is:
- enable an implicit data-binding lookup against the hiera-puppet
backend for a value of 'classname::variable' in the file
'modules/classname/manifests/params.pp', which simplifies class
definition and provides consistency with other hiera backends. As a
module author, you'd still leave your logic for variables in
params.pp, but they'd be implicitly looked up via data bindings as
the class is declared, after consulting site-wide hiera.



Do I understand correctly that you set out to get rid of the ::params
class pattern, but now you favor an approach that depends on that
pattern?  Why is that better than being more general: enable an implicit
lowest-priority hierarchy level for values of form
'modulename::variable', drawing on data from per-module data files such
as modules/modulename/data.yaml?


AIUI, prototyping revealed that many params classes contain logic that 
cannot be expressed in .yaml to calculate default values. Making the 
fallback lookup check the params class, the logic can be preserved while 
avoiding verbose boilerplate code like the params_lookup calls in 
Alessandro's modules.



Regards, David

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: Status of Data in modules

[jcbollinger]

On Friday, October 11, 2013 1:09:23 PM UTC-5, Eric Sorenson wrote:
>
>
> Thanks to everyone who kicked the tires on the experimental data in 
> modules feature included in Puppet 3.3.0. We got a lot of feedback, some 
> cool proof-of-concept modules, and a definitive conclusion to the 
> experiment. 
>
> The idea of including a module-specific hiera backend is centered around 
> one primary use case: replacing the 'params class pattern', a common idiom 
> in Puppet modules that's described in the [Using Parameterized 
> Classes][param-classes] guide.



I guess I wasn't following this closely enough to realize that getting rid 
of the "params" class pattern was an objective.  I thought this was a 
somewhat more general initiative.

[...]

So the proposed solution at this point is: 
> - enable an implicit data-binding lookup against the hiera-puppet backend 
> for a value of 'classname::variable' in the file 
> 'modules/classname/manifests/params.pp', which simplifies class definition 
> and provides consistency with other hiera backends. As a module author, 
> you'd still leave your logic for variables in params.pp, but they'd be 
> implicitly looked up via data bindings as the class is declared, after 
> consulting site-wide hiera. 
>


Do I understand correctly that you set out to get rid of the ::params class 
pattern, but now you favor an approach that depends on that pattern?  Why 
is that better than being more general: enable an implicit lowest-priority 
hierarchy level for values of form 'modulename::variable', drawing on data 
from per-module data files such as modules/modulename/data.yaml?

 
John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: Status of Data in modules

[Chuck]

 

I see hiera data in modules just being a very useful extension of the 
current hiera implementation.

Why

1)  separate out the variables my dev teams use into the modules so that it 
is easier to package their changes.  Promoting their module includes all of 
their heir data.

2)  In an enterprise environment this allows other modules to access that 
data when the modules are not totally independent.  Eg.. I have a 
apache_module_v1 and a apache_business_unit_v2 that uses the apache module.

3)  access controls by adding hiera into modules as opposed to a main hiera 
directory structure.  The developers just need access to their modules. 

4)  We version our modules so that they call all be active at the same time 
and the old modules only have maintenance updates.  Hiera in modules means 
the central hiera is not touched by the developers.

eg.  apache_module_v1

   apache_module_v2

   apache_module_v3


What would be nice but not necessary:

1) defined variable automatically have classname added to avoid global 
conflicts.


eg.

Module: apache

  variable: port

  becomes global hiera:   apache::port



On Friday, October 11, 2013 6:26:09 PM UTC-5, Chuck wrote:
>
>
> I see the best aspect of data in modules is that it allows the clear 
> separation of variables per module in hiera.  This is important because 
> module developers don't need access to a central global hiera that is 
> "static".  For our use we need to break variables down by environment and 
> datacenter, hiera is great for this.  And data in modules created a very 
> clear scope of control for module authors.  We were basically treating data 
> in modules as a distributed hiera v1 which works really well.  The only 
> addition that would have been nice is the adding of the classname in front 
> of all variables as you are proposing for your params.pp.
>
> Managing a "central" hiera directory structure can be painful when you 
> have 20 - 50 developers that need to create hiera variables.  I really 
> think data in modules helps this considerably.  I agree that about moving 
> logic into hiera is not beneficial as you just end up with more code.
>
>
>
> On Friday, October 11, 2013 1:09:23 PM UTC-5, Eric Sorenson wrote:
>>
>>
>> Thanks to everyone who kicked the tires on the experimental data in 
>> modules feature included in Puppet 3.3.0. We got a lot of feedback, some 
>> cool proof-of-concept modules, and a definitive conclusion to the 
>> experiment. 
>>
>> The idea of including a module-specific hiera backend is centered around 
>> one primary use case: replacing the 'params class pattern', a common idiom 
>> in Puppet modules that's described in the [Using Parameterized 
>> Classes][param-classes] guide. The problem that most testers ran into 
>> though is that for non-trivial modules they ended up having to re-implement 
>> the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious 
>> ways. The solutions to this led to more contortions until we'd ended up 
>> with the ability to execute parser functions in the right-hand-side of a 
>> yaml value. So something which started out trying to help separate data 
>> from code ended up putting code back into data!  
>>
> Additionally, even after multiple attempts to simplify the surface area 
>> and user experience with the bindings system (described in ARM-9) that 
>> underlay the data-in-modules implementation, users still found its 
>> complexity daunting. There are some important bits of scaffolding (like an 
>> actual type system for Puppet!) that will prove valuable as more of the 
>> future parser and evaluator work that Henrik is building makes its way into 
>> the product, but in the final analysis the data in modules feature was the 
>> wrong vehicle to introduce them. 
>>
>> Refocusing on the problems users were trying to solve (and here I have to 
>> give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the 
>> dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] 
>> module) and the problems with the 'params' pattern lent some clarity. We've 
>> gotten into a situation of disparity with regard to hiera and data 
>> bindings, because data bindings enable module _users_ to use their 
>> site-wide hiera data but don't provide moduel _authors_ the same 
>> affordance. But rather than introduce additional complexity, we can close 
>> the gap for existing code patterns. 
>>
>> So the proposed solution at this point is: 
>> - enable an implicit data-binding lookup against the hiera-puppet backend 
>> for a value of 'classname::variable' in the file 
>> 'modules/classname/manifests/params.pp', which simplifies class definition 
>> and provides consistency with other hiera backends. As a module author, 
>> you'd still leave your logic for variables in params.pp, but they'd be 
>> implicitly looked up via data bindings as the class is declared, after 
>> consulting site-wide hiera. 
>> - remove th

[Puppet Users] Re: Status of Data in modules

[Chuck]

I see the best aspect of data in modules is that it allows the clear 
separation of variables per module in hiera.  This is important because 
module developers don't need access to a central global hiera that is 
"static".  For our use we need to break variables down by environment and 
datacenter, hiera is great for this.  And data in modules created a very 
clear scope of control for module authors.  We were basically treating data 
in modules as a distributed hiera v1 which works really well.  The only 
addition that would have been nice is the adding of the classname in front 
of all variables as you are proposing for your params.pp.

Managing a "central" hiera directory structure can be painful when you have 
20 - 50 developers that need to create hiera variables.  I really think 
data in modules helps this considerably.  I agree that about moving logic 
into hiera is not beneficial as you just end up with more code.



On Friday, October 11, 2013 1:09:23 PM UTC-5, Eric Sorenson wrote:
>
>
> Thanks to everyone who kicked the tires on the experimental data in 
> modules feature included in Puppet 3.3.0. We got a lot of feedback, some 
> cool proof-of-concept modules, and a definitive conclusion to the 
> experiment. 
>
> The idea of including a module-specific hiera backend is centered around 
> one primary use case: replacing the 'params class pattern', a common idiom 
> in Puppet modules that's described in the [Using Parameterized 
> Classes][param-classes] guide. The problem that most testers ran into 
> though is that for non-trivial modules they ended up having to re-implement 
> the Puppet DSL logic encoded in their params.pp in convoluted, non-obvious 
> ways. The solutions to this led to more contortions until we'd ended up 
> with the ability to execute parser functions in the right-hand-side of a 
> yaml value. So something which started out trying to help separate data 
> from code ended up putting code back into data!  
>
Additionally, even after multiple attempts to simplify the surface area and 
> user experience with the bindings system (described in ARM-9) that underlay 
> the data-in-modules implementation, users still found its complexity 
> daunting. There are some important bits of scaffolding (like an actual type 
> system for Puppet!) that will prove valuable as more of the future parser 
> and evaluator work that Henrik is building makes its way into the product, 
> but in the final analysis the data in modules feature was the wrong vehicle 
> to introduce them. 
>
> Refocusing on the problems users were trying to solve (and here I have to 
> give shout-outs to Ashley Penney for his [puppetlabs-ntp][] branch and the 
> dynamic duo of Spencer Krug/William van Hevelingen for their [startrek][] 
> module) and the problems with the 'params' pattern lent some clarity. We've 
> gotten into a situation of disparity with regard to hiera and data 
> bindings, because data bindings enable module _users_ to use their 
> site-wide hiera data but don't provide moduel _authors_ the same 
> affordance. But rather than introduce additional complexity, we can close 
> the gap for existing code patterns. 
>
> So the proposed solution at this point is: 
> - enable an implicit data-binding lookup against the hiera-puppet backend 
> for a value of 'classname::variable' in the file 
> 'modules/classname/manifests/params.pp', which simplifies class definition 
> and provides consistency with other hiera backends. As a module author, 
> you'd still leave your logic for variables in params.pp, but they'd be 
> implicitly looked up via data bindings as the class is declared, after 
> consulting site-wide hiera. 
> - remove the user-facing '--binder' functionality 
> - fix known problems with the hiera-puppet lookups ([Redmine 
> 15746][15746], namely, but if there are others that are important to you 
> please speak up!) 
>
> To show how this would work, I'll rework the ['smart parameter defaults' 
> example][param-classes] I linked above, with my commentary behind `##` 
> comments: 
>
> # /etc/puppet/modules/webserver/manifests/params.pp 
> 
> class webserver::params {   ## nothing changes here... 
>  $packages = $operatingsystem ? { 
>/(?i-mx:ubuntu|debian)/=> 'apache2', 
>/(?i-mx:centos|fedora|redhat)/ => 'httpd', 
>  } 
>  $vhost_dir = $operatingsystem ? { 
>/(?i-mx:ubuntu|debian)/=> '/etc/apache2/sites-enabled', 
>/(?i-mx:centos|fedora|redhat)/ => '/etc/httpd/conf.d', 
>  } 
> } 
> 
> # /etc/puppet/modules/webserver/manifests/init.pp 
> 
> class webserver(  ## inheritance is gone, and 
>  $packages,   ## data bindings look up the defaults 
>  $vhost_dir   ## as webserver::params::vhost_dir 
> ) { 
> 
>  package { $packages: ensure => present } 
> 
>  file { 'vhost_dir': 
>path   => $vhost_dir, 
>ensure => directory, 
>mode   => '0750', 
>