Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Felix Frank]

On 05/07/2014 07:04 PM, Alex Scoble wrote:
> All in all, I love the product, don't get me wrong, and maybe it looks
> pretty stable when you're at the guru level and figuring out new stuff
> is fairly easy, but for me, it's just me and a coworker trying to mature
> our linux and puppet infrastructure, processes and workflows. Neither of
> us are developers, so are learning that side of things as best as we can
> while we learn Puppet, git, hiera, Geppetto (which is the bee's knees,
> just be super careful when you try to merge two significantly different
> branches together using it), beaker, rspec, ruby and so on.

Thanks for the in-debt write-up, Alex.

So what I took from it is that, for all the documentation and buzzing
blog-sphere, it can be daunting to pick the patterns, solutions and tool
chain that are best fitted for one's particular needs and goals. Is that
a fair summary?

I can get behind that, I guess, but I have a hard time coming up with a
real good strategy to handle this. Yes, it's a rich ecosystems, and that
has its own downsides. Perhaps the Learning Puppet course should
culminate in the ultimate advice to join the list and ask for hints
regarding one's own goals and environment?

But then such threads run an elevated risk of starving, I feel, because
some new users tend to ask quite broadly with too little to go on for
the senior members to take interest in resolving the challenge.



Felix

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/5379EFF2.7020409%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

And this is aside from the significant resistance we've been getting from 
our traditional Windows admins over using Puppet, which is a problem that 
Puppet definitely can't solve. Their partnership with Microsoft is a very 
good thing, in my opinion though.

Oh, and Chocolatey. :D

--Alex

On Wednesday, May 7, 2014 10:04:23 AM UTC-7, Alex Scoble wrote:
>
> I'm sorry, I misspoke, I should have said that for us Puppet (PE actually) 
> has been a moving target in a number of ways.
>
> For instance, we started out heavily using the PE dashboard to define what 
> classes specific nodes would get and related variables, pretty quickly 
> realized that that wasn't very scalable or repeatable and found hiera and 
> environments.
>
> Puppet does a decent job of explaining environments, but their 
> documentation in regards to integrating with git and gitolite is 
> inadequate, particularly considering that they strongly recommend going 
> down that path. Oh and while you are researching how to make dynamic 
> environments work, you invariably run into R10k which sounds really 
> awesome, but then you wonder how do you use it?
>
> Dynamic environments are great, extremely useful, but centrally managing 
> what nodes are in what environment wasn't possible until we discovered the 
> console_env module, which unfortunately isn't working for us after we 
> upgraded to PE 3.2 (and to make a point about how the product is a moving 
> target, they are currently working on a new way of assigning roles to 
> systems). Hiera is also great as it's much faster to work with than the 
> dashboard, but it's a lot easier to make mistakes with hiera. Also, for 
> many situations, you need a hiera yaml file for individual nodes (like when 
> you want a group of systems to use a specific class). So now there's the 
> roles and profiles pattern, which Puppet recommends using along with hiera, 
> but they don't provide good documentation at all on how to use both 
> together.
>
> Plus if you pay attention to R.I. Pienaar (and you definitely should, in 
> my opinion), you see that he's proposed a new pattern where you use hiera 
> data within modules, but getting that to work requires using one of the 
> modules that aren't yet in a finished state...
>
> I've talked to people who don't think that Puppet has handled the way 
> they've added features and patterns and deprecated other features and 
> patterns, but we haven't personally run in to that.
>
> All in all, I love the product, don't get me wrong, and maybe it looks 
> pretty stable when you're at the guru level and figuring out new stuff is 
> fairly easy, but for me, it's just me and a coworker trying to mature our 
> linux and puppet infrastructure, processes and workflows. Neither of us are 
> developers, so are learning that side of things as best as we can while we 
> learn Puppet, git, hiera, Geppetto (which is the bee's knees, just be super 
> careful when you try to merge two significantly different branches together 
> using it), beaker, rspec, ruby and so on.
>
> I'm always questioning our choice in tools because that's how you stay 
> ahead in this game and when I go looking for why people are using Salt 
> Stack and Ansible instead of Puppet or Chef, the number one reason I've 
> seen is complexity of the code needed to do something.
>
> Having said that, now that I've figured out a number of things and can 
> pretty much do anything I want with Puppet, albeit crudely (you can see my 
> kvm/webvirtmgr module as evidence of this), I don't see a reason to switch, 
> but I can certainly understand why some people may look at the DevOps space 
> and gravitate towards Salt Stack and Ansible over Puppet.
>
> Thanks,
>
> Alex
>
> On Wednesday, May 7, 2014 1:18:49 AM UTC-7, Felix.Frank wrote:
>>
>> On 05/07/2014 12:52 AM, Ramin K wrote: 
>> > 
>> > I find it best not to change my workflow or methodology until it 
>> > makes sense on my system regardless of what the community or even 
>> Puppet 
>> > Labs has said. 
>>
>> Ramin, I could hardly agree more. Even your ignored practices resemble 
>> my own personal choices very closely (those perhaps come rather natural 
>> to old schoolers). 
>>
>> If I understood Alex right though, he also feels that the apparent flux 
>> might be hindering broader acceptance of Puppet. If that is indeed the 
>> case, we have a problem that we should talk about. (Note that apparent 
>> stability is more important than the technicalities.) 
>>
>> However, it has been my feeling that general adoption is not one of 
>> Puppet's problems. On the contrary, Puppet users usually form the 
>> largest crowd in any kind of forum concerned with the configuration 
>> management problem. The user base keeps growing and the community is 
>> literally buzzing with activity. 
>>
>> Alex, are there concrete issues that you have faced concerning the ease 
>> of adoption? 
>>
>> Thanks, 
>> Felix 
>>
>

-- 
You received this message because you are subscribed 

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

I'm sorry, I misspoke, I should have said that for us Puppet (PE actually) 
has been a moving target in a number of ways.

For instance, we started out heavily using the PE dashboard to define what 
classes specific nodes would get and related variables, pretty quickly 
realized that that wasn't very scalable or repeatable and found hiera and 
environments.

Puppet does a decent job of explaining environments, but their 
documentation in regards to integrating with git and gitolite is 
inadequate, particularly considering that they strongly recommend going 
down that path. Oh and while you are researching how to make dynamic 
environments work, you invariably run into R10k which sounds really 
awesome, but then you wonder how do you use it?

Dynamic environments are great, extremely useful, but centrally managing 
what nodes are in what environment wasn't possible until we discovered the 
console_env module, which unfortunately isn't working for us after we 
upgraded to PE 3.2 (and to make a point about how the product is a moving 
target, they are currently working on a new way of assigning roles to 
systems). Hiera is also great as it's much faster to work with than the 
dashboard, but it's a lot easier to make mistakes with hiera. Also, for 
many situations, you need a hiera yaml file for individual nodes (like when 
you want a group of systems to use a specific class). So now there's the 
roles and profiles pattern, which Puppet recommends using along with hiera, 
but they don't provide good documentation at all on how to use both 
together.

Plus if you pay attention to R.I. Pienaar (and you definitely should, in my 
opinion), you see that he's proposed a new pattern where you use hiera data 
within modules, but getting that to work requires using one of the modules 
that aren't yet in a finished state...

I've talked to people who don't think that Puppet has handled the way 
they've added features and patterns and deprecated other features and 
patterns, but we haven't personally run in to that.

All in all, I love the product, don't get me wrong, and maybe it looks 
pretty stable when you're at the guru level and figuring out new stuff is 
fairly easy, but for me, it's just me and a coworker trying to mature our 
linux and puppet infrastructure, processes and workflows. Neither of us are 
developers, so are learning that side of things as best as we can while we 
learn Puppet, git, hiera, Geppetto (which is the bee's knees, just be super 
careful when you try to merge two significantly different branches together 
using it), beaker, rspec, ruby and so on.

I'm always questioning our choice in tools because that's how you stay 
ahead in this game and when I go looking for why people are using Salt 
Stack and Ansible instead of Puppet or Chef, the number one reason I've 
seen is complexity of the code needed to do something.

Having said that, now that I've figured out a number of things and can 
pretty much do anything I want with Puppet, albeit crudely (you can see my 
kvm/webvirtmgr module as evidence of this), I don't see a reason to switch, 
but I can certainly understand why some people may look at the DevOps space 
and gravitate towards Salt Stack and Ansible over Puppet.

Thanks,

Alex

On Wednesday, May 7, 2014 1:18:49 AM UTC-7, Felix.Frank wrote:
>
> On 05/07/2014 12:52 AM, Ramin K wrote: 
> > 
> > I find it best not to change my workflow or methodology until it 
> > makes sense on my system regardless of what the community or even Puppet 
> > Labs has said. 
>
> Ramin, I could hardly agree more. Even your ignored practices resemble 
> my own personal choices very closely (those perhaps come rather natural 
> to old schoolers). 
>
> If I understood Alex right though, he also feels that the apparent flux 
> might be hindering broader acceptance of Puppet. If that is indeed the 
> case, we have a problem that we should talk about. (Note that apparent 
> stability is more important than the technicalities.) 
>
> However, it has been my feeling that general adoption is not one of 
> Puppet's problems. On the contrary, Puppet users usually form the 
> largest crowd in any kind of forum concerned with the configuration 
> management problem. The user base keeps growing and the community is 
> literally buzzing with activity. 
>
> Alex, are there concrete issues that you have faced concerning the ease 
> of adoption? 
>
> Thanks, 
> Felix 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/f7a80605-ed1e-4e45-a5c0-188d58767339%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Felix Frank]

On 05/07/2014 12:52 AM, Ramin K wrote:
> 
> I find it best not to change my workflow or methodology until it
> makes sense on my system regardless of what the community or even Puppet
> Labs has said.

Ramin, I could hardly agree more. Even your ignored practices resemble
my own personal choices very closely (those perhaps come rather natural
to old schoolers).

If I understood Alex right though, he also feels that the apparent flux
might be hindering broader acceptance of Puppet. If that is indeed the
case, we have a problem that we should talk about. (Note that apparent
stability is more important than the technicalities.)

However, it has been my feeling that general adoption is not one of
Puppet's problems. On the contrary, Puppet users usually form the
largest crowd in any kind of forum concerned with the configuration
management problem. The user base keeps growing and the community is
literally buzzing with activity.

Alex, are there concrete issues that you have faced concerning the ease
of adoption?

Thanks,
Felix

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/5369EC69.3010304%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Ramin K]

On 5/5/2014 7:32 AM, Alex Scoble wrote:

If you are trying to follow Puppet Labs recommended best practices, it's
definitely a moving target.


	I find it best not to change my workflow or methodology until it makes 
sense on my system regardless of what the community or even Puppet Labs 
has said.


A few of the things I have completely ignored,
 - inheriting params.pp class to get local variables
 - ENC
 - parameterized classes
 - 99.8% of Forge modules
 - data bindings

A few of the things I do/have done that Puppet no longer recommends or 
has never recommended,

 - use stages
 - include subclasses to change behavior of modules, class 
postfix::config::relay inherits postfix::config {

 - specific hiera calls
 - used RVM to run my Puppet master

	On the flip side I did have a 2000+ line commit last year when I 
switched over to role/profile. However the massive benefits in 
organization (at least for me) were worth the time investment. At some 
point I'll probably bite the parameterized classes/data binding bullet 
as well.


	I ran across a paper recently with the following quote. I think keeping 
this definition of Engineering in mind would be a good thing for the 
Puppet community instead of searching for the elusive best solution.


http://www.lunduniversity.lu.se/o.o.i.s?id=24965&postid=2968043
"Engineering has no hint of the absolute, the deterministic, the 
guaranteed, the true. Instead it fairly reeks of the uncertain, the 
provisional and the doubtful. The engineer instinctively recognizes this 
and calls his ad hoc method “doing the best you can with what you’ve 
got,” “finding a seat-of-the-pants solution,” or just “muddling 
through”. (Koen, 1985, p. 23)."


Ramin

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/536967B4.7000303%40badapple.net.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[jcbollinger]

On Monday, May 5, 2014 9:32:05 AM UTC-5, Alex Scoble wrote:
>
> If you are trying to follow Puppet Labs recommended best practices, it's 
> definitely a moving target.
>
>

Obviously, "best practices" are not at all the same thing as the DSL itself.

Still, I'm curious what you define as "PuppetLabs recommended best 
practices".  PL has indeed at times published various best practices guides 
on their site, but the current one is so high-level as to hardly recommend 
anything close enough to the DSL as to lead anyone to confuse the two.  The 
guides they have published in the past have come and gone as Puppet has 
grown and experience with it has grown, both at PL and in the community.

PL is indeed a fast-moving company, though, and Puppet is a faster-moving 
product than many.  Nevertheless, Felix is right: the Puppet core is pretty 
stable.  The key to best practices, whether with Puppet or any other 
complex system, is to understand the system well enough to know why one 
practice is better than another.  That's something we can help you with 
here, though you will have to contend with a plurality of opinions.


Best,

John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/30a0ffe0-a86b-4464-9951-c05507588b6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

If you are trying to follow Puppet Labs recommended best practices, it's
definitely a moving target.


On Mon, May 5, 2014 at 5:37 AM, Felix Frank  wrote:

> On 04/30/2014 04:28 PM, Alex Scoble wrote:
> > I sometimes wonder how the moving target that is the Puppet DSL slows
> > adoption of the product.
>
> Well, I don't know what part of the discussion led you to the conclusion
> of the Puppet DSL being a "moving target", but I can confidently state
> that it is just not true.
>
> Yes, the language is seeing constant additions. This is inherently a
> good thing, because those strive to ever make the user's life easier.
> Such changes are usually opt-in, you don't have to use things you don't
> need.
>
> Sure, users face recurring issues that get iterated time and again on
> the list, because solving them is a less than trivial design problem.
> When those can eventually be solved, the offending parts may be taken
> out of the language (see Dynamic Scoping), but never without an
> appropriate deprecation phase. The dev team goes to great lengths to
> ensure that your manifest will not just break.
>
> As for the future parser, that is just an elegant way to allow people to
> preview the Puppet 4 language additions before its release (yes, Puppet
> 4 *will* have an enhanced DSL - I would argue against the view that
> turning a new page for a major release makes you a "moving target").
>
> I feel compelled to add - the future parser is a preview system. It is
> *not* something you should enable on a production master for a realistic
> field test. Production grade for this parser will be reached in Puppet 4.
>
> Regards,
> Felix
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/QNZyd4ipB0U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/536785FA.5030203%40alumni.tu-berlin.de
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAETmVfTTaC8zOauVdPKpQ3tiDHpuO6fVjNKmoKET6UXxb%2BpkLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Felix Frank]

On 04/30/2014 04:28 PM, Alex Scoble wrote:
> I sometimes wonder how the moving target that is the Puppet DSL slows
> adoption of the product.

Well, I don't know what part of the discussion led you to the conclusion
of the Puppet DSL being a "moving target", but I can confidently state
that it is just not true.

Yes, the language is seeing constant additions. This is inherently a
good thing, because those strive to ever make the user's life easier.
Such changes are usually opt-in, you don't have to use things you don't
need.

Sure, users face recurring issues that get iterated time and again on
the list, because solving them is a less than trivial design problem.
When those can eventually be solved, the offending parts may be taken
out of the language (see Dynamic Scoping), but never without an
appropriate deprecation phase. The dev team goes to great lengths to
ensure that your manifest will not just break.

As for the future parser, that is just an elegant way to allow people to
preview the Puppet 4 language additions before its release (yes, Puppet
4 *will* have an enhanced DSL - I would argue against the view that
turning a new page for a major release makes you a "moving target").

I feel compelled to add - the future parser is a preview system. It is
*not* something you should enable on a production master for a realistic
field test. Production grade for this parser will be reached in Puppet 4.

Regards,
Felix

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/536785FA.5030203%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[jcbollinger]

On Wednesday, April 30, 2014 9:23:49 AM UTC-5, Alex Scoble wrote:
>
> Hi John,
>
> Thanks for the info.
>
> By the way, I started using the inherit pattern with params because of the 
> myriad of Splunk modules out there that do it, including this one from 
> Puppet Labs https://github.com/puppetlabs/puppetlabs-splunk
>


Where that PL module uses class inheritance, it is for one or both of the 
two reasons I gave earlier.

Some of the classes inherit class splunk::params in order to use variables 
of that class as default values in their own parameter lists.  Inheriting 
splunk::params ensures that that class has already been evaluated when the 
child class is evaluated.  Where the child classes in fact refer to 
variables from splunk::params, they do so via their qualified names.

The classes in the 'platform' subpackage inherit splunk::virtual.  They do 
so in order to override parameters of resources declared by 
splunk::virtual.  As far as I can tell, they do not refer to any of 
splunk::virtual's class variables via unqualifed names.

So actually, that module provides a pretty good example and model of 
several of my above points about class inheritance.  It does have a few 
flaws, though -- most notably, it parameterizes class splunk::params that 
is intended to be inherited.  Notwithstanding my practical arguments, the 
language reference explicitly forbids inheriting from parameterized 
classes,
 
so the PL crew really should know better.

 

>
> I understand why the pattern is used, it makes it a bit easier to deal 
> with variables, particularly with hiera.
>


If that's what you're gleaning from the PL module then no, it seems that 
you don't understand why the pattern is used (there).

 

> I also understand what you are saying. This is the sort of thing that can 
> drive one nuts, heh. It would be nice if there was only one way to do 
> things and the one way was also the right way, but such nirvana is hard to 
> find.
>
>

I do not claim to be keeper of the One True Way, but I am fairly well 
informed about this community's current opinions about best practices for 
Puppet manifest code (and at times I have disagreed with aspects of them).  
You will certainly code things as you think most appropriate, but you 
should make the best-informed decisions you can about how that actually 
is.  Myself, I am not much swayed by the "Somebody else is doing it" 
rationale.


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/c7094feb-af21-4afd-9f14-a4df10c87636%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

Thanks, R.I.Pienaar,

I sometimes wonder how the moving target that is the Puppet DSL slows
adoption of the product.

I'll keep an eye on the future parser. Hopefully, it's in PE soon. Of
course, then I'll need to wait until someone creates a useful module that
shows how to use the pattern. I'm not quite smart enough (or knowledgeable
enough yet) to create a new pattern on my own.

Thanks,

Alex


On Wed, Apr 30, 2014 at 7:19 AM, R.I.Pienaar  wrote:

>
>
> - Original Message -
> > From: "jcbollinger" 
> > To: puppet-users@googlegroups.com
> > Sent: Wednesday, April 30, 2014 3:15:03 PM
> > Subject: [Puppet Users] Re: Looking for a better way to use hiera hashes
> than create_resources
> >
> >
> >
> > On Tuesday, April 29, 2014 11:37:02 AM UTC-5, Alex Scoble wrote:
> > >
> > > Hi John,
> > >
> > > Thanks so much for your feedback. It's extremely useful for me at this
> > > stage of my education in the Puppet DSL.
> > >
> > > Here is the Puppet Users group thread where R.I. Pienaar said that he
> felt
> > > that using create_resources() was bad:
> > >
> https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ
> > >
> >
> >
> > For the record, then, R.I. said "I dont tend to use create_resources as I
> > consider it about as bad as eval() :) ".  I hardly find that an emphatic
> > and categorical  condemnation.  In fact, I think that lines up relatively
> > well with my own position on the matter, expressed earlier in this
> thread.
>
> And I still think that, unfortunately there are often few other options.
>
> Today you should just suck it up and use the future parser and loop the
> data
> though, it seems to be speedy enough and relatively stable, syntax has been
> locked down etc.
>
> I've not followed this thread too closely but I'd hope the f.parser
> iterating
> your hashes/arrays should do all you need.  If not it's probably worth
> highlighting
> those.
>
> ie. the closer to reality the f.parser becomes the less you should be using
> create_resources.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/QNZyd4ipB0U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/1444896807.322.1398867543907.JavaMail.zimbra%40devco.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAETmVfRzC-Wd6vOzohynFj125X3Lp8b-L1p45J0yfTQsnCkCZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

Hi John,

Thanks for the info.

By the way, I started using the inherit pattern with params because of the
myriad of Splunk modules out there that do it, including this one from
Puppet Labs https://github.com/puppetlabs/puppetlabs-splunk

I understand why the pattern is used, it makes it a bit easier to deal with
variables, particularly with hiera. I also understand what you are saying.
This is the sort of thing that can drive one nuts, heh. It would be nice if
there was only one way to do things and the one way was also the right way,
but such nirvana is hard to find.

Thanks,

Alex


On Wed, Apr 30, 2014 at 7:15 AM, jcbollinger wrote:

>
>
> On Tuesday, April 29, 2014 11:37:02 AM UTC-5, Alex Scoble wrote:
>>
>> Hi John,
>>
>> Thanks so much for your feedback. It's extremely useful for me at this
>> stage of my education in the Puppet DSL.
>>
>> Here is the Puppet Users group thread where R.I. Pienaar said that he
>> felt that using create_resources() was bad: https://groups.google.
>> com/forum/?fromgroups#!searchin/puppet-users/create_
>> resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ
>>
>
>
> For the record, then, R.I. said "I dont tend to use create_resources as I
> consider it about as bad as eval() :) ".  I hardly find that an emphatic
> and categorical  condemnation.  In fact, I think that lines up relatively
> well with my own position on the matter, expressed earlier in this thread.
>
>
>
>> Responses to comments:
>>
>> 2. As far as I can see, I'm using class inheritance solely to make sure
>> that class parameters are available to the main and sub classes. Maybe you
>> are seeing that I'm using inheritance where it's not yet strictly needed?
>> I'll have to look more closely at my code and fix any instances of
>> inheritance where it's unnecessary. Thanks.
>>
>>
> I think I was looking primarily at class kvm::virtnet.  It is not
> parameterized itself and it does not contain any resource parameter
> overrides, so it cannot have any need to inherit from class kvm.  Moreover,
> class kvm is parameterized, so *nobody* should inherit from it.  It looks
> like you may be inheriting simply to avoid prefixing class kvm's variables
> with 'kvm::' where you reference them, but that's poor form even when you
> do use inheritance.
>
> If you are concerned about a possibility that class kvm::virtnet will be
> evaluated before class kvm, so that the references to kvm's variables are
> not resolved correctly, then 'include' class kvm instead of inheriting from
> it.  But that's dangerous and nearly pointless.  It is dangerous because if
> class kvm is declared via a parameterized-style declaration, and class
> kvm::virtnet really is evaluated first, then catalog compilation will fail
> -- as it would also do under the same circumstances with classes as you
> present.  It is nearly pointless because if you can rely on class kvm being
> evaluated first so that catalog compilation succeeds, then class
> kvm::virtnet doesn't need to inherit, include, or do anything else along
> those lines to work properly.
>
> The caveat is that if class kvm is declared via 'include' or (I think)
> hiera_include(), and class kvm::virtnet is also independently declared,
> then it is possible for class kvm::virtnet to be evaluated first without
> the catalog compiler crashing later.  One possible approach to that problem
> is basically just to say "don't do that".  If class kvm::virtnet is
> intended only for internal use by the module, as appears may be the case,
> then document that clearly and be done with it.  None of this is an issue
> with respect to class kvm::virtnet being declared by class kvm, by the way;
> if that's the only declaration of class kvm::virtnet then class kvm is
> certain to be evaluated first.
>
>
> John
>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/QNZyd4ipB0U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/35a23857-cc2c-445e-adb0-aa8ce404e760%40googlegroups.com
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAETmVfTQ-SNAksDYGpDEbZs_7wA0ka--z_%2Bc6v49fc5qHZwSaA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[R.I.Pienaar]

- Original Message -
> From: "jcbollinger" 
> To: puppet-users@googlegroups.com
> Sent: Wednesday, April 30, 2014 3:15:03 PM
> Subject: [Puppet Users] Re: Looking for a better way to use hiera hashes than 
> create_resources
> 
> 
> 
> On Tuesday, April 29, 2014 11:37:02 AM UTC-5, Alex Scoble wrote:
> >
> > Hi John,
> >
> > Thanks so much for your feedback. It's extremely useful for me at this
> > stage of my education in the Puppet DSL.
> >
> > Here is the Puppet Users group thread where R.I. Pienaar said that he felt
> > that using create_resources() was bad:
> > https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ
> >
> 
> 
> For the record, then, R.I. said "I dont tend to use create_resources as I
> consider it about as bad as eval() :) ".  I hardly find that an emphatic
> and categorical  condemnation.  In fact, I think that lines up relatively
> well with my own position on the matter, expressed earlier in this thread.

And I still think that, unfortunately there are often few other options.

Today you should just suck it up and use the future parser and loop the data
though, it seems to be speedy enough and relatively stable, syntax has been
locked down etc.

I've not followed this thread too closely but I'd hope the f.parser iterating
your hashes/arrays should do all you need.  If not it's probably worth 
highlighting
those.

ie. the closer to reality the f.parser becomes the less you should be using
create_resources.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/1444896807.322.1398867543907.JavaMail.zimbra%40devco.net.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[jcbollinger]

On Tuesday, April 29, 2014 11:37:02 AM UTC-5, Alex Scoble wrote:
>
> Hi John,
>
> Thanks so much for your feedback. It's extremely useful for me at this 
> stage of my education in the Puppet DSL.
>
> Here is the Puppet Users group thread where R.I. Pienaar said that he felt 
> that using create_resources() was bad: 
> https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ
>


For the record, then, R.I. said "I dont tend to use create_resources as I 
consider it about as bad as eval() :) ".  I hardly find that an emphatic 
and categorical  condemnation.  In fact, I think that lines up relatively 
well with my own position on the matter, expressed earlier in this thread.
 


> Responses to comments:
>
> 2. As far as I can see, I'm using class inheritance solely to make sure 
> that class parameters are available to the main and sub classes. Maybe you 
> are seeing that I'm using inheritance where it's not yet strictly needed? 
> I'll have to look more closely at my code and fix any instances of 
> inheritance where it's unnecessary. Thanks.
>
>
I think I was looking primarily at class kvm::virtnet.  It is not 
parameterized itself and it does not contain any resource parameter 
overrides, so it cannot have any need to inherit from class kvm.  Moreover, 
class kvm is parameterized, so *nobody* should inherit from it.  It looks 
like you may be inheriting simply to avoid prefixing class kvm's variables 
with 'kvm::' where you reference them, but that's poor form even when you 
do use inheritance.

If you are concerned about a possibility that class kvm::virtnet will be 
evaluated before class kvm, so that the references to kvm's variables are 
not resolved correctly, then 'include' class kvm instead of inheriting from 
it.  But that's dangerous and nearly pointless.  It is dangerous because if 
class kvm is declared via a parameterized-style declaration, and class 
kvm::virtnet really is evaluated first, then catalog compilation will fail 
-- as it would also do under the same circumstances with classes as you 
present.  It is nearly pointless because if you can rely on class kvm being 
evaluated first so that catalog compilation succeeds, then class 
kvm::virtnet doesn't need to inherit, include, or do anything else along 
those lines to work properly.

The caveat is that if class kvm is declared via 'include' or (I think) 
hiera_include(), and class kvm::virtnet is also independently declared, 
then it is possible for class kvm::virtnet to be evaluated first without 
the catalog compiler crashing later.  One possible approach to that problem 
is basically just to say "don't do that".  If class kvm::virtnet is 
intended only for internal use by the module, as appears may be the case, 
then document that clearly and be done with it.  None of this is an issue 
with respect to class kvm::virtnet being declared by class kvm, by the way; 
if that's the only declaration of class kvm::virtnet then class kvm is 
certain to be evaluated first.


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/35a23857-cc2c-445e-adb0-aa8ce404e760%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

Hi John,

Thanks so much for your feedback. It's extremely useful for me at this 
stage of my education in the Puppet DSL.

Here is the Puppet Users group thread where R.I. Pienaar said that he felt 
that using create_resources() was 
bad: 
https://groups.google.com/forum/?fromgroups#!searchin/puppet-users/create_resources$20pienaar/puppet-users/lxYDKf7dgc0/TppS_7BFB9IJ

Responses to comments:

1. That makes perfect sense. I'll do that from now on. Thanks.
2. As far as I can see, I'm using class inheritance solely to make sure 
that class parameters are available to the main and sub classes. Maybe you 
are seeing that I'm using inheritance where it's not yet strictly needed? 
I'll have to look more closely at my code and fix any instances of 
inheritance where it's unnecessary. Thanks.
3. You are right. I'm using the wrong kind of 'undef' here.
4. Yep, I'll have to look at the parameters and clean this up as well.

Thanks much!

Alex

On Tuesday, April 29, 2014 7:10:36 AM UTC-7, jcbollinger wrote:
>
>
>
> On Monday, April 28, 2014 2:34:43 PM UTC-5, Alex Scoble wrote:
>>
>> Hi All,
>>
>> I'm working on a module that builds KVM/libvirt hosts and populates them 
>> with predefined VMs.
>>
>> So far I have the module to where it can create any number of virtual 
>> nets, storage pools and volumes using virsh, but it isn't pretty.
>>
>> I've read on various threads here that create_resources is not a good 
>> function to use. This was stated quite emphatically by R.I. Pienaar and 
>> others.
>>
>
>
> If anyone has been vocal (here) about Puppet functions not to use then 
> it's me, and create_resources() is not one I rant about.  As a general rule 
> I do prefer expressing configuration in DSL where that's feasible, and you 
> can, in fact, do the job of create_resources() almost entirely with 
> ordinary Puppet DSL (you need to add only stdlib's keys() function).  
> Nevertheless, I'm not persuaded that going to substantial effort and length 
> to avoid create_resources() is necessarily a win.
>
> On the other hand, I have come out rather strongly against the 
> ensure_resource() function.  Were you perhaps thinking of that?
>
>  
>
>>
>> Already I've run into the situation where it's hard to control order in 
>> which two separate create_resources functions are run and I've seen some 
>> kludges to fix it, but I'm looking for a better way and hoping that it 
>> doesn't involve the use of custom types because that's currently more than 
>> I want to deal with.
>>
>>
>
> Indeed, one of the issues that does exist with create_resources() is that 
> you cannot express resource references in Hiera data (though I have seen 
> one recent report that claims otherwise).  When part of the point of using 
> that function in the first place is that you don't want to have to know 
> exactly which resources you are actually declaring, that can present a 
> challenge.  It is, nevertheless, a challenge that can be worked around.  In 
> some cases, the "workaround" is a mechanism, such as collector chaining,  
> that you might reasonably have used anyway.
>
> > It would be super helpful if anyone could point me to a puppet module on 
> github that presents me with a better pattern to use with the hiera data.
>
> [...]
>
> > Any other help or criticisms are also welcome.
>
>
> That looks pretty clean to me already.  It does appear that maybe you want 
> to express relationships between your Kvm::Virtpool and Kvm::Virtvol 
> instances, but Garrett appears already to have gotten you pointed in a 
> useful direction with that (collector chaining).
>
> If you want responses to particular commentary about the 
> create_resources() function then it would be helpful to provide a pointer 
> to that commentary.  R.I. is a pretty smart guy, and he has thought a lot 
> about how Puppet can and should use data, so I'm not remotely going to try 
> to respond to anything he may have said on the topic without actually 
> reading it for myself
>
> Additional, minor comments about your code:
>
>1. For any but the most simple Execs, I recommend using descriptive 
>resource titles, and expressing the commands themselves via Exec's 
>'command' property.  It is much, much easier that way to understand their 
>purpose, especially in log output.
>2. There are only two reasons for using class inheritance: to make a 
>variable of the inherited class reliably available for use as a class 
>parameter default, and to enable overriding properties of a resource 
>declared by the inherited class.  The latter is the original purpose, and 
>the former is merely a side effect, but the former is by far the more 
>common reason for class inheritance these days.  In all other cases, an 
>ordinary class declaration (include 'foo') can and should be used instead, 
>if indeed anything of the sort is needed at all.  Avoid class inheritance 
>where it is unneeded.  Especially avoid inheriting from a parameterized 
>  

[Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[jcbollinger]

On Monday, April 28, 2014 2:34:43 PM UTC-5, Alex Scoble wrote:
>
> Hi All,
>
> I'm working on a module that builds KVM/libvirt hosts and populates them 
> with predefined VMs.
>
> So far I have the module to where it can create any number of virtual 
> nets, storage pools and volumes using virsh, but it isn't pretty.
>
> I've read on various threads here that create_resources is not a good 
> function to use. This was stated quite emphatically by R.I. Pienaar and 
> others.
>


If anyone has been vocal (here) about Puppet functions not to use then it's 
me, and create_resources() is not one I rant about.  As a general rule I do 
prefer expressing configuration in DSL where that's feasible, and you can, 
in fact, do the job of create_resources() almost entirely with ordinary 
Puppet DSL (you need to add only stdlib's keys() function).  Nevertheless, 
I'm not persuaded that going to substantial effort and length to avoid 
create_resources() is necessarily a win.

On the other hand, I have come out rather strongly against the 
ensure_resource() function.  Were you perhaps thinking of that?

 

>
> Already I've run into the situation where it's hard to control order in 
> which two separate create_resources functions are run and I've seen some 
> kludges to fix it, but I'm looking for a better way and hoping that it 
> doesn't involve the use of custom types because that's currently more than 
> I want to deal with.
>
>

Indeed, one of the issues that does exist with create_resources() is that 
you cannot express resource references in Hiera data (though I have seen 
one recent report that claims otherwise).  When part of the point of using 
that function in the first place is that you don't want to have to know 
exactly which resources you are actually declaring, that can present a 
challenge.  It is, nevertheless, a challenge that can be worked around.  In 
some cases, the "workaround" is a mechanism, such as collector chaining,  
that you might reasonably have used anyway.

> It would be super helpful if anyone could point me to a puppet module on 
github that presents me with a better pattern to use with the hiera data.

[...]

> Any other help or criticisms are also welcome.


That looks pretty clean to me already.  It does appear that maybe you want 
to express relationships between your Kvm::Virtpool and Kvm::Virtvol 
instances, but Garrett appears already to have gotten you pointed in a 
useful direction with that (collector chaining).

If you want responses to particular commentary about the create_resources() 
function then it would be helpful to provide a pointer to that commentary.  
R.I. is a pretty smart guy, and he has thought a lot about how Puppet can 
and should use data, so I'm not remotely going to try to respond to 
anything he may have said on the topic without actually reading it for 
myself

Additional, minor comments about your code:

   1. For any but the most simple Execs, I recommend using descriptive 
   resource titles, and expressing the commands themselves via Exec's 
   'command' property.  It is much, much easier that way to understand their 
   purpose, especially in log output.
   2. There are only two reasons for using class inheritance: to make a 
   variable of the inherited class reliably available for use as a class 
   parameter default, and to enable overriding properties of a resource 
   declared by the inherited class.  The latter is the original purpose, and 
   the former is merely a side effect, but the former is by far the more 
   common reason for class inheritance these days.  In all other cases, an 
   ordinary class declaration (include 'foo') can and should be used instead, 
   if indeed anything of the sort is needed at all.  Avoid class inheritance 
   where it is unneeded.  Especially avoid inheriting from a parameterized 
   class.
   3. Your class kvm has the strung 'undef' as the defined default value 
   for a bunch of parameters.  That's quite different from the undef keyword; 
   is it what you really mean?  If it were then I would expect to see some 
   kind of test for those values, but I don't.
   4. More generally, don't define class and type parameter default values 
   unless you really mean it.  If you forget to declare a needed resource or 
   class parameter then Puppet will diagnose the issue if and only if the 
   parameter has no defined default value.  Do not write definitions or 
   classes such that failure to declare all the needed parameters will 
   silently yield an invalid configuration, as appears may be the case for 
   some of your defined parameters.
   

John


-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/cf039141-1935-4380-bfc5-65cc99410726%40googlegro

[Puppet Users] Re: Looking for a better way to use hiera hashes than create_resources

[Alex Scoble]

By the way, if it's not clear, my ultimate goal is to automate everything 
so that bare metal servers pxe boot to a razor server, get their CentOS 
install ISO and kickstart from razor, get installed, boot up, get their 
network interfaces and logical volumes set up, get all the necessary 
libvirt packages installed and configured, all the necessary bits required 
to get VMs up and running (storage pools, virtual volumes, virtual 
networks), VMs get spun up, they connect to razor and get their OSes 
installed and everything configured through Puppet.

And the only thing I have to do is turn on the bare metal servers and sign 
the puppet certs, although I will likely set the Puppet server to autosign 
those as well so everything is automated. This way we can have a set of lab 
systems that can be brought up and torn down as quickly as possible.

On Monday, April 28, 2014 12:34:43 PM UTC-7, Alex Scoble wrote:
>
> Hi All,
>
> I'm working on a module that builds KVM/libvirt hosts and populates them 
> with predefined VMs.
>
> So far I have the module to where it can create any number of virtual 
> nets, storage pools and volumes using virsh, but it isn't pretty.
>
> I've read on various threads here that create_resources is not a good 
> function to use. This was stated quite emphatically by R.I. Pienaar and 
> others.
>
> Already I've run into the situation where it's hard to control order in 
> which two separate create_resources functions are run and I've seen some 
> kludges to fix it, but I'm looking for a better way and hoping that it 
> doesn't involve the use of custom types because that's currently more than 
> I want to deal with.
>
> Here's what I have and hopefully someone can help me do this in a cleaner 
> way. Sorry that it's not in Github as I'm not ready to put it there yet.
>
> *This is an example of the hiera data that I'm working with:*
>
> kvm::servertype: 'kvm'
> kvm::virtnet_name: 'br0'
> kvm::virtnet_forwardmode: 'bridge'
> kvm::virtbridge_name: 'br0'
> kvm::virtnet_macaddress: '52:54:00:1F:95:6C'
> kvm::virtpool_hash:
>   default:
> virtpool_size: '429496729601'
>   iso-images:
> virtpool_target: '/var/lib/libvirt/iso-images'
> virtpool_format: 'iso'
> kvm::virtvol_hash:
>   dtlrazorts1.img:
> volcapacity: '80G'
> volformat: 'qcow2'
>   dtlrepots1.img:
> volcapacity: '80G'
> volformat: 'qcow2'
>   dtlwebvirtmants1.img:
> volcapacity: '60G'
> volformat: 'qcow2'
>
> *My init.pp*
>
> class kvm (
>   $servertype  = $kvm::params::servertype,
>   $virtnet_name= 'undef',
>   $virtnet_forwardmode = 'undef',
>   $virtbridge_name = 'undef',
>   $virtnet_macaddress  = 'undef',
>   $virtpool_hash   = 'undef',
>   $virtvol_hash= 'undef',
> ) inherits kvm::params {
>
>   include kvm::fw
>
>   if ($servertype == 'kvm') {
> 
> file { '/var/opt/lib/pe-puppet/temp':
>   ensure => directory,
>   owner  => 'pe-puppet',
>   group  => 'pe-puppet',  
> }
> 
> package { ['libvirt',
>'python-virtinst',
>'qemu-kvm',
>'qemu-kvm-tools',
>'bridge-utils',
>'virt-manager',
>'libguestfs-tools',]:
>   ensure => present,
> }
>
> service { 'libvirtd':
>   ensure => running,
>   enable => true,
>   hasstatus  => true,
>   hasrestart => true,
>   require=> Package['libvirt'],
> }
> 
> if $virtpool_hash {
> create_resources('kvm::virtpool', $virtpool_hash)
> }
> include kvm::virtnet
> if $virtvol_hash {
> create_resources('kvm::virtvol', $virtvol_hash)
> }
>   }
>
>   if $servertype == 'kvmwebmgr' {
> include kvm::kvmwebmgr
>   }
>
> }
>
> *The virtpool.pp file:*
>
> define kvm::virtpool (
>   $virtpool= $name,
>   $virtpool_size   = undef,
>   $virtpool_target = '/var/lib/libvirt/images',
>   $virtpool_type   = 'dir',
>   $virtpool_format = 'raw',
> ) {
>   
>   file { "${virtpool_target}":
> ensure => directory,
>   } ->  
>   exec { "virsh pool-define-as ${name} --target ${virtpool_target} --type 
> ${virtpool_type} --source-format ${virtpool_format} && virsh pool-autostart 
> ${name} && virsh pool-start ${name}":
> path=> '/usr/bin',
> unless  => "virsh pool-list | /bin/grep ${name}",
> #refreshonly => true,
>   }
> 
> }
>
> *The virtnet.pp file:*
>
> class kvm::virtnet inherits kvm {
>   
>   file { 
> "/var/opt/lib/pe-puppet/temp/${hostname}_virtnet_${virtnet_name}.xml":
> ensure  => file,
> content => template('kvm/virtnet.xml.erb'),
> owner   => 'pe-puppet',
> group   => 'pe-puppet',
>   } ->  
>   exec { 'virsh net-destroy default && virsh net-undefine default':
> path   => '/usr/bin',
> onlyif => 'virsh net-list | /bin/grep default',
>   } ->  
>   exec { "virsh net-define 
> /var/opt/lib/pe-puppet/temp/${hostname}_virtnet_${virtnet_name}.xml && 
> virsh net-autostart