On Wed, Mar 2, 2016 at 1:43 AM R.I.Pienaar <r...@devco.net> wrote: > > > ----- Original Message ----- > > From: "Kylo Ginsberg" <k...@puppetlabs.com> > > To: "puppet-dev" <puppet-dev@googlegroups.com> > > Sent: Wednesday, 2 March, 2016 06:59:08 > > Subject: Re: [Puppet-dev] metaparam question > > > On Fri, Feb 5, 2016 at 12:02 AM, R.I.Pienaar <r...@devco.net> wrote: > > > >> hello, > >> > >> Thanks for the replies, think I'll reply to both in one go > >> > >> ----- Original Message ----- > >> > From: "Luke Kanies" <l...@puppetlabs.com> > >> > To: "puppet-dev" <puppet-dev@googlegroups.com> > >> > Sent: Friday, February 5, 2016 6:15:28 AM > >> > Subject: Re: [Puppet-dev] metaparam question > >> > >> >> On Feb 4, 2016, at 3:35 PM, Kylo Ginsberg <k...@puppetlabs.com> > wrote: > >> >> > >> >>> On Wed, Feb 3, 2016 at 7:47 AM, R.I.Pienaar <r...@devco.net> wrote: > >> >>> hello, > >> >>> > >> >>> I would like to add a metaparameter - which I think is easy now via > >> >>> Type.newmetaparam. > >> >> > >> >> We haven't been thinking of metaparameters as a general purpose > >> extension point. > >> >> This came up once before that I know of, about a year ago, and > there's > >> a little > >> >> discussion of this in https://tickets.puppetlabs.com/browse/PUP-4281 > . > >> The > >> >> conclusion we reached at the time was, more or less, to explore > whether > >> the > >> >> desired change could be accomplished with a puppet function and/or a > >> change to > >> >> core puppet. > >> > > >> > I don't remember my original logic for metaparams not being an > extension > >> point. > >> > Given the system's support for easily loading this kind of plugin, it > >> should be > >> > easy to add. > >> > > >> > The discussion on the ticket is pretty limited. We've got multiple use > >> cases > >> > here, and I have seen use cases in the past brought up, but they were > >> mostly on > >> > lists or in person, so I don't think they made it into the ticketing > >> system. > >> > > >> > RI - you up for a pull request to add the autoloading? Have you > thought > >> much > >> > about how this might go wrong? > >> > >> Unfortunately my time is pretty limited and I'll prefer to see what I > can > >> do > >> with this information in the catalog rather than making a merge worthy > >> patch > >> at present. > >> > >> I'm happy to just patch my actual puppet code with no modules to get to > >> that > >> point, I'll have a look at the validate_cmd in file to see if there is a > >> way > >> in the provider base class to do it so all providers get it. > >> > > > > I kept meaning to get back to this thread and it slipped my mind. Mea > culpa > > on that. > > > > I'm curious where you got with this approach? E.g. did you end up > patching > > an additional metaparameter and were there any lessons worth sharing in > > that? > > Unfortunately work is a bit busy right now so I've not had time to really > give it > any screen time :( > > >> For me this being in every provider is a nice to have I guess - it > would be > >> great to be able to improve puppets ability to check if something worked > >> in the broad sense, and it's going to apply to all resource types > equally. > >> > > > >> I could take the route that https://github.com/nanliu/puppet-transport > >> does > >> and instead of making the meta param do something use it just to store > this > >> information and then some other module do something with this > information > >> and > >> that would let me play with what I want, but it seems a small step to > >> improve > >> puppet in general. > >> > > > > I'm intrigued by the idea of a generic 'validate' or 'health check'. It > > sounds like you're thinking of it as separate from the rest of the > > type/provider, i.e. an external command that could be run, but I also > > wonder (this is half-baked) if, rather than a separate metaparameter, it > > could use a standard provider method to do the validation. I.e. when > puppet > > is applying the catalog it has to essentially ask the same question that > a > > 'validate' method would to figure out if it needs to remediate, so can > the > > same need be met with some refinement of the generic type/provider api? > > As a POC yes, I'd like it external - there's a ton of Nagios plugins out > there > and they would magically all be useful without any extra work, with the > time > I have minimising the yaks have value :) > > But there are common patterns to these things, so for sure you could mark > up a module/resource with some additional properties or allow provider > authors > to do better than we do today - but then they more or less can already > code their > modules to do in depth health tests if they wanted. Probably the > type/provider > API needs some love in general and if that was something you'd make front > and > centre it would be great. I imagine puppet might get some built in > 'checks' > like this, say a generic built in TCP checker that we can reference here > instead > of calling out to a nagios plugin (which you then have to manage), and > that would > be pretty neat and a good place to add pretty good deep introspection value > for module authors. Putting some nasty CLI call with a bunch of arguments > hard coded into a resource is not elegant at all - like nagios based check > validation would cause, that would not be my final destination but a > reasonable > stop along the way. > > The meta parameter though has value - I see this to a large extend as user > serviceable. I get a module from the forge that has some provider, for me > a 'responds well' check might be < 10ms for others it might be < 100ms and > for others it might not even be a time based expression.. And the > parameters > that makes up what is good - here I used response time - are actually very > varied > so no doubt some space for getting this in a magical way for now a meta > param > seems a easy thing to reason about for now while I look at how this works > and > how feasible it is. > > Another approach might be to introduce a more formal observer pattern into > the > system, I've toyed with this a few times and there's a interesting example > over > at https://forge.puppetlabs.com/ryanuber/tell though it's not what I want > but > as a thing to look at and think about it's quite interesting. My thoughts > are > not really well formed around it, but I did built a network wide observer > for > puppet events and could do pretty crazy things with it, not sure they were > good ideas though. > > > > > > (And btw, thanks for the pointer to puppet-transport. I looked at that > long > > ago and had forgotten (or more likely didn't internalize) the use there > of > > a do-nothing metaparameter to store connection information.) > > > > > >> I've told many people this story so no doubt you remember the context > Luke > >> but what I'd like to do is allow people to attach these checks to > resources > >> and of course use them in catalog apply stage to get a deeper sense of > >> 'done' > >> in the same way that the File validate_cmd does but later I want to > consume > >> the catalog in other tools and extract the same - or rather hopefully > speak > >> to some API and get this. > >> > >> Imagine I had a bridge to Sensu where it loads the catalog and just > checks > >> everything that has a validate attached for free with no configuration > >> required, > >> this is actually really easy with sensu as it doesnt require you to > declare > >> the checks upfront on the server and all you have to do is write JSON > to a > >> socket for every check, it'll promiscuously take them and show them and > do > >> alerts etc. > >> > >> Once you have this information and can surface it additionally into the > >> Puppet > >> eco system you can gather the state regularly like a monitoring system > does > >> and it becomes quite interesting later down the line when perhaps you > can > >> use this to influence the node assignment logic in the orchastrator so > that > >> a Web app will be configured to talk to a known good DB server - or that > >> you can detect that the web app should be reconfigured once it's chosen > >> DB isn't good anymore. That's quite far down the line though, I think > just > >> being able to do the monitoring part into sensu would already be a huge > win > >> for many. > >> > > > > I haven't heard this story before but it's super interesting. Since you > > mention the orchestrator and you're thinking about monitoring/validation > to > > gate or to assign connections between nodes, have you seen > > > http://docs.puppetlabs.com/pe/latest/app_orchestration_availability_tests.html#using-availability-test > ? > > Availability tests as described there are just a start but they seem like > > they're in the neighborhood of what you're describing above. > > Yes, I saw those after Eric mentioned it to me at the contributor summit, > they > are interesting for sure and along the lines of what I mean above about > the generic > patterns in checks and module authors bringing their own built in ones. > Though > you'll find few/no successful monitoring systems that told people to > rewrite every > check from fresh when thousands of nagios ones exist. The nagios layer > could well > be one of these checks though. > > > The added idea here of integrating such a thing, be it availability test > or > > validate metaparameter, with a monitoring system could be really > powerful. > > yes in as much as monitoring isn't what monitoring is about. It's about > the data > thats the side effect of monitoring and what you can do with it. I think > Puppet > is fairly uniquely situated to do things with that data very few others > can do. > > > Wrt catalog format (or perhaps report format?) it would be interesting to > > explore whether we could adapt the format to facilitate such > integrations. > > Currently both catalog format and report format are more serializations > of > > internal data structures than they are designed as external integration > > points. I'd have to noodle on this some more, but it's definitely a > course > > worth exploring. > > Yeah, I've done nasty little things to extract information from catalogs > and > they all broke recently with 4 :P Public APIs around interacting with it > would be good. A library to do so would be fantastic >
+1 for having an API to access resource attributes in the catalog. This would solve the entire class of issues where we need configuration data for types/providers. Here's an example: https://github.com/puppetlabs/puppetlabs-splunk/pull/14/files Without this information, the splunk config files must exist where the the path is hardcoded in the resource type. Thanks, Nan -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CACqVBqB5KfVe6na9SDRb_U8UJcfz_67-vFo4B%2BoOzxxDMhoXaQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
