On Wed, Mar 2, 2016 at 2:50 AM, Trevor Vaughan <[email protected]> wrote:
> Hi all, > > This is a very interesting conversation and I'm monitoring to see where it > ends up. > I see what you did there ;) Thanks for the chuckle. > > You're going to want to have a central location for organizing this > information between nodes since there are no guarantees that any two nodes > can get bi-directional communications between each other. > > Take, for instance, a highly regulated HIPAA environment. You may be able > to send data one way between two managed environments but bi-directional > communication would be unable to be processed. Therefore, you would need > some level of controlled and shared trigger between your host catalogs. > (OK, fine, I still want language-level semaphores). > > Trevor > > On Wed, Mar 2, 2016 at 4:43 AM, R.I.Pienaar <[email protected]> wrote: > >> >> >> ----- Original Message ----- >> > From: "Kylo Ginsberg" <[email protected]> >> > To: "puppet-dev" <[email protected]> >> > Sent: Wednesday, 2 March, 2016 06:59:08 >> > Subject: Re: [Puppet-dev] metaparam question >> >> > On Fri, Feb 5, 2016 at 12:02 AM, R.I.Pienaar <[email protected]> wrote: >> > >> >> hello, >> >> >> >> Thanks for the replies, think I'll reply to both in one go >> >> >> >> ----- Original Message ----- >> >> > From: "Luke Kanies" <[email protected]> >> >> > To: "puppet-dev" <[email protected]> >> >> > Sent: Friday, February 5, 2016 6:15:28 AM >> >> > Subject: Re: [Puppet-dev] metaparam question >> >> >> >> >> On Feb 4, 2016, at 3:35 PM, Kylo Ginsberg <[email protected]> >> wrote: >> >> >> >> >> >>> On Wed, Feb 3, 2016 at 7:47 AM, R.I.Pienaar <[email protected]> wrote: >> >> >>> hello, >> >> >>> >> >> >>> I would like to add a metaparameter - which I think is easy now via >> >> >>> Type.newmetaparam. >> >> >> >> >> >> We haven't been thinking of metaparameters as a general purpose >> >> extension point. >> >> >> This came up once before that I know of, about a year ago, and >> there's >> >> a little >> >> >> discussion of this in >> https://tickets.puppetlabs.com/browse/PUP-4281. >> >> The >> >> >> conclusion we reached at the time was, more or less, to explore >> whether >> >> the >> >> >> desired change could be accomplished with a puppet function and/or a >> >> change to >> >> >> core puppet. >> >> > >> >> > I don't remember my original logic for metaparams not being an >> extension >> >> point. >> >> > Given the system's support for easily loading this kind of plugin, it >> >> should be >> >> > easy to add. >> >> > >> >> > The discussion on the ticket is pretty limited. We've got multiple >> use >> >> cases >> >> > here, and I have seen use cases in the past brought up, but they were >> >> mostly on >> >> > lists or in person, so I don't think they made it into the ticketing >> >> system. >> >> > >> >> > RI - you up for a pull request to add the autoloading? Have you >> thought >> >> much >> >> > about how this might go wrong? >> >> >> >> Unfortunately my time is pretty limited and I'll prefer to see what I >> can >> >> do >> >> with this information in the catalog rather than making a merge worthy >> >> patch >> >> at present. >> >> >> >> I'm happy to just patch my actual puppet code with no modules to get to >> >> that >> >> point, I'll have a look at the validate_cmd in file to see if there is >> a >> >> way >> >> in the provider base class to do it so all providers get it. >> >> >> > >> > I kept meaning to get back to this thread and it slipped my mind. Mea >> culpa >> > on that. >> > >> > I'm curious where you got with this approach? E.g. did you end up >> patching >> > an additional metaparameter and were there any lessons worth sharing in >> > that? >> >> Unfortunately work is a bit busy right now so I've not had time to really >> give it >> any screen time :( >> >> >> For me this being in every provider is a nice to have I guess - it >> would be >> >> great to be able to improve puppets ability to check if something >> worked >> >> in the broad sense, and it's going to apply to all resource types >> equally. >> >> >> > >> >> I could take the route that https://github.com/nanliu/puppet-transport >> >> does >> >> and instead of making the meta param do something use it just to store >> this >> >> information and then some other module do something with this >> information >> >> and >> >> that would let me play with what I want, but it seems a small step to >> >> improve >> >> puppet in general. >> >> >> > >> > I'm intrigued by the idea of a generic 'validate' or 'health check'. It >> > sounds like you're thinking of it as separate from the rest of the >> > type/provider, i.e. an external command that could be run, but I also >> > wonder (this is half-baked) if, rather than a separate metaparameter, it >> > could use a standard provider method to do the validation. I.e. when >> puppet >> > is applying the catalog it has to essentially ask the same question >> that a >> > 'validate' method would to figure out if it needs to remediate, so can >> the >> > same need be met with some refinement of the generic type/provider api? >> >> As a POC yes, I'd like it external - there's a ton of Nagios plugins out >> there >> and they would magically all be useful without any extra work, with the >> time >> I have minimising the yaks have value :) >> >> But there are common patterns to these things, so for sure you could mark >> up a module/resource with some additional properties or allow provider >> authors >> to do better than we do today - but then they more or less can already >> code their >> modules to do in depth health tests if they wanted. Probably the >> type/provider >> API needs some love in general and if that was something you'd make front >> and >> centre it would be great. I imagine puppet might get some built in >> 'checks' >> like this, say a generic built in TCP checker that we can reference here >> instead >> of calling out to a nagios plugin (which you then have to manage), and >> that would >> be pretty neat and a good place to add pretty good deep introspection >> value >> for module authors. Putting some nasty CLI call with a bunch of arguments >> hard coded into a resource is not elegant at all - like nagios based check >> validation would cause, that would not be my final destination but a >> reasonable >> stop along the way. >> >> The meta parameter though has value - I see this to a large extend as user >> serviceable. I get a module from the forge that has some provider, for me >> a 'responds well' check might be < 10ms for others it might be < 100ms and >> for others it might not even be a time based expression.. And the >> parameters >> that makes up what is good - here I used response time - are actually >> very varied >> so no doubt some space for getting this in a magical way for now a meta >> param >> seems a easy thing to reason about for now while I look at how this works >> and >> how feasible it is. >> >> Another approach might be to introduce a more formal observer pattern >> into the >> system, I've toyed with this a few times and there's a interesting >> example over >> at https://forge.puppetlabs.com/ryanuber/tell though it's not what I >> want but >> as a thing to look at and think about it's quite interesting. My >> thoughts are >> not really well formed around it, but I did built a network wide observer >> for >> puppet events and could do pretty crazy things with it, not sure they were >> good ideas though. >> >> >> > >> > (And btw, thanks for the pointer to puppet-transport. I looked at that >> long >> > ago and had forgotten (or more likely didn't internalize) the use there >> of >> > a do-nothing metaparameter to store connection information.) >> > >> > >> >> I've told many people this story so no doubt you remember the context >> Luke >> >> but what I'd like to do is allow people to attach these checks to >> resources >> >> and of course use them in catalog apply stage to get a deeper sense of >> >> 'done' >> >> in the same way that the File validate_cmd does but later I want to >> consume >> >> the catalog in other tools and extract the same - or rather hopefully >> speak >> >> to some API and get this. >> >> >> >> Imagine I had a bridge to Sensu where it loads the catalog and just >> checks >> >> everything that has a validate attached for free with no configuration >> >> required, >> >> this is actually really easy with sensu as it doesnt require you to >> declare >> >> the checks upfront on the server and all you have to do is write JSON >> to a >> >> socket for every check, it'll promiscuously take them and show them >> and do >> >> alerts etc. >> >> >> >> Once you have this information and can surface it additionally into the >> >> Puppet >> >> eco system you can gather the state regularly like a monitoring system >> does >> >> and it becomes quite interesting later down the line when perhaps you >> can >> >> use this to influence the node assignment logic in the orchastrator so >> that >> >> a Web app will be configured to talk to a known good DB server - or >> that >> >> you can detect that the web app should be reconfigured once it's chosen >> >> DB isn't good anymore. That's quite far down the line though, I think >> just >> >> being able to do the monitoring part into sensu would already be a >> huge win >> >> for many. >> >> >> > >> > I haven't heard this story before but it's super interesting. Since you >> > mention the orchestrator and you're thinking about >> monitoring/validation to >> > gate or to assign connections between nodes, have you seen >> > >> http://docs.puppetlabs.com/pe/latest/app_orchestration_availability_tests.html#using-availability-test >> ? >> > Availability tests as described there are just a start but they seem >> like >> > they're in the neighborhood of what you're describing above. >> >> Yes, I saw those after Eric mentioned it to me at the contributor summit, >> they >> are interesting for sure and along the lines of what I mean above about >> the generic >> patterns in checks and module authors bringing their own built in ones. >> Though >> you'll find few/no successful monitoring systems that told people to >> rewrite every >> check from fresh when thousands of nagios ones exist. The nagios layer >> could well >> be one of these checks though. >> >> > The added idea here of integrating such a thing, be it availability >> test or >> > validate metaparameter, with a monitoring system could be really >> powerful. >> >> yes in as much as monitoring isn't what monitoring is about. It's about >> the data >> thats the side effect of monitoring and what you can do with it. I think >> Puppet >> is fairly uniquely situated to do things with that data very few others >> can do. >> >> > Wrt catalog format (or perhaps report format?) it would be interesting >> to >> > explore whether we could adapt the format to facilitate such >> integrations. >> > Currently both catalog format and report format are more serializations >> of >> > internal data structures than they are designed as external integration >> > points. I'd have to noodle on this some more, but it's definitely a >> course >> > worth exploring. >> >> Yeah, I've done nasty little things to extract information from catalogs >> and >> they all broke recently with 4 :P Public APIs around interacting with it >> would be good. A library to do so would be fantastic >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-dev/454494855.237167.1456911800479.JavaMail.zimbra%40devco.net >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Trevor Vaughan > Vice President, Onyx Point, Inc > (410) 541-6699 > > -- This account not approved for unencrypted proprietary information -- > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXjF%3DTh1o5vJBU9hW%3DHhmxNDcDf5CHAkc-TUM9HF4hxDA%40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXjF%3DTh1o5vJBU9hW%3DHhmxNDcDf5CHAkc-TUM9HF4hxDA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Kylo Ginsberg | [email protected] | irc: kylo | twitter: @kylog -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALsUZFEe3jinCgPRWtt1yZngGhK54dbkC0ZXajUJPdAK_Fuudw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
