On Monday, February 11, 2013 11:09:28 AM UTC-6, Roman Shaposhnik wrote: > > On Mon, Feb 11, 2013 at 6:39 AM, jcbollinger > <john.bo...@stjude.org<javascript:>> > wrote: > > I am inclined to suspect, however, that you are > > rejecting Hiera based on mis- or limited understanding of what it can do > for > > you. If Hiera can in fact do the job adequately, then I think it makes > the > > most sense to use it instead of building your own custom solution. > > Honestly, I started this thread to be educated on various options. If you > have the patience to bear with me -- that'll be mostly appreciated. I may > ask some silly questions, though -- so patience is the keyword here ;-) > > Now, before you read further let me state a few assumptions that I have as > far as using Hiera with a very custom ENC is concerned and see if they > hold true. Here's my ideal state of things as far as how I'd like to use > Puppet: > #1 I'd like to have a very flexible set of Puppet classes capable of > deploying radically different topologies of Bigtop Hadoop > clusters. > ASSUMPTION: to me this means a set of module(s) full of > highly parameterized classes >
False. Any parametrized class you write can be trivially rewritten without parameters, instead pulling data from an external source (with hiera being the canonical example of such a source). That changes only the protocol by which data is fed to your classes, not their expressive capability. > #2 Given that there's nothing static about my environment (I'm not > running a datacenter -- I'm running a utility that lets users > deploy > random configurations of Bigtop Hadoop clusters on VMs) I'd rather > minimize the # of things I have to configure/interact with > when describing > how the next N VMs should look like. > ASSUMPTION: to me this means relying exclusively on a very custom > ENC that would instantiate classes > with precisely the > right set of parameters, instead of > updating Hiera backend > everytime a new deployment is asked for. > Questionable. You posit a large and diverse collection of data characterizing the configuration that will be deployed. You will certainly need to record that data somewhere, somehow, for each configuration. You or your users will need to do this at least as part of the process of developing a configuration. You or your users will probably want to have it on an ongoing basis if they want to *maintain* their configuration via Puppet or deploy the same configuration on a new set of VMs. You or your users may also want to have it as a starting point for creating variant configurations, instead of starting from scratch each time. So, since you need to record your data, why not do so in a form that Puppet can use directly? As a bonus, your ENC can then be a lot simpler. > #3 Even though I can make my ENC spew out a setting for every single > parameter that each class needs I'd rather check with the > collective > Puppet intelligence first to figure out whether there's a > DSL-level language > construct that would allow a parameter in a 'base' class > affect the defaults > in the 'child' one. NOTE: I'm NOT talking about dragging > data/settings > into my classes, I'm purely asking whether there's a sane way for > the > following bit of my ENC output: > classes: > cluster::hdfs > auth: kerberos > to affect all the children such as cluster::hdfs::namenode so the > ENC doesn't have to do it explicitly. > ASSUMPTION: I do realize that Hiera is one option to make this > happen, > however, it seems that in my case the > trade off is: I need > to teach my ENC to manipulate the > state of Hiera backend. > If I decide to do that it seems that > I might as well make my > ENC output extremely verbose. > Questionable. Although you could, in principle, make your ENC write or rewrite data for the hiera back end, that seems a strange way to go about it. One would normally maintain the data separately from, but in parallel with the ENC itself. Otherwise you're recomputing the data every time, or else you're reading it from somewhere else and transliterating it into the form that Puppet will consume, which seems like wasted effort to me. To answer your question more directly, Puppet parametrized classes can draw on *non-parametrized* classes' variables for their own parameter defaults, but the mechanism for doing so reliably uses class inheritance, so 1. Doing so prevents you using class inheritance for anything else (which may be tolerable), and 2. You cannot safely use parametrized classes as a source of class parameter defaults, because parametrized base classes are not supported. > > > Separating configuration data from your manifests is precisely what > Hiera is > > for. The prevailing opinion around here is that that's a good thing. > > Besides, if you want the data in Puppet DSL then you can just declare it > as > > variables of some class or classes. Or your ENC can set it as the > values of > > global variables. > > An ideal place for me to keep this data is at the level of ENC. Personally, I tend to put an hiera data store as on the same conceptual level as an ENC. I suppose you mean that the ideal place would be embedded in the ENC itself. You know your own needs better than I do, but find such a plan surprising. > I > really *DO NOT* > want it at the level of the DSL itself. Ok, though you seemed to say differently before. > What I do need at the level of the DSL > are the hooks to make the kind of interfacing with the ENC possible. > There are few such hooks. The protocol between Puppet and ENC is: Puppet feeds a node name to the ENC, and the ENC spits back global variables, classes, and, optionally, the classes' parameters. A sufficiently intrepid ENC can also weasel the target node's facts out of Puppet, though that's not part of the official protocol. An ENC could also, as you imagine, modify data files on which the master will later rely, or otherwise muck with the master, supposing that you are not running in a multi-master environment such as under apache/passenger. I really would not advise such an approach, but it would be possible. > > > Furthermore, Puppet 3 integrates Hiera with parameterized classes by > using > > hiera to look up values for class parameters that are not explicitly > > declared, before ultimately falling back to any defaults written into > the > > class itself. There are good reasons to rely on that function > completely, > > instead of declaring parameters explicitly anywhere, though those > reasons > > are somewhat weaker if you use a custom ENC. > > That's precisely my use case -- a very custom ENC. In fact, I'd rather > have > my ENC be a single source of truth for all *parameterization* > The reasons are weaker with a custom ENC because in that case you can assume the responsibility to program the ENC to declare classes in an order that works -- which is not necessarily easy. It means extra effort must be spent developing and maintaining the ENC and manifest set, because it is not safe in general to assume that any class can declare its own dependencies. It also, therefore, requires much tighter coupling between ENC and manifest set than is otherwise necessary. Although that trade-off is available, it's not one that I would advise anyone to make. > > > Is that a bona fide use case for you? That would be unusual. > > It seems to be (unless given my description above you will draw > a different conclusion). > I don't have enough information to draw a conclusion, and certainly what you've said so far hasn't persuaded me. Specifically, what I was referring to was a situation such as your example of systems flipping back and forth between auth => 'simple' and auth => 'kerberos' in alternating hours. Is that a realistic example of something you need to be able to do? If so, then Puppet may be the wrong tool altogether. To support something like that you probably want a tool that will proactively modify configuration cluster-wide, on demand, with immediate success / failure feedback. If that's what you need, then Puppet is not it. > > > I really don't understand what you mean by "too global". > > Sorry for the poor choice of words. What I really meant was that > in my case, where: > # the knowledge of how defaults cascade changes with every > single deployment > I still don't understand. How is that not a property of your classes? Isn't that what you started off asking about anyway? Perhaps what varies with each deployment is which defaults you override? > # such knowledge originates with an ENC > having Hiera in the mix would mean yet another subsytem which > state I'd have to keep consistent with the state of my ENC. > > Your ENC is *stateful*? How does that even work? > > That sounds like a workable solution, but whether it makes sense depends > on > > information I don't have. > > Well, you know know the full story. If you could help making the right > choice > here -- that'll be, as I said, appreciated. > > I'm not sure I do know the full story, but here's my take on what you've given us: 1. I think you are creating a false (or at least unneeded) dichotomy between the ENC's data and hiera-curated data. To the extent that the ENC itself relies on external data -- a configuration file or a database, for instance -- that data could instead be served directly via hiera. The ENC's data could thus BE your hiera data store. 2. If you are looking for synchronous cluster-wide configuration changes, then you probably want to look somewhere else. Mcollective might suit your needs better, for example. 3. Declaring explicit parameter values for your classes, whether via an ENC or in DSL, incurs a larger development and maintenance cost for your ENC and manifests than not. How much larger depends on the size and nature of your manifest set, but it could be significant. 4. I can't think of anything in Puppet other than hiera that supports anything like your idea for "cascading defaults" down a tree of parametrized classes. 5. Overall, I get the sense that you've chosen a design concept and you're trying to fit it to Puppet. It may be worthwhile to take a step back and re-evaluate your design concept in light of what you are learning about Puppet. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
