Hi Luke. Thanks for a thoughtful and detailed response. Quite a similar question was posted about two weeks back, you might find > that very interesting: > https://groups.google.com/forum/#!topic/puppet-users/NW2yuHMJvsY >
I saw this, and have been kicking around the idea leading to this post If you are a confident Puppet Coder, you might prefer to import the source, > patch the module to add your feature, then submit the patch back upstream. > This is likely part of my problem. I am not a confident puppet coder, probably closer to barely competent. > When using roles and profiles you end up declaring all the module >> parameters again to avoid losing functionality and flexibility. >> > > ... Not sure I agree with that statement. That sounds odd. Why would you > be re-declaring module parameters if you're not changing something from the > defaults? And if you are intending to change something, then of course you > are supplying different parameters? > Lets say a module has 10 parameters and supplies defaults for most of them. When writing a profile you have to choose how many of the class parameters can remain defaults, how many to override, and how many to expose as profile parameters. It's sounds fine to limit the number of parameters at the profile, right up until you hit an edge case that doesn't work with the default values and the parameter you need to change now requires a profile update... > You also need to be familiar with all the classes, types, and parameters >> from all modules in order to use them effectively. >> > > Ideally the README page of a module would contain amazing user level > documentation of how the module should work... but not that many do. I > often find I have to go read the Puppet code itself to figure out exactly > what a parameter does. > Ditto on the documentation. Some modules are better than others, and of course you can review the manifests, but with my admitted weakness in Puppet DSL it's not always immediately apparent to me what some classes are doing. > To avoid all of the above, I put together the 'basic' module and posted it >> on the forge: https://forge.puppet.com/southalc/basic >> > > Ok :-) I'm beginning to see what the core of your problem is. The fact > that you've created your own module to effectively do create_resources() > hash definitions says to me that you haven't quite grasped the concepts of > the Role / Profile design pattern. I know I have a very strong view on > this subject and many others will disagree, but personally I think the Role > / Profile pattern and the "do-everything-with-Hiera-data" pattern are > practically incompatible. > I'd like to think I grasp the roles/profiles concept, but am just not convinced it's a better approach. Abstracting away configuration details and exposing a limited set of parameters results in uniform configurations. In doing so it also seems it limits flexibility and ensures that you'll continue to spend a good deal of time maintaining your collection of profiles/modules. > This module uses the hiera_hash/create_resources model for all the native >> puppet (version 5.5) types, using module parameters that match the type >> (exceptions for metaparameters, per the README). The module also includes >> the 'file_line' type from puppetlabs/stdlib, the 'archive' type from >> puppet/archive, and the local defined type 'binary', which together provide >> a simple and powerful way to create complex configurations from hiera. All >> module parameters default to an empty hash and also have a merge strategy >> of 'hash' to enable a great deal of flexibility. With this approach I've >> found it possible to replace many single purpose modules it's much faster >> and easier to get the results I'm looking for. >> > > A Hiera-based, data-driven approach will always be faster to produce a > "new" result (just like writing Ansible YAML is faster to produce than > Puppet code)... It's very easy to brain dump configuration into YAML and > have it work, and that's efficient up to a certain point. For your simple > use cases, yes, I can completely see why you would be looking at the Role > Profile pattern and saying to yourself "WTF for?". I think the tipping > point of which design method becomes more efficient directly relates to how > complicated (or how much control) you want over your systems. > A number of people I've talked to like Ansible because of the easy learning curve and great time-to-results. > The more complicated you go, the more I think you will find that Hiera > just doesn't quite cut it. Hiera is a key value store. You can start > using some neat tricks like hash merging, you can look up other keys to > de-duplicate data... When you start to model more and more complicated > infrastructure, I think you will find that you don't have enough power in > Hiera to describe what you want to describe, and that you need an > imperative programming language (eg: if statements, loops, map-reduce). > The Puppet DSL is imperative. > Speaking of hiera tricks, I created an exec resource with the command defined as a multi-line script to include variables and function declarations. I use this to collect data and create local facts. The next puppet run creates additional resources based on the presence of these facts. This is basically the same as creating a module with external facts, but doesn't require a module. An upside is that the fact script doesn't need to execute on every puppet agent run, with the downside being that the host takes a second puppet run to create all resources. I'm not sure if I should be proud or ashamed of what I did, but it works! > Yes, the hiera data can become quite large, but I find it much easier to > manage data in hiera than coding modules with associated logic, parameters, > templates, etc. Is this suitable for hyper-scale deployment? Maybe not, > but for a few hundred servers with a few dozen configuration variants it > seems to work nicely. Is everyone else using puppet actually happy with > the roles/profiles method? > If you are only making small-to-medium changes to a standard operating system, and/or your machines are short-lived cloud systems that get thrown away after half an hour, then sure, a Hiera-only approach will work fine at the scale you are suggesting. In my situation, we have a Red Hat Satellite serving as a Puppet ENC. We also use automated OS provisioning where we can add custom facts. With this we're able to target groups of hosts with specific hiera data, creating a fairly comprehensive configuration. > I also think team size and composition is a big factor. If I was in a > team of one or two people I'm sure I'd be saying "Yeah! Hiera! I can change > anything really really easily!". If I was in a team of a dozen engineers > geographically spread across the world with vastly different levels of > Puppet knowledge I think I'd be saying "Oh god... Everything's in Hiera... > It's so easy for someone to mess up. What on earth has someone changed > now". If you haven't guessed already, I've been here before. > This may be the greatest factor to influence the decision. In my case we have 2 people working with puppet, and the system we're building is to be handed over to team with little to no puppet experience. This system runs at a single site with only a couple hundred managed nodes and maybe a couple dozen unique configurations. > Personally I think the most useful part of the Role Profile design pattern > is the encapsulation of implementation details behind business-specific > Profiles. Jesus, what a mouthful. How about "hiding away all the details > behind an interface that makes sense to me and my team"? > While I like the idea of automating business processes, I dislike the idea of hard-coding them. Business processes change, and the code base will be constantly chasing. Given the choice, I'd rather change data than code. Best demonstrated with a real life example we use here... > > https://gist.github.com/lukebigum/1a8b9735f604493da38d7b1d924ba4bb > > The above is the Profile for an LMAX "Statistics Collection Server". A > statistics collection server collects statistics. If someone wants to > collect statistics, all they have to do is put: > > include ::profile::statistics::collection_server > > Somewhere in a node definition and set *AT MOST* nine Hiera parameters for > that Profile. That's the real win - an LMAX statistics collection server > has only 9 parameters that can be changed. They don't really have to > understand exactly what goes into building a Statistics Collection Server > if they don't want to (in practice they might need to browse the code to > check what a parameter does though, because we are lazy and don't document > our Profiles). > > If you go read that profile in detail you'll see I pull in several > component modules: Puppetlabs Apache, Influxdb, a private LVM module that's > a wrapper for Puppetlabs' LVM, Grafana, and Chronograf. Apache (with SSL) > is set up to proxy Grafana and Chronograf. Our LVM module creates the file > system underneath Influx before is installed. Most of the parameters to > the component modules are hard coded, and this is a great thing because > that means every single one of our Statistics Collection Servers are > exactly the same. I even pull in a (private) Nagios module to define > monitoring resources, so when one of my Engineers uses that profile they > get the monitoring _automatically_. > > I count 81 parameters to component modules in that Profile, so that would > be at least 81 lines of Hiera needed to reproduce that functionality in > YAML (and even then, good luck ensuring that the LVM disk is there before > Influx is installed). I have condensed that to 9 possible parameters where > I think someone should legitimately change something. Otherwise, you use > my defaults, and that keeps things the same, reducing entropy across our > estate. Yes, writing this profile took a lot longer than doing it in YAML, > but our engineers shouldn't need to "figure out" how to build an InfluxDB > server ever again. > Thanks for sharing your example. I won't argue that roles/profiles isn't a great way to standardize deployments. I guess my complaint is that it takes so much investment to get there. > Another big win for me: testing. I can write puppet-rspec units tests for > the above Profile to make sure that if someone tries to change it, they > don't break any existing functionality. Our current workflow has our > engineers committing onto branches and creating Merge Requests in our > private Git Lab. All tests must pass before they can merge code to Master. > They usually get notified within minutes if something they've pushed hasn't > passed tests. > You can do testing of Hiera-defined infrastructure, however all approaches > I've read about seem awfully cumbersome and wasteful. I won't rant about > that today. > When I looked into writing rspec tests it was a bit daunting. Much worse than beginning with Puppet DSL. My puppet tests generally consist of applying my changes locally and observing results. Cumbersome and wasteful are the correct terms. So tell me, how did I go at convincing you? :-) > Well, you have caused me some guilt that maybe I've taken the easy way out rather than becoming more proficient with puppet. Once you've had that first hit and instant high from the hiera crack pipe... it's hard not to go back. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/ea77b61a-1ac9-40ea-878b-20c93e37180a%40googlegroups.com.
