[Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
On May 14, 8:52 am, Simon J Mudd sjm...@pobox.com wrote: sjm...@pobox.com (Simon J Mudd) writes: john.bollin...@stjude.org (jcbollinger) writes: In fact, Puppet Labs's own recently updated style guide recommends against using extlookup(), though that position is controversial. I found the URL:http://docs.puppetlabs.com/guides/style_guide.html The extlookup() Function Modules should avoid the use of extlookup() in favor of ENCs or other alternatives. The statement is clear, but it would be nice to know the reasoning. That is if there's a problem with extlookup() then it would be good to know what it is. This may have been discussed elsewhere but it shouldn't be necessary to have to go and search for this. In the end I did.http://groups.google.com/group/puppet-users/browse_thread/thread/34e4... It is a good read. Now you know what I meant when I called Puppet Labs's style recommendation controversial. Personally, I remain unpersuaded that the recommended style is a good technical choice for most people, but I will not rehash the arguments that you have just read. However, from the discussion a few things strike me: 1. the use of parameterised classes is recommended heavily. I've just found out about this new feature inspite of using puppet for a long time. Hence many recipes that I'm using are not paremeterised and I have a large number of similar classes. This is certainly worth fixing but is quite a painful transition to make given the number of systems in use and the chance of creating heavy breakage during that change. So if you haven't used parameterised classes much that's the first thing that needs looking at. If you have a complex set of manifests that do not use parameterized classes, then I think it much safer to coalesce similar classes with the help of extlookup() than by attempting to switch to parameterized classes. There are a couple of important ways (other than parameterization itself) by which parameterized classes differ from non-parameterized ones, and the more complex your current manifests, the more likely these are to bite you if you parameterize existing classes. Note, however, that it is the use of extlookup() *as an alternative to class parameterization* that the style guide recommends against. Do not take it as a blanket recommendation against using the feature. Note also that another part of the Style Guide's approach is that classes should avoid including other classes. Although it may not be immediately obvious, that goes hand-in-hand with extensive use of parameterized classes. Do take that into consideration as you evaluate the Style Guide's recommendations. [...] if you already have working recipes perhaps and aren't using an ENC [, extlookup] gives you a way to clearly remove the conditional logic the recipes currently have. It's then a smaller step up to using an ENC if you want to go that way. For those that don't it still becomes reasonably clear where the configuration is parameterised and how/why that's done, so it's still a more flexible approach. I agree. Furthermore, I maintain that if you are not using an ENC, then parameterized classes offer few advantages to offset their disadvantages. Therefore, if you are not already using an ENC and do not want to switch to one right now, then extlookup() is the best generally available way to externalize your configuration's data. John -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
john.bollin...@stjude.org (jcbollinger) writes:
On May 14, 8:52 am, Simon J Mudd sjm...@pobox.com wrote:
However, from the discussion a few things strike me:
1. the use of parameterised classes is recommended heavily. I've just
found out about this new feature inspite of using puppet for a long
time. Hence many recipes that I'm using are not paremeterised and I
have a large number of similar classes. This is certainly worth fixing
but is quite a painful transition to make given the number of systems
in use and the chance of creating heavy breakage during that change.
So if you haven't used parameterised classes much that's the first
thing that needs looking at.
If you have a complex set of manifests that do not use parameterized
classes, then I think it much safer to coalesce similar classes with
the help of extlookup() than by attempting to switch to parameterized
classes.
Perhaps. To some extend my non-parameterised classes are _very_ similar
in many ways except for various parameters (creation of logical volumes
and filesystems, version of mysql to use, creation of certain cron jobs
for importing informtation into a configuration database, and other crons
for doing standard jobs, etc) Most of the completely common stuff
has been taken out. However, using extlookup() doesn't really work for
me right now for a simple reason I haven't yet mentioned. I'd further
like to have a lookup_by_key() function [let's say similar to the extlookup()
one provided now] but also a lookup_by_regexp() function where I can match
groups of boxes, in my case mainly based on regexps of their hostnames
rather than on specific $hostname, or $domain, etc Without that
I need to add more entries to match by lots of hostnames, which doesn't
work.
So I'd like something like
$mount_point = lookup_by_regexp( 'mount_point', $hostname, 'regexp_file', '' )
# parameters: lookup_item, lookup_value, regexp file to lookup, default
$lvsize = lookup_by_regexp( 'lvsize', $hostname, 'regexp_file', '' )
class db_lvconfig { db_lvconfig: mount_point = $mount_point, lvsize = $lvsize
} # note: other default parameters not shown like: owner, group, mode, vgname,
lvname
and have in the .csv file something like
mount_point,/dbservera/,/mount_point/for/servera
mount_point,/dbserverb/,/mount_point/for/serverb
mount_point,/dbserverc/,/mount_point/for/serverc
lvsize,/dbservera/,100G
lvsize,/dbserverb/,200G
lvsize,/dbserverc/,300G
Then if $hostname = 'dbservera-001'
it would get $lvsize = '100G', and $mount_point = '/mount_point/for/servera'
If I have 10 dbservera-xxx boxes then this keeps the config simple and the
.csv files much smaller.
There are a couple of important ways (other than
parameterization itself) by which parameterized classes differ from
non-parameterized ones, and the more complex your current manifests,
the more likely these are to bite you if you parameterize existing
classes.
Perhaps. for most of my cases I don't think that's the case. I'm
slowly adjusting the configuration of several classes to use
parameterised classes but even these still work nicely with
extlookup() alternatives such as the currently unimplemented
lookup_by_key() or lookup_by_regexp() alternatives I'm thinking of
using.
Note, however, that it is the use of extlookup() *as an alternative to
class parameterization* that the style guide recommends against. Do
not take it as a blanket recommendation against using the feature.
I'm not. You need to have a very considered mindset to build
_everything_ top down using an ENC and classes or globalvariables. If
using global variables as heavily as puppet seem to imply then ensuing
that you have a clear naming policy for this is quite important to
avoid unexpected name clashes (or mistakes) which might go unnoticed.
So I'd certainly like to see real world examples of complete
configurations to be fully convinced this works well. Perhaps after
moving a configuration to use extlookup() type mechanisms it then
becomes easier to make a further step up to get to the ENCs style
Puppetlabs reccommends. However, with system configuration being a
continual ongoing process I tend to think that a 100% ENC-only style
build using only parameterised classes is going to be a hard thing to
achieve in practice. And the global variables just don't convince me
yet. Perhaps I still have to see the light.
Note also that another part of the Style Guide's approach is that
classes should avoid including other classes. Although it may not be
immediately obvious, that goes hand-in-hand with extensive use of
parameterized classes. Do take that into consideration as you
evaluate the Style Guide's recommendations.
Hm.. well building bigger blocks from smaller ones tends to be a good
practice to follow, so this recommendation seems counterintuitive.
I'll have to lookup the section and see if they explain why, but guess
it's because the intention is you build your server in
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
- Original Message - john.bollin...@stjude.org (jcbollinger) writes: Perhaps. To some extend my non-parameterised classes are _very_ similar in many ways except for various parameters (creation of logical volumes and filesystems, version of mysql to use, creation of certain cron jobs for importing informtation into a configuration database, and other crons for doing standard jobs, etc) Most of the completely common stuff has been taken out. However, using extlookup() doesn't really work for me right now for a simple reason I haven't yet mentioned. I'd further like to have a lookup_by_key() function [let's say similar to the extlookup() one provided now] but also a lookup_by_regexp() function where I can match groups of boxes, in my case mainly based on regexps of their hostnames rather than on specific $hostname, or $domain, etc Without that I need to add more entries to match by lots of hostnames, which doesn't work. I think you're really trying to just avoid thinking in the puppet way instead trying to force some previous ideas on how things should work on it rather than figure out how it works. For example, you could just define a fact on your machines that indicate what role they have - perhaps based on your regex hostnames - and then use that in extlookup to select csv file to use. That would achieve your goal and it would be the puppet way of solving it. Puppetlabs reccommends. However, with system configuration being a continual ongoing process I tend to think that a 100% ENC-only style build using only parameterised classes is going to be a hard thing to achieve in practice. And the global variables just don't convince me yet. Perhaps I still have to see the light. ENCs can (now) pass parameters to param classes. So you dont have to do it all with global vars. Note also that another part of the Style Guide's approach is that classes should avoid including other classes. Although it may not be immediately obvious, that goes hand-in-hand with extensive use of parameterized classes. Do take that into consideration as you evaluate the Style Guide's recommendations. Hm.. well building bigger blocks from smaller ones tends to be a good practice to follow, so this recommendation seems counterintuitive. I'll have to lookup the section and see if they explain why, but guess it's because the intention is you build your server in terms of smallish components. +1, I think this is mostly down to the bad design in param classes - they really do not promote the use of many small classes that creates a larger module. Making modules harder to read and making dependencies harder to specify. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
Hi John, john.bollin...@stjude.org (jcbollinger) writes: ... Let's not cast things in terms of correctness, except insomuch as whether they reliably produce the desired effect on clients. Indeed. As always there is more than one way to solve a problem. Depending on what you're trying to do with Puppet, however, it is certainly true that some approaches to structuring your manifests will make them easier to write and maintain than will others. Personally, when writing specializations into my manifests, I find it useful to keep in mind the question of *why* a particular host or group of hosts is different from others. I have yet to run into an answer that didn't fall into one of these categories, or a combination of them: 1) It has a functional role that some other systems do not In my case 90% of the differences are functional differences, often expressed as part of the hostname. 2) Its network environment requires differences from some other systems. There may be some minor differences here but they are the exceptions rather than the rule. 3) It's just special I have a few of these too. It has worked well for me so far to model roles via (unparameterized) classes, assigned to nodes via their node declarations. That leaves only one level between common and node-specific where I might need to customize data. Yes, I have db_server::type_a and db_server::type_b, ... (That intermediate level could in principal be parameterized by network domain, but in my case it is by subnet.) Most of my nodes do not require per-node customization, so I don't end up with many data files for extlookup. In my case one db_server is pretty much similar to another one. Things vary such as mysql version to use (normally pretty constant), partitions to mount and their location(s), cron jobs to setup. All of these are lower level classes which are almost the same except for a few parameters. I also started with non-parameterised classes so created a large number of nearly duplicate classes and now I'm beginning to parameterise some of these the total number of classes should drop much more easily. Of course if I can lookup the parameters from a data source (extlookup or similar) then I can pretty much make many classes identical: all they do is pickup the required parameters and apply them. Moreover, I disagree with several of the opinions and conclusions in your post: 1) You write: The extlookup() functionality only allows [... specifying implicitly ...] where to look for this value. That is false. Extlookup does provide for configuration of a standard set of CSV files to search (which can be parameterized by nodes' facts), but the function also has an optional parameter to specify a specific file to be searched first on any given invocation. Perhaps coming from a database background, I'd like to mirror what seems more _natural_ and having values spread around over potentially a large number of files seems non-intuitive. If extlookup use would indeed require you to maintain a large number of separate files then that might be a good reason to find something better, but in all likelihood you can avoid that, or else structure it in a sane way. Consider also: When you work with a database, do you normally focus on how it maps data to the host filesystem? Focus no. Again as mentioned I try to simplify the structure by normalising it. Here I see I want to lookup a parameter ( say partition size ) from somewhere and I need to find it for a server called $hostname. If I can't find a parameter for $hostname, I may be happy if I find the same parameter for $domain, or if not I may be happy with some default value. _My_ preference is to look that up in one place. Also as mentioned in a different message, doing this lookup via a regexp might nicely enable me to keep the list of entries short. Given the diverse data parameterization you described, if you created a database for your configuration data, would you really organize everything into a single table? (And what would be its key?) yes: CREATE TABLE lookup_table ( config_item VARCHAR(200) NOT NULL, lookup_value VARCHAR(200) NOT NULL, return_value VARCHAR(200) NOT NULL PRIMARY KEY ( config_item, lookup_value ) ) SELECT return_value FROM lookup_table WHERE config_item = 'lvsize' AND lookup_value = 'myhostname001' provides a fast lookup of the value. If that fails you can do SELECT return_value FROM lookup_table WHERE config_item = 'lvsize' AND lookup_value = 'example.com' for a more generic response. If that fails you can do: SELECT return_value FROM lookup_table WHERE config_item = 'lvsize' AND lookup_value = 'DEFAULT' Yes, I'm fully aware this could be normalised better but given the limited number of entries it's trivial to understand, trivial to maintain fast to access. The 3 SELECTs could be optimised into a _single_ more complex SELECT adding
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
r...@devco.net (R.I.Pienaar) writes: - Original Message - john.bollin...@stjude.org (jcbollinger) writes: Perhaps. To some extend my non-parameterised classes are _very_ similar in many ways except for various parameters (creation of logical volumes and filesystems, version of mysql to use, creation of certain cron jobs for importing informtation into a configuration database, and other crons for doing standard jobs, etc) Most of the completely common stuff has been taken out. However, using extlookup() doesn't really work for me right now for a simple reason I haven't yet mentioned. I'd further like to have a lookup_by_key() function [let's say similar to the extlookup() one provided now] but also a lookup_by_regexp() function where I can match groups of boxes, in my case mainly based on regexps of their hostnames rather than on specific $hostname, or $domain, etc Without that I need to add more entries to match by lots of hostnames, which doesn't work. I think you're really trying to just avoid thinking in the puppet way instead trying to force some previous ideas on how things should work on it rather than figure out how it works. For example, you could just define a fact on your machines that indicate what role they have - perhaps based on your regex hostnames - and then use that in extlookup to select csv file to use. That would achieve your goal and it would be the puppet way of solving it. Currently I build nodes.pp from an external source and that provides the node with it's basic class (and hence role). So that's not needed. One of the irritating things I see when we discuss puppet is the inevitable I can't describe my complete problem as it's [confidential] my site specific, so you can't have a full understanding of what I'm doing or why I'm doing it wrong, or provide me a precise, clear explanation of how better to solve my original problem. We talk in generics and that means we never fully understand each other. I guess the reasons are obvious but it's probably frustrating to us all. Nevertheless this thread I started has helped me clarify several things so I appreciate the time you've spent discussing them with me. Simon -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
- Original Message - r...@devco.net (R.I.Pienaar) writes: - Original Message - john.bollin...@stjude.org (jcbollinger) writes: Perhaps. To some extend my non-parameterised classes are _very_ similar in many ways except for various parameters (creation of logical volumes and filesystems, version of mysql to use, creation of certain cron jobs for importing informtation into a configuration database, and other crons for doing standard jobs, etc) Most of the completely common stuff has been taken out. However, using extlookup() doesn't really work for me right now for a simple reason I haven't yet mentioned. I'd further like to have a lookup_by_key() function [let's say similar to the extlookup() one provided now] but also a lookup_by_regexp() function where I can match groups of boxes, in my case mainly based on regexps of their hostnames rather than on specific $hostname, or $domain, etc Without that I need to add more entries to match by lots of hostnames, which doesn't work. I think you're really trying to just avoid thinking in the puppet way instead trying to force some previous ideas on how things should work on it rather than figure out how it works. For example, you could just define a fact on your machines that indicate what role they have - perhaps based on your regex hostnames - and then use that in extlookup to select csv file to use. That would achieve your goal and it would be the puppet way of solving it. Currently I build nodes.pp from an external source and that provides the node with it's basic class (and hence role). So that's not needed. so if you just set a variable in the node scope you can use those in extlookup as I described - much simpler than writing an entire new type of function. And combine this with the optional paramter to extlookup that let you specify a search-first file and you get additional key/dimension you wish. Still, just write the functions you need - sounds like you know exactly what you want and its quite specific so just impliment it :) If its good and all, you can share it on forge.puppetlabs.com -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
Hi John,
While you obviously disagree with me, thanks for taking the time to
reply. I may be looking at the problem the wrong way which is why
I'm trying to figure out if that's the case and why.
john.bollin...@stjude.org (jcbollinger) writes:
On May 13, 8:07 am, jcbollinger john.bollin...@stjude.org wrote:
I agree that it is silly to suppose that extlookup() is a universal
solution, but personally, I don't take that impression from web
comments in the first place. In fact, Puppet Labs's own recently
updated style guide recommends against using extlookup(), though that
position is controversial.
Interesting. I wasn't aware of that. A colleague is starting to
use extlookup() for other functionality and if this may not be
ideal we may need to reconsider.
I also agree that for some sites, a richer / more powerful lookup
feature might be needed.
On the other hand, extlookup() is easy to set up and use, and it is
powerful enough for most needs. If you need a more powerful data
lookup function then you would be lucky to find a canned one that
meets all your needs.
Perhaps but I don't see it as being that bad, and it seems to be
the closest thing to a solution I could find at the moment. My current
recipes (perhaps incorrectly) are heavily customised in a way
which often is not just host specific or domain specific. Hence
the need to override the default settings. Other settings though
may need this thus requiring this to be more dynamic.
Moreover, I disagree with several of the opinions and conclusions in
your post:
1) You write: The extlookup() functionality only allows [...
specifying implicitly ...] where to look for this value. That is
false. Extlookup does provide for configuration of a standard set of
CSV files to search (which can be parameterized by nodes' facts), but
the function also has an optional parameter to specify a specific file
to be searched first on any given invocation.
Perhaps coming from a database background, I'd like to mirror what
seems more _natural_ and having values spread around over potentially
a large number of files seems non-intuitive.
2) You would prefer looking up data via a compound key (config_item,
lookup_value) rather than via a simple key (config_item). You rightly
observe that this would sometimes allow fewer files to be used than
are needed in the current implementation, because different values for
the same item could be distinguished by the lookup_value instead of
by the file in which they reside. The trade-off, however, is that
every lookup is more expensive, because on average the function would
need to read more lines, and to perform a more complex analysis on
each one. In your blog you suggest a DB backend, but that's not
really relevant because by that point you have a completely different
function than extlookup(). And even a DB performs multi-key queries
slower than single-key searches.
Not if they are part of the primary key. That's part of the point.
Any modern database can sustain thousands of queries a second doing this.
However, you are right small sites may not want to go this way to
start with unless configuration using a database is trivial.
3) You argue that your suggested formulation of extlookup would be
clearer as the configuration is more explicit then the current
extlookup() definition. I think you're missing the point. It would
indeed be clearer from which file the data would come, but the
objective of extlookup is to separate the *definition* of the data as
much as possible from the *use* of the data. And I like that. I
prefer that my manifests _not_ specify a bunch of details to every
extlookup() call, because that would obscure the details that are
important at the manifest level.
ok. This perhaps is a different point of view we have. extlookup()
looks like a function. As such it's nice if the function given the same
input provides the same output. Otherwise confusion is easy.
As written it's clear if you do extlookup('dns_server') that you are
getting a dns server result. In many of the examples provided
the expected use case is: check first for the host, then the domain,
then ., but that's not specific, _and_ may be important.
Where the dns_server info is coming from is IMO important, as is how
you are trying to look it up (the order of the parameters in the search).
_How_ it gets retrieved (.csv file, db backend, ...) is _not_
something which I care about as probably that mechanism is likely to
be static and stable amongst all recipes.
4) You seem to want to use an arbitrary-length list of values as your
lookup key, but I don't see how that can work while retaining the
apparent objective of reducing the number of data files that must be
associated with your configuration and retaining the use of CSV.
The .csv files are currently lookup_key,lookup_value. AFAIK there is
no limit on either parameter's size. I'm only suggesting a third
grouping/column item_type,
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
r...@devco.net (R.I.Pienaar) writes:
Let me not be taken as an extlookup() zealot. I do like the design
as it is, but that doesn't mean it is ideal -- or even good -- for every
purpose. One of the advantages of Puppet, however, is that it is
fairly easy to modify and extend. If you would like an alternative
lookup facility then by all means, write one. If you are community-
minded then you could consider sharing it; the Puppet Forge (http://
forge.puppetlabs.com/) makes it easy to do that.
You may also be interested in some of the extlookup() author's
comments about it (http://www.devco.net/archives/2009/08/31/
complex_data_and_puppet.php). He hangs out here, so you may also
hear from him directly.
So extlookup wasnt written as a be-all and end-all of data sources.
I recognized we want data and code separation and of course I could write
an ENC but I also recognized that I could do something generic that would
solve a large % of cases for people to push out the point at which they
need an ENC a bit later or in many cases entirely. I hoped I could solve
it in a way that adds some value like make it easier to write configurable
modules and to do so in a way that people without programming backgrounds
can grasp and use and make their manifests a bit better.
With this in mind I wrote extlookup aiming to solve the most common problems.
I am not against ENCs I think big/complex shops absolutely must have them and
I
think a lot of the extlookup bashers missed this point - if you are big
enough or complex enough where extlookup doesnt work for you then you've
probably reached a legitimate point where investing time in writing an
ENC is time well spent. And you're also not any more the the target audience
extlookup was created for.
That's an interesting thought and also something I'm not sure of the answer to.
Personally I'm responsible for managing quite a few database servers, but the
company of course has several more severs of different types what we manage
besides that.
When I looked at the ENCs it seems to approach (if I understand how
they should be written) the problem from a different point of
view. That is provide: a list of global variables, and a list of
classes which should be defined for each node. That requires thinking
about the problem in quite a different way to the extlookup()
functionality which basically says: give me this value, and now based
on that I can decide what to do... That's my understanding of the view
views of how to configure a system. If that's right one thing I don't like
with the ENCs is the fact that it appears to use and rely on a large number
of global variables which are then used by the classes. As systems get more
complex that strikes me as not being good as the class behaviour may
change depending on the scope of where something comes from and how it's
included
and that's not clear. I've already been tripped up by not setting a global
variable
correctly (or using the wrong name) and not noticing and of course behaviour
changes. So I still think it's clearer if you can do:
$ip_address = extlookup( parameters to get an ip address )
$interface = extlookup( parameters to get an interface name )
interface::check{ $interface: ip_address = $ip_address }
than
inteface::check{ $global_interface1: ip_address = $global_ip_address1 }
as in the latter case it's not clear where these 2 values come
from. Their names are also less intuitive and if you have a number of
these to set then it gets worse.
You should though just look at extlookups code, its extremely simple and easy
to understand (especially since Jesse rewrote my ancient crappy code) and
this
should let you hack up solutions that solves your problems.
Ive tried to keep away from Ruby. Perhaps that's a real issue but yes, given
the idea is certainly sound, extending in a way that's more appropriate to my
perceived needs seems a sane approach.
Puppet is a framework, its very hackable, its easy to write functions and
types,
easy to copy them out to all your servers and make it work in new and
interesting
ways. So I totally encourage you to use extlookup as a example and solve the
problem in your own way or even to evaluate writing your own ENC.
One way or another I am probably going to go down that path. For the reasons
stated earlier building a complete ENC means that everything depends on it.
using a extlookup() type functionaliy initially at least makes the change seem
less imposing. However, if used heavily in the end it's doing just the same
thing,
but a different way.
Thanks for your comments.
Simon
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
On May 14, 2011, at 12:55 AM, Simon J Mudd wrote: Hi John, While you obviously disagree with me, thanks for taking the time to reply. I may be looking at the problem the wrong way which is why I'm trying to figure out if that's the case and why. john.bollin...@stjude.org (jcbollinger) writes: On May 13, 8:07 am, jcbollinger john.bollin...@stjude.org wrote: 2) You would prefer looking up data via a compound key (config_item, lookup_value) rather than via a simple key (config_item). You rightly observe that this would sometimes allow fewer files to be used than are needed in the current implementation, because different values for the same item could be distinguished by the lookup_value instead of by the file in which they reside. The trade-off, however, is that every lookup is more expensive, because on average the function would need to read more lines, and to perform a more complex analysis on each one. In your blog you suggest a DB backend, but that's not really relevant because by that point you have a completely different function than extlookup(). And even a DB performs multi-key queries slower than single-key searches. Not if they are part of the primary key. That's part of the point. Any modern database can sustain thousands of queries a second doing this. However, you are right small sites may not want to go this way to start with unless configuration using a database is trivial. As a side note, putting the information in a csv file should make it much easier to version along with the rest of your files. I assume, though I could easily be wrong, that most DBs aren't easy to version control using GIT and Subversion. That means (using csv files) you can keep your data in lockstep with the version of the config files. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
kc7...@gmail.com (Patrick) writes: On May 14, 2011, at 12:55 AM, Simon J Mudd wrote: ... Not if they are part of the primary key. That's part of the point. Any modern database can sustain thousands of queries a second doing this. However, you are right small sites may not want to go this way to start with unless configuration using a database is trivial. As a side note, putting the information in a csv file should make it much easier to version along with the rest of your files. I assume, though I could easily be wrong, that most DBs aren't easy to version control using GIT and Subversion. Using GIT or subversion, no. But versioning the configuration in the database is hardly difficult to do. (If you got that far you have enough infrastructure and expertise to solve this). That means (using csv files) you can keep your data in lockstep with the version of the config files. Indeed and I think that's nice. So the only time this becomes an issue is when the load on the puppet master to provide the lookup based on the extlookup() calls is sufficiently high to be a problem. When it's a problem you override the csv lookup to work the same based on some sort of db lookup alternative. Simon -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
john.bollin...@stjude.org (jcbollinger) writes: In fact, Puppet Labs's own recently updated style guide recommends against using extlookup(), though that position is controversial. I found the URL: http://docs.puppetlabs.com/guides/style_guide.html The extlookup() Function Modules should avoid the use of extlookup() in favor of ENCs or other alternatives. The statement is clear, but it would be nice to know the reasoning. That is if there's a problem with extlookup() then it would be good to know what it is. This may have been discussed elsewhere but it shouldn't be necessary to have to go and search for this. Simon -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
sjm...@pobox.com (Simon J Mudd) writes: john.bollin...@stjude.org (jcbollinger) writes: In fact, Puppet Labs's own recently updated style guide recommends against using extlookup(), though that position is controversial. I found the URL: http://docs.puppetlabs.com/guides/style_guide.html The extlookup() Function Modules should avoid the use of extlookup() in favor of ENCs or other alternatives. The statement is clear, but it would be nice to know the reasoning. That is if there's a problem with extlookup() then it would be good to know what it is. This may have been discussed elsewhere but it shouldn't be necessary to have to go and search for this. In the end I did. http://groups.google.com/group/puppet-users/browse_thread/thread/34e4dfb4828a2835/d9f6d8d6c7bba841?show_docid=d9f6d8d6c7bba841 It is a good read. However, from the discussion a few things strike me: 1. the use of parameterised classes is recommended heavily. I've just found out about this new feature inspite of using puppet for a long time. Hence many recipes that I'm using are not paremeterised and I have a large number of similar classes. This is certainly worth fixing but is quite a painful transition to make given the number of systems in use and the chance of creating heavy breakage during that change. So if you haven't used parameterised classes much that's the first thing that needs looking at. 2. Does puppet provide an ENC? The dashboard? Again if you're not using one switching over is a heavy tasks as you have to define _everything_ first before you can use it. (that's how it seems to me). I recently attended a puppet master course and it's content was interesting, but despite there was little emphasis on this sort of thing getting the high level design right and more on the nitty gritty of the usage of the lower-level resource types. To push the use of an ENC it seems that this higher level planning would be most helpful. Also a real-life overview of how this would be implemented. Saying how to configure an ssh server doesn't count. I'd be interested in the deployment of a series of database servers, their configuration, setup, and how to take into account differences between one set of boxes and another. If I can relate to a use case that fits my own (using an ENC) and there's a way to slowly bring in more systems into using this ENC then it makes it easier to follow that path. The alternative of using an extlookup() behaviour does indeed have the inconvenience of hiding some behaviour depending on external aspects which are picked up by the extlookups, but if you already have working recipes perhaps and aren't using an ENC this gives you a way to clearly remove the conditional logic the recipes currently have. It's then a smaller step up to using an ENC if you want to go that way. For those that don't it still becomes reasonably clear where the configuration is parameterised and how/why that's done, so it's still a more flexible approach. Judging from people I've listened to at the last puppet camp there are many ways to use puppet. Sites vary from small to very large and there's no clear one solution fits all. So I'm inclined to agree that a extlookup() [ or get_config_by_key lookup] mechanism may work fine for a lot of people. As long as you know and understand the limitations of the approach you are taking that's fine. Again the ENC setup described to me at the course I went on did not mention parameterised classes, but simple variable settings and classes. I guess that will change as perhaps if you use parameterised classes the enc should do nothing other than provide a definition of the parameterised classes to use and their initial setup valus. Removing the global variables would ensure that each class's behaviour is encapsulated only by it's input parameters and that does sound much more reasonable. Simon -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
On May 13, 1:25 am, Simon J Mudd sjm...@pobox.com wrote: Hi, I have been trying to improve the coding of some of my puppet recipes and had some trouble so wrote this:http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/ Comments on the web seem to indicate that extlookup() solves all problems but I don't really see that and hence have proposed a possible way to keep the data closer together and make the extlookup() behaviour more explicit and thus IMO clearer. What do you think? Simon -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Thoughts about extlookup: http://blog.wl0.org/2011/05/thoughts-about-extlookup-in-puppet/
- Original Message - Let me not be taken as an extlookup() zealot. I do like the design as it is, but that doesn't mean it is ideal -- or even good -- for every purpose. One of the advantages of Puppet, however, is that it is fairly easy to modify and extend. If you would like an alternative lookup facility then by all means, write one. If you are community- minded then you could consider sharing it; the Puppet Forge (http:// forge.puppetlabs.com/) makes it easy to do that. You may also be interested in some of the extlookup() author's comments about it (http://www.devco.net/archives/2009/08/31/ complex_data_and_puppet.php). He hangs out here, so you may also hear from him directly. So extlookup wasnt written as a be-all and end-all of data sources. I recognized we want data and code separation and of course I could write an ENC but I also recognized that I could do something generic that would solve a large % of cases for people to push out the point at which they need an ENC a bit later or in many cases entirely. I hoped I could solve it in a way that adds some value like make it easier to write configurable modules and to do so in a way that people without programming backgrounds can grasp and use and make their manifests a bit better. With this in mind I wrote extlookup aiming to solve the most common problems. I am not against ENCs I think big/complex shops absolutely must have them and I think a lot of the extlookup bashers missed this point - if you are big enough or complex enough where extlookup doesnt work for you then you've probably reached a legitimate point where investing time in writing an ENC is time well spent. And you're also not any more the the target audience extlookup was created for. You should though just look at extlookups code, its extremely simple and easy to understand (especially since Jesse rewrote my ancient crappy code) and this should let you hack up solutions that solves your problems. Puppet is a framework, its very hackable, its easy to write functions and types, easy to copy them out to all your servers and make it work in new and interesting ways. So I totally encourage you to use extlookup as a example and solve the problem in your own way or even to evaluate writing your own ENC. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
