Hi John,
While you obviously disagree with me, thanks for taking the time to
reply. I may be looking at the problem the wrong way which is why
I'm trying to figure out if that's the case and why.
john.bollin...@stjude.org (jcbollinger) writes:
> On May 13, 8:07 am, jcbollinger <john.bollin...@stjude.org> wrote:
>
> I agree that it is silly to suppose that extlookup() is a universal
> solution, but personally, I don't take that impression from web
> comments in the first place. In fact, Puppet Labs's own recently
> updated style guide recommends against using extlookup(), though that
> position is controversial.
Interesting. I wasn't aware of that. A colleague is starting to
use extlookup() for other functionality and if this may not be
ideal we may need to reconsider.
> I also agree that for some sites, a richer / more powerful lookup
> feature might be needed.
>
> On the other hand, extlookup() is easy to set up and use, and it is
> powerful enough for most needs. If you need a more powerful data
> lookup function then you would be lucky to find a canned one that
> meets all your needs.
Perhaps but I don't see it as being that bad, and it seems to be
the closest thing to a solution I could find at the moment. My current
recipes (perhaps incorrectly) are heavily customised in a way
which often is not just "host specific" or "domain" specific. Hence
the need to override the default settings. Other settings though
may need this thus requiring this to be more dynamic.
> Moreover, I disagree with several of the opinions and conclusions in
> your post:
>
> 1) You write: "The extlookup() functionality only allows [...
> specifying implicitly ...] where to look for this value." That is
> false. Extlookup does provide for configuration of a standard set of
> CSV files to search (which can be parameterized by nodes' facts), but
> the function also has an optional parameter to specify a specific file
> to be searched first on any given invocation.
Perhaps coming from a database background, I'd like to mirror what
seems more _natural_ and having values spread around over potentially
a large number of files seems non-intuitive.
> 2) You would prefer looking up data via a compound key (config_item,
> lookup_value) rather than via a simple key (config_item). You rightly
> observe that this would sometimes allow fewer files to be used than
> are needed in the current implementation, because different values for
> the same item could be distinguished by the "lookup_value" instead of
> by the file in which they reside. The trade-off, however, is that
> every lookup is more expensive, because on average the function would
> need to read more lines, and to perform a more complex analysis on
> each one. In your blog you suggest a DB backend, but that's not
> really relevant because by that point you have a completely different
> function than extlookup(). And even a DB performs multi-key queries
> slower than single-key searches.
Not if they are part of the primary key. That's part of the point.
Any modern database can sustain thousands of queries a second doing this.
However, you are right small sites may not want to go this way to
start with unless configuration using a database is trivial.
> 3) You argue that your suggested formulation of extlookup would be
> "clearer as the configuration is more explicit then the current
> extlookup() definition." I think you're missing the point. It would
> indeed be clearer from which file the data would come, but the
> objective of extlookup is to separate the *definition* of the data as
> much as possible from the *use* of the data. And I like that. I
> prefer that my manifests _not_ specify a bunch of details to every
> extlookup() call, because that would obscure the details that are
> important at the manifest level.
ok. This perhaps is a different point of view we have. extlookup()
looks like a function. As such it's nice if the function given the same
input provides the same output. Otherwise confusion is easy.
As written it's clear if you do extlookup('dns_server') that you are
getting a "dns server" result. In many of the examples provided
the expected use case is: check first for the host, then the domain,
then ....., but that's not specific, _and_ may be important.
Where the dns_server info is coming from is IMO important, as is how
you are trying to look it up (the order of the parameters in the search).
_How_ it gets retrieved (.csv file, db backend, ...) is _not_
something which I care about as probably that mechanism is likely to
be static and stable amongst all recipes.
> 4) You seem to want to use an arbitrary-length list of values as your
> lookup key, but I don't see how that can work while retaining the
> apparent objective of reducing the number of data files that must be
> associated with your configuration and retaining the use of CSV.
The .csv files are currently lookup_key,lookup_value. AFAIK there is
no limit on either parameter's size. I'm only suggesting a third
grouping/column item_type, and that is unlikely to change the width
of the .csv files much. My suggestion is the structure would be:
dns_server,host1.example.com,ns1.example.com
dns_server,host2.example.com,ns1.example.com
dns_server,all_domains,ns1.maindomain.com
...
The suggestion (though maybe it is not possible to implement this in
ruby or the puppet recipes) was that the keys that are looked up:
($hostname, $domain, ....) are provided as a single parameter (a list)
to the extlookup() call. Currently this is hardcoded in a list assigned to a
variable.
> 5) You write "Using the current (puppet 2.6) mechanism would require
> me, if I understand it correctly, to configure different .csv files to
> be searched for each 'configuration parameter'." Perhaps, but I think
> that's a bit of a straw man in that you are supposing a usage mode for
> extlookup that isn't well attuned to its design. Although it can
> support them, extlookup is not designed for data domains that are
> parameterized along many different dimensions. More to the point,
> such extensive parameterization is rarely needed in practice.
ok. That's perhaps the main "misunderstanding on my side". If I want to
configure a variable to be looked up, _I_ would like that information
to be kept mainly in the same place. Depending on the variable
I may want to lookup the result based on $hostname, $domain, ...
but I may want to lookup based on other variables. So having this
list not visibly part of the extlookup() call appears to make this
harder.
> 6) You write "I may also need in several places to override the
> $extlookup_precedence. In any case this can’t be overridden several
> times within the same module which is what I would need if I want to
> lookup different variables." I don't follow. In the first place, you
> can always use extlookup's third parameter to specify a particular
> file to check first.
No, not the file to check, but the parameters to lookup. that is I'd
like the extlookup() to be a generic lookup function which allows me
to choose the values I lookup, and depending on each call.
> Let me not be taken as an extlookup() zealot. I do like the design as
> it is, but that doesn't mean it is ideal -- or even good -- for every
> purpose. One of the advantages of Puppet, however, is that it is
> fairly easy to modify and extend. If you would like an alternative
> lookup facility then by all means, write one. If you are community-
> minded then you could consider sharing it; the Puppet Forge (http://
> forge.puppetlabs.com/) makes it easy to do that.
I partly wrote this to try to get an idea as to whether I'm missing the
point, something which often happens when you don't use a tool the way
it's designed, or whether perhaps more flexibility is desirable.
So far I've seen many of the extlookup() usage examples show the simple
setup a DNS server, setup ...., but not application related where the
configuration gets more complicated because the required parameters
depend less on the hostname/domain/etc but on groups of boxes setup
a certain way. Maybe I'm just missing examples to indicate
how this can be done. Nevertheless the "lack of transparency" of how
the extlookup() makes it's calls is what troubles me somewhat.
I guess I need to see why Puppet Labs recommend against using
extlookup() even though they have added it recently in 2.6.
In any case thanks for sharing your thoughts with me. Hopefully
this will help me get a better understanding of how to configure what
I want.
Simon
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
