On 02/09/17 13:02, John Baird wrote:
Henrik,
I really appreciate your patience here, but I feel like there is
something obvious that I am either not grasping or isn't spelled out in
the documentation.
Let's say I make a function called `lookup_file` which will do the
File.Read in whatever manner I need. I can call that from my manifests
to lookup the file, but how, then do I lookup the context from hiera?
It's the combination of these two things that seems abstract to me and
is preventing me from moving forward. I understand how to do both
individually, but I'm not sure how to retrieve the hiera context for the
path from within this "other" function....
I can see that any file_backend function I create chomps the key on the
first "." as we previously discussed, which means that I need to call a
function BEFORE that in order to call my file_backend from within my
file_lookup function, for instance.
How do I get the options and context to be used from the function my
manifest will be using? Let's assume in my manifest:
content => lookup_file('/tmp/somefile.txt')
does this work in my backend?
Puppet::Functions.create_function(:file_backend) do
dispatch :file_backend do
param 'String', :key
param 'Hash', :options
param 'Puppet::LookupContext', :context
end
def file_backend(key, options, context)
searchroot = options['path']
filename = "#{searchroot}/#{filename}"
You probably wanted to interpolate #{key} here, not #{filename} since it
will always be nil here.
# If file is found, return contents of file
# If file is not found, context.not_found()
end
def lookup_file(key)
options = :options
context = :context
file_backend(key, :options, :context)
end
end
That looks like a decent start of that backend. You want to give it a
name that is in your module's namespace.
I think that's the part I am struggling with... hiera knows about the
context and options, but my manifest knows (or should know) the proper
key to look up. How do I combine them so I retrieve the contents of the
file?
You (and the users of your module) interact with the backend via lookup.
The backend function should not be called directly from a manifest -
that is difficult since you have to context in which to do the lookup.
(There are ways, for testing purposes, or when delegating - but that is
not the way it is supposed to be done normally).
Your backend is called from hiera because it is configured to be called
in a hiera.yaml where you also give the path to the root of where the
files are - and you get that in the options parameter (as you already
have figured out).
There are some alternatives how you can support users - which I am
showing below:
(I use a module name of "my" here.)
Alternative 1
---
A user needs to add escapes to all '.' in the filename manually when
looking up. (Try this first to make sure your backend works).
lookup("/tmp/foo'.'bar")
Not so great.
Alternative 2
---
A user calls a function that escapes all '.' in the filename.
lookup(my::escape_dot("/tmp/foo.bar"))
You provide that escape function (see further down)
Alternative 3
---
Make your own lookup wrapping function - tell users to use that as a
convenience.
function my::lookup_file(String $file_name) {
lookup(my::escape_dot($file_name))
}
Users call that instead of lookup()
my::lookup_file('/tmp/foo.bar')
They can also call lookup() directly if they want to use the options
lookup() provides, but then they need to call the my::escape_dot() function.
my::escape_dot function
---
The my::escape_dot is something like:
Puppet::Functions.create_function(:'my::escape_dot') do
dispatch :escape_dot do
param 'String', :key
end
def escape_dot(str)
str.split('.').join("'.'")
end
end
With that escaping in place, your backend function will get the key with
the '.' without the quotes.
There are a couple of corner cases with the quoting function if file
names used as keys can contain single quotes, or if it can contain both
single and double quotes. I hope you do not have to support those cases.
(I suggest you simply document that it is illegal to use file names
containing quotes, and possible check for that and give an error in the
quoting function).
- henrik
On Sat, Sep 2, 2017 at 3:29 AM Henrik Lindberg
<henrik.lindb...@puppet.com <mailto:henrik.lindb...@puppet.com>> wrote:
On 01/09/17 01:32, John Baird wrote:
> Henrik,
>
> I think the disconnect is coming from the fact that the documentation
> only goes so far. In that, I mean, what should my lookup be from my
> manifest?
>
> Should my manifest be:
> $somefile = lookup('/etc/example.txt')
> and then specify the file_backend in my hiera.yaml?
>
> Or should my manifest be:
> $somefile = file_backend('/etc/example.txt')
> and then specify the file_backed in my hiera.yaml?
>
> If I do the former, is specifying the ":key" just implied?
> If I do the latter, I end up in a situation where the file_backend
> function is looking for more arguments than just the simple filename.
>
> I feel like that's what's not really clear, since the YAML and JSON
> backends simply append the key to the options path without ever
> specifying the actual key existing in the backend. So it's a little
> confusing how I should actually be interacting with the custom
backend.
>
The big difference between the YAML and JSON backends is that they load
files with keys in them. In your case the key is the "name" of the file!
The JSON and YAML backends are of the data_hash kind. They are called
once and must produce a hash of all the keys in the file pointed to by
the path the backend is given. You do not want that since it would mean
that you would have to slurp in all of the data files on the initial
call. That could be a problem if you have lots of large files and they
are not all needed for every compilation.
Instead you need to deliver the value for a key per lookup of that key.
The simple implementation takes the key appends it to the root path,
reads that file and returns it as the value. If the file does not exist
it should instead call $context.not_found().
- henrik
> On Monday, August 28, 2017 at 2:03:09 AM UTC-5, Henrik Lindberg
wrote:
>
> On 28/08/17 00:45, John Baird wrote:
> > The backend function receives a path in the options hash -
> that path is
> > guaranteed to exist - all non existing paths are simply
> skipped by
> > hiera
> > (see the linked docs how it works). Thus, you do not
need to
> use any of
> > the find file, etc. You can use relative paths in the
> hiera.yaml config
> > and hiera will figure it out (see the docs).
> >
> >
> > Henrik,
> >
> > I believe I fully understand the context here, however, is the
> "options"
> > hash not fully referenced within the "hiera.yaml" ?
Basically,
> how do I
> > access the lookup of the key itself from within a puppet
manifest
> as an
> > appendage of the options['path']. I really believe that
is the
> piece I
> > am just not able to connect. Can you help me understand
how to
> do this?
> > Thanks!
> >
>
> Not quite sure exactly where there is a piece of information
> missing, so
> pardon if I explain too much here.
>
> When hiera 5 processes a lookup it will visit each entry in the
> hiera.yaml. Such an entry may use one out of several ways to
specify a
> set of file paths to check for existence (path, paths, glob).
If the
> path corresponds to something existing, a call is made to the
function
> associated with that entry. It receives an options hash where
path is
> set to the path that was found to be existing. This is
repeated for all
> the existing paths found in that entry.
>
> A user may have given additional options in the options hash.
>
> If the mode of the lookup is "priority" (first found) the
search for a
> value stops when a function produces a value for the key
being searched
> for. If the search is a kind of "merge" then the search
continues until
> all values for the key have been found.
>
> You can use the fact that the hiera framework performs the
existence
> check. If you do a glob to match all of the files your
function will be
> called with each - then simply check if the leaf file name
matches the
> key being looked up. Call "not_found" for all others, and
read and
> return the content for the one that matches.
>
> By doing that you get a small advantage in that you will not be
> incurring checks for lots of file existence for all of the
keys that
> are
> irrelevant and never will have a matching file.
>
> Does this help with what you were wondering about?
> - henrik
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
> <http://puppet-on-the-edge.blogspot.se/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it,
send
> an email to puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users%2bunsubscr...@googlegroups.com>
> <mailto:puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users%2bunsubscr...@googlegroups.com>>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/puppet-users/9f084cd5-73b7-4b17-9c81-8a386af77dc4%40googlegroups.com
>
<https://groups.google.com/d/msgid/puppet-users/9f084cd5-73b7-4b17-9c81-8a386af77dc4%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to a topic in
the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/puppet-users/TNDd6f8K6H8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users%2bunsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/oodq2v%241su%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-users+unsubscr...@googlegroups.com
<mailto:puppet-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CACSjNA4Uk7hHP5zbODY4YT9BTboEVuu5WqKSJeAR83sO7ScD6A%40mail.gmail.com
<https://groups.google.com/d/msgid/puppet-users/CACSjNA4Uk7hHP5zbODY4YT9BTboEVuu5WqKSJeAR83sO7ScD6A%40mail.gmail.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/oogeir%24qon%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.
