Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title
Eric Sorenson updated an issue
Puppet / PUP-7362
Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Change By:
Eric Sorenson
Hi,This issue was orginaly posted on Pupepet user lsit https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/puppet-users/0Pgxjk5Ru4I/goQU3tz3BQAJ . Being suggested by Eric Thompson and Henrik Lindberg, I file a ticket for PUP here. After upgrading opensource Puppet to version 4.9.4, I followed the Hiera guide https://docs.puppet.com/puppet/4.9/hiera_hierarchy.html#most-hierarchies-interpolate-variables to feed some data to it. I found that for some node, 'puppet lookup' failed to interpolate variables in yaml setting, which lead to incorrect lookup results.Steps to reproduce this issue 1. Install Puppet server on puppet.pi.sjtu.edu.cn . 2. Install Puppet agents on puppet.pi.sjtu.edu.cn, slurm1.pi.sjtu.edu.cn and mu05.pi.sjtu.edu.cn. Sign certificates for them. 3. Delete global hiera setting `/etc/puppetlabs/code/hiera.yaml`. # rm -f /etc/puppetlabs/code/hiera.yaml 4. Create an environment named `production`. # mkdir -p /etc/puppetlabs/code/environments/production/{hieradata,maifests,modules} 5. Create the Hiera conf file for the production environment in `/etc/puppetlabs/code/environments/production/hiera.yaml`{code:java}---version: 5defaults: datadir: hieradata data_hash: yaml_datahierarchy: - name: "Per-node data"path: "nodes/%{trusted.certname}.yaml" - name: "common"path: "common.yaml"{code} 6. Add common data in `/etc/puppetlabs/code/environments/production/hieradata/common.yaml`:{code:java}---group: Compute Nodes{code} 7a. Add per-node hiera data for slurm1.pi.sjtu.edu.cn in `/etc/puppetlabs/code/environments/production/hieradata/nodes/slurm1.pi.sjtu.edu.cn.yaml`:{code:java}---group: SLURM{code} 7b. Add per-node hiera data for puppet.pi.sjtu.edu.cn in `/etc/puppetlabs/code/environments/production/hieradata/nodes/puppet.pi.sjtu.edu.cn.yaml`:{code:java}---group: Puppet Server{code} 8a. Lookup group for puppet.pi.sjtu.edu.cn and successfully get "Puppet Server" stored in `puppet.pi.sjtu.edu.cn.yaml`.{code:java}# puppet lookup group --node puppet.pi.sjtu.edu.cn --explainSearching for "lookup_options" Global Data Provider (hiera configuration version 5)No such key: "lookup_options" Environment Data Provider (hiera configuration version 5)Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"Merge strategy hash Hierarchy entry "Per-node data"Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/puppet.pi.sjtu.edu.cn.yaml" Original path: "nodes/%{trusted.certname}.yaml" No such key: "lookup_options" Hierarchy entry "common"Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml" Original path: "common.yaml" No such key: "lookup_options"Searching for "group" Global Data Provider (hiera configuration version 5)No such key: "group" Environment Data Provider (hiera configuration version 5)Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"Hierarchy entry "Per-node data" Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/puppet.pi.sjtu.edu.cn.yaml"Original path: "nodes
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Henrik Lindberg assigned an issue to Lindsey Smith Puppet / PUP-7362 Puppet lookup in 4.9.4 failed to interpolate Hiera data path Change By: Henrik Lindberg Assignee: Henrik Lindberg Lindsey Smith Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Ethan Brown updated an issue Puppet / PUP-7362 Puppet lookup in 4.9.4 failed to interpolate Hiera data path Change By: Ethan Brown Team: Puppet Developer Experience Agent Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Ethan Brown updated an issue Puppet / PUP-7362 Puppet lookup in 4.9.4 failed to interpolate Hiera data path Change By: Ethan Brown Labels: triaged Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Lindsey Smith commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Jianwen Wei could you say more about why you are running lookup on a node that has not checked in with the master? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Lindsey Smith commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Nicholas Fagerlund when that happens, would you also include Rick Monro and myself Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Nicholas Fagerlund commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Yeah, these are some good concerns. Another interface issue or two: There's no way in the command to "blank out" the discovered facts; you can only overlay facts with a data file. How can the user tell whether the facts are from PuppetDB, local YAML cache for requested node, or local puppet master facts? This seems worth bringing up at the CLI leads meeting, if one or both of you would be willing to drop in. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Henrik Lindberg commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Thomas Hallgren agree, if this does not get the right UX a user may believe they got the correct answer, but it was not. It is more complicated than you first may think - as you have to consider: node data present in PDB / not present facts given on command line trusted facts given on command line given node is master node (i.e. facts can be obtained) no node given = use this node (i.e. master) A good UX should warn, but warning about too much is not good - ideally there should be warnings about missing values. The user may only use simple things like fqdn and the like in their hierarchy. Learning what exactly was missing would help in adding the missing things to facts/trusted given on the CLI. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.co
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Thomas Hallgren commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Let's assume that we set node name associated facts from the command line in case the node isn't found. I guess we could parse the name and derive domain, fqdn, hostname, trusted[domain], trusted[certname], and trusted[hostname]. Isn't there a risk that this will mask the underlying problem in a bad way and add to the confusion? An alternative approach would be to make it very apparent to the user that the node isn't found, and hence, all it's facts are missing. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Nicholas Fagerlund commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Thomas Hallgren Sure, but no one uses puppet master --compile whereas lots of people use hiera and will be switching to puppet lookup as its official replacement. So this ticket will be the first of many, since every Hiera user will now encounter this behavior. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Henrik Lindberg commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Seems reasonable to add the certname from the command line if node is not found. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Thomas Hallgren commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Nicholas Fagerlund, this is not new. The puppet master --compile has exposed this behavior for a very long time but I don't think this ticket is a duplicate. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Nicholas Fagerlund commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path I have questions still. If Jianwen had PuppetDB working, would this lookup have worked correctly? As in, was the problem just that there were no cached facts available for that node? Or did the cached facts fail to pick up trusted.certname? If it's the latter, I would call that a defect. If it's the former, I would actually still ask us to consider setting trusted.certname if the user specifies - node and we can't find an existing value for trusted.certname. Because: is there any possible situation where someone would set ` -node ` and NOT expect the facts to include that node's name? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Jianwen Wei commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Thank you all for the explaination. Still kind of confused (I'm new to Puppet), but I am confident that this is a CLI-related issue and won't affect Puppet's overall usability. I agree to mark this issue as "won't fix" and look forward to better documentation on testing your Hiera settings. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Henrik Lindberg commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Thanks Jianwen Wei for taking the time to report and giving us a chance to discuss the situation you encountered. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Henrik Lindberg commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Specifically for the lookup CLI is that you can provide the facts (including trusted) - that is required when the node has never checked in. The same thing applies to other commands - without facts for a node, what is a master --compile supposed to do? - divine the type of operating system of the node? The problem is much bigger than just 'certname'. IMO: Since it is actually impossible to do the right thing due to missing information the right behavior of puppet should really be to warn or refuse to compute. The current behavior of silently divining (by using local facts) is probably not the right thing. I think that this branch of my answer is a UX issue and is probably best addressed by taking this to the mailing list for debate - and to create follow on tickets based on what is being agreed. Then specifically to the use of $facts['clientcert'] - this is used to increase security over the old style where any node could send a fact and pretend to be the node in question. It is therefore right to not always have values (use cases where the trusted facts are indeed missing). If you want to serve values for those cases, you must then author your hierarchy and provide data that is suitable for that situation (or as stated above, make sure trusted facts exists, or alternatively provide them when testing). Documentation could also point out this difference between the old (hiera 3, use of insecure fact vs. hiera 5 recommendation to use secure fact), and what it means from a practical standpoint when testing/using lookup CLI, master compile etc. Ping Nicholas Fagerlund for this part (and Nick feel free if you want to turn this ticket into a docs ticket). Maybe this falls under a more general "how to test and troubleshoot your hiera data" (ping Lindsey Smith). Also, a ping to Charlie Sharpsteen from a support standpoint - is this is a common mistake / problem in the field? I am inclined to close this with a "won't fix" as puppet works as designed. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Thomas Hallgren assigned an issue to Henrik Lindberg Puppet / PUP-7362 Puppet lookup in 4.9.4 failed to interpolate Hiera data path Change By: Thomas Hallgren Assignee: Thomas Hallgren Henrik Lindberg Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Thomas Hallgren commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Henrik Lindberg, how do we proceed with this? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title
Thomas Hallgren commented on PUP-7362
Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path
This is, from what I can tell, not an issue with lookup. Instead, it's related to CLI commands in general and the setting of trusted.certname. It can be exemplified by using the puppet master command with an environment that has a manifests/site.pp that looks like this:
notify { $::trusted['certname']: }
and watch this command succeed:
puppet master --compile $(hostname)
while this command will fail:
puppet master --compile 'some.other.host'
since the title now is undefined. This is likely because no facts are found by the 'facts' terminus for the node in question. The 'clientcert' is a local fact. It later gets turned into the trusted['certname'].
Add Comment
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Thomas Hallgren assigned an issue to Thomas Hallgren Puppet / PUP-7362 Puppet lookup in 4.9.4 failed to interpolate Hiera data path Change By: Thomas Hallgren Assignee: Thomas Hallgren Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Henrik Lindberg commented on PUP-7362 Re: Puppet lookup in 4.9.4 failed to interpolate Hiera data path Ping Thomas Hallgren can you take a look? Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-7362) Puppet lookup in 4.9.4 failed to interpolate Hiera data path
Title: Message Title Josh Cooper updated an issue Puppet / PUP-7362 Puppet lookup in 4.9.4 failed to interpolate Hiera data path Change By: Josh Cooper Summary: Puppet lookup in 4.9. 5 4 failed to interpolate Hiera data path Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
