Issue #9109 has been updated by Jacob Helwig.
Status changed from Accepted to Investigating
Assignee set to Jacob Helwig
I'm able to reproduce this with a fresh CentOS 6 x86_64 VM, with the following
setup (and their dependencies):
* Puppet 2.7.5-1.el6 (From yum.puppetlabs.com)
* Facter 1.6.2-1.el6 (From yum.puppetlabs.com)
* epel-release 6-5
* mongrel 1.1.5
* nginx 0.8.54-1.el6
* ruby 1.8.7.299-5.el6_0.1
`nginx.conf`:
user daemon daemon;
worker_processes 4;
error_log /var/log/nginx-puppet.log notice;
pid /var/run/nginx-puppet.pid;
events {
worker_connections 1024;
}
http {
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
large_client_header_buffers 16 4k;
proxy_buffers 128 4k;
keepalive_timeout 65;
tcp_nodelay on;
ssl on;
ssl_certificate
/var/lib/puppet/ssl/certs/centos-6-x86-64-master.testing-vm.perlninja.com.pem;
ssl_certificate_key
/var/lib/puppet/ssl/private_keys/centos-6-x86-64-master.testing-vm.perlninja.com.pem;
ssl_client_certificate /var/lib/puppet/ssl/ca/ca_crt.pem;
ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_session_cache shared:SSL:8m;
ssl_session_timeout 5m;
ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;
upstream puppet-production {
server 127.0.0.1:18140;
}
server {
listen 8140;
ssl_verify_client optional;
root /var/empty;
access_log /var/log/nginx/access-8140.log;
location / {
proxy_pass http://puppet-production;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify SUCCESS;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
}
Invoking a single worker on the command line via: `puppet master --servertype
mongrel --masterport 18140 --no-daemonize --verbose`
With this setup, I get the following for `/tmp/facts.yaml`, when running
`puppet agent --test`:
---
name: main
title: main
servername: localhost.localdomain
serverip: 10.0.2.15
module_name: ""
environment: production
serverversion: 2.7.5
----------------------------------------
Bug #9109: fact and node caches not written to disk with Mongrel and Puppet >
2.7.0
https://projects.puppetlabs.com/issues/9109
Author: John Guthrie
Status: Investigating
Priority: High
Assignee: Jacob Helwig
Category: server
Target version:
Affected Puppet version: 2.7.3
Keywords: puppet master discard facts
Branch:
Hello all,
I seem to have stumbled onto a possible bug where the puppet master seems to be
discarding any and all facts that a client sends to it. I've been trying a
large number of configuration changes, and nothing seems to work.
First of all, the setup:
puppet master:
<pre>
Oracle enterprise Linux 5.6 (A RHEL derivative)
puppet 2.7.3
mongrel 1.1.5
apache 2.2.3
ruby 1.8.5
</pre>
puppet client:
<pre>
Oracle enterprise Linux 5.6
puppet 2.7.3
ruby 1.8.5
</pre>
Next, the symptom:
Whenever I try to access a fact in a puppet configuration or a template it
comes back as an empty string. This holds for any fact, and seems to be true
for every client that I have tried. I do know that the puppet master is
receiving the facts since I have done packet captures using wireshark on the
puppet master machine, and one can clearly see the facts being sent to the
puppet master. And then the puppet master sends back a configuration that one
would expect if you had sent no facts at all. I see the same result when I
connect directly to the server using wget or telnet. Even if this is some
configuration error on my part, I can't think of any reason why this should
ever happen. Hence, I am filing this as a bug.
I also have a second puppet server running puppet 2.7.0 that I had set up a
while ago for testing purposes, and I am seeing the same behaviour on that
machine as well.
What I have tried to fix this:
At first, I thought this might have something to do with my storeconfigs. So I
turned that off. It didn't make any difference. In addition, I have tried all
of the following configuration changes. None of them have helped:
* disabled and removed all custom facts
* disabled pluginsync
* twiddled inventory service settings
* Turned off external node configuration and switched to using AST
* Tried to explicitly set the environment for the client
* Switched to using a simple SSL PKI for puppet instead of a certificate
heirarchy
* Opened up auth.conf to allow any REST API request from anywhere (And yes, I
am fully aware that this is a bad idea in general. This is a temporary measure
to eliminate any ACLs as culprits.)
* Tried using the global forms of the fact variables, i.e. $::ipaddress instead
of $ipaddress
* Tried using the --ignorecache flag
* Upgraded from 2.7.1 to 2.7.3
I know some of them might not make sense, but I've been in a position of not
knowing what else to do.
I have also placed some debugging messages in the following files in an attempt
to catch puppet in the act of discarding the facts:
* puppet/indirector/indirection.rb
* puppet/indirector/catalog/compiler.rb
* puppet/indirector/catalog/static_compiler.rb
Alas, the facts had already been discarded by the time the execution reached
this code. (This might be obvious to some of the core developers, but this was
my first deep look at the puppet ruby code.) If someone could tell me where in
puppet code it first picks up the request, that would be great. Then I could
actually trace the data path from its start.
I have briefly looked at the ruby gem ruby-debug, but I have not had much to
start using it.
Tests that I have been using:
I have been using two pieces of puppet configuration to test this. The first
is the following notify resource:
<pre>
notify {
"Echo-IP":
message => "IP = $::ipaddress",
withpath => true;
}
</pre>
I would expect the following:
<pre>
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: IP =
xxx.xxx.xxx.xxx
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: defined
'message' as 'IP = xxx.xxx.xxx.xxx'
info: /Stage[main]/Subversion::Server/Notify[Echo-IP]: Scheduling refresh of
(completed_Subversion::Server)
</pre>
Instead, I see the following:
<pre>
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: IP =
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: defined
'message' as 'IP = '
info: /Stage[main]/Subversion::Server/Notify[Echo-IP]: Scheduling refresh of
(completed_Subversion::Server)
</pre>
I have also been using the following suggestion from
<http://www.puppetcookbook.com/posts/see-all-client-variables.html>:
<pre>
file {
"/tmp/facts.yaml":
content => inline_template("<%= scope.to_hash.reject { |k,v| !
(k.is_a?(String) && v.is_a?(String) ) }.to_yaml %>");
}
</pre>
This places all of the variables that are available to the client in file
called /tmp/facts.yaml. I would expect to see my facts in there. Instead, I
only see the following:
<pre>
---
name: subversion::server
title: subversion::server
servername: puppet.server.in.my.domain
serverip: xxx.yyy.zzz.www
caller_module_name: &id001 subversion
domain: my.domain
puppetserver: puppet.server
ntpserver_prefer: ntp.my.domain
environment: production
module_name: *id001
serverversion: 2.7.3
</pre>
(As you can see some variables have been redacted.) Of these, domain,
ntpserver_prefer, and puppetserver, are all set manually by me in the
configuration. The rest I assume are generated by the puppet master. There is
*nothing* that is generated specifically by the client. ipaddress,
ipaddress_*, netmask, netmask_*, hostname, fqdn, kernel*, uptime, etc. are all
notably absent.
Thank you very much in advance for any help.
--
You have received this notification because you have either subscribed to it,
or are involved in it.
To change your notification preferences, please click here:
http://projects.puppetlabs.com/my/account
--
You received this message because you are subscribed to the Google Groups
"Puppet Bugs" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-bugs?hl=en.
