Hello It is still broken.
I set soft_write_failure=false I upgraded puppet on the nodes, so now the puppet master and nodes are in version 3.4.3 This is the result of the puppetdb ssl-setup : [root@el6 lofic]# puppetdb ssl-setup PEM files in /etc/puppetdb/ssl already exists, checking integrity. Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct. [root@el6 lofic]# puppetdb ssl-setup -f PEM files in /etc/puppetdb/ssl already exists, checking integrity. Overwriting existing PEM files due to -f flag Copying files: /var/lib/puppet/ssl/certs/ca.pem, /var/lib/puppet/ssl/private_keys/el6.labolinux.fr.pem and /var/lib/puppet/ssl/certs/el6.labolinux.fr.pem to /etc/puppetdb/ssl Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct. Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct. I restarted the puppetdb The catalogs are still absent. When I launch the master in debug +trace mode, I see : Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead. Warning: ActiveRecord-based storeconfigs and inventory are deprecated. See http://links.puppetlabs.com/activerecord-deprecation (at /usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:334:in `new') Debug: Using settings: adding file resource 'dblocation': 'File[/var/lib/puppet/state/clientconfigs.sqlite3]{:path=>"/var/lib/puppet/state/clientconfigs.sqlite3", :mode=>"660", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'railslog': 'File[/var/log/puppet/rails.log]{:path=>"/var/log/puppet/rails.log", :mode=>"600", :owner=>"puppet", :group=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Finishing transaction 23034320 Info: Connecting to sqlite3 database: /var/lib/puppet/state/clientconfigs.sqlite3 Debug: Configuring PuppetDB terminuses with config file /etc/puppet/puppetdb.conf The name resolution seems fine for the master, the puppetd and the nodes [root@el6 conf.d]# host beaker.labolinux.fr beaker.labolinux.fr has address 192.168.0.10 [root@el6 conf.d]# host 192.168.0.10 10.0.168.192.in-addr.arpa domain name pointer beaker.labolinux.fr. [root@el6 conf.d]# host el6.labolinux.fr el6.labolinux.fr has address 192.168.0.16 [root@el6 conf.d]# host 192.168.0.16 16.0.168.192.in-addr.arpa domain name pointer el6.labolinux.fr. [root@el6 conf.d]# host el6d.labolinux.fr el6d.labolinux.fr has address 192.168.0.63 [root@el6 conf.d]# host 192.168.0.63 63.0.168.192.in-addr.arpa domain name pointer el6d.labolinux.fr. I still have the SSL problem : # puppet node status el6.labolinux.fr --verbose --debug --trace Debug: Configuring PuppetDB terminuses with config file /etc/puppet/puppetdb.conf Debug: Failed to load library 'selinux' for feature 'selinux' Debug: Using settings: adding file resource 'confdir': 'File[/etc/puppet]{:path=>"/etc/puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Puppet::Type::User::ProviderPw: file pw does not exist Debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/uuidgen does not exist Debug: Puppet::Type::User::ProviderUser_role_add: file roleadd does not exist Debug: Failed to load library 'ldap' for feature 'ldap' Debug: Puppet::Type::User::ProviderLdap: feature ldap is missing Debug: /User[puppet]: Provider useradd does not support features libuser; not managing attribute forcelocal Debug: Puppet::Type::Group::ProviderPw: file pw does not exist Debug: Puppet::Type::Group::ProviderDirectoryservice: file /usr/bin/dscl does not exist Debug: Failed to load library 'ldap' for feature 'ldap' Debug: Puppet::Type::Group::ProviderLdap: feature ldap is missing Debug: /Group[puppet]: Provider groupadd does not support features libuser; not managing attribute forcelocal Debug: Using settings: adding file resource 'vardir': 'File[/var/lib/puppet]{:path=>"/var/lib/puppet", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'logdir': 'File[/var/log/puppet]{:path=>"/var/log/puppet", :mode=>"750", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'statedir': 'File[/var/lib/puppet/state]{:path=>"/var/lib/puppet/state", :mode=>"1755", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'rundir': 'File[/var/run/puppet]{:path=>"/var/run/puppet", :mode=>"755", :owner=>"puppet", :group=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'libdir': 'File[/var/lib/puppet/lib]{:path=>"/var/lib/puppet/lib", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hiera_config': 'File[/etc/puppet/hiera.yaml]{:path=>"/etc/puppet/hiera.yaml", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'certdir': 'File[/var/lib/puppet/ssl/certs]{:path=>"/var/lib/puppet/ssl/certs", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'ssldir': 'File[/var/lib/puppet/ssl]{:path=>"/var/lib/puppet/ssl", :mode=>"771", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:path=>"/var/lib/puppet/ssl/public_keys", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'requestdir': 'File[/var/lib/puppet/ssl/certificate_requests]{:path=>"/var/lib/puppet/ssl/certificate_requests", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'privatekeydir': 'File[/var/lib/puppet/ssl/private_keys]{:path=>"/var/lib/puppet/ssl/private_keys", :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'privatedir': 'File[/var/lib/puppet/ssl/private]{:path=>"/var/lib/puppet/ssl/private", :mode=>"750", :owner=>"puppet", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hostprivkey': 'File[/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem]{:path=>"/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem", :mode=>"600", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hostpubkey': 'File[/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem]{:path=>"/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'localcacert': 'File[/var/lib/puppet/ssl/certs/ca.pem]{:path=>"/var/lib/puppet/ssl/certs/ca.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'hostcrl': 'File[/var/lib/puppet/ssl/crl.pem]{:path=>"/var/lib/puppet/ssl/crl.pem", :mode=>"644", :owner=>"puppet", :ensure=>:file, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: Using settings: adding file resource 'pluginfactdest': 'File[/var/lib/puppet/facts.d]{:path=>"/var/lib/puppet/facts.d", :ensure=>:directory, :loglevel=>:debug, :links=>:follow, :backup=>false}' Debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] Debug: /File[/etc/puppet/hiera.yaml]: Autorequiring File[/etc/puppet] Debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet] Debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/certificate_requests]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/ssl/private_keys/beaker.labolinux.fr.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] Debug: /File[/var/lib/puppet/ssl/public_keys/beaker.labolinux.fr.pem]: Autorequiring File[/var/lib/puppet/ssl/public_keys] Debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] Debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl] Debug: /File[/var/lib/puppet/facts.d]: Autorequiring File[/var/lib/puppet] Debug: Finishing transaction 19965520 Error: Could not retrieve status for el6.labolinux.fr: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A In the puppetdb.log I see : 2014-02-28 14:13:11,984 INFO [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started SelectChannelConnector@localhost:8080 2014-02-28 14:13:12,229 INFO [clojure-agent-send-off-pool-2] [ssl.SslContextFactory] Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] 2014-02-28 14:13:12,241 INFO [clojure-agent-send-off-pool-2] [server.AbstractConnector] Started sslselectchannelconnec...@el6.labolinux.fr:8081 And something like : 2014-02-28 14:13:14,120 WARN [qtp1396798521-46] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain when I try the command = puppet node status When I run the agent on a node I see for example : 2014-02-28 14:19:09,268 INFO [command-proc-52] [puppetdb.command] [87024599-94b4-4b2c-a324-b8ea39d26bf0] [replace facts] el6d.labolinux.fr 2014-02-28 14:19:19,985 INFO [command-proc-52] [puppetdb.command] [bd719c3f-aeb4-41a1-98ac-a3524acb1107] [store report] puppet v3.4.3 - el6d.labolinux.fr and no errors, but still no catalogs in the db. It seems that /var/lib/puppet/state/clientconfigs.sqlite3 is refreshed after each agent run. Is this a normal store ? I had to install the activerecord and sqlite3 gems, otherwise the run of puppet on the nodes was complaining with a lack of activerecord. When I do a puppet run I see in the logs of the master : Info: Caching node for el6f.labolinux.fr Debug: Saved catalog to database in 1.19 seconds Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: catalog supports formats: pson b64_zlib_yaml yaml dot raw (... many messages like the previous 2 lines...) Debug: Failed to load library 'msgpack' for feature 'msgpack' Debug: file_metadata supports formats: pson b64_zlib_yaml yaml raw Debug: Received report to process from el6f.labolinux.fr Debug: Processing report from el6f.labolinux.fr with processor Puppet::Reports::Store Debug: Processing report from el6f.labolinux.fr with processor Puppet::Reports::Puppetdb Info: 'store report' command for el6f.labolinux.fr submitted to PuppetDB with UUID 860538fe-4f58-4b35-888b-86b1678df601 I'm stuck there. Louis 2014-02-25 14:51 GMT+01:00 Ken Barber <k...@puppetlabs.com>: > > with my puppetdb > > - I can't query any resource or catalog > > - exported resources are not working > > > > > > This is working : > > > > # echo '["=", ["fact", "rubyversion"], "1.8.7"]' > queryfile > > # curl -X GET http://localhost:8080/v3/nodes --data-urlencode > > query@queryfile 2>/dev/null | tail -7 > > }, { > > "name" : "el6.labolinux.fr", > > "deactivated" : null, > > "catalog_timestamp" : null, > > "facts_timestamp" : "2014-02-25T08:18:07.529Z", > > "report_timestamp" : "2014-02-25T08:18:10.018Z" > > > > I can also query with success the facts, metrics, reports > > (reports=store,puppetdb) endpoints. > > > > But it is not working with the resources or catalog endpoint : > > > > curl -X GET 'http://localhost:8080/v3/resources/User' > > -> [] > > > > curl -X GET 'http://localhost:8080/v3/resources/Package' > > -> [] > > > > curl -X GET 'http://localhost:8080/v3/resources/File' > > -> [] > > > > curl -X GET http://localhost:8080/v3/catalogs/el6.labolinux.fr > > -> { "error" : "Could not find catalog for el6.labolinux.fr" } > > > > curl -X GET http://localhost:8080/v3/nodes 2>/dev/null | grep name | > grep > > el6.labolinux.fr > > -> "name" : "el6.labolinux.fr", > > > > In addition, this is not working from the puppet master : > > > > # puppet node status el6.labolinux.fr > > Error: Could not retrieve status for el6.labolinux.fr: SSL_connect > SYSCALL > > returned=5 errno=0 state=SSLv3 read finished A > > This is the smoking gun. It looks like an SSL error is being thrown. > > > # puppet cert list el6.labolinux.fr > > + "el6.labolinux.fr" (SHA256) > > > 76:00:C9:B9:0C:31:61:9C:A5:D9:B4:49:D7:17:39:76:15:9D:18:2C:E0:07:41:6B:6C:3A:4D:68:E1:BF:65:0D > > > > I think that a consequence is that my exported resources don't work. > > > > Here is my configuration. > > > > On the master : > > > > # dpkg-query -W | egrep 'puppet(master|db)' > > puppetdb-terminus 1.6.2-1puppetlabs1 > > puppetmaster 3.4.3-1puppetlabs1 > > puppetmaster-common 3.4.3-1puppetlabs1 > > > > In puppet.conf on the master : > > > > [master] > > storeconfigs=true > > storeconfig_backend=puppetdb > > reports=store,puppetdb > > > > In routes.yaml on the master : > > > > --- > > master: > > facts: > > terminus: puppetdb > > cache: yaml > > > > In puppetdb.conf on the master : > > > > [main] > > server=el6.labolinux.fr > > port=8081 > > soft_write_failure=true > > Set the soft_write_failure to false, and you should be seeing far more > errors relating to SSL I bet. I think the problem stems from the > errors being masked and probably just being stored in the masters log. > This is the correct behaviour when this setting is true, so as to > allow the master to continue to run when PuppetDB is not operational. > > Switching to false will make the real error surface most probably. Can > you try changing that setting, restarting the puppet master and > display the results for us? Or find the error in the location where > you puppet master outputs its logs (daemon.log on Debian usually I > think?). > > > On the puppetdb node : > > > > [root@el6 ~]# rpm -qa | grep '^puppet' > > puppet-3.3.2-1.el6.noarch > > Why is your Puppet agent on the puppetdb node running an older > revision then the master? I presume you are running Puppet on the > PuppetDB node as well to manage that host correct? > > > puppetdb-1.6.2-1.el6.noarch > > > > Database backend configuration : > > > > [database] > > classname = org.postgresql.Driver > > subprotocol = postgresql > > subname = //127.0.0.1:5432/puppetdb > > username = puppetdb > > > > What am I missing ? > > This looks like a basic SSL setup issue but I have limited data to > work on. Try running "puppetdb ssl-setup", storing the results then > using "puppetdb ssl-setup -f" to force a manual repair of the > certificates PuppetDB uses. Make sure you restart PuppetDB before > trying again. > > ken. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAE4bNTmPhj2xpBiitN2e3q3%2BDmt43w%2BDLidA3j8yX_oS9h3cuA%40mail.gmail.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE9jN33eYAhA32i0zqdHi0a5bUimpQVb4oE2L8EtG3y6%2B5M%2B%3DA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
