When setting up some scenario runs with Gatling, we ran into a one-time condition after the initial Puppet / Puppet Server startup where concurrent catalog / node requests can fail. Here is the sequence of steps:
1) Install Puppet Server 2.5.0 (bringing in Puppet Agent 1.6.1).
2) In the /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf file, ensure that the jruby-puppet.max-active-instances is set to something greater than 1, e.g., 4, and that use-legacy-auth-conf is set to true.
3) Replace the contents of the /etc/puppetlabs/puppetserver/conf.d/auth.conf file with the following:
authorization: {
|
version: 1
|
rules: [
|
{
|
# Allow nodes to retrieve their own catalog
|
match-request: {
|
path: "/"
|
type: path
|
}
|
allow-unauthenticated: true
|
sort-order: 1
|
name: "allow all"
|
}
|
]
|
}
|
This just disables authorization so that the catalog requests being made later don't need to provide unique client certificates.
4) Start the Puppet Server service.
Note that the "node" and "facts" directories under the /opt/puppetlabs/server/data/puppetserver/yaml directory do not exist.
5) Run the following script:
#! /bin/bash
|
|
server_name=localhost
|
agent_base_name=agent1
|
|
for i in `seq 1 3`; do
|
full_agent_name="${agent_base_name}${i}"
|
curl -k -X POST "https://${server_name}:8140/puppet/v3/catalog/${full_agent_name}?environment=production" --data-urlencode "environment=production" --data-urlencode "facts_format=pson" --data-urlencode "facts={\"name\":\"${full_agent_name}\",\"values\":{\"somefact\":\"whatever\"}}" &
|
done
|
The script will perform 3 catalog requests via curl as background processes - roughly in parallel.
Expected results:
All of the catalog requests succeed.
Actual results:
In most cases, one or more of the catalog requests will fail with a message like the following being written to the puppetserver.log file:
2016-08-24 17:35:42,461 ERROR [qtp678770218-76] [puppetserver] Puppet Server Error: Failed when searching for node agent12: File exists - /opt/puppetlabs/server/data/puppetserver/yaml/node
|
In the curl output for one of the failing requests, the following stack trace could be seen:
{
|
"issue_kind": "RUNTIME_ERROR",
|
"message": "Server Error: Failed when searching for node agent12: File exists - /opt/puppetlabs/server/data/puppetserver/yaml/node",
|
"stacktrace": [
|
"org/jruby/RubyDir.java:461:in `mkdir'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/yaml.rb:27:in `save'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:200:in `find'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:295:in `find_node'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:292:in `find_node'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:332:in `node_from_request'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:50:in `find'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:121:in `do_find'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'",
|
"org/jruby/RubyArray.java:1613:in `each'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'",
|
"/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'",
|
"file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'",
|
"Puppet$$Server$$Master_917301204.gen:13:in `handleRequest'",
|
"request_handler_core.clj:273:in `invoke'",
|
"jruby_request.clj:46:in `invoke'",
|
"jruby_request.clj:31:in `invoke'",
|
"request_handler_service.clj:34:in `handle_request'",
|
"request_handler.clj:3:in `invoke'",
|
"request_handler.clj:3:in `invoke'",
|
"core.clj:2493:in `invoke'",
|
"core.clj:211:in `invoke'",
|
"core.clj:45:in `invoke'",
|
"core.clj:330:in `invoke'",
|
"core.clj:51:in `invoke'",
|
"ringutils.clj:86:in `invoke'",
|
"master_core.clj:428:in `invoke'",
|
"ring.clj:21:in `invoke'",
|
"ring.clj:12:in `invoke'",
|
"comidi.clj:249:in `invoke'",
|
"jetty9_core.clj:424:in `invoke'",
|
"normalized_uri_helpers.clj:80:in `invoke'"
|
]
|
}
|
—
I think this failure occurs because one of the catalog requests successfully causes the appropriate yaml directories to be created whereas the other requests fail on the attempt to create the directory because it already exists at the point the creation attempt is made. It seems like the master should handle this in a more benign way - by ignoring the failure to create a directory that already exists and proceed on with handling the catalog request.
This is pretty easy to workaround by just doing an initial agent run on the master before letting requests from other agents come in. Probably fairly unlikely to happen in real world deployments. All subsequent catalog requests succeed if the yaml directories exist by the time the request checks for them.
|