On 9/15/14 7:28 AM, Ryan Anderson wrote:
My 3.4.3 puppet master on RHEL6.5 is using apache+passenger (4.0.39). As
I've added more agents, I've increasingly hit situations where I get
numerous errors from the web server. The problem shows up as errors on
file resources from agents such as this:
Error 503 on SERVER: <h1>This website is under heavy load</h1><p>We're
sorry, too many people are accessing this website at the same time.
We're working on this problem. Please try again later.</p> Could not
retrieve file metadata for puppet:///modules/foo/etc/cron.d/foo: Error
503 on SERVER: <h1>This website is under heavy load</h1><p>We're sorry,
too many people are accessing this website at the same time. We're
working on this problem. Please try again later.</p>
I used the default passenger settings recommended in the Puppet Pro 2nd
Edition book, and have since increased the values thinking it would
help, but it hasn't. Here's the relevant apache settings:
LoadModule passenger_module
/usr/lib/ruby/gems/1.8/gems/passenger-4.0.39/buildout/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-4.0.39
PassengerDefaultRuby /usr/bin/ruby
PassengerHighPerformance On
PassengerMaxPoolSize 20
PassengerMaxRequests 3000
PassengerPoolIdleTime 600
Any help is appreciated.
The 503 is coming from Passenger and is a sign that there are no free
processes to handle incoming requests. If you run sudo passenger-status
during these times you should see a number of requests in the queue.
Based on your config I would first check the performance of the master
during these periods. If you are using 100% of the CPU, then you may
want to reduce PassengerMaxPoolSize. The recommended
PassengerMaxPoolSize is # cores * 2 on physical hardware and as low as 1
on virtual machines. I'd try 1.5 as a starting point.
I would also check RAM usage. A Puppet master process can clock in as
high as 250 MB depending on catalog, exported resources, etc. You may
limit the PoolSize again based on RAM.
Lastly a properly set PassengerMaxPoolSize is the number of concurrent
requests you Puppet master can handle. If you reduce concurrent
requests, you may not need to increase your hardware allocation. You
might consider having your servers check in once per hour or using splay
to keep Puppet agent runs from clumping together. The usual sign is that
your Puppet master is using 5-10% CPU and then it jumps to 100% for 3-5
minutes.
If you like more information I've recently tried to write the
definitive Passenger/Puppet tuning guide after realizing most of the
information available is very poor and mostly out of date.
http://ask.puppetlabs.com/question/13433/how-should-i-tune-passenger-to-run-puppet/?answer=13434#post-id-13434
Ramin
--
You received this message because you are subscribed to the Google Groups "Puppet
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/5417035F.60703%40badapple.net.
For more options, visit https://groups.google.com/d/optout.
