I ended up restart the mysql database on the storeconfigs server which helped this problem dramatically. But, I'm still open for more tuning options if they're out there : )
On Tuesday, July 31, 2012 8:34:45 AM UTC-5, Luke Baker wrote: > > After restarting our puppetmaster host, we are seeing high CPU load and a > ton of queued passenger requests. I would suspect that our puppetmaster is > falling victim to the thundering herd of puppet agents, but we have > deployed most of our agents using a randomized cron job. The others are > using 'splay = true' if they are running puppet agent. > > Some info: > Puppetmaster is RHEL6 x86_64 (VM) > 4 vCPU 8GB memory > Apache 2.2 > Passenger 3.0.7 > > The vhost for our puppetmaster includes these passenger options: > PassengerHighPerformance on > PassengerUseGlobalQueue on > PassengerPoolIdleTime 1500 > PassengerMaxPoolSize 24 > PassengerMaxRequests 4000 > PassengerStatThrottleRate 120 > PassengerSpawnMethod smart > RackAutoDetect Off > RailsAutoDetect Off > > > Look at all of these queued requests! > > Passenger-status: > ----------- General information ----------- > max = 24 > count = 24 > active = 22 > inactive = 2 > Waiting on global queue: 234 > > ----------- Application groups ----------- > /etc/puppet/rack: > App root: /etc/puppet/rack > * PID: 24840 Sessions: 1 Processed: 528 Uptime: 9h 41m 19s > * PID: 24703 Sessions: 1 Processed: 513 Uptime: 9h 42m 17s > * PID: 24642 Sessions: 1 Processed: 567 Uptime: 9h 42m 29s > * PID: 24827 Sessions: 1 Processed: 572 Uptime: 9h 41m 28s > * PID: 24869 Sessions: 1 Processed: 530 Uptime: 9h 41m 4s > * PID: 24586 Sessions: 1 Processed: 572 Uptime: 9h 42m 32s > * PID: 24715 Sessions: 1 Processed: 539 Uptime: 9h 42m 17s > * PID: 29046 Sessions: 1 Processed: 487 Uptime: 8h 52m 35s > * PID: 24609 Sessions: 1 Processed: 575 Uptime: 9h 42m 30s > * PID: 24605 Sessions: 1 Processed: 488 Uptime: 9h 42m 30s > * PID: 24831 Sessions: 1 Processed: 555 Uptime: 9h 41m 27s > * PID: 29036 Sessions: 1 Processed: 549 Uptime: 8h 52m 35s > * PID: 24583 Sessions: 1 Processed: 565 Uptime: 9h 42m 32s > * PID: 24768 Sessions: 1 Processed: 521 Uptime: 9h 41m 56s > * PID: 24850 Sessions: 1 Processed: 579 Uptime: 9h 41m 16s > * PID: 29050 Sessions: 1 Processed: 497 Uptime: 8h 52m 35s > * PID: 24567 Sessions: 1 Processed: 536 Uptime: 9h 42m 37s > * PID: 24761 Sessions: 1 Processed: 570 Uptime: 9h 42m 5s > * PID: 24709 Sessions: 1 Processed: 575 Uptime: 9h 42m 17s > * PID: 29042 Sessions: 1 Processed: 488 Uptime: 8h 52m 35s > * PID: 24865 Sessions: 1 Processed: 559 Uptime: 9h 41m 5s > * PID: 29039 Sessions: 1 Processed: 506 Uptime: 8h 52m 35s > > /usr/share/foreman: > App root: /usr/share/foreman > * PID: 24724 Sessions: 0 Processed: 5743 Uptime: 9h 42m 16s > * PID: 24717 Sessions: 0 Processed: 5099 Uptime: 9h 42m 17s > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/hNtnLgeBLzYJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
