After restarting our puppetmaster host, we are seeing high CPU load and a ton of queued passenger requests. I would suspect that our puppetmaster is falling victim to the thundering herd of puppet agents, but we have deployed most of our agents using a randomized cron job. The others are using 'splay = true' if they are running puppet agent.
Some info: Puppetmaster is RHEL6 x86_64 (VM) 4 vCPU 8GB memory Apache 2.2 Passenger 3.0.7 The vhost for our puppetmaster includes these passenger options: PassengerHighPerformance on PassengerUseGlobalQueue on PassengerPoolIdleTime 1500 PassengerMaxPoolSize 24 PassengerMaxRequests 4000 PassengerStatThrottleRate 120 PassengerSpawnMethod smart RackAutoDetect Off RailsAutoDetect Off Look at all of these queued requests! Passenger-status: ----------- General information ----------- max = 24 count = 24 active = 22 inactive = 2 Waiting on global queue: 234 ----------- Application groups ----------- /etc/puppet/rack: App root: /etc/puppet/rack * PID: 24840 Sessions: 1 Processed: 528 Uptime: 9h 41m 19s * PID: 24703 Sessions: 1 Processed: 513 Uptime: 9h 42m 17s * PID: 24642 Sessions: 1 Processed: 567 Uptime: 9h 42m 29s * PID: 24827 Sessions: 1 Processed: 572 Uptime: 9h 41m 28s * PID: 24869 Sessions: 1 Processed: 530 Uptime: 9h 41m 4s * PID: 24586 Sessions: 1 Processed: 572 Uptime: 9h 42m 32s * PID: 24715 Sessions: 1 Processed: 539 Uptime: 9h 42m 17s * PID: 29046 Sessions: 1 Processed: 487 Uptime: 8h 52m 35s * PID: 24609 Sessions: 1 Processed: 575 Uptime: 9h 42m 30s * PID: 24605 Sessions: 1 Processed: 488 Uptime: 9h 42m 30s * PID: 24831 Sessions: 1 Processed: 555 Uptime: 9h 41m 27s * PID: 29036 Sessions: 1 Processed: 549 Uptime: 8h 52m 35s * PID: 24583 Sessions: 1 Processed: 565 Uptime: 9h 42m 32s * PID: 24768 Sessions: 1 Processed: 521 Uptime: 9h 41m 56s * PID: 24850 Sessions: 1 Processed: 579 Uptime: 9h 41m 16s * PID: 29050 Sessions: 1 Processed: 497 Uptime: 8h 52m 35s * PID: 24567 Sessions: 1 Processed: 536 Uptime: 9h 42m 37s * PID: 24761 Sessions: 1 Processed: 570 Uptime: 9h 42m 5s * PID: 24709 Sessions: 1 Processed: 575 Uptime: 9h 42m 17s * PID: 29042 Sessions: 1 Processed: 488 Uptime: 8h 52m 35s * PID: 24865 Sessions: 1 Processed: 559 Uptime: 9h 41m 5s * PID: 29039 Sessions: 1 Processed: 506 Uptime: 8h 52m 35s /usr/share/foreman: App root: /usr/share/foreman * PID: 24724 Sessions: 0 Processed: 5743 Uptime: 9h 42m 16s * PID: 24717 Sessions: 0 Processed: 5099 Uptime: 9h 42m 17s -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/p1uXXzKJhCcJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
