for posterity - a new puppetlabs ticket was created for this, https://tickets.puppetlabs.com/browse/PUP-5380, where the root cause of the problem is pinpointed.
On Tue, Oct 13, 2015 at 3:34 PM, Adam Winberg <adam.winb...@gmail.com> wrote: > ok, so I have been tinkering a bit more with this and I my problems start > when I upgrade to puppet 3.7.5. So 3.7.4 works fine, but 3.7.5 and 3.8.x is > showing some really terrible performance for me. > > Running puppetserver (jvm) with 3.7.4, 9 jruby instances and 12G Xmx I get > really good performance, catalog compilation times ranging from 1-3s. > Upgrade to 3.7.5, restart puppetserver, and the compilation takes forever > (40-90s) which in turn causes the server to get overloaded with puppet > agents which causes a lot of failed runs. There only error messages in the > logs are > "java.nio.channels.WritePendingException: null" > which probably is some timeout error caused by the excessive load on the > server. > > I can't see anything in the release notes for 3.7.5 that would explain > this, but someone out there is sure to have a better eye for that (I hope). > > I also added this to https://tickets.puppetlabs.com/browse/SERVER-819, > since it seems to be a similar problem even though I do not use puppetdb. > I'm hoping that someone out there can think of any reason for my problems. > > Regards > Adam > > > > On Thursday, September 17, 2015 at 7:03:50 AM UTC+2, Adam Winberg wrote: >> >> I actually tried the --profile option but never got it to actually record >> any 'PROFILE' log data. The logdest and debug flags works as expected, but >> no 'PROFILE' entries in the logfile. >> >> On Thu, Sep 17, 2015 at 1:02 AM, Josh Cooper <j...@puppetlabs.com> wrote: >> >>> Hi Adam, >>> >>> On Wed, Sep 16, 2015 at 3:47 AM, Adam Winberg <adam.winb...@gmail.com> >>> wrote: >>> >>>> Upgraded from 3.7.3 to 3.8.2 and have after that had very slow >>>> compilation and file serving from the puppet master. We're running our >>>> puppetmaster with apache/passenger, ca. 900 hosts. Complete puppet runs >>>> earlier took 10-15s and after update take 30-300s. Very unstable >>>> performance. >>>> >>>> Even running with just a few clients the runtimes are really slow. >>>> Decreasing the 'PassengerMaxPoolSize' makes the runs faster, but then it >>>> cant cope with an increased load. >>>> >>>> I have also tried with the jvm puppetserver with roughly the same >>>> performance (or lack thereof). I noticed the same pattern, if i set >>>> 'max-active-instances' in puppetserver.conf to 2, I get acceptable >>>> performance with a handful of clients, but with increased load it doesnt >>>> cope. If I raise the 'max-active-instances' the performance is even worse. >>>> >>>> This is running on an vmware server, 8 cpu and 12GB RAM. >>>> >>>> Cant find anything in the release notes that explain this performance >>>> drops, does anyone recognize the problem? >>>> >>>> I have now downgraded to 3.7.3 and my puppet runs are as fast as ever >>>> again. >>>> >>>> Any help appreciated. >>>> >>>> Regards >>>> Adam >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to puppet-users+unsubscr...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/puppet-users/CAFCL3HvvEC4HA4xD655sAPMBMznVdsqcQCYp5FHn%3D_SQ8TG2qg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> I would take a look at the `--profile` option as described in >>> https://puppetlabs.com/blog/tune-puppet-performance-profiler, making >>> sure to update config.ru as needed. >>> >>> Josh >>> >>> -- >>> Josh Cooper >>> Developer, Puppet Labs >>> >>> *PuppetConf 2015 <http://2015.puppetconf.com/>** is right around the >>> corner! Join us October 5-9 in Portland, OR. **Register now >>> <https://puppetconf2015.eventbrite.com/>**.* >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to puppet-users+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com >>> <https://groups.google.com/d/msgid/puppet-users/CA%2Bu97umgvfrNbJFeSRk8Rvnxkys-MAaSHEQKCeKqETy1OrXG1w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/95f54835-40bf-4b9e-9ece-e9c0187876be%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/95f54835-40bf-4b9e-9ece-e9c0187876be%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAFCL3Hug%3D0pZ79D9Xdotxi8YPivxi97%3D_F1muW3gRw_YGGdgCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
