So we saw similar and with only the default maximum 4 puppetserver jruby instances would often get 5 or 6 clients connecting at once which in turn led to blocking and then a queue building as more clients connected. We would check port 8140 and often see over 80 established connections.
Now that we have doubled the max-active-instances to 8 and increased the JVM heap size to 4GB the concurrent connections are able to be handled and a queue no longer builds so puppet runs are much quicker and the server does not get so bogged down. I hope this helps. On Thursday, February 6, 2020 at 10:51:42 AM UTC, Martijn Grendelman wrote: > > Hi, > > A question about Puppetserver performance. > > For quite a while now, our primary Puppet server is suffering from severe > slowness and high CPU usage. We have tried to tweak its settings, giving it > more memory (Xmx = 6 GB at the moment) and toying with the > 'max-active-instances' setting to no avail. The server has 8 virtual cores > and 12 GB memory in total, to run Pupperserver, PuppetDB and PostgreSQL. > > Notably, after a restart, the performance is acceptable for a while > (several hours, up to a almost day), but then it plummets again. > > We figured that the server was just unable to cope with the load (we had > over 270 nodes talking to it in 30 min intervals), so we added a second > master that now takes more than half of that load (150 nodes). That did not > make any difference at all for the primary server. The secondary server > however, has no trouble at all dealing with the load we gave it. > > In the graph below, that displays catalog compilation times for both > servers, you can see the new master in green. It has very constant high > performance. The old master is in yellow. After a restart, the compile > times are good (not great) for a while.The first dip represents ca. 4 > hours, the second dip was 18 hours. At some point, the catalog compilation > times sky-rocket, as does the server load. 10 seconds in the graph below > corresponds to a server load of around 2, while 40 seconds corresponds to a > server load of around 5. It's the Puppetserver process using the CPU. > > The second server, the green line, has a consistent server load of around > 1, with 4 GB memory (2 GB for the Puppetserver JVM) and 2 cores (it's an > EC2 t3.medium). > > > > If I have 110 nodes, doing two runs per hour, that each take 30 seconds to > run, I would still have a concurrency of less than 2, so Puppet causing a > consistent load of 5 seems strange. My first thought would be that it's > garbage collection or something like that, but the server plenty of memory > (OS cache has 2GB). > > Any ideas on what makes the Puppetserver starting using so much CPU? What > can we try to keep it down? > > Thanks, > Martijn Grendelman > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f2d5182f-cf48-4612-9806-e7a29c9cb7c2%40googlegroups.com.
