Hi, About the growing state.yaml : I have upgraded to version 6.7 in the mean time. I've seen in the code that the issue with :statettl was fixed. As the default setting is 32days, I still need to wait a couple of days before being sure it works, as I still had a "tidy" statement for cleanup up Puppet reports until recently. I still have a huge state.yaml file, containing 200'000 lines, and referencing 60'000 reports that have already been removed, but they are only reports generated during the last month, with the oldest entries with a ":checked:" date of 2020-01-06 ... So I guess it works !
Thank you for clarifying this issue ! Yvan On Thu, Feb 6, 2020 at 6:04 PM Justin Stoller <[email protected]> wrote: > Yvan your issue sounds like https://tickets.puppetlabs.com/browse/PUP-3647, > do you know if that is fixed now, or has regressed since then? > > Your issue does sound like a CodeCache or Metaspace issue. > > One tunable you didn't mention was "max-active-instances" I've found a > bunch of folks that turned that very low to combat leaky code in 5.x or > 4.x, despite it causing Puppet & the ruby runtime to be reloaded > frequently. In 6.x that loading became much more expensive so small values > of "max-active-instances" can be very detrimental to performance (and > contribute to excessive Metaspace/CodeCache usage). > > This is also assuming that your servers are both 6.x and both at the same > version. Can you confirm that? There are recent improvements in Server > performance that could contribute (though probably not completely explain) > the difference in performance your seeing if your new Server is the latest > version and your old server hasn't been upgraded in a few months. > > HTH, > Justin > > > > On Thu, Feb 6, 2020 at 8:43 AM KevinR <[email protected]> wrote: > >> Hi Martijn, >> >> it sounds like you have a sub-optimal combination of: >> >> - The amount of JRubies >> - The total amount of java heap memory for puppetserver >> - The size of your code base >> >> This typically causes the kind of problems you're experiencing. What's >> happening in a nutshell is that puppet is loading so much code in memory >> that is starts running out of it and starts performing garbage collection >> more and more aggressively. At the end, 95% of all cpu cycles are spent on >> garbage collection and you don't have any cpu cycles left over to actually >> do work like compile catalogs... >> >> To understand how Puppet loads code into memory: >> >> Your code base is: ( [ size of your control-repo ] + [ size of all the >> modules from the Puppetfile ] ) x [ the amount of puppet code >> environments] >> So let's say: >> >> - your control repo is 5MB in size >> - all modules together are 95MB in size >> - you have 4 code environments: development, testing, acceptance and >> production >> >> That's 100MB of code to load in memory, per environment. For 4 >> environments, that's 400MB. >> A different way to get this amount directly is to run *du -h >> /etc/puppetlabs/code/environments* on the puppet master and look at the >> size reported for */etc/puppetlabs/code/environments* >> >> Now every JRuby will load that entire code base into memory. So if you >> have 4 JRubies, that's 1600MB of java heap memory that's actually needed. >> You can imagine what problems will happen if there isn't this much heap >> memory configured... >> >> If you're using the defaults, Puppet will create the same amount of >> JRubies as the number of cpu cores on your master, minus 1, with a maximum >> of 4 JRubies for the system. >> If you override the defaults, you can specify any number of JRubies you >> want with the max-active-instances setting. >> >> So by default a 2-cpu puppet master will create 1 JRuby, a 4-cpu puppet >> master will create 3 JRubies, an 8-cpu puppet master will create 4 JRubies. >> >> So now you know how to determine the amount of java heap memory you need >> to configure, which you can do by configuring the -Xmx and -Xms options in >> the JAVA_ARGS section of the puppetserver startup command. >> Then finally make sure the host has enough physical memory available to >> provide this increased amount of java heap memory. >> >> Once enough java heap memory is provided, you'll see the cpu usage stay >> stable. >> >> Kind regards, >> >> Kevin Reeuwijk >> >> Principal Sales Engineer @ Puppet >> >> On Thursday, February 6, 2020 at 11:51:42 AM UTC+1, Martijn Grendelman >> wrote: >>> >>> Hi, >>> >>> A question about Puppetserver performance. >>> >>> For quite a while now, our primary Puppet server is suffering from >>> severe slowness and high CPU usage. We have tried to tweak its settings, >>> giving it more memory (Xmx = 6 GB at the moment) and toying with the >>> 'max-active-instances' setting to no avail. The server has 8 virtual cores >>> and 12 GB memory in total, to run Pupperserver, PuppetDB and PostgreSQL. >>> >>> Notably, after a restart, the performance is acceptable for a while >>> (several hours, up to a almost day), but then it plummets again. >>> >>> We figured that the server was just unable to cope with the load (we had >>> over 270 nodes talking to it in 30 min intervals), so we added a second >>> master that now takes more than half of that load (150 nodes). That did not >>> make any difference at all for the primary server. The secondary server >>> however, has no trouble at all dealing with the load we gave it. >>> >>> In the graph below, that displays catalog compilation times for both >>> servers, you can see the new master in green. It has very constant high >>> performance. The old master is in yellow. After a restart, the compile >>> times are good (not great) for a while.The first dip represents ca. 4 >>> hours, the second dip was 18 hours. At some point, the catalog compilation >>> times sky-rocket, as does the server load. 10 seconds in the graph below >>> corresponds to a server load of around 2, while 40 seconds corresponds to a >>> server load of around 5. It's the Puppetserver process using the CPU. >>> >>> The second server, the green line, has a consistent server load of >>> around 1, with 4 GB memory (2 GB for the Puppetserver JVM) and 2 cores >>> (it's an EC2 t3.medium). >>> >>> >>> >>> If I have 110 nodes, doing two runs per hour, that each take 30 seconds >>> to run, I would still have a concurrency of less than 2, so Puppet causing >>> a consistent load of 5 seems strange. My first thought would be that it's >>> garbage collection or something like that, but the server plenty of memory >>> (OS cache has 2GB). >>> >>> Any ideas on what makes the Puppetserver starting using so much CPU? >>> What can we try to keep it down? >>> >>> Thanks, >>> Martijn Grendelman >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com >> <https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXRUJPpkv%3D333Fgk0EgV7h%3DkmK-ZYji5sThpZeQEXHbFQ%40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXRUJPpkv%3D333Fgk0EgV7h%3DkmK-ZYji5sThpZeQEXHbFQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAMuuXv2PzP7LxniTbSWDnHF59BNFMYwPqe3oH-9wn2%3DwBgrV4Q%40mail.gmail.com.
