On Fri, Feb 7, 2020 at 2:11 AM Yvan Broccard <[email protected]> wrote:
> Hi, > About the growing state.yaml : > I have upgraded to version 6.7 in the mean time. I've seen in the code > that the issue with :statettl was fixed. > As the default setting is 32days, I still need to wait a couple of days > before being sure it works, as I still had a "tidy" statement for cleanup > up Puppet reports until recently. > I still have a huge state.yaml file, containing 200'000 lines, and > referencing 60'000 reports that have already been removed, but they are > only reports generated during the last month, with the oldest entries with > a ":checked:" date of 2020-01-06 > That's odd. Are those yaml reports that puppetserver received and stored locally due to the default puppet setting "reports=store"? Puppetserver should be storing reports in /opt/puppetlabs/server/data/puppetserver/reports/<client>/*yaml, so those should not appear in state.yaml. Can you confirm whether puppetserver is loading/updating state.yaml or only the local puppet agent running on the server. For example to confirm puppetserver isn't accessing that file: $ sudo strace -p <puppetserver java pid> -P /opt/puppetlabs/puppet/cache/state/state.yaml Only the agent should ever access the state file and I wouldn't expect reports to be in there. But it's possible a puppetserver extension (function, report processor, hiera backend, ...) could be calling Puppet::Util::Storage.load/store erroneously. > ... So I guess it works ! > > Thank you for clarifying this issue ! > > Yvan > > On Thu, Feb 6, 2020 at 6:04 PM Justin Stoller <[email protected]> wrote: > >> Yvan your issue sounds like >> https://tickets.puppetlabs.com/browse/PUP-3647, do you know if that is >> fixed now, or has regressed since then? >> >> Your issue does sound like a CodeCache or Metaspace issue. >> >> One tunable you didn't mention was "max-active-instances" I've found a >> bunch of folks that turned that very low to combat leaky code in 5.x or >> 4.x, despite it causing Puppet & the ruby runtime to be reloaded >> frequently. In 6.x that loading became much more expensive so small values >> of "max-active-instances" can be very detrimental to performance (and >> contribute to excessive Metaspace/CodeCache usage). >> >> This is also assuming that your servers are both 6.x and both at the same >> version. Can you confirm that? There are recent improvements in Server >> performance that could contribute (though probably not completely explain) >> the difference in performance your seeing if your new Server is the latest >> version and your old server hasn't been upgraded in a few months. >> >> HTH, >> Justin >> >> >> >> On Thu, Feb 6, 2020 at 8:43 AM KevinR <[email protected]> wrote: >> >>> Hi Martijn, >>> >>> it sounds like you have a sub-optimal combination of: >>> >>> - The amount of JRubies >>> - The total amount of java heap memory for puppetserver >>> - The size of your code base >>> >>> This typically causes the kind of problems you're experiencing. What's >>> happening in a nutshell is that puppet is loading so much code in memory >>> that is starts running out of it and starts performing garbage collection >>> more and more aggressively. At the end, 95% of all cpu cycles are spent on >>> garbage collection and you don't have any cpu cycles left over to actually >>> do work like compile catalogs... >>> >>> To understand how Puppet loads code into memory: >>> >>> Your code base is: ( [ size of your control-repo ] + [ size of all the >>> modules from the Puppetfile ] ) x [ the amount of puppet code >>> environments] >>> So let's say: >>> >>> - your control repo is 5MB in size >>> - all modules together are 95MB in size >>> - you have 4 code environments: development, testing, acceptance and >>> production >>> >>> That's 100MB of code to load in memory, per environment. For 4 >>> environments, that's 400MB. >>> A different way to get this amount directly is to run *du -h >>> /etc/puppetlabs/code/environments* on the puppet master and look at the >>> size reported for */etc/puppetlabs/code/environments* >>> >>> Now every JRuby will load that entire code base into memory. So if you >>> have 4 JRubies, that's 1600MB of java heap memory that's actually needed. >>> You can imagine what problems will happen if there isn't this much heap >>> memory configured... >>> >>> If you're using the defaults, Puppet will create the same amount of >>> JRubies as the number of cpu cores on your master, minus 1, with a maximum >>> of 4 JRubies for the system. >>> If you override the defaults, you can specify any number of JRubies you >>> want with the max-active-instances setting. >>> >>> So by default a 2-cpu puppet master will create 1 JRuby, a 4-cpu puppet >>> master will create 3 JRubies, an 8-cpu puppet master will create 4 JRubies. >>> >>> So now you know how to determine the amount of java heap memory you need >>> to configure, which you can do by configuring the -Xmx and -Xms options in >>> the JAVA_ARGS section of the puppetserver startup command. >>> Then finally make sure the host has enough physical memory available to >>> provide this increased amount of java heap memory. >>> >>> Once enough java heap memory is provided, you'll see the cpu usage stay >>> stable. >>> >>> Kind regards, >>> >>> Kevin Reeuwijk >>> >>> Principal Sales Engineer @ Puppet >>> >>> On Thursday, February 6, 2020 at 11:51:42 AM UTC+1, Martijn Grendelman >>> wrote: >>>> >>>> Hi, >>>> >>>> A question about Puppetserver performance. >>>> >>>> For quite a while now, our primary Puppet server is suffering from >>>> severe slowness and high CPU usage. We have tried to tweak its settings, >>>> giving it more memory (Xmx = 6 GB at the moment) and toying with the >>>> 'max-active-instances' setting to no avail. The server has 8 virtual cores >>>> and 12 GB memory in total, to run Pupperserver, PuppetDB and PostgreSQL. >>>> >>>> Notably, after a restart, the performance is acceptable for a while >>>> (several hours, up to a almost day), but then it plummets again. >>>> >>>> We figured that the server was just unable to cope with the load (we >>>> had over 270 nodes talking to it in 30 min intervals), so we added a second >>>> master that now takes more than half of that load (150 nodes). That did not >>>> make any difference at all for the primary server. The secondary server >>>> however, has no trouble at all dealing with the load we gave it. >>>> >>>> In the graph below, that displays catalog compilation times for both >>>> servers, you can see the new master in green. It has very constant high >>>> performance. The old master is in yellow. After a restart, the compile >>>> times are good (not great) for a while.The first dip represents ca. 4 >>>> hours, the second dip was 18 hours. At some point, the catalog compilation >>>> times sky-rocket, as does the server load. 10 seconds in the graph below >>>> corresponds to a server load of around 2, while 40 seconds corresponds to a >>>> server load of around 5. It's the Puppetserver process using the CPU. >>>> >>>> The second server, the green line, has a consistent server load of >>>> around 1, with 4 GB memory (2 GB for the Puppetserver JVM) and 2 cores >>>> (it's an EC2 t3.medium). >>>> >>>> >>>> >>>> If I have 110 nodes, doing two runs per hour, that each take 30 seconds >>>> to run, I would still have a concurrency of less than 2, so Puppet causing >>>> a consistent load of 5 seems strange. My first thought would be that it's >>>> garbage collection or something like that, but the server plenty of memory >>>> (OS cache has 2GB). >>>> >>>> Any ideas on what makes the Puppetserver starting using so much CPU? >>>> What can we try to keep it down? >>>> >>>> Thanks, >>>> Martijn Grendelman >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com >>> <https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXRUJPpkv%3D333Fgk0EgV7h%3DkmK-ZYji5sThpZeQEXHbFQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXRUJPpkv%3D333Fgk0EgV7h%3DkmK-ZYji5sThpZeQEXHbFQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAMuuXv2PzP7LxniTbSWDnHF59BNFMYwPqe3oH-9wn2%3DwBgrV4Q%40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-users/CAMuuXv2PzP7LxniTbSWDnHF59BNFMYwPqe3oH-9wn2%3DwBgrV4Q%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Josh Cooper | Software Engineer [email protected] -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CA%2Bu97ukhTOsrSnvanNRjWMug%3Do4ogRSTOnEsrFCwJEPO7H4-8A%40mail.gmail.com.
