Hi,
About the growing state.yaml :
I have upgraded to version 6.7 in the mean time. I've seen in the code that
the issue with :statettl was fixed.
As the default setting is 32days, I still need to wait a couple of days
before being sure it works, as I still had a "tidy" statement for cleanup
up Puppet reports until recently.
I still have a huge state.yaml file, containing 200'000 lines, and
referencing 60'000 reports that have already been removed, but they are
only reports generated during the last month, with the oldest entries with
a ":checked:" date of  2020-01-06 ... So I guess it works !

Thank you for clarifying this issue !

Yvan

On Thu, Feb 6, 2020 at 6:04 PM Justin Stoller <[email protected]> wrote:

> Yvan your issue sounds like https://tickets.puppetlabs.com/browse/PUP-3647,
> do you know if that is fixed now, or has regressed since then?
>
> Your issue does sound like a CodeCache or Metaspace issue.
>
> One tunable you didn't mention was "max-active-instances" I've found a
> bunch of folks that turned that very low to combat leaky code in 5.x or
> 4.x, despite it causing Puppet & the ruby runtime to be reloaded
> frequently. In 6.x that loading became much more expensive so small values
> of "max-active-instances" can be very detrimental to performance (and
> contribute to excessive Metaspace/CodeCache usage).
>
> This is also assuming that your servers are both 6.x and both at the same
> version. Can you confirm that? There are recent improvements in Server
> performance that could contribute (though probably not completely explain)
> the difference in performance your seeing if your new Server is the latest
> version and your old server hasn't been upgraded in a few months.
>
> HTH,
> Justin
>
>
>
> On Thu, Feb 6, 2020 at 8:43 AM KevinR <[email protected]> wrote:
>
>> Hi Martijn,
>>
>> it sounds like you have a sub-optimal combination of:
>>
>>    - The amount of JRubies
>>    - The total amount of java heap memory for puppetserver
>>    - The size of your code base
>>
>> This typically causes the kind of problems you're experiencing. What's
>> happening in a nutshell is that puppet is loading so much code in memory
>> that is starts running out of it and starts performing garbage collection
>> more and more aggressively. At the end, 95% of all cpu cycles are spent on
>> garbage collection and you don't have any cpu cycles left over to actually
>> do work like compile catalogs...
>>
>> To understand how Puppet loads code into memory:
>>
>> Your code base is:  ( [ size of your control-repo ] + [ size of all the
>> modules from the Puppetfile ] )  x  [ the amount of puppet code
>> environments]
>> So let's say:
>>
>>    - your control repo is 5MB in size
>>    - all modules together are 95MB in size
>>    - you have 4 code environments: development, testing, acceptance and
>>    production
>>
>> That's 100MB of code to load in memory, per environment. For 4
>> environments, that's 400MB.
>> A different way to get this amount directly is to run *du -h
>> /etc/puppetlabs/code/environments* on the puppet master and look at the
>> size reported for */etc/puppetlabs/code/environments*
>>
>> Now every JRuby will load that entire code base into memory. So if you
>> have 4 JRubies, that's 1600MB of java heap memory that's actually needed.
>> You can imagine what problems will happen if there isn't this much heap
>> memory configured...
>>
>> If you're using the defaults, Puppet will create the same amount of
>> JRubies as the number of cpu cores on your master, minus 1, with a maximum
>> of 4 JRubies for the system.
>> If you override the defaults, you can specify any number of JRubies you
>> want with the max-active-instances setting.
>>
>> So by default a 2-cpu puppet master will create 1 JRuby, a 4-cpu puppet
>> master will create 3 JRubies, an 8-cpu puppet master will create 4 JRubies.
>>
>> So now you know how to determine the amount of java heap memory you need
>> to configure, which you can do by configuring the -Xmx and -Xms options in
>> the JAVA_ARGS section of the puppetserver startup command.
>> Then finally make sure the host has enough physical memory available to
>> provide this increased amount of java heap memory.
>>
>> Once enough java heap memory is provided, you'll see the cpu usage stay
>> stable.
>>
>> Kind regards,
>>
>> Kevin Reeuwijk
>>
>> Principal Sales Engineer @ Puppet
>>
>> On Thursday, February 6, 2020 at 11:51:42 AM UTC+1, Martijn Grendelman
>> wrote:
>>>
>>> Hi,
>>>
>>> A question about Puppetserver performance.
>>>
>>> For quite a while now, our primary Puppet server is suffering from
>>> severe slowness and high CPU usage. We have tried to tweak its settings,
>>> giving it more memory (Xmx = 6 GB at the moment) and toying with the
>>> 'max-active-instances' setting to no avail. The server has 8 virtual cores
>>> and 12 GB memory in total, to run Pupperserver, PuppetDB and PostgreSQL.
>>>
>>> Notably, after a restart, the performance is acceptable for a while
>>> (several hours, up to a almost day), but then it plummets again.
>>>
>>> We figured that the server was just unable to cope with the load (we had
>>> over 270 nodes talking to it in 30 min intervals), so we added a second
>>> master that now takes more than half of that load (150 nodes). That did not
>>> make any difference at all for the primary server. The secondary server
>>> however, has no trouble at all dealing with the load we gave it.
>>>
>>> In the graph below, that displays catalog compilation times for both
>>> servers, you can see the new master in green. It has very constant high
>>> performance. The old master is in yellow. After a restart, the compile
>>> times are good (not great) for a while.The first dip represents ca. 4
>>> hours, the second dip was 18 hours. At some point, the catalog compilation
>>> times sky-rocket, as does the server load. 10 seconds in the graph below
>>> corresponds to a server load of around 2, while 40 seconds corresponds to a
>>> server load of around 5. It's the Puppetserver process using the CPU.
>>>
>>> The second server, the green line, has a consistent server load of
>>> around 1, with 4 GB memory (2 GB for the Puppetserver JVM) and 2 cores
>>> (it's an EC2 t3.medium).
>>>
>>>
>>>
>>> If I have 110 nodes, doing two runs per hour, that each take 30 seconds
>>> to run, I would still have a concurrency of less than 2, so Puppet causing
>>> a consistent load of 5 seems strange. My first thought would be that it's
>>> garbage collection or something like that, but the server plenty of memory
>>> (OS cache has 2GB).
>>>
>>> Any ideas on what makes the Puppetserver starting using so much CPU?
>>> What can we try to keep it down?
>>>
>>> Thanks,
>>> Martijn Grendelman
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com
>> <https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXRUJPpkv%3D333Fgk0EgV7h%3DkmK-ZYji5sThpZeQEXHbFQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/puppet-users/CA%2B%3DBEqXRUJPpkv%3D333Fgk0EgV7h%3DkmK-ZYji5sThpZeQEXHbFQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAMuuXv2PzP7LxniTbSWDnHF59BNFMYwPqe3oH-9wn2%3DwBgrV4Q%40mail.gmail.com.

Reply via email to