Hi Kevin and others who have responded,
Thanks all for your tips. Unfortunately, no breakthroughs yet.
The current state is this:
* Both Puppetservers typically run at the latest version, currently
both 6.8.0.
* The primary server has 8 virtual cores and 12 GB of physical
(virtualized) RAM, Java is running with -Xms6g -Xmx6g.
* Max-active-instances is currently set to 7.
* This morning, I added -XX:ReservedCodeCacheSize=1g to the JVM
startup config.
* The size of our 'environments' directory is 131 MB. We currently
have 3 environments.
I've been looking at JVM stats with 'jstat', and the server doesn't
appear to spend any significant amount of time doing GC (seems to be
about 1%).
After a server restart, compilation times typically drop to 9 seconds on
average (on the secondary server, it's 5 seconds consistently), but
after a while, they go back to 30 or 40 seconds.
As I noted in my first post, our server has an average of less than 2
concurrent agents talking to it, so I can't imagine this happening due
to lack of resources. The fact that our secondary server handles a
bigger load than the primary, with a third of the memory and only 2
cores, seems to confirm this.
So:
- enough CPU power (I would think)
- enough memory
- no significant garbage collection
- Puppetserver causing a load of 5
Any more tips? Would it make sense to run PuppetDB and PostgreSQL on a
different VM?
Thanks,
Martijn Grendelman.
Op 6-2-2020 om 17:43 schreef KevinR:
> Hi Martijn,
>
> it sounds like you have a sub-optimal combination of:
>
> * The amount of JRubies
> * The total amount of java heap memory for puppetserver
> * The size of your code base
>
> This typically causes the kind of problems you're experiencing. What's
> happening in a nutshell is that puppet is loading so much code in
> memory that is starts running out of it and starts performing garbage
> collection more and more aggressively. At the end, 95% of all cpu
> cycles are spent on garbage collection and you don't have any cpu
> cycles left over to actually do work like compile catalogs...
>
> To understand how Puppet loads code into memory:
>
> Your code base is: ( [ size of your control-repo ] + [ size of all
> the modules from the Puppetfile ] ) x [ the amount of puppet code
> environments]
> So let's say:
>
> * your control repo is 5MB in size
> * all modules together are 95MB in size
> * you have 4 code environments: development, testing, acceptance and
> production
>
> That's 100MB of code to load in memory, per environment. For 4
> environments, that's 400MB.
> A different way to get this amount directly is to run *du -h
> /etc/puppetlabs/code/environments* on the puppet master and look at
> the size reported for */etc/puppetlabs/code/environments*
>
> Now every JRuby will load that entire code base into memory. So if you
> have 4 JRubies, that's 1600MB of java heap memory that's actually
> needed. You can imagine what problems will happen if there isn't this
> much heap memory configured...
>
> If you're using the defaults, Puppet will create the same amount of
> JRubies as the number of cpu cores on your master, minus 1, with a
> maximum of 4 JRubies for the system.
> If you override the defaults, you can specify any number of JRubies
> you want with the max-active-instances setting.
>
> So by default a 2-cpu puppet master will create 1 JRuby, a 4-cpu
> puppet master will create 3 JRubies, an 8-cpu puppet master will
> create 4 JRubies.
>
> So now you know how to determine the amount of java heap memory you
> need to configure, which you can do by configuring the -Xmx and -Xms
> options in the JAVA_ARGS section of the puppetserver startup command.
> Then finally make sure the host has enough physical memory available
> to provide this increased amount of java heap memory.
>
> Once enough java heap memory is provided, you'll see the cpu usage
> stay stable.
>
> Kind regards,
>
> Kevin Reeuwijk
>
> Principal Sales Engineer @ Puppet
>
>
> On Thursday, February 6, 2020 at 11:51:42 AM UTC+1, Martijn Grendelman
> wrote:
>
> Hi,
>
> A question about Puppetserver performance.
>
> For quite a while now, our primary Puppet server is suffering from
> severe slowness and high CPU usage. We have tried to tweak its
> settings, giving it more memory (Xmx = 6 GB at the moment) and
> toying with the 'max-active-instances' setting to no avail. The
> server has 8 virtual cores and 12 GB memory in total, to run
> Pupperserver, PuppetDB and PostgreSQL.
>
> Notably, after a restart, the performance is acceptable for a
> while (several hours, up to a almost day), but then it plummets again.
>
> We figured that the server was just unable to cope with the load
> (we had over 270 nodes talking to it in 30 min intervals), so we
> added a second master that now takes more than half of that load
> (150 nodes). That did not make any difference at all for the
> primary server. The secondary server however, has no trouble at
> all dealing with the load we gave it.
>
> In the graph below, that displays catalog compilation times for
> both servers, you can see the new master in green. It has very
> constant high performance. The old master is in yellow. After a
> restart, the compile times are good (not great) for a while.The
> first dip represents ca. 4 hours, the second dip was 18 hours. At
> some point, the catalog compilation times sky-rocket, as does the
> server load. 10 seconds in the graph below corresponds to a server
> load of around 2, while 40 seconds corresponds to a server load of
> around 5. It's the Puppetserver process using the CPU.
>
> The second server, the green line, has a consistent server load of
> around 1, with 4 GB memory (2 GB for the Puppetserver JVM) and 2
> cores (it's an EC2 t3.medium).
>
>
>
> If I have 110 nodes, doing two runs per hour, that each take 30
> seconds to run, I would still have a concurrency of less than 2,
> so Puppet causing a consistent load of 5 seems strange. My first
> thought would be that it's garbage collection or something like
> that, but the server plenty of memory (OS cache has 2GB).
>
> Any ideas on what makes the Puppetserver starting using so much
> CPU? What can we try to keep it down?
>
> Thanks,
> Martijn Grendelman
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/sYARGwznkOs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/e8cff298-a0a6-48a9-9bbc-a7f000926467%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/d5fc4c74-e9c0-c5da-88a2-07784767b6a2%40grendelman.net.