But the way I'm reading this, the question of the OP is to reduce cpu
load on the agents, not the master. Puppet is currently unable to see
wether or not something changed on the machine since last run without
actually checking. I guess there's a bunch of indications that you
could use depending on the OS, but I doubt you can make that 100%
waterproof.
Walter
On Wed, Jun 20, 2012 at 5:34 PM, Pablo Fernandez
<pablo.fernan...@cscs.ch> wrote:
> Hi,
>
> I really think the original question is very good: "why do you need to
> compile all manifests again and again when there is no change on the sources
> (files/ENC/whatever input)?"
>
> Tricks like the proposed ones are clearly not the solution, and even if the
> language is not prepared for that today, this is probably something worth
> developing for the future. How many changes do you perform to the system? I
> believe 95% of the compile time (probably closer to 99%) produces exactly
> the same output again and again.
>
> BR/Pablo
>
>
>
> On 19/06/12 15:59, Brian Gallew wrote:
>
> There actually is a way to do this, though you may find it to be more
> painful to work with.
>
> Imagine, if you will, two environments: production and maintenance.
>
> The production environment is the one you're running right now, for
> production. It fully manages everything and ensures that your systems are
> all fully up-to-spec. It takes about 5 minutes for a full run of this
> manifest.
>
> The maintenance environment, on the other hand, manages /etc/passwd,
> exported resources, and a couple critical resources that change frequently.
> It doesn't check package versions, update /etc/ssh/ssh_known_hosts,
> configure backup software, etc. It's main purpose is to keep puppet
> running.
>
> Once you have these two environment configured, you move the majority of
> your hosts from "production" to "maintenance", and your puppet runtime
> drops. When you make actual changes to the manifests, you temporarily move
> all those hosts back into the production manifest so they get applied, and
> then revert them to maintenance.
>
> Another possibility for reducing overall CPU usage is to reduce the number
> of times a day that Puppet runs. If you cut it back to twice daily, then
> your total CPU usage goes from 120 minutes/host to 10 minutes/host. That
> is, in fact, how we run Puppet where I work, though we do that out of a
> culture of a "no changes during production" mindset rather than saving CPU
> cycles.
>
> Finally, consider the actual reasons for your long run times. If it's
> primarily that you are checksumming large file trees, you may want to
> consider other alternatives. While Puppet is fabulous for templated files,
> perhaps the bulk of those files could go into a bzr/svn/git/hg/whatever
> repository? Then your manifest for that directory is reduced to an exec{}
> for creating it, and either an exec{} or perhaps a cron{} for running the
> appropriate update.
>
> On Tue, Jun 19, 2012 at 6:38 AM, jcbollinger <john.bollin...@stjude.org>
> wrote:
>>
>>
>>
>> On Tuesday, June 19, 2012 5:23:42 AM UTC-5, Duncan wrote:
>>>
>>> Hi folks, I'm scratching my head with a problem with system load.
>>>
>>> When Puppet checks in every hour, runs through all our checks, then exits
>>> having confirmed that everything is indeed as expected, the vast majority of
>>> the time no changes are made. But we still load our systems with this work
>>> every hour just to make sure. Our current configuration isn't perhaps the
>>> most streamlined, taking 5 minutes for a run.
>>>
>>> The nature of our system, however, is highly virtualised with hundreds of
>>> servers running on a handful of physical hosts. It got me thinking about
>>> how to reduce the system load of Puppet runs as much as possible.
>>> Especially when there may be a move to outsource to virtualisation hosts who
>>> charge per CPU usage (but that's a business decision, not mine).
>>>
>>> Is there a prescribed method for reducing Puppet runs to only be done
>>> when necessary? Running an md5sum comparison on a file every hour isn't
>>> much CPU work, but can it be configured so that Puppet runs are triggered by
>>> file changes? I know inotify can help me here, but I was wondering if
>>> there's anything already built-in?
>>
>>
>> You seem to be asking whether there's a way to make the Puppet agent run
>> to see whether it should run. Both "no, obviously not" and "yes, it's
>> automatic" can be construed as correct answers. In a broader context,
>> anything you run to perform the kind of monitoring you suggest will consume
>> CPU. You'd have to test to see whether there was a net improvement.
>>
>> Consider also that although file checksumming is one of the more expensive
>> operations Puppet performs, files are not the only managed resources in most
>> Puppet setups. You'll need to evaluate whether it meets your needs to
>> manage anything only when some file changes.
>>
>> There are things you can do to reduce Puppet's CPU usage, however. Here
>> are some of them:
>>
>> You can lengthen the interval between runs (more than you already have
>> done).
>> You can apply a lighter-weight file checksum method (md5lite or even
>> mtime).
>> You can employ schedules to reduce the frequency at which less important
>> resources are managed.
>> You can minimize the number of resources managed on each node.
>>
>> John
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/puppet-users/-/5UkHTsXNKIsJ.
>>
>> To post to this group, send email to puppet-users@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
--
Walter Heck
--
Check out my startup: Puppet training and consulting @ http://www.olindata.com
Follow @olindata on Twitter and/or 'Like' our Facebook page at
http://www.facebook.com/olindata
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
