date:20110913

Re: [Puppet Users] Storeconfigs seem slow

2011-09-13 Thread Ohad Levy

On Tue, Sep 13, 2011 at 12:41 AM, Justin Lambert
jlamb...@localmatters.com wrote:
Thanks for the response. We're using Posrgres, and the catalog build seems
a bit slow, but nothing compared to the client runtime which is where I've
been focusing. Your assessment is correct, it is just the nagios server
that is extremely slow (~20 mins), there is minimal/no impact to the client
machines.
We're at about the 100 hosts, but have closer to 1500 services - maybe we
have exceeded what storeconfigs can do then. If that is the case, is there
a recommended alternative that isn't manually maintaining config files? It
seems like most of the processing time is spent client side and I haven't
been able to figure out why. Even doing an md5sum on all of the files from
the CLI takes less than 2 seconds.

While it would require you to generate the templates yourself, you can
use foreman query script [1] to get the data you need based on all
sort of conditions.

Ohad

[1] -
https://github.com/ohadlevy/puppet-foreman/blob/master/foreman/lib/puppet/parser/functions/foreman.rb

On Mon, Sep 12, 2011 at 3:30 PM, Gabriel Filion lelu...@gmail.com wrote:

Hi,

On 11-09-12 04:43 PM, Justin Lambert wrote:
We are moving to have our nagios servers generate their nagios configs
based on what services are installed on specific hosts (as well as the
hosts registering themselves). What we have found is that our runtimes
have gone through the roof on this and I'm trying to figure out why
(summary below from a puppet run). The config pull takes a while, but
the majority of the time is spent on the client side. Running puppet
with -d has a large chunk of this time with nothing being updated on the
screen and one processor core being pegged. We're running 2.6.9 on
SL6.0 x86_64.

What db backend are you using for stored configs?

If you're using the sqlite3 backend, I'd recommend switching to mysql or
postgresql. The sqlite3 backend is mainly there for easing puppet dev,
but it's way too slow for production use..

I'm not sure if I have an unreasonable number of resources and I need to
do things differently or if I have a problem on my client I need to
address. Any insight or direction to go down to continue debugging?

Normally the client run time shouldn't change much with or without
exporting nagios resources, except on the Nagios server (the one
extracting the puppet resources).

In my experience, exporting native Nagios resources on Nagios clients
and collecting them on the Nagios server doesn't seem to be scaling very
well. But still, it's usable with around 100 hosts and 500 services..

--
Gabriel Filion

--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Storeconfigs seem slow

2011-09-13 Thread Peter Meier

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 We're at about the 100 hosts, but have closer to 1500 services - maybe we
 have exceeded what storeconfigs can do then.  If that is the case, is there
 a recommended alternative that isn't manually maintaining config files? 

Just to be clear: After the catalog has been compiled and sent to the
client, storeconfigs is not anymore involved at all.

So if your compile time is reasonable, but execution time on the client
is slow, then you have to look on the client side and not storeconfigs.

Storeconfigs just adds a few selects (and inserts) of additional
resources, that weren't directly in your manifest. But they end up the
same way in the catalog as any other resource does.

~pete
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5u+AkACgkQbwltcAfKi38OJACfXT9sumhvPpIoK0qUlH5x4JfN
AaMAnROsu3S4u3w82sI9NzDeBWdlwKvP
=xUF9
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] Storeconfigs seem slow

2011-09-13 Thread Peter Meier

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 The good thing with this method is that you can manage the module
 directory (where the different config file excerpts are stored) with
 'purge = true' so that only exported resources are present in the final
 nagios configuration (something that native types don't handle very well
 -- or actually handle very badly).

Yes, because purging the directory would unexport the resource of the
decomissioned host. But, as other resources might have been exported as
well, that can't be purged by that trick, I would still recommend to
clean up decommissioned hosts with the puppet node clean face-action,
that got partially merged into 2.7.3 and hopefully will be fully (with
the important parts for our discussion) merged into 2.7.4.

~pete
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5u+bAACgkQbwltcAfKi39iQQCeIf2CPVgA3f11wE0VxPCefZDb
OvEAn1Tw5UTSTnons6wJGyqUdO50lspD
=c84z
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

[Puppet Users] Re: Storeconfigs seem slow

2011-09-13 Thread Bart Descamps

I had a similar problem with my nagios server where the catalog run
took about 500seconds for about 100 nodes with about 1000 services,
most of which were generated with exported resources/stored config. We
use the Naginator-resources in puppet.

However, the main speed issue was not with the fetching of these
resources but with calculating the checksums of all the files I used
(to see if there were changes compared to the master). As was
suggested in the 0.25.x-documentation we put our hosts and our
services in different .cfg-files (in my case we choose to have a file
per host or hostgroup with all of its services included). Yesterday I
changed this however to only a couple of files. In services.cfg for
example all services are kept now... same thing for hosts.cfg,
hostgroups.cfg and commands.cfg.

My puppetrun now takes about 160 seconds, which is still very slow in
my opinion but a big gain compared to the 500 seconds before. The
compile time has pretty much stayed the same (about 80 seconds), but
at clientside we gained a lot of time.

Maybe you can try if the same action works for you?

The MD5-checksum of Puppet seems to be very slow indeed. We also don't
understand why it takes so long, but apparently it does.

Kind regards,
Bart

On Sep 13, 8:35 am, Peter Meier peter.me...@immerda.ch wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

  The good thing with this method is that you can manage the module
  directory (where the different config file excerpts are stored) with
  'purge = true' so that only exported resources are present in the final
  nagios configuration (something that native types don't handle very well
  -- or actually handle very badly).

 Yes, because purging the directory would unexport the resource of the
 decomissioned host. But, as other resources might have been exported as
 well, that can't be purged by that trick, I would still recommend to
 clean up decommissioned hosts with the puppet node clean face-action,
 that got partially merged into 2.7.3 and hopefully will be fully (with
 the important parts for our discussion) merged into 2.7.4.

 ~pete
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/

 iEYEARECAAYFAk5u+bAACgkQbwltcAfKi39iQQCeIf2CPVgA3f11wE0VxPCefZDb
 OvEAn1Tw5UTSTnons6wJGyqUdO50lspD
 =c84z
 -END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

[Puppet Users] Re: Storeconfigs seem slow

2011-09-13 Thread Bart Descamps

And as an additional note, there is a related bug/feature open for it:

http://projects.puppetlabs.com/issues/5650


On Sep 13, 9:49 am, Bart Descamps barty.desca...@gmail.com wrote:
 I had a similar problem with my nagios server where the catalog run
 took about 500seconds for about 100 nodes with about 1000 services,
 most of which were generated with exported resources/stored config. We
 use the Naginator-resources in puppet.

 However, the main speed issue was not with the fetching of these
 resources but with calculating the checksums of all the files I used
 (to see if there were changes compared to the master). As was
 suggested in the 0.25.x-documentation we put our hosts and our
 services in different .cfg-files (in my case we choose to have a file
 per host or hostgroup with all of its services included). Yesterday I
 changed this however to only a couple of files. In services.cfg for
 example all services are kept now... same thing for hosts.cfg,
 hostgroups.cfg and commands.cfg.

 My puppetrun now takes about 160 seconds, which is still very slow in
 my opinion but a big gain compared to the 500 seconds before. The
 compile time has pretty much stayed the same (about 80 seconds), but
 at clientside we gained a lot of time.

 Maybe you can try if the same action works for you?

 The MD5-checksum of Puppet seems to be very slow indeed. We also don't
 understand why it takes so long, but apparently it does.

 Kind regards,
 Bart

 On Sep 13, 8:35 am, Peter Meier peter.me...@immerda.ch wrote:







  -BEGIN PGP SIGNED MESSAGE-
  Hash: SHA1

   The good thing with this method is that you can manage the module
   directory (where the different config file excerpts are stored) with
   'purge = true' so that only exported resources are present in the final
   nagios configuration (something that native types don't handle very well
   -- or actually handle very badly).

  Yes, because purging the directory would unexport the resource of the
  decomissioned host. But, as other resources might have been exported as
  well, that can't be purged by that trick, I would still recommend to
  clean up decommissioned hosts with the puppet node clean face-action,
  that got partially merged into 2.7.3 and hopefully will be fully (with
  the important parts for our discussion) merged into 2.7.4.

  ~pete
  -BEGIN PGP SIGNATURE-
  Version: GnuPG v1.4.11 (GNU/Linux)
  Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/

  iEYEARECAAYFAk5u+bAACgkQbwltcAfKi39iQQCeIf2CPVgA3f11wE0VxPCefZDb
  OvEAn1Tw5UTSTnons6wJGyqUdO50lspD
  =c84z
  -END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

AW: [Puppet Users] Dashboard - Pending Tasks

2011-09-13 Thread Bernd Adamowicz

Yes, I was simply missing the worker process.

Thanks, Craig and Michael!

 -Ursprüngliche Nachricht-
 Von: puppet-users@googlegroups.com [mailto:puppet-
 us...@googlegroups.com] Im Auftrag von Craig White
 Gesendet: Montag, 12. September 2011 22:00
 An: puppet-users@googlegroups.com
 Betreff: Re: [Puppet Users] Dashboard - Pending Tasks
 
 
 On Sep 12, 2011, at 11:42 AM, Michael Stahnke wrote:
 
  There is a puppet-dashboard-workers init script that will run a
 daemon
  for you in the background to do this.
 
 I presume you are referring to my script and yes, I am aware that there
 is a script to run the daemon. My cron script also executes the same
 init but my script makes sure every hour that it is still running and
 if not, start it.
 
 --
 Craig White ~
 craig.wh...@ttiltd.com
 1.800.869.6908 ~~
 www.ttiassessments.com
 
 Need help communicating between generations at work to achieve your
 desired success? Let us help!
 
 --
 You received this message because you are subscribed to the Google
 Groups Puppet Users group.
 To post to this group, send email to puppet-users@googlegroups.com.
 To unsubscribe from this group, send email to puppet-
 users+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/puppet-users?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

[Puppet Users] Re: Facter variable $puppetversion

2011-09-13 Thread Luke Bigum

Err, what is that 0.25-5 doc folder and what RPM owns it?

rpm -qf /usr/share/doc/puppet-0.25.5

If nothing owns it, you've pretty much proved your system has old
Puppet artefacts lying around. Personally I wouldn't trust any of the
content in /usr/lib/ruby now. Is this a production system? Anything
else use Ruby on it?

I'd start to get heavy handed as this point:

tar -cvzf /tmp/usrlibruby.tar.gz /usr/lib/ruby (take a backup)
yum remove ruby puppet facter (remove all your RPMs)
find /usr/lib/ruby (what's left in your Ruby libdir?)
locate puppet (again, what's left over, should be almost nothing but /
var/lib/puppet, /var/run stuff and config files)

Now you could try reinstall and compare your backed up version of /usr/
lib/ruby with your new one.

On Sep 12, 11:47 pm, Douglas Garstang doug.garst...@gmail.com wrote:

 [root@hproxy11 ~]# locate puppet
...
 /usr/share/doc/puppet-0.25.5

-- 
You received this message because you are subscribed to the Google Groups 
Puppet Users group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

63 matches

Mail list logo