[Puppet Users] Re: Performance issues
if you use summarize = true on your client configs, you should get a block at the end that looks like this: #puppetd -vt Changes: Total: 1 Resources: Applied: 1 Out of sync: 1 Restarted: 1 Scheduled: 317 Total: 266 Time: Config retrieval: 139.14 Cron: 0.16 Exec: 5.28 File: 19.54 Filebucket: 0.00 Mount: 0.27 Package: 2.87 Schedule: 0.00 Service: 3.08 User: 0.06 Yumrepo: 0.01 Total: 170.43 info: Sent transaction report in 0.28 seconds notice: Finished catalog run in 35.99 seconds > > We have quite a bit of puppet clients handled by Mongrel server > (4instance) and off late been noticing performance issues with the > client mostly. It takes almost 2-3mins to complete puppet runs on > clients and takes almost 100%cpus. > > I know its hard to know what the problem is unless you see what the > manifests are. My question is more on what are the tools or options > puppet has to look more into details on what its doing or spending > most time. When i run with "--debug --test" i see it mostly doing > multiples "fileserver.describe" and we are not doing any file > bucketing on the server. > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Yum issues
Thanks guys. I will give both your suggestions a try and see how i get on. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Performance issues
We have quite a bit of puppet clients handled by Mongrel server (4instance) and off late been noticing performance issues with the client mostly. It takes almost 2-3mins to complete puppet runs on clients and takes almost 100%cpus. I know its hard to know what the problem is unless you see what the manifests are. My question is more on what are the tools or options puppet has to look more into details on what its doing or spending most time. When i run with "--debug --test" i see it mostly doing multiples "fileserver.describe" and we are not doing any file bucketing on the server. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Yum issues
Hello, > We have majority of RHEL5 servers in our environment. I have noticed > in the process of trying to deploy puppet, that i am getting errors > when yum is trying to install packages. Most of the time it works > flawlessly, however sometime the following will be reported: > [...] > Does anyone have any information as to what is going wrong here? Or > what can be done to stop this from happening. The repo's i am using > are local mirrors I have configured for our company. If you're not using redhat-network. It's probably an idea to put: [main] enabled = 0 in your /etc/yum/pluginconf.d/rhnplugin.conf The only fact that yum won't try to log into RHN each time it gets run might be enough to solve your issue. Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: template flapping / classes lost?
Peter,
I made that change in both affected templates late last night. Both
services are still experiencing strange issues (flapping), but it's no
longer as regular as before (now, say, each one will be OK for 4 or 5
cycles and then flap back and forth).
I've confirmed via debugging output to syslog that the external node
classifier script *is* getting the correct FQDN and sending the correct
list of classes and parameters. I've confirmed via notice() statements
that the master is doing things right and is entering all of the correct
classes, and entering the correct branches of the conditionals.
Any other tips from you guys? Is there any strong way to debug the
templates, like do a state dump of defined classes and facts to an error
log every time the template generates?
Thanks,
Jason
Peter wrote:
> I tried using the same check in a template of mine and I got it to
> work
> I found that the closures for the if and end statements are %> and not
> -%>
>
> so your statement would look like this.
>
> <% if scope.compiler.classlist.include?("edu_rutgers_css_resnet") then
> %>
> # PUPPET: set due to presence of class 'edu_rutgers_css_resnet'
> ServerAdmin f...@bar.com
> # END PUPPET
> <% else %>
> # PUPPET: default value
> ServerAdmin r...@localhost
> # END PUPPET
> <% end %>
>
> give that a try.
>
> On Jul 28, 11:37 pm, Jason Antman wrote:
>
>> I was just about to move my first Puppet-built box into production, and
>> I'm getting a terribly troublesome error. I'm using an external node
>> classifier and am not using storedconfigs.
>>
>> We have a more-or-less generic httpd config for all of our boxen -
>> except this one. I'm generating httpd.conf from a template, so in the
>> template I have something like:
>>
>> <% if scope.compiler.classlist.include?("edu_rutgers_css_resnet") then -%>
>> # PUPPET: set due to presence of class 'edu_rutgers_css_resnet'
>> ServerAdmin f...@bar.com
>> # END PUPPET
>> <% else %>
>> # PUPPET: default value
>> ServerAdmin r...@localhost
>> # END PUPPET
>> <% end -%>
>>
>> I also generate /etc/sysconfig/iptables from a template, which includes:
>>
>> <% if scope.compiler.classlist.include?("httpd") then -%>
>> # accept port 80, added by Puppet template with "httpd" class
>> -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j
>> ACCEPT
>> <% else -%>
>> # puppet doesn't see the class 'httpd' defined, NOT opening port 80...
>> <% end -%>
>>
>> It seems that Puppet is "missing" the httpd class. When Puppet runs
>> every half hour, flaps between the if and else values for these two
>> statements, causing it to replace the config files and restart the
>> associated services. I originally thought that this was happening at
>> exactly each cycle, but it seems somewhat intermittent:
>>
>> Jul 28 02:19:56 resnet2 puppetd[24046]:
>> (//iptables/File[/etc/sysconfig/iptables]/content) content changed
>> '{md5}654732a5f76e975e1fef1907ce6c5b46' to
>> '{md5}fffc88c34693d4ee67a22c4190d1e608'
>> Jul 28 02:19:57 resnet2 puppetd[24046]: (//iptables/Service[iptables])
>> Triggering 'refresh' from 2 dependencies
>> Jul 28 04:20:20 resnet2 puppetd[24046]:
>> (//iptables/File[/etc/sysconfig/iptables]/content) content changed
>> '{md5}fffc88c34693d4ee67a22c4190d1e608' to
>> '{md5}654732a5f76e975e1fef1907ce6c5b46'
>> Jul 28 04:20:20 resnet2 puppetd[24046]: (//iptables/Service[iptables])
>> Triggering 'refresh' from 2 dependencies
>> Jul 28 06:50:44 resnet2 puppetd[24046]:
>> (//iptables/File[/etc/sysconfig/iptables]/content) content changed
>> '{md5}654732a5f76e975e1fef1907ce6c5b46' to
>> '{md5}fffc88c34693d4ee67a22c4190d1e608'
>> Jul 28 06:50:44 resnet2 puppetd[24046]: (//iptables/Service[iptables])
>> Triggering 'refresh' from 2 dependencies
>> Jul 28 07:50:56 resnet2 puppetd[24046]:
>> (//iptables/File[/etc/sysconfig/iptables]/content) content changed
>> '{md5}fffc88c34693d4ee67a22c4190d1e608' to
>> '{md5}654732a5f76e975e1fef1907ce6c5b46'
>> Jul 28 07:50:56 resnet2 puppetd[24046]: (//iptables/Service[iptables])
>> Triggering 'refresh' from 2 dependencies
>>
>> Jul 28 04:20:18 resnet2 puppetd[24046]:
>> (//httpd/File[/etc/httpd/conf/httpd.conf]/content) content changed
>> '{md5}e20ffe121bc385871c43c1c8d4c83376' to
>> '{md5}d42e551de04a44d8e9121de93795ad33'
>> Jul 28 04:20:19 resnet2 puppetd[24046]: (//httpd/Service[httpd])
>> Triggering 'refresh' from 1 dependencies
>> Jul 28 06:20:39 resnet2 puppetd[24046]:
>> (//httpd/File[/etc/httpd/conf/httpd.conf]/content) content changed
>> '{md5}d42e551de04a44d8e9121de93795ad33' to
>> '{md5}e20ffe121bc385871c43c1c8d4c83376'
>> Jul 28 06:20:40 resnet2 puppetd[24046]: (//httpd/Service[httpd])
>> Triggering 'refresh' from 1 dependencies
>> Jul 28 06:50:45 resnet2 puppetd[24046]:
>> (//httpd/File[/etc/httpd/conf/httpd.conf]/content) content changed
>> '{md5}e20ffe121bc385871c43c1c8d4c83376' to
>> '{md5}d42e551de04a44d8e9121de93795ad33'
>> Jul 28 06:50:46 resnet2 puppetd[24046]: (//httpd/Service[httpd])
>> Triggering 'refresh' from
[Puppet Users] Re: puppet recipes
On Wed, Jul 29, 2009 at 12:35 PM, David Schmitt wrote:
>
> Asif Iqbal wrote:
>> On Wed, Jul 29, 2009 at 1:57 AM, David Schmitt wrote:
>>> Asif Iqbal wrote:
So I think I should start small and simple and it may grow to a
solution that will be really useful to others.
Lets start w/ real basic.
I have 300 hosts. I like a push a user to about 100 hosts (dns
resolver type hosts) out of 300 total.
How do I set that up within puppet ?
>>> The very simplest stuff:
>>>
>>> | node "dns1", ..., "dns100" {
>>> | user { "foo": ... }
>>> | }
>>
>> this recipe worked perfect. I have seen the links you posted below and
>> I like to use them
>> slowly. I will move to that direction gradually.
>>
>> For now, the user account created perfectly. Here is the complete recipe
>>
>> (root)@sys-ubuntu:/etc/puppet/manifests# cat site.pp
>> # site.pp
>> # the .pp extension is default and not needed to add
>>
>> node "puppet-client1","puppet-client2",..."puppet-client10" {
>> user { "testuser":
>> ensure => "present",
>> uid => "102",
>> gid => "1",
>> comment => "test user",
>> home => "/export/home/testuser",
>> shell => "/bin/bash",
>> managehome => "true",
>> }
>> }
>>
>> How do I add this user to User_Alias TESTUSERS in the sudoers file on
>> all these hosts?
>> Without puppet I would ssh in to all the hosts and run `visudo' and
>> add the user in that User_Alias.
>>
>> I looked at the puppet recipe where sudeors file is kept in puppet
>> server and is pushed to
>> the puppet clients. For this I need to edit the sudoers file and my
>> recipe depends on it.
>> I like it more dynamic. I want puppet client to run the visudo and
>> append the user in User_Alias.
>> This way even if my environment grows I don't have to manage multiple
>> sudoers file on puppet master.
>
> Since there is currently no native sudo type I know of, I'd recommend
> using the concatenated_file and concatenated_file_part defines[1] from
> my "common" module[2]. Using them you can build your sudoers file on the
> nodes from a locally editable header and various parts from your manifests:
>
>
> class sudo {
> concatenated_file { "/etc/sudoers": }
> }
>
> class admin1 {
> user { admin1: }
> concatenated_file_part {
> "admin1":
> dir => "/etc/sudoers.d",
> content => "..."
> }
> }
>
> node ... {
> include admin1
> }
I am little lost. I dont see my user `testuser' here. I guess `admin1'
could be `testuser' instead if I want
to be consistent with my initial recipe?
Also what would go in the content => .. so that I can append user
`testuser' to the following entry
User_Alias TESTUSERS = user1, user2
in the sudoers file
>
>
>
>
> Regards, DavidS
>
> [1]http://git.black.co.at/?p=module-common;a=blob;f=manifests/defines/concatenated_file.pp;hb=HEAD
> [2]http://git.black.co.at/?p=module-common
>
> >
>
--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~--~~~~--~~--~--~---
[Puppet Users] Re: puppet recipes
Asif Iqbal wrote:
> On Wed, Jul 29, 2009 at 1:57 AM, David Schmitt wrote:
>> Asif Iqbal wrote:
>>> So I think I should start small and simple and it may grow to a
>>> solution that will be really useful to others.
>>>
>>> Lets start w/ real basic.
>>>
>>> I have 300 hosts. I like a push a user to about 100 hosts (dns
>>> resolver type hosts) out of 300 total.
>>>
>>> How do I set that up within puppet ?
>> The very simplest stuff:
>>
>> | node "dns1", ..., "dns100" {
>> | user { "foo": ... }
>> | }
>
> this recipe worked perfect. I have seen the links you posted below and
> I like to use them
> slowly. I will move to that direction gradually.
>
> For now, the user account created perfectly. Here is the complete recipe
>
> (root)@sys-ubuntu:/etc/puppet/manifests# cat site.pp
> # site.pp
> # the .pp extension is default and not needed to add
>
> node "puppet-client1","puppet-client2",..."puppet-client10" {
> user { "testuser":
> ensure => "present",
> uid => "102",
> gid => "1",
> comment => "test user",
> home => "/export/home/testuser",
> shell => "/bin/bash",
> managehome => "true",
> }
> }
>
> How do I add this user to User_Alias TESTUSERS in the sudoers file on
> all these hosts?
> Without puppet I would ssh in to all the hosts and run `visudo' and
> add the user in that User_Alias.
>
> I looked at the puppet recipe where sudeors file is kept in puppet
> server and is pushed to
> the puppet clients. For this I need to edit the sudoers file and my
> recipe depends on it.
> I like it more dynamic. I want puppet client to run the visudo and
> append the user in User_Alias.
> This way even if my environment grows I don't have to manage multiple
> sudoers file on puppet master.
Since there is currently no native sudo type I know of, I'd recommend
using the concatenated_file and concatenated_file_part defines[1] from
my "common" module[2]. Using them you can build your sudoers file on the
nodes from a locally editable header and various parts from your manifests:
class sudo {
concatenated_file { "/etc/sudoers": }
}
class admin1 {
user { admin1: }
concatenated_file_part {
"admin1":
dir => "/etc/sudoers.d",
content => "..."
}
}
node ... {
include admin1
}
Regards, DavidS
[1]http://git.black.co.at/?p=module-common;a=blob;f=manifests/defines/concatenated_file.pp;hb=HEAD
[2]http://git.black.co.at/?p=module-common
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~--~~~~--~~--~--~---
[Puppet Users] Re: puppet recipes
On Wed, Jul 29, 2009 at 1:57 AM, David Schmitt wrote:
>
> Asif Iqbal wrote:
>> So I think I should start small and simple and it may grow to a
>> solution that will be really useful to others.
>>
>> Lets start w/ real basic.
>>
>> I have 300 hosts. I like a push a user to about 100 hosts (dns
>> resolver type hosts) out of 300 total.
>>
>> How do I set that up within puppet ?
>
> The very simplest stuff:
>
> | node "dns1", ..., "dns100" {
> | user { "foo": ... }
> | }
this recipe worked perfect. I have seen the links you posted below and
I like to use them
slowly. I will move to that direction gradually.
For now, the user account created perfectly. Here is the complete recipe
(root)@sys-ubuntu:/etc/puppet/manifests# cat site.pp
# site.pp
# the .pp extension is default and not needed to add
node "puppet-client1","puppet-client2",..."puppet-client10" {
user { "testuser":
ensure => "present",
uid => "102",
gid => "1",
comment => "test user",
home => "/export/home/testuser",
shell => "/bin/bash",
managehome => "true",
}
}
How do I add this user to User_Alias TESTUSERS in the sudoers file on
all these hosts?
Without puppet I would ssh in to all the hosts and run `visudo' and
add the user in that User_Alias.
I looked at the puppet recipe where sudeors file is kept in puppet
server and is pushed to
the puppet clients. For this I need to edit the sudoers file and my
recipe depends on it.
I like it more dynamic. I want puppet client to run the visudo and
append the user in User_Alias.
This way even if my environment grows I don't have to manage multiple
sudoers file on puppet master.
I am going to be using puppet mainly to manage user accounts (password
and group files) and sudoers
file of various formats. Once I am comfortable with it, I will plan to
use it to install packages and
then down the road may be install patches as well. Most of my servers
are solaris.
>
> That's of course very trivial. The next steps would be to put the user
> into his own class/module where you can encapsulate the user and his
> environment (ssh key, shell configuration, ...) and use an external
> nodes classifier[1] to find your nodes instead of typing them all out.
>
> You can read many more examples on the wiki [2] and [3]. Also look at
> the references linked from the documentation main page[4].
>
>
>
> Regards, DavidS
>
>
>
> [1] http://reductivelabs.com/trac/puppet/wiki/ExternalNodes
> [2] http://reductivelabs.com/trac/puppet/wiki/PuppetModules
> [3] http://reductivelabs.com/trac/puppet/wiki/Recipes
> [4] http://reductivelabs.com/trac/puppet/wiki/DocumentationStart
>
>
>
>
>
> >
>
--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~--~~~~--~~--~--~---
[Puppet Users] Re: Yum issues
2009/7/29 josbal : > > Hi, > > We have majority of RHEL5 servers in our environment. I have noticed > in the process of trying to deploy puppet, that i am getting errors > when yum is trying to install packages. Most of the time it works > flawlessly, however sometime the following will be reported: > > change from absent to present failed: Execution of '/usr/bin/yum -d 0 - > e 0 -y install modcluster' returned 1: This system is not registered > with RHN. RHN support will be disabled. Error Downloading > Packages: modcluster-0.12.1-2.el5.x86_64: failure: > modcluster-0.12.1-2.el5.x86_64.rpm from core-1: [Errno 256] No more > mirrors to try. This is unlikely to be a puppet issue. You are most likely to be hitting an issue in your yum configuration with metadata_expire being to high. metadata_expire Time (in seconds) after which the metadata will expire. So that if the current metadata downloaded is less than this many sec- onds old then yum will not update the metadata against the repository. If you find that yum is not downloading information on updates as often as you would like lower the value of this option. If say you are updating a package in the repo (and removing the old) and it's less than metadata_expire your yum client may be looking for the old removed package. Paul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Metrics and storeconfigs
Hi Dan, I wasn't planning to include reporting so quickly, but you've inspired me to push it up a bit. you'll find some basic info at http://ohadlevy.github.com/gni/ in the code, look in app/model/report.rb and db/migrate/*report* have fun Ohad On Wed, Jul 29, 2009 at 7:40 PM, Dan Bode wrote: > > Hi Ohad, > > --ah, we extend the schema... :) > > Can you say what the reporting schema looks like? And more importantly for > me, are you combining logs and metrics in some way (if so how?) > > -thanks, > > Dan > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: storeconfigs storms?
Brice Figureau writes:
> One thing I noticed is the following: I had in two separate places
> (modules) the following pattern:
>
> if ! defined(File["a"]) {
> file { "a":
> ...
> }
> }
>
> Then when a host was coming to get its config in puppetmaster, it could
> get the File[a] defined from place1. If it was connected to another
> master process, then the same host could get the File[a] from place2.
> It isn't an issue per se, but that means that this resource has possibly
> too different tag set (one mentioning place1, the other place2).
> So each time, the tags for this resource was deleted and recreated,
> generating database load.
>
> Maybe you have such pattern in your manifests?
I don't think we have this. If we did wouldn't it be generating the
extra load all the time instead of twice a month (our sample size is
still just 2 so twice a month might not be quite right )
> But maybe the cause is external to MySQL. I have seen this kind of stuff
> happen when:
> * I/O degrades because another process is eating all the available
> disks throughput (usually backup processes or snapshotting)
> * I/O degrades because the machine is swapping (something eating
> memory), either because the swap area is on the same disks as the mysql
> data, or simply because mysql innodb buffer pool is being swapped in &
> out.
This box is dedicated to puppetmaster and MySQL and never has iowait so
I don't think this is it.
> What I suggest you is if it happens next time, is to use innotop and
> have a look to the live running queries. You might find that you have
> one or more "slow" queries.
I installed innotop. Of course it says everything's bored right now :)
I assume that when there's a slow query it sticks around on the screen?
Everything's so fast now that it disappears before I can read the SQL.
I also installed maatkit. Hopefully we'll learn something during the
next storm!
Thanks for the help!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~--~~~~--~~--~--~---
[Puppet Users] Re: Metrics and storeconfigs
Hi Ohad, --ah, we extend the schema... :) Can you say what the reporting schema looks like? And more importantly for me, are you combining logs and metrics in some way (if so how?) -thanks, Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: storeconfigs storms?
On Wed, 2009-07-29 at 09:56 +0100, Trevor Hemsley wrote: > Brice Figureau wrote: > > What I suggest you is if it happens next time, is to use innotop and > > have a look to the live running queries. You might find that you have > > one or more "slow" queries. > > > The slow query log might be a better alternative. Of course, but it needs to be activated, and to activate it you must restart mysql which might not be an option, hence my suggestion of viewing live traffic with innotop (did I say this tool is very good). -- Brice Figureau My Blog: http://www.masterzen.fr/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: storeconfigs storms?
Brice Figureau wrote: > What I suggest you is if it happens next time, is to use innotop and > have a look to the live running queries. You might find that you have > one or more "slow" queries. > The slow query log might be a better alternative. -- Trevor Hemsley Infrastructure Engineer . * C A L Y P S O * Brighton, UK OFFICE +44 (0) 1273 666 350 FAX +44 (0) 1273 666 351 . www.calypso.com This electronic-mail might contain confidential information intended only for the use by the entity named. If the reader of this message is not the intended recipient, the reader is hereby notified that any dissemination, distribution or copying is strictly prohibited. * P * /*/ Please consider the environment before printing this e-mail /*/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: storeconfigs storms?
On Tue, 2009-07-28 at 21:09 -0400, Mark Plaksin wrote:
> Brice Figureau writes:
>
> >> Our mysql questions, com_select and com_insert stats spike first.
> >> com_select and com_update are normally at around 5 and spike to 40;
> >> questions is normally around 150 and spikes to 600. Threads connected
> >> goes from around 15 to 30. After that it looks like everything queues
> >> up behind MySQL and we start getting timeouts on our ~450 clients.
>
> ...
>
> > The only reason for a storeconfig storm is that Puppet deletes all the
> > resources/tags belonging to a particular host and then recreates them,
> > so you see a lots of Inserts.
> >
> > Now the real question is why Puppet thinks there is such discrepencies
> > between the database and the live compilation.
> >
> > Are you sure you're not removing hosts from the database?
>
> Yes. Hosts that no longer exist are still in the database :)
One thing I noticed is the following: I had in two separate places
(modules) the following pattern:
if ! defined(File["a"]) {
file { "a":
...
}
}
Then when a host was coming to get its config in puppetmaster, it could
get the File[a] defined from place1. If it was connected to another
master process, then the same host could get the File[a] from place2.
It isn't an issue per se, but that means that this resource has possibly
too different tag set (one mentioning place1, the other place2).
So each time, the tags for this resource was deleted and recreated,
generating database load.
Maybe you have such pattern in your manifests?
> > What would be interesting is to activate the mysql general query log
> > (warning it will increase your load), and dig in the large log around
> > the timeframe you see the storm (you can also activate the rails log for
> > the same effect).
>
> I meant to ask whether some MySQL expert could look at our binary logs
> and figure out what happened :) Oh, I see there's a mysqlbinlog command!
> Who knew? Some quick greps of its output say the total number of
> updates and inserts from yesterday is about the same as any other day.
> Same for various hours yesterday--the hour that we got slammed doesn't
> seem to have more updates or inserts than other hours when we didn't get
> slammed.
The binlog contains only write queries (ie INSERT, UPDATE, DELETE), so
you're only seeing a part of the story.
It's easy to trigger a "storm", if you have a few queries that takes a
long time.
But maybe the cause is external to MySQL. I have seen this kind of stuff
happen when:
* I/O degrades because another process is eating all the available
disks throughput (usually backup processes or snapshotting)
* I/O degrades because the machine is swapping (something eating
memory), either because the swap area is on the same disks as the mysql
data, or simply because mysql innodb buffer pool is being swapped in &
out.
What I suggest you is if it happens next time, is to use innotop and
have a look to the live running queries. You might find that you have
one or more "slow" queries.
> > Or I remember reading that maatkit now contains a query log extractor
> > from tcpdump captures files; it is worth capturing the traffic between
> > Puppet and mysql and analyze the queries performed. Maybe you'll find
> > the issue.
>
> Maybe there's a tool which reads binary logs and tells you what caused
> the storm :)
If only :-)
--
Brice Figureau
My Blog: http://www.masterzen.fr/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~--~~~~--~~--~--~---
