[Puppet Users] Re: Command Exceeded Timeout on Basic Shell Commands
In case anyone is interested, we figured out what was causing this issue. Being that we run a Ruby on Rails application and naturally Puppet runs on Ruby as well, we actually load TC Malloc memory allocator for our application. By running one of the applying configuration processes in gstack, we were able to discover that it was getting stuck on libtcmalloc.so. The version of TC Malloc we were running that caused this issue was 4.2.0 but since upgrading to latest 4.2.6 we have not experienced another occurrence of this issue. On Monday, June 15, 2015 at 10:12:03 PM UTC-4, Franck wrote: We've been experiencing a lot of Command exceeded timeouts on basic shell commands using the exec type for tasks that should execute fairly fast: Jun 15 15:45:44 host1 puppet-agent[57648]: (/Stage[main]/Timezone::Utc/Exec[/bin/rm -f /etc/localtime /bin/ln -s /usr/share/zoneinfo/UTC /etc/localtime]) Command exceeded timeout Jun 10 21:15:24 host1 puppet-agent[57081]: (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/ vmware-uninstall-tools.pl]/onlyif) Check /usr/bin/test -f /usr/bin/ vmware-uninstall-tools.pl exceeded timeout Jun 10 23:56:02 host1 puppet-agent[40286]: (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/yum install -y open-vm-tools.x86_64]/unless) Check /bin/rpm -q open-vm-tools exceeded timeout All these commands can be run locally to the host and return fairly quickly, but when puppet executes them they time out. Extending the timeout is an option but ridiculous since default is 300 seconds and none of these commands should take 5 minutes or more to return. Some of the things observed is that this only affects CentOS 6.x hosts as we also have Ubuntu 14.x hosts and they do not experience these problems. Also, we've played around with different versions of the puppet agent along with different versions of Ruby and none of them had any effect as this condition persists regardless. Also, this does not seem to affect all of our CentOS 6.x hosts but only certain ones -- randomly. Running puppet agent in debug mode does not seem to uncover what's going on as it just hangs when it gets to the exec. It's very annoying and actually dangerous in some cases as the puppet agent will continue spawning multiple applying configuration processes which will cause hosts to swap memory as each takes up more and more memory and in some instances will hose them entirely. We've had to remove these manifests that cause these conditions in the interim but right now we have a lot of hosts we need to manage with puppet so we need to be able to use this. Basic info on the hosts in question: - Puppet: 3.7.5 - Ruby: 2.1.2 - CentOS 6.6 Anyone have any ideas as to what could be causing this? Thanks. Franck -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/64526946-314f-46c2-8ba5-d7fbf1ee09e8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Re: Command Exceeded Timeout on Basic Shell Commands
Thank you for your response John. You made some good suggestions especially with running it in strace to retrieve more information. In regards to your question, whether we have observed the behavior of puppet spawning multiple processes, I can definitely confirm this. When some of our nodes suffer from this, we are alerted to memory hitting a threshold and when we look at the process on these nodes, we clearly see multiple applying configurations in the list of processes. At that point, we stop the puppet daemon and kill all the applying configurations. We'll try retrieving some more information this week to see if we can narrow down some more possibilities. This does seem to affect a few random nodes but on a consistent basis so we can definitely reproduce this. Thanks again for your response. Franck On Wednesday, June 17, 2015 at 9:19:40 AM UTC-4, jcbollinger wrote: On Monday, June 15, 2015 at 9:12:03 PM UTC-5, Franck wrote: We've been experiencing a lot of Command exceeded timeouts on basic shell commands using the exec type for tasks that should execute fairly fast: Jun 15 15:45:44 host1 puppet-agent[57648]: (/Stage[main]/Timezone::Utc/Exec[/bin/rm -f /etc/localtime /bin/ln -s /usr/share/zoneinfo/UTC /etc/localtime]) Command exceeded timeout Jun 10 21:15:24 host1 puppet-agent[57081]: (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/ vmware-uninstall-tools.pl]/onlyif) Check /usr/bin/test -f /usr/bin/ vmware-uninstall-tools.pl exceeded timeout Jun 10 23:56:02 host1 puppet-agent[40286]: (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/yum install -y open-vm-tools.x86_64]/unless) Check /bin/rpm -q open-vm-tools exceeded timeout All these commands can be run locally to the host and return fairly quickly, but when puppet executes them they time out. Very strange. Extending the timeout is an option but ridiculous since default is 300 seconds and none of these commands should take 5 minutes or more to return. No, probably not a viable option. If these particular commands are not completing within the standard timeout, then there's no particular reason to think that they would *ever* complete, no matter what timeout you set. Some of the things observed is that this only affects CentOS 6.x hosts as we also have Ubuntu 14.x hosts and they do not experience these problems. Also, we've played around with different versions of the puppet agent along with different versions of Ruby and none of them had any effect as this condition persists regardless. Also, this does not seem to affect all of our CentOS 6.x hosts but only certain ones -- randomly. There is surely some pattern to which machines are affected and which not. Discovering that pattern would be a big step in solving the problem. Running puppet agent in debug mode does not seem to uncover what's going on as it just hangs when it gets to the exec. You could try running Puppet under strace to get a low-level view of exactly what Puppet gets stuck on. Nevertheless, if the problem sticks to particular computers across different Puppet versions and different Ruby versions, then the root of the problem must be outside Puppet itself. You could compare the lists of installed packages between an affected machine and a non-affected one. Perhaps the problem is caused by a specific package or package version. You should compare the catalogs applied to the machines that suffer from this problem with those for the machines that are not affected. It may help to narrow down the problem if you find that it is associated with a small number of specific resources. You should check how Puppet is running on affected machines vs. non-affected ones. Is it running as a privileged user? The same one? It's very annoying and actually dangerous in some cases as the puppet agent will continue spawning multiple applying configuration processes which will cause hosts to swap memory as each takes up more and more memory and in some instances will hose them entirely. Have you actually observed that behavior? If so, then something is dreadfully wrong. Puppet should never start a new catalog run when one is already underway. It has safeguards in place to prevent that. If you have stumbled across a way in which those can be circumvented, then I'm sure PuppetLabs would appreciate a bug report. We've had to remove these manifests that cause these conditions in the interim but right now we have a lot of hosts we need to manage with puppet so we need to be able to use this. Basic info on the hosts in question: - Puppet: 3.7.5 - Ruby: 2.1.2 - CentOS 6.6 Anyone have any ideas as to what could be causing this? You haven't given us much to work with, and I, at least, have never before heard of such an issue. I do not know what is causing it, but I suggest you try narrowing down
[Puppet Users] Command Exceeded Timeout on Basic Shell Commands
We've been experiencing a lot of Command exceeded timeouts on basic shell commands using the exec type for tasks that should execute fairly fast: Jun 15 15:45:44 host1 puppet-agent[57648]: (/Stage[main]/Timezone::Utc/Exec[/bin/rm -f /etc/localtime /bin/ln -s /usr/share/zoneinfo/UTC /etc/localtime]) Command exceeded timeout Jun 10 21:15:24 host1 puppet-agent[57081]: (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/vmware-uninstall-tools.pl]/onlyif) Check /usr/bin/test -f /usr/bin/vmware-uninstall-tools.pl exceeded timeout Jun 10 23:56:02 host1 puppet-agent[40286]: (/Stage[main]/Open-vm-tools::Package/Exec[/usr/bin/yum install -y open-vm-tools.x86_64]/unless) Check /bin/rpm -q open-vm-tools exceeded timeout All these commands can be run locally to the host and return fairly quickly, but when puppet executes them they time out. Extending the timeout is an option but ridiculous since default is 300 seconds and none of these commands should take 5 minutes or more to return. Some of the things observed is that this only affects CentOS 6.x hosts as we also have Ubuntu 14.x hosts and they do not experience these problems. Also, we've played around with different versions of the puppet agent along with different versions of Ruby and none of them had any effect as this condition persists regardless. Also, this does not seem to affect all of our CentOS 6.x hosts but only certain ones -- randomly. Running puppet agent in debug mode does not seem to uncover what's going on as it just hangs when it gets to the exec. It's very annoying and actually dangerous in some cases as the puppet agent will continue spawning multiple applying configuration processes which will cause hosts to swap memory as each takes up more and more memory and in some instances will hose them entirely. We've had to remove these manifests that cause these conditions in the interim but right now we have a lot of hosts we need to manage with puppet so we need to be able to use this. Basic info on the hosts in question: - Puppet: 3.7.5 - Ruby: 2.1.2 - CentOS 6.6 Anyone have any ideas as to what could be causing this? Thanks. Franck -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f2071040-32e1-4c9b-8656-5a33650c67ca%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Puppetd daemon dies after running
): Autorequiring File[/var/lib/puppet] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/certs] (debug): Autorequiring File[/var/lib/puppet/ssl] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/private_keys/ ps-prod-util1.plansource.com.pem] (debug): Autorequiring File[/var/lib/ puppet/ssl/private_keys] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/public_keys/ ps-prod-util1.plansource.com.pem] (debug): Autorequiring File[/var/lib/ puppet/ssl/public_keys] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/facts] (debug): Autorequiring File[/var/lib/puppet] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/ certificate_requests] (debug): Autorequiring File[/var/lib/puppet/ssl] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/certs/ca.pem] (debug): Autorequiring File[/var/lib/puppet/ssl/certs] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/state] (debug): Autorequiring File[/var/lib/puppet] Thu Jan 27 21:30:13 + 2011 /File[/etc/puppet/puppet.conf] (debug): Autorequiring File[/etc/puppet] Thu Jan 27 21:30:13 + 2011 Puppet (debug): Finishing transaction 218723380 Thu Jan 27 21:30:13 + 2011 Puppet (notice): Reopening log files orequiring File[/var/lib/puppet/ssl] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/private_keys/ ps-prod-util1.pem] (debug): Autorequiring File[/var/lib/puppet/ssl/ private_keys] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/public_keys/ ps-prod-util1.pem] (debug): Autorequiring File[/var/lib/puppet/ssl/ public_keys] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/facts] (debug): Autorequiring File[/var/lib/puppet] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/ certificate_requests] (debug): Autorequiring File[/var/lib/puppet/ssl] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/ssl/certs/ca.pem] (debug): Autorequiring File[/var/lib/puppet/ssl/certs] Thu Jan 27 21:30:13 + 2011 /File[/var/lib/puppet/state] (debug): Autorequiring File[/var/lib/puppet] Thu Jan 27 21:30:13 + 2011 /File[/etc/puppet/puppet.conf] (debug): Autorequiring File[/etc/puppet] Thu Jan 27 21:30:13 + 2011 Puppet (debug): Finishing transaction 218723380 Here's some more relevant information: [root@ps-prod-util1 /]# ruby -v ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux] Ruby Enterprise Edition 20090421 [root@ps-prod-util1 /]# which ruby /opt/ruby-enterprise-1.8.6-20090421/bin/ruby [root@ps-prod-util1 /]# /opt/ruby-enterprise-1.8.6-20090421/bin/ruby / opt/ruby-enterprise-1.8.6-20090421/sbin/puppetd [root@ps-prod-util1 /]# ps -ef | grep puppet[d] [root@ps-prod-util1 /]# Can someone please help me out and guide me in the right direction? Thanks, Franck -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppetd Cron Not Running
Thanks for that tip Daniel.
I went ahead and switched to use the fqdn_rand function as it's a
lot more consistent which is what I wanted.
I'm new to Puppet so I'm still learning some of the functions that can
be used.
On Aug 23, 7:54 pm, Daniel Pittman dan...@rimspace.net wrote:
Franck ffallat...@gmail.com writes:
So I've decided to use crontab for all my puppet clients rather than
the daemon. I've set-up a puppetmaster with seven puppet clients.
I'm using the following pattern:
class cron {
$minute = generate('/usr/bin/env', 'sh', '-c', 'printf $((RANDOM
%60+0))')
$minute = fqdn_rand(59)
That generates a random, but consistent, number in the 0-59 range based on the
name of the host. This will give some spread, but not the complete randomness
of the method above.
Daniel
--
✣ Daniel Pittman ✉ dan...@rimspace.net ☎ +61 401 155 707
♽ made with 100 percent post-consumer electrons
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppetd Cron Not Running
Yes, I did take that into account when using the original function's
implementation.
But the issue wasn't residing with that, rather one the puppet clients
is not executing through cron.
On Aug 24, 5:43 am, Ohad Levy ohadl...@gmail.com wrote:
Another options which I've used successfully in the past
is:http://projects.puppetlabs.com/projects/puppet/wiki/Cron_Patterns- see the
section title: Setting Cron using a Puppet custom function
One important thing to note is that when you get a random value, puppet will
change it on each puppet run.
Ohad
On Tue, Aug 24, 2010 at 2:54 AM, Daniel Pittman dan...@rimspace.net wrote:
Franck ffallat...@gmail.com writes:
So I've decided to use crontab for all my puppet clients rather than
the daemon. I've set-up a puppetmaster with seven puppet clients.
I'm using the following pattern:
class cron {
$minute = generate('/usr/bin/env', 'sh', '-c', 'printf $((RANDOM
%60+0))')
$minute = fqdn_rand(59)
That generates a random, but consistent, number in the 0-59 range based on
the
name of the host. This will give some spread, but not the complete
randomness
of the method above.
Daniel
--
✣ Daniel Pittman ✉ dan...@rimspace.net ☎ +61 401 155
707
♽ made with 100 percent post-consumer electrons
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.compuppet-users%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Puppetd Cron Not Running
So I've decided to use crontab for all my puppet clients rather than
the daemon. I've set-up a puppetmaster with seven puppet clients.
I'm using the following pattern:
class cron {
$minute = generate('/usr/bin/env', 'sh', '-c', 'printf $((RANDOM
%60+0))')
cron { manual-puppet:
command = /usr/sbin/puppetd --server=ps-dev-web1 --logdest=/
var/log/puppet/puppet.log --verbose --onetime --no-daemonize,
user= root,
hour= *,
minute = $minute,
ensure = present,
}
}
Six of the puppet clients are running the cronjob just fine at random
intervals, except one. The thing about this host is that if I run the
crontab entry command manually, it's successful:
[r...@ps-dev-app1 puppet]# /usr/sbin/puppetd --server=ps-dev-web1 --
logdest=/var/log/puppet/puppet.log --verbose --onetime --no-daemonize
info: Caching catalog for ps-dev-app1.plansourcedev.com
info: Applying configuration version '1282328744'
notice: /Stage[main]/Cron/Cron[manual-puppet]/minute: minute changed
'5' to '31'
notice: /Stage[main]/Cron/Cron[manual-puppet]/hour: defined 'hour' as
'*'
notice: Finished catalog run in 0.39 seconds
However, when I run this through crontab, nothing happens.
Here's what the /var/log/cron shows, when it executes the command
through crontab:
Aug 23 16:01:01 ps-dev-app1 crond[16367]: (root) CMD (/usr/sbin/
puppetd --server=ps-dev-web1 --logdest=/var/log/puppet/puppet.log --
verbose --debug --onetime --no-daemonize)
And although the command seems to execute through crontab, there's no
evidence of this otherwise. For one, the /var/log/puppet.log does
not get created or append to existing log. The crontab entry should
also reflect a change in interval time, but there's no evidence of
this either.
I've tried several different things, including removing the crontab
file - crontab -r and re-running the command manually to re-create
the crontab entry. I've, verified, my crond is running normally, by
running random commands and verifying they executed through cron. It
would seem only puppetd is not success at running through
crontab.
Fortunately, only one of my puppet clients is experiencing this
issue.
Anyone have an idea of what could be perpetuating this behavior ?
Thanks.
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppetd Cron Not Running
I have not considered --splay, but I definitely will give that a
shot.
I meant that the puppet.log was to be created in the /var/log/puppet
directory, that was a typo.
Thanks Brandon.
On Aug 23, 5:23 pm, Brandon Evans bev...@adicio.com wrote:
On 8/23/10 9:15 AM, Franck wrote:
So I've decided to use crontab for all my puppet clients rather than
the daemon. I've set-up a puppetmaster with seven puppet clients.
I'm using the following pattern:
class cron {
$minute = generate('/usr/bin/env', 'sh', '-c', 'printf $((RANDOM
%60+0))')
cron { manual-puppet:
command = /usr/sbin/puppetd --server=ps-dev-web1 --logdest=/
var/log/puppet/puppet.log --verbose --onetime --no-daemonize,
user = root,
hour = *,
minute = $minute,
ensure = present,
}
}
Have you considered using the --splay --splaylimit options built into
puppet rather than constantly editing the crontab?
http://docs.puppetlabs.com/references/stable/configuration.html?actio...
Six of the puppet clients are running the cronjob just fine at random
intervals, except one. The thing about this host is that if I run the
crontab entry command manually, it's successful:
[r...@ps-dev-app1 puppet]# /usr/sbin/puppetd --server=ps-dev-web1 --
logdest=/var/log/puppet/puppet.log --verbose --onetime --no-daemonize
info: Caching catalog for ps-dev-app1.plansourcedev.com
info: Applying configuration version '1282328744'
notice: /Stage[main]/Cron/Cron[manual-puppet]/minute: minute changed
'5' to '31'
notice: /Stage[main]/Cron/Cron[manual-puppet]/hour: defined 'hour' as
'*'
notice: Finished catalog run in 0.39 seconds
However, when I run this through crontab, nothing happens.
Here's what the /var/log/cron shows, when it executes the command
through crontab:
Aug 23 16:01:01 ps-dev-app1 crond[16367]: (root) CMD (/usr/sbin/
puppetd --server=ps-dev-web1 --logdest=/var/log/puppet/puppet.log --
verbose --debug --onetime --no-daemonize)
And although the command seems to execute through crontab, there's no
evidence of this otherwise. For one, the /var/log/puppet.log does
not get created or append to existing log. The crontab entry should
also reflect a change in interval time, but there's no evidence of
this either.
Maybe just a typo, but the command you are running shows the puppet.log
should be created in /var/log/puppet/puppet.log but you are looking in
/var/log/puppet.log. Perhaps there is a permission issue or
/var/log/puppet/ does not exist?
I've tried several different things, including removing the crontab
file - crontab -r and re-running the command manually to re-create
the crontab entry. I've, verified, my crond is running normally, by
running random commands and verifying they executed through cron. It
would seem only puppetd is not success at running through
crontab.
Fortunately, only one of my puppet clients is experiencing this
issue.
Anyone have an idea of what could be perpetuating this behavior ?
Thanks.
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
