[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
We are also having this similar issue with v2.7.10.1. Short term fix is
deleting the lockfile that hasn't updated in 2x{puppet-run-cycle} and
restarting puppet.
On Tuesday, February 10, 2009 2:25:53 AM UTC-8, Keith Edmunds wrote:
>
> I'm just starting a roll out of Puppet and I'm seeing a problem on maybe
> 25% of client nodes. The symptoms are that the clients stop updating. In
> the Puppetmaster log, I'm seeing things like:
>
> Feb 9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for in
> 0.05 seconds
> Feb 9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for in
> 0.05 seconds
> Feb 9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for in
> 1.83 seconds
> Feb 9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for in
> 0.91 seconds
>
> These are all for the same client; everything appears normal until 21:41,
> then no more checks from the client (it's now 10:17 on Feb 10).
>
> On the client, I tried running puppetd manually:
>
> # puppetd -t
> notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping
> catalog run
>
> A look at the lock file:
>
> # ls -l /var/lib/puppet/state/puppetdlock
> -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock
>
> ...shows that it was probably created at the next run after the last one
> logged on the Puppetmaster (above).
>
> Looking at the lock file:
>
> # echo $(cat /var/lib/puppet/state/puppetdlock)
> 32400
> # ps -fp 32400
> UIDPID PPID C STIME TTY TIME CMD
> root 32400 1 0 Feb06 ?00:01:41 ruby /usr/sbin/puppetd -w 5
>
> ...shows that the puppetd is still running.
>
> Why would the lock file be created and not subsequently deleted?
>
> If it helps, it is likely that the Puppetmaster was very busy at that
> time, but even so I would expect the client to deal with that graciously.
>
> Maybe related, maybe not: I can't stop puppetd in the usual way:
>
> # /etc/init.d/puppet stop
> Stopping puppet configuration management tool.
> # ps -fp 32400
> UIDPID PPID C STIME TTY TIME CMD
> root 32400 1 0 Feb06 ?00:01:41 ruby /usr/sbin/puppetd -w 5
>
> If I 'kill -9' the puppetd process, remove the /var/run/puppetd.pid file
> and remove the lock file, I can restart puppetd and it runs OK for a
> while, but eventually the puppetdlock file causes this problem again.
>
> Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch.
>
> Grateful for any suggestions / pointers / etc.
>
> Thanks,
> Keith
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
On Mon, 16 Feb 2009 21:39:55 -0600, l...@madstop.com said: > Note that you should be able to run 'puppetd --enable' to remove that > stale lock file. Old thread revisited. Some clients occasionally don't clear the lockfile in the subject. A couple of people have suggested, as above, that "puppetd --enable" should clear it. It doesn't: # ls -l /var/lib/puppet/state/puppetdlock -rw-r--r-- 1 root root 5 2009-03-20 14:50 /var/lib/puppet/state/puppetdlock # puppetd --enable # ls -l /var/lib/puppet/state/puppetdlock -rw-r--r-- 1 root root 5 2009-03-20 14:50 /var/lib/puppet/state/puppetdlock # My only recourse now would seem to be a cron job that checks the age of the puppetdlock and restarts the puppet client if the lockfile is older than a given age. This is with: # puppetd --version 0.24.5 # Any ideas as to why this is happening? Thanks, Keith --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
We have had this happening occasionally (approx. 1 out of ~120 VMs, every other week or so) on our OpenVZ VMs as well. We're running Debian Etch, and over the past week upgraded to 0.24.5-3, so I'm keeping an eye on things to see if that resolved it for us. - Jeff Paul Lathrop wrote: > On Mon, Feb 16, 2009 at 7:39 PM, Luke Kanies wrote: >> Otherwise I've no idea, because you're the first to run into it >> that I know of. >> >> Note that you should be able to run 'puppetd --enable' to remove that >> stale lock file. > > I periodically run into this issue as well; it only happens on our Xen > VMs. Not sure if that helps. > > I just have a cron job that runs puppetd --enable on these systems. > > --Paul > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
On Mon, Feb 16, 2009 at 7:39 PM, Luke Kanies wrote: > Otherwise I've no idea, because you're the first to run into it > that I know of. > > Note that you should be able to run 'puppetd --enable' to remove that > stale lock file. I periodically run into this issue as well; it only happens on our Xen VMs. Not sure if that helps. I just have a cron job that runs puppetd --enable on these systems. --Paul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
On Feb 10, 2009, at 4:25 AM, Keith Edmunds wrote: > > I'm just starting a roll out of Puppet and I'm seeing a problem on > maybe > 25% of client nodes. The symptoms are that the clients stop > updating. In > the Puppetmaster log, I'm seeing things like: > > Feb 9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for in > 0.05 seconds > Feb 9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for in > 0.05 seconds > Feb 9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for in > 1.83 seconds > Feb 9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for in > 0.91 seconds > > These are all for the same client; everything appears normal until > 21:41, > then no more checks from the client (it's now 10:17 on Feb 10). > > On the client, I tried running puppetd manually: > > # puppetd -t > notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping > catalog run > > A look at the lock file: > > # ls -l /var/lib/puppet/state/puppetdlock > -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/ > puppetdlock > > ...shows that it was probably created at the next run after the last > one > logged on the Puppetmaster (above). > > Looking at the lock file: > > # echo $(cat /var/lib/puppet/state/puppetdlock) > 32400 > # ps -fp 32400 > UIDPID PPID C STIME TTY TIME CMD > root 32400 1 0 Feb06 ?00:01:41 ruby /usr/sbin/ > puppetd -w 5 > > ...shows that the puppetd is still running. > > Why would the lock file be created and not subsequently deleted? > > If it helps, it is likely that the Puppetmaster was very busy at that > time, but even so I would expect the client to deal with that > graciously. > > Maybe related, maybe not: I can't stop puppetd in the usual way: > > # /etc/init.d/puppet stop > Stopping puppet configuration management tool. > # ps -fp 32400 > UIDPID PPID C STIME TTY TIME CMD > root 32400 1 0 Feb06 ?00:01:41 ruby /usr/sbin/ > puppetd -w 5 > > If I 'kill -9' the puppetd process, remove the /var/run/puppetd.pid > file > and remove the lock file, I can restart puppetd and it runs OK for a > while, but eventually the puppetdlock file causes this problem again. > > Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch. > > Grateful for any suggestions / pointers / etc. I haven't seen this for absolute ages, so whatever used to cause it is unlikely to be the problem now. Is there maybe a signal being sent to puppetd? If not, any chance you can get a stack trace from the clients?You'd have to leave a client running in the foreground with stdout redirected. Otherwise I've no idea, because you're the first to run into it that I know of. Note that you should be able to run 'puppetd --enable' to remove that stale lock file. -- One of the keys to happiness is a bad memory. -- Rita Mae Brown - Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
> Does puppet changes your time > (ntp)? If the time changes while puppet is running it could lead for a > very long time that puppet is waiting No, although all servers are running NTP. I don't think that's the problem: the problem is the stale puppetdlock. Thanks for the suggestion - Keith --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
[Puppet Users] Re: Lock file /var/lib/puppet/state/puppetdlock
maybe not related but similar problem... Does puppet changes your time (ntp)? If the time changes while puppet is running it could lead for a very long time that puppet is waiting On 2/10/09, Keith Edmunds wrote: > > I'm just starting a roll out of Puppet and I'm seeing a problem on maybe > 25% of client nodes. The symptoms are that the clients stop updating. In > the Puppetmaster log, I'm seeing things like: > > Feb 9 20:10:23 vs4 puppetmasterd[17942]: Compiled catalog for in > 0.05 seconds > Feb 9 20:40:41 vs4 puppetmasterd[17942]: Compiled catalog for in > 0.05 seconds > Feb 9 21:11:16 vs4 puppetmasterd[17942]: Compiled catalog for in > 1.83 seconds > Feb 9 21:41:37 vs4 puppetmasterd[17942]: Compiled catalog for in > 0.91 seconds > > These are all for the same client; everything appears normal until 21:41, > then no more checks from the client (it's now 10:17 on Feb 10). > > On the client, I tried running puppetd manually: > > # puppetd -t > notice: Lock file /var/lib/puppet/state/puppetdlock exists; skipping > catalog run > > A look at the lock file: > > # ls -l /var/lib/puppet/state/puppetdlock > -rw-r--r-- 1 root root 5 2009-02-09 22:11 /var/lib/puppet/state/puppetdlock > > ...shows that it was probably created at the next run after the last one > logged on the Puppetmaster (above). > > Looking at the lock file: > > # echo $(cat /var/lib/puppet/state/puppetdlock) > 32400 > # ps -fp 32400 > UIDPID PPID C STIME TTY TIME CMD > root 32400 1 0 Feb06 ?00:01:41 ruby /usr/sbin/puppetd -w 5 > > ...shows that the puppetd is still running. > > Why would the lock file be created and not subsequently deleted? > > If it helps, it is likely that the Puppetmaster was very busy at that > time, but even so I would expect the client to deal with that graciously. > > Maybe related, maybe not: I can't stop puppetd in the usual way: > > # /etc/init.d/puppet stop > Stopping puppet configuration management tool. > # ps -fp 32400 > UIDPID PPID C STIME TTY TIME CMD > root 32400 1 0 Feb06 ?00:01:41 ruby /usr/sbin/puppetd -w 5 > > If I 'kill -9' the puppetd process, remove the /var/run/puppetd.pid file > and remove the lock file, I can restart puppetd and it runs OK for a > while, but eventually the puppetdlock file causes this problem again. > > Versions: 0.24.5-3, the Debian Lenny package compiled for Debian Etch. > > Grateful for any suggestions / pointers / etc. > > Thanks, > Keith > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~--~~~~--~~--~--~---
