[Puppet Users] Re: Puppet freezes in the middle of runs
Puppet is version 0.25.4 The machines are all running under similar hardware, sorry if the word class was misleading. The database machines are failing to run puppet, while all of the other servers (mostly application servers) have no problem. On Mar 18, 12:41 pm, Michael DeHaan mich...@reductivelabs.com wrote: On Thu, Mar 18, 2010 at 12:36 PM, Thomas tommyfis...@gmail.com wrote: Hello, I'm running puppet on a bunch of Red Hat Enterprise Linux 5 machines. I am seeing an issue where puppetd will freeze in the middle of a run, regularly. If I run with --debug, the last line of output I see is usually a command that completes quickly, or a notification that something has succeeded - so I don't think it's related to the freeze. The strange thing is this problem only occurs on one class of machine that I am running puppet on - none of the other machines (which have different configurations under puppet) have this problem. I'm not sure how to get more information to debug what puppet is getting hung up on. Can anyone point me in the right direction? Thanks! For starters, what is the class and what versions of Puppet are you using? -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Puppet freezes in the middle of runs
On Thu, Mar 18, 2010 at 12:45 PM, Thomas tommyfis...@gmail.com wrote: Puppet is version 0.25.4 The machines are all running under similar hardware, sorry if the word class was misleading. The database machines are failing to run puppet, while all of the other servers (mostly application servers) have no problem. Can you also check http://projects.reductivelabs.com/search/index/puppet?q=hangsscope=all_words=1issues=1submit=Submit for any open items that may apply to your case? --Michael -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppet freezes in the middle of runs
pstree doesn't show anything interesting:
├─sshd,1354
│ ├─sshd,2008
│ │ └─bash,2010
│ ├─sshd,4942
│ │ └─bash,4944
│ │ └─puppetd,5459 /usr/sbin/puppetd --onetime --no-
daemonize --test --pluginsync --factpath=/var/lib/puppet/lib/facter --
server puppet1.sermo.dev
│ │ └─{puppetd},5505
When I do strace, however, I get the following. Seems like it's stuck
on something:
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, 90BD46D9DB1','145D4600-3186-11DF..., 4096) = 4096
read(6, 1sL21hcnNoYWx1bm1hcnNoYWwvcGx1Z2..., 4096) = 4096
read(6, F89CA8A12257','106FB820-3186-11D..., 4096) = 4096
read(6, Nlc2IiPnF1ZXVlPC9qYm9zc2VzYjpkZX..., 4096) = 4096
read(6, VsYzJJdWJXVnpjMkZuWlM1MGNtRnVjM0..., 4096) = 4096
read(6, 53B15','0F809D30-3186-11DF-9D30-..., 4096) = 4096
read(6, y' ORDER BY M.NAME DESC,M.LAST_U..., 4096) = 4096
read(6, cGVydHk+PEtleT5iM0puTG1waWIzTnpM..., 4096) = 4096
read(6, _SERVICE_CATEGORY C WHERE C.SERV..., 4096) = 4096
read(6, 46amJvc3MvZXNiL2Vwci90eXBlL2ptcz..., 4096) = 4096
read(6, ((UPPER(NAME) = 'PANELSSNOOPSERV..., 4096) = 4096
read(6, _UPDATE FROM JUDDI_BUSINESS_SERV..., 4096) = 4096
read(6, Lmpib3NzLmNvbS93cy8yMDA3LzAxL2pi..., 4096) = 4096
read(6, MlZSZFdWMVpTQSs8L1ZhbHVlPjwvUHJv..., 4096) = 4096
read(6, -3186-11DF-BF20-B7B2D7785F29','1..., 4096) = 4096
read(6, JXMWhibVF0WlhabGJuUSs8L21hcnNoYW..., 4096) = 4096
read(6, 53B5A7C523B','0EC51560-3186-11DF..., 4096) = 4096
read(6, B3B0-3186-11DF-B3B0-F2756F2630FE..., 4096) = 4096
read(6, ib3NzZXNiIj51cm46amJvc3MvZXNiL2V..., 4096) = 4096
read(6, 122062 Query\tcommit\n\t\t122062 Que..., 4096) = 4096
read(6, 7','0FBCE2E0-3186-11DF-A2E0-92D5..., 4096) = 4096
read(6, Wx1bm1hcnNoYWwvcGx1Z2luL3Nlcmlhb..., 4096) = 4096
read(6, N ('CBA178E0-3186-11DF-B8E0-C39C..., 4096) = 4096
read(6, A6Ly9zY2hlbWFzLmpib3NzLmNvbS93cy..., 4096) = 4096
read(6, ZoWkUxbGMzTmhaMlZSZFdWMVpTQSs8L1..., 4096) = 4096
read(6, N.NAME FROM JUDDI_BUSINESS_SERVI..., 4096) = 4096
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, '1516AAF0-3186-11DF-AAF0-F9DB38C..., 4096) = 4096
read(6, XNiOnBlcnNpc3RlbnQ+PGpib3NzZXNiO..., 4096) = 4096
read(6, UE','RDLVR')\n\t\t122063 Query\tcomm..., 4096) = 4096
read(6, 51cm46amJvc3MvZXNiL2Vwci90eXBlL2..., 4096) = 4096
read(6, BY M.NAME DESC,M.LAST_UPDATE DES..., 4096) = 4096
read(6, 6-11DF-B3B0-F2756F2630FE','10F5F..., 4096) = 4096
read(6, 1DF-AD30-E4944AD03A09','0F3F9C90..., 4096) = 4096
read(6, lPnJPMEFCWFZ5QUFKYlFxenpGL2dHQ0Z..., 4096) = 4096
read(6, 1DF-9DA0-B2D00126D089','15719C30..., 4096) = 4096
read(6, lVZFdhR051VW1sYVYwWXdURzVTY0ZveV..., 4096) = 4096
read(6, ','1354FDC0-3186-11DF-BDC0-9CD52..., 4096) = 4096
read(6, F6EF0896BCFD','1516AAF0-3186-11D..., 4096) = 4096
read(6, pib3NzZXNiOmphdmEubmFtaW5nLnByb3..., 4096) = 4096
read(6, lYZ3VhbTF6TG0xbGMzTmhaMlV1Y21Wa1..., 4096) = 4096
read(6, from message where uuid='urn:jb..., 4096) = 4096
read(6, uTG1waWIzTnpMbk52WVM1bGMySXVaR1Z..., 4096) = 4096
read(6, THORIZED_NAME,OPERATOR,NAME,LANG..., 4096) = 4096
read(6, 85F29','10A59530-3186-11DF-9530-..., 4096) = 4096
read(6, nRlbnQ+PEtleT5iM0puTG1waWIzTnpMb..., 4096) = 4096
read(6, ATE,N.NAME FROM JUDDI_BUSINESS_S..., 4096) = 4096
read(6, SERVICE_NAME N WHERE N.SERVICE_K..., 4096) = 4096
read(6, bGJuUWdkR0Z5WjJWMFBTSndZVzVsYkhN..., 4096) = 4096
read(6, DF-8FA0-9C0B5EA0244B','12F55190-..., 4096) = 4096
read(6, ZXh0RmFjdG9yeTwvamJvc3Nlc2I6amF2..., 4096) = 4096
read(6, YldWemMyRm5aUzV6YjNWeVkyVT08L0tl..., 4096) = 4096
read(6, 1E0-3186-11DF-A1E0-B52AF3A5E0C9'..., 4096) = 4096
read(6, 3186-11DF-9C90-F7F201853B15','0F..., 4096) = 4096
read(6, V1YxWlM1dVlXMWw8L0tleT48VmFsdWU+..., 4096) = 4096
read(6, TMODEL_KEY,M.LAST_UPDATE,M.NAME,..., 4096) = 4096
read(6, tczovLzEwLjE4LjIuNTM6MTA5OS9xdWV..., 4096) = 4096
read(6, 9PTwvS2V5PjxWYWx1ZT5yTzBBQlhOeUF..., 4096) = 4096
read(6, E = 'org.jboss.soa.esb.:category..., 4096) = 4096
read(6, mFtaW5nLnByb3ZpZGVyLnVybD48amJvc..., 4096) = 4096
read(6, mhaMlV1Y21Wa1pXeHBkbVZ5WldRPTwvS..., 4096) = 4096
read(6, 90-C8786D5E36C7','129C0E00-3186-..., 4096) = 4096
read(6, ZXM+PFByb3BlcnR5PjxLZXk+YjNKbkxt..., 4096) = 4096
read(6, 0-3186-11DF-A1E0-DC63C5C390D7','..., 4096) = 4096
read(6, 86-11DF-9530-F89CA8A12257','106F..., 4096) = 4096
read(6, 11DF-9D30-F53B5A7C523B','0EC5156..., 4096) = 4096
read(6, WhMbVZ6WWk1VFpYSjJhV05sQUFBQUFBQ..., 4096) = 4096
read(6, PmptczovLzEwLjE4LjIuNTM6MTA5OS9x..., 4096) = 4096
read(6, ZEE9PTwvS2V5PjxWYWx1ZT5yTzBBQlhO..., 4096) = 4096
read(6, DI_TMODEL WHERE TMODEL_KEY='uuid..., 4096) = 4096
read(6, .NAME DESC,M.LAST_UPDATE DESC\n\t\t..., 4096) = 4096
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, bWFzLmpib3NzLmNvbS93cy8yMDA3LzAx..., 4096) = 4096
read(6, ,'14AF5DA0-3186-11DF-9DA0-F90BD4..., 4096) = 4096
read(6, -F90BD46D9DB1','145D4600-3186-11..., 4096) = 4096
read(6, EZNRWxGVmtWV1FUMDlDbDFkUGp3dlkyO...,
[Puppet Users] Re: Puppet freezes in the middle of runs
Looking into it more, I think I found the root of the problem. The
following is a snippet from my puppet configuration:
# create /var/log/mysql/archived_logs
file { archived_logs:
path= /var/log/mysql/archived_logs,
ensure = directory,
mode= 775,
owner = mysql,
group = mysql,
recurse = true,
require = File[mysql_log]
}
What I see happening in strace is puppet trying to read in data from
one of the files in the archived_logs directory. Did I make a mistake
in my configuration here?
On Mar 18, 2:18 pm, Thomas tommyfis...@gmail.com wrote:
pstree doesn't show anything interesting:
├─sshd,1354
│ ├─sshd,2008
│ │ └─bash,2010
│ ├─sshd,4942
│ │ └─bash,4944
│ │ └─puppetd,5459 /usr/sbin/puppetd --onetime --no-
daemonize --test --pluginsync --factpath=/var/lib/puppet/lib/facter --
server puppet1.sermo.dev
│ │ └─{puppetd},5505
When I do strace, however, I get the following. Seems like it's stuck
on something:
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, 90BD46D9DB1','145D4600-3186-11DF..., 4096) = 4096
read(6, 1sL21hcnNoYWx1bm1hcnNoYWwvcGx1Z2..., 4096) = 4096
read(6, F89CA8A12257','106FB820-3186-11D..., 4096) = 4096
read(6, Nlc2IiPnF1ZXVlPC9qYm9zc2VzYjpkZX..., 4096) = 4096
read(6, VsYzJJdWJXVnpjMkZuWlM1MGNtRnVjM0..., 4096) = 4096
read(6, 53B15','0F809D30-3186-11DF-9D30-..., 4096) = 4096
read(6, y' ORDER BY M.NAME DESC,M.LAST_U..., 4096) = 4096
read(6, cGVydHk+PEtleT5iM0puTG1waWIzTnpM..., 4096) = 4096
read(6, _SERVICE_CATEGORY C WHERE C.SERV..., 4096) = 4096
read(6, 46amJvc3MvZXNiL2Vwci90eXBlL2ptcz..., 4096) = 4096
read(6, ((UPPER(NAME) = 'PANELSSNOOPSERV..., 4096) = 4096
read(6, _UPDATE FROM JUDDI_BUSINESS_SERV..., 4096) = 4096
read(6, Lmpib3NzLmNvbS93cy8yMDA3LzAxL2pi..., 4096) = 4096
read(6, MlZSZFdWMVpTQSs8L1ZhbHVlPjwvUHJv..., 4096) = 4096
read(6, -3186-11DF-BF20-B7B2D7785F29','1..., 4096) = 4096
read(6, JXMWhibVF0WlhabGJuUSs8L21hcnNoYW..., 4096) = 4096
read(6, 53B5A7C523B','0EC51560-3186-11DF..., 4096) = 4096
read(6, B3B0-3186-11DF-B3B0-F2756F2630FE..., 4096) = 4096
read(6, ib3NzZXNiIj51cm46amJvc3MvZXNiL2V..., 4096) = 4096
read(6, 122062 Query\tcommit\n\t\t122062 Que..., 4096) = 4096
read(6, 7','0FBCE2E0-3186-11DF-A2E0-92D5..., 4096) = 4096
read(6, Wx1bm1hcnNoYWwvcGx1Z2luL3Nlcmlhb..., 4096) = 4096
read(6, N ('CBA178E0-3186-11DF-B8E0-C39C..., 4096) = 4096
read(6, A6Ly9zY2hlbWFzLmpib3NzLmNvbS93cy..., 4096) = 4096
read(6, ZoWkUxbGMzTmhaMlZSZFdWMVpTQSs8L1..., 4096) = 4096
read(6, N.NAME FROM JUDDI_BUSINESS_SERVI..., 4096) = 4096
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, '1516AAF0-3186-11DF-AAF0-F9DB38C..., 4096) = 4096
read(6, XNiOnBlcnNpc3RlbnQ+PGpib3NzZXNiO..., 4096) = 4096
read(6, UE','RDLVR')\n\t\t122063 Query\tcomm..., 4096) = 4096
read(6, 51cm46amJvc3MvZXNiL2Vwci90eXBlL2..., 4096) = 4096
read(6, BY M.NAME DESC,M.LAST_UPDATE DES..., 4096) = 4096
read(6, 6-11DF-B3B0-F2756F2630FE','10F5F..., 4096) = 4096
read(6, 1DF-AD30-E4944AD03A09','0F3F9C90..., 4096) = 4096
read(6, lPnJPMEFCWFZ5QUFKYlFxenpGL2dHQ0Z..., 4096) = 4096
read(6, 1DF-9DA0-B2D00126D089','15719C30..., 4096) = 4096
read(6, lVZFdhR051VW1sYVYwWXdURzVTY0ZveV..., 4096) = 4096
read(6, ','1354FDC0-3186-11DF-BDC0-9CD52..., 4096) = 4096
read(6, F6EF0896BCFD','1516AAF0-3186-11D..., 4096) = 4096
read(6, pib3NzZXNiOmphdmEubmFtaW5nLnByb3..., 4096) = 4096
read(6, lYZ3VhbTF6TG0xbGMzTmhaMlV1Y21Wa1..., 4096) = 4096
read(6, from message where uuid='urn:jb..., 4096) = 4096
read(6, uTG1waWIzTnpMbk52WVM1bGMySXVaR1Z..., 4096) = 4096
read(6, THORIZED_NAME,OPERATOR,NAME,LANG..., 4096) = 4096
read(6, 85F29','10A59530-3186-11DF-9530-..., 4096) = 4096
read(6, nRlbnQ+PEtleT5iM0puTG1waWIzTnpMb..., 4096) = 4096
read(6, ATE,N.NAME FROM JUDDI_BUSINESS_S..., 4096) = 4096
read(6, SERVICE_NAME N WHERE N.SERVICE_K..., 4096) = 4096
read(6, bGJuUWdkR0Z5WjJWMFBTSndZVzVsYkhN..., 4096) = 4096
read(6, DF-8FA0-9C0B5EA0244B','12F55190-..., 4096) = 4096
read(6, ZXh0RmFjdG9yeTwvamJvc3Nlc2I6amF2..., 4096) = 4096
read(6, YldWemMyRm5aUzV6YjNWeVkyVT08L0tl..., 4096) = 4096
read(6, 1E0-3186-11DF-A1E0-B52AF3A5E0C9'..., 4096) = 4096
read(6, 3186-11DF-9C90-F7F201853B15','0F..., 4096) = 4096
read(6, V1YxWlM1dVlXMWw8L0tleT48VmFsdWU+..., 4096) = 4096
read(6, TMODEL_KEY,M.LAST_UPDATE,M.NAME,..., 4096) = 4096
read(6, tczovLzEwLjE4LjIuNTM6MTA5OS9xdWV..., 4096) = 4096
read(6, 9PTwvS2V5PjxWYWx1ZT5yTzBBQlhOeUF..., 4096) = 4096
read(6, E = 'org.jboss.soa.esb.:category..., 4096) = 4096
read(6, mFtaW5nLnByb3ZpZGVyLnVybD48amJvc..., 4096) = 4096
read(6, mhaMlV1Y21Wa1pXeHBkbVZ5WldRPTwvS..., 4096) = 4096
read(6, 90-C8786D5E36C7','129C0E00-3186-..., 4096) = 4096
read(6, ZXM+PFByb3BlcnR5PjxLZXk+YjNKbkxt..., 4096) = 4096
read(6,
Re: [Puppet Users] Re: Puppet freezes in the middle of runs
On Mar 18, 2010, at 12:24 PM, Thomas wrote:
Looking into it more, I think I found the root of the problem. The
following is a snippet from my puppet configuration:
# create /var/log/mysql/archived_logs
file { archived_logs:
path= /var/log/mysql/archived_logs,
ensure = directory,
mode= 775,
owner = mysql,
group = mysql,
recurse = true,
require = File[mysql_log]
}
What I see happening in strace is puppet trying to read in data from
one of the files in the archived_logs directory. Did I make a mistake
in my configuration here?
What exactly are you trying to do here? Are you trying to push logs from the
server to the clients? If not, why do you have recurse = true? If you are
trying to push permissions, this isn't a good way to do it. Sadly, Puppet is
slow enough that an exec with chown -R mysql:mysql
/var/log/mysql/archived_logs and a chmod -R ... would probably be much
faster.
One wild guess would be that a file in that folder is locked, and puppet is
blocking on trying to do something to that file.
--
You received this message because you are subscribed to the Google Groups
Puppet Users group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Puppet freezes in the middle of runs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 What exactly are you trying to do here? Are you trying to push logs from the server to the clients? If not, why do you have recurse = true? If you are trying to push permissions, this isn't a good way to do it. Sadly, Puppet is slow enough that an exec with chown -R mysql:mysql /var/log/mysql/archived_logs and a chmod -R ... would probably be much faster. yeah have a look at: http://groups.google.com/group/puppet-users/browse_thread/thread/84ab151c4935524f and http://groups.google.com/group/puppet-dev/browse_thread/thread/2297b1e36c590d2e/ba9ab386bedf3f35 One wild guess would be that a file in that folder is locked, and puppet is blocking on trying to do something to that file. which might additionally be a problem. cheers pete -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuipLEACgkQbwltcAfKi39wGQCeMtIo7WoIyoVCl6eNTaJv8JnN VU0AoLC9RtAAylpFikfkB1jU7kw5eYJt =cwJc -END PGP SIGNATURE- -- You received this message because you are subscribed to the Google Groups Puppet Users group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
