Re: [Puppet Users] Re: Puppet freezes in the middle of runs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > What exactly are you trying to do here? Are you trying to push logs > from the server to the clients? If not, why do you have "recurse => > true"? If you are trying to push permissions, this isn't a good way > to do it. Sadly, Puppet is slow enough that an exec with "chown -R > mysql:mysql /var/log/mysql/archived_logs" and a "chmod -R ..." would > probably be much faster. yeah have a look at: http://groups.google.com/group/puppet-users/browse_thread/thread/84ab151c4935524f and http://groups.google.com/group/puppet-dev/browse_thread/thread/2297b1e36c590d2e/ba9ab386bedf3f35 > One wild guess would be that a file in that folder is locked, and > puppet is blocking on trying to do something to that file. which might additionally be a problem. cheers pete -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuipLEACgkQbwltcAfKi39wGQCeMtIo7WoIyoVCl6eNTaJv8JnN VU0AoLC9RtAAylpFikfkB1jU7kw5eYJt =cwJc -END PGP SIGNATURE- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Re: [Puppet Users] Re: Puppet freezes in the middle of runs
On Mar 18, 2010, at 12:24 PM, Thomas wrote:
> Looking into it more, I think I found the root of the problem. The
> following is a snippet from my puppet configuration:
>
> # create /var/log/mysql/archived_logs
> file { "archived_logs":
> path=> "/var/log/mysql/archived_logs",
> ensure => directory,
> mode=> 775,
> owner => "mysql",
> group => "mysql",
> recurse => "true",
> require => File[mysql_log]
> }
>
> What I see happening in strace is puppet trying to read in data from
> one of the files in the archived_logs directory. Did I make a mistake
> in my configuration here?
>
What exactly are you trying to do here? Are you trying to push logs from the
server to the clients? If not, why do you have "recurse => true"? If you are
trying to push permissions, this isn't a good way to do it. Sadly, Puppet is
slow enough that an exec with "chown -R mysql:mysql
/var/log/mysql/archived_logs" and a "chmod -R ..." would probably be much
faster.
One wild guess would be that a file in that folder is locked, and puppet is
blocking on trying to do something to that file.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppet freezes in the middle of runs
Looking into it more, I think I found the root of the problem. The
following is a snippet from my puppet configuration:
# create /var/log/mysql/archived_logs
file { "archived_logs":
path=> "/var/log/mysql/archived_logs",
ensure => directory,
mode=> 775,
owner => "mysql",
group => "mysql",
recurse => "true",
require => File[mysql_log]
}
What I see happening in strace is puppet trying to read in data from
one of the files in the archived_logs directory. Did I make a mistake
in my configuration here?
On Mar 18, 2:18 pm, Thomas wrote:
> pstree doesn't show anything interesting:
>
> ├─sshd,1354
> │ ├─sshd,2008
> │ │ └─bash,2010
> │ ├─sshd,4942
> │ │ └─bash,4944
> │ │ └─puppetd,5459 /usr/sbin/puppetd --onetime --no-
> daemonize --test --pluginsync --factpath=/var/lib/puppet/lib/facter --
> server puppet1.sermo.dev
> │ │ └─{puppetd},5505
>
> When I do strace, however, I get the following. Seems like it's stuck
> on something:
>
> --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
> rt_sigreturn(0x1a) = 4096
> read(6, "90BD46D9DB1','145D4600-3186-11DF"..., 4096) = 4096
> read(6, "1sL21hcnNoYWx1bm1hcnNoYWwvcGx1Z2"..., 4096) = 4096
> read(6, "F89CA8A12257','106FB820-3186-11D"..., 4096) = 4096
> read(6, "Nlc2IiPnF1ZXVlPC9qYm9zc2VzYjpkZX"..., 4096) = 4096
> read(6, "VsYzJJdWJXVnpjMkZuWlM1MGNtRnVjM0"..., 4096) = 4096
> read(6, "53B15','0F809D30-3186-11DF-9D30-"..., 4096) = 4096
> read(6, "y' ORDER BY M.NAME DESC,M.LAST_U"..., 4096) = 4096
> read(6, "cGVydHk+PEtleT5iM0puTG1waWIzTnpM"..., 4096) = 4096
> read(6, "_SERVICE_CATEGORY C WHERE C.SERV"..., 4096) = 4096
> read(6, "46amJvc3MvZXNiL2Vwci90eXBlL2ptcz"..., 4096) = 4096
> read(6, "((UPPER(NAME) = 'PANELSSNOOPSERV"..., 4096) = 4096
> read(6, "_UPDATE FROM JUDDI_BUSINESS_SERV"..., 4096) = 4096
> read(6, "Lmpib3NzLmNvbS93cy8yMDA3LzAxL2pi"..., 4096) = 4096
> read(6, "MlZSZFdWMVpTQSs8L1ZhbHVlPjwvUHJv"..., 4096) = 4096
> read(6, "-3186-11DF-BF20-B7B2D7785F29','1"..., 4096) = 4096
> read(6, "JXMWhibVF0WlhabGJuUSs8L21hcnNoYW"..., 4096) = 4096
> read(6, "53B5A7C523B','0EC51560-3186-11DF"..., 4096) = 4096
> read(6, "B3B0-3186-11DF-B3B0-F2756F2630FE"..., 4096) = 4096
> read(6, "ib3NzZXNiIj51cm46amJvc3MvZXNiL2V"..., 4096) = 4096
> read(6, "122062 Query\tcommit\n\t\t122062 Que"..., 4096) = 4096
> read(6, "7','0FBCE2E0-3186-11DF-A2E0-92D5"..., 4096) = 4096
> read(6, "Wx1bm1hcnNoYWwvcGx1Z2luL3Nlcmlhb"..., 4096) = 4096
> read(6, "N ('CBA178E0-3186-11DF-B8E0-C39C"..., 4096) = 4096
> read(6, "A6Ly9zY2hlbWFzLmpib3NzLmNvbS93cy"..., 4096) = 4096
> read(6, "ZoWkUxbGMzTmhaMlZSZFdWMVpTQSs8L1"..., 4096) = 4096
> read(6, "N.NAME FROM JUDDI_BUSINESS_SERVI"..., 4096) = 4096
> --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
> rt_sigreturn(0x1a) = 4096
> read(6, "'1516AAF0-3186-11DF-AAF0-F9DB38C"..., 4096) = 4096
> read(6, "XNiOnBlcnNpc3RlbnQ+PGpib3NzZXNiO"..., 4096) = 4096
> read(6, "UE','RDLVR')\n\t\t122063 Query\tcomm"..., 4096) = 4096
> read(6, "51cm46amJvc3MvZXNiL2Vwci90eXBlL2"..., 4096) = 4096
> read(6, "BY M.NAME DESC,M.LAST_UPDATE DES"..., 4096) = 4096
> read(6, "6-11DF-B3B0-F2756F2630FE','10F5F"..., 4096) = 4096
> read(6, "1DF-AD30-E4944AD03A09','0F3F9C90"..., 4096) = 4096
> read(6, "lPnJPMEFCWFZ5QUFKYlFxenpGL2dHQ0Z"..., 4096) = 4096
> read(6, "1DF-9DA0-B2D00126D089','15719C30"..., 4096) = 4096
> read(6, "lVZFdhR051VW1sYVYwWXdURzVTY0ZveV"..., 4096) = 4096
> read(6, "','1354FDC0-3186-11DF-BDC0-9CD52"..., 4096) = 4096
> read(6, "F6EF0896BCFD','1516AAF0-3186-11D"..., 4096) = 4096
> read(6, "pib3NzZXNiOmphdmEubmFtaW5nLnByb3"..., 4096) = 4096
> read(6, "lYZ3VhbTF6TG0xbGMzTmhaMlV1Y21Wa1"..., 4096) = 4096
> read(6, " from message where uuid='urn:jb"..., 4096) = 4096
> read(6, "uTG1waWIzTnpMbk52WVM1bGMySXVaR1Z"..., 4096) = 4096
> read(6, "THORIZED_NAME,OPERATOR,NAME,LANG"..., 4096) = 4096
> read(6, "85F29','10A59530-3186-11DF-9530-"..., 4096) = 4096
> read(6, "nRlbnQ+PEtleT5iM0puTG1waWIzTnpMb"..., 4096) = 4096
> read(6, "ATE,N.NAME FROM JUDDI_BUSINESS_S"..., 4096) = 4096
> read(6, "SERVICE_NAME N WHERE N.SERVICE_K"..., 4096) = 4096
> read(6, "bGJuUWdkR0Z5WjJWMFBTSndZVzVsYkhN"..., 4096) = 4096
> read(6, "DF-8FA0-9C0B5EA0244B','12F55190-"..., 4096) = 4096
> read(6, "ZXh0RmFjdG9yeTwvamJvc3Nlc2I6amF2"..., 4096) = 4096
> read(6, "YldWemMyRm5aUzV6YjNWeVkyVT08L0tl"..., 4096) = 4096
> read(6, "1E0-3186-11DF-A1E0-B52AF3A5E0C9'"..., 4096) = 4096
> read(6, "3186-11DF-9C90-F7F201853B15','0F"..., 4096) = 4096
> read(6, "V1YxWlM1dVlXMWw8L0tleT48VmFsdWU+"..., 4096) = 4096
> read(6, "TMODEL_KEY,M.LAST_UPDATE,M.NAME,"..., 4096) = 4096
> read(6, "tczovLzEwLjE4LjIuNTM6MTA5OS9xdWV"..., 4096) = 4096
> read(6, "9PTwvS2V5PjxWYWx1ZT5yTzBBQlhOeUF"..., 4096) = 4096
> read(6, "E = 'org.jboss.soa.esb.:category"..., 4096) = 4096
> read(6, "mFtaW5nLnByb3ZpZGVyLnVybD48amJvc"..., 4096) = 4096
>
[Puppet Users] Re: Puppet freezes in the middle of runs
pstree doesn't show anything interesting:
├─sshd,1354
│ ├─sshd,2008
│ │ └─bash,2010
│ ├─sshd,4942
│ │ └─bash,4944
│ │ └─puppetd,5459 /usr/sbin/puppetd --onetime --no-
daemonize --test --pluginsync --factpath=/var/lib/puppet/lib/facter --
server puppet1.sermo.dev
│ │ └─{puppetd},5505
When I do strace, however, I get the following. Seems like it's stuck
on something:
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, "90BD46D9DB1','145D4600-3186-11DF"..., 4096) = 4096
read(6, "1sL21hcnNoYWx1bm1hcnNoYWwvcGx1Z2"..., 4096) = 4096
read(6, "F89CA8A12257','106FB820-3186-11D"..., 4096) = 4096
read(6, "Nlc2IiPnF1ZXVlPC9qYm9zc2VzYjpkZX"..., 4096) = 4096
read(6, "VsYzJJdWJXVnpjMkZuWlM1MGNtRnVjM0"..., 4096) = 4096
read(6, "53B15','0F809D30-3186-11DF-9D30-"..., 4096) = 4096
read(6, "y' ORDER BY M.NAME DESC,M.LAST_U"..., 4096) = 4096
read(6, "cGVydHk+PEtleT5iM0puTG1waWIzTnpM"..., 4096) = 4096
read(6, "_SERVICE_CATEGORY C WHERE C.SERV"..., 4096) = 4096
read(6, "46amJvc3MvZXNiL2Vwci90eXBlL2ptcz"..., 4096) = 4096
read(6, "((UPPER(NAME) = 'PANELSSNOOPSERV"..., 4096) = 4096
read(6, "_UPDATE FROM JUDDI_BUSINESS_SERV"..., 4096) = 4096
read(6, "Lmpib3NzLmNvbS93cy8yMDA3LzAxL2pi"..., 4096) = 4096
read(6, "MlZSZFdWMVpTQSs8L1ZhbHVlPjwvUHJv"..., 4096) = 4096
read(6, "-3186-11DF-BF20-B7B2D7785F29','1"..., 4096) = 4096
read(6, "JXMWhibVF0WlhabGJuUSs8L21hcnNoYW"..., 4096) = 4096
read(6, "53B5A7C523B','0EC51560-3186-11DF"..., 4096) = 4096
read(6, "B3B0-3186-11DF-B3B0-F2756F2630FE"..., 4096) = 4096
read(6, "ib3NzZXNiIj51cm46amJvc3MvZXNiL2V"..., 4096) = 4096
read(6, "122062 Query\tcommit\n\t\t122062 Que"..., 4096) = 4096
read(6, "7','0FBCE2E0-3186-11DF-A2E0-92D5"..., 4096) = 4096
read(6, "Wx1bm1hcnNoYWwvcGx1Z2luL3Nlcmlhb"..., 4096) = 4096
read(6, "N ('CBA178E0-3186-11DF-B8E0-C39C"..., 4096) = 4096
read(6, "A6Ly9zY2hlbWFzLmpib3NzLmNvbS93cy"..., 4096) = 4096
read(6, "ZoWkUxbGMzTmhaMlZSZFdWMVpTQSs8L1"..., 4096) = 4096
read(6, "N.NAME FROM JUDDI_BUSINESS_SERVI"..., 4096) = 4096
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, "'1516AAF0-3186-11DF-AAF0-F9DB38C"..., 4096) = 4096
read(6, "XNiOnBlcnNpc3RlbnQ+PGpib3NzZXNiO"..., 4096) = 4096
read(6, "UE','RDLVR')\n\t\t122063 Query\tcomm"..., 4096) = 4096
read(6, "51cm46amJvc3MvZXNiL2Vwci90eXBlL2"..., 4096) = 4096
read(6, "BY M.NAME DESC,M.LAST_UPDATE DES"..., 4096) = 4096
read(6, "6-11DF-B3B0-F2756F2630FE','10F5F"..., 4096) = 4096
read(6, "1DF-AD30-E4944AD03A09','0F3F9C90"..., 4096) = 4096
read(6, "lPnJPMEFCWFZ5QUFKYlFxenpGL2dHQ0Z"..., 4096) = 4096
read(6, "1DF-9DA0-B2D00126D089','15719C30"..., 4096) = 4096
read(6, "lVZFdhR051VW1sYVYwWXdURzVTY0ZveV"..., 4096) = 4096
read(6, "','1354FDC0-3186-11DF-BDC0-9CD52"..., 4096) = 4096
read(6, "F6EF0896BCFD','1516AAF0-3186-11D"..., 4096) = 4096
read(6, "pib3NzZXNiOmphdmEubmFtaW5nLnByb3"..., 4096) = 4096
read(6, "lYZ3VhbTF6TG0xbGMzTmhaMlV1Y21Wa1"..., 4096) = 4096
read(6, " from message where uuid='urn:jb"..., 4096) = 4096
read(6, "uTG1waWIzTnpMbk52WVM1bGMySXVaR1Z"..., 4096) = 4096
read(6, "THORIZED_NAME,OPERATOR,NAME,LANG"..., 4096) = 4096
read(6, "85F29','10A59530-3186-11DF-9530-"..., 4096) = 4096
read(6, "nRlbnQ+PEtleT5iM0puTG1waWIzTnpMb"..., 4096) = 4096
read(6, "ATE,N.NAME FROM JUDDI_BUSINESS_S"..., 4096) = 4096
read(6, "SERVICE_NAME N WHERE N.SERVICE_K"..., 4096) = 4096
read(6, "bGJuUWdkR0Z5WjJWMFBTSndZVzVsYkhN"..., 4096) = 4096
read(6, "DF-8FA0-9C0B5EA0244B','12F55190-"..., 4096) = 4096
read(6, "ZXh0RmFjdG9yeTwvamJvc3Nlc2I6amF2"..., 4096) = 4096
read(6, "YldWemMyRm5aUzV6YjNWeVkyVT08L0tl"..., 4096) = 4096
read(6, "1E0-3186-11DF-A1E0-B52AF3A5E0C9'"..., 4096) = 4096
read(6, "3186-11DF-9C90-F7F201853B15','0F"..., 4096) = 4096
read(6, "V1YxWlM1dVlXMWw8L0tleT48VmFsdWU+"..., 4096) = 4096
read(6, "TMODEL_KEY,M.LAST_UPDATE,M.NAME,"..., 4096) = 4096
read(6, "tczovLzEwLjE4LjIuNTM6MTA5OS9xdWV"..., 4096) = 4096
read(6, "9PTwvS2V5PjxWYWx1ZT5yTzBBQlhOeUF"..., 4096) = 4096
read(6, "E = 'org.jboss.soa.esb.:category"..., 4096) = 4096
read(6, "mFtaW5nLnByb3ZpZGVyLnVybD48amJvc"..., 4096) = 4096
read(6, "mhaMlV1Y21Wa1pXeHBkbVZ5WldRPTwvS"..., 4096) = 4096
read(6, "90-C8786D5E36C7','129C0E00-3186-"..., 4096) = 4096
read(6, "ZXM+PFByb3BlcnR5PjxLZXk+YjNKbkxt"..., 4096) = 4096
read(6, "0-3186-11DF-A1E0-DC63C5C390D7','"..., 4096) = 4096
read(6, "86-11DF-9530-F89CA8A12257','106F"..., 4096) = 4096
read(6, "11DF-9D30-F53B5A7C523B','0EC5156"..., 4096) = 4096
read(6, "WhMbVZ6WWk1VFpYSjJhV05sQUFBQUFBQ"..., 4096) = 4096
read(6, "PmptczovLzEwLjE4LjIuNTM6MTA5OS9x"..., 4096) = 4096
read(6, "ZEE9PTwvS2V5PjxWYWx1ZT5yTzBBQlhO"..., 4096) = 4096
read(6, "DI_TMODEL WHERE TMODEL_KEY='uuid"..., 4096) = 4096
read(6, ".NAME DESC,M.LAST_UPDATE DESC\n\t\t"..., 4096) = 4096
--- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
rt_sigreturn(0x1a) = 4096
read(6, "bWFzLmpib3NzLmNvbS93cy8yMDA3LzAx"..., 4096) = 4096
read(6, ",'14AF5DA0
Re: [Puppet Users] Re: Puppet freezes in the middle of runs
On Thu, Mar 18, 2010 at 12:45 PM, Thomas wrote: > Puppet is version 0.25.4 > > The machines are all running under similar hardware, sorry if the word > "class" was misleading. The database machines are failing to run > puppet, while all of the other servers (mostly application servers) > have no problem. > > Can you also check http://projects.reductivelabs.com/search/index/puppet?q=hangs&scope=&all_words=1&issues=1&submit=Submit for any open items that may apply to your case? --Michael -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppet freezes in the middle of runs
Thomas writes: > Hello, > > I'm running puppet on a bunch of Red Hat Enterprise Linux 5 machines. > I am seeing an issue where puppetd will freeze in the middle of a run, > regularly. If I run with --debug, the last line of output I see is > usually a command that completes quickly, or a notification that > something has succeeded - so I don't think it's related to the freeze. > The strange thing is this problem only occurs on one class of machine > that I am running puppet on - none of the other machines (which have > different configurations under puppet) have this problem. > > I'm not sure how to get more information to debug what puppet is > getting hung up on. Can anyone point me in the right direction? I've seen this happen when puppet is running a command that hangs. So pstree might help you find the problem. In our case it was trying to do an NFS mount and the 'mount' command itself was hanging. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
[Puppet Users] Re: Puppet freezes in the middle of runs
Puppet is version 0.25.4 The machines are all running under similar hardware, sorry if the word "class" was misleading. The database machines are failing to run puppet, while all of the other servers (mostly application servers) have no problem. On Mar 18, 12:41 pm, Michael DeHaan wrote: > On Thu, Mar 18, 2010 at 12:36 PM, Thomas wrote: > > Hello, > > > I'm running puppet on a bunch of Red Hat Enterprise Linux 5 machines. > > I am seeing an issue where puppetd will freeze in the middle of a run, > > regularly. If I run with --debug, the last line of output I see is > > usually a command that completes quickly, or a notification that > > something has succeeded - so I don't think it's related to the freeze. > > The strange thing is this problem only occurs on one class of machine > > that I am running puppet on - none of the other machines (which have > > different configurations under puppet) have this problem. > > > I'm not sure how to get more information to debug what puppet is > > getting hung up on. Can anyone point me in the right direction? > > > Thanks! > > For starters, what is the class and what versions of Puppet are you using? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-us...@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
