Jira (PUP-5028) 100% cpu
Title: Message Title
Peter Huene commented on PUP-5028
Re: 100% cpu
Ignore my last comment. I was incorrectly looking at the polls in virNetClientSetTLSSession and not virNetClientIOEventLoop where the backtrace is showing the problem lies.
Looking at virNetClientIOEventLoop, a bad descriptor to poll should break it out of the loop:
repoll:
ret = poll(fds, ARRAY_CARDINALITY(fds), timeout);
if (ret < 0 && (errno == EAGAIN || errno == EINTR))
goto repoll;
ignore_value(pthread_sigmask(SIG_SETMASK, &oldmask, NULL));
virObjectLock(client);
if (ret < 0) {
Jira (PUP-5028) 100% cpu
Title: Message Title
Peter Huene commented on PUP-5028
Re: 100% cpu
Indeed, it looks like libvirt ignores legitimate errors from poll and only checks for needing to poll again.
I suspect this would fix it:
if (ret < 0) {
if (errno == EAGAIN || errno == EINTR) {
goto repoll;
}
}
ignore_value(pthread_sigmask(SIG_SETMASK, &oldmask, NULL));
if (ret < 0) {
goto error;
}
Note that there's two places where poll is called that could enter an infinite loop.
Unfortunately, there's little Puppet or Facter can do to work around this bug, especially if it's occurring during a Ruby GC. This will need to be
Jira (PUP-5028) 100% cpu
Title: Message Title Maxim Kozin updated an issue Puppet / PUP-5028 100% cpu Custom facts with libvirt-lxc and external commands Change By: Maxim Kozin Attachment: lxc.rb Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5028) 100% cpu
Title: Message Title Maxim Kozin commented on PUP-5028 Re: 100% cpu We have hardware nodes with libvirt-lxc and LXC containers under puppet. Puppet run on both type servers, but hang only on libvirt-lxc nodes. Yes, we use custom facts which communicate with libvirt and execute some external command Code in attachment. Moreover - backtrace in gdb show that we loop in libvirt code, in poll() on nonexistent fd with timeout = -1: #0 pthread_sigmask (how=how@entry=2, newmask=, newmask@entry=0x7fff36a73900, oldmask=oldmask@entry=0x0) at ../nptl/sysdeps/pthread/pthread_sigmask.c:53 #1 0x7f185bcc213d in virNetClientIOEventLoop (client=client@entry=0x45c4ea0, thiscall=thiscall@entry=0x3748920) at rpc/virnetclient.c:1518 #2 0x7f185bcc28eb in virNetClientIO (thiscall=0x3748920, client=0x45c4ea0) at rpc/virnetclient.c:1786 #3 virNetClientSendInternal (client=client@entry=0x45c4ea0, msg=msg@entry=0x45c31b0, expectReply=expectReply@entry=true, nonBlock=nonBlock@entry=false) at rpc/virnetclient.c:1958 #4 0x7f185bcc3cf3 in virNetClientSendWithReply (client=client@entry=0x45c4ea0, msg=msg@entry=0x45c31b0) at rpc/virnetclient.c:1986 #5 0x7f185bcc4502 in virNetClientProgramCall (prog=prog@entry=0x45c4e30, client=client@entry=0x45c4ea0, serial=serial@entry=7, proc=proc@entry=2, noutfds=noutfds@entry=0, outfds=outfds@entry=0x0, ninfds=ninfds@entry=0x0, infds=infds@entry=0x0, args_filter=args_filter@entry=0x7f186c795700 <__GI_xdr_void>, args=args@entry=0x0, ret_filter=ret_filter@entry=0x7f186c795700 <__GI_xdr_void>, ret=ret@entry=0x0) at rpc/virnetclientprogram.c:329 #6 0x7f185bc9c992 in callFull (priv=priv@entry=0x45c9140, flags=flags@entry=0, fdin=fdin@entry=0x0, fdinlen=fdinlen@entry=0, fdout=fdout@entry=0x0, fdoutlen=fdoutlen@entry=0x0, proc_nr=proc_nr@entry=2, args_filter=0x7f186c795700 <_GI_xdr_void>, args=args@entry=0x0, ret_filter=0x7f186c795700 <_GI_xdr_void>, ret=ret@entry=0x0, conn=0x45c9140) at remote/remote_driver.c:6575 #7 0x7f185bcb494c in call (conn=0x45c9140, ret=0x0, ret_filter=, args=0x0, args_filter=, proc_nr=2, flags=0, priv=0x45c9140) at remote/remote_driver.c:6597 #8 doRemoteClose (priv=0x45c9140, conn=0x45c4ad0) at remote/remote_driver.c:1228 #9 0x7f185bcb4b7c in remoteConnectClose (conn=0x45c4ad0) at remote/remote_driver.c:1271 #10 0x7f185bc4bd5f in virConnectDispose (obj=0x45c4ad0) at datatypes.c:154 #11 0x7f185bbcb183 in virObjectUnref (anyobj=) at util/virobject.c:265 #12 0x7f185bc4bca3 in virDomainDispose (obj=0x45c4cf0) at datatypes.c:255 #13 0x7f185bbcb183 in virObjectUnref (anyobj=anyobj@entry=0x45c4cf0) at util/virobject.c:265 #14 0x7f185bc57969 in virDomainFree (domain=0x45c4cf0) at libvirt.c:2272 #15 0x7f1860495419 in domain_free (d=) at domain.c:95 #16 0x7f186d609191 in run_final (objspace=0x1766940, objspace=0x1766940, obj=63470080) at gc.c:1418 #17 finalize_list (objspace=objspace@entry=0x1766940, p=0x3c87a00) at gc.c:1432 #18 0x7f186d609240 in finalize_deferred (objspace=objspace@entry=0x1766940) at gc.c:1452 #19 0x7f186d60a03e in rb_gc_finalize_deferred () at gc.c:1461 #20 0x7f186d7107a5 in rb_threadptr_execute_interrupts (th=th@entry=0x1766590, blocking_timing=blocking_timing@entry=0) at thread.c:1956 #21 0x7f186d7109ec in rb_threadptr_execute_interrupts (th=th@entry=0x1766590, blocking_timing=blocking_timing@entry=0) at thread.c:1898 #22 0x7f186d6f35e4 in vm_call0_body (th=th@entry=0x1766590, ci=ci@entry=0x7fff36a74120, argv=) at vm_eval.c:238 #23 0x7f186d6f497b in vm_call0 (defined_class=, me=0x1857bf0, argv=, argc=1, id=1408, recv=59393760, th=0x1766590) at vm_eval.c:49 #24 rb_call0 (recv=59393760, mid=1408, argc=1, argv=, scope=CALL_FCALL, self=24898920) at vm_eval.c:324 #25 0x7f186d63fe21 in rb_class_new_instance (argc=1, argv=0x7f186dafba90, klass=) at object.c:1799 #26 0x7f186d6f24f2 in vm_call_cfunc_with_fra
Jira (PUP-5028) 100% cpu
Title: Message Title Peter Huene commented on PUP-5028 Re: 100% cpu Looks like the fact might be coming from this module, which has been updated not to call back into Puppet's type system to query pip version: https://github.com/stankevich/puppet-python/commit/fda0c748790ace9a1822fdfa912fc85499ddcf80 Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5028) 100% cpu
Title: Message Title Josh Cooper assigned an issue to Maxim Kozin Puppet / PUP-5028 100% cpu Change By: Josh Cooper Assignee: Kylo Ginsberg Maxim Kozin Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5028) 100% cpu
Title: Message Title Josh Cooper commented on PUP-5028 Re: 100% cpu Maxim Kozin can you run puppet with --debug as it should show the commands that puppet is attempting to execute. Also it looks like you have a custom fact that is calling into puppet? Can you share the fact code? Add Comment This message was sent by Atlassian JIRA (v6.4.5#64020-sha1:78acd6c) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5028) 100% cpu
Title: Message Title Maxim Kozin created an issue Puppet / PUP-5028 100% cpu Issue Type: Bug Affects Versions: PUP 3.7.5 Assignee: Kylo Ginsberg Components: Client Created: 2015/08/12 10:35 AM Environment: CentOS Linux release 7.0.1406 (Core) 3.10.0-123.13.2.el7.x86_64 Priority: Normal Reporter: Maxim Kozin puppet service periodically get >100% cpu load: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17976 root 20 0 408432 50620 1780 R 129.5 0.2 2616:17 puppet Use described here method (https://gist.github.com/jashmenn/4069637), I get ruby backtrace: /usr/share/ruby/vendor_ruby/puppet/util.rb:295:in `new' /usr/share/ruby/vendor_ruby/puppet/util.rb:295:in `block (2 levels) in safe_posix_fork' /usr/share/ruby/vendor_ruby/puppet/util.rb:295:in `upto' /usr/share/ruby/vendor_ruby/puppet/util.rb:295:in `block in safe_posix_fork' /usr/share/ruby/vendor_ruby/puppet/util.rb:290:in `fork' /usr/share/ruby/vendor_ruby/puppet/util.rb:290:in `safe_posix_fork' /usr/share/ruby/vendor_ruby/puppet/util/execution.rb:237:in
