Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
On 05/28/2015 07:56 PM, George Shuklin wrote: Hello. Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. Just to let everyone to know: Update to proposed version of python-eventlet fixes the problem. Proposed debs can be found here: https://launchpad.net/ubuntu/+source/python-eventlet ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
On 28 May 2015, at 19:10, Tim Bell tim.b...@cern.ch wrote: Using UDP is a great workaround but it does not feel like a fix... can't the daemons realise that the syslog socket is not alive and reconnect. Given it affects most of the OpenStack projects, a fix inside one of the oslo logging libraries (if the error can be detected there) would be great. We too have been bitten hard by this issue in the past - way before Juno - when using rsyslog logging to a remote target (i.e Logstash). We eventually went down the route of using log-courier [1] on x86 and beaver [2] on ARM (due to the lack of support for Go). Both have worked out well for us - if you’re using Logstash it might be worth looking into either of these as a solution instead of switching to UDP and hoping that you don’t lose any messages that you might care about. [1] https://github.com/driskell/log-courier https://github.com/driskell/log-courier [2] https://github.com/josegonzalez/python-beaver https://github.com/josegonzalez/python-beaver — -Nick -- DataCentred Limited registered in England and Wales no. 05611763 ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
Hi, As I commented on the bug, it sounds similar with: https://bugs.launchpad.net/ubuntu/+source/python-eventlet/+bug/1452312 https://github.com/eventlet/eventlet/issues/192 2015-05-29 1:56 GMT+09:00 George Shuklin george.shuk...@gmail.com: Hello. Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators -- Nobuto Murata / 村田信人 ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
-Original Message- From: Christian Schwede [mailto:cschw...@redhat.com] Sent: 28 May 2015 20:03 To: George Shuklin; openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted On 28.05.15 18:56, George Shuklin wrote: Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. Yes, the Swift project hit this bug in the past too, though it was fixed awhile ago: Using UDP is a great workaround but it does not feel like a fix... can't the daemons realise that the syslog socket is not alive and reconnect. Given it affects most of the OpenStack projects, a fix inside one of the oslo logging libraries (if the error can be detected there) would be great. https://bugs.launchpad.net/swift/+bug/780025 https://review.openstack.org/#/c/8841/ Summarized: using UDP instead of a unix socket fixes this. -- Christian Tim ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
Hello, Yeah, I ran into it last fall: http://www.gossamer-threads.com/lists/openstack/operators/41876 Good to know that this issue still exists in Juno (we're still on Icehouse). Thanks for the note. :) Joe On Thu, May 28, 2015 at 10:56 AM, George Shuklin george.shuk...@gmail.com wrote: Hello. Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
I bet you can workaround this by switching to UDP remote syslog. You'd loose messages, but your processes *should* fire and forget On May 28, 2015, at 9:56 AM, George Shuklin george.shuk...@gmail.com wrote: Hello. Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted
On 28.05.15 18:56, George Shuklin wrote: Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. Yes, the Swift project hit this bug in the past too, though it was fixed awhile ago: https://bugs.launchpad.net/swift/+bug/780025 https://review.openstack.org/#/c/8841/ Summarized: using UDP instead of a unix socket fixes this. -- Christian ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] 100% CPU and hangs if syslog is restarted
Hello. Today we've discover a very serious bug in juno: https://bugs.launchpad.net/nova/+bug/1459726 In short: if you're using syslog, and restart rsyslog, all APIs processes will eventually stuck with 100% CPU usage without doing anything. Is anyone hits this bug before? It looks like very nasty. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
