[openstack-dev] [devstack] libvirt default log level
Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. [1] https://review.openstack.org/#/c/63992/ https://review.openstack.org/#/c/63992/ [2] https://review.openstack.org/#/c/66630/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On 01/15/2014 11:56 AM, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. [1] https://review.openstack.org/#/c/63992/ https://review.openstack.org/#/c/63992/ [2] https://review.openstack.org/#/c/66630/ Why not just make this a shell variable/localrc option that defaults to the original and have the gate jobs set it to debug? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
+1 for your change. I've been hit by the very same issue today. Simon On 15/01/2014 17:56, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. [1] https://review.openstack.org/#/c/63992/ https://review.openstack.org/#/c/63992/ [2] https://review.openstack.org/#/c/66630/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On Wed, Jan 15, 2014 at 05:28:07PM +, Daniel P. Berrange wrote: On Wed, Jan 15, 2014 at 09:56:58AM -0700, Steven Dake wrote: Agreed, it is complete insanity to set 'log_level=1' on *any* hosts. The level of debug info that generates is so enourmous that you'd never wanted to look at it. Debugging from that would be like trying to find a specific piece of hay in a haystack. That original devstack change should just be completely reverted. It would be useful if Jenkins machines had an elevated log level, but even there you definitely don't want to set log_level=1. I would suggest that for Jenkin's machines *only* we want to try log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util log_outputs=1:file:/var/log/libvirt/libvirtd.log I see that something close to this has already been added to devstack: https://review.openstack.org/#/c/65834/2/lib/nova which is the right way to tailor logging levels. So we should definitely revert https://review.openstack.org/#/c/63992/ Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On Wed, Jan 15, 2014 at 09:56:58AM -0700, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. Agreed, it is complete insanity to set 'log_level=1' on *any* hosts. The level of debug info that generates is so enourmous that you'd never wanted to look at it. Debugging from that would be like trying to find a specific piece of hay in a haystack. That original devstack change should just be completely reverted. It would be useful if Jenkins machines had an elevated log level, but even there you definitely don't want to set log_level=1. I would suggest that for Jenkin's machines *only* we want to try log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util log_outputs=1:file:/var/log/libvirt/libvirtd.log Don't touch log settings at all on general developers' machines. If when developers want to toubleshoot something they are perfectly able to do so without us making unhelpful decisions for them. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On Wed, Jan 15, 2014 at 11:35 AM, Daniel P. Berrange berra...@redhat.comwrote: I see that something close to this has already been added to devstack: https://review.openstack.org/#/c/65834/2/lib/nova which is the right way to tailor logging levels. So we should definitely revert https://review.openstack.org/#/c/63992/ [Who approved that anyway?? Oh...] FWIW I lean in this direction unless there is a strong need to make this configurable. We have enough config vars to track already so if there is a strong response to that I'll make it so this can be set using the local.conf mechanism. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On 01/15/2014 10:28 AM, Daniel P. Berrange wrote: On Wed, Jan 15, 2014 at 09:56:58AM -0700, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. Agreed, it is complete insanity to set 'log_level=1' on *any* hosts. The level of debug info that generates is so enourmous that you'd never wanted to look at it. Debugging from that would be like trying to find a specific piece of hay in a haystack. That original devstack change should just be completely reverted. It would be useful if Jenkins machines had an elevated log level, but even there you definitely don't want to set log_level=1. I would suggest that for Jenkin's machines *only* we want to try log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util log_outputs=1:file:/var/log/libvirt/libvirtd.log Don't touch log settings at all on general developers' machines. If when developers want to toubleshoot something they are perfectly able to do so without us making unhelpful decisions for them. Daniel, Makes sense to me. https://review.openstack.org/#/c/66900/ Regards, Daniel ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On 01/15/2014 10:13 AM, David Kranz wrote: On 01/15/2014 11:56 AM, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. [1] https://review.openstack.org/#/c/63992/ https://review.openstack.org/#/c/63992/ [2] https://review.openstack.org/#/c/66630/ Why not just make this a shell variable/localrc option that defaults to the original and have the gate jobs set it to debug? David, I don't think devstack should be mucking with system config files including libvirt in the default case. If someone wants to propose a special case change to handle the gate jobs, that would solve the gate not having enough log information, as indicated in a followup thread from Daniel. Since the gate jobs are throw-away VMs, the problem of polluting developer workstation config files with devstack changes doesn't exist. Regards -steve -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
On 01/15/2014 01:00 PM, Steven Dake wrote: On 01/15/2014 10:28 AM, Daniel P. Berrange wrote: On Wed, Jan 15, 2014 at 09:56:58AM -0700, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. Agreed, it is complete insanity to set 'log_level=1' on *any* hosts. The level of debug info that generates is so enourmous that you'd never wanted to look at it. Debugging from that would be like trying to find a specific piece of hay in a haystack. That original devstack change should just be completely reverted. It would be useful if Jenkins machines had an elevated log level, but even there you definitely don't want to set log_level=1. I would suggest that for Jenkin's machines *only* we want to try log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util log_outputs=1:file:/var/log/libvirt/libvirtd.log Don't touch log settings at all on general developers' machines. If when developers want to toubleshoot something they are perfectly able to do so without us making unhelpful decisions for them. Daniel, Makes sense to me. https://review.openstack.org/#/c/66900/ We already do all sorts of changing of things on developer machines for devstack, so changing libvirt config does not seem out of scope. I agree that current level of debug is probably not right, which is why I wanted this on the list, so let's figure out what the right level is and set it. Especially given that there are bad interactions between nova and libvirt at the moment (which was the reason for the first patch). I'd also be ok with setting this behind a DEBUG_LIBVIRT flag. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [devstack] libvirt default log level
Hi, -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: Thursday, January 16, 2014 4:54 AM To: Daniel P. Berrange; OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [devstack] libvirt default log level On 01/15/2014 02:45 PM, Daniel P. Berrange wrote: On Wed, Jan 15, 2014 at 01:42:34PM -0500, Sean Dague wrote: On 01/15/2014 01:00 PM, Steven Dake wrote: On 01/15/2014 10:28 AM, Daniel P. Berrange wrote: On Wed, Jan 15, 2014 at 09:56:58AM -0700, Steven Dake wrote: Hi, Ken'ichi Omichi submitted a change [1] in devstack to change the default log level to 1 for libvirt. This results in continual spam to /var/log/messages in my development system, even after exiting devstack. The spam looks like: Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 8 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 9 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 10 Jan 14 08:13:49 bigiron libvirtd: 2014-01-14 15:13:49.334+: 1480: debug : virFileClose:90 : Closed fd 11 in a continual loop I submitted a change [2] that sets the default to level 2 (info + warnings + errors) which was -1'ed by Sean Dague. His suggestion was to take the discussion upstream so consensus around what the default should be can be made so this doesn't end up getting changed every week. The core mission of devstack is to provide a development environment for developers. The fact that it is being used in the gate seems somewhat ancillary to it's mission, and forcing a default of spam the system logs with tons of libvirt messages seems counter to the core mission of devstack. As is, without modification devstack makes looking at anything useful in my system logs impossible without grep -v libvirt and is intrusive to developer's workstations. Agreed, it is complete insanity to set 'log_level=1' on *any* hosts. The level of debug info that generates is so enourmous that you'd never wanted to look at it. Debugging from that would be like trying to find a specific piece of hay in a haystack. That original devstack change should just be completely reverted. It would be useful if Jenkins machines had an elevated log level, but even there you definitely don't want to set log_level=1. I would suggest that for Jenkin's machines *only* we want to try log_filters=1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 1:util log_outputs=1:file:/var/log/libvirt/libvirtd.log Don't touch log settings at all on general developers' machines. If when developers want to toubleshoot something they are perfectly able to do so without us making unhelpful decisions for them. Daniel, Makes sense to me. https://review.openstack.org/#/c/66900/ We already do all sorts of changing of things on developer machines for devstack, so changing libvirt config does not seem out of scope. I agree that current level of debug is probably not right, which is why I wanted this on the list, so let's figure out what the right level is and set it. A more suitable debugging level for troubleshooting is already set by this change https://review.openstack.org/#/c/65834/2/lib/nova based on a previous recommendation of mine in the referenced bug, which is close to what I suggested above Ok, sorry, I hadn't correctly connected the two reviews. Based on that I'm +2 on the previous revert. I'm sorry for my change, and thank you for reverting it. My purpose is for digging some gate failures related to libvirt. As the first step, we need to get a log file of libvirtd from gate and I am proposing https://review.openstack.org/#/c/61892/ now. (This is not a review request to openstack-dev ml;-) Thanks Ken'ichi Ohmichi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
