Re: [Openstack] LinuxBridge Agent Error

Brian Haley Wed, 12 Apr 2017 08:34:42 -0700

On 04/11/2017 06:35 PM, Daniel Russell wrote:

This is somewhat unrelated to the original email, but more of a question of an 
issue I have recently come across that seems tangentially related.


When run_one_command is executed and the message is sent to rootwrap via that pipe, if 
rootwrap errors for some reason, I get " Unserializable message: ('#ERROR',  
ValueError('I/O operation on closed file',))" returned with a stack trace.

In this case it appears the problem was that conntrack wasn't installed.  In my 
case it was that the rootwrap filters denied the request to kill the process.  
I only tracked it down because I configured syslog on the rootwrap daemon and 
killed the process (so l3-agent would respawn it, retry the command, and error 
again).  The error then appeared very clearly in /var/log/messages, so rootwrap 
does understand what is going on but it doesn't seem to feed it back to the 
caller.

Is this a known listed bug?  It would make diagnosis a lot easier if an easy to 
understand error message came back instead of the above.


Hi Dan,

https://bugs.launchpad.net/neutron/+bug/1677742 was the bug I opened forthe original rootwrap error, and the change for that merged upstream.That will catch the error and print the command being tried, which is atleast a bread crumb and finding the failure. Could be there is moreinfo we can gather, if you can try that change and add info to the bugit would be great.


-Brian

-----Original Message-----
From: Brian Haley [mailto:haleyb....@gmail.com]
Sent: Tuesday, 11 April 2017 6:24 AM
To: openstack@lists.openstack.org
Subject: Re: [Openstack] LinuxBridge Agent Error

On 04/10/2017 04:15 PM, Georgios Dimitrakakis wrote:

Installing the "conntrack-tools" package from the
"centos-openstack-ocata" repository seems to have fixed the problem.

My question is if this should have happened automatically during
installation and if yes why it didn't happen?


 From your comment it was a manual install.  Looking at the 
neutron-sanity-check tool it doesn't seem to check for conntrack.  Fix proposed 
at https://review.openstack.org/455450

Thanks!

-Brian

On Fri, 07 Apr 2017 19:26:02 +0300, Georgios Dimitrakakis wrote:

Hello,

I am on CentOS 7 and this is a manual Ocata installation as described
on the deployment guide.

My cloud seems to be operating normally even with these messages. The
thing is that I don't like all these (there are a whole bunch of them
every second) in my log files.

I will try to see if I am missing the "conntrack" and try to install
the patch that will provide more info..


Best,

G.

Is this a packstack installation?

Inviato da iPhone

Il giorno 07 apr 2017, alle ore di 07:19, Kevin Benton  ha scritto:

You may be missing whatever package in your distro provides the
'conntrack' util.

On Apr 7, 2017 03:52, "Georgios Dimitrakakis" wrote:

Hello,

I am having an Ocata installation and on my Compute Node the
"linuxbridge-agent.log" file is constantly filling out with
messages like this:

2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[1]ent._common_agent Traceback (most recent call last):
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[2]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
_common_agent.py",

line 453, in daemon_loop
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[3]ent._common_agent sync =
self.process_network_devices(device_info)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[4]ent._common_agent File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
153, in wrapper
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[5]ent._common_agent return f(*args, **kwargs)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[6]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
_common_agent.py",

line 213, in process_network_devices
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[7]ent._common_agent resync_b =
self.treat_devices_removed(device_info['removed'])
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[8]ent._common_agent File
"/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
153, in wrapper
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[9]ent._common_agent return f(*args, **kwargs)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[10]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
_common_agent.py",

line 331, in treat_devices_removed
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[11]ent._common_agent self.sg_agent.remove_devices_filter(devices)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[12]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.p
y",

line 238, in remove_devices_filter
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[13]ent._common_agent self.firewall.remove_port_filter(device)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[14]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
all.py",

line 222, in remove_port_filter
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[15]ent._common_agent
self._remove_conntrack_entries_from_port_deleted(port)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[16]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
all.py",

line 194, in _remove_conntrack_entries_from_port_deleted
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[17]ent._common_agent [device_info], ethertype, set())
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[18]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
y",

line 121, in delete_conntrack_state_by_remote_ips
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[19]ent._common_agent
self._delete_conntrack_state(device_info_list, rule)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[20]ent._common_agent File


"/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
y",

line 103, in _delete_conntrack_state
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[21]ent._common_agent extra_ok_codes=[1])
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[22]ent._common_agent File
"/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
line 128, in execute
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[23]ent._common_agent execute_rootwrap_daemon(cmd, process_input,
addl_env))
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[24]ent._common_agent File
"/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
line 115, in execute_rootwrap_daemon
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[25]ent._common_agent return client.execute(cmd, process_input)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[26]ent._common_agent File
"/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line
128, in execute
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[27]ent._common_agent res = proxy.run_one_command(cmd, stdin)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[28]ent._common_agent File "", line 2, in run_one_command
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[29]ent._common_agent File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
_callmethod
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[30]ent._common_agent raise convert_to_error(kind, result)
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[31]ent._common_agent RemoteError:
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[32]ent._common_agent


--------------------------------------------------------------------
-------

2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[33]ent._common_agent Unserializable message: ('#ERROR',
ValueError('I/O operation on closed file',))
2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[34]ent._common_agent


--------------------------------------------------------------------
-------

2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
[35]ent._common_agent
2017-04-07 13:37:47.710 1397 INFO neutron.plugins.ml2.drivers.ag
[36]ent._common_agent [req-93157f5e-9f2d-4547-9002-c473211df719 -
- - - -] Linux bridge agent Agent out of sync with plugin!

Any idea what might be the problem???

The full log of a total circle of messages can be found here:
https://pastebin.com/6Yzn0FuY [37]

Looking forward for an answer!

Best regards,

G.

_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [38]
Post to : openstack@lists.openstack.org [39] Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [40]

!DSPAM:1,58e777a158574020391806!

_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [42]
Post to : openstack@lists.openstack.org [43] Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [44]

!DSPAM:1,58e777a158574020391806!



Links:
------
[1] http://neutron.plugins.ml2.drivers.ag
[2] http://neutron.plugins.ml2.drivers.ag
[3] http://neutron.plugins.ml2.drivers.ag
[4] http://neutron.plugins.ml2.drivers.ag
[5] http://neutron.plugins.ml2.drivers.ag
[6] http://neutron.plugins.ml2.drivers.ag
[7] http://neutron.plugins.ml2.drivers.ag
[8] http://neutron.plugins.ml2.drivers.ag
[9] http://neutron.plugins.ml2.drivers.ag
[10] http://neutron.plugins.ml2.drivers.ag
[11] http://neutron.plugins.ml2.drivers.ag
[12] http://neutron.plugins.ml2.drivers.ag
[13] http://neutron.plugins.ml2.drivers.ag
[14] http://neutron.plugins.ml2.drivers.ag
[15] http://neutron.plugins.ml2.drivers.ag
[16] http://neutron.plugins.ml2.drivers.ag
[17] http://neutron.plugins.ml2.drivers.ag
[18] http://neutron.plugins.ml2.drivers.ag
[19] http://neutron.plugins.ml2.drivers.ag
[20] http://neutron.plugins.ml2.drivers.ag
[21] http://neutron.plugins.ml2.drivers.ag
[22] http://neutron.plugins.ml2.drivers.ag
[23] http://neutron.plugins.ml2.drivers.ag
[24] http://neutron.plugins.ml2.drivers.ag
[25] http://neutron.plugins.ml2.drivers.ag
[26] http://neutron.plugins.ml2.drivers.ag
[27] http://neutron.plugins.ml2.drivers.ag
[28] http://neutron.plugins.ml2.drivers.ag
[29] http://neutron.plugins.ml2.drivers.ag
[30] http://neutron.plugins.ml2.drivers.ag
[31] http://neutron.plugins.ml2.drivers.ag
[32] http://neutron.plugins.ml2.drivers.ag
[33] http://neutron.plugins.ml2.drivers.ag
[34] http://neutron.plugins.ml2.drivers.ag
[35] http://neutron.plugins.ml2.drivers.ag
[36] http://neutron.plugins.ml2.drivers.ag
[37] https://pastebin.com/6Yzn0FuY
[38] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[39] mailto:openstack@lists.openstack.org
[40] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[41] mailto:gior...@acmac.uoc.gr
[42] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[43] mailto:openstack@lists.openstack.org
[44] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[45] mailto:ke...@benton.pub



_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to     : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] LinuxBridge Agent Error

Reply via email to