Public bug reported: Sometimes the following error appears in the linuxbridge q-agt log: [1]
Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1066', 'up'] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected This happens after a vxlan device has been created and should be set up. Interesting is, that it's always the first vxlan device that has been created (Had a look at 3 different logs). All vxlan devices that have been created after that could be set up fine. After that a agent resync is triggered. The ensure_vxlan method detects, that the interface is already there and does not create it again. But it does not check if it is up as well - so it may still be in the down state (don't know, can't locally reproduce it) and never put up. This might be an issue. Along logstash, the issue seemed to be occured the first time at January 13 [2]. The tests do not seem to fail due to this error, there are about as many succeeded runs as failed runs when this error shows up in the log. Probably that's because it's just a single node testing, and no package will ever leave the devstack node.... A workaround could be to modify ensure_vxlan to add a check if it is up (and not just checking if it exists). More details from [1]: 2016-01-20 16:44:35.226 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Creating vxlan interface vxlan-1066 for VNI 1066 ensure_vxlan /opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:265 2016-01-20 16:44:35.228 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Running command (rootwrap daemon): ['ip', 'link', 'add', 'vxlan-1066', 'type', 'vxlan', 'id', '1066', 'group', '224.0.0.1', 'dev', 'eth0'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:100 2016-01-20 16:44:35.316 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Exit code: 0 execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:142 2016-01-20 16:44:35.316 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1066', 'up'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:100 2016-01-20 16:44:35.329 ERROR neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected 2016-01-20 16:44:35.333 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Error in agent loop. Devices info: {'current': set(['tapce39edcc-d9']), 'removed': set([]), 'added': set(['tapce39edcc-d9']), 'updated': set([])} 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last): 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1161, in daemon_loop 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 967, in process_network_devices 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent resync_a = self.treat_devices_added_updated(devices_added_updated) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1001, in treat_devices_added_updated 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent device_details['port_id'], device_details['device_owner']) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 473, in add_interface 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent tap_device_name, device_owner) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 435, in add_tap_interface 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent segmentation_id) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 391, in ensure_physical_in_bridge 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent return self.ensure_vxlan_bridge(network_id, segmentation_id) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 200, in ensure_vxlan_bridge 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent interface = self.ensure_vxlan(segmentation_id) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 288, in ensure_vxlan 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent int_vxlan.link.set_up() 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 461, in set_up 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent return self._as_root([], ('set', self.name, 'up')) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 321, in _as_root 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent use_root_namespace=use_root_namespace) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 94, in _as_root 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent log_fail_as_error=self.log_fail_as_error) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 103, in _execute 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent log_fail_as_error=log_fail_as_error) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 140, in execute 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent raise RuntimeError(msg) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected [1] http://logs.openstack.org/69/143169/69/check/gate-tempest-dsvm-neutron-linuxbridge/cd66b94/logs/screen-q-agt.txt.gz [2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Transport%20endpoint%20is%20not%20connected%5C%22%20AND%20build_name%3A%5C%22gate-tempest-dsvm-neutron-linuxbridge%5C%22 ** Affects: neutron Importance: Undecided Status: New ** Tags: linuxbridge -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1536942 Title: lb: "RTNETLINK answers: Transport endpoint is not connected" when setting vxlan device Status in neutron: New Bug description: Sometimes the following error appears in the linuxbridge q-agt log: [1] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1066', 'up'] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected This happens after a vxlan device has been created and should be set up. Interesting is, that it's always the first vxlan device that has been created (Had a look at 3 different logs). All vxlan devices that have been created after that could be set up fine. After that a agent resync is triggered. The ensure_vxlan method detects, that the interface is already there and does not create it again. But it does not check if it is up as well - so it may still be in the down state (don't know, can't locally reproduce it) and never put up. This might be an issue. Along logstash, the issue seemed to be occured the first time at January 13 [2]. The tests do not seem to fail due to this error, there are about as many succeeded runs as failed runs when this error shows up in the log. Probably that's because it's just a single node testing, and no package will ever leave the devstack node.... A workaround could be to modify ensure_vxlan to add a check if it is up (and not just checking if it exists). More details from [1]: 2016-01-20 16:44:35.226 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Creating vxlan interface vxlan-1066 for VNI 1066 ensure_vxlan /opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:265 2016-01-20 16:44:35.228 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Running command (rootwrap daemon): ['ip', 'link', 'add', 'vxlan-1066', 'type', 'vxlan', 'id', '1066', 'group', '224.0.0.1', 'dev', 'eth0'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:100 2016-01-20 16:44:35.316 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Exit code: 0 execute /opt/stack/new/neutron/neutron/agent/linux/utils.py:142 2016-01-20 16:44:35.316 DEBUG neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1066', 'up'] execute_rootwrap_daemon /opt/stack/new/neutron/neutron/agent/linux/utils.py:100 2016-01-20 16:44:35.329 ERROR neutron.agent.linux.utils [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected 2016-01-20 16:44:35.333 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-8731d5a6-8d07-4312-8d28-5746487f21cb None None] Error in agent loop. Devices info: {'current': set(['tapce39edcc-d9']), 'removed': set([]), 'added': set(['tapce39edcc-d9']), 'updated': set([])} 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last): 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1161, in daemon_loop 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent sync = self.process_network_devices(device_info) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 967, in process_network_devices 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent resync_a = self.treat_devices_added_updated(devices_added_updated) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 1001, in treat_devices_added_updated 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent device_details['port_id'], device_details['device_owner']) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 473, in add_interface 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent tap_device_name, device_owner) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 435, in add_tap_interface 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent segmentation_id) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 391, in ensure_physical_in_bridge 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent return self.ensure_vxlan_bridge(network_id, segmentation_id) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 200, in ensure_vxlan_bridge 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent interface = self.ensure_vxlan(segmentation_id) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 288, in ensure_vxlan 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent int_vxlan.link.set_up() 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 461, in set_up 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent return self._as_root([], ('set', self.name, 'up')) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 321, in _as_root 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent use_root_namespace=use_root_namespace) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 94, in _as_root 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent log_fail_as_error=self.log_fail_as_error) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 103, in _execute 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent log_fail_as_error=log_fail_as_error) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 140, in execute 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent raise RuntimeError(msg) 2016-01-20 16:44:35.333 23050 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Transport endpoint is not connected [1] http://logs.openstack.org/69/143169/69/check/gate-tempest-dsvm-neutron-linuxbridge/cd66b94/logs/screen-q-agt.txt.gz [2] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Transport%20endpoint%20is%20not%20connected%5C%22%20AND%20build_name%3A%5C%22gate-tempest-dsvm-neutron-linuxbridge%5C%22 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1536942/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp