Public bug reported: Tests inside neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase seems to fail intermittently. The root cause seems like to be the same for all the tests...
Here are two different stack traces from two different patches... ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:49:54.288 | 2016-02-19 09:49:54.264 | ============================== 2016-02-19 09:49:54.312 | 2016-02-19 09:49:54.279 | Failed 1 tests - output below: 2016-02-19 09:49:54.321 | 2016-02-19 09:49:54.292 | ============================== 2016-02-19 09:49:54.332 | 2016-02-19 09:49:54.308 | 2016-02-19 09:49:54.335 | 2016-02-19 09:49:54.310 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_disable_network_port(native) 2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.312 | --------------------------------------------------------------------------------------------------------------- 2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.314 | 2016-02-19 09:49:54.343 | 2016-02-19 09:49:54.316 | Captured traceback: 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.317 | ~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.322 | Traceback (most recent call last): 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.328 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 242, in test_arp_spoof_disable_network_port 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.351 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2) 2016-02-19 09:49:54.378 | 2016-02-19 09:49:54.354 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping 2016-02-19 09:49:54.380 | 2016-02-19 09:49:54.357 | dst_ip]) 2016-02-19 09:49:54.385 | 2016-02-19 09:49:54.361 | File "neutron/agent/linux/ip_lib.py", line 885, in execute 2016-02-19 09:49:54.386 | 2016-02-19 09:49:54.363 | log_fail_as_error=log_fail_as_error, **kwargs) 2016-02-19 09:49:54.389 | 2016-02-19 09:49:54.365 | File "neutron/agent/linux/utils.py", line 140, in execute 2016-02-19 09:49:54.392 | 2016-02-19 09:49:54.368 | raise RuntimeError(msg) 2016-02-19 09:49:54.395 | 2016-02-19 09:49:54.371 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 2016-02-19 09:49:54.397 | 2016-02-19 09:49:54.374 | 2016-02-19 09:49:54.420 | 2016-02-19 09:49:54.376 | --- 192.168.0.2 ping statistics --- 2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.378 | 2 packets transmitted, 0 received, 100% packet loss, time 1005ms 2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.379 | 2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.382 | ; Stderr: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ============================== 2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below: 2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ============================== 2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 | 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native) 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ---------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 | 2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging: 2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 | 2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 | 2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback: 2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last): 2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic 2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2) 2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping 2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip]) 2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute 2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs) 2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute 2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg) 2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 | 2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics --- 2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms 2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 | 2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr: 2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ============================== 2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below: 2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ============================== 2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 | 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native) 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ---------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 | 2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging: 2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 | 2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 | 2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback: 2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last): 2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic 2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2) 2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping 2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip]) 2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute 2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs) 2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute 2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg) 2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 | 2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics --- 2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms 2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 | 2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr: 2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 | ** Affects: neutron Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1547486 Title: ARPSpoofOFCtlTestCase functional gate tests failing intermittently Status in neutron: New Bug description: Tests inside neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase seems to fail intermittently. The root cause seems like to be the same for all the tests... Here are two different stack traces from two different patches... ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:49:54.288 | 2016-02-19 09:49:54.264 | ============================== 2016-02-19 09:49:54.312 | 2016-02-19 09:49:54.279 | Failed 1 tests - output below: 2016-02-19 09:49:54.321 | 2016-02-19 09:49:54.292 | ============================== 2016-02-19 09:49:54.332 | 2016-02-19 09:49:54.308 | 2016-02-19 09:49:54.335 | 2016-02-19 09:49:54.310 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_disable_network_port(native) 2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.312 | --------------------------------------------------------------------------------------------------------------- 2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.314 | 2016-02-19 09:49:54.343 | 2016-02-19 09:49:54.316 | Captured traceback: 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.317 | ~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.322 | Traceback (most recent call last): 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.328 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 242, in test_arp_spoof_disable_network_port 2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.351 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2) 2016-02-19 09:49:54.378 | 2016-02-19 09:49:54.354 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping 2016-02-19 09:49:54.380 | 2016-02-19 09:49:54.357 | dst_ip]) 2016-02-19 09:49:54.385 | 2016-02-19 09:49:54.361 | File "neutron/agent/linux/ip_lib.py", line 885, in execute 2016-02-19 09:49:54.386 | 2016-02-19 09:49:54.363 | log_fail_as_error=log_fail_as_error, **kwargs) 2016-02-19 09:49:54.389 | 2016-02-19 09:49:54.365 | File "neutron/agent/linux/utils.py", line 140, in execute 2016-02-19 09:49:54.392 | 2016-02-19 09:49:54.368 | raise RuntimeError(msg) 2016-02-19 09:49:54.395 | 2016-02-19 09:49:54.371 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 2016-02-19 09:49:54.397 | 2016-02-19 09:49:54.374 | 2016-02-19 09:49:54.420 | 2016-02-19 09:49:54.376 | --- 192.168.0.2 ping statistics --- 2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.378 | 2 packets transmitted, 0 received, 100% packet loss, time 1005ms 2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.379 | 2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.382 | ; Stderr: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ============================== 2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below: 2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ============================== 2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 | 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native) 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ---------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 | 2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging: 2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 | 2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 | 2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback: 2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last): 2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic 2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2) 2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping 2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip]) 2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute 2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs) 2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute 2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg) 2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 | 2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics --- 2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms 2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 | 2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr: 2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ============================== 2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below: 2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ============================== 2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 | 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native) 2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ---------------------------------------------------------------------------------------------------------------------- 2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 | 2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging: 2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json 2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 | 2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 | 2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback: 2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~ 2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last): 2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic 2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2) 2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping 2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip]) 2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute 2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs) 2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute 2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg) 2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. 2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 | 2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics --- 2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms 2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 | 2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr: 2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 | To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1547486/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp