Public bug reported: A watchful reviewer may notice that sometimes test_dashboard_help_redirection fails. Initially I thought that it was caught by an increased network timeout from Jenkins worker node to docs.openstack.org site. So I made a patch https://github.com/openstack/horizon/commit/6474215840d4cdde3751c335f46f32cb317238c0
Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root): delay.sh <BEGIN OF FILE> delay=$1 ip=$2 tc qdisc del dev eth0 root if [ -n $delay ]; then tc qdisc add dev eth0 root handle 1: prio tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1 tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms fi <END OF FILE> I realized that first the window is opened, then it waits for the page to become accessible, and if I set timeout large enough (say, 10 seconds), I get the following traceback: Error Traceback (most recent call last): File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__ return self.run(*args, **kwds) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run return run_test.run(result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run return self._run_one(actual_result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one return self._run_prepared_result(ExtendedToOriginalDecorator(result)) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result self._run_core() File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core self.case._run_teardown, self.result): File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user return self._got_user_exception(sys.exc_info()) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception self.case.onException(exc_info, tb_label=tb_label) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException handler(exc_info) File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot self.driver.get_screenshot_as_file(filename) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file png = self.get_screenshot_as_png() File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png return base64.b64decode(self.get_screenshot_as_base64().encode('ascii')) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64 return self.execute(Command.SCREENSHOT)['value'] File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute response = self.command_executor.execute(driver_command, params) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute return self._request(command_info[0], url, body=data) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request self._conn.request(method, parsed_url.path, body, headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused It's different from the traceback that follows intermittent failure on gate: 2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last): 2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/tests/test_dashboard_help_redirection.py", line 28, in test_dashboard_help_redirection 2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL") 2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual 2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat(observed, matcher, message) 2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat 2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error 2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.matchers._impl.MismatchError: 'http://docs.openstack.org/' != u'http://localhost/dashboard/project/': help link did not redirect to the right URL The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial. ** Affects: horizon Importance: Undecided Status: New ** Tags: integration-tests ** Description changed: A watchful reviewer may notice that sometimes test_dashboard_help_redirection fails. Initially I thought that it was caught by an increased network timeout from Jenkins worker node to docs.openstack.org site. So I made a patch https://github.com/openstack/horizon/commit/6474215840d4cdde3751c335f46f32cb317238c0 Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root): delay.sh <BEGIN OF FILE> delay=$1 ip=$2 tc qdisc del dev eth0 root if [ -n $delay ]; then - tc qdisc add dev eth0 root handle 1: prio - tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1 - tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms + tc qdisc add dev eth0 root handle 1: prio + tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1 + tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms fi <END OF FILE> - I realized that first the window is opened, then it hangs for the page - to become accessible, and if I set timeout large enough (say, 30 + I realized that first the window is opened, then it waits for the page + to become accessible, and if I set timeout large enough (say, 10 seconds), I get the following traceback: Error Traceback (most recent call last): - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run - self.runTest(result) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest - test(result) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__ - return self.run(*args, **kwds) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run - return run_test.run(result) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run - return self._run_one(actual_result) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one - return self._run_prepared_result(ExtendedToOriginalDecorator(result)) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result - self._run_core() - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core - self.case._run_teardown, self.result): - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user - return self._got_user_exception(sys.exc_info()) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception - self.case.onException(exc_info, tb_label=tb_label) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException - handler(exc_info) - File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot - self.driver.get_screenshot_as_file(filename) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file - png = self.get_screenshot_as_png() - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png - return base64.b64decode(self.get_screenshot_as_base64().encode('ascii')) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64 - return self.execute(Command.SCREENSHOT)['value'] - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute - response = self.command_executor.execute(driver_command, params) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute - return self._request(command_info[0], url, body=data) - File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request - self._conn.request(method, parsed_url.path, body, headers) - File "/usr/lib/python2.7/httplib.py", line 979, in request - self._send_request(method, url, body, headers) - File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request - self.endheaders(body) - File "/usr/lib/python2.7/httplib.py", line 975, in endheaders - self._send_output(message_body) - File "/usr/lib/python2.7/httplib.py", line 835, in _send_output - self.send(msg) - File "/usr/lib/python2.7/httplib.py", line 797, in send - self.connect() - File "/usr/lib/python2.7/httplib.py", line 778, in connect - self.timeout, self.source_address) - File "/usr/lib/python2.7/socket.py", line 571, in create_connection - raise err + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run + self.runTest(result) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest + test(result) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__ + return self.run(*args, **kwds) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run + return run_test.run(result) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run + return self._run_one(actual_result) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one + return self._run_prepared_result(ExtendedToOriginalDecorator(result)) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result + self._run_core() + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core + self.case._run_teardown, self.result): + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user + return self._got_user_exception(sys.exc_info()) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception + self.case.onException(exc_info, tb_label=tb_label) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException + handler(exc_info) + File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot + self.driver.get_screenshot_as_file(filename) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file + png = self.get_screenshot_as_png() + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png + return base64.b64decode(self.get_screenshot_as_base64().encode('ascii')) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64 + return self.execute(Command.SCREENSHOT)['value'] + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute + response = self.command_executor.execute(driver_command, params) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute + return self._request(command_info[0], url, body=data) + File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request + self._conn.request(method, parsed_url.path, body, headers) + File "/usr/lib/python2.7/httplib.py", line 979, in request + self._send_request(method, url, body, headers) + File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request + self.endheaders(body) + File "/usr/lib/python2.7/httplib.py", line 975, in endheaders + self._send_output(message_body) + File "/usr/lib/python2.7/httplib.py", line 835, in _send_output + self.send(msg) + File "/usr/lib/python2.7/httplib.py", line 797, in send + self.connect() + File "/usr/lib/python2.7/httplib.py", line 778, in connect + self.timeout, self.source_address) + File "/usr/lib/python2.7/socket.py", line 571, in create_connection + raise err error: [Errno 111] Connection refused It's different from the traceback that follows intermittent failure on gate: 2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last): 2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/tests/test_dashboard_help_redirection.py", line 28, in test_dashboard_help_redirection 2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL") 2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual 2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat(observed, matcher, message) 2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat 2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error 2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.matchers._impl.MismatchError: 'http://docs.openstack.org/' != u'http://localhost/dashboard/project/': help link did not redirect to the right URL The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial. ** Tags added: integration-tests -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1520582 Title: Recent fix to integration tests didn't fix the intermittent test_dashboard_help_redirection failure Status in OpenStack Dashboard (Horizon): New Bug description: A watchful reviewer may notice that sometimes test_dashboard_help_redirection fails. Initially I thought that it was caught by an increased network timeout from Jenkins worker node to docs.openstack.org site. So I made a patch https://github.com/openstack/horizon/commit/6474215840d4cdde3751c335f46f32cb317238c0 Recently I found that this intermittent failure still occurs. This time I reproduced the network delay by means of script (run as root): delay.sh <BEGIN OF FILE> delay=$1 ip=$2 tc qdisc del dev eth0 root if [ -n $delay ]; then tc qdisc add dev eth0 root handle 1: prio tc filter add dev eth0 parent 1:0 protocol ip pref 55 handle ::55 u32 match ip dst $ip flowid 2:1 tc qdisc add dev eth0 parent 1:1 handle 2: netem delay ${delay}ms fi <END OF FILE> I realized that first the window is opened, then it waits for the page to become accessible, and if I set timeout large enough (say, 10 seconds), I get the following traceback: Error Traceback (most recent call last): File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run self.runTest(result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest test(result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/unittest2/case.py", line 673, in __call__ return self.run(*args, **kwds) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 606, in run return run_test.run(result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 80, in run return self._run_one(actual_result) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 94, in _run_one return self._run_prepared_result(ExtendedToOriginalDecorator(result)) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 108, in _run_prepared_result self._run_core() File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 149, in _run_core self.case._run_teardown, self.result): File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 193, in _run_user return self._got_user_exception(sys.exc_info()) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/runtest.py", line 213, in _got_user_exception self.case.onException(exc_info, tb_label=tb_label) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 558, in onException handler(exc_info) File "/home/timur/develop/horizon/openstack_dashboard/test/integration_tests/helpers.py", line 132, in _save_screenshot self.driver.get_screenshot_as_file(filename) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 758, in get_screenshot_as_file png = self.get_screenshot_as_png() File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 777, in get_screenshot_as_png return base64.b64decode(self.get_screenshot_as_base64().encode('ascii')) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 787, in get_screenshot_as_base64 return self.execute(Command.SCREENSHOT)['value'] File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 199, in execute response = self.command_executor.execute(driver_command, params) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute return self._request(command_info[0], url, body=data) File "/home/timur/develop/horizon/.venv/local/lib/python2.7/site-packages/selenium/webdriver/remote/remote_connection.py", line 425, in _request self._conn.request(method, parsed_url.path, body, headers) File "/usr/lib/python2.7/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request self.endheaders(body) File "/usr/lib/python2.7/httplib.py", line 975, in endheaders self._send_output(message_body) File "/usr/lib/python2.7/httplib.py", line 835, in _send_output self.send(msg) File "/usr/lib/python2.7/httplib.py", line 797, in send self.connect() File "/usr/lib/python2.7/httplib.py", line 778, in connect self.timeout, self.source_address) File "/usr/lib/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused It's different from the traceback that follows intermittent failure on gate: 2015-11-24 19:48:29.619 | 2015-11-24 19:48:29.606 | Traceback (most recent call last): 2015-11-24 19:48:29.620 | 2015-11-24 19:48:29.607 | File "/opt/stack/new/horizon/openstack_dashboard/test/integration_tests/tests/test_dashboard_help_redirection.py", line 28, in test_dashboard_help_redirection 2015-11-24 19:48:29.621 | 2015-11-24 19:48:29.608 | "help link did not redirect to the right URL") 2015-11-24 19:48:29.623 | 2015-11-24 19:48:29.609 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 350, in assertEqual 2015-11-24 19:48:29.624 | 2015-11-24 19:48:29.610 | self.assertThat(observed, matcher, message) 2015-11-24 19:48:29.625 | 2015-11-24 19:48:29.611 | File "/opt/stack/new/horizon/.tox/py27integration/local/lib/python2.7/site-packages/testtools/testcase.py", line 435, in assertThat 2015-11-24 19:48:29.626 | 2015-11-24 19:48:29.613 | raise mismatch_error 2015-11-24 19:48:29.627 | 2015-11-24 19:48:29.614 | testtools.matchers._impl.MismatchError: 'http://docs.openstack.org/' != u'http://localhost/dashboard/project/': help link did not redirect to the right URL The second one looks more like the command to switch the window was issued before window was actually opened. If that's truly the root cause, fix should be trivial. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1520582/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp