kennknowles opened a new issue, #19241: URL: https://github.com/apache/beam/issues/19241
I'm looking at a flake in a postcommit suite that happened last week in Python Dataflow integration test (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are not sufficient to debug the problem. The test suite failed, however: 1. Test Result section in Jenkins postcommit only includes unit tests, and does not include any integration tests. 2. The only available information about the failure is buried in the Full Console Log and is not very informative (copy-pasted below). 3. There is a line in the console log pointing to: "XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml", however as a non-committer I don't have access to Jenkins machines. This makes rootcausing integration tests failures quite difficult, since we cannot even reliably identify an ID of a failed Dataflow job from the test output. It would be helpful to capture and persist in Jenkins the invocation of integration test, and all the console output that SDK produces during test execution. Sample available logs from console output. 13:41:54 test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... ERROR 13:41:54 13:41:54 \==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\==\== 13:41:54 ERROR: test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) 13:41:54 \--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\-- 13:41:54 Traceback (most recent call last): 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 812, in run 13:41:54 test(orig) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 45, in __call__ 13:41:54 return self.run(*arg, **kwarg) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 133, in run 13:41:54 self.runTest(result) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py", line 151, in runTest 13:41:54 test(result) 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__ 13:41:54 return self.run(*args, **kwds) 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run 13:41:54 testMethod() 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py", line 161, in test_leader_board_it 13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts)) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py", line 345, in run 13:41:54 'total_score': 'INTEGER', 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 414, in __exit__ 13:41:54 self.run().wait_until_finish() 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 394, in run 13:41:54 self.to_runner_api(), self.runner, self._options).run(False) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py", line 407, in run 13:41:54 return self.runner.run_pipeline(self) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline 13:41:54 self.result.cancel() 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1167, in cancel 13:41:54 self._update_job() 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 1078, in _update_job 13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id()) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py", line 184, in wrapper 13:41:54 return fun(*args, **kwargs) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py", line 629, in get_job 13:41:54 response = self._client.projects_locations_jobs.Get(request) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 659, in Get 13:41:54 config, request, global_params=global_params) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py", line 720, in _RunMethod 13:41:54 http, http_request, **opts) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 346, in MakeRequest 13:41:54 check_response_func=check_response_func) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py", line 396, in _MakeRequestNoRetry 13:41:54 redirections=redirections, connection_type=connection_type) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 193, in new_request 13:41:54 redirections, connection_type) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py", line 282, in request 13:41:54 connection_type=connection_type) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1694, in request 13:41:54 (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1434, in _request 13:41:54 (response, content) = self._conn_request(conn, request_uri, method, body, headers) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1390, in _conn_request 13:41:54 response = conn.getresponse() 13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse 13:41:54 response.begin() 13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin 13:41:54 version, status, reason = self._read_status() 13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status 13:41:54 line = self.fp.readline(_MAXLINE **** 1) 13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline 13:41:54 data = self._sock.recv(self._rbufsize) 13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv 13:41:54 return self.read(buflen) 13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read 13:41:54 v = self._sslobj.read(len) 13:41:54 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 276, in signalhandler 13:41:54 raise TimedOutException() 13:41:54 TimedOutException: 'test_leader_board_it (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)' 13:41:54 13:41:54 \--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\-- 13:41:54 XML: /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml 13:41:54 \--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\--\-- 13:41:54 Ran 14 tests in 3001.679s 13:41:54 13:41:54 FAILED (errors=1) 13:41:54 Imported from Jira [BEAM-5492](https://issues.apache.org/jira/browse/BEAM-5492). Original Jira may contain additional context. Reported by: tvalentyn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
