[
https://issues.apache.org/jira/browse/BEAM-5492?focusedWorklogId=767362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-767362
]
ASF GitHub Bot logged work on BEAM-5492:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/May/22 18:42
Start Date: 06/May/22 18:42
Worklog Time Spent: 10m
Work Description: andoni-guzman commented on PR #17530:
URL: https://github.com/apache/beam/pull/17530#issuecomment-1119914944
Hi @tvalentyn . I think is ready, but I would like to another test using for
example Python CoGBK Dataflow Streaming to verify that now the format is
correct.
Related to why in some tests publish relevant errors, in this particular
case I believe it's for the kind of error, in the second one ( 5191) I saw an
error that is not present in 2605 . Maybe that error invoke a log for that
problem.
Also , doing some research for this task, I notice that some tests use
test_dataflow_runner and other dataflow_runner.
Test_dataflow_runner had some logs that could help. But I added that kind of
logs in both files to get more info of a failure.
Yes hunting logs adds a lot of friction, maybe we can look in a future to
try to add logging.error in exceptions or something else. If you have an idea
of how we can improve this, I'll with the disposition to help
Issue Time Tracking
-------------------
Worklog Id: (was: 767362)
Time Spent: 4.5h (was: 4h 20m)
> Python Dataflow integration tests should export the pipeline console output
> to Jenkins Test Result section
> ----------------------------------------------------------------------------------------------------------
>
> Key: BEAM-5492
> URL: https://issues.apache.org/jira/browse/BEAM-5492
> Project: Beam
> Issue Type: New Feature
> Components: testing
> Reporter: Valentyn Tymofieiev
> Assignee: Andoni Guzman
> Priority: P3
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> I'm looking at a flake in a postcommit suite that happened last week in
> Python Dataflow integration test
> (https://issues.apache.org/jira/browse/BEAM-5415), and the existing logs are
> not sufficient to debug the problem. The test suite failed, however:
> 1. Test Result section in Jenkins postcommit only includes unit tests, and
> does not include any integration tests.
> 2. The only available information about the failure is buried in the Full
> Console Log and is not very informative (copy-pasted below).
> 3. There is a line in the console log pointing to: "XML:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml",
> however as a non-committer I don't have access to Jenkins machines.
> This makes rootcausing integration tests failures quite difficult, since we
> cannot even reliably identify an ID of a failed Dataflow job from the test
> output.
> It would be helpful to capture and persist in Jenkins the invocation of
> integration test, and all the console output that SDK produces during test
> execution.
> Sample available logs from console output.
> 13:41:54 test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ...
> ERROR
> 13:41:54
> 13:41:54
> ======================================================================
> 13:41:54 ERROR: test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 Traceback (most recent call last):
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
> line 812, in run
> 13:41:54 test(orig)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 45, in __call__
> 13:41:54 return self.run(*arg, **kwarg)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 133, in run
> 13:41:54 self.runTest(result)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
> line 151, in runTest
> 13:41:54 test(result)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
> 13:41:54 return self.run(*args, **kwds)
> 13:41:54 File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> 13:41:54 testMethod()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
> line 161, in test_leader_board_it
> 13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
> line 345, in run
> 13:41:54 'total_score': 'INTEGER',
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 414, in __exit__
> 13:41:54 self.run().wait_until_finish()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 394, in run
> 13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
> line 407, in run
> 13:41:54 return self.runner.run_pipeline(self)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 68, in run_pipeline
> 13:41:54 self.result.cancel()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1167, in cancel
> 13:41:54 self._update_job()
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1078, in _update_job
> 13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
> line 184, in wrapper
> 13:41:54 return fun(*args, **kwargs)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
> line 629, in get_job
> 13:41:54 response = self._client.projects_locations_jobs.Get(request)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
> line 659, in Get
> 13:41:54 config, request, global_params=global_params)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
> line 720, in _RunMethod
> 13:41:54 http, http_request, **opts)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
> line 346, in MakeRequest
> 13:41:54 check_response_func=check_response_func)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
> line 396, in _MakeRequestNoRetry
> 13:41:54 redirections=redirections, connection_type=connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
> line 193, in new_request
> 13:41:54 redirections, connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py",
> line 282, in request
> 13:41:54 connection_type=connection_type)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1694, in request
> 13:41:54 (response, content) = self._request(conn, authority, uri,
> request_uri, method, body, headers, redirections, cachekey)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1434, in _request
> 13:41:54 (response, content) = self._conn_request(conn, request_uri,
> method, body, headers)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py",
> line 1390, in _conn_request
> 13:41:54 response = conn.getresponse()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
> 13:41:54 response.begin()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 453, in begin
> 13:41:54 version, status, reason = self._read_status()
> 13:41:54 File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
> 13:41:54 line = self.fp.readline(_MAXLINE + 1)
> 13:41:54 File "/usr/lib/python2.7/socket.py", line 480, in readline
> 13:41:54 data = self._sock.recv(self._rbufsize)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 756, in recv
> 13:41:54 return self.read(buflen)
> 13:41:54 File "/usr/lib/python2.7/ssl.py", line 643, in read
> 13:41:54 v = self._sslobj.read(len)
> 13:41:54 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
> line 276, in signalhandler
> 13:41:54 raise TimedOutException()
> 13:41:54 TimedOutException: 'test_leader_board_it
> (apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)'
> 13:41:54
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 XML:
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml
> 13:41:54
> ----------------------------------------------------------------------
> 13:41:54 Ran 14 tests in 3001.679s
> 13:41:54
> 13:41:54 FAILED (errors=1)
> 13:41:54
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
