Hi everyone, The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron" is failing[1]. The task :sdks:python:docs is failing with this traceback:
Traceback (most recent call last): File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/bin/sphinx-apidoc", line 8, in <module> from sphinx.apidoc import main File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/apidoc.py", line 27, in <module> from sphinx.quickstart import EXTENSIONS File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/quickstart.py", line 35, in <module> from docutils.utils import column_width File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/utils/__init__.py", line 21, in <module> import docutils.io File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/io.py", line 348 (self.destination.mode, mode)), file=self._stderr) ^ SyntaxError: invalid syntax It seems to be due to a bug in docutils which is a dependency of sphinx. Unfortunately, I don't have much context about this. I'd love to help with fixing this. There is another failure in test_wordcount_it of task :sdks:python:test-suites:dataflow:preCommitIT, but it seems to be a flaky test. [1] https://scans.gradle.com/s/ygctjiqxcz6qa/console-log?task=:sdks:python:docs On Mon, Jul 15, 2019 at 11:16 PM Yifan Zou <yifan...@google.com> wrote: > We saw similar timeouts of the python precommit and it usually break the > Jenkins build workers. I've run the precommit manually several times. It > sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build > VMs eventually ran out of memory (104G memory available in total) then > disconnected. Not sure what happened during that job. > > > > On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <ttanay...@gmail.com> > wrote: > >> Yes. It passed on the second attempt. >> >> But, I'm yet to figure out why it hangs for ~1.5 hours. >> >> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <rakeshku...@lyft.com> >> wrote: >> >>> >>> >>> Even I am running into the same issue. Though my test passed but >>> <https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow >>> the task didn't terminate, eventually the task was aborted. I have already >>> tried a couple of times to retrigger tye python precommit but it failed >>> every time. >>> >>> @Tanay did it pass it for you? >>> >>> >>> >>> >>> >>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <ttanay...@gmail.com> >>> wrote: >>> >>>> Thank You Valentyn! >>>> >>>> I'll retest it. >>>> Hopefully, it's a transient issue. >>>> >>>> Regards, >>>> - Tanay Tummalapalli >>>> >>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>>>> No, we did not reduce the timeout recently. Looking at console logs, >>>>> nothing happened for an hour or so, >>>>> >>>>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :) >>>>> 06:57:50 * >>>>> >>>>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out >>>>> (after 120 minutes). Marking the build as aborted. >>>>> >>>>> >>>>> However, we can also see in the logs that py36-cython suite never >>>>> started, not sure way. I assume gradle waited for this suite to finish. >>>>> Try "retest this please", hopefully this is a transient gradle issue. >>>>> I did not observe it before. >>>>> >>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli < >>>>> ttanay...@gmail.com> wrote: >>>>> >>>>>> Hi Udi, >>>>>> >>>>>> I rebased another PR[1] onto the fix mentioned above. The lint error >>>>>> is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing >>>>>> because of a timeout at 120 minutes[2]. >>>>>> The log says "Build timed out (after 120 minutes). Marking the build >>>>>> as aborted." >>>>>> Another PR's Python PreCommit job aborted with the same error[3]. >>>>>> >>>>>> I found this issue - "[BEAM-3040] Python precommit timed out after >>>>>> 150 minutes"[4]. >>>>>> Was the timeout reduced recently? >>>>>> >>>>>> Regards, >>>>>> - Tanay Tummalapalli >>>>>> >>>>>> [1] https://github.com/apache/beam/pull/8871 >>>>>> [2] >>>>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull >>>>>> >>>>>> [3] https://github.com/apache/beam/pull/9050 >>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040 >>>>>> >>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote: >>>>>> >>>>>>> This is due to >>>>>>> https://github.com/apache/beam/pull/8969 >>>>>>> and >>>>>>> https://github.com/apache/beam/pull/8934 >>>>>>> being merged today. >>>>>>> >>>>>>> Fix is here: https://github.com/apache/beam/pull/9044 >>>>>>> >>>>>>
