[
https://issues.apache.org/jira/browse/BEAM-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930762#comment-16930762
]
Chad Dombrova commented on BEAM-8213:
-------------------------------------
Here's a hierarchy of the relevant pre-commit tasks:
{code}
pythonPreCommit()
dependsOn ":sdks:python:test-suites:tox:py2:preCommitPy2"
dependsOn "docs"
dependsOn "testPy2Cython"
dependsOn "testPython2"
dependsOn "testPy2Gcp"
dependsOn "lint"
dependsOn ":sdks:python:test-suites:tox:py35:preCommitPy35"
dependsOn "testPython35"
dependsOn "testPy35Gcp"
dependsOn "testPy35Cython"
dependsOn "lint"
dependsOn ":sdks:python:test-suites:tox:py36:preCommitPy36"
dependsOn "testPython36"
dependsOn "testPy36Gcp"
dependsOn "testPy36Cython"
dependsOn ":sdks:python:test-suites:tox:py37:preCommitPy37"
dependsOn "testPython37"
dependsOn "testPy37Gcp"
dependsOn "testPy37Cython"
dependsOn ":sdks:python:test-suites:dataflow:py2:preCommitIT"
dependsOn ":sdks:python:test-suites:dataflow:py37:preCommitIT"
{code}
I think an easy place to start is to split up pythonPreCommit into 6 separate
jobs
- preCommitPy2
- preCommitPy35
- preCommitPy36
- preCommitPy37
- preCommitITPy2
- preCommitITPy37
Then we pull out all the lint and doc tasks into another task:
- preCommitPyLint
It's a lot of jobs, but each one will finish faster, and the output for each
will be much more clear.
> Run and report python tox tasks separately within Jenkins
> ---------------------------------------------------------
>
> Key: BEAM-8213
> URL: https://issues.apache.org/jira/browse/BEAM-8213
> Project: Beam
> Issue Type: Improvement
> Components: build-system
> Reporter: Chad Dombrova
> Priority: Major
>
> As a python developer, the speed and comprehensibility of the jenkins
> PreCommit job could be greatly improved.
> Here are some of the problems
> - when a lint job fails, it's not reported in the test results summary, so
> even though the job is marked as failed, I see "Test Result (no failures)"
> which is quite confusing
> - I have to wait for over an hour to discover the lint failed, which takes
> about a minute to run on its own
> - The logs are a jumbled mess of all the different tasks running on top of
> each other
> - The test results give no indication of which version of python they use. I
> click on Test results, then the test module, then the test class, then I see
> 4 tests named the same thing. I assume that the first is python 2.7, the
> second is 3.5 and so on. It takes 5 clicks and then reading the log output
> to know which version of python a single error pertains to, then I need to
> repeat for each failure. This makes it very difficult to discover problems,
> and deduce that they may have something to do with python version mismatches.
> I believe the solution to this is to split up the single monolithic python
> PreCommit job into sub-jobs (possibly using a pipeline with steps). This
> would give us the following benefits:
> - sub job results should become available as they finish, so for example,
> lint results should be available very early on
> - sub job results will be reported separately, and there will be a job for
> each py2, py35, py36 and so on, so it will be clear when an error is related
> to a particular python version
> - sub jobs without reports, like docs and lint, will have their own failure
> status and logs, so when they fail it will be more obvious what went wrong.
> I'm happy to help out once I get some feedback on the desired way forward.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
