By the way, the outcome on this was that splitting the python precommit job into one job per python version resulted in increasing the total test completion time by 66%, which is obviously not good. This is because we are using Gradle to run the python tests tasks in parallel (the jenkins VMs have 16 cores each, utilized across 2 slots, IIRC), but after the split there were only 1-2 gradle tasks per test. Since the python test runner, nose, is currently not using parallel execution, there were not enough concurrent tasks to make proper use of the VM's CPUs.
tl;dr I'm going to create a followup PR to split out just the Lint job (same as we have Spotless for Java). This is our best ROI for now. -chad On Fri, Sep 27, 2019 at 3:27 PM Kyle Weaver <kcwea...@google.com> wrote: > > Do we have good pypi caching? > > Building Python SDK harness containers takes 2 mins each (times 4, the > number of versions) on my machine, even if nothing has changed. But we're > already paying that cost, so I don't think splitting the jobs should make > it any worse. (https://issues.apache.org/jira/browse/BEAM-8277 if anyone > has any ideas) > > Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com > > > On Wed, Sep 25, 2019 at 11:21 AM Pablo Estrada <pabl...@google.com> wrote: > >> Thanks Chad, and thank you for notifying on the dev list. >> >> On Wed, Sep 25, 2019 at 10:59 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> Nice. >>> >>> Do we have good pypi caching? If not this could add a lot of overhead to >>> our already-backed-up CI queue. (btw I still think your change is good, and >>> just makes proper caching more important) >>> >>> Kenn >>> >>> On Tue, Sep 24, 2019 at 9:55 PM Chad Dombrova <chad...@gmail.com> wrote: >>> >>>> Hi all, >>>> I'm working to make the CI experience with python a bit better, and my >>>> current initiative is splitting up the giant Python PreCommit job into 5 >>>> separate jobs into separate jobs for Lint, Py2, Py3.5, Py3.6, and Py3.7. >>>> >>>> Around 11am Pacific time tomorrow I'm going to initiate the seed jobs, >>>> at which point all PRs will start to run the new precommit jobs. It's a >>>> bit of a chicken-and-egg scenario with testing this, so there could be >>>> issues that pop up after the seed jobs are created, but I'll be working to >>>> resolve those issues as quickly as possible. >>>> >>>> If you run into problems because of this change, please let me know on >>>> the github PR. >>>> >>>> Here's the PR: https://github.com/apache/beam/pull/9642 >>>> Here's the Jira: https://issues.apache.org/jira/browse/BEAM-8213# >>>> >>>> The upshot is that after this is done you'll get better feedback on >>>> python test failures! >>>> >>>> Let me know if you have any concerns. >>>> >>>> thanks, >>>> chad >>>> >>>>
