> Do we have good pypi caching? Building Python SDK harness containers takes 2 mins each (times 4, the number of versions) on my machine, even if nothing has changed. But we're already paying that cost, so I don't think splitting the jobs should make it any worse. (https://issues.apache.org/jira/browse/BEAM-8277 if anyone has any ideas)
Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Wed, Sep 25, 2019 at 11:21 AM Pablo Estrada <pabl...@google.com> wrote: > Thanks Chad, and thank you for notifying on the dev list. > > On Wed, Sep 25, 2019 at 10:59 AM Kenneth Knowles <k...@apache.org> wrote: > >> Nice. >> >> Do we have good pypi caching? If not this could add a lot of overhead to >> our already-backed-up CI queue. (btw I still think your change is good, and >> just makes proper caching more important) >> >> Kenn >> >> On Tue, Sep 24, 2019 at 9:55 PM Chad Dombrova <chad...@gmail.com> wrote: >> >>> Hi all, >>> I'm working to make the CI experience with python a bit better, and my >>> current initiative is splitting up the giant Python PreCommit job into 5 >>> separate jobs into separate jobs for Lint, Py2, Py3.5, Py3.6, and Py3.7. >>> >>> Around 11am Pacific time tomorrow I'm going to initiate the seed jobs, >>> at which point all PRs will start to run the new precommit jobs. It's a >>> bit of a chicken-and-egg scenario with testing this, so there could be >>> issues that pop up after the seed jobs are created, but I'll be working to >>> resolve those issues as quickly as possible. >>> >>> If you run into problems because of this change, please let me know on >>> the github PR. >>> >>> Here's the PR: https://github.com/apache/beam/pull/9642 >>> Here's the Jira: https://issues.apache.org/jira/browse/BEAM-8213# >>> >>> The upshot is that after this is done you'll get better feedback on >>> python test failures! >>> >>> Let me know if you have any concerns. >>> >>> thanks, >>> chad >>> >>>
