+1 to separating ITs from precommit. Downside would be, when Chad tried to do something similar [1] it was noted that the total time to run all precommit tests would increase and also potentially increase the queue time.
Another alternative, we could run a smaller set of IT tests in precommits and run the whole suite as part of post commit tests. [1] https://github.com/apache/beam/pull/9642 On Thu, Oct 24, 2019 at 12:15 PM Valentyn Tymofieiev <valen...@google.com> wrote: > One improvement could be move to Precommit IT tests into a separate suite > from precommit tests, and run it in parallel. > > On Thu, Oct 24, 2019 at 11:41 AM Brian Hulette <bhule...@google.com> > wrote: > >> Python Precommits are taking quite a while now [1]. Just visually it >> looks like the average length is 1.5h or so, but it spikes up to 2h. I've >> had several precommit runs get aborted due to the 2 hour limit. >> >> It looks like there was a spike up above 1h back on 9/6 and the duration >> has been steadily rising since then. Is there anything we can do about this? >> >> Brian >> >> [1] >> http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1&from=now-90d&to=now&fullscreen&panelId=4 >> >
