I took another look at this and precommit ITs are already running in parallel, albeit in the same suite. However it appears Python precommits became slower, especially Python 2 precommits [35 min per suite x 3 suites], see [1]. Not sure yet what caused the increase, but precommits used to be faster. Perhaps we have added a slow test or a lot of new tests.
[1] https://scans.gradle.com/s/jvcw5fpqfc64k/timeline?task=ancsbov425524 On Thu, Oct 24, 2019 at 4:53 PM Ahmet Altay <al...@google.com> wrote: > Ack. Separating precommit ITs to a different suite sounds good. Anyone is > interested in doing that? > > On Thu, Oct 24, 2019 at 2:41 PM Valentyn Tymofieiev <valen...@google.com> > wrote: > >> This should not increase the queue time substantially, since precommit >> ITs are running *sequentially* with precommit tests, unlike multiple >> precommit tests which run in parallel to each other. >> >> The precommit ITs we run are batch and streaming wordcount tests on Py2 >> and one Py3 version, so it's not a lot of tests. >> >> On Thu, Oct 24, 2019 at 1:07 PM Ahmet Altay <al...@google.com> wrote: >> >>> +1 to separating ITs from precommit. Downside would be, when Chad tried >>> to do something similar [1] it was noted that the total time to run all >>> precommit tests would increase and also potentially increase the queue time. >>> >>> Another alternative, we could run a smaller set of IT tests in >>> precommits and run the whole suite as part of post commit tests. >>> >>> [1] https://github.com/apache/beam/pull/9642 >>> >>> On Thu, Oct 24, 2019 at 12:15 PM Valentyn Tymofieiev < >>> valen...@google.com> wrote: >>> >>>> One improvement could be move to Precommit IT tests into a separate >>>> suite from precommit tests, and run it in parallel. >>>> >>>> On Thu, Oct 24, 2019 at 11:41 AM Brian Hulette <bhule...@google.com> >>>> wrote: >>>> >>>>> Python Precommits are taking quite a while now [1]. Just visually it >>>>> looks like the average length is 1.5h or so, but it spikes up to 2h. I've >>>>> had several precommit runs get aborted due to the 2 hour limit. >>>>> >>>>> It looks like there was a spike up above 1h back on 9/6 and the >>>>> duration has been steadily rising since then. Is there anything we can do >>>>> about this? >>>>> >>>>> Brian >>>>> >>>>> [1] >>>>> http://104.154.241.245/d/_TNndF2iz/pre-commit-test-latency?orgId=1&from=now-90d&to=now&fullscreen&panelId=4 >>>>> >>>>
