Alex, I changed my mind: I'm okay retrying single tests, just not entire suites of tests (e.g. if precommits take an hour, retrying the run takes up an additional hour on the Jenkins machine). This is more of an issue in Python, where gradle does not (currently) have insight into which tests failed and how to retry just them.
On Fri, Apr 26, 2019 at 2:17 PM Alex Amato <[email protected]> wrote: > @Udi Meiri <[email protected]>, Is this true if the specific tests are > rerun? I don't think we should rerun all tests. > > On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev <[email protected]> > wrote: > >> Preinstalling dependencies may affect the dependency resolution, and we >> may end up testing a different configuration than a user would have after >> installing beam into a clean environment. >> >> I do think pip uses cache, unless one specifies "--no-cache-dir". By >> default the cache is ~/.cache/pip. Looking up the log message in OP, we can >> see several "Using cached..." log entries. Not sure why futures was not >> fetched from cache or PyPi. Perhaps it is also a pip flake. >> >> I would be against wiping flakes under the rug by rerunning the whole >> suite after an error, but re-rerunning parts of the test environment set >> up, that are prone to environmental flakes, such as setupVirtualEnv seems >> reasonable. I agree with Udi that care should be taken to not overload >> Jenkins (e.g. retries should be limited) >> >
smime.p7s
Description: S/MIME Cryptographic Signature
