We do retry certain inherently flaky tests, for example, see[1]. This practice should be used with caution, see discussion [2].
However retrying an individual test would not avoid the flake that Alex brought up in this thread, we'd have to retry setupVirtualEnv task that is executed once per suite of tests. Retrying just that task is different from retrying the whole suite. [1] https://github.com/apache/beam/blob/516cdb6401d9fb7adb004de472771fb1fb3a92af/sdks/python/apache_beam/runners/worker/statesampler_test.py#L41, this was discussed [2] https://lists.apache.org/thread.html/16060fd7f4d408857a5e4a2598cc96ebac0f744b65bf4699001350af@%3Cdev.beam.apache.org%3E discussed On Fri, Apr 26, 2019 at 3:30 PM Udi Meiri <eh...@google.com> wrote: > Alex, I changed my mind: I'm okay retrying single tests, just not entire > suites of tests (e.g. if precommits take an hour, retrying the run takes up > an additional hour on the Jenkins machine). > This is more of an issue in Python, where gradle does not (currently) have > insight into which tests failed and how to retry just them. > > > > On Fri, Apr 26, 2019 at 2:17 PM Alex Amato <ajam...@google.com> wrote: > >> @Udi Meiri <eh...@google.com>, Is this true if the specific tests are >> rerun? I don't think we should rerun all tests. >> >> On Fri, Apr 26, 2019 at 12:11 PM Valentyn Tymofieiev <valen...@google.com> >> wrote: >> >>> Preinstalling dependencies may affect the dependency resolution, and we >>> may end up testing a different configuration than a user would have after >>> installing beam into a clean environment. >>> >>> I do think pip uses cache, unless one specifies "--no-cache-dir". By >>> default the cache is ~/.cache/pip. Looking up the log message in OP, we can >>> see several "Using cached..." log entries. Not sure why futures was not >>> fetched from cache or PyPi. Perhaps it is also a pip flake. >>> >>> I would be against wiping flakes under the rug by rerunning the whole >>> suite after an error, but re-rerunning parts of the test environment set >>> up, that are prone to environmental flakes, such as setupVirtualEnv seems >>> reasonable. I agree with Udi that care should be taken to not overload >>> Jenkins (e.g. retries should be limited) >>> >>
